I want to scrape all the pages and subpages of a web site (by scrape, I mean save all the online content to local HTML files).

What is the best utility to crawl all the pages? Ideally, I would like to specify how many layers deep to scrape.

2 Answers 2


You have two options:

You can use the wget command-line utility like so:

wget -rl 10

Replace the 10 with the number of levels you would like to recurse down into.

Or, you can use SiteSucker. A GUI application that recursively downloads websites. You can also specify how far down to recurse with SiteSucker.

  • 1
    SiteSucker is great.
    – neoneye
    Commented Feb 21, 2011 at 0:23


GUI for wget, $15.



  • Deepvacuum makes me think of Dire Straits: "Money for nothing" :D
    – Pitto
    Commented Aug 2, 2011 at 16:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .