Ask Different is a question and answer site for power users of Apple hardware and software. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to scrape all the pages and subpages of a web site (by scrape, I mean save all the online content to local HTML files).

What is the best utility to crawl all the pages? Ideally, I would like to specify how many layers deep to scrape.

share|improve this question
up vote 18 down vote accepted

You have two options:

You can use the wget command-line utility like so:

wget -rl 10

Replace the 10 with the number of levels you would like to recurse down into.

Or, you can use SiteSucker. A GUI application that recursively downloads websites. You can also specify how far down to recurse with SiteSucker.

share|improve this answer
SiteSucker is great. – neoneye Feb 21 '11 at 0:23
+1 for SiteSucker. Great tool. Does the job perfectly. – Mahm00d Apr 13 '12 at 13:31


GUI for wget, $15.



share|improve this answer
Deepvacuum makes me think of Dire Straits: "Money for nothing" :D – Pitto Aug 2 '11 at 16:53

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.