Take the 2-minute tour ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.

I want to scrape all the pages and subpages of a web site (by scrape, I mean save all the online content to local HTML files).

What is the best utility to crawl all the pages? Ideally, I would like to specify how many layers deep to scrape.

share|improve this question
add comment

2 Answers

up vote 16 down vote accepted

You have two options:

You can use the wget command-line utility like so:

wget -rl 10

Replace the 10 with the number of levels you would like to recurse down into.

Or, you can use SiteSucker. A GUI application that recursively downloads websites. You can also specify how far down to recurse with SiteSucker.

share|improve this answer
1  
SiteSucker is great. –  neoneye Feb 21 '11 at 0:23
    
+1 for SiteSucker. Great tool. Does the job perfectly. –  Mahm00d Apr 13 '12 at 13:31
add comment

DeepVacuum

GUI for wget, $15.

cocoawget

(free)

share|improve this answer
    
Deepvacuum makes me think of Dire Straits: "Money for nothing" :D –  Pitto Aug 2 '11 at 16:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.