I want to scrape all the pages and subpages of a web site (by scrape, I mean save all the online content to local HTML files).
What is the best utility to crawl all the pages? Ideally, I would like to specify how many layers deep to scrape.
Ask Different is a question and answer site for power users of Apple hardware and software. It only takes a minute to sign up.Sign up to join this community
You have two options:
You can use the
wget command-line utility like so:
wget -rl 10
10 with the number of levels you would like to recurse down into.
Or, you can use SiteSucker. A GUI application that recursively downloads websites. You can also specify how far down to recurse with SiteSucker.