Free OS X utility to scrape web sites?

Question

I want to scrape all the pages and subpages of a web site (by scrape, I mean save all the online content to local HTML files).

What is the best utility to crawl all the pages? Ideally, I would like to specify how many layers deep to scrape.

Mark Szymanski · Accepted Answer · 2011-02-20 19:19:41Z

up vote 14 down vote accepted

You have two options:

You can use the wget command-line utility like so:

wget -rl 10

Replace the 10 with the number of levels you would like to recurse down into.

Or, you can use SiteSucker. A GUI application that recursively downloads websites. You can also specify how far down to recurse with SiteSucker.

answered Feb 20 '11 at 19:19

Mark Szymanski
875415

1

SiteSucker is great. – neoneye Feb 21 '11 at 0:23

+1 for SiteSucker. Great tool. Does the job perfectly. – Mahm00d Apr 13 '12 at 13:31

Lauri Ranta · Answer 2 · 2011-08-02 15:52:57Z

up vote 0 down vote

GUI for wget, $15.

(free)

answered Aug 2 '11 at 15:52

Lauri Ranta
18.2k22168

Deepvacuum makes me think of Dire Straits: "Money for nothing" :D – Pitto Aug 2 '11 at 16:53

2 Answers