3

Is it possible to automate the saving of a webpage (using the .webarchive format) using either Automator (in a background process) or using Terminal?

2

Downloading & saving as webarchive

A command line tool named webarchiver will download URLs and save them to .webarchive format. You can install this tool via MacPorts (alas, not homebrew!) or compile it with XCode. I am a XCode dummy, but succeeded with instructions found here.

How to operate:

webarchiver 0.5
Usage: webarchiver -url URL -output FILE 
Example: webarchiver -url http://www.google.com -output google.webarchive
-url    http:// or path to local file
-output File to write webarchive to

Nice file names

This lenghty one-liner for terminal allows you to configure the desired URL and will download a YYYY-MM-DD-prefixed webarchive file:

URL="www.nytimes.com"; ./webarchiver -url "http://$URL" -output "/Users/<your username>/Desktop/$(date +"%Y-%m-%d-$URL.webarchive")"

This will save a webarchive to your Desktop:

2014-02-10-www.nytimes.com.webarchive

If you are not sure what <your username> is, enter whoami in Terminal.app (and press enter, of course).

Cron

I would rather use launchd, as "the use of cron on OS X is discouraged". There is a nice launchd editor named Lingon. Have fun!

4
  • Brilliant find! Thank you so much! I'm planning on automating this with Automator (running each website archive within the 'Run Shell Script' option. Thing is, is there any way to select where the output will be? (Right now the default area seems to be ~/user instead of ~/user/desktop) Thanks. – JFW Jul 26 '14 at 14:02
  • You are welcome! Please see the updated answer. – myhd Jul 28 '14 at 12:40
  • 1
    Thanks again for finding it. I've marked your answer as the one that has solved the question (after the question was first asked 275 days ago). I've been using webarchiver since and it's been incredible. I've not had the guts to automate this, so what I did was just put the line of code into Automator, and output that as a clickable file on the desktop. It's been pretty fantastic so far. Unfortunately the file sizes of the output are pretty big (~3MB, as opposed to the usual 0.5MB), but the fact that it is self-contained means that it's a lot more portable. – JFW Oct 9 '14 at 21:40
  • Just a heads up: webarchiver has been made available through homebrew as well (have a look: brew info webarchiver)! – ƘɌỈSƬƠƑ Aug 8 '17 at 7:58
2

Yes is the simple answer with either.

I am on my iPad at the mo. But you can use unix command curl to download the webpage to and pipe it to the unix command textutil which can output it to a webarchive file.

If I get a chance I will post an example.


Here is a small example (quick ) of what I was thinking. Written in Applescript running do shell script commands.

  property agent : "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3"

property outPutFormat : "rtf"
property saveDIR : "/Users/USERNAME/Desktop/"
property fileName : "test2"

set theData to do shell script "curl " & " -A" & space & quoted form of agent & space & "http://weather.yahoo.com/france/auvergne/france-29332634/" as string


do shell script "echo " & quoted form of theData & "|textutil -format html -convert" & space & outPutFormat & space & "-stdin -output " & space & saveDIR & fileName & "." & outPutFormat

Although this works. I am not very happy with the results. This is due to curl and textutil only processing the html code but not resources.

So am working on something else that will save a webArchive in a much better way. 90% there but will take a little longer for me to write

9
  • I really would appreciate your following up on this. Would it also be possible for you to show how this can be done in the context of a cron(job)? Thanks in advance! – JFW Jan 7 '14 at 14:13
  • what is the webpage. Also in my haste to help you I forgot doing it this way does work. But only for the front facing page. But some resource file etc. will not be included so the page will look wonky. Still looking at way to resolve that. But can you supply more deatil on why and what you want from the page. – markhunte Jan 7 '14 at 16:37
  • Right. I was looking to capture pages of news sites like ft.com, nytimes.com, ..., so I am able to see the headline trends throughout the months and years. My memory isn't great so this would be a great aid. – JFW Jan 8 '14 at 7:38
  • So rather than a webarchive would a text file be better. Maybe formatted. I think it would work better – markhunte Jan 8 '14 at 10:38
  • Yeah, wouldn't mind having a formatted text file. Would it be possible for you to post an example? – JFW Jan 9 '14 at 19:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .