3

I have a .txt file with URL's on a separate line.

http://www.apple.com
http://www.google.com
http://www.reuters.com

I would like to download these webpages as a page source (a .html file) so I can open them offline in my webbrowser.

I tried to do this with automator, but it doesn't seem to work properly. My Automator workflow consists of 2 steps: "Extract data from text" and "Download URLs". I've looked on the web for already existing solutions, but I haven't found anything I understand.

Can someone create a program with Automator or Applescript (or something else) so I can download these webpages?

The program should work as follows:

  1. The program reads a .txt file with URLs on a separate line. (The filetype doesn't really matter, as long as it is simple for your program: .csv, .pages, .doc, ...)
  2. The program reads each URL in the file and downloads it as a .html file in order that the webpages are accessible without an internet connection.
  3. All the .html files should be saved in a folder, preferably a folder on my desktop with the name "Downloaded html files"

Thanks in advance, If there are any questions, don't hesitate to ask. I will respond asap.

Improve this question
2
  • 1
    What is a "map" (item 3)? Do you just want the page the URL points to OR the page including all images etc. OR all subpages as well (aka the whole site the URL points to)?
    – nohillside ♦
    Nov 4, 2015 at 14:33
  • I want a full copy of the webpage. I don't want to save the whole site with all subpages. just the page the URL points to with images etc. Each .html file should be saved in a folder on my desktop. With 'map', i mean folder (translation mistake). Thanks for your response!
    – OlivierVR
    Nov 4, 2015 at 14:39

2 Answers 2

Reset to default
1

The command line is easier (https://superuser.com/a/168625):

Use wget -i urls.txt

An alternate way, using this answer, using Extract URLs from text, then Download URLs. Save it as a service. Select all URLs, right click > Services > Download URLs

Improve this answer
3
  • wget is not installed on OS X.
    – 3871968
    Nov 4, 2015 at 19:24
  • @jBot-42 you can install it easily using homebrew.
    – At0mic
    Nov 4, 2015 at 21:47
  • Concerning your alternate way: I made the automator program (saved as a service). While using this program, the URLs are downloaded as html-files and saved in my download folder. Instead, I want the files saved in a new folder on my desktop. How do I automatically create a new folder on my desktop (named "Downloaded html files") where the files are downloaded? Can someone adapt the automator program that I just made so it fits all my needs? Thanks in advance!
    – OlivierVR
    Nov 4, 2015 at 21:59
1

To use the following method, you will need to install wget.

Make a file with the extension .sh in the same directory as your file containing the links and add this text to it:

mkdir ~/Desktop/download

while read line; do wget -E -H --directory-prefix=/Users/username/Desktop/download -k -p $line; done < file.txt

cd ~/Desktop/download

Make sure to edit the script and change username to your username

This reads file.txt for the URLs and runs the wget command multiple times with all the links one-by-one and saves them to a folder named download on your desktop.

Run it in terminal with ./script.sh or whatever you named it. If it shows Permission Denied, then run chmod a+x script.sh

Improve this answer
6
  • 2
    wget is not part of OS X by default
    – nohillside ♦
    Nov 4, 2015 at 14:31
  • @patrix I put the mv command into the shell script, because when I tested it, wget made a directory called -directory-prefix=~ in the directory with my script, which contained Desktop and download.
    – John K
    Nov 4, 2015 at 14:38
  • 1
    Yes, because you had both -P and ----directory-prefix, which do the same thing, so you only need one or the other. I've fixed the wget options.
    – nohillside ♦
    Nov 4, 2015 at 14:41
  • Could this be done with curl?
    – amanthethy
    Nov 4, 2015 at 15:01
  • @amanthethy Yes. Just change the wget command after the do in the loop into curl. I dont know the optio s though
    – John K
    Nov 4, 2015 at 15:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .