How to add a timer between wget html downloads?

Question

I found this very useful script that does the job for what I'm looking to achieve.

It's been taken from this post: Batch download URLs from a .txt file

To use the following method, you will need to install wget. Make a file with the extension .sh in the same directory as your file containing the links and add this text to it:

mkdir ~/Desktop/download

while read line; do wget -E -H --directory-prefix=/Users/username/Desktop/download -k -p $line; done < file.txt

cd ~/Desktop/download

Make sure to edit the script and change username to your username This reads file.txt for the URLs and runs the wget command multiple times with all the links one-by-one and saves them to a folder named download on your desktop.

I just need to make a small edit to make sure that between one request and another the script waits 5 minutes.

Could you please tell me how to edit it?

Answer 1

wget can read URLs directly from a file (-i file), and knows how to pause between downloads (-w seconds), so you don’t actually need a loop at all. Also, wget downloads into the current directory by default so you don't really need --directory-prefix either.

Just run

mkdir -p ~/Desktop/download
cd ~/Desktop/download
wget -E -H -k -p -w $((5*60)) -i file.txt

PS: You might also want to add --random-wait to have wget wait between 0.5 and 1.5 times the number of seconds you've specified. This might help to avoid problems with sites who take extreme steps to detect and block wget requests.