Use curl to download images from website using wildcard?

Question

Is there a way to use curl to download non-sequential images with a wildcard? I want to download all the panoramas I upload to my account at 360 Panorama to a local folder so I don't have to do this manually each time.

The images files follow the pattern of: http://occipital.com/images/viewer/XYZ_flat.jpg with the XYZ being random. It also seems from that URL that I need to be logged in or seen as logged in to the site.

Answer 1

You can use this bash code for the actual URL you provided in your comment.

for i in $(for j in {1..8}; do
    curl http://occipital.com/user/01ca-255/george-coghill/$j/;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  curl -L -o "${i##*/}" "$i";
done

Feel free to write this command as one line; all line breaks were only added to increase legibility. You may copy or remove them, whatever you like.

What this code does:

1. Iterate over the 8 pages of your account gallery
2. Extract the image names from the preview images
3. Fetch all full-sized images using this list of names

If you want to only download files which don't exist, and don't know the number of gallery pages up front, you can adjust the code to this:

for i in $(
  j=1;
  while curl --max-redirs 0 -L -f \
      http://occipital.com/user/01ca-255/george-coghill/$((j++))/; do
    :;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  [[ -f "${i##*/}" ]] || curl -L -o "${i##*/}" "$i";
done

The first code now increments the page number until a redirect or error occurs. The server will redirect you to the last existing page if you try to visit a page after the existing ones. The [[ -f … ]] || part will execute curl only if the corresponding file does not exist yet.

Answer 2

So you want to download pictures from http://occipital.com/images/viewer/*_flat.jpg?

This is not possible. You can't treat viewer as a folder. The web is a service that provides you with things when you ask for it. It doesn't have a list of all of the files. Unless you try every file from AAA to ZZZ, this is not possible without a list.

Answer 3

While you can't use a true wildcard you can specify parts within braces ie. curl mysite.{alpha,beta,gamma}.com or number ranges mysite.com/files[0001-0100].txt

See the curl man page for more info

http://curl.haxx.se/docs/manpage.html

So for your files that are random you might do mages/viewer/[000-999]_flat.jpg You'd end up with lots of file not found but it should get all the files that are there.

Answer 4

Like ClarkGoble said, use something like this "brute-force" solution:

    curl http://occipital.com/images/viewer/[1-999:1]_flat.jpg -f -o -image_#1_flat.jpg

The "-f" option will prevent curl from writing output files if it gets a 404 error. So you will get output only for the files found on the server.

-o stands for "output"

#1 matches the replacement pattern, i.e. in the case, the numbers starting from 1, up to 999, with a step of 1.