Take the 2-minute tour ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.

Is there a way to use curl to download non-sequential images with a wildcard? I want to download all the panoramas I upload to my account at 360 Panorama to a local folder so I don't have to do this manually each time.

The images files follow the pattern of: http://occipital.com/images/viewer/XYZ_flat.jpg with the XYZ being random. It also seems from that URL that I need to be logged in or seen as logged in to the site.

share|improve this question
    
I'm not too familiar with curl, as I only just discovered it when looking for a way to accomplish this task and curl (or wget) seemed the solution. –  George C Aug 11 '12 at 3:34
    
See CURL to download a directory on Super User. –  Arjan Aug 11 '12 at 7:42
    
Can you add some specific (real) URLs to your question? Maybe we can find a pattern there which would be possible to guess when using curl. –  patrix Aug 11 '12 at 8:18
    
@patrix here's an actual URL: occipital.com/images/viewer/qSJGuD_flat.jpg — seems they don't offer a direct path that's tied to the username. Here's the link to my user account page when I am not logged in: occipital.com/user/01ca-255/george-coghill –  George C Aug 12 '12 at 13:23
    
@Arjan I am trying to just download the newer images as they are posted, not the entire directory, and not just one time. Trying to avoid having to go to my account and manually download each image every time I upload new panoramas. –  George C Aug 12 '12 at 13:28
show 3 more comments

3 Answers 3

up vote 4 down vote accepted

You can use this bash code for the actual URL you provided in your comment.

for i in $(for j in {1..8}; do
    curl http://occipital.com/user/01ca-255/george-coghill/$j/;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  curl -L -o "${i##*/}" "$i";
done

Feel free to write this command as one line; all line breaks were only added to increase legibility. You may copy or remove them, whatever you like.

What this code does:

  1. Iterate over the 8 pages of your account gallery
  2. Extract the image names from the preview images
  3. Fetch all full-sized images using this list of names

If you want to only download files which don't exist, and don't know the number of gallery pages up front, you can adjust the code to this:

for i in $(
  j=1;
  while curl --max-redirs 0 -L -f \
      http://occipital.com/user/01ca-255/george-coghill/$((j++))/; do
    :;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  [[ -f "${i##*/}" ]] || curl -L -o "${i##*/}" "$i";
done

The first code now increments the page number until a redirect or error occurs. The server will redirect you to the last existing page if you try to visit a page after the existing ones. The [[ -f … ]] || part will execute curl only if the corresponding file does not exist yet.

share|improve this answer
    
This did the trick, I have to say it's way over my head but I really appreciate the help! How would I tweak the code to specify the download directory? –  George C Nov 3 '12 at 21:11
    
@GeorgeC: "${i##*/}" is the name of the file on your local machine; you may prepend a directory name to that, e.g. "some/dir/${i##*/}". Or you simply cd into the desired directory first. –  MvG Nov 4 '12 at 20:28
    
Thank you for the additional code. –  George C Nov 14 '12 at 4:11
add comment

So you want to download pictures from http://occipital.com/images/viewer/*_flat.jpg?

This is not possible. You can't treat viewer as a folder. The web is a service that provides you with things when you ask for it. It doesn't have a list of all of the files. Unless you try every file from AAA to ZZZ, this is not possible without a list.

share|improve this answer
    
That's what I was thinking was going to be the problem. But let's say there was a URL that this could work on—how would one use curl with non-sequential URLs to automatically check and download new files? Or is it just not possible without a sequential URL pattern? –  George C Aug 12 '12 at 13:25
add comment

While you can't use a true wildcard you can specify parts within braces ie. curl mysite.{alpha,beta,gamma}.com or number ranges mysite.com/files[0001-0100].txt

See the curl man page for more info

http://curl.haxx.se/docs/manpage.html

So for your files that are random you might do mages/viewer/[000-999]_flat.jpg You'd end up with lots of file not found but it should get all the files that are there.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.