1

I am using python requests to get itunes URL responses.

for some links it works and for others it doesn't, for example:

working - https://itunes.apple.com/SA/app/id944846798?ls=1&mt=8

not working - https://itunes.apple.com/app/id1029783189?mt=8 (getting a 404 error)

I noticed that the difference between the 2 links is that if i paste the working one in the browser, it opens it correctly, and if i paste the non-working one, it tries to open itunes and get stuck.

I tried few user-agents, for example:

"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3188.0 Safari/537.36"

Since spoofing a user agent isn't enough for my python tool to appear like a web browser, Is there something else I can try?

  • 1
    If you get a 404 response - that is a valid response to any URL. What's the real problem here? (and be sure to acknowledge that scraping iTunes links programmatically will surely get you blocked and IP banned if you are scraping data from Apple at a high volume. Apple's CDN is quite sophisticated and designed to let people in web browsers access links and block bots and high usage clients that hammer on the site.) – bmike Mar 24 '18 at 17:31
  • I don't scrape at high volume. and i know it is a valid response if the page is not found, but it exists, and i know because i can open it from my phone with Australia proxy (and i call the request with a proxy too) – slashms Mar 24 '18 at 17:47
  • So - you might need to show a little code or turn this into a curl question so others might be able to help. You don't have to edit, but from your comment - you're explaining that this really is about Apple's servers sending a different response to two different web browsers. Your exact code and IP and other items is what's causing your error - you're being blocked in effect. I'l try an edit to make this clear you need a solution to interact with the store from a script and not that you can't retrieve that URL – bmike Mar 24 '18 at 18:21
  • +1 either way if you want to roll back my edit or improve / refine it. Let's see what people have to offer other than - use a web browser to access store links and previews. – bmike Mar 24 '18 at 18:27
1

I found a way to bypass it. It seems that itunes is sending back a 404 response for some applications, especially (from what i've seen) if they have restricted content such as gambling.

the way get around it is instead of sending a request to the main apple store like here

https://itunes.apple.com/app/id1029783189?mt=8

one should send the request to the local appstore. In this example case, the Australian one:

https://itunes.apple.com/au/app/id1029783189

Thanks for everyone who tried to help.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .