0

QUESTION: i need a working method to be used in applescript to get the right source of the page, WITHOUT loading the page.

sample link: https://www.idealista.it/immobile/16679597/

result: wrong html, it talks about authentication.

INITIAL CODE (always present for all the tries below):

set MyUser to username@dom.com
set MyPass to password
set UrlOfPage to "https://www.idealista.it/immobile/16679597/"

TRIES (all of the tries below are given in this webpage https://ec.haxx.se/http-auth.html):

  • works but need the page to be loaded in Safari

    tell front document of application "Safari" to set StrHtml to (get source) as string
    
  • returns wrong html

    set StrHtml to (do shell script "curl --user " & MyUser & ":" & MyPass & " " & UrlOfPage)
    
    set StrHtml to (do shell script "curl --anyauth --user " & MyUser & ":" & MyPass & " " & UrlOfPage)
    
    set StrHtml to (do shell script "curl --digest --user " & MyUser & ":" & MyPass & " " & cellurl)
    
    set StrHtml to (do shell script "curl --negotiate --user " & MyUser & ":" & MyPass & " " & cellurl)
    
    set StrHtml to (do shell script "curl --ntlm --user " & MyUser & ":" & MyPass & " " & cellurl)
    
  • doesn't work: unknown token

    set StrHtml to (do shell script "curl --proxy-anyauth --proxy-user " & MyUser & ":" & MyPass & " https://www.idealista.it/immobile/16679597/ \ --proxy https://proxy.idealista.it/immobile/16679597:80/")
    

could somebody help me now please?

  • Have you been able to get the right HTML using just curl in the Terminal? Adding AppleScript at this stage may be complicating the problem. Considering perfecting the curl command, then adding AppleScript to parse the result. – Graham Miln Jul 16 at 14:28
  • @GrahamMiln yeah, i tried but unfortunately it gives exactly the same results as using curl from shell script in applescript and the same results as using a simple curl without authentication. i'm stuck at this point. – Francesco Sigismondi Jul 16 at 15:20
  • 1
    @nohillside i'd like to grab the string just below "Statistiche", at the bottom of the page, right below the position on the map. The exact text i'd like to grab is "Annuncio aggiornato il 15 luglio" which in english it sounds something like "Updated on July, 15th". – Francesco Sigismondi Jul 16 at 15:25
  • 1
    You actually don't need to login, I get the same content without a login. Also, what actually happens is that the webserver recognizes your access as being from a script, and displays a dedicated page then (to verify this, set the User-Agent string to "curl" in Safari and reload). – nohillside Jul 16 at 16:58
  • 1
    I would start with the tools recommended in the answer below. – nohillside Jul 18 at 19:57
1

Dedicated Tools

Given the problems encountered with curl and AppleScript, consider using an alternative dedicated tool such as Beautiful Soup. See How To Scrape Web Pages with Beautiful Soup and Python 3 for a comprehensive introduction.

Alternatively, there are numerous tools other that can help, see Web scraping software on Wikipedia. Many of these tools are free, open-source, and can be called from the command line.

I have previously used Web::Scraper for extracting property listings.

  • Does it have a cost? Does it grab part of html and give it to applescript? Does it work on High Sierra OS X? – Francesco Sigismondi Jul 16 at 16:13
  • Beautiful Soup is open source and free. Many of the other tools are also free and can be used from the command line and thus interacted with using AppleScript. – Graham Miln Jul 16 at 16:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .