0

I am trying to use the command line to extract a specific value from a web page.

The page is from the VS Code marketplace: https://marketplace.visualstudio.com/items?itemName=HCLTechnologies.hclappscancodesweep and I am after the number of installs. I managed to get to a point where I can isolate up to this:

{"statisticName":"install","value":206.0},{"statisticName":"averagerating","value":5.0},{"statisticName":"ratingcount","value":8.0},{"statisticName":"trendingdaily","value":0.0},{"statisticName":"trendingmonthly","value":88.950276243093924},{"statisticName":"trendingweekly","value":28.7292817679558},{"statisticName":"weightedRating","value":4.668447071838485}],"installationTargets":[{"target":"Microsoft.VisualStudio.Code","targetVersion":""}],"deploymentType":0}

I am on a mac and using zsh. The command I'm currently using is:

curl -G "https://marketplace.visualstudio.com/items?itemName=HCLTechnologies.hclappscancodesweep" | grep -Eo {\"statisticName\":\"install\",\"value\":.+\}

The desired output of this command should be: 206.0 which is the number I'm after. If you can trim it down to 206 that's even better.

I have 2 main problems: 1) I can't isolate the match to start right before my number. I tried \K but that doesn't seem to work on the mac. I did not see an option to use groups either.

2) I can't make the matching after my number stop after the first match of a curly bracket }. Keep in mind the number can vary in length, so I want to stop it where the JSON stops. I did actually try using [:digit:] and [0-9] to restrict to the number, but that did not work either and that's why I went down the curly bracket path.

I did see some options out there to work with the JSON part and extract things there, but the environment I'm working with does not have the flexibility of installing other utilities, so I need help to achieve this with a "vanilla" environment.

P.S. I'm working on Catalina 10.15.4

5
  • So turns out piping in sed was the easier way of doing this. The command now looks like this: curl -G "marketplace.visualstudio.com/…" | grep -Eo {\"statisticName\":\"install\",\"value\":.+\} | sed -E 's/.*"install","value":"?([^,"]*)}"?.*/\1/' – sir_k Apr 22 '20 at 12:32
  • 1
    I had the grep to isolate part of my JSON and since I'm not that good with sed, decided to use only a sample that extracts the number I wanted from the grep output. Elegant? No. Does it do the job? Yes. – sir_k Apr 22 '20 at 12:45
  • 1
    FWIW Your command does not work for me, it returns nothing at all; however, this works for me: curl -sG "https://marketplace.visualstudio.com/items?itemName=HCLTechnologies.hclappscancodesweep" | grep -Eom 1 '[0-9]+[ ]installs' | awk '{print $1}' – user3439894 Apr 22 '20 at 13:47
  • Please include your own solution in the question, post it as an answer below instead. And actually it will even work if you omit the grep part :-) – nohillside Apr 22 '20 at 15:31
  • I did realise later on that parsing directly for that JSON would do the trick. Thanks for pointing me that way, learned something new today! – sir_k Apr 22 '20 at 16:29
3

grep isn't well suited for data scrapping, you are better of with sed in such cases.

curl -sG 'https://marketplace.visualstudio.com/items?itemName=HCLTechnologies.hclappscancodesweep' \
    | sed -Ene 's/.*\"install\",\"value\":([0-9\.]+).*/\1/p'

The first part of the substition matches only once in the HTML file so there isn't even a need to select the correct line first.

To get the number without the decimal point use

curl -sG 'https://marketplace.visualstudio.com/items?itemName=HCLTechnologies.hclappscancodesweep' \
    | sed -Ene 's/.*\"install\",\"value\":([0-9]+).*/\1/p'
11
  • FWIW Your command does not work for me, it returns nothing at all; however, this works for me: curl -sG "https://marketplace.visualstudio.com/items?itemName=HCLTechnologies.hclappscancodesweep" | grep -Eom 1 '[0-9]+[ ]installs' | awk '{print $1}' What's interesting is it appears what's returned by curl using the same URL is different for the OP then what's returned for me. – user3439894 Apr 22 '20 at 15:34
  • @user3439894 Interesting. I just copy/pasted the commands form my answer back into Terminal, they still work here -) – nohillside Apr 22 '20 at 15:36
  • @user3439894 Location-depending content on the web page maybe? Both the OP and I are connecting from Europe, maybe the page looks different in the States. – nohillside Apr 22 '20 at 15:38
  • I'm sure it does as I know you wouldn't post something that doesn't work, but the only explanation I have is I'm not getting back the same content as you or the OP from the curl cmd for the same URL. I wonder why that is? – user3439894 Apr 22 '20 at 15:38
  • 1
    This is the first time I went to that URL, so no cache on my system! We are getting served two different pages from the same URL and it's beyond our control. – user3439894 Apr 22 '20 at 17:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .