6

I'd like to extra the top-level domain and the second-level domain from a URL like "https://apple.stackexchange.com/questions/ask"

Example URL with desired result below. 

https://apple.stackexchange.com/questions/ask
   stackexchange.com

https://www.nytimes.com/2019/07/16/science/5g-cellphones-wireless-cancer.html
   nytimes.com

https://nextdoor.com/news_feed/?post=117602&ct=-A17-ghvVOF0tfn9vptW_5a7JOBEyP4w6_hJAZUnMQqN56952&ec=OWKiQRDj9vEHefhwfGYAE0s%3D&lc=1002&is=tpe
   nextdoor.com

https://www.amazon.com/gp/product/B007B60SCG/ref=ox_sc_act_title_1?smid=ATVPDKIKX0DER&psc=1
   amazon.com

http://www.verizon.net/index.php
   verizon.net

I'm ignoring those multi-tier domains. I'd prefer to use Bash on macOS.

There are lots of pages on getting the full domain name:

  1. Extract domain name from URL using bash shell parameter substitution

    https://www.cyberciti.biz/faq/get-extract-domain-name-from-url-in-linux-unix-bash/

  2. echo http://example.com/index.php | awk -F[/:] '{print $4}'

    https://stackoverflow.com/a/11385736/1360075

I do not need this level of perfection.

https://github.com/john-kurkowski/tldextract

Improve this question
5
  • If no one has an answer after a couple days, consider adding a small edit with a table of 3 or 4 examples what top/second actually mean. The library you mention has a lookup table so we don’t know if you care about a pattern or just a smaller lookup table? – bmike♦ Jul 20 '19 at 0:37
  • 3
    Do you only need to extract .com domain names? Are you concerned about multi-tier domains, such as .co.uk? – Graham Miln Jul 20 '19 at 8:06
  • I didn't have any idea of what the components of a URL were called. In my looking around these forum, domain name seemed to include www.example.com which I didn't want. I wanted just example.com. I turned to google where I found those names. There doesn't seem to be a uniform definition of the components. mattcutts.com/blog/seo-glossary-url-definitions and doepud.co.uk/blog/anatomy-of-a-url and doepud.co.uk/blog/anatomy-of-a-url For some reason, folks in the UK are most prolific with the explaination of URL components, but they have the funny names too. – historystamp Jul 20 '19 at 18:13
  • 1
    A domain name is just that, a name. The only truly standardized part is the last element (TLD, top level domain), everything else is just a group of strings concatenated by '.' The UK and some other countries have a second layer „beneath“ the TLD, but even is just a convention. – nohillside♦ Jul 20 '19 at 18:22
  • 1
    See rfc-editor.org/info/rfc3986 for the formal definition and en.wikipedia.org/wiki/Uniform_Resource_Identifier for a more approachable description of the URL. – Graham Miln Jul 21 '19 at 7:33
6

As you are already using awk and are looking for a simple solution:

awk -F/ '{n=split($3, a, "."); printf("%s.%s", a[n-1], a[n])}' <<< 'http://www.example.com/index.php'
      ^ ^   ^^^^^^^^^^^^^^^^^^                  ^^^^^^^^^^^^
      | |          |                                  |
      | |          |                            last two elements 
      | |          |
      | |          +--- Split the 3rd field (aka the part after //) into
      | |               the array 'a', using '.' as the separator for splitting.
      | |               Returns the number of created array elements in 'n'.
      | |
      | +-------------- The awk code between the '' gets run once for every
      |                 input line, with the fields split by -F/ stored in
      |                 $1, $2 etc. In our case $1 contains "http:", $2 is 
      |                 empty, $3 contains "www.example.com" and $4 etc. the
      |                 various path elements (if there are any)
      |
      +---------------- Split the input lines into fields, separated by '/'
Improve this answer
6
  • 1
    One line & one command: 👏🏻 ! – dan Jul 20 '19 at 9:22
  • You might add a newline character to printf - printf("%s.%s\n" and could you explain how your solution works ? – fd0 Jul 20 '19 at 12:35
  • 1
    @fd0 Depends on what you want to do with the result afterwards. If you assign it to a variable you don't want to have a trailing newline. – nohillside♦ Jul 20 '19 at 12:36
  • @Dan I see a ; in the one line, but I see it's in single quotes, so I guess it squeaks by. Whatever, it's short and understandable after reading the explanation. – historystamp Jul 20 '19 at 17:52
  • @historystamp the ; is the awk line separator, and here it is read by awk and not the shell because the whole awk script is the part between the '. – dan Jul 20 '19 at 17:56
4

Parsing URLs with Bash

The following questions should provide a good starting point:

@pjz's answer breaks apart a URL into more manageable parts:

#!/bin/sh

INPUT_URL="https://www.amazon.com/gp/product/B007B60SCG/ref=ox_sc_act_title_1?smid=ATVPDKIKX0DER&psc=1"

# extract the protocol
proto="`echo $INPUT_URL | grep '://' | sed -e's,^\(.*://\).*,\1,g'`"
# remove the protocol
url=`echo $INPUT_URL | sed -e s,$proto,,g`

# extract the user and password (if any)
userpass="`echo $url | grep @ | cut -d@ -f1`"
pass=`echo $userpass | grep : | cut -d: -f2`
if [ -n "$pass" ]; then
    user=`echo $userpass | grep : | cut -d: -f1`
else
    user=$userpass
fi

# extract the host -- updated
hostport=`echo $url | sed -e s,$userpass@,,g | cut -d/ -f1`
port=`echo $hostport | grep : | cut -d: -f2`
if [ -n "$port" ]; then
    host=`echo $hostport | grep : | cut -d: -f1`
else
    host=$hostport
fi

# extract the path (if any)
path="`echo $url | grep / | cut -d/ -f2-`"

echo $hostport

Given the $hostport, you should now be able to strip back the domain as desired.

Improve this answer
2
  • "Given the $hostport". I could have hacked something together, but it would not have been elegant. – historystamp Jul 20 '19 at 17:49
  • If you need a robust solution, use an existing URL parsing library. If you need something to pass over a set of constrained URLs once, a Bash script is a reasonable choice. – Graham Miln Jul 21 '19 at 7:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .