Extract parent domain name from a list of url through Bash ShellScripting

awk, bash, linux, unix

Solution

Using awk

awk -F \/ '{l=split($3,a,"."); print (a[l-1]=="com"?a[l-2] OFS:X) a[l-1] OFS a[l]}' OFS="." file|sort -u

contatoruy.in
dicadodia.com.br
doomyjupe.com
forterins.com
gaelsyaray.com
livrariacultura.com.br
maxxivrimoveis.com.br
meguiatramandai.com.br
prategama.com
quetxviii.com
smilecire.com
suleacatan.com
theirpoem.com
toneyvaws.com
visionwebmkt.com
woadsbevy.com
yournjuju.com
zonalrems.com
zrobimystrone.pl

Problem

I have a list of urls like this: ``` http://noto.zrobimystrone.pl/pucenter/images/NGdocs/ http://visionwebmkt.com/unsubscribe.php?M=879552&C=b744d324e38f5f3b0bcf549f1d57a3ab&L=20&N=497 http://www.meguiatramandai.com.br/unsubscribe.php?M=722&C=8410431be55bf12faac13d18982d71cd&L=1&N=3 http://www.contatoruy.in/link.php?M=86457&N=4&L=1&F=H http://www.maxxivrimoveis.com.br/ http://www.meguiatramandai.com.br/unsubscribe.php?M=722&C=8410431be55bf12faac13d18982d71cd&L=1&N=2 http://arm.smilecire.com/ch+urch38146263923bpa.stor/imp-roved258021029his+health212149011 http://hurl.zonalrems.com/ge.tyo-ur584372780599hea+lth247408058un/der+control21211901 http://harp.doomyjupe.com/see.this-better/life+58291551346csexdrive663295668+better/how.981692016 http://beefy.toneyvaws.com/no+tice/how/35306640b+see/app=5429204last/attempt=457943182 http://kirk.yournjuju.com/shop/sam.sclub-win=ter/58387369768esame+673844946.bett.er-loo.k981686408 http://idly.theirpoem.com/veri-fy/notice-7853508818b2glob/al=who.43639603inc.lusion-610549278 http://wva188.suleacatan.com/credit-score/review/-551694841511001sfdghsfdgsdfg63887839 http://cop.forterins.com/app.lyto=face962540097dtolo+oko.ung268570307yo.un-ger8752507 http://vni116.gaelsyaray.com/qertqetert//-dghjghjghd5531864856415612229498430 http://ticket.prategama.com/shop/sam.sclub-win=ter/752490935same+226373195.bett.er-loo.k212801 http://cbu125.quetxviii.com/cvbnvbn7551116db537203--swrtytry664896546 http://c5a.dicadodia.com.br/pass4sp09/NetAffProTeste-1.html http://snub.woadsbevy.com/ama/zing-753773417oppe-tun/ity+217801.is-here/now=236922473 http://mkt.livrariacultura.com.br/pub/cc?_ri_=X0Gzc2X%3DWQpglLjHJlYQGgzfB7tPi0PuyyJ71ES ``` I wanna extract only the parents domain names, for example: ``` http://noto.zrobimystrone.pl/pucenter/images/NGdocs/ http://visionwebmkt.com/unsubscribe.php?M=879552&C=b744d324e38f5f3b0bcf549f1d57a3ab&L=20&N=497 http://www.meguiatramandai.com.br/unsubscribe.php?M=722&C=8410431be55bf12faac13d18 ``` Into ``` zrobimystrone.pl visionwebmkt.com meguiatramandai.com.br ``` I have tried ``` awk '{gsub("http://|/.*","")}1' list.txt ``` and got the following results: ``` noto.zrobimystrone.pl visionwebmkt.com www.meguiatramandai.com.br www.contatoruy.in www.maxxivrimoveis.com.br www.meguiatramandai.com.br arm.smilecire.com hurl.zonalrems.com harp.doomyjupe.com beefy.toneyvaws.com ``` but dont know how to get only the parent name from noto.zrobimystrone.pl for instance.

Original source