awk + filter Log files
awk, bash, linux, perl, sed
Solution
With GNU awk for gensub():
$ awk '!seen[gensub(/([^,]*,){3}/,"","")]++' file
[INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08
With any awk that supports RE intervals (most modern awks):
$ awk '{key=$0; sub(/([^,]*,){3}/,"",key)} !seen[key]++' file
[INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08
Problem
I used the following nice awk command in order to filter duplicate lines example: ``` cat LogFile | awk '!seen[$0]++' ``` the problem is that in some cases we need to filter duplicate lines in spite some fields are different and they no so important for example LogFile: ``` [INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08 [INFO],[02/Jun/2014-19:31:25],EXE,ds1a,INHT VERION , 1.4.4.3-08 [INFO],[02/Jun/2014-19:32:40],EXE,ds1a,INHT VERION , 1.4.4.3-08 ``` please take a look on this file - LogFile I need to remove the duplicate lines from the third delimiter "," until the end of the line , and no matter what is before the third delimiter so finally I should get this filtered file: ( should get always the first one in the list ) ``` [INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08 ``` so please help me to complete my task how to filter the LofFile from the third delimiter "," , and ignore the fields: [INFO],[...........],EXE, Remark – implantation can be also with perl one liner line