grep -f alternative for huge files
grep, large-files, scripting, unix
Solution
Try using LC_ALL=C . It turns the searching pattern from UTF-8 to ASCII which speeds up by 140 time the original speed. I have a 26G file which would take me around 12 hours to do down to a couple of minutes. Source: Grepping a huge file (80GB) any way to speed it up?
So what I do is:
LC_ALL=C fgrep "pattern" <input >output
Problem
``` grep -F -f file1 file2 ``` file1 is 90 Mb (2.5 million lines, one word per line) file2 is 45 Gb That command doesn't actually produce anything whatsoever, no matter how long I leave it running. Clearly, this is beyond grep's scope. It seems grep can't handle that many queries from the `-f` option. However, the following command does produce the desired result: ``` head file1 > file3 grep -F -f file3 file2 ``` I have doubts about whether sed or awk would be appropriate alternatives either, given the file sizes. I am at a loss for alternatives... please help. Is it worth it to learn some `sql` commands? Is it easy? Can anyone point me in the right direction?