"sort filename | uniq" does not work on large files

Question

I can remove duplicate entries from small text files, but not large text files. I have a file that's 4MB. The beginning of the file looks like this: ``` aa aah aahed aahed aahing aahing aahs aahs aal aalii aalii aaliis aaliis ... ``` I want to remove the duplicates. For example, "aahed" shows up twice, and I would only like it to show up once. No matter what one-liner I've tried, the big list will not change. If It type: sort big_list.txt | uniq | less I see: ``` aa aah aahed aahed <-- didn't get rid of it aahing aahing <-- didn't get rid of it aahs aahs <-- didn't get rid of it aal ... ``` However, If I copy a small chunk of words from the top of this text file and re-run the command on the small chunk of data, it does what's expected. Are these programs refusing to sort because the file is too big? I didn't think 4MB was very big. It doesn't output a warning or anything. I quickly wrote my own "uniq" program, and it has the same behavior. It works on a small subset of the list, but…

Accepted Answer

You can normalize line delimeters (convert CR+LF to LF):

```
sed 's/\r//' big_list.txt | sort -u
```

"sort filename | uniq" does not work on large files

Solution

Problem