How to sort a large file on two levels efficiently?
perl, sorting, unix
Solution
The UNIX `sort` utility can handle sorting large data (e.g. larger than your working 16GB of RAM) by creating temporary working files on disk space.
So, I'd recommend simply using UNIX `sort` for this as you've suggested, invoking the option `-T tmp_dir`, and making sure that `tmp_dir` has enough disk space to hold all of the temporary working files that will be created there.
By the way, this is discussed in a previous SO question.
Problem
I have a very large file, over 100GB (many billions of lines), and I would like to conduct a two-level sort as quick as possible on a unix system with limited memory. This will be one step in a large perl script, so I'd like to use perl if possible. So, how can I do this? My data looks like this: ``` A 129 B 192 A 388 D 148 D 911 A 117 ``` ...But for billions of lines. I need to first sort by letter, and then by number. Would it be easier to use a unix sort, like... ``` sort -k1,2 myfile ``` Or can I do this all in perl somehow? My system will have something like 16GB ram, but the file is about 100GB. Thanks for any suggestions!