How does sort work out how much RAM there is?
linux, memory, sorting
Solution
If you're using GNU `sort`, the answer is it calculates a default based on the rlimits for data (set by `ulimit -d`) and RSS (set by `ulimit -m`) as well as the sysconf values for available memory and total memory.
Regardless of your ulimit, the default memory size won't exceed more than 3/4ths of either your currently available memory, or 1/8th of your total memory, whichever is greater.
/* Let MEM be available memory or 1/8 of total memory, whichever
is greater. */
double avail = physmem_available ();
double total = physmem_total ();
double mem = MAX (avail, total / 8);
/* Leave a 1/4 margin for physical memory. */
if (total * 0.75 < size)
size = total * 0.75;
With GNU `sort`, you can use the `-S` option to specify sorting buffer size:
-S, --buffer-size=SIZE
use SIZE for main memory buffer
This value can either be a number of kilobytes, can be suffixed with another unit (e.g. `-S 100M`), or can be a percentage of total memory (e.g. `-S 55%`)
Problem
If I do `ulimit -v 200000` and the run `sort largefile` I can see from `top` that sort uses at most 142232 Virt and 92764 Res but this decreases even more after a while. - How does sort know what the ulimit limit was set to? - Why doesn't it use all the 200MB I have given it?