glibc application holding onto unused memory until just before exit

Question

I have a C++ application (gcc 4.9.1, glibc 2.17) running on Linux (Centos 7). It uses various third-party libraries, notably Boost 1.61. As the application runs, I can watch its memory usage increasing steadily via `htop`'s `VIRT` and `RES` columns, or the `ps` command, etc. If I let it go long enough, it will use enormous amounts of that memory and swamp the box. Sounds like a leak, but it passes `valgrind` with only a few bytes leaked, all in places I'd expect. Debug print messages indicate program flow is as expected. Digging further via the debugger, I find that most of this memory is still in use when `__run_exit_handlers` is invoked at the end of `main`. I can step through various calls to `free` as it works through the global destructor chain. After it finishes those, I observe only a minimal downward change in the apparent memory usage. Then, finally it calls `_exit()`, and only then is the memory restored to the operating system, all at once. Can anyone offer me additional ti…

Accepted Answer

Everything here is based on GNU libc implementation of `malloc` running on Linux.

Test program below does not seem to give away any memory to the system after freeing memory (`strace` does not show `sbrk` calls that return memory back to the kernel):

```
int main()
{
    static const int N = 5000000;
    static void *arr[N];

for (int i = 0; i < N; i++)
        arr[i] = std::malloc(1024);

// reverse to simplify allocators job
    for (int i = N - 1; i >= 0; i--)
        std::free(arr[i]);
}
```

Looks like glibc does not give away memory back at all. According to `mallopt(3)` man page, parameter `M_TRIM_THRESHOLD` is responsible for giving away memory. By default it is 128kb, while test program allocates and frees 5 GB of memory. Looks like some other details of `malloc` implementation do not let it free memory.

At the moment I can recommend following solutions:

- If you can, try calling `malloc_trim` once in a while or after freeing a lot of memory. This should force trimming and should give memory back to OS using `MADV_DONTNEED`.

- Avoid allocating large number of small objects using `malloc` or `operator new`, instead allocate them from a memory pool of a size greater than `M_MMAP_THRESHOLD`. Try destroing that pool afterwards if program logic allows this. Memory chunks of size greater than `M_MMAP_THRESHOLD` are immediately released back to OS.

- Same as previous one, but should be faster: allocate memory pools for small objects using `mmap` and release memory back to OS using `madvise` and `MADV_DONTNEED`/`MADV_FREE`.

- Try using another allocator that might take advantage of `MADV_FREE` to return memory back to the system (jemalloc?).

I have found this old (2006) ticket on glibc's bugzilla. It says there that `free` never returns memory back to the kernel, unless `malloc_trim` is called.

Newer versions of `free` seem to have code that executes internal `systrim` function that should trim top of the arena, but I wasn't able to make it work.

glibc application holding onto unused memory until just before exit

Solution

Problem

Related problems