malloc returns null when run with Valgrind

c, c++, linux, memory-management, valgrind

Solution

Note that `malloc(3)` is lying to you -- it doesn't actually allocate all the memory at once, it just asks the OS for it, and the OS lies to `malloc(3)`. This is perfectly normal behavior that most of the time works fine. The description for `/proc/sys/vm/overcommit_memory` in `proc(5)` contains the details:

   /proc/sys/vm/overcommit_memory
          This file contains the kernel virtual memory
          accounting mode.  Values are:

                 0: heuristic overcommit (this is the default)
                 1: always overcommit, never check
                 2: always check, never overcommit

          In mode 0, calls of mmap(2) with MAP_NORESERVE are not
          checked, and the default check is very weak, leading
          to the risk of getting a process "OOM-killed".  Under
          Linux 2.4 any nonzero value implies mode 1.  In mode 2
          (available since Linux 2.6), the total virtual address
          space on the system is limited to (SS + RAM*(r/100)),
          where SS is the size of the swap space, and RAM is the
          size of the physical memory, and r is the contents of
          the file /proc/sys/vm/overcommit_ratio.

Valgrind cannot be so flippant; it actually keeps track of allocated, initialized, and uninitialized memory for a process. It therefore requires more memory than the process does on its own, and it does not have the same tolerance for over-committing memory.

I do not know how much more memory you will need to run the program under valgrind, but try adding a few more gigabytes of swap space. You can make a new swap file by using `dd` to write zeros to a file -- do not use a sparse file -- and then run `mkswap(8)` on the file to initialize it and run `swapon(8)` with the filename to tell the system to use it as a swap file.

Problem

I have a huge allocation (tens of GBs) happening with a malloc call which works in normal scenario. The system does have a huge RAM and is a 64 bit machine running 2.6 x86_64 kernel. The mem rlimit has been done as INFINITY with setrlimit. I thought of running it with Valgrind to do the memory analysis and check for the leaks. But malloc fails with NULL pointer returned when run with valgrind. I tried reducing the size of allocation but that doesn't help. Any inputs ? Regards, -J

Original source