Releasing memory in Python
memory-management, python
Solution
Memory allocated on the heap can be subject to high-water marks. This is complicated by Python's internal optimizations for allocating small objects (`PyObject_Malloc`) in 4 KiB pools, classed for allocation sizes at multiples of 8 bytes -- up to 256 bytes (512 bytes in 3.3). The pools themselves are in 256 KiB arenas, so if just one block in one pool is used, the entire 256 KiB arena will not be released. In Python 3.3 the small object allocator was switched to using anonymous memory maps instead of the heap, so it should perform better at releasing memory.
Additionally, the built-in types maintain freelists of previously allocated objects that may or may not use the small object allocator. The `int` type maintains a freelist with its own allocated memory, and clearing it requires calling `PyInt_ClearFreeList()`. This can be called indirectly by doing a full `gc.collect`.
Try it like this, and tell me what you get. Here's the link for psutil.Process.memory_info.
import os
import gc
import psutil
proc = psutil.Process(os.getpid())
gc.collect()
mem0 = proc.memory_info().rss
# create approx. 10**7 int objects and pointers
foo = ['abc' for x in range(10**7)]
mem1 = proc.memory_info().rss
# unreference, including x == 9999999
del foo, x
mem2 = proc.memory_info().rss
# collect() calls PyInt_ClearFreeList()
# or use ctypes: pythonapi.PyInt_ClearFreeList()
gc.collect()
mem3 = proc.memory_info().rss
pd = lambda x2, x1: 100.0 * (x2 - x1) / mem0
print "Allocation: %0.2f%%" % pd(mem1, mem0)
print "Unreference: %0.2f%%" % pd(mem2, mem1)
print "Collect: %0.2f%%" % pd(mem3, mem2)
print "Overall: %0.2f%%" % pd(mem3, mem0)
Output:
Allocation: 3034.36%
Unreference: -752.39%
Collect: -2279.74%
Overall: 2.23%
Edit:
I switched to measuring relative to the process VM size to eliminate the effects of other processes in the system.
The C runtime (e.g. glibc, msvcrt) shrinks the heap when contiguous free space at the top reaches a constant, dynamic, or configurable threshold. With glibc you can tune this with `mallopt` (M_TRIM_THRESHOLD). Given this, it isn't surprising if the heap shrinks by more -- even a lot more -- than the block that you `free`.
In 3.x `range` doesn't create a list, so the test above won't create 10 million `int` objects. Even if it did, the `int` type in 3.x is basically a 2.x `long`, which doesn't implement a freelist.
Problem
I have a few related questions regarding memory usage in the following example. If I run in the interpreter, ``` foo = ['bar' for _ in xrange(10000000)] ``` the real memory used on my machine goes up to `80.9mb`. I then, ``` del foo ``` real memory goes down, but only to `30.4mb`. The interpreter uses `4.4mb` baseline so what is the advantage in not releasing `26mb` of memory to the OS? Is it because Python is "planning ahead", thinking that you may use that much memory again? Why does it release `50.5mb` in particular - what is the amount that is released based on? Is there a way to force Python to release all the memory that was used (if you know you won't be using that much memory again)? NOTE This question is different from How can I explicitly free memory in Python? because this question primarily deals with the increase of memory usage from baseline even after the interpreter has freed objects via garbage collection (with use of `gc.collect` or not).