Why does peak memory usage increase when there are more elements to loop/apply over?
apply, garbage-collection, memory, memory-management, r
Solution
I think this sentence from the Memory Chapter of Advanced R Programming by Hadley Wickham best summarizes the reason for the discrepancy.
Garbage collection normally happens lazily: R calls gc() when it needs more space. In reality, that R might hold onto the memory after the function has terminated, but it will release it as soon as it's needed
The chapter also has a good function called `mem()` that allows you to see more clearly how much memory a block of code is using than `gc()` allows. If time allows, I would redo the test with Wickham's `mem()` function.
Edit: As Peter noted, the `mem()` function is deprecated. Use the `mem_used()` function from the pryr package instead.
Problem
I am trying to reduce the memory footprint of an R package and have noticed behaviour that I can't seem to suppress. See the below example: ``` x <- matrix(runif(1.5e7), ncol = 200) ## CASE 1: Test with half of columns gc(reset = TRUE) a <- apply(x[, 1:100], 2, quantile) gc() # used (Mb) gc trigger (Mb) max used (Mb) # Ncells 190549 10.2 407500 21.8 222055 11.9 # Vcells 15292303 116.7 35490421 270.8 35484249 270.8 object.size(a) # 4696 bytes rm(a) ## CASE 2: Test with all columns gc(reset = TRUE) b <- apply(x, 2, quantile) gc() # used (Mb) gc trigger (Mb) max used (Mb) # Ncells 190824 10.2 407500 21.8 245786 13.2 # Vcells 15293740 116.7 39292189 299.8 39286529 299.8 object.size(b) # 8696 bytes rm(b) ## CASE 3: Test with all columns + call gc gc(reset = TRUE) c <- apply(x, 2, function(i) { r <- quantile(i); gc(); r }) gc() # used (Mb) gc trigger (Mb) max used (Mb) # Ncells 191396 10.3 407500 21.8 197511 10.6 # Vcells 15294307 116.7 45737818 349.0 30877185 235.6 object.size(c) # 8696 bytes rm(c) ``` `a` and `b` differ by only ~4kb yet the garbage collector reports a difference of ~30mb between the peak memory usage of cases 1 and 2. `c` uses less memory than both `a` and `c`, I imagine not without a considerable penalty in runtime. The peak memory allocation seems to positively correlate with the number of columns considered in the call to `apply`, but why? Does the call to `apply` result in memory allocation living beyond the scope of an iteration? I would have expected any internal temporaries to be freed (or marked as being unused) by the `gc` before the end of each iteration. This behaviour can be reproduced using `lapply` over `data.frame`s and also with different functions in lieu of `quantile`. I am under the impression that I am overlooking a very fundamental aspect of memory usage behaviour in `R` but still can't wrap my head around it. Ultimately, my question is: how do I further reduce the memory footprint in cases like the example above? Thanks in advance and do not hesitate to point out any inaccuracies in my question. EDIT: As per @ChristopherLouden's suggestion, I used calls to `mem` in place of `gc` and all three cases were described as taking ~126.9182mb. ``` ## http://adv-r.had.co.nz/memory.html#garbarge-collection mem <- function() { bit <- 8L * .Machine$sizeof.pointer if (!(bit == 32L || bit == 64L)) { stop("Unknown architecture", call. = FALSE) } node_size <- if (bit == 32L) 28L else 56L usage <- gc() sum(usage[, 1] * c(node_size, 8)) / (1024 ^ 2) } ```