Node.js heap memory limit for single object
javascript, memory-management, node.js, v8
Solution
It turns out that there are hard-limits put on the maximum size of strings, objects and arrays. The limts are a remnant of the old garbage collector. Here is the relevant ticket:
https://code.google.com/p/v8/issues/detail?id=3505
Problem
Does v8 have limits on the heap allocations for single objects? `a = new Array(1024*1024*102)` fails on node command-line with `FATAL ERROR: JS Allocation failed - process out of memory` Also, this fails with the same error when run as a script `node --expose-gc --nouse-idle-notification --max-old-space-size=8192` `FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory` ``` var util = require('util'); o = {}; while(1) { o["hahahahahaha" + String(ctr)] = ctr; ctr++; if (ctr % 100000 === 0) { console.log(util.inspect(process.memoryUsage())); if (ctr % 10000000 === 0) gc(); } } ``` Last output: `{ rss: 1009557504, heapTotal: 993408824, heapUsed: 964980592 }` However, ``` var a = []; while(1) { var o = {}; o["hahahahahaha" + String(ctr)] = ctr; a.push(o); ctr++; if (ctr % 100000 === 0) { console.log(ctr); console.log(util.inspect(process.memoryUsage())); console.log(); if (ctr % 10000000 === 0) gc(); } } ``` is just fine `{ rss: 5466140672, heapTotal: 1091224368, heapUsed: 1070460592 }` Edit: `node -v` `v0.10.25` `uname -a` `Linux 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux` Edit 2: Even this works! It seems that v8's limit applies to number of properties an object can have? ``` while(1) { if (!o["hahahahahaha" + String(Math.floor(ctr/1000000))]) { o["hahahahahaha" + String(Math.floor(ctr/1000000))] = {}; console.log(Object.keys(o)) } o["hahahahahaha" + String(Math.floor(ctr/1000000))][String(ctr)] = ctr; ctr++; if (ctr % 100000 === 0) { console.log(ctr); console.log(util.inspect(process.memoryUsage())); console.log(); if (ctr % 10000000 === 0) gc(); } } ``` `{ rss: 2474512384, heapTotal: 2466405768, heapUsed: 2431583008 }` Also, I found this: https://github.com/v8/v8/blob/master/src/objects.h#L2928 I wonder if it's relevant.