OutOfMemory error when processing a big file in Clojure
clojure, out-of-memory
Solution
You wouldn't believe how much overhead a realized lazy seq imposes. I tested this on a 64-bit OS: it's something like 120 bytes. That's pure overhead for every lazy seq member. The vector, on the other hand, has quite a low overhead and is basically the same as a Java array, given a large enough vector. So try replacing `doall` with `vec`.
Let's also see how much memory you are spending without the overhead. You've got 5e6 pairs of integers -- that's 5e6 x 8 = 40 MB. You could save by using shorts and get a 50% saving (I repeat---that's not counting the overhead of the parent collection, and each vector instance holding the pair has its own overhead).
The next step in the saving is to use a raw array for both the outer collection and for the pairs. It could still be a very practical solution since an array is seqable and integrates quite well with the language. To do that, you'd just have to replace the two occurrences of `vec` with `to-array`.
UPDATE
The difference between `Integer` and `Short` is not that big due to both still being full-fledged objects. It would save much more to store the number pairs as primitive arrays, using `short-array` (or `int-array`) instead of `to-array`.
Problem
I'm following the algo-class.org course, and one of its programming assignment provide a file with format as below: ``` 1 2 1 5 2 535 ``` ... There are over 5 million such lines, I want to read in the file and convert it to a vector of integer vector like this: [[1 2][1 5][2 535]...]. ``` (defn to-int-vector [s] (vec (map #(Integer/parseInt %) (re-seq #"\w+" s)))) (def ints (with-open [rdr (clojure.java.io/reader "<file>")] (doall (map to-int-vector (line-seq rdr))))) ``` So i believe in this way, I'm not holding the entire file in memory, and only generating a large integer vector. But i get OutOfMemoryError from this. I did try to generate a vector of the same size and same format by running rand-int, and that works fine. Looks like the memory issue is caused by the temp objects generated? What is the ideal way in clojure to handle a case like this? Update: yes, i realize that I am hold the entire integer vector. I have raised the heap size and it now works. I'm interested that a vector and 5 million elements(10 million intergers) can take up so much memory -- I have to allocate 3g for the jvm. Is there any other way that will take the memory down?