Unexpected Scala collections memory behaviour

collections, memory-management, scala, stream

Solution

Here is what happens. `Stream` is always evaluated lazily but already calculated elements are "cached" for later. Lazy evaluation is crucial. Look at this piece of code:

a = a.flatMap( v => Some( v ) )

Although it looks as if you were transforming one `Stream` to another and discarding the old one, this is not what happens. The new `Stream` still keeps a reference to the old one. That's because result `Stream` should not eagerly compute all elements of underlying stream but do that on demand. Take this as an example:

io.Source.fromFile("very-large.file").getLines().toStream.
  map(_.trim).
  filter(_.contains("X")).
  map(_.substring(0, 10)).
  map(_.toUpperCase)

You can chain as many operations as you want, but file is barely touched to read first line. Each subsequent operation just wraps the previous `Stream`, holding a reference to child stream. The moment you ask for `size` or do `foreach`, evaluation starts.

Back to your code. In the second iteration you create third stream, holding a reference to the second one, which in turns keeps a reference to the one you initially defined. Basically you have a stack of pretty big objects growing.

But this doesn't explain why memory leaks so fast. The crucial part is... `println()`, or `a.size` to be precise. Without printing (and thus evaluating the whole `Stream`) `Stream` remains "unevaluated". Unevaluated stream doesn't cache any values, so it's very slim. Memory would still leak due to growing chain of streams in one another, but much, much slower.

This begs a questions: why it works with `toList` It's quite simple. `List.map()` eagerly creates new `List`. Period. The previous one is no longer referenced and eligible for GC.

Problem

The following Scala code (on 2.9.2): ``` var a = ( 0 until 100000 ).toStream for ( i <- 0 until 100000 ) { val memTot = Runtime.getRuntime().totalMemory().toDouble / ( 1024.0 * 1024.0 ) println( i, a.size, memTot ) a = a.map(identity) } ``` uses an ever increasing amount of memory on every iteration of the loop. If `a` is defined as `( 0 until 100000 ).toList`, then the memory usage is stable (give or take GC). I understand that streams evaluate lazily but retain elements once they are generated. But it appears that in my code above, each new stream (generated by the last line of code) somehow keeps a reference to previous streams. Can someone help explain?

Original source