What's the memory usage of LINQ Join?
.net, c#, linq, memory
Solution
Looking in Reflector at `Enumerable.JoinIterator`, it seems that the whole `inner` (second) sequence is loaded into memory (into a `Lookup`), but that the `outer` (first) sequence is then streamed (i.e. not all loaded into memory).
There doesn't seem to be anything clever going on if the `inner` sequence is too big to fit into memory.
Jon Skeets seems to agree on the first point:
The real Join operator uses the same behaviour as Except and Intersect when it comes to how the input sequences are consumed:
...
When MoveNext is called on the result sequence for the first time, it immediately consumes the whole of the inner sequence, buffering it.
The outer sequence is streamed - it's only read one element at a time. By the time the result sequence has started yielding results from the second element of outer, it's forgotten about the first element.
Problem
If the two IEnumerable(s) are in memory, what would be the memory usage for joining them? Assume selecting all columns. Is it size of left table + size of right table + number of rows in the joined table? If the two IEnumerable(s) are defined by file streaming, will Join throw out of memory exception if they are too big to fit in memory? Or will it load until near out of memory and run the scans multiple times (similar to database join)?