Which is the best way to improve memory usage when you collect a large data set before processing it? (.NET)

.net, c#

Solution

You ought to leverage C#'s iterator blocks and use the `yield return` statement to do something like this:

 public class Program
 {
     public IEnumerable<SomeClass> GetObjects()
     {
         while( // get implementation
             yield return object;
         }
     }

     public void ProcessObjects(IEnumerable<SomeClass> objects)
     {
         foreach(var object in objects)
             // process implementation
     }

     void Main()
     {
         var objects = GetObjects();
         ProcessObjects(objects);
     }
 }

This would allow you to stream each object and not keep the entire sequence in memory - you would only need to keep one object in memory at a time.

Problem

When I have to get GBs of data, save it on a collection and process it, I have memory overflows. So instead of: ``` public class Program { public IEnumerable<SomeClass> GetObjects() { var list = new List<SomeClass>(); while( // get implementation list.Add(object); } return list; } public void ProcessObjects(IEnumerable<SomeClass> objects) { foreach(var object in objects) // process implementation } void Main() { var objects = GetObjects(); ProcessObjects(objects); } } ``` I need to: ``` public class Program { void ProcessObject(SomeClass object) { // process implementation } public void GetAndProcessObjects() { var list = new List<SomeClass>(); while( // get implementation Process(object); } return list; } void Main() { var objects = GetAndProcessObjects(); } } ``` There is a better way?

Original source