BackgroundWorker limit number of workers and create them as become available

backgroundworker, c#, multithreading

Solution

Honestly? Don't use `BackgroundWorker`. Use TPL manually, TPL Dataflow or even Reactive Extensions (Rx) if you want this kind of control.

Personally I would use TPL Dataflow and I would setup an `TransformBlock<MyObject, MyObject>` which you configure with a `MaxDegreeOfParallelism` that makes sense for you (e.g. you want to process 20 at a time) and then link that back to a UI updating `ActionBlock<MyObject>` that is configured to run on the UI (`Dispatcher`) thread. Such code would look something like this...

Somewhere in your initialization logic, possibly as part of Window constructor (NOTE: must be executed on the main Dispatcher thread to work correctly)

TransformBlock<MyObject, MyObject> myProcessingBlock = new ActionBlock<MyObject, MyObject>(
   myObject =>
   {
     // ... perform your processing of this object here ...

     return myObject;
   },
   new ExecutionDataflowBlockOptions
   {
      MaxDegreeOfParallelism = 20
   });

ActionBlock<MyObject> myUINotificationBlock = new ActionBlock<MyObject>(
   myObject =>
   {
       // ... update the UI details for this data here ...
   },
   new ExecutionDataflowBlockOptions
   {
       TaskScheduler = TaskScheduler.FromCurrentSynchronizationContext() // must be executed on the Dispatcher block!
   });

 myProcessingBlock.LinkTo(myUINotificationBlock);

And this is how you would post new work to it:

MyObject someObjectToProcess = GetSomeObjectToProcess(...);

myProcessingBlock.Post(someObjectToProcess);

The rest is all handled for you by the magic of the TPL Dataflow Library. Just declare it and set it free. TPL Dataflow even supports `async` methods, so if you know you're going to be making web service calls you can just use the `async` keyword on the method you define for the `TransformBlock<MyObject, MyObject>` like so:

TransformBlock<MyObject, MyObject> myProcessingBlock = new ActionBlock<MyObject, MyObject>(
   async myObject =>
   {
     HttpClient someHttpClient = new HttpClient();

     HttpResponseMessage responseMessage = await someHttpClient.PostAsync(..., ...);

     return myObject;
   },
   new ExecutionDataflowBlockOptions
   {
      MaxDegreeOfParallelism = 20
   });

This way you won't even block a CPU thread while the HTTP network call is outstanding which is even more full of win.

Problem

I have a function that is creating a `BackgroundWorker` for each object in the list, what I want to do now is limit the number of workers created to 20, and as they finished, process the next item in the list. I was thinking on using a queue and before processing the worker check the queue size if it's <20 then create the worker and add it to the queue. My question is how can I remove that item from the queue once it's finished? and how can I set the loop to wait until a worker becomes available? EDIT I think I'm close and this is what I have now, but the problem is the program is getting stuck at the `while` loop: ``` var myObjectList = new List<myObject>(); myObjectList = PopulateList(); BackgroundWorker bgw; foreach (var obj in myObjectList) { bgw = new BackgroundWorker(); while(BgwList.Count >= 20); //getting stuck here, why? BgwList.Add(bgw); if(!bgw.IsBusy) { bgw.RunWorkerAsync(obj); } } ``` Removing the `BackgroundWorker` once it is finished: ``` void BackgroundWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e) { BgwList.Remove((BackgroundWorker)sender); //verified that bgw is being removed from list //... other code } ```

Original source