Wait for blocking collection (queue) to decrease in size in C#
.net, c#, event-driven, multithreading, queue
Solution
Instead of using a `Queue<T>`, you can use a `BlockingCollection<T>` for Q2. If you set its `BoundedCapacity`, calls to `Q2.Add()` will block when the capacity is reached. This will automatically throttle the processing of Q1, as the N tasks will begin blocking if they can't add to the final queue.
Problem
I'm working on project with the following workflow: Part One: - Event arrives asynchronously and is queued in blocking queue, we'll call that Q1 - Thread picks up next available item from that queue - Item ends up running {N} number of tasks in parallel - Each task queues its result on a second queue, we'll call that Q2. - When processing of item finishes, the next item is read off the queue. Part Two: - Another thread reads off of Q2 one object at a time and works on the result So, the problem here is, every item on the first queue ends up running a large number of tasks in parallel, and each task queues its result. The second queue must be processed serially, one item at a time though, and it's getting flooded. My Question I need a mechanism that will make the thread processing Q1 wait until the number of items in Q2 is below a specific threshhold. What's the best way to achieve this? Is there any way to have an event driven solution rather than a polling solution?