Limit number of Threads in Task Parallel Library

.net, azure-storage, c#, c#-5.0, task-parallel-library

Solution

You should not be using threads for this at all. There's a `Task`-based API for this, which is naturally asynchronous: CloudBlockBlob.UploadFromFileAsync. Use it with `async/await` and `SemaphoreSlim` to throttle the number of parallel uploads.

Example (untested):

const MAX_PARALLEL_UPLOADS = 5;

async Task UploadFiles()
{
    var files = new List<string>();
    // ... add files to the list

    // init the blob block and
    // upload files asynchronously
    using (var blobBlock = new CloudBlockBlob(url, credentials))
    using (var semaphore = new SemaphoreSlim(MAX_PARALLEL_UPLOADS))
    {
        var tasks = files.Select(async(filename) => 
        {
            await semaphore.WaitAsync();
            try
            {
                await blobBlock.UploadFromFileAsync(filename, FileMode.Create);
            }
            finally
            {
                semaphore.Release();
            }
        }).ToArray();

        await Task.WhenAll(tasks);
    }
}

Problem

I have few hundreds of files i need to upload to Azure Blob Storage. I want to use parallel task library. But instead of running all the 100 threads to upload in a foreach on list of files, how can i put a limit on max number of threads that it can use and finish the job in parallel. or does it balance the things automatically?

Original source