Monitor the total number of tasks when using Async / Await in a recursive function
I wrote this code. It recursively creates folders on the web system by making REST Calls. So basically it creates a folder for the root node, then gets all the child nodes and calls itself in parallel and recursively. (for each child)
the only problem with the code is that if a node can have children too OR if the hierarchy is too deep I start getting "TaskCancellation" errors.
I already tried to increase the timeout to 10 minutes .. but that doesn't solve the problem.
So my question is, how can I start talking about 50 tasks and then wait for something to free up and only continue when there is an open slot of 50.
Currently I think my code continues to create tasks without any constraints as the flow goes through the hierarchy.
public async Task CreateSPFolder(Node node, HttpClient client, string docLib, string currentPath = null)
{
string nodeName = Uri.EscapeDataString(nodeName);
var request = new { __metadata = new { type = "SP.Folder" }, ServerRelativeUrl = nodeName };
string jsonRequest = JsonConvert.SerializeObject(request);
StringContent strContent = new StringContent(jsonRequest);
strContent.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json;odata=verbose");
HttpResponseMessage resp = await client.PostAsync(cmd, strContent);
if (resp.IsSuccessStatusCode)
{
currentPath = (currentPath == null) ? nodeName : currentPath + "/" + nodeName;
}
else
{
string content = await resp.Content.ReadAsStringAsync();
Console.WriteLine(content);
throw new Exception("Failed to create folder " + content);
}
}
List<Task> taskList = new List<Task>();
node.Children.ToList().ForEach(c => taskList.Add(CreateSPFolder(c, client, docLib, currentPath)));
Task.WaitAll(taskList.ToArray());
}
source to share
You can use SemaphoreSlim
to manage the number of parallel tasks. You initialize the semaphore to the maximum number of tasks you want to have, then every time you do the task you acquire the semaphore, then release it when you finish the task.
This is a somewhat simplified version of your code that runs forever using random numbers and does no more than two tasks at a time.
class Program
{
private static SemaphoreSlim semaphore = new SemaphoreSlim(2, 2);
public static async Task CreateSPFolder(int folder)
{
try
{
await semaphore.WaitAsync();
Console.WriteLine("Executing " + folder);
Console.WriteLine("WaitAsync - CurrentCount " + semaphore.CurrentCount);
await Task.Delay(2000);
}
finally
{
Console.WriteLine("Finished Executing " + folder);
semaphore.Release();
Console.WriteLine("Release - CurrentCount " + semaphore.CurrentCount);
}
var rand = new Random();
var next = rand.Next(10);
var children = Enumerable.Range(1, next).ToList();
Task.WaitAll(children.Select(CreateSPFolder).ToArray());
}
static void Main(string[] args)
{
CreateSPFolder(1).Wait();
Console.ReadKey();
}
}
source to share
First of all, I think your problem is not the number of tasks, but the number of blocked threads waiting in Task.WaitAll(taskList.ToArray());
. Better to wait asynchronously in such cases (i.e.await Task.WhenAll(taskList);
Second, you can use TPL Dataflow
ActionBlock
with MaxDegreeOfParallelism
set to 50 and just submit it for every folder you create. This way you have a flat queue of work to be done and when it's empty, you're done.
Pseudocode:
var block = new ActionBlock<FolderInfo>(
async folderInfo => {
await CreateFolderAsync(folderInfo);
foreach (var subFolder in GetSubFolders(folderInfo))
{
block.Post(subFolder);
}
},
new DataFlowExecutionOptions {MaxDegreeOfParallelism = 5});
block.Post(rootFolderInfo);
source to share