How can I control the number of threads when I use "Task.WhenAll"
I am validating image urls by making an asynchronous HTTP request. Everything works fine with the code below, but when I have so many images, our firewall will block my internet access due to so many threads requesting at the same time. So I was looking for a solution on how to limit the number of concurrent threads. I ended up with this thread suggesting that I use SemaphoreSlim , but I am somehow unable to figure out and how to implement it?
- is that SemaphoreSlim is waiting or waitAsnyc (what's the difference anyway?) should be inside foreach when tasks are added? Can I just create a task list with linq like in my code?
- why the job is being used. Run?
- Does the thread run after the line is executed? after task.run or task.whenall?
If this is not the best approach, suggest a better one. I'm not sure if using MaxDegreeOfParallelism with parallelism. Does better also make sense?
Dim tasks = myImages.Select(Function(x) testUrl_async(x))
Dim results = Await Task.WhenAll(tasks)
Async Function testUrl_async(ByVal myImage As image) As Task(Of image)
Dim myImageurl as string=myImage.imageurl
myHttpResponse = Await myHttpClient.GetAsync(myImageurl)
If myHttpResponse.IsSuccessStatusCode Then
Return myImage
Else
Return Nothing
End If
End Function
source to share
our firewall will block my internet access due to so many threads simultaneously requesting. So I was looking for a solution on how to limit the number of concurrent threads.
It is pretty accurate that the firewall limits the number of your connections, and hence you want to limit the number of connections (not threads).
is that SemaphoreSlim is waiting or waitAsnyc (what's the difference anyway?)
Wait
is a synchronous wait - it will block the calling thread. WaitAsync
is an asynchronous wait - it releases the calling thread and resumes execution of the current method when a semaphore is available.
should be inside foreach when adding tasks? Can I just create a task list using linq like in my code?
You can do it either way: create the list explicitly, or use LINQ.
why the job is being used. Run?
This is a bug in this answer. Task.Run
certainly not needed or needed here.
after which the line is run? after task.run or task.whenall?
When you call Task.Run
, this delegate is immediately put into the thread pool. But as I said above, you don't want to use Task.Run
(it shouldn't be used in the original answer either).
So, something like this should be sufficient:
Private _mutex As New SemaphoreSlim(20)
Async Function testUrl_async(myImage As image) As Task(Of image)
Await _mutex.WaitAsync()
Try
Dim myImageurl = myImage.imageurl
Dim myHttpResponse = Await myHttpClient.GetAsync(myImageurl)
Return If(myHttpResponse.IsSuccessStatusCode, myImage, Nothing)
Finally
_mutex.Release()
End Try
End Function
source to share
You can use Task
and SemaphoreSlim
like this:
var MaxAllowedConcurrentRequestsCount = 20;
var guard = new SemaphoreSlim(0, MaxAllowedConcurrentRequestsCount);
foreach(var imageUrl in imageUrls)
{
guard.Wait()
var image = await Task.Factory.StartNew((imageUrl) => return myHttpClient.Get(imageUrl));
guard.Release();
// You can do whaterver you like with image. For example add it to you concurrent list.
}
source to share
I am assuming you are not in WPF or Windows Form Thread. If you only have one thread running after await
.
As we assume you are not in these threads, it is ThreadPool
used to execute continuations after await
. You can change the number of threads used by the pool from ThreadPool.SetMaxThreads
, but I suggest you not and leave it up to .NET. doing the right thing. This is usually the best way.
source to share