Throttling Asynchronous Calls Made by Actors
We have a fairly high bandwidth that makes asynchronous calls to external systems over http. We find the downlink systems are overloaded due to the number of calls they receive from us.
Downstream calls are made using the "pipe to" scheme described here: https://petabridge.com/blog/akkadotnet-async-actors-using-pipeto/
The reason so many calls go to downstream systems is because the actor does not wait for a response from the asynchronous call before it processes the next message in its mailbox (it is completed with a message when the asynchronous call starts). Obviously this is by design, but in these cases it results in a very large number of asynchronous calls being made by the external service.
We need a way to regulate incoming calls. I can think of several possible solutions to this problem.
-
Execute external service calls synchronously while waiting for the task to complete. Set up a pool router for the actor, which will basically be the way to regulate the number of calls this external service makes.
-
Use the ReceiveAsync method, not the Receive method. This is basically the same as option 1. On the petabride page I posted above, although it says about this method - "just don't do it" :)
-
Before making the async call, start all incoming messages and then unblock them after the async task completes. Obviously, the use of this bandwidth is much more limited.
I was wondering if anyone has a similar problem when working with akka and was able to solve it?
Edit:
So, in the end, only option 1 worked for us. That is, having a pool router with Receive (), which specifically waited for the IO calls that it needed to make (api call for the external system). This works very well and we can control the "throttling" by setting the pool size.
We tried option 2 (ReceiveAsync), but we found that at some point the system would stop and stop responding without throwing any errors. We suspect he is at a dead end. This has to do with how the async key works, just waiting for the task to complete using .Wait () or .Result. Now I can understand why Petabridge recommends against using ReceiveAsync :)
We did not try option 3 as it would have meant more significant changes.
source to share
As for me, I solved this by using creating sub-actors with a router as workers that can only handle one message at a time. Thus, you can adjust the load of the external system to the number of employees. It can also give you the option to use consistent hashing to avoid concurrent processing of certain messages.
As for the workers - in one project I used the 1st method, but used a dispatcher for workers - so that they always have one thread to handle messages and just don't affect other parts of the system. This is good if you have a fairly consistent load.
source to share
Actually the second option ( ReceiveAsync
) is the perfect solution for your problem. The only risk is that you are slowing down the sender in this case, as the actor will now wait asynchronously for the HTTP request to complete. This means that the actor himself may be overwhelmed if the high level of messages keeps pushing him constantly.
If so, you can:
- Increase the number of users (listeners on the other end of the HTTP connection) to keep up with the pace.
- Simulate your problem using Akka streams instead of actors. Streams have built-in backpressure support that can be applied upstream until they reach the original source of the request trace.
source to share