How to automate workers in AWS stepping functions
We will implement the process using AWS step functions. Some of the tasks in this process take a long time. We use actions (workers) for this step running on ECS instances. The state machine looks like this:
Trigger -> step 1 (Lambda) -> step2 (ECS) -> end step
Is there a way to find the number of tasks in the queue waiting to be received by the step2 workers and use this information to auto-scale the workers (ECS cluster)?
source to share
You can call the api list_executions
to get a list of running executions and what is called for each of those executions get_execution_history
.
In the response, you can iterate over the execution events. If you find it on the list ActivityScheduled
and you don't see it ActivityStarted
, it means that it is waiting for a worker to complete this task.
source to share