Scaling ECS EC2 instances when a task cannot be hosted
I am using an ECS cluster for Jenkins agents / slaves with Jenkins ECS plugin .
The plugin pushes the ECS task when the task requests build-node. Now I want to scale the EC2 instances in the Autoscaling group associated with the ECS cluster as required.
- Jenkins is often idle. In this case, I do not want any instances to be present in the autosave group.
- If a node request (and therefore an ECS task) is requested and cannot be pushed, I want to add the EC2 instance to the autoscale group.
- If an instance is inactive and just before billing hour, I want that instance to be deleted.
3. The point can be run by a cronjob on EC2 instances, which regularly checks to see if the conditions are met and removes the EC2 instance.
But how can I fulfill the 2. point? I am unable to create a cloud call alert that is triggered if a task cannot be hosted.
How can i do this?
source to share
For point 2, one way to deal with this problem would be to autoscale if there are not enough cpu blocks to accommodate the new jenkins slave.
You must use the cluster processor sparing metric to scale. http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html#cluster_reservation
source to share
A rather hackish way to achieve this: you can use a lambda function to detect when a service has runningCount + pendingCount < desiredCount
more than X seconds. (I haven't tested this yet.) Similar solutions are suggested here .
There seems to be no correct scaling solution only when tasks cannot be accommodated. Perhaps AWS wants us to override our clusters, which might be good practice for high availability, but not always the best or cheapest solution.
source to share
If a task cannot be allocated, it means that the placement of this task in your ECS cluster will exceed your memory or reservation. You can configure Cloudwatch alarms for one or both of these ECS metrics and an autoscaling policy that will add and remove EC2 instances in your ECS cluster.
This, combined with an autoscaling policy that scales your ECS services to ecs: service: DesiredCount must be large enough for you to add the base EC2 instances required by your ECS cluster.
For example, your ScalingPolicy for an ECS service might be "when we use 70% of our allocated memory for this service, add 2 to DesiredCount". After adding 1 service task, the ECS Cluster MemoryReservation score may exceed the "80" threshold, after which the Cloudwatch alarm will trigger a certain threshold in ECS MemoryReservation, and the autoscaling policy will add another EC2 node where the second task can now be placed.
source to share