EC2 instances: how to start tasks, how to stop them?
I have a long batch job that I would like to run on Spot AWS EC2 instances to save money. However, I cannot find an answer to two seemingly critical questions:
-
When a new instance is created, I need to load the code onto it, configure it and run the code. How is this done for Spot Instances that are automatically and non-responsive?
-
When the instance is stopped, I would prefer to have some type of notification so that the state can be persisted. (This is not critical, as the batch job will run fine if it ends abruptly, but a clean shutdown is preferred.)
What is the standard way to deploy spot instances? Is there a way to do a manual tweak, turn it into a spot instance, and then let it sleep until a spot price is available?
source to share
As for # 1, if you are creating an AMI (Amazon car image), you can have whatever you want pre-installed on a "sleeping" image that you can use as a basis for the initial spot image:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances-getting-started.html
For # 2, you can get notified when an instance of an instance shuts down using SNS:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-autoscaling-notifications.html http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html
BTW: You can get a notification that the instance was aborted, but only after it exits. You can't get notified that the instance will shut down and save state gracefully - you need to design your solution to be okay with unexpected shutdown.
No matter how high you bid, there is always a risk that your Spot Instances will be interrupted. We strongly recommend against bidding above the demand price or using Spot for applications that cannot be interrupted.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-protect-interruptions.html
source to share