Costs associated with an AWS Lambda Fault Tolerant Function?
I am learning serverless technology (specifically Python, Django and Zappa on AWS Lambda) and one thing about error handling hit me. The Zappa docs say:
By default AWS Lambda will try to retry the call based on events (non-API gateway like CloudWatch) if an exception was thrown.
In the AWS Lambda Documentation , I read:
Depending on the source of the event, AWS Lambda may retry a failed lambda function. For example, if Kinesis is the source of the event, AWS Lambda will retry the failed call until the Lambda function completes or the writes in the stream expire.
Does this mean that the function will be called an infinite number of times when it raises an unhandled exception? If this happens out of control, costs must go through the roof.
Concerning this; what does "before the entries in the stream expire" mean? What records and what stream?
source to share
According to AWS docs :
-
Event sources that are not streaming : for example, S3, API Gateway, etc.
-
Synchronous Calling : If you called Lambda using the SDK or API gateway, if an exception is thrown, you are responsible for deciding when / when / how the request should be retried.
-
Asynchronous Call : If Lambda was called by an async call (like S3), it will automatically retry the call twice with delays between attempts. If you have specified a dead letter queue, then the failed event is sent to SQS / SNS. If DLQ is not specified, the event will be discarded.
-
-
Stream-based event sources such as DynamoDB and Kinesis.
- If the Lambda function fails, it will keep trying until the data expires (maximum 7 days for Kinesis). It retries with an exponential backoff with a ceiling of 1 minute between two tries. You will be charged for all retries, but you can create an alert to start and stop the stream when the source is offline.
The documentation on stream based event source is not very well defined, but you can read this stream on the AWS forums where an AWS employee answered the question:
Question
Specifically, when my Lambda receives Kinesis events and writes data to another service ... but the other service goes down for a while (like a few hours) ... will my Lambda keep getting (and throwing errors) at a constant rate?
Lambda retry is good because I want guaranteed delivery of events, but ideally in this situation I also don't want to be evaluated at a high rate when my Lambda was consistently unsuccessful for a while
Answer
If a function starts execution but fails due to downstream dependencies, then you get a score for the duration of the function execution. Lambda backs off exponentially if your function fails, down to about one minute. You can also track this as the ShardIteratorAge increases and take action to pause the thread processing if necessary until you resolve the downstream dependency
source to share