Invalid SQS queue caused by AWS Lambda call errors
I have an AWS Lambda function that subscribes to a DynamoDB stream and configured with an SQS dead letter queue (DLQ) . I can see that the correct queue is configured in the management console. Also I took care of granting my permissions for sqs:SendMessage
on my DLQ.
The subscription works, but still hangs on call errors as if no DLQ has been configured. Ie, if there is a message that results in an unhandled exception, the function keeps repeating that message until it is removed from the stream. I can see that the number of access errors is increasing and DLX errors are not showing in the Cloudwatch dashboard. The SQS remains empty.
What I want is that the failed messages are forwarded to my DLQ and the subscription continues until the next message. Any ideas?
Edit
As Jonathan Seed stated below, DLQ does not currently work with streaming signatures. AWS Support has confirmed that they are working to implement it.
source to share
I believe this is because DynamoDB streams are stream based event sources. The lambda documentation says that when dealing with stream-based event sources, "if the Lambda function does not work, AWS Lambda tries to process the erroneous batch of records until the data expires"
From my understanding, the lambda function will retry until the event is successfully processed or expires and disappears from the thread, the event is never "discarded" by the lambda function as they are in non-thread event sources.
You may have to implement your own failover as part of your main lambda function if you want to cancel certain events by manually sending the event to the queue / topic and return successfully.
source to share