AWS Kinesis and Lambda Version Control
I created an AWS Firehose endpoint (could change to plain Kinesis) that receives logs from producers and stores them in an S3 bucket and a lambda function that consumes the data, processes it, and stores the output to db.
Everything works perfectly. I am now planning on doing middleware and streaming development for this whole framework. When I release a new version, I cannot immediately replace all manufacturers, so I need to keep the old versions of the products until there is no producer left, because I might break the change protocols in the new versions.
I'm not sure what the best approach would be to create a version using kinesi and lambda. Should I copy the whole structure for new versions (including dev and staging) and force producers to write a specific stream of versions?
or should I create a middle lambda function that checks for packages (which contain their version information) and outputs events to a specific s3 that has folder versions? So lambda functions will only consume data they know about. This will allow me to use versioning for lambda functions.
Here is a picture of the structure for the first idea
Here is the second structure
I wonder what the best solution would be or if there are better ways to do this.
source to share
First, Lambdas can be launched directly with Kinesis - no need for Kinesis Firehose or S3.
Second, your question really boils down to: do you need a separate Kinesis + Lambda pipeline per version or not. I would go with the following solution:
- One Kinesis stream for all data versions.
- One Lambda function in this thread. It internally handles different versions separately. Roughly speaking, think of the various if-else checks by version number.
Advantages of the above approach to a single Kinesis + Lambda pipeline per version:
- The first is operatively simpler. In the latter case, you will need to set up a new pipeline every time a new version is introduced.
- You will have a small number of active versions at any given time. So a few if-else checks in your code should work fine.
Of course, keep the Dev and Prod pipelines separate to minimize the explosion radius of the wrong code in the old one.
source to share