Clever Geek Handbook

Amazon S3 Conditional Input Object

I have a system where I receive a lot of messages. Each message has a unique identifier, but can also receive updates during its lifetime. Since the time between sending messages and processing can be very long (weeks), they are stored in S3. Only the latest version is required for each post. My problem is that sometimes two messages of the same ID come together, but they have two versions (older and newer).

Is there a way for S3 to have a conditional PutObject request where I can declare "put this object if I don't have a newer version in S3"?

+3


source to share


1 answer


I need an atomic operation here

This is not a precedent for S3, which is ultimately consistent. Some ideas:



  • You can try to split your messages - all messages starting with AL go to one box, MZ goes to another block. Then each window locally checks for duplicates.

  • Your best bet is probably some kind of database. Depending on your use case, you can use a regular SQL database, or perhaps a simple RAM-only database like Redis. Write to multiple Redis DBs at the same time to avoid SPOF.

  • There is SWF that can create a unique processing queue for each item, but that probably means more HTTP requests than just checking against S3.

  • An interesting idea from David about enabling version control. You may have a daemon that periodically shuts down older versions. When reading, you will need to "read repair" where you look for versions that look for a new object.

+2


source






More articles:


All Articles