Must consume highly available messages
Is there a standard design pattern I can use to consume messages from a queue in order, but still have high availability?
I can of course split the load into separate queues by the last digit account number (the order is only important for each account), which gives me scalability, but if the host account numbers ending in "2" fail, for example I need that something to lift this load.
I would have thought there was a standard template for this kind of processing. unfortunately I cannot make the messages idempotent since the queue source is related to third party integration.
Any thoughts are appreciated.
source to share
Despite the fact that it was a few days ago, I had to answer a question in which the word "idempotent" was indicated. Don't think there is a design pattern here, but I have an approach.
I would go with separate worker queues as you suggested handling messages in a scalable way. A simple brain-sorted reader would read from the third queue and send messages to the appropriate work queue.
The HA part will work with work queue readers. Each of them will have a friend queue reader. Each queue reader sent a heart rate message to their interlocutor at a regular interval. If the buddy hasn't received a message (or a certain number of messages), he:
- Start handling your now dead friends queue as well as your own
- Inform sysadmin about dead queue reader
In order to prevent two readers from hitting the same queue, when the animated dead queue is rebuilt, she will first rebuild her heartbeat with her buddy. As soon as the interlocutor recognizes that he is no longer processing messages in the original queue of the read queue, the updated reader will start working again.
You can get more redundancy by increasing the number of friends in the group, or queue readers will have a new buddy when their original buddy dies, but that adds more complications when readers die or come back.
One approach to this is to have a token for each queue. The reader could only read the queue when he possessed his token. Each reader will start using one token and broadcast the heartbeat to all other readers. Heartbeat will include all the tokens for the queues that the reader is currently processing. This will provide all readers with a picture of the system as a whole without requiring centralized management. When a reader notices that a token has not been broadcast for a certain timeframe, he will require a token if he:
- Has the smallest number of tokens among the surviving readers.
- Or in the case of a tie, has the lowest ID # among the surviving readers with the fewest tokens.
Once it declares the token, it will start processing the queue and send a notification to the system administrator.
When the reader is online again, he will listen to the heartbeat and restore his picture of the system. It will then determine which reader has:
- Most tokens
- Or in the case of a link, the reader with the highest ID #
The expected reader will state that the last token is in another other reader list. Once another reader confirms the claim and which no longer processes the queue represented by the token, the animated reader will start processing the queue again.
One possible benefit of this approach is that a one-to-one relationship between readers and queues is not required. It allows you to create any number of queues that make sense and find the appropriate number of readers that can handle the load.
source to share