How do I achieve accessibility while handling events in order?
purpose
I want to implement robust event handling and still process them in the order they arrived.
Description
The listener task listens for the events of many clients, wraps them in a message, and enqueues them. Event handler tasks are read from the queue and process events. Events from one client must be processed in order; events from different clients may be handled out of order. A simple solution:
Create N queues that listens for event queues in queue[client_id % N]
. Each queue has one event handler that reads it.
This works, but I should also consider the case where the event handler fails. I can think of two ways to handle this:
- Election of a leader among queue readers - one sample for each queue
- Use RabbitMQ Confirms function (or equivalent in other MQ systems) - each queue has multiple event handlers. The event handler removes the message and sends an acknowledgment after it has been processed.
Scale of the task
There are about 5K events / sec, events are small 50-200 bytes. The message in the queue is likely to be a cluster of 1000 events to reduce MQ overhead. This means that an MQ system that supports Option 2 and processes tens of messages / sec will work.
Question
Can I do option 2 or will it have too many queue conflicts? Is this supported with confirmations? Is there a better way to implement such a design? I can send event clusters to event handlers and wait for an ACK before sending the next chunk, but I think using MQ is probably less reinventing the wheel.
source to share
Another way to do this is by using a split theme in Kafka. Kafka can split a topic by ID and maintain order within a topic. Therefore, in this case, it is useful to split into client_id
. Kafka then manages the load between clients by reworking the theme.
Also see: How to provide distributed processing and high availability at the same time in Kafka?
source to share
I have used RabbitMQ before, but cannot confirm if it supports your scenario of only the next message, when the first message is successfully processed, I will leave this part to someone else with better knowledge of this tool.
for implementation with any common service / message bus:
The event source queue can help solve the problem using the following technique:
- You have a pool of handlers (in the queue, or can handle all queues altogether, if it is the same event), when the handler does not work, we still have many others, and we can instantiate both multiple and multiple handlers. lag behind demand.
- Each queue must be blocked when an event is processed by a handler to maintain the order in which events are processed.
- A blocked queue must time out to automatically unblock and recover the last event (which was blocked) in case the handler selects an event and fails to process it.
- The event handler, upon completion, unlocks the queue and removes the event that has just finished processing.
This technology partially peeps, but blocks the entire queue, not just one event.
You won't be able to increase throughput with this approach, possibly more stability and fault tolerance, mainly due to blocking queues. Your best bet is to think about how to remove the message order dependency and let the handlers select those messages in any order and still achieve the business logic you want (maybe correlation can help).
source to share
First, I strongly believe that the most efficient way to handle requests is to process them sequentially with a single handler. Hence, the solution must provide "proximity", so each client must be bound to a handler (as you described in your "simple" solution). We just needed to create a dispatch procedure that was efficient and compatible with handler failures.
How can affinity be realized? How to reassign an affinity if the handler should be disabled?
As a key measure, I would suggest limiting the length of the handler queues and keeping the exceptional load in one large "inbound" queue:
clients -> listener -> (big incoming queue) -> load balancer -> handler queues -> handlers
\________<-__system events_<-____/
Once we have some guarantees on the length of the handler queue (for example, "the queue is short enough to handle 0.1 seconds"), we have access to a wide variety of load balancing methods that might not be available in general. The queue size limit should be adjusted as low as possible without sacrificing performance.
The Load Balancer procedure should be able to:
- maintain a bi-directional map (or ever simple array) that assigns handlers to clients;
- send messages from the queue of handlers entering the queue in accordance with the map;
- receive notifications when a handler is idle (then decouple clients from that handler)
- receive notifications about the death of a handler (then redistribute clients from this handler);
- reload dead handler queue (unless data loss is allowed)
In such an architecture, the logic would be highly customizable. For example, we will get many options for how to implement "queue overflow" (a situation when the handler's queue has reached its limit):
- stop the whole processing.
- destroy the handler and reallocate its queue
- find out (in some way, for example, using time limits or counters in shared memory) that the handler queue does not contain any requests from this client, and then reassigns the client;
- defer the request in some way;
- prohibit the client; and etc.
My suggestion is not a hi-tech solution, it looks more like homemade programming. But my guess is that such a design will be scalable and efficient enough to provide high availability.
source to share