In the publish / subscribe model in microservices, how to receive / consume a message only once per service type
We are developing a microservices architecture model where service A publishes a message and services B and C wants to receive / consume a message. However, for high availability, multiple instances of service B and C are running concurrently. Now the question is how do we design such that only one instance of service B and one instance of service C receive the message, and not all other service instances.
As far as I know about RabbitMQ, this behavior is not easy to achieve. I wonder if Kafka or some other messaging framework has built-in support for this scenario, which I think should be very common in microservices architecture.
source to share
Kafka has a feature called Consumer Groups that does exactly what you describe.
Each identical instance of B can declare its group group.id with a single string (for example, "serviceB"), and Kafka ensures that each instance is assigned a mutually exclusive set of topic sections for all topics it subscribes to.
Since all instances of C will have a different .id group (say serviceC), then they will also receive the same messages as instances of B, but they will be in an independent consumer group, so messages are only sent to 1 of N instances of C, the maximum copies is the total number of sections.
You can dynamically and independently increase or decrease the number of instances of B and C. If any instance dies, the remaining instances will automatically rebalance the assigned partitions and take over message processing for the deceased instance.
Data should never be stored more than once, so there is another commit log or "source of truth" for all of these service instances.
source to share
Kafka has built-in support for this scenario.
You can create two Customer Groups , one for B
and one for C
. Both Consumer Groups
sign messages from A
.
Any message posted A
will be sent to both groups. However, only one member of each group can receive the message.
source to share
You can also test this use case in kafka using command line tools.
You create a manufacturer with
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Then you can create two different user groups (cgB, cgC) with
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=cgB
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=cgC
Once you post a message to a topic, both groups (B, C) will receive the message, but keep which message they handle independently.
Explain better here: Kafka Quickstart
source to share
These are the changes that need to be made to achieve the same result with MQ Rabbit
- Create 2 separate queues, one for each service B and C.
- Change your logic to read a message from the queue so that only one instance will read the message from the queue using the rabbitmq connection blocking thing.
This way, when multiple instances of B and C are run, both will receive the message and are still scalable.
source to share