Redis vs Kafka vs RabbitMQ for 1MB messages
I am currently investigating a queuing solution for handling 1MB medium sized messages. Apart from the differences in the features of Redis, Kafka and RabbitMQ, I cannot find a good answer to their performance on messages around 1MB.
- Any of you guys know how many 1MB messages any of these descriptors can handle?
- Do you know of any other queuing solutions that might work better?
source to share
I am in charge of Kafka.
Kafka herself has very good performance even for large messages. In our tests with 2 Kafka nodes, we achieve p2p communication with a smaller 150MB / sec message with messages over 150MB / sec.
The only thing you need to remember is to configure the broker to accept large messages.
Good article: Configuring Kafka for Performance and Resource Management - Handling Large Messages
I know of another p2p solution that might be interesting if you have specific requirements see YAMI4
I've used Redis, but only for very small messages, so I can't say anything about 1MB.
source to share
When evaluating Kafka vs Redis in your case, there are other factors you should consider besides the message size. Here are some that I can think of:
- How many producers / consumers? Redis performance can be affected by more producers / consumers due to the nature of Redis (forced queue). This is due to the fact that Redis delivers a message to all consumers at the same time, the moment the message is placed on the queue.
- Do you need speed or reliability first? If speed is of the utmost importance, use Redis as it does not store messages and delivers them faster. If you want reliability, use Kafka as it retains messages even after they've been delivered.
- Do you want your consumers to receive messages when they are ready, or do you want messages to be sent to consumers immediately? In the first case, use Kafka because it is a pull based mechanism (the consumer has to request a message). In the second case, use Redis as it uses a push based mechanism (the message is sent to the consumer as soon as it enters the queue). RabbitMQ is also push based (although there is a pull API with poor performance)
- How many messages are expected? If that's not a lot, use Redis as you are limited in memory. Otherwise, use Kafka. Best practice for RabbitMQ is to keep queues short. This means that you can use messages as fast as they appear in the queue. So if you have some kind of long running consumer-side operation, RabbitMQ might not be the best choice.
- Scaling? Kafka scales very well horizontally (it's built with scalability in mind). RabbitMQ usually scales vertically. Redis also scales well horizontally if needed.
Obviously, there is more than one criterion when evaluating the correct queue solution. There are best practices and guidelines for each of the queue engines you are looking at. Think more about your particular use case, it is definitely worth the time as it will save you time if you choose the wrong queuing mechanism.
source to share