What are the possible use cases for Amazon SQS or any queue?
So, I was trying to get Amazon AWS
since my company is using all the infrastructure.
One component that I could never get it right is Queue Service
, I searched Google quite a bit, but I could not get a satisfactory answer. I think the work Cron
and are Queue Service
somewhat similar, correct me if I am wrong.
So what exactly does it do SQS
? As I understand it, it stores simple messages that will be used by other components in AWS
order to complete tasks, and you can send messages to do this.
In this question Can someone explain to me what Amazon Web Services components are used in a regular web service? ; the answer mentioned what they used SQS
to queue up the tasks they want to execute asynchronously. Why not just return the message to the user and do the processing later? Why wait SQS
to get your stuff done?
Also, let's say I have a web application that allows the user to schedule some daily tasks, how SQS
would that fit?
source to share
No, cron and SQS are not alike. One (cron) schedules jobs and the other (SQS) saves messages. Queues are used to decouple message producers from message consumers. This is one way architects can scale and be reliable.
Let's say you've created a mobile voting app for a popular TV show and 5 to 25 million viewers all vote at the same time (at the end of each speech). How are you going to deal with that many votes in such a short amount of time (say 15 seconds)? You could create a sizable layer of web server and database that could handle millions of messages per second, but that would be expensive, you would need to anticipate the maximum expected workload and it would not be resilient (e.g. a database crash or throttling). If several people voted, you overpay for the infrastructure; if the vote goes crazy, then the votes may be lost.
The best solution would be to use some kind of queuing mechanism that would separate the voting applications from your service, where the voting queue was very scalable so that it could happily consume 10 messages per second or 10 million messages per second. Then you will have an application layer that pulls messages from this queue as quickly as possible to count the votes.
source to share
One thing I would add to @jarmod's excellent and concise answer is that post size of posts matters. For example, in the AWS maximum size is only 256 KB , if you are not using the extended client library, which increases to a maximum of 2 GB. But note that it uses S3 as temporary storage.
In RabbitMQ, the practical limit is around 100KB. There is no hard-coded limit in RabbitMQ, but the system just stops more or less often. From personal experience, RabbitMQ can process a constant stream of messages of about 1MB for 1 to 2 hours non-stop, but then it starts to behave erratically, often becoming a zombie, and you will need to restart the process.
source to share