Clever Geek Handbook

Reading data sequentially in an AWS SQS queue

I am very new to AWS SQS queues and I am playing with boto right now. I noticed that when I try to read a queue full of messages in a while loop, I see that after reading 10-25 messages, the queue does not return any message (although the queue has over 1000+ messages). It starts filling in another set of 10-25 messages after a few seconds, or stops and restarts the program. while true: read_queue () // connection has already been established with the correct queue.

Any thoughts on this behavior or point me in the right direction. Just to reiterate, I'm only a couple of days at SQS!

thank

+3


source to share


2 answers


How SQS queues work by default (short poll). If you have not changed any parameters after configuring the queue, a message from a weighted random sample of machines is accepted by default. If you are using more than one computer and want all the messages that you can use at that moment (on all machines), you need to use long polling. See Amazon documentation here. I don't think boto supports this directly by ATM.



+2


source


Long polling is more efficient because it allows you to leave the HTTP connection open for a period of time while you wait for more results. However, you can still do your own polling in boto by simply setting up a loop and waiting for a period of time between reading the queue. You can get good overall throughput with this polling strategy.



+2


source






More articles:


All Articles