Clever Geek Handbook

Queue based processing

So I am developing an application that relies on something similar to a messaging bus. The idea is to be extremely resilient. I have a queue of tasks to be completed and here are the steps that I believe will be the ultimate goal in a queue based system.

  • Server A / B initially adds the item to the queue for work.
  • Server C listens on the queue for a new item in the queue and starts working on it. I believe it should block the item with a timeout (if the server goes down, etc., I need other workers to be able to work on it.)

Now one of two things happens:

  • Server C is not responding to the specified task, or it has timed out a long time ago and the queue unblocks it and passes it to Server D for processing

- OR -

  • Server C completes the task and eventually removes the entry from the queue.

I have looked for different solutions and I see that a lot of people are using REDIS as a backend to do this operation, but the queue is pretty simple. For example RPOPLPUSH

will remove a key from the queue. What happens if the server crashes? Now the queue thinks that it has processed this item, and we have a lost task.

What are the recommended steps to ensure task completion and notes on task failures that can be reproduced by another server? I intend to write tasks in go and I am open to using cloud services like AWS.

+3


source to share


2 answers


Redis is the basic component on which you can create a queuing system. However, implementing a real guaranteed delivery system on top of Redis is not trivial, especially if you require transactional behavior.

Here are some of the queuing systems implemented with Redis in different languages:

Things like this can be designed in Go, but when it comes to true guaranteed delivery semantics, the devil is in the details.



You will probably be better served by a dedicated queuing system like RabbitMQ or ActiveMQ. While they are more complex, they offer more options and probably better guarantees.

Here is a Go client for RabbitMQ: https://github.com/streadway/amqp

You might also be interested in disque (a dedicated queue solution from the author of Redis) and the corresponding Go client at https://github.com/EverythingMe/go-disque

Finally, beanstalkd is another lightweight solution; you can find the Go client at: https://github.com/kr/beanstalk

+2


source


We might point out the obvious here, but SQS ( http://aws.amazon.com/sqs/ ) gives you what you need out of the box. You don't have to worry about managing the queue system, it scales automatically for you, you focus on writing the application.



you post messages to the queue. workers pull them out of the queue, process them, and receive a message when done. if workers do not confirm the message after the time specified by you, the message will be returned to the other worker.

+1


source






More articles:


All Articles