Which queue is most appropriate?
I am building a mobile photo sharing site in Python similar to TwitPic and have been researching various queues for image processing. I've looked into RabbitMQ and ActiveMQ, but I think there is a better solution for my use case. I'm looking for something a little lighter. I am open to any suggestions.
source to share
You can write a daemon that uses the built-in multiprocessing library and its Queue .
All you have to do is create a pool of workers and make them wait for jobs from the queue. Your main process can queue new jobs and you are good to go.
source to share
The good thing about Gearman is that it allows you to synchronize multiple tasks performed by multiple employees at will.
I have used beanstalkd successfully in several high volume applications.
The latter is better suited for asynchronous jobs, and the former gives you more flexibility if you want to block the execution of a job.
source to share
Are you considering a single machine architecture or a cluster of machines? Sending an image to an available worker process on the same computer or on a different computer isn't much different, especially if you're using TCP sockets. Knowing what workers are available, create more and resources if needed, having a failover mechanism if a worker fails, etc. Gradually complicate the problem.
It could be something as simple as using httplib to push an image to a personal server running Apache or twisted and a set of cgi applications. When you add another server, combine the request between them.
source to share