Propagating a task with python and zeromq
I have a working application using python and zeromq and I would like to optimize it.
In short, the master node sends the same request to all workers (about 200) and then collects the responses. Based on the answer, it sends a message back to one node and node response.
I have now implemented a very simple pattern. Each worker has one REP socket, and the server has a list of REQ sockets. The server iterates over all sockets, sending a generic message, and then iterates over all sockets to collect responses. Finally, based on the responses, the server selects one worker, sends him a message and waits for a response.
This is, of course, rather slow. The slowest part is sending the same message 200 times. The collection is also slow. The solutions I have found for distributing tasks and collecting responses require load balancing which is not what I need. I need every worker to receive a message and respond.
What is the recommended pattern for this situation?
thank
I don't know zmq. Here's a pattern that might not work, just to start:
master node sends the same request to all workers (about 200)
master PUB bind *: 3140 send
working SUB connect masterhost: 3140 SUBSCRIBE recv
then collect answers
working PUSH connect masterhost: 3141 send
master PULL bind *: 3141 recv
Based on the response, it sends a message back to one response node and node.
master REQ connect workerhost: 3142 send recv
worker REP bind *: 3142 recv send
If each worker needs a different job, Pub / sub wont works. Then you need a working pool implementation. Either you push jobs in a circular fashion (just use a push socket bound to the server and every client pulls it, zeromq will do a round robin) or every worker requests a job from the server if the jobs at least have minimal complexity and the difference between jobs is high, which is the best approach. The zeromq manual has numerous examples on the home page and on the net:
- http://zguide.zeromq.org/page:all (look at the paronoid drawing of a pirate)
- http://blog.garambrogne.net/post/2010/10/23/simple-python-work-queue-with-zeromq
- https://github.com/marklit/zeromq-worker-queue
The exact implementation also depends on whether you need to process jobs reliably.