Clever Geek Handbook

Propagating a task with python and zeromq

I have a working application using python and zeromq and I would like to optimize it.

In short, the master node sends the same request to all workers (about 200) and then collects the responses. Based on the answer, it sends a message back to one node and node response.

I have now implemented a very simple pattern. Each worker has one REP socket, and the server has a list of REQ sockets. The server iterates over all sockets, sending a generic message, and then iterates over all sockets to collect responses. Finally, based on the responses, the server selects one worker, sends him a message and waits for a response.

This is, of course, rather slow. The slowest part is sending the same message 200 times. The collection is also slow. The solutions I have found for distributing tasks and collecting responses require load balancing which is not what I need. I need every worker to receive a message and respond.

What is the recommended pattern for this situation?

thank

+3


source to share


2 answers


I don't know zmq. Here's a pattern that might not work, just to start:

master node sends the same request to all workers (about 200)

master PUB bind *: 3140 send

working SUB connect masterhost: 3140 SUBSCRIBE recv

then collect answers



working PUSH connect masterhost: 3141 send

master PULL bind *: 3141 recv

Based on the response, it sends a message back to one response node and node.

master REQ connect workerhost: 3142 send recv

worker REP bind *: 3142 recv send

0


source


If each worker needs a different job, Pub / sub wont works. Then you need a working pool implementation. Either you push jobs in a circular fashion (just use a push socket bound to the server and every client pulls it, zeromq will do a round robin) or every worker requests a job from the server if the jobs at least have minimal complexity and the difference between jobs is high, which is the best approach. The zeromq manual has numerous examples on the home page and on the net:



The exact implementation also depends on whether you need to process jobs reliably.

+2


source






More articles:


All Articles