Strange multiprocessing issues with zeroRPC
So I am playing with ZeroRPC and Tornado for a pet project and I am facing some problems using ZeroRPC in combination with a multiprocessing library python
. In particular, I create and start new ZeroRPC servers programmatically, but usually the server zerorpc
locks up on startup , so I thought it should transfer it to a different process:
server = zerorpc.Server(FuncWrapper())
server.bind(server_address)
process = multiprocessing.Process(target=server.run)
process.start()
However, when I do this, the server call RPC
just hangs, which is typical behavior when the endpoint has not been created correctly. However, if I just let the server run the block and call it like this:
serhouldver = zerorpc.Server(FuncWrapper())
server.bind(server_address)
server.run()
Everything works perfectly. I understand that these two implementations should be equivalent, but somehow they are not.
Any ideas?
source to share
zerorpc uses gevent for shared asynchronous I / O. You can watch tornado, multiprocessing and gevent play together.
For which I can say:
server = zerorpc.Server(FuncWrapper()) server.bind(server_address) process = multiprocessing.Process(target=server.run) process.start()
Lines 1 and 2 create and bind the port to the current process. But on lines 3 and 4, what I can guess is the following is happening:
- Called
- fork, all threads in the process are lost in fork
- any zeromq socket and context are already dead (no more thread). Good news: a context can be destroyed and a new one created (see http://lists.zeromq.org/pipermail/zeromq-dev/2014-January/024536.html ).
- You now have a port open in your local process with an active zeromq socket, but no one is reading from that socket (hence no reaction when you talk to it).
- On the other hand, in a multiprocessor process, zerorpc is executed by invoking recv on the zmq source socket relic. Nothing ever comes, zeromq is already dead.
Without testing, I can only guess that a running zerorpc in a completely new process should work:
def ZeroRPC_Service():
server = zerorpc.Server(FuncWrapper())
server.bind(server_address)
server.run()
process = multiprocessing.Process(target=ZeroRPC_Service)
process.start()
Then, through some Manager object or other shared memory services offered by the multiprocess, you can make the zerorpc server in the new process available and communicate with your local process.
On the side of a note, if what you are trying to do is call server.run () without blocking, if you only use gevent, I would tell you to just run it in your own coroutine:
server = zerorpc.Server(FuncWrapper()) server.bind(server_address) server_coro = gevent.spawn(server.run)
Perhaps you can call server.run
directly from the coroutine / asyunc tornado function. Maybe there is a way to integrate gevent and tornado (for example this link suggests: https://bmmh.wordpress.com/2012/10/07/tornado-and-gevent/ ), I don't know enough to help you with this point ...
source to share