Different startup and startup behavior in Python multiprocessing
I am trying to start multiple processes in a Python program using multiprocessing.Queue
to exchange data between them.
My code is shown as follows TestClass
- it is a process that receives packets from the zmq socket and passes them to the queue. There is another process (I took it out of the code) continues to receive messages from the queue. I also have a script to post posts to this zmq feed.
from multiprocessing import Process, Queue
import zmq
import time
class TestClass(Process):
def __init__(self, queue):
super(TestClass, self).__init__()
# Setting up connections
self.context = zmq.Context()
self.socket = self.context.socket(zmq.SUB)
self.socket.connect("tcp://192.168.0.6:8577")
self.socket.setsockopt(zmq.SUBSCRIBE, b'')
self.queue = queue
def run(self):
while True:
msg = self.socket.recv()
self.queue.put(msg)
queue = Queue()
c = TestClass(queue)
c.run()
# Do something else
If I use c.run()
to start a process, it works fine, but it doesn't start as a process because it blocks the next statement.
Then I switched to c.start()
to start the process, but it is stuck on the line socket.recv()
and cannot receive incoming messages. Can anyone explain this and suggest a good solution? Thanks to
source to share
The problem is that you create a socket zmq
in the parent process, but then try to use it in the child. Something in the forking process is breaking the socket, so it doesn't work when you try to use it. You can fix this by simply creating the socket on the child and not the parent. This has no negative side effects as you are not trying to use the socket on the parent to begin with.
from multiprocessing import Process, Queue
import zmq
import time
class TestClass(Process):
def __init__(self, queue):
super(TestClass, self).__init__()
self.queue = queue
def run(self):
# Setting up connections
self.context = zmq.Context()
self.socket = self.context.socket(zmq.SUB)
self.socket.connect("tcp://192.168.0.6:8577")
self.socket.setsockopt(zmq.SUBSCRIBE, b'')
while True:
msg = self.socket.recv()
self.queue.put(msg)
if __name__ == "__main__":
queue = Queue()
c = TestClass(queue)
c.start() # Don't use run()
# Do something else
source to share