How to run generator code in parallel?
I have code like this:
def generator():
while True:
# do slow calculation
yield x
I would like to move the slow calculation in order to split the process (s).
I am working in python 3.6, so I have concurrent.futures.ProcessPoolExecutor
. It is just not clear how to use a parallel generator with this.
The differences from the usual parallel script using map
is that there is nothing to display here (the generator runs forever) and we don't want all the results at once, we want to queue them up and wait for the queue to not fill up before calculating more results ...
I don't need to use concurrent
, multiprocessing
fine too. This is a similar problem, it is not clear how to use this inside a generator.
Minor twist: each value returned by the generator is an array with a lot (10 megabytes or so). How to convey this without etching and sawing? I've seen the docs for multiprocessing.Array
, but it's not entirely obvious how to pass a numpy array using this.
source to share
In this type of situation, I usually use the joblib library . It is a parallel computing environment based on multiprocessing. It supports memmapping exactly for cases where you have to handle large numpy arrays. I think it's worth checking you out.
Maybe the joblib documentation is not clear enough on this, showing only examples with loops, since you want to use a generator, I must point out that it actually works with generators. An example that will achieve what you want is the following:
from joblib import Parallel, delayed
def my_long_running_job(x):
# do something with x
# you can customize the number of jobs
Parallel(n_jobs=4)(delayed(my_long_running_job)(x) for x in generator())
Edit: I don't know what kind of processing you want to do, but if it releases the GIL, you might consider using streams as well. This way you will have no problem carrying large numpy arrays between processes and still use true parallelism.
source to share