Python: exit to multiprocessing pool
I have to Parallelize a function that turns on a specific "output". This is just a simple copy of the entire program that I have to work on, but sums up the problems I am facing. Here I am trying to understand multiprocessing, apply_async and profitability for my project In this example, I used multiprocessing.pool and used apply_async to parallelize. I have put several print statements in a "parallel" function, but they don't print. When I return a yield with a refund, the seal approvals are reflected. I'm not sure about the nature of the crop. I know his generator and can only be used once after returning it. Please advise how to do this.
import multiprocessing as mp
results=[]
def parallel(x, y, z):
print "aim in parallel"
count=0
result=[]
for line in range(10000):
count+=1
result.append(count)
p=x**3+y+z
print " result"
print result
print p
if p > 0:
return result
# yield result, p
# count += 1
# yield p, result
# count += 1
def collect_results(result):
print "aim in callback"
results.append(result)
#print results
def apply_async_with_callback():
pool = mp.Pool(processes=10)
r = range(10)
[pool.apply_async(parallel, args=(2,5, 7),callback=collect_results) for i in r ]
pool.close()
pool.join()
print "length"
print len(results)
print results
if __name__ == "__main__":
apply_async_with_callback()
source to share
When a function containing a statement is called yield
, it doesn't actually run the code, but instead returns a generator:
>>> p = parallel(1, 2, 3)
>>> p
<generator object parallel at 0x7fde9c1daf00>
Then, when the next value is required, the code will run until the value is received:
>>> next(p)
([10000], 6)
>>> next(p)
(6, [10000])
In your case, results
contains 10 generators that were created asynchronously, but they were never executed.
If you want to use a generator, you can slightly modify your code for the target function that creates a list from the generator:
def parallel2(x, y, z):
return list(parallel(x, y, z))
def collect_results(lst):
results.extend(lst)
def apply_async_with_callback():
pool = mp.Pool()
for _ in range(10):
pool.apply_async(parallel2, args=(2, 5, 7),
callback=collect_results)
source to share