Clever Geek Handbook

Can I use map / imap / imap_unordered with functions without arguments?

Sometimes I need to use multiprocessing with functions without arguments. I wish I could do something like:

from multiprocessing import Pool

def f():  # no argument
    return 1

# TypeError: f() takes no arguments (1 given)
print Pool(2).map(f, range(10))

I could do Process(target=f, args=())

, but I prefer the map

/ imap

/ syntax imap_unordered

. Is there a way to do this?

+7


source to share


3 answers


map

The first argument function must be a function and must take one argument. This is required because the iterable passed as the second argument will be iterated over, and the values ​​will be passed to the function, one at each iteration.

So your best bet is to override f

to take one argument and ignore it, or write a wrapper function with one argument, ignore the argument and return a return value f

like this



from multiprocessing import Pool

def f():  # no argument
    return 1

def throw_away_function(_):
    return f()

print(Pool(2).map(throw_away_function, range(10)))
# [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

You cannot use functions lamdba

with pools because they are not matched.

+8


source


You can use pool.starmap()

instead .map()

like this:

from multiprocessing import Pool

def f():  # no argument
    return 1

print Pool(2).starmap(f, [() for _ in range(10)])


starmap

will pass all the iteration data items as arguments to f

. The iterations should be empty in your case.

+3


source


Is there something wrong with the use Pool.apply_async

?

with multiprocessing.Pool() as pool:
    future_results = [pool.apply_async(f) for i in range(n)]
    results = [f.get() for f in future_results]
+1


source






More articles:


All Articles