Python multiprocessing - why am I getting so many threads per process?
I am using joblib (a wrapper for a multiprocessing package) to start a loop over some function over enumerated arguments. When I do htop
, I see that the number of processes is equal to the number of cpu counts ( n_jobs=-1
does this for you automatically). However, I also see that each process has as many threads as cpu_count - 1
... Is this expected? How does the second layer of parallelism come about and where is it from?
source to share
It looks like it is a problem with the backlind JobLib called Loky
which is used as the default JobLib, I had exactly the same problem and the performance increased significantly due to too many threads. To only use kernels and not threads, you must force joblib to be used multiprocessing
as a backend like this:
from joblib import Parallel, delayed
my_list_of_results = Parallel(n_jobs=-1, backend="multiprocessing")(delayed(my_function)(my_stuff, ) for my_stuff in whatever)
source to share