Using OMP_NUM_THREADS = 1 for Python Multiprocessing
Since you said in a comment that your Python program is calling a C module that uses OpenMP :
OpenMP does multithreading within a process, and the default thread count is the number that the CPU can actually run concurrently. (This is usually the number of CPU cores, or a multiple of that number if the processor has an SMT feature like Intel Hyper-Threading.) So, if you have, for example, a quad-core processor without hyper-threading, OpenMP wants to run 4 threads by default ...
When you use a Python module multiprocessing
, your program starts multiple Python processes that can run concurrently. You can control the number of processes, but often you want it to be the number of cores or CPU threads. returns multiprocessing.cpu_count()
.
So what happens on this quad-core processor if you run a program multiprocessing
that starts 4 Python processes and each call to an OpenMP function starts 4 threads? As a result, you start 16 threads on 4 cores. This will work, but not with maximum efficiency, as each core will have to spend some time switching between tasks.
The setting OMP_NUM_THREADS=1
basically disables OpenMP multithreading, so each of your Python processes remains single-threaded.
Make sure you run enough Python processes if you do! If you have 4 processor cores and only run 2 single-threaded Python processes, you will have 2 cores and the other 2 are idle. (In that case, you can install OMP_NUM_THREADS=2
.)
source to share
Solved in the comments:
OMP_NUM_THREADS is an option for OpenMP, C / C ++ / Fortran API to perform multithreading in a process.
It is not clear how this even relates to Python multiprocessing.
Is your Python programming program modules written in C that use OpenMP internally? - Wyzard
source to share