Multithreading PySide / Cython and the GIL

I have struggled with PySide / Cython multi-threaded application for many days .
Separating concerns into parts, I came up with a test I run with valgrind / helgrind (see one comment below).
Function C (CHLone load

below) uses Python with its GIL (no directive for Cython), Shiboken also accesses the GIL.
The results show that there is a conflict with GIL access (if I'm not mistaken), but my understanding is that the GIL was there to avoid such a conflict.

Shiboken is doing some dealloc that collides with PyList_New

in another thread ...

I am lost,
I thought the GIL was in charge of blocking PyList_New

and another thread was modifying some shared data in Python. Or forget to block Shiboken ?

Line listobject.c

161 is the call _PyObject_GC_TRACK()

that (I suppose) asked the GC to keep track of the new object.
Line classobject.c

2360 is a call _PyObject_GC_UNTRACK()

that strongly resembles a GC request to stop tracking an object ...
I see that the helgrind dialog is a possible data race, but it actually results in a core dump, and I don't like this possible Python GC word when using streams. I would like to fix this first.

==26535== ----------------------------------------------------------------
==26535== 
==26535== Possible data race during read of size 8 at 0x4FE8488 by thread #2
==26535== Locks held: none
==26535==    at 0x4C92A80: PyList_New (listobject.c:161)
==26535==    by 0x742C43F: s2p_parseAndReadHDF (SIDStoPython.c:949)
==26535==    by 0x742C5C4: s2p_parseAndReadHDF (SIDStoPython.c:968)
==26535==    by 0x742E638: s2p_loadAsHDF (SIDStoPython.c:1485)
==26535==    by 0x741C3CC: __pyx_f_6CHLone_load (pyCHLone.c:2182)
==26535==    by 0x741D2AD: __pyx_pf_6CHLone_12load (pyCHLone.c:2422)
==26535==    by 0x741D1C3: __pyx_pw_6CHLone_13load (pyCHLone.c:2392)
==26535==    by 0x4D0A48F: PyEval_EvalFrameEx (ceval.c:4013)
==26535==    by 0x4D0C3DC: PyEval_EvalCodeEx (ceval.c:3253)
==26535==    by 0x4C8B641: function_call (funcobject.c:526)
==26535==    by 0x4C5F652: PyObject_Call (abstract.c:2529)
==26535==    by 0x4C7279E: instancemethod_call (classobject.c:2578)
==26535== 
==26535== This conflicts with a previous write of size 8 by thread #1
==26535== Locks held: none
==26535==    at 0x4C6C53F: instancemethod_dealloc (classobject.c:2360)
==26535==    by 0x5AB248A: Shiboken::AutoDecRef::~AutoDecRef() (in /tmp/tools-2/local/x86z/lib/python2.7/site-packages/PySide/QtCore.so)
==26535==    by 0x5F9736F: PySide::GlobalReceiverV2::qt_metacall(QMetaObject::Call, int, void**) (in /tmp/tools-2/local/x86z/lib/libpyside-python2.7.so.1.0.9)
==26535==    by 0x659BCA5: QObject::event(QEvent*) (in /tmp/tools-2/local/x86z/lib/libQtCore.so.4.8.0)
==26535==    by 0x5B038C5: QCoreApplicationWrapper::notify(QObject*, QEvent*) (in /tmp/tools-2/local/x86z/lib/python2.7/site-packages/PySide/QtCore.so)
==26535==    by 0x6586F8B: QCoreApplication::notifyInternal(QObject*, QEvent*) (in /tmp/tools-2/local/x86z/lib/libQtCore.so.4.8.0)
==26535==    by 0x658A5A7: QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) (in /tmp/tools-2/local/x86z/lib/libQtCore.so.4.8.0)
==26535==    by 0x65B60F2: ??? (in /tmp/tools-2/local/x86z/lib/libQtCore.so.4.8.0)
==26535==

      

Questions:
 1. Is the behavior I accept for the GIL correct?

 2. If No ,
... Does this mean that I have to manage the GIL locking on each branch myself?
... Then I need to create my own Python mutex?! ??!?
 3. If Yes ,
... Is there a problem in my Cython or C lib regarding the use of the GIL?
... Could this be a side effect of QThread / QMutex?

+3


source to share


1 answer


In CPython, a global interpreter lock, or GIL, is a mutex that prevents multiple native threads of Python bytecode from executing concurrently.

Thus, all you can expect from the GIL is that it runs two threads of Python code at the same time. You should not expect any blocking action. It is generally bad practice to write code that depends on the GIL to protect you. The GIL is a wart that many want to remove, and it doesn't exist in some forms of python like IronPython and jython. Where it exists, it reduces concurrency, imposes overhead, and makes python code more complex than it would otherwise be.



It is not clear to me from the above that there is any GIL failure. The second trail seems to be very deep in the land of C ++, not in areas that I understand the GIL is limiting.

+2


source







All Articles