Boost.Python and Boost.Signals2: segmentation errors

If MyClass::some_later_event()

called from a C ++ thread that does not explicitly control the Global Interpreter Lock (GIL), then it could result in undefined mode.

Python and C ++ streams.

Let's look at a case where C ++ threads interact with Python. For example, a C ++ thread can be configured to call a signal MyClass

some time later MyClass.event_in(seconds, value)

.

This example can get pretty pretty, so let's start with the basics: the Python GIL. In short, the GIL is a mutex around the translator. If the thread is doing anything that affects the reference counting of the python-based managed object, then it should get the GIL. In the GDB trace, the Boost.Signals2 library was probably trying to call a non-GIL Python object, which resulted in a crash. While managing the GIL is fairly simple, it can get quite complex.

First, the module must have Python to initialize the GIL for streaming.

BOOST_PYTHON_MODULE(example)
{
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.
  // ...
}

      

        
        
        
      

    

For convenience, let's create a simple class to help manage the GIL across the scope:

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};

      

        
        
        
      

    

Lets you determine when a C ++ thread will need the GIL:

• boost::signals2::signal
  
  can create additional copies of connected objects, as is done when called.
• Calling a Python object linked through boost::signals2::signal
  
  . The callback will certainly affect python objects. For example, an argument self
  
  provided to a method __call__
  
  will increment and decrement the object reference count.

Class MyClass

.

Here is a basic layout class based on the source code:

/// @brief Mockup class.
class MyClass
{
public:
  /// @brief Connect a slot to the signal.
  template <typename Slot>
  void connect_slot(const Slot& slot)
  {
    signal_.connect(slot);
  }

  /// @brief Send an event to the signal.
  void event(int value)
  {
    signal_(value);
  }

private:
  boost::signals2::signal<void(int)> signal_;
};

      

        
        
        
      

    

Since a C ++ thread can trigger a signal MyClass

, the lifetime MyClass

should be at least as long as the thread. A good candidate for this is that Boost.Python manages MyClass

with boost::shared_ptr

.

BOOST_PYTHON_MODULE(example)
{
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.

  namespace python = boost::python;
  python::class_<MyClass, boost::shared_ptr<MyClass>,
                 boost::noncopyable>("MyClass")
    .def("event", &MyClass::event)
    // ...
    ;
}

      

        
        
        
      

    

boost::signals2::signal

interacts with python objects.

boost::signals2::signal

can make copies when called. Also, C ++ slots can be connected to the signal, so it would be ideal to block the GIL only when calling Python slots. However, it signal

does not provide interceptors so that we can get the GIL before making copies of slots or invoking a slot.

To avoid signal

creating copies of slots boost::python::object

, you can use a wrapper class that creates a copy boost::python::object

to keep the reference counting accurate and manage the copy across shared_ptr

. This allows signal

free shared_ptr

copying instead of copying boost::python::object

without the GIL.

This GIL security slot can be encapsulated in a helper class.

/// @brief Helepr type that will manage the GIL for a python slot.
///
/// @detail GIL management:
///           * Caller must own GIL when constructing py_slot, as 
///             the python::object will be copy-constructed (increment
///             reference to the object)
///           * The newly constructed python::object will be managed
///             by a shared_ptr.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion.
///           * When py_slot is invoked (operator()), it will acquire
///             the GIL then delegate to the managed python::object.
struct py_slot
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_slot(const boost::python::object& object)
    : object_(
        new boost::python::object(object),  // GIL locked, so copy.
        [](boost::python::object* object)   // Delete needs GIL.
        {
          gil_lock lock;
          delete object;
        }
      )
  {}

  // Use default copy-constructor and assignment-operator.
  py_slot(const py_slot&) = default;
  py_slot& operator=(const py_slot&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    gil_lock lock;
    (*object_)(args...); 
  }

private:
  boost::shared_ptr<boost::python::object> object_;
};

      

        
        
        
      

    

A helper function will be available to Python for type adaptation.

/// @brief MyClass::connect_slot helper.
template <typename ...Args>
void MyClass_connect_slot(
  MyClass& self,
  boost::python::object object)
{
  py_slot slot(object); // Adapt object to a py_slot for GIL management.

  // Using a lambda here allows for the args to be expanded automatically.
  // If bind was used, the placeholders would need to be explicitly added.
  self.connect_slot([slot](Args... args) mutable { slot(args...); });
}

      

        
        
        
      

    

And the updated binding opens a helper function:

python::class_<MyClass, boost::shared_ptr<MyClass>,
               boost::noncopyable>("MyClass")
  .def("connect_slot", &MyClass_connect_slot<int>)
  .def("event",        &MyClass::event)
  // ...
  ;

      

        
        
        
      

    

The flow itself.

The thread functionality is pretty simple: it sleeps, then triggers a signal. However, it is important to understand the context of the GIL.

/// @brief Sleep then invoke an event on MyClass.
template <typename ...Args>
void MyClass_event_in_thread(
  boost::shared_ptr<MyClass> self,
  unsigned int seconds,
  Args... args)
{
  // Sleep without the GIl.
  std::this_thread::sleep_for(std::chrono::seconds(seconds));

  // We do not want to hold the GIL while invoking or copying 
  // C++-specific slots connected to the signal.  Thus, it is the 
  // responsibility of python slots to manage the GIL via the 
  // py_slot wrapper class.
  self->event(args...);
}

/// @brief Function that will be exposed to python that will create
///        a thread to call the signal.
template <typename ...Args>
void MyClass_event_in(
  boost::shared_ptr<MyClass> self,
  unsigned int seconds,
  Args... args)
{
  // The caller may or may not have the GIL.  Regardless, spawn off a 
  // thread that will sleep and then invoke an event on MyClass.  The
  // thread will not be joined so detach from it.  Additionally, as
  // shared_ptr is thread safe, copies of it can be made without the
  // GIL.
  std::thread(&MyClass_event_in_thread<Args...>, self, seconds, args...)
      .detach();
}

      

        
        
        
      

    

<sub>Note that it MyClass_event_in_thread

can be expressed as a lambda, but unpacking the template package into a lambda does not work on some compilers.</sub>

Updated bindings MyClass

.

python::class_<MyClass, boost::shared_ptr<MyClass>,
               boost::noncopyable>("MyClass")
  .def("connect_slot", &MyClass_connect_slot<int>)
  .def("event",        &MyClass::event)
  .def("event_in",     &MyClass_event_in<int>)
  ;

      

        
        
        
      

    

The final solution looks like this:

#include <thread> // std::thread, std::chrono
#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/signals2/signal.hpp>

/// @brief Mockup class.
class MyClass
{
public:
  /// @brief Connect a slot to the signal.
  template <typename Slot>
  void connect_slot(const Slot& slot)
  {
    signal_.connect(slot);
  }

  /// @brief Send an event to the signal.
  void event(int value)
  {
    signal_(value);
  }

private:
  boost::signals2::signal<void(int)> signal_;
};

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};    

/// @brief Helepr type that will manage the GIL for a python slot.
///
/// @detail GIL management:
///           * Caller must own GIL when constructing py_slot, as 
///             the python::object will be copy-constructed (increment
///             reference to the object)
///           * The newly constructed python::object will be managed
///             by a shared_ptr.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion.
///           * When py_slot is invoked (operator()), it will acquire
///             the GIL then delegate to the managed python::object.
struct py_slot
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_slot(const boost::python::object& object)
    : object_(
        new boost::python::object(object),  // GIL locked, so copy.
        [](boost::python::object* object)   // Delete needs GIL.
        {
          gil_lock lock;
          delete object;
        }
      )
  {}

  // Use default copy-constructor and assignment-operator.
  py_slot(const py_slot&) = default;
  py_slot& operator=(const py_slot&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    gil_lock lock;
    (*object_)(args...); 
  }

private:
  boost::shared_ptr<boost::python::object> object_;
};

/// @brief MyClass::connect_slot helper.
template <typename ...Args>
void MyClass_connect_slot(
  MyClass& self,
  boost::python::object object)
{
  py_slot slot(object); // Adapt object to a py_slot for GIL management.

  // Using a lambda here allows for the args to be expanded automatically.
  // If bind was used, the placeholders would need to be explicitly added.
  self.connect_slot([slot](Args... args) mutable { slot(args...); });
}

/// @brief Sleep then invoke an event on MyClass.
template <typename ...Args>
void MyClass_event_in_thread(
  boost::shared_ptr<MyClass> self,
  unsigned int seconds,
  Args... args)
{
  // Sleep without the GIL.
  std::this_thread::sleep_for(std::chrono::seconds(seconds));

  // We do not want to hold the GIL while invoking or copying 
  // C++-specific slots connected to the signal.  Thus, it is the 
  // responsibility of python slots to manage the GIL via the 
  // py_slot wrapper class.
  self->event(args...);
}

/// @brief Function that will be exposed to python that will create
///        a thread to call the signal.
template <typename ...Args>
void MyClass_event_in(
  boost::shared_ptr<MyClass> self,
  unsigned int seconds,
  Args... args)
{
  // The caller may or may not have the GIL.  Regardless, spawn off a 
  // thread that will sleep and then invoke an event on MyClass.  The
  // thread will not be joined so detach from it.  Additionally, as
  // shared_ptr is thread safe, copies of it can be made without the
  // GIL.
  // Note: MyClass_event_in_thread could be expressed as a lambda,
  //       but unpacking a template pack within a lambda does not work
  //       on some compilers.
  std::thread(&MyClass_event_in_thread<Args...>, self, seconds, args...)
      .detach();
}

BOOST_PYTHON_MODULE(example)
{
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.

  namespace python = boost::python;
  python::class_<MyClass, boost::shared_ptr<MyClass>,
                 boost::noncopyable>("MyClass")
    .def("connect_slot", &MyClass_connect_slot<int>)
    .def("event",        &MyClass::event)
    .def("event_in",     &MyClass_event_in<int>)
    ;
}

      

        
        
        
      

    

And the testing script:

from time import sleep
import example

def spam1(x):
  print "spam1: ", x

def spam2(x):
  print "spam2: ", x

c = example.MyClass()
c.connect_slot(spam1)
c.connect_slot(spam2)
c.event(123)
print "Sleeping"
c.event_in(3, 321)
sleep(5)
print "Done sleeping"

      

        
        
        
      

    

The results are as follows:

spam1:  123
spam2:  123
Sleeping
spam1:  321
spam2:  321
Done sleeping