Dump a non-const but not copyable element in Boost Python

I am having trouble reproducing your results, but here is some information that might help in solving the problem.

With simple classes:

class Signal
{
public:
  void connect() { std::cout << "connect called" << std::endl; }
private:
  boost::signals2::signal<void()> signal_;
};

class MyClass
{
public:
  Signal on_event;
};

      

        
        
        
      

    

And the main bindings:

namespace python = boost::python;
python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
  .def("connect", &Signal::connect)
  ;

python::class_<MyClass>("MyClass")
  .def_readonly("on_event", &MyClass::on_event)
  ;

      

        
        
        
      

    

The code fails to compile. When exposing a class, Boost.Python defaults to registering converters. These converters require copy constructors as a means of copying a C ++ class object into a store that can be manipulated by a Python object. This behavior can be disabled for a class by supplying it boost::noncopyable

as an argument for the type class_

.

In this case, binding MyClass

does not suppress copy constructors. Boost.Python will try to use copy constructors inside the bindings and fail with a compiler error as the member variable on_event

cannot be copied. Signal

not copied because it contains a member variable with a type boost::signal2::signal

that inherits from boost::noncopyable

.

Adding boost:::noncopyable

an argument to bindings as a type MyClass

allows the code to compile.

namespace python = boost::python;
python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
  .def("connect", &Signal::connect)
  ;

python::class_<MyClass, boost::noncopyable>("MyClass")
  .def_readonly("on_event", &MyClass::on_event)
  ;

      

        
        
        
      

    

Using:

>>> import example
>>> m = example.MyClass()
>>> m.on_event.connect()
connect called
>>> m.on_event = None
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: can't set attribute
>>> 

      

        
        
        
      

    

While this setup allows for the required bindings and invocation syntax, it looks like this is the first step in the final goal.

---

My apologies if this is too presumptuous. However, based on other recent questions, I'd like to take the time to expand on the original example to cover what appears to be the ultimate goal: the ability to hook Python callbacks to signal2::signal

. I will talk about two different approaches as the mechanics and difficulty level are different, but they can provide insight into the details to be considered for the final solution.

Python streams only.

For this first scenario, let's assume that only Python threads interact with the library.

One technique that makes it relatively easy is to use inheritance. Start by defining a helper class Slot

that can connect to Signal

.

class Slot
  : public boost::python::wrapper<Slot>
{
public:
  void operator()()
  {
    this->get_override("__call__")();
  }
};

      

        
        
        
      

    

The class Slot

inherits from a boost::python::wrapper

class that unintrusively provides hooks to allow Python classes to override functions in the base class.

When the callable type connects to boost::signals2::signal

, the signal can copy the argument into an internal list. Thus, it is important that the functor can extend the lifetime of the instance Slot

as long as it stays connected to Signal

. The easiest way to do this is to manage Slot

with boost::shared_ptr

.

The resulting class Signal

looks like this:

class Signal
{
public:
  template <typename Callback>
  void connect(const Callback& callback)
  {
    signal_.connect(callback);
  }

  void operator()() { signal_(); }
private:
  boost::signals2::signal<void()> signal_;
};

      

        
        
        
      

    

And the helper function helps to maintain Signal::connect

generic, use other C ++ types if needed.

void connect_slot(Signal& self, 
                  const boost::shared_ptr<Slot>& slot)
{
  self.connect(boost::bind(&Slot::operator(), slot));
}

      

        
        
        
      

    

This results in the following bindings:

BOOST_PYTHON_MODULE(example) {
  namespace python = boost::python;
  python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
    .def("connect",  &connect_slot)
    .def("__call__", &Signal::operator())
    ;

  python::class_<MyClass, boost::noncopyable>("MyClass")
    .def_readonly("on_event", &MyClass::on_event)
    ;

  python::class_<Slot, boost::shared_ptr<Slot>, 
                 boost::noncopyable>("Slot")
    .def("__call__", python::pure_virtual(&Slot::operator()))
    ;
}

      

        
        
        
      

    

And its usage looks like this:

>>> from example import *
>>> class Foo(Slot):
...     def __call__(self):
...          print "Foo::__call__"
... 
>>> m = MyClass()
>>> foo = Foo()
>>> m.on_event.connect(foo)
>>> m.on_event()
Foo::__call__
>>> foo = None
>>> m.on_event()
Foo::__call__

      

        
        
        
      

    

Despite its success, it has the unfortunate characteristic that it is not pythonic. For example:

>>> def spam():
...     print "spam"
... 
>>> m = MyClass()
>>> m.on_event.connect(spam)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    Signal.connect(Signal, function)
did not match C++ signature:
    connect(Signal {lvalue}, boost::shared_ptr<Slot>)

      

        
        
        
      

    

It would be ideal if any callable could be connected to the signal. One easy way to do this is for the monkey to fix the bindings in Python. To be transparent to the end user:

• Change the name of the C ++ binding module from example
  
  to _example
  
  . Make sure to change the name of the library.
• Create example.py
  
  which will fix Signal.connect()
  
  to wrap the argument in a type that inherits from Slot
  
  .

example.py

might look something like this:

from _example import *

class _SlotWrap(Slot):

    def __init__(self, fn):
        self.fn = fn
        Slot.__init__(self)

    def __call__(self):
        self.fn()

def _signal_connect(fn):
    def decorator(self, slot):
        # If the slot is not an instance of Slot, then aggregate it
        # in SlotWrap.
        if not isinstance(slot, Slot):
            slot = _SlotWrap(slot)
        # Invoke the decorated function with the slot.
        return fn(self, slot)
    return decorator

# Patch Signal.connect.
Signal.connect = _signal_connect(Signal.connect)

      

        
        
        
      

    

The end user patch is seamless.

>>> from example import *
>>> def spam():
...     print "spam"
... 
>>> m = MyClass()
>>> m.on_event.connect(spam)
>>> m.on_event()
spam

      

        
        
        
      

    

With this patch, any callable type can connect to Signal

without explicitly inheriting from Slot

. Thus, it becomes much more pythonic than the original solution. Never underestimate the benefits of keeping the bindings simple and non-pythonic, but fix them like pythons in python.

---

Python and C ++ streams.

In the following scenario, consider a case where C ++ threads interact with Python. For example, a C ++ thread can be configured to trigger a signal after a while.

This example can get pretty attractive, so let's start with the basics: Python Global Interpreter Lock (GIL). In short, the GIL is a mutex around the translator. If the thread is doing anything that affects the reference counting of the python-based managed object, then it should get the GIL. In the previous example, since there were no C ++ threads, everything was done during the acquisition of the GIL. While this is pretty straight forward, it can get quite complex.

First, the module must have Python to initialize the GIL for streaming.

BOOST_PYTHON_MODULE(example) {
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.
  ...
}

      

        
        
        
      

    

For convenience, create a simple class to help manage the GIL:

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};

      

        
        
        
      

    

A signal will be triggered on the stream MyClass

. Thus, it should extend the lifetime MyClass

while the stream is alive. A good candidate for this purpose is the management of MyClass

using shared_ptr

.

Lets you determine when a C ++ thread will need the GIL:

• MyClass
  
  removed shared_ptr
  
  .
• boost::signals2::signal
  
  can create additional copies of connected objects, as is done when called simultaneously .
• Calling a Python object linked through boost::signals2::signal
  
  . The callback will certainly affect python objects. For example, an argument self
  
  provided to a method __call__
  
  will increment and decrement the object reference count.

Support MyClass

is removed from the C ++ stream.

A custom debugger is required to ensure that the GIL is held when MyClass

removed shared_ptr

from the C ++ stream . This also requires a binding to suppress the default constructor and use a custom constructor instead.

/// @brief Custom deleter.
template <typename T>
struct py_deleter
{
  void operator()(T* t)
  {
    gil_lock lock;    
    delete t;
  }
};

/// @brief Create Signal with a custom deleter.
boost::shared_ptr<MyClass> create_signal()
{
  return boost::shared_ptr<MyClass>(
    new MyClass(),
    py_deleter<MyClass>());
}

...

BOOST_PYTHON_MODULE(example) {

  ...

  python::class_<MyClass, boost::shared_ptr<MyClass>,
                 boost::noncopyable>("MyClass", python::no_init)
    .def("__init__", python::make_constructor(&create_signal))
    .def_readonly("on_event", &MyClass::on_event)
    ;
}

      

        
        
        
      

    

The flow itself.

The thread functionality is pretty simple: it sleeps, then triggers a signal. However, it is important to understand the context of the GIL.

/// @brief Wait for a period of time, then invoke the
///        signal on MyClass.
void call_signal(boost::shared_ptr<MyClass>& shared_class,
                 unsigned int seconds)
{
  // The shared_ptr was created by the caller when the GIL was
  // locked, and is accepted as a reference to avoid modifying
  // it while the GIL is not locked.

  // Sleep without the GIL so that other python threads are able
  // to run.
  boost::this_thread::sleep_for(boost::chrono::seconds(seconds));

  // We do not want to hold the GIL while invoking C++-specific
  // slots connected to the signal.  Thus, it is the responsibility of
  // python slots to lock the GIL.  Additionally, the potential
  // copying of slots internally by the signal will be handled through
  // another mechanism.
  shared_class->on_event();

  // The shared_class has a custom deleter that will lock the GIL
  // when deletion needs to occur.
}

/// @brief Function that will be exposed to python that will create
///        a thread to call the signal.
void spawn_signal_thread(boost::shared_ptr<MyClass> self,
                       unsigned int seconds)
{
  // The caller owns the GIL, so it is safe to make copies.  Thus,
  // spawn off the thread, binding the arguments via copies.  As
  // the thread will not be joined, detach from the thread.
  boost::thread(boost::bind(&call_signal, self, seconds)).detach();
}

      

        
        
        
      

    

And the bindings MyClass

are updated.

python::class_<MyClass, boost::shared_ptr<MyClass>,
               boost::noncopyable>("MyClass", python::no_init)
  .def("__init__", python::make_constructor(&create_signal))
  .def("signal_in", &spawn_signal_thread)
  .def_readonly("on_event", &MyClass::on_event)
  ;

      

        
        
        
      

    

boost::signals2::signal

interacts with python objects.

boost::signals2::signal

can create copies when called. Also, C ++ slots can be connected to the signal, so it would be ideal that the GIL does not block when the signal is called. However, it Signal

does not provide interceptors so that we can get the GIL before creating copies of slots or invoking a slot.

To add to the complexity, when the bindings map to a C ++ function that accepts a C ++ class HeldType

that is not a smart pointer, then Boost.Python will retrieve a C ++ object that is not referencing a referenced object, counted the python object. This can be safely done because the calling thread in Python has a GIL. To maintain reference counting on slots trying to connect to Python, and also allow any callable type to connect, we can use an opaque type boost::python::object

.

To avoid making Signal

copies of what is provided boost::python::object

, you can create a copy boost::python::object

so that the reference counting remains accurate and manages the copy across shared_ptr

. This allows Signal

you to freely create copies shared_ptr

instead of creating boost::python::object

without the GIL.

This GIL security slot can be encapsulated in a helper class.

/// @brief Helper type that will manage the GIL for a python slot.
class py_slot
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_slot(const boost::python::object& object)
    : object_(new boost::python::object(object),   // GIL locked, so copy.
              py_deleter<boost::python::object>()) // Delete needs GIL.
  {}

  void operator()()
  {
    // Lock the gil as the python object is going to be invoked.
    gil_lock lock;
    (*object_)(); 
  }

private:
  boost::shared_ptr<boost::python::object> object_;
};

      

        
        
        
      

    

A helper function will be available to Python for type adaptation.

/// @brief Signal connect helper.
void signal_connect(Signal& self,
                    boost::python::object object)
{
  self.connect(boost::bind(&py_slot::operator(), py_slot(object)));
}

      

        
        
        
      

    

And the updated binding opens a helper function:

python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
  .def("connect",  &signal_connect)
  .def("__call__", &Signal::operator())
  ;

      

        
        
        
      

    

The final solution looks like this:

#include <boost/bind.hpp>
#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/signals2/signal.hpp>
#include <boost/thread.hpp>

class Signal
{
public:
  template <typename Callback>
  void connect(const Callback& callback)
  {
    signal_.connect(callback);
  }

  void operator()() { signal_(); }
private:
  boost::signals2::signal<void()> signal_;
};

class MyClass
{
public:
  Signal on_event;
};

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};    

/// @brief Custom deleter.
template <typename T>
struct py_deleter
{
  void operator()(T* t)
  {
    gil_lock lock;    
    delete t;
  }
};

/// @brief Create Signal with a custom deleter.
boost::shared_ptr<MyClass> create_signal()
{
  return boost::shared_ptr<MyClass>(
    new MyClass(),
    py_deleter<MyClass>());
}

/// @brief Wait for a period of time, then invoke the
///        signal on MyClass.
void call_signal(boost::shared_ptr<MyClass>& shared_class,
                 unsigned int seconds)
{
  // The shared_ptr was created by the caller when the GIL was
  // locked, and is accepted as a reference to avoid modifying
  // it while the GIL is not locked.

  // Sleep without the GIL so that other python threads are able
  // to run.
  boost::this_thread::sleep_for(boost::chrono::seconds(seconds));

  // We do not want to hold the GIL while invoking C++-specific
  // slots connected to the signal.  Thus, it is the responsibility of
  // python slots to lock the GIL.  Additionally, the potential
  // copying of slots internally by the signal will be handled through
  // another mechanism.
  shared_class->on_event();

  // The shared_class has a custom deleter that will lock the GIL
  // when deletion needs to occur.
}

/// @brief Function that will be exposed to python that will create
///        a thread to call the signal.
void spawn_signal_thread(boost::shared_ptr<MyClass> self,
                       unsigned int seconds)
{
  // The caller owns the GIL, so it is safe to make copies.  Thus,
  // spawn off the thread, binding the arguments via copies.  As
  // the thread will not be joined, detach from the thread.
  boost::thread(boost::bind(&call_signal, self, seconds)).detach();
}

/// @brief Helepr type that will manage the GIL for a python slot.
struct py_slot
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_slot(const boost::python::object& object)
    : object_(new boost::python::object(object),   // GIL locked, so copy.
              py_deleter<boost::python::object>()) // Delete needs GIL.
  {}

  void operator()()
  {
    // Lock the gil as the python object is going to be invoked.
    gil_lock lock;
    (*object_)(); 
  }

private:
  boost::shared_ptr<boost::python::object> object_;
};

/// @brief Signal connect helper.
void signal_connect(Signal& self,
                    boost::python::object object)
{
  self.connect(boost::bind(&py_slot::operator(), py_slot(object)));
}

BOOST_PYTHON_MODULE(example) {
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.

  namespace python = boost::python;
  python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
    .def("connect",  &signal_connect)
    .def("__call__", &Signal::operator())
    ;

  python::class_<MyClass, boost::shared_ptr<MyClass>,
                 boost::noncopyable>("MyClass", python::no_init)
    .def("__init__", python::make_constructor(&create_signal))
    .def("signal_in", &spawn_signal_thread)
    .def_readonly("on_event", &MyClass::on_event)
    ;
}

      

        
        
        
      

    

And testing script ( test.py

):

from time import sleep
from example import *

def spam():
    print "spam"

m = MyClass()
m.on_event.connect(spam)
m.on_event()

m.signal_in(2)
m = None
print "Sleeping"
sleep(5)
print "Done sleeping"

      

        
        
        
      

    

The results are as follows:

spam
Sleeping
spam
Done sleeping

---

Finally, when an object is passed through the Boost.Python layer, take some time to consider how to manage its lifespan and the context in which it will be used. This often requires an understanding of how the other libraries in use will handle the object. This is not an easy task and a pythonic solution could be a problem.