Boost python: bind argument lifetime to return value using return_internal_reference
I am starting to learn to use boost python and ask rookie.
I would like to write a function that can bind the lifetime of its argument to its results, so that when I call r = func(a)
, the argument a
will never be destroyed if I still have a link r
, The documentation suggests using a call policy return_internal_reference
for this type of request. But does it require it to r
be an internal link a
, as the title suggests?
In the (more simplified) example below, let's say I want to associate the lifetime of an input array a
with a generated lambda function that is not an internal input reference a
.
#include <functional>
#include <boost/python.hpp>
#include <boost/python/return_internal_reference.hpp>
using namespace std;
using namespace boost::python;
function<float(int)> func(const float* a) {
return [=](int n) { return a[n]; };
}
BOOST_PYTHON_MODULE(test) {
def("func", func, return_internal_reference<1>());
}
I hope I can do the following in python:
f = func(a) # 'a' can be a temporary variable, say returned by another function
f(5) # but 'a' should not be destroyed at this step,
# because its lifetime is tied to 'f'
When I tried to compile the above code, I got a wall of errors listed below, but if I remove the invocation policy return_internal_reference<1>()
, the code compiles successfully.
I'm sure I used this calling policy incorrectly, but not sure how to get it right. Any pointer would be much appreciated. Many thanks!
$ g++ -std=c++11 -shared Test.cc -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -L/opt/local/lib -lboost_python-mt -lpython2.7 -o test.so
In file included from /opt/local/include/boost/preprocessor/iteration/detail/iter/forward1.hpp:52:0,
from /opt/local/include/boost/python/detail/invoke.hpp:63,
from /opt/local/include/boost/python/detail/caller.hpp:16,
from /opt/local/include/boost/python/object/function_handle.hpp:8,
from /opt/local/include/boost/python/converter/arg_to_python.hpp:19,
from /opt/local/include/boost/python/call.hpp:15,
from /opt/local/include/boost/python/object_core.hpp:14,
from /opt/local/include/boost/python/args.hpp:25,
from /opt/local/include/boost/python.hpp:11,
from Test.cc:2:
/opt/local/include/boost/python/detail/invoke.hpp: In instantiation of 'PyObject* boost::python::detail::invoke(boost::python::detail::invoke_tag_<false, false>, const RC&, F&, AC0&) [with RC = boost::python::detail::reference_existing_object_requires_a_pointer_or_reference_return_type<std::function<float(int)> >; F = std::function<float(int)> (*)(const float*); AC0 = boost::python::arg_from_python<const float*>; PyObject = _object]':
/opt/local/include/boost/python/detail/caller.hpp:223:13: required from 'PyObject* boost::python::detail::caller_arity<1u>::impl<F, Policies, Sig>::operator()(PyObject*, PyObject*) [with F = std::function<float(int)> (*)(const float*); Policies = boost::python::return_internal_reference<>; Sig = boost::mpl::vector2<std::function<float(int)>, const float*>; PyObject = _object]'
/opt/local/include/boost/python/object/py_function.hpp:38:33: required from 'PyObject* boost::python::objects::caller_py_function_impl<Caller>::operator()(PyObject*, PyObject*) [with Caller = boost::python::detail::caller<std::function<float(int)> (*)(const float*), boost::python::return_internal_reference<>, boost::mpl::vector2<std::function<float(int)>, const float*> >; PyObject = _object]'
Test.cc:14:1: required from here
/opt/local/include/boost/python/detail/invoke.hpp:75:82: error: no match for call to '(const boost::python::detail::reference_existing_object_requires_a_pointer_or_reference_return_type<std::function<float(int)> >) (std::function<float(int)>)'
return rc(f( BOOST_PP_ENUM_BINARY_PARAMS_Z(1, N, ac, () BOOST_PP_INTERCEPT) ));
^
In file included from /opt/local/include/boost/python/object/function_handle.hpp:8:0,
from /opt/local/include/boost/python/converter/arg_to_python.hpp:19,
from /opt/local/include/boost/python/call.hpp:15,
from /opt/local/include/boost/python/object_core.hpp:14,
from /opt/local/include/boost/python/args.hpp:25,
from /opt/local/include/boost/python.hpp:11,
from Test.cc:2:
/opt/local/include/boost/python/detail/caller.hpp: In instantiation of 'static const PyTypeObject* boost::python::detail::converter_target_type<ResultConverter>::get_pytype() [with ResultConverter = boost::python::detail::reference_existing_object_requires_a_pointer_or_reference_return_type<std::function<float(int)> >; PyTypeObject = _typeobject]':
/opt/local/include/boost/python/detail/caller.hpp:240:19: required from 'static boost::python::detail::py_func_sig_info boost::python::detail::caller_arity<1u>::impl<F, Policies, Sig>::signature() [with F = std::function<float(int)> (*)(const float*); Policies = boost::python::return_internal_reference<>; Sig = boost::mpl::vector2<std::function<float(int)>, const float*>]'
/opt/local/include/boost/python/object/py_function.hpp:48:35: required from 'boost::python::detail::py_func_sig_info boost::python::objects::caller_py_function_impl<Caller>::signature() const [with Caller = boost::python::detail::caller<std::function<float(int)> (*)(const float*), boost::python::return_internal_reference<>, boost::mpl::vector2<std::function<float(int)>, const float*> >]'
Test.cc:14:1: required from here
/opt/local/include/boost/python/detail/caller.hpp:102:109: error: 'struct boost::python::detail::reference_existing_object_requires_a_pointer_or_reference_return_type<std::function<float(int)> >' has no member named 'get_pytype'
return create_result_converter((PyObject*)0, (ResultConverter *)0, (ResultConverter *)0).get_pytype();
^
source to share
With CallPolicy . This policy allows return types to be returned by value, while extending the life of another object to at least the same as for the returned object.with_custodian_and_ward_postcall
BOOST_PYTHON_MODULE(test) {
def("func", func, with_custodian_and_ward_postcall<0, 1>());
}
As noted in the documentation return_internal_reference
, the object returned references to an existing internal object:
return_internal_reference
[...] allow you to specify pointers and references to objects stored inside [...] so that they can be returned safely without creating a copy of the referent.
The documentation also briefly mentions its use with_custodian_and_ward_postcall
. Thus, it return_internal_reference
has two notable effects on the displayed function:
- The returned Python object has no explicit or shared access to the C ++ referenced object.
- The returned Python object is a custodian and extends the lifetime of the arrival object designated
owner_arg
at least as long as the custodian.
Since the Python object returned is an existing object for which it is neither explicitly nor shared, Boost.Python performs type checking to prevent dangling reference creation. The code example func()
returns a functor by value, which results in a compiler error indicating that the return type must be either a pointer or a reference:
struct boost :: python :: detail :: reference_existing_object_requires_a_pointer_or_reference_return_type
To explicitly control the lifetime of the returned object, you should consider using return_value_policy
and models ResultConverterGenerators . For example, if I func()
returned a functor with a pointer created new()
and wanted to transfer ownership of the object to Python while maintaining the custodian-custodian relationship, then one could bind the policies using the policy composition :
BOOST_PYTHON_MODULE(test) {
def("func", func,
return_value_policy<manage_new_object,
with_custodian_and_ward_postcall<0, 1> >());
}
Below is a complete minimal example based on source code with verbose output to demonstrate behavior with_custodian_and_ward_postcall
:
#include <boost/python.hpp>
#include <iostream>
/// @brief Mockup class with verbose construction and destruction.
class foo
{
public:
foo() { std::cout << "foo() " << this << std::endl; }
foo(const foo&) { std::cout << "foo(const foo&) " << this << std::endl; }
~foo() { std::cout << "~foo() " << this << std::endl; }
};
/// @brief Mockup class with verbose construction and destruction.
class bar
{
public:
bar() { std::cout << "bar() " << this << std::endl; }
bar(const bar&) { std::cout << "bar(const bar&) " << this << std::endl; }
~bar() { std::cout << "~bar() " << this << std::endl; }
};
/// @brief Mockup factory function.
foo make_foo(bar& /* unused */)
{
return foo();
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Do not allow Foo to be explicitly created from its type.
python::class_<foo>("Foo", python::no_init);
python::class_<bar>("Bar", python::init<>());
// Expose make_foo, that returns a foo object when provided a
// bar object. The bar object lifetime will be extended to
// be at least as long as that of the returned foo object.
python::def("make_foo", &make_foo,
python::with_custodian_and_ward_postcall<
0, // custodian = returned Foo object
1 // ward = provided Bar object
>());
}
Interactive use:
>>> import example
>>> bar = example.Bar()
bar() 0x125ac30
>>> foo = example.make_foo(bar)
foo() 0x7fffa9b5efff
foo(const foo&) 0x7f1fcbe40090
~foo() 0x7fffa9b5efff
>>> bar = None
>>> foo = None
~foo() 0x7f1fcbe40090
~bar() 0x125ac30
Note that even though the variable is bar
set to None
, the actual arrival object remains alive until the keeper foo
is destroyed.
source to share