Seg due to multithreading (using boost libraries)
We have a program that uses both boost and sparse matrices and we are trying to integrate boost streams. However, when we go from single-threaded to multi-threaded application, we get segmentation errors that do not occur in the single-threaded case.
We are debugging using gdb (in Eclipse) and I found that seg failures occur during function calls in forced code i.e. I am trying to access a sparse matrix entry and the stack trace goes to the boost code and dies at some point (not always the same point) in these files.
I am confused because these matrices are allocated inside separate threads and all shared resources are protected by mutex locks. Also, I didn't think that streams were throwing seg errors often, just multiple access and bad data. However, I have obviously not experienced multi-threaded programming, so I was hoping that someone more experienced in this area could provide some advice.
I am using boost driven system and example compile command
g++ -DNDEBUG -I"/usr/local/include/boost-1_38" -I/usr/local/include -I"/home/scandido/workspace/BigSHOT/src" -I"/home/scandido/workspace/BigSHOT/src/base" -O3 -Wall -c -fmessage-length=0 `freetype-config --cflags` -pthread -MMD -MP -MF"src/utils/timing_info.d" -MT"src/utils/timing_info.d" -o"src/utils/timing_info.o" "../src/utils/timing_info.cpp"
and the linker command
g++ -L"/usr/local/lib" -L/usr/local/lib -o"BigSHOT" ./src/utils/timing_info.o ... many more objects ... ./src/base/pomdp/policy_fn/EventDriven.o ./src/base/pomdp/policy_fn/Greedy.o ./src/anotheralgorithm.o -lboost_serialization-gcc43-mt -lpthread -lboost_thread-gcc43-mt -lboost_program_options-gcc43-mt -lboost_iostreams-gcc43-mt -lpng -lpngwriter -lz -lfreetype
Here is the stack trace for a thread that has the following errors:
Thread [5] (Suspended: Signal 'SIGSEGV' received. Description: Segmentation fault.)
17 boost::numeric::ublas::mapped_matrix<bool, boost::numeric::ublas::basic_row_major<unsigned long, long>, boost::numeric::ublas::map_std<unsigned long, bool, std::allocator<std::pair<unsigned long const, bool> > > >::operator() /usr/local/include/boost-1_38/boost/numeric/ublas/matrix_sparse.hpp:377 0x000000000041c328
16 BigSHOT::Fire1FireState::get_cell() /home/scandido/workspace/BigSHOT/src/systems/fire1/pomdp/Fire1State.cpp:51 0x0000000000419a75
15 BigSHOT::Fire1SquareRegionProbObsFn::operator() /home/scandido/workspace/BigSHOT/src/systems/fire1/obs_fn/Fire1SquareRegionProbObsFn.cpp:92 0x000000000042ac37
14 BigSHOT::Fire1SquareRegionProbObsFn::operator() /home/scandido/workspace/BigSHOT/src/systems/fire1/obs_fn/Fire1SquareRegionProbObsFn.cpp:66 0x000000000042a8bf
13 BigSHOT::BayesFilterFn<BigSHOT::Fire1Belief, BigSHOT::Fire1State, BigSHOT::Fire1Action, BigSHOT::Fire1Observation>::update() /home/scandido/workspace/BigSHOT/src/base/pomdp/filter_fn/BayesFilterFn.h:50 0x0000000000445c3b
12 BigSHOT::HyperParticleFilter<BigSHOT::Fire1Belief, BigSHOT::Fire1Action, BigSHOT::Fire1Observation>::future_evolution() /home/scandido/workspace/BigSHOT/src/base/hpf/HyperParticleFilter.h:127 0x00000000004308e0
11 BigSHOT::HyperParticleFilter<BigSHOT::Fire1Belief, BigSHOT::Fire1Action, BigSHOT::Fire1Observation>::hyperfilter() /home/scandido/workspace/BigSHOT/src/base/hpf/HyperParticleFilter.h:86 0x000000000043149b
10 BigSHOT::HyperParticleFilterSystem<BigSHOT::HyperCostFn<BigSHOT::Fire1Belief, BigSHOT::Fire1Action>, BigSHOT::PolicyFn<BigSHOT::Fire1Belief, BigSHOT::Fire1Action>, BigSHOT::Fire1Belief, BigSHOT::Fire1Action, BigSHOT::Fire1Observation>::next_stage() /home/scandido/workspace/BigSHOT/src/base/hpf/HyperParticleFilter.h:189 0x0000000000446180
9 hyperfilter() /home/scandido/workspace/BigSHOT/src/anotheralgorithm.cpp:126 0x0000000000437798
8 hf_thread_wrapper() /home/scandido/workspace/BigSHOT/src/anotheralgorithm.cpp:281 0x0000000000437cd9
7 boost::_bi::list1<boost::_bi::value<int> >::operator()<void (*)(int), boost::_bi::list0>() /usr/local/include/boost-1_38/boost/bind.hpp:232 0x000000000043f25a
6 boost::_bi::bind_t<void, void (*)(int), boost::_bi::list1<boost::_bi::value<int> > >::operator() /usr/local/include/boost-1_38/boost/bind/bind_template.hpp:20 0x000000000043f298
5 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(int), boost::_bi::list1<boost::_bi::value<int> > > >::run() /usr/local/include/boost-1_38/boost/thread/detail/thread.hpp:56 0x000000000043f2b6
4 thread_proxy() 0x00007f28241c893f
3 start_thread() 0x00007f28243d93ba
2 clone() 0x00007f2822a4ffcd
1 <symbol is not available> 0x0000000000000000
The line of code where this happens is the second of this block:
const_reference operator () (size_type i, size_type j) const {
const size_type element = layout_type::element (i, size1_, j, size2_);
const_subiterator_type it (data ().find (element));
I would like to reiterate that the seg error does not always occur in the same place in the code, but always when doing something in the step-up code.
Thanks in advance for your help!
source to share
Segfaults can arise from other well-debugged libraries (or even from the standard library!) If you damage the heap or free store, for example by double freeing (or double deleting) a pointer, a pointer that has already been freed (or deleted) by freeing ( or by deleting) a pointer that was not allocated using delete, where you should have used delete [] or vice versa, etc.
This accident often occurs in a completely different place and at a completely different time from when and where the error occurred. If you have variables other than matrices shared across multiple threads and you have a race condition that, for example, causes the shared variable to be deleted twice, this can corrupt the free storage and then crash inside the matrix boosting code ...
You should run your code with a tool like valgrind to try and track down heap / free storage corruption.
source to share