Runtime linker error using Thrust in MATLAB MEX file
I am having trouble using CUDA Thrust library in MATLAB MEX code.
I have an example that works fine externally, but if I compile and run it as a MEX file, it throws "missing characters" errors at runtime.
This is similar to the Thrust library. If thrust::device_vector
I use cudaMalloc
with cudaMemcpy
or instead cublasSetVector
, then everything will be fine.
Minimal example
thrustDemo.cu:
#ifdef MATLAB_MEX_FILE
#include "mex.h"
#include "gpu/mxGPUArray.h"
#endif
#include <thrust/device_vector.h>
#include <vector>
void thrustDemo() {
std::vector<double> foo(65536, 3.14);
thrust::device_vector<double> device_foo(foo);
}
#ifdef MATLAB_MEX_FILE
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, mxArray const *prhs[]) {
thrustDemo();
}
#else
int main(void) { thrustDemo(); }
#endif
Problem
I can compile this from the command line ( nvcc thrustDemo.cu
) and run the executable just fine.
When I try to build this as a MATLAB MEX file ( mexcuda thrustDemo.cu
from MATLAB R2017a), it compiles and links just fine:
>> mexcuda thrustDemo.cu
Building with 'nvcc'.
MEX completed successfully.
But when I try to run it I get the following error:
>> thrustDemo()
Invalid MEX-file '/home/kqs/thrustDemo.mexa64':
Missing symbol '_ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE5c_strEv' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE5emptyEv' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt12length_errorC1EPKc' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt13runtime_errorC2EPKc' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEaSEPKc' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1EPKcRKS3_' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1ERKS4_' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1Ev' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEpLEPKc' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEpLERKS4_' required by '/home/kqs/thrustDemo.mexa64'.
This is quite foreign to me; can someone tell me what this means? They look like linker errors, but they are generated at runtime. Also, I thought Thrust was a templating library, so what can be linked there?
Finally, replacing thrust::device_vector
with cudaMalloc
and with either cudaMemcpy
or cublasSetVector
works just fine. So for now I'm stuck with a bunch cudaMalloc
of my code, which seems ... frustrating. I would really like to be able to use Thrust.
Version
MATLAB R2017a
nvcc
V8.0.61, gcc
5.4.0, Ubuntu 16.04.2
NVidia 375.39 driver, GTX 1060 video card (Compute Capability 6.1)
Update: ldd
conclusion
In the comments, I checked the dependencies of the MEX file with ldd thrustDemo.mexa64
:
linux-vdso.so.1 => (0x00007ffdd35ea000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f097eccf000)
libcudart.so.8.0 => /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0 (0x00007f097ea69000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f097e852000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f097e489000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f097e180000)
/lib64/ld-linux-x86-64.so.2 (0x0000562df178c000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f097df7b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f097dd5e000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f097db56000)
I tried to find one of these missing symbols and was able to find it:
$ nm -D /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep "_ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE5c_strEv"
0000000000120be0 W _ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE5c_strEv
So it seems like MATLAB should look in the wrong place.
source to share
It turns out this has nothing to do with Thrust, but rather a problem with MATLAB having its own version of the C ++ standard libraries.
Thanks to @Navan and @talonmies for their helpful comments.
Error interpretation
First, MATLAB raises these errors when loading the MEX file. The MEX file has external dependencies and MATLAB could not find them.
After checking these dependencies with a Linux utility ldd
, then using nm
to display the symbols defined by those libraries, I found that the system version of the libstdc++
shared library actually contains these "missing symbols". Hence why the compiled version works fine.
Solution to the problem
So the root problem is that MATLAB comes with its older version libstdc++
that lacks these features. Knowing the root cause, I found questions like this:
GLIBCXX_3.4.11 version not found (buildW.mexglx required)
which describe workarounds that have actually been successful for my problem.
Specifically, I used LD_PRELOAD
when starting MATLAB to force MATLAB to use the system libstdc++
instead of its own copy:
$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/local/MATLAB/R2017a/bin/matlab
Update: better solution
It turns out that the people at GCC are well aware of this incompatibility and discuss it here :
With the release of GCC 5.1 libstdC ++ introduced a new ABI library that includes new implementations of std :: string and std :: list. These changes were necessary to comply with the 2011 C ++ standard, which disallows "Copy-On-Write" strings and requires lists to keep track of their size.
To maintain backward compatibility for existing libstdC ++ related code, the library name has not changed, and old implementations are still supported alongside new ones .... The _GLIBCXX_USE_CXX11_ABI macro (see Macros) determines whether the declarations in the library headers use the old or new ABI.
To tell us to gcc
use an older ABI we just need to define _GLIBCXX_USE_CXX11_ABI
how 0
before we include any library headers like. passing the parameter to the -D
compiler:
-D_GLIBCXX_USE_CXX11_ABI=0
For the sake of completeness, I mentioned that my complete call mexcuda
looks like this:
nvcc_opts = [...
'-gencode=arch=compute_30,code=sm_30 ' ...
'-gencode=arch=compute_50,code=sm_50 ' ...
'-gencode=arch=compute_60,code=sm_60 ' ...
'-std=c++11 ' ...
'-D_GLIBCXX_USE_CXX11_ABI=0 ' % MATLAB libstdc++ uses old library ABI
];
mexcuda_opts = {
'-lcublas' % Link to cuBLAS
'-lmwlapack' % Link to LAPACK
'-lcufft' % Link to cuFFT
['NVCCFLAGS="' nvcc_opts '"']
'-L/usr/local/cuda/lib64' % Location of CUDA libraries
};
mexcuda(mexcuda_opts{:}, src_file);
source to share