Caffe compilation error: gflags.cc 'binds both statically and dynamically to this executable
I am trying to install caffe following this tutorial
Basically I have the following error when entering the last make command:
me@dl-01:/home/me/caffe-master$ make runtest
.build_release/tools/caffe
caffe: command line brew
usage: caffe command args
commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time
Flags from tools/caffe.cpp:
-gpu (Run in GPU mode on given device ID.) type: int32 default: -1
-iterations (The number of iterations to run.) type: int32 default: 50
-model (The model definition protocol buffer text file..) type: string
default: ""
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-weights (Optional; the pretrained weights to initialize finetuning. Cannot
be set simultaneously with snapshot.) type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle
ERROR: something wrong with flag 'flagfile' in file '/root/glog-0.3.3/gflags-master/src/gflags.cc'. One possibility: file '/root/glog-0.3.3/gflags-master/src/gflags.cc' is being linked both statically and dynamically into this executable.
make: *** [runtest] Error 1
I don't understand how to resolve this error. Has anyone found this error before? how can i solve it?
source to share
Regardless if you solve it somewhere else, I am posting an answer here, otherwise others are facing the same problem.
First of all, this problem seems to have arisen due to the fact that we do not always read things correctly and blindly follow all instructions, thinking that they are all relevant to our business. hint: they don't.
There is a section in the installation instructions for Caffe (assuming Ubuntu instructions) that reads:
Everything is packed at 14.04.
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev protobuf-compiler
Blindly ignoring the following heading, which clearly states:
Remaining dependencies, 12.04
we keep installing these dependencies, building and installing as needed, which leads to the unfortunate side effect of having two versions of libgflags, one dynamic (in /usr/lib[/x86_x64]
and one static in/usr/local/lib
Resolution
- Promise yourself bad, we'll read the instructions next time.
-
Remove libgflags
sudo apt-get remove -y libgflags
-
Delete
make install
versionssudo rm -f /usr/local/lib/libgflags.a /usr/local/lib/libgflags_nothreads.a sudo rm -rf /usr/local/include/gflags
-
Clean Caffe Design
cd <path>/<to>/caffe make clean
-
Reinstall the libgflags package
sudo apt-get install -y libgflags-dev
-
Restore coffee
make all make test make runtest
Et Voila. All tests should now be running and you are ready to rock the deep boat.
source to share
I also had two libraries installed, a shared library .so and a static library .a. I removed all of them as well as the / usr / local / include / glog folder. The .so file I migrated when I (cross) compiled the system and the .a was from a native and updated build. Ultimately it boiled down to creating glog (initially) such that it provided .so files. I started with a clean boot:
git clone git://github.com/google/glog
Then I edited CMakeLists.txt. Where he says:
add_library (glog
${GLOG_SRCS}
)
I changed it to:
add_library (glog SHARED
${GLOG_SRCS}
)
Then you have to follow other instructions. For my particular case, I had to use slightly different instructions without saying that you need to do this. For me it was: mkdir build cd build
export CXXFLAGS="-fPIC"
cmake ..
make
sudo make install
This gave me the .so files and put them in the right place. Then I started with coffee and it fixed the error for me.
source to share
I have developed a way to debug this problem analytically. In my case, I was cross-compiling for an older ABI, so apt-get was not an option and I was compiling all dependencies by hand.
First, let's take a look at what this problem really is. In the Google GFlags library, flags are declared through global objects. When the global object constructor runs, it calls the GFlags library to register this command line flag. If the global constructor is run multiple times (due to multiple versions of the library containing it loaded into memory), then the GFlags register method fails.
What does GLog have to do with it? Well, GLog uses GFlags and it has globally declared flag objects. Even though GFlags is linked correctly, if the GLog library is loaded multiple times, you get an error pointing to logging.cc in GLog.
It looks like a mess, yes. Even though most of the time GLog and GFlags are linked as shared, if another library references a static or some other version, kaboom.
Luckily, we can debug this problem using GDB and other tools if you want to get into some tricky symbol analysis. First, you'll want to run GDB on a Python interpreter when it tries to import caffe:
gdb --args python -c 'import caffe'
Now run the program once so that GDB can build all the libraries it imports:
(gdb) r
We can now set the FlagRegistry::RegisterFlag()
breakpoint in place in the ( FlagRegistry::RegisterFlag()
) function that prints the error message and run it again. Note that this line number is from my version of GFlags (2.2.2), you might have to look at the source code for your version of GFlags and get the line number.
(gdb) break gflags.c:728
(gdb) r
Hopefully GDB should terminate when the error first appears (if not, check that gflags was built with debug symbols). Look at the trail:
(gdb) bt
#0 google::(anonymous namespace)::FlagRegistry::RegisterFlag (this=0xa33b30, flag=0x1249d20) at dev/gflags-2.2.2/src/gflags.cc:728
#1 0x00007ffff0f3247a in _GLOBAL__sub_I_logging.cc () from prefix/lib/libcaffe2.so
#2 0x00007ffff7de76ca in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0x7fffffffdb08, env=env@entry=0x7fffffffdb28) at dl-init.c:72
#3 0x00007ffff7de77db in call_init (env=0x7fffffffdb28, argv=0x7fffffffdb08, argc=3, l=<optimized out>) at dl-init.c:30
#4 _dl_init (main_map=main_map@entry=0xd9c2a0, argc=3, argv=0x7fffffffdb08, env=0x7fffffffdb28) at dl-init.c:120
#5 0x00007ffff7dec8f2 in dl_open_worker (a=a@entry=0x7fffffffcf70) at dl-open.c:575
#6 0x00007ffff7de7574 in _dl_catch_error (objname=objname@entry=0x7fffffffcf60, errstring=errstring@entry=0x7fffffffcf68, mallocedp=mallocedp@entry=0x7fffffffcf5f,
operate=operate@entry=0x7ffff7dec4e0 <dl_open_worker>, args=args@entry=0x7fffffffcf70) at dl-error.c:187
#7 0x00007ffff7debdb9 in _dl_open (file=0x9aee70 "prefix/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state.so", mode=-2147483646,
caller_dlopen=0x51bb39 <_PyImport_GetDynLoadFunc+233>, nsid=-2, argc=<optimized out>, argv=<optimized out>, env=0x7fffffffdb28) at dl-open.c:660
#8 0x00007ffff75ecf09 in dlopen_doit (a=a@entry=0x7fffffffd1a0) at dlopen.c:66
#9 0x00007ffff7de7574 in _dl_catch_error (objname=0xabf9f0, errstring=0xabf9f8, mallocedp=0xabf9e8, operate=0x7ffff75eceb0 <dlopen_doit>, args=0x7fffffffd1a0) at dl-error.c:187
#10 0x00007ffff75ed571 in _dlerror_run (operate=operate@entry=0x7ffff75eceb0 <dlopen_doit>, args=args@entry=0x7fffffffd1a0) at dlerror.c:163
#11 0x00007ffff75ecfa1 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#12 0x000000000051bb39 in _PyImport_GetDynLoadFunc ()
<snip>
Okay, there is a lot to do, but let's focus on the line that really matters:
#1 0x00007ffff0f3247a in _GLOBAL__sub_I_logging.cc () from prefix/lib/libcaffe2.so
This is a constructor call for global variables in logging.cc (which is part of GLog). As you can see, this call is in libcaffe2.so, which means that GLog was statically linked to libcaffe2.so [I used caffe2, but the procedure should be the same for both].
Then you can set google::(anonymous namespace)::FlagRegistry::RegisterFlag
breakpoints on google::(anonymous namespace)::FlagRegistry::RegisterFlag
and restart the program from the beginning. Look at each call to RegisterFlag () and find out exactly where this flag was registered the first time. If the library providing the flag is a shared library, then it should only be registered from that .so file, and nowhere else.
You can use to confirm the diagnosis
nm <library> | grep _GLOBAL__sub_I_logging.cc
check this initialization function in the library file. Once you've found your culprit, you'll need to rebuild it so that it doesn't statically link to GFlags / GLog.
source to share