Cuda-memcheck: internal error (7)
I tried to debug my CUDA-C program and I found out that cuda-memcheck can be useful. My problem is that I cannot use this tool because every time I run it, even in the Hello World program taken from the Internet, I get this result:
========= CUDA-MEMCHECK
*program output*
========= Internal error (7)
========= No CUDA-MEMCHECK results found
Here's also the bash screen: cuda-memcheck output
I haven't found anything on the internet, so hope someone else here can help me. Thanks everyone!
source to share
It seems to be just a permissions issue as first mentioned in the comments by Robert Crovella. Make sure you have permission to use the relevant files /dev
and you can see there /dev/nvidia-uvm-tools
.
For me this issue happened inside a docker container. Then I restarted the container with the preferred flag:
nvidia-docker run ... --privilieged ...
and that completely solved my problem. See question: https://github.com/NVIDIA/nvidia-docker/issues/454 .
So, either using memcheck with:
sudo cuda-memcheck ...
or trying to reinstall everything with correct permissions and working with sudo should fix this problem (as you found).
One thing that was obvious to me was that it cuda-memcheck
worked pretty fast as long as I ran my program normally. This should not happen: the command can slow down your cores by more than 10x, so if you notice that the command is running at normal speed before normal execution, it is a sign that cuda-memcheck
it cannot even get into your kernels to measure them.
source to share