Failed to create handle cudnn: CUDNN_STATUS_INTERNAL_ERROR

I managed to get it to work by deleting the .nv folder in my home folder:

sudo rm -rf ~/.nv/

      

        
        
        
      

    

In my case, it seems that the problem was caused by a mismatch between tensorflow and cudnn version. The following worked for me (I was working on Ubuntu 16.04 with NVidia Tesla K80 on Google Cloud, tensorflow 1.5 finally worked with cudnn 7.0.4 and cuda 9.0):

• Remove cuDNN completely:

  sudo rm /usr/local/cuda/include/cudnn.h
  sudo rm /usr/local/cuda/lib64/libcudnn*
  
        
  
          
          
          
        
  
      

  After that import tensorflow should throw an error.

• Download the appropriate version of cuDNN. Note that there is cuDNN 7.0.4 for CUDA 9.0 and cuDNN 7.0.4 for CUDA 8.0. You should choose the one that matches your CUDA version. Be careful during this step, otherwise you will get a similar problem again. Install cuDNN as usual:

  tar -xzvf cudnn-9.0-linux-x64-v7.tgz
  cd cuda
  sudo cp -P include/cudnn.h /usr/include
  sudo cp -P lib64/libcudnn* /usr/lib/x86_64-linux-gnu/
  sudo chmod a+r /usr/lib/x86_64-linux-gnu/libcudnn*
  
        
  
          
          
          
        
  
      

  In this example, I've installed cuDNN 7.0.x for CUDA 9.0 (x doesn't really matter). Observe your version of CUDA.

• Restart your computer. In my case, the problem went away. If the error still occurs, consider installing a different version of TensorFlow.

Hope this helps someone.

I also get the same error and I solved the problem. My system properties were as follows:

• Operating system: Ubuntu 14.04
• GPU: GTX 1050Ti
• Nvidia Driver: 375.66
• Tensorflow: 1.3.0
• Cudnn: 6.0.21 (cudnn-8.0-linux-x64-v6.0.deb)
• Cuda: 8.0.61
• Keras: 2.0.8

How I solved the problem:

• I copied the cudnn files to their respective locations (/ usr / local / cuda / include and / usr / local / cuda / lib 64)

• I am setting environment variables like:

  * export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
  * export CUDA_HOME=/usr/local/cuda
  
        
  
          
          
          
        
  
      

• I am also running sudo ldconfig -v command
  
  to cache shared libraries for the runtime linker.

I hope these steps help those who are about to go crazy too.

This is a cudnn compatible problem. Check what you installed using the GPU for example tensorflow-gpu

. Which version? Is the version compatible with your version cudnn

and is cudnn installed the correct version for your cuda ?.

I noticed that: cuDNN v7.0.3

for Cuda 7.*

cuDNN v7.1.2

for Cuda 9.0

cuDNN v7.3.1

for Cuda 9.1

and so on.

So also check the correct TensorFlow version for your cuda configurations. For example -using tensorflow-gpu

: TF v1.4

for cudnn 7.0.*

TF v1.7

and above for cudnn 9.0.*

etc.

Therefore, all you have to do is reinstall the appropriate version of cudnn. Hope it helps!

For anyone else who ran into a problem in Jupyter notebook :

I have managed two Jupyter notebooks. After closing one of them, the problem was resolved.

Adding the following code worked for me:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

      

        
        
        
      

    

In my environment, there is no mismatch between CuDNN and Cuda versions. OS: Ubuntu-18.04; Flux tensor: 1.14; CuDNN: 7.6; where: 10.1 (418.87.00).

I faced the same problem too:

Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX 1050
major: 6 minor: 1 memoryClockRate (GHz) 1.493 pciBusID 0000:01:00.0
Total memory: 3.95GiB
Free memory: 3.60GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0)
E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
F tensorflow/core/kernels/conv_ops.cc:532] Check failed:  stream->parent()->GetConvolveAlgorithms(&algorithms)

Aborted (core dumped)

      

        
        
        
      

    

But in my case, using sudo with the command worked perfect.

I ran into this problem when I accidentally installed CUDA 9.2 libcudnn7_7.2.1.38-1 + cuda9.2_amd64.deb instead of libcudnn7_7.0.5.15-1 + cuda9.0_amd64.deb on a system with CUDA 9.0 installed.

I got there because I had CUDA 9.2 installed and I downgraded to CUDA 9.0 and obviously libcudnn is version specific.

For me, re-running the CUDA installation as described here solved the problem:

# Add NVIDIA package repository
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
sudo apt install ./cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt install ./nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt update

# Install CUDA and tools. Include optional NCCL 2.x
sudo apt install cuda9.0 cuda-cublas-9-0 cuda-cufft-9-0 cuda-curand-9-0 \
    cuda-cusolver-9-0 cuda-cusparse-9-0 libcudnn7=7.2.1.38-1+cuda9.0 \
    libnccl2=2.2.13-1+cuda9.0 cuda-command-line-tools-9-0


      

        
        
        
      

    

During the setup time apt-get

reduced cudnn7

which I think is the culprit here. It was probably updated by accident with an upgrade apt-get upgrade

to a version that is incompatible with some other part of the system.

Please remember to close your tensor board terminal / cmd or other terminals that interact with / with the directory. Then you can resume your workout, everything should work on this.