Tensorflow allocating GPU memory when using tf.device ('/ cpu: 0')
System info : 1.1.0, GPU, Windows, Python 3.5, code runs in ipython consoles.
I am trying to run two different Tensorflow sessions, one on the GPU (which does some batch work) and one on the CPU which I use for quick tests, while the other is running.
The problem is, when I start the second session defining with tf.device('/cpu:0')
, the session tries to allocate GPU memory and crashes another session.
My code:
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import time
import tensorflow as tf
with tf.device('/cpu:0'):
with tf.Session() as sess:
# Here 6 GBs of GPU RAM are allocated.
time.sleep(5)
How do I make Tensorflow ignore the GPU?
UPDATE:
As suggested in a comment by @Nicolas I looked at this answer and ran
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import tensorflow as tf
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
which prints:
[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 2215045474989189346
, name: "/gpu:0"
device_type: "GPU"
memory_limit: 6787871540
locality {
bus_id: 1
}
incarnation: 13663872143510826785
physical_device_desc: "device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0"
]
It seems to me that even if I explicitly tell the script to ignore any CUDA devices, it still finds and uses them. Could this be a TF 1.1 bug?
source to share
It turns out that setting CUDA_VISIBLE_DEVICES
to an empty string does not mask the CUDA devices visible to the script.
From the documentationCUDA_VISIBLE_DEVICES
(highlighted by me):
Only devices whose index is present in the sequence are visible to CUDA, and they are listed in the order of sequence. If one of the indexes is invalid, only devices whose index precedes the invalid index visible to CUDA applications . For example, setting CUDA_VISIBLE_DEVICES to 2.1 causes device 0 to be invisible and device 2 to be listed before device 1. Setting CUDA_VISIBLE_DEVICES to 0.2, -1.1 forces devices 0 and 2 to be visible and device 1 should be invisible.
It looks like an empty string is being used to treat it as "no valid devices", but changed the value as it is not mentioned in the documentation.
Changing the code to os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
fixes the problem. Running
import os
os.environ["CUDA_VISIBLE_DEVICES"]="-1"
import tensorflow as tf
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
now prints
[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 14097726166554667970
]
and instantiation tf.Session
no longer handles GPU memory.
source to share
Could you try one of these configuration options?
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
# or config.gpu_options.per_process_gpu_memory_fraction = 0.0
with tf.Session(config=config) as sess:
...
According to the documentation , this should help you manage your GPU memory for that particular session, and thus your second session should be able to run on the GPU.
EDIT: According to this answer, you should also try this:
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="-1"
source to share