How to get reproducible output when running Keras with Tensorflow backend
Every time I run an LSTM network with Keras in a jupyter notebook I get a different result and have googled a lot and have tried several different solutions but none of them work, here are some of my solutions:
-
set the number of random numpy seeds
random_seed=2017 from numpy.random import seed seed(random_seed)
-
set random tensorflow seed
from tensorflow import set_random_seed set_random_seed(random_seed)
-
set inline random seed
import random random.seed(random_seed)
-
install PYTHONHASHSEED
import os os.environ['PYTHONHASHSEED'] = '0'
-
add PYTHONHASHSEED to jupyter notebook kernel.json
{ "language": "python", "display_name": "Python 3", "env": {"PYTHONHASHSEED": "0"}, "argv": [ "python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ] }
and my env version:
Keras: 2.0.6
Tensorflow: 1.2.1
CPU or GPU: CPU
and this is my code:
model = Sequential()
model.add(LSTM(16, input_shape=(time_steps,nb_features), return_sequences=True))
model.add(LSTM(16, input_shape=(time_steps,nb_features), return_sequences=False))
model.add(Dense(8,activation='relu'))
model.add(Dense(1,activation='linear'))
model.compile(loss='mse',optimizer='adam')
source to share
The seed is definitely missing from the definition of the model. Detailed documentation can be found here: https://keras.io/initializers/ .
Basically, your layers use random variables as the basis for their parameters. Therefore, you get different outputs each time.
One example:
model.add(Dense(1, activation='linear',
kernel_initializer=keras.initializers.RandomNormal(seed=1337),
bias_initializer=keras.initializers.Constant(value=0.1))
Keras themselves have a section on getting reproducible results in their FAQ section: ( https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development ). They have the following piece of code to get reproducible results:
import numpy as np
import tensorflow as tf
import random as rn
# The below is necessary in Python 3.2.3 onwards to
# have reproducible behavior for certain hash-based operations.
# See these references for further details:
# https://docs.python.org/3.4/using/cmdline.html#envvar-PYTHONHASHSEED
# https://github.com/fchollet/keras/issues/2280#issuecomment-306959926
import os
os.environ['PYTHONHASHSEED'] = '0'
# The below is necessary for starting Numpy generated random numbers
# in a well-defined initial state.
np.random.seed(42)
# The below is necessary for starting core Python generated random numbers
# in a well-defined state.
rn.seed(12345)
# Force TensorFlow to use single thread.
# Multiple threads are a potential source of
# non-reproducible results.
# For further details, see: https://stackoverflow.com/questions/42022950/which-seeds-have-to-be-set-where-to-realize-100-reproducibility-of-training-res
session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
from keras import backend as K
# The below tf.set_random_seed() will make random number generation
# in the TensorFlow backend have a well-defined initial state.
# For further details, see: https://www.tensorflow.org/api_docs/python/tf/set_random_seed
tf.set_random_seed(1234)
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
K.set_session(sess)
source to share
Keras + Tensorflow.
Step 1, disable the GPU.
import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = ""
Step 2, run those libraries that are included in your code, say "tensorflow, numpy, random".
import tensorflow as tf
import numpy as np
import random as rn
sd = 1 # Here sd means seed.
np.random.seed(sd)
rn.seed(sd)
os.environ['PYTHONHASHSEED']=str(sd)
from keras import backend as K
config = tf.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)
tf.set_random_seed(sd)
sess = tf.Session(graph=tf.get_default_graph(), config=config)
K.set_session(sess)
Make sure these two pieces of code are included at the beginning of your code, then the result will be reproducible.
source to share