Reusing a Tensorflow variable

Question

Reusing a Tensorflow variable

I have built my LSTM model. Ideally, I want to use variable reuse to define the LSTM test model later.

with tf.variable_scope('lstm_model') as scope:
    # Define LSTM Model
    lstm_model = LSTM_Model(rnn_size, batch_size, learning_rate,
                     training_seq_len, vocab_size)
    scope.reuse_variables()
    test_lstm_model = LSTM_Model(rnn_size, batch_size, learning_rate,
                     training_seq_len, vocab_size, infer=True)

The code above is giving me an error

Variable lstm_model/lstm_vars/W already exists, disallowed. Did you mean to set reuse=True in VarScope?

If I set Reuse = True as shown in below code block

with tf.variable_scope('lstm_model', reuse=True) as scope:

I am getting another error

Variable lstm_model/lstm_model/lstm_vars/W/Adam/ does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?

I have added the relevant model code below as a reference. Relevant section in LSTM model where I have weights

with tf.variable_scope('lstm_vars'):
    # Softmax Output Weights
    W = tf.get_variable('W', [self.rnn_size, self.vocab_size], tf.float32, tf.random_normal_initializer())

Relevant section where I have Adam's optimizer:

optimizer = tf.train.AdamOptimizer(self.learning_rate)

+3

variables tensorflow

caramelslice 23 jul. '17 at 9:43

source to share

2 answers

caramelslice · Answer 1 · 2017-07-23T12:40:58+0000

It seems that instead of:

with tf.variable_scope('lstm_model') as scope:
    # Define LSTM Model
    lstm_model = LSTM_Model(rnn_size, batch_size, learning_rate,
                     training_seq_len, vocab_size)
    scope.reuse_variables()    
    test_lstm_model = LSTM_Model(rnn_size, batch_size, learning_rate,
                     training_seq_len, vocab_size, infer_sample=True)

This fixes the problem

# Define LSTM Model
lstm_model = LSTM_Model(rnn_size, batch_size, learning_rate,
                        training_seq_len, vocab_size)

# Tell TensorFlow we are reusing the scope for the testing
with tf.variable_scope(tf.get_variable_scope(), reuse=True):
    test_lstm_model = LSTM_Model(rnn_size, batch_size, learning_rate,
                                 training_seq_len, vocab_size, infer_sample=True)

Vladimir Bystricky · Answer 2 · 2017-07-23T10:06:41+0000

If you use the same variable twice (or multiple times), you must use it first the with tf.variable_scope('scope_name', reuse=False):

next time with tf.variable_scope('scope_name', reuse=True):

.

Or you can use the method tf.variable_scope.reuse_variables()

with tf.variable_scope("foo") as scope:
    v = tf.get_variable("v", [1])
    scope.reuse_variables()
    v1 = tf.get_variable("v", [1])

in the code above v

and v1

are the same variable.

Reusing a Tensorflow variable

More articles: