TensorFlow Exponential Moving Average
I can't figure out how to get it to tf.train.ExponentialMovingAverage
work. Below is a simple code to find w
in a simple equation y_ = x * w
. m
- moving average. Why does the code return None
for m
? How can I get it to return a moving average?
import numpy as np
import tensorflow as tf
w = tf.Variable(0, dtype=tf.float32)
ema = tf.train.ExponentialMovingAverage(decay=0.9)
m = ema.apply([w])
x = tf.placeholder(tf.float32, [None])
y = tf.placeholder(tf.float32, [None])
y_ = tf.multiply(x, w)
with tf.control_dependencies([m]):
loss = tf.reduce_sum(tf.square(tf.subtract(y, y_)))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
train = optimizer.minimize(loss)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(20):
_, w_, m_ = sess.run([train, w, m], feed_dict={x: [1], y: [10]})
print(w_, ',', m_)
Output:
0.02 , None
0.03996 , None
0.0598801 , None
0.0797603 , None
0.0996008 , None
0.119402 , None
0.139163 , None
0.158884 , None
0.178567 , None
0.19821 , None
0.217813 , None
0.237378 , None
0.256903 , None
0.276389 , None
0.295836 , None
0.315244 , None
0.334614 , None
0.353945 , None
0.373237 , None
0.39249 , None
source to share
This is because the m
(python) variable does not contain the result of the operation other than the operation itself. See Document:
Returns:
An Operation that updates the moving averages.
To access the mean, you need to create a new item in your graph:
av = ema.average(w)
and then:
_, w_, av_ = sess.run([train, w, av], feed_dict={x: [1], y: [10]})
print(w_, ',', av_)
will print
[0.020000001, 0.0]
[0.039960001, 0.0020000006]
[0.059880082, 0.0057960013]
[0.07976032, 0.01120441]
[0.099600799, 0.018060002]
Complete code
import tensorflow as tf w = tf.Variable(0, dtype=tf.float32) ema = tf.train.ExponentialMovingAverage(decay=0.9) m = ema.apply([w]) av = ema.average(w) x = tf.placeholder(tf.float32, [None]) y = tf.placeholder(tf.float32, [None]) y_ = tf.multiply(x, w) with tf.control_dependencies([m]): loss = tf.reduce_sum(tf.square(tf.subtract(y, y_))) optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001) train = optimizer.minimize(loss) with tf.Session() as sess: sess.run(tf.global_variables_initializer()) for i in range(20): _, w_, av_ = sess.run([train, w, av], feed_dict={x: [1], y: [10]}) print(w_, ',', av_)
source to share