Tensorflow: CNN doesn't learn at all
I just created my own CNN that reads data from disk and tries to learn. But the scales don't seem to learn at all, they all remain randomized.
Deviations change only slightly. I have already tried using grayscale images without success. I was also tired of reducing my dataset to only 2 classes, which I thought should have worked. But the measured accuracy is below 50% (I may be calculating the accuracy false)
Here's some code:
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, classes])
weights = {
'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),
'wd1': tf.Variable(tf.random_normal([12*12*64, 1024])),
'out': tf.Variable(tf.random_normal([1024, classes]))
}
biases = {
'bc1': tf.Variable(tf.random_normal([32])),
'bc2': tf.Variable(tf.random_normal([64])),
'bd1': tf.Variable(tf.random_normal([1024])),
'out': tf.Variable(tf.random_normal([classes]))
}
pred = model.conv_net(x, weights, biases, keep_prob, imgSize)
with tf.name_scope("cost"):
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
with tf.name_scope("optimizer"):
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
with tf.name_scope("accuracy"):
correct_pred = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
while(step < epochs):
batch_x, batch_y = batch_creator(batch_size, train_x.shape[0], 'train')
sess.run(optimizer, feed_dict={x: batch_x, y: batch_y, keep_prob: dropout})
if(step % display_step == 0):
batchv_x, batchv_y = batch_creator(batch_size, val_x.shape[0], 'val')
summary, loss, acc = sess.run([merged, cost, accuracy], feed_dict={x: batchv_x, y: batchv_y})
train_writer.add_summary(summary, step)
I have looked at the created batches which look great. batch_x is an array of 2304 floats that represent a 48x48 image batch_y is an array with one_hot labels: [0 0 ... 0 1 0 ... 0 0]
This is my model:
def conv2d(x, W, b, strides=1):
x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
x = tf.nn.bias_add(x, b)
return tf.nn.relu(x)
def maxpool2d(x, k=2):
return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],
padding='SAME')
def conv_net(x, weights, biases, dropout, imgSize):
with tf.name_scope("Reshaping_data") as scope:
x = tf.reshape(x, shape=[-1, imgSize, imgSize, 1], name="inp") #(?, 48, 48, 1)
with tf.name_scope("Conv1") as scope:
conv1 = conv2d(x, weights['wc1'], biases['bc1'])
conv1 = maxpool2d(conv1, k=2) #(?, 24, 24, 32)
with tf.name_scope("Conv2") as scope:
conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
conv2 = maxpool2d(conv2, k=2) #(?, 12, 12, 64)
with tf.name_scope("FC1") as scope:
fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]]) #(?, 9216)
fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1']) #(?, 1024)
fc1 = tf.nn.relu(fc1) #(?, 1024)
out = tf.add(tf.matmul(fc1, weights['out']), biases['out'], name="out") #(?, 43)
return out
Thanks for your help!
PS: This is how some filters of the second convolutional layer look like (it doesn't matter how many epochs later)
source to share
I tried your network with cifar-10 database.
I am afraid the problem is caused by huge parameters, especially in the fc1 layer. You can try to reduce the number of cores, for example, divided by 2, in convolutional layers and use 4 or 6 as k in layer pools to reduce spatial dimensions. Then you can decrease the number of weights in the fc1 layer.
And when the parameters are numerous, be careful when initializing the weight. Use tf.contrib.layers.xavier_initializer()
or tf.random_normal_initializer(stddev=np.sqrt(2.0 / n))
for better initialization.
After decreasing the parameters and better initializing the weights, the losses by the figure-10 begin to converge. You can try this with your own database.
source to share