Learning and Loss Don't Change in Keras CNN Model
I run CNN to classify left and right strokes. I have 190,000 educational images and I use 10% for validation. My model is set up as shown below. I get the paths of all images, read them and resize them. I normalize the image and then fit it to the model. My problem is that I was sticking to a learning accuracy of 62.5% and a loss of about 0.6615-0.6619. Is there something wrong, what am I doing? How can I stop this?
Just a few interesting notes:
1) I first tested this on 10 images, I had the same problem but changed the optimizer to adam and the batch size to 4.
2) Then I tested more and more images, but each time I needed to change the batch size to get improvements in accuracy and loss. With 10,000 images, I had to use a batch size of 500 and the rmsprop optimizer. However, accuracy and loss only really started to change after epoch 10.
3) Now I am training on 190,000 images and I cannot increase the batch size since my GPU is at max.
imageWidth = 50
imageHeight = 150
def get_filepaths(directory):
file_paths = []
for filename in files:
filepath = os.path.join(root, filename)
file_paths.append(filepath) # Add it to the list.
return file_paths
def cleanUpPaths(fullFilePaths):
cleanPaths = []
for f in fullFilePaths:
if f.endswith(".png"):
cleanPaths.append(f)
return cleanPaths
def getTrainData(paths):
trainData = []
for i in xrange(1,190000,2):
im = image.imread(paths[i])
im = image.imresize(im, (150,50))
im = (im-255)/float(255)
trainData.append(im)
trainData = np.asarray(trainData)
right = np.zeros(47500)
left = np.ones(47500)
trainLabels = np.concatenate((left, right))
trainLabels = np_utils.to_categorical(trainLabels)
return (trainData, trainLabels)
#create the convnet
model = Sequential()
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(imageWidth,imageHeight,1),strides=1))#32
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), activation='relu',strides=1))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(1, 3)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (1, 2), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 1)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax'))
sgd = SGD(lr=0.01)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop',metrics=['accuracy'])
#prepare the training data*/
trainPaths = get_filepaths("better1/train")
trainPaths = cleanUpPaths(trainPaths)
(trainData, trainLabels) = getTrainData(trainPaths)
trainData = np.reshape(trainData,(95000,imageWidth,imageHeight,1)).astype('float32')
trainData = (trainData-255)/float(255)
#train the convnet***
model.fit(trainData, trainLabels, batch_size=500, epochs=50, validation_split=0.2)
#/save the model and weights*/
model.save('myConvnet_model5.h5');
model.save_weights('myConvnet_weights5.h5');
source to share
I had this problem several times, so I thought to make a small summary and possible solutions, etc. to help people in the future.
Problem . The model predicts one of 2 (or more) possible classes for all data it sees *
An approving problem arises . Method 1: The accuracy for the model remains around 0.5 during training (or 1 / n, where n is the number of classes). Method 2: Obtain the scores of each class in the predictions and confirm that it predicts all of one class.
Fixes / Checks (in some order):
- Double Check Model Architecture : use
model.summary()
, check model. - Check Data Labels : Make sure your train data labeling hasn't been obfuscated somewhere in preprocessing, etc. (it happens!)
- Validation of data delivery across the channel is randomized : make sure you are not feeding train data to the model one class at a time. For example, if you are using
ImageDataGenerator().flow_from_directory(PATH)
, check that paramshuffle=True
isbatch_size
greater than 1. - Check if any pre-prepared layers have been prepared. : ** If you are using a pretrained model, make sure all layers that use pretrained weights are NOT initially trainable. For the first epochs, it is only necessary to train new (randomly initialized) layers;
for layer in pretrained_model.layers: layer.trainable = False
should be somewhere in your code. - Ramp Down Training Level : Decrease Training Speed by 10 and try again. Note that you will have to completely re-initialize the layers you are trying to train each time you try to get a new learning rate. (For example, I had this problem that was only solved after I got to
lr=1e-6
, so keep going!)
If any of you know of more fixes / checks that could get the model trained properly, please contribute and I will try to update the list.
** Note that it is common to get most of the pretrained model once the new layers have been pretrained “enough”.
* Other problem names to help you search here ... keras tensorflow theano CNN convolutional neural network bad learning stuck fixed non static broken error tried stuck training optimization optimization only 0.5 precision does not change only predicts one class will not train , model stuck on class model reloading between keras CNN eras, same conclusion
source to share
I would try a couple of things. A lower learning rate should help with more data. In general, adapting the optimizer should help. Also, your network seems very small, you may need to increase the capacity of the model by adding layers or increasing the number of filters in the layers.
For a more detailed description of how to put deep learning into practice, see ( http://www.deeplearningbook.org/contents/guidelines.html ).
source to share
I have two more things to add to DBCerigo's large list.
- Check the activation functions : some layers have an activation function by default
linear
, if you don't insert some non-linearity into the model, it won't be able to generalize, so the network will try to learn how to linearly divide the object space it's not linear. Making sure you have a non-linearity set set is a good reference point. - Check the complexity of the model : If you have a relatively simple model and it only learns until the 1st or 2nd era and then stops, it may be trying to learn something too complex. Try to make the model deeper. This usually happens when working with frozen models with only 1 or 2 layers unfrozen.
While the second one may be obvious, I ran into his problem once and I wasted a lot of time checking everything (data, batches, LR ...) before I figured out.
Hope this helps
source to share