Caffe network gets very low loss, but very poor accuracy when tested
I am a bit new to work and I am getting strange behavior. I am trying to use fine tuning on bvlc_reference_caffenet to do an OCR task.
I took my preprocessed net, changed the last FC level to the number of output classes I have, and retrained. After several thousand iterations, I get ~ .001 size loss and over 90 percent accuracy when testing the network. However, when I try to run my network over data, I get terrible results, no more than 7 or 8 percent.
The code I am using to start the network:
[imports]
net = caffe.Classifier('bvlc_reference_caffenet/deploy.prototxt', 'bvlc_reference_caffenet/caffenet_train_iter_28000.caffemodel',
image_dims=(227, 227, 1))
input_image = caffe.io.load_image('/Training_Processed/6/0.png')
prediction = net.predict([input_image]) # predict takes any number of images, and formats them for the Caffe net automatically
cls = prediction[0].argmax()
Any thoughts on why this performance might be so poor?
Thank!
PS: Some additional information that may or may not be used. When classifying as shown below, the classifier appears to favor certain classes. Even though I have class 101 problem, it seems to me that at most 15 different classes are assigned
PPS: I am also sure that I am not over-recycling. I've tested this with snapshots and they all show the same poor results.
source to share
In your code to test the model you posted, omit some components:
- It looks like you are not subtracting the average of the image.
- You have not swapped channels from RGB to BGR.
- You have not scaled the inputs to the range [0..255].
In cases like this, caffe.Classifier
you might see something like:
net = caffe.Classifier('bvlc_reference_caffenet/deploy.prototxt',
'bvlc_reference_caffenet/caffenet_train_iter_28000.caffemodel',
mean = NP.load( 'ilsvrc_2012_mean.npy' ),
input_scale=1.0, raw_scale=255,
channel_swap=(2,1,0),
image_dims=(227, 227, 1))
it is important to have the same input transformations in the test as in training.
source to share