Accuracy degrades the longer I train the Keras model

I am currently using the resnet built in keras to categorize the two classes. I use a model breakpoint to keep the best models based on validation accuracy. Improved and better models persist until I go through all my data a few times. Keras retains the new models demonstrating that they have higher accuracy, but when I test the models they perform worse than the previous models.

Here is the test result for each model with validation data. The first number in the model name is the epoch, the last number is the precision according to keras.

• 03-0.90.hdf5, Wrong predictions for A: 176, Wrong predictions for B: 1652, general errors 1828
• 04-0.91.hdf5, Wrong predictions for A: 246, Wrong predictions for B: 1448, general errors 1694
• 06-0.92.hdf5, Wrong predictions for A: 135, Wrong predictions for B: 1738, general errors 1873
• 09-0.92.hdf5, Wrong predictions for A: 117, Wrong predictions for B: 1738, general errors 1855
• 10-0.92.hdf5, Wrong predictions for A: 183, Wrong predictions for B: 1208, general errors 1391
• 15-0.92.hdf5, Wrong predictions for A: 64, Wrong predictions for B: 2973, general errors 3037
• 23-0.93.hdf5, Wrong predictions for A: 119, Wrong predictions for B: 2511, general errors 2630
• 26-0.93.hdf5, Wrong predictions for A: 113, Wrong predictions for B: 1794, common errors 1907
• 27-0.93.hdf5, Wrong predictions for A: 85, Wrong predictions for B: 2821, common errors 2906

...

• 1049-0.97.hdf5, Wrong predictions for A: 31, Wrong predictions for B: 5949, sum of errors 5980
• 1089-0.97.hdf5, Wrong predictions for A: 3, Wrong predictions for B: 6137, general errors 6140
• 1092-0.97.hdf5, Wrong predictions for A: 6, Wrong predictions for B: 6325, general errors 6331
• 1095-0.97.hdf5, Wrong predictions for A: 0, Wrong predictions for B: 6408, general errors 6408
• 1126-0.97.hdf5, Wrong predictions for A: 0, Wrong predictions for B: 6413, general errors 6413

As you can see, the model seems to improve accuracy when predicting A, even if accuracy falls on B more than A improves.

Edit: Here is my code https://github.com/tan2684/Modified-Keras-Resnet