I am medical student with only a very basic background in CNN's and TensorFlow, so I appreciate any advice.
I implemented a CNN with ~20K black and white images that are 32×32 pixels. I chose to partition my dataset into ~19K images as training data and ~1K images as testing data. Using Tensorflow and TFLearn, I classified my images into one of three classes and plotted accuracy over time with TensorBoard.
The resulting accuracy graph looks like this:
My questions are:
- Why does the accuracy suddenly spike down randomly into the mid 90 percents, then shoot back up?
- How can I tell if I am "overfitting" my data set?
- Is my breakdown of training vs. testing data optimal? (i.e. I chose to use 5% of my images to test and 95% of my images to train. Is there a better combination?)
Best Answer
It depends on your parameter update strategy, your parameter update strategy's parameters (e.g., learning rate), your network, and your data set.
E.g. see the spike of Rmsprop even though the network had almost converged:
(Images credit: Alec Radford.)
As long as you have the same spike on your validation set, it shouldn't be an issue. If you wish to further investigate, you may want to look at the gradients, and whether the spikes happen on some particular samples.
See https://datascience.stackexchange.com/a/627/843.
Since it looks like the accuracy on the test set is pretty much 1, I wouldn't worry about overfitting.
The test set should reflect as best as possible the actual samples on which your system will be used in production.