I have a model that is trained on n classes and has more than 95% accuracy on the test set.
The model is going to receive a mixture of images that are either from one of the n classes or from unknown classes. My guess / expectation was that in the latter case, the distribution of values on the last layer (softmax) would be more or less uniform. Then to spot an alien image, it would be enough to threshold the prediction layer.
Unfortunately that is not the case, and the model seem to express a very high level of confidence every time. Even worse (using keras):
>>> max(model.predict(np.random.rand(1, 299, 299, 3))[0])
1.0
So basically the model is 100% sure that random noise is one of the class.
How to detect the images that do not belong to one of the classes that the model was trained on?
Note: everything has been normalized and in Keras predict returns probability, not class numbers.
Best Answer
You have not informed your model that there are other options, so in some ways it may be unsurprising that it guesses one. The softmax function in particular is designed to accentuate even slight differences in its inputs.
I would suggest adding a "none of the above" class to your model with appropriate training images of this class. Depending on what your application is, these "none of the above" images may be pure noise (as in your example), or actual images that are not from any of your target classes. Since the class of images not from any of your target classes will likely be quite large, I'd suggest picking images that are likely to appear in your application.
If this is insufficient, then you may want to look into estimating distributions to more explicitly find the images that fall far outside the distribution of images observed during training.