I've been working with Convolutional Neural Networks (CNNs) for some time now, mostly on image data for semantic segmentation/instance segmentation. I've often visualized the softmax of the network output as a "heat map" to see how high per pixel activations for a certain class are.
I've interpreted low activations as "uncertain" / "unconfident" and high activations as "certain" / "confident" predictions. Basically this means interpreting the softmax output (values within $(0,1)$) as a probability or (un)certainty measure of the model.
(E.g. I've interpreted an object/area with a low softmax activation averaged over its pixels to be difficult for the CNN to detect, hence the CNN being "uncertain" about predicting this kind of object.)
In my perception this often worked, and adding additional samples of "uncertain" areas to the training results improved results on these. However I've heard quite often now from different sides that using/interpreting softmax output as an (un)certainty measure is not a good idea and is generally discouraged. Why?
EDIT:
To clarify what I'm asking here I'll elaborate on my insights so far in answering this question. However none of the following arguments made clear to me ** why it is generally a bad idea**, as I was repeatedly told by colleagues, supervisors and is also stated e.g. here in section "1.5"
In classification models, the probability vector obtained at the end of the pipeline (the softmax output) is often erroneously interpreted as model confidence
or here in section "Background" :
Although it may be tempting to interpret the values given by the final softmax layer of a convolutional neural network as confidence scores, we need to be careful not to read too much into this.
The sources above reason that using the softmax output as uncertainty measure is bad because:
imperceptible perturbations to a real image can change a deep network’s softmax output to arbitrary values
This means that softmax output isn't robust to "imperceptible perturbations" and hence it's output isn't usuable as probability.
Another paper picks up on the "softmax output = confidence" idea and argues that with this intuition networks can be easily fooled, producing "high confidence outputs for unrecognizable images".
(…) the region (in the input domain) corresponding to a particular class may be much larger than the space in that region occupied by training examples from that class. The result of this is that an image may lie within the region assigned to a class and so be classified with a large peak in the softmax output, while still being far from images that occur naturally in that class in the training set.
This means that data that is far away from training data should never get a high confidence, since the model "can't" be sure about it (as it has never seen it).
However: Isn't this generally simply questioning the generalization properties of NNs as a whole? I.e. that the NN's with softmax loss don't generalize well to (1) "imperceptible perturbations" or (2) input data samples that are far away from the training data, e.g. unrecognizable images.
Following this reasoning I still don't understand, why in practice with data that is not abstractly and artifically altered vs. the training data (i.e. most "real" applications), interpreting the softmax output as a "pseudo-probability" is a bad idea. After all, they seem to represent well what my model is sure about, even if it isn't correct (in which case I need to fix my model). And isn't model uncertainty always "only" an approximation?
Best Answer
This question can be answered more precisely than the current answers. Fixing the deviation between the predicted probabilities (the output of the softmax layer of a neural network) and their true probabilities (which represent a notion of confidence), is known as calibration or reliability curves.
The issue with many deep neural networks is that, although they tend to perform well for prediction, their estimated predicted probabilities produced by the output of a softmax layer can not reliably be used as the true probabilities (as a confidence for each label). In practice, they tend to be too high - neural networks are 'too confident' in their predictions.
Chuan Go et. al., working with Kilian Weinberger, developed an effective solution for calibrating the predicted probabilities of neural networks in this paper: https://arxiv.org/abs/1706.04599
This paper also explains how predicted probabilities can be interpreted as confidence measures when the predicted probabilities are correctly calibrated.