My problem has 3 classes.
I want to implement a single neural network, which has only one neural at the output layer, to classify these 3 classes.
Is there any way to implement it?
Solved – Multi-class classification using a single neural network with only one output neural
artificial intelligenceclassificationmachine learningneural networks
Related Solutions
I would use a single network with multiple outputs (instead of multiple single-output networks, see this question for the differences), with the following approach:
The whole network will be train with some hidden layers and final softmax layer as the output. Assuming that most of the learned features will be useful for multiple classes (which is the reason why we prefer a single network), when adding a new class, most of the network might remain fixed. The last hidden layer would be regarded as inputs to the last layer of the original network. So only a single layer should be trained when adding a new class.
Of course, it will be useful to allow training of the whole network for fine tuning, but as the number of classes is increased the importance of such fine tuning is decreased.
Tensorflow has a loss function weighted_cross_entropy_with_logits, which can be used to give more weight to the 1's. So it should be applicable to a sparse multi-label classification setting like yours.
From the documentation:
This is like sigmoid_cross_entropy_with_logits() except that pos_weight, allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error.
The argument pos_weight is used as a multiplier for the positive targets
If you use the tensorflow backend in Keras, you can use the loss function like this (Keras 2.1.1):
import tensorflow as tf
import keras.backend.tensorflow_backend as tfb
POS_WEIGHT = 10 # multiplier for positive targets, needs to be tuned
def weighted_binary_crossentropy(target, output):
"""
Weighted binary crossentropy between an output tensor
and a target tensor. POS_WEIGHT is used as a multiplier
for the positive targets.
Combination of the following functions:
* keras.losses.binary_crossentropy
* keras.backend.tensorflow_backend.binary_crossentropy
* tf.nn.weighted_cross_entropy_with_logits
"""
# transform back to logits
_epsilon = tfb._to_tensor(tfb.epsilon(), output.dtype.base_dtype)
output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)
output = tf.log(output / (1 - output))
# compute weighted loss
loss = tf.nn.weighted_cross_entropy_with_logits(targets=target,
logits=output,
pos_weight=POS_WEIGHT)
return tf.reduce_mean(loss, axis=-1)
Then in your model:
model.compile(loss=weighted_binary_crossentropy, ...)
I have not found many resources yet which report well working values for the pos_weight in relation to the number of classes, average active classes, etc.
Best Answer
If your problem is truly classification you shouldn't implement output as one neuron, since you add strong a prior assumption about your class correlation. You can use one neuron only for binary class problems. Use 3 neurons outputs for 3-class problems. Check this example https://janakiev.com/notebooks/keras-iris/