Solved – Does it make sense to use an Early Stopping Metric like “mae” instaed of “val_loss” for regression problems

kerasmachine learningmetricneural networks

I am performing a regression on a Dataset and try to replace a mathematical Model with a Neural Network.

To avoid overfitting I decided to use the Early Stopping Callback Function of Keras. So far I have been told, that the metrics to monitor for this problem should be "val_loss" but when I try to do that, the neural networks stops training very early.

I tried to monitor the mean absolute Error 'mae' and seem to get much better results, but I found no other example doing this so I am not sure if I am making another mistake which I am not aware of. Is there a reason not to do this? Or is there maybe even another metric that i should consider using?

dataset = np.loadtxt("output_20180804.out", delimiter=",")

X  = dataset[0:5000,4:7]
Y  = dataset[0:5000,0:4]

tbCallBack = TensorBoard(log_dir='./Graph{}', histogram_freq=0, write_graph=True, write_images=True) #TensorBoard Monitoring
esCallback = EarlyStopping(monitor='val_loss',
                           min_delta=0,
                           patience=2,
                           verbose=1,
                           mode='auto')


#create Layers
visible = Input(shape=(3,))
x = Dropout(.1)(visible)
x = Dense(60)(x)
output = Dense(4)(x)


Optimizer = optimizers.Adam(lr=0.0001
                            #amsgrad = True
                           )
model = Model(inputs=visible, outputs = output)
model.compile(optimizer=Optimizer,
              loss=['mse'],
              metrics=['mae']
              )
model.fit(X, Y, epochs=100000, batch_size=200, shuffle=True, validation_split=0.35, callbacks=[tbCallBack, esCallback])




print(model.get_weights())



# evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))


savestring = 'dense_100_epo100000_batch800_val02_20180807.h5'
model.save(savestring)

model.summary()

Best Answer

In order to prevent overfitting, EarlyStopping should monitor a validation metric. Because your loss function is the mse, by monitoring val_loss you are essentially monitoring the validation Mean Squared Error. If you think that mae is a better metric for your task, you should monitor val_mae instead.

Why monitor a validation metric when performing early stopping?

Early stopping, is mostly intended to combat overfitting in your model. Overfitting is a phenomenon, commonly occurring in Machine Learning, where a model performs worse on a validation/test set than the training set.

Take a look at the image below:

The red and blue lines depict the model's loss, during training, on the training/validation sets respectively. As you can see, while the training loss continues to drop, the validation loss initially plateaus and then starts to raise. This happens because after some point the model learns to memorize the training data, while forgetting how to effectively differentiate between the classes. When we reach this point, even though we think the model is improving (indicated by all the training set metrics), actually has begun to diverge from its goal (to perform well on unseen data).

Early stopping is a method of combating this. By terminating the model, before it has completed its training we might get a better performance on unseen data. This works by monitoring a validation metric and terminating the model when this metric stops dropping.

Related Question