Neural Networks – Understanding Abrupt Jumps in H2O DeepLearning Model Score History

h2oneural networksr

I train deep learning models on the fashion MNIST dataset using H2O with the R interface. My score history often looks strange: the end is jumping up or down as two different values are reported for the last epoch.

Here is an example:
enter image description here

How could the training RMSE be 0.38416 and also 0.39117 at the same time after 10 epochs? (Sometimes I get even more expressed differences where the last value is almost double of the previous one.) The same is true for other metrics as well. See the plot for the classification error:
enter image description here

Could you please explain how rows 11 and 12 differ?

Here is the reproducible code (you can download the data from here:

library(tidyverse)
library(h2o)

h2o.init()

fmnist_data <- read_csv("fashion-mnist_train.csv")

# standardization
fmnist_data <- mutate(fmnist_data, 
    label = as.factor(label),
    across(-label, ~./255)
)

# data splitting
data_split <- h2o.splitFrame(as.h2o(fmnist_data), ratios = 0.05, seed = 1234)
fmnist_train <- data_split[[1]]
fmnist_holdout <- data_split[[2]]

# training
dl_fit <- h2o.deeplearning(
    x = 2:785,
    y = "label",
    hidden = c(8, 8),
    training_frame = fmnist_train,
    validation_frame = fmnist_holdout,
    seed = 12345,
    reproducible = TRUE
)
plot(dl_fit, metric = "classification_error")
h2o.scoreHistory(dl_fit)

Best Answer

I think what is happening is that the training does 10 epochs of training, and then chooses the epoch with the lowest validation score as the model to use. Which is at epoch 9 in the example you give.

So the final line of the scoring history just repeats the metrics for the best version of the model it chose. I.e. it is saying "this is the score of the best model after 10 epochs".

Related Question