Solved – How is the standard error of the estimated prediction error calculated

data miningpredictive-modelsregression

I'm reading the book "The Elements of Statistical Learning" (Hastie, Tibshirani, and Friedman). At page 62, they present the estimated prediction curves with the standard errors for best subset selection, ridge regression, lasso and PCR.
enter image description here
The question is: how is the standard error of a prediction error calculated?
Thank you.

Best Answer

You can see that the $y$-axis is labeled CV-Error, which gives a clue.

Using cross validation, we can do the following:

for each cross validation fold train[i], test[i]:
    for each model parameter p:
        train model with parameter p on train[i]
        score model with parameter p on test[i]
        set s_{ip} = the score above

Then the line plot in the above graphs is

mean(s_{ip}) where p is fixed and i varaies

And the error bars are derived from

sd(s_{ip}) where p is fixed and i varaies

So, in words, for each parameter the mean out of fold score is calculated across all the folds, and the standard error of the out of fold score is calculated across all the folds.

Related Question