There is no free lunch in statistics. Elastic Net reduces overfitting (lowers variance) at the cost of increasing bias. With OLS, you could fit a model with all 50 variables. This OLS model would have very low bias (under certain assumptions, the coefficient estimates may be unbiased) but suffer from high variance (overfitting).
In your case, you mentioned that the OLS coefficients look very different than the Elastic Net coefficients, even though both models use the same 10 variables. The difference may be due to bias introduced by the fact that Elastic Net does not compute the coefficients by minimizing the residual sum of squares (which is how OLS computes the coefficients). Elastic net computes the coefficients by minimizing the "penalized" residual sum of squares.
Alternatively, the coefficient estimates may be different between OLS and Elastic Net due to sample size. With small sizes, p-values from OLS may not be reliable. With small sample sizes, the bias from elastic net may also be high.
Here's a simulated example using $n=25$. The "true model" contains only two variables, $x1$ and $x2$, with the "true coefficients" of 2 and 3. Due to small sample size and high irreducible error, the p-value for x1 is high (>22%). The coefficient differences between the two models are also high.
set.seed(1983)
nobs <- 25
x1 <- rnorm(nobs, 10, 10)
x2 <- rnorm(nobs, 20, 20)
x3 <- rnorm(nobs, 30, 30)
x4 <- rnorm(nobs, 40, 40)
y <- 100 + 2*x1 + 3*x2 + rnorm(nobs,0,100)
df <- data.frame(y=y, x1=x1, x2=x2, x3=x3, x4=x4)
### fit a linear model
lm.mod <- lm(y ~ ., data=df)
summary(lm.mod)
### fit an elastic net model using 5-fold CV
library(caret)
set.seed(1984)
enet.mod <- train(y ~ ., data=df, method="glmnet", tuneLength=5, trControl=trainControl(method="cv", number=5))
coef(enet.mod$finalModel, enet.mod$bestTune$lambda)
### compute diffs between coefs
lm.mod$coefficients - t(coef(enet.mod$finalModel, enet.mod$bestTune$lambda))[1,]
When the sample size is increased to $n = 1000$, the p-value for $x1$ is very low and the coefficient differences between the two models are small.
set.seed(1983)
nobs <- 1000
x1 <- rnorm(nobs, 10, 10)
x2 <- rnorm(nobs, 20, 20)
x3 <- rnorm(nobs, 30, 30)
x4 <- rnorm(nobs, 40, 40)
y <- 100 + 2*x1 + 3*x2 + rnorm(nobs,0,100)
df <- data.frame(y=y, x1=x1, x2=x2, x3=x3, x4=x4)
### fit a linear model
lm.mod <- lm(y ~ ., data=df)
summary(lm.mod)
### fit an elastic net model using 5-fold CV
library(caret)
set.seed(1984)
enet.mod <- train(y ~ ., data=df, method="glmnet", tuneLength=5, trControl=trainControl(method="cv", number=5))
coef(enet.mod$finalModel, enet.mod$bestTune$lambda)
### compute diffs between coefs
lm.mod$coefficients - t(coef(enet.mod$finalModel, enet.mod$bestTune$lambda))[1,]
Finally I got the same values with the following code :
Python
# normalize function that gives the same with R
def mystandardize(D):
S = np.std(D, axis=0, ddof=1)
M = np.mean(D, axis = 0)
D_norm = (D-M)/S
return [D_norm, M, S]
Y_norm_train = pd.DataFrame(mystandardize(Y_train)[0])
glmnet_regr = linear_model.ElasticNet(alpha=1, l1_ratio = 0.01,
fit_intercept = True, normalize = False, tol=0.0000001, max_iter = 100000)
glmnet_regr.fit(X_train, Y_norm_train)
R
y_norm_train <- scale(y[train_idx])
glmnet_obj_norm <- glmnet(x_train, y_norm_train, alpha=0.01, lambda = 1,
thresh = 1e-07, standardize = FALSE, intercept=TRUE, standardize.response = FALSE)
print_coef(glmnet_obj_norm)
Best Answer
The script above gives:
Using naive=FALSE for ElasticNet is just transforming the coefficients of naive=TRUE according to formula: coef(ENet) = (1 + lambda) * coef(NaiveENet)
The glmnet gives "ready-to-use" coefficients. However they are different from those of ElasticNet. (maybe different algorithms)
You'll get ElasticNet and glmnet "aligned" if you don't normalize the predictors.
How predict works?
enet prediction is linear in enet naive prediction (formula):
enet naive is almost identical to glmnet prediction: