Solved – How to perform stepwise regression without intercept

regressionstepwise regression

I have to implement a regression model and I have about 30 variables in the model. Some of the variables do not have much influence on the model, but I need to use a formalized method for eliminating variables. I ended up with the AIC and BIC methods and I have been using the step function in R. The problem is that I need to do a regression model that goes through the origin. In R I use the following statement for regression through the origin,

g <- lm(Pcubes ~ 0 + ., data)

and a regression with intercept

g <- lm(Pcubes ~ ., data)

I then use the step function as

l <- step(g)

and the summary for the new the regression model l, summary(l). Whenever I use the regression through the origin, I get a summary from summary(l) where some variables are not significant to the model and I also get a new regression model that looks as follows,

lm(formula = Pcubes ~ X1 + X4 + X6 + X7 + X8 + X13 + X14 + X17 + 
    X18 + X23 + X24 + X25 + X29 + X30 + X40 + X41 + X44 + X45 + 
    X46 + X48 + X52 + X53 + X54 + X55 + X71 + X82 + X91 + X96 - 1, data = data)

The new model subtracts a 1. And the step function also results in different variables for regression through the origin and regression with intercept. So my questions are, can I use the step function in R for regression through the origin or can I just ignore the -1 in the model or should I use the regression with intercept to choose the best variables and implement the regression through the origin with those variables?

Best Answer

For the lm model with intercept lm(y~.), the simplest model to consider by step function is lm(y~1). So it will try to drop the intercept at all.

For the lm model without intercept lm(y~-1+.), step function will compare the model family all without intercept but different predictor x.

Related Solutions

Solved – Multiple regression with no origin and mix of directly entered and stepwise entered variables using R

I think you can set up your base model, that is the one with your 12 IVs and then use add1() with the remaining predictors. So, say you have a model mod1 defined like mod1 <- lm(y ~ 0+x1+x2+x3) (0+ means no intercept), then

add1(mod1, ~ .+x4+x5+x6, test="F")

will add and test one predictor after the other on top of the base model.

More generally, if you know in advance that a set of variables should be included in the model (this might result from prior knowledge, or whatsoever), you can use step() or stepAIC() (in the MASS package) and look at the scope= argument.

Here is an illustration, where we specify a priori the functional relationship between the outcome, $y$, and the predictors, $x_1, x_2, \dots, x_{10}$. We want the model to include the first three predictors, but let the selection of other predictors be done by stepwise regression:

set.seed(101)
X <- replicate(10, rnorm(100))
colnames(X) <- paste("x", 1:10, sep="")
y <- 1.1*X[,1] + 0.8*X[,2] - 0.7*X[,5] + 1.4*X[,6] + rnorm(100)
df <- data.frame(y=y, X)

# say this is one of the base model we think of
fm0 <- lm(y ~ 0+x1+x2+x3+x4, data=df)

# build a semi-constrained stepwise regression
fm.step <- step(fm0, scope=list(upper = ~ 0+x1+x2+x3+x4+x5+x6+x7+x8+x9+x10, 
                                lower = ~ 0+x1+x2+x3), trace=FALSE)
summary(fm.step)

The results are shown below:

Coefficients:
   Estimate Std. Error t value Pr(>|t|)    
x1   1.0831     0.1095   9.888 2.87e-16 ***
x2   0.6704     0.1026   6.533 3.17e-09 ***
x3  -0.1844     0.1183  -1.558    0.123    
x6   1.6024     0.1142  14.035  < 2e-16 ***
x5  -0.6528     0.1029  -6.342 7.63e-09 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 1.004 on 95 degrees of freedom
Multiple R-squared: 0.814,  Adjusted R-squared: 0.8042 
F-statistic: 83.17 on 5 and 95 DF,  p-value: < 2.2e-16

You can see that $x_3$ has been retained in the model, even if it proves to be non-significant (well, the usual caveats with univariate tests in multiple regression setting and model selection apply here -- at least, its relationship with $y$ was not specified).

Solved – Interpretation of coefficients in multiple regression without intercept

The only situations where you might consider excluding the constant are models where it is trivially true that the constant has to be 0. The fact that you looked at the significance level is in itself sufficient evidence that it is not trivially true that the constant is 0. So I would leave it in.

Best Answer

Related Solutions

Solved – Multiple regression with no origin and mix of directly entered and stepwise entered variables using R

Solved – Interpretation of coefficients in multiple regression without intercept

Related Question