Solved – My p-values increase when adding variables: is the model still valid

p-valueregression

Forewarning- I'm not a very advanced in regression.
(Updated with 2 edits)

I'm running multiple regressions with Excel and noticed that my p-values are becoming insignificant when adding more variables. My model is very simple.
N=9
with a statistically significant variable, then I add another variable (which by itself is significant) and the p-value jumps into non-significant land.

I've read that this could be because of multicollinearity: should I be concerned with this since I'm only using the model to predict? How do I confirm this in Excel, and if so is my model not valid?

Edit – Thanks for your input guys:

My process is to test each variable one by one, and if it registers a significant p value (less than .05) and a high R2'd then I keep it in the model and add another variable.

This is where I am getting confused, as I add another variable the R2 (and adjusted R2) increase but the p values both increase above .05 (but independently are less that .05).

What does this mean? Is there any way to run a good multiple regression model in excel using a small sample size (N of 9-15) for prediction without the above problem.

Thanks again.

Edit #2 – I read through some of the other threads and a recurring theme is that this happens b/c of collinearity. I did a VIF test and the value is 1.97 which is below 2.5 so doesn't set off any alarms.
So if my two variables don't have collinearity, whats happening to the p values? i.e. both are significant independently but only 1 is when I regress both variables?

Best Answer

What you are describing is a variant of stepwise model building, which, whether based on the p-values of individual predictors, or on measures of overall model performance like $R^{2}$ or AIC results in a host of problems rendering inference from such models suspect:

  • deflated p values
  • inflated overall model performance values
  • inflated coefficients
  • inflated F statistics for the whole model
  • highly probable exclusion of true predictors
  • highly probable inclusion of false predictors

Most of these problems arise because you are neither accounting for nor reporting the string of invisible "conditional on all these previous rejection decisions" at each step of the model building process.

So how to build a model if not by a stepwise approach? By theoretically justifying your model variables a priori and embrace reporting negative effects for a given model (i.e. don't just report coefficients with p-values below your significance threshold).


References
Babyak, M. A. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66:411–421.

Henderson, D. A. and Denison, D. R. (1989). Stepwise regression in social and psychological research. Psychological Reports, 64:251–257.

Huberty, C. J. (1989). Problems with stepwise methods—better alternatives. Advances in Social Science Methodology, 1:43–70.

Hurvich, C. M. and Tsai, C.-L. (1990). The impact of model selection on inference in linear regression. The American Statistician, 44(3):214–217.

Lovell, M. C. (1983). Data mining. The Review of Economics and Statistics, 65(1):1–12.

Malek, M. H. and Coburn, D. E. B. J. W. (2007). On the inappropriateness of stepwise regression analysis for model building and testing. European Journal of Applied Physiology, 101(2):263–264.

McIntyre, S. H., Montgomery, D. B., Srinivasan, V., and Weitz, B. A. (1983). Evaluating the statistical significance of models developed by stepwise regression. Journal of Marketing Research, 20(1):1–11.

Pope, P. T. and Webster, J. T. (1972). The use of an $F$-statistic in stepwise regression procedures. Technometrics, 14(2):327–340.

Rencher, A. C. and Pun, F. C. (1980). Inflation of R$^2$ in best subset regression. Technometrics, 22(1):49–53.

Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4):1237–1282.

Sribney, B., Harrell, F., and Conroy, R. (2011). Problems with stepwise regression.

Steyerberg, E. W., Eijkemans, M. J., and Habbema, J. D. F. (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of clinical epidemiology, 52(10):935–942.

Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4):525–534.

Wilkinson, L. (1979). Tests of significance in stepwise regression. Psychological Bulletin, 86(1):168–174.

Related Question