I'm running a multiple linear regression on a set of sports data. When I run the regression on one season, which has 380 data points and which I thought was a fair amount, I get quite a high p-value on one of my independent variables. However, when I run the regression on all my data points (I have more than 3000 data points in total), the p-value decreases from .97 to .02. As I add more data points, the p-value decreases further. My question is: is my variable really significant or am I just decreasing the p-value by adding more data points?
Solved – Does it make sense for p-value to decrease with more data points
multiple regressionp-value
Related Solutions
Both options are the same answer.
Sampling error depends on sample size. If I understand allright, you are increasing the amount of evidence when moving from day 8 to day 15. It is expected that the "true relationships" among independent variables and between them and the dependent one in the population will start to reveal as sample size increases. Think in a signal-to-noise ratio increasing. Think in the number of times you need to throw a coin until you are confident that is a fair one.
Anyway, I would measure the progress in those relationships by observing the stability of the coefficients instead of the changes in p-values. P-values are informing you about null hipotheses, that I guess in this case are testing that each of the coefficients equals zero in the population. The confidence intervals will be reduced as sample size increases (under the proper assumptions being valid) so any constant coefficient will start to "be different from zero" (reject the null hipothesis) with enough data.
[Note 1. You have to also be aware of colineality among the "independent" variables. This may also make your (partial regression) coefficients to change]
[Note 2. I would also try to follow the advice by Peter Flom above]
What you are describing is a variant of stepwise model building, which, whether based on the p-values of individual predictors, or on measures of overall model performance like $R^{2}$ or AIC results in a host of problems rendering inference from such models suspect:
- deflated p values
- inflated overall model performance values
- inflated coefficients
- inflated F statistics for the whole model
- highly probable exclusion of true predictors
- highly probable inclusion of false predictors
Most of these problems arise because you are neither accounting for nor reporting the string of invisible "conditional on all these previous rejection decisions" at each step of the model building process.
So how to build a model if not by a stepwise approach? By theoretically justifying your model variables a priori and embrace reporting negative effects for a given model (i.e. don't just report coefficients with p-values below your significance threshold).
References
Babyak, M. A. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66:411–421.
Henderson, D. A. and Denison, D. R. (1989). Stepwise regression in social and psychological research. Psychological Reports, 64:251–257.
Huberty, C. J. (1989). Problems with stepwise methods—better alternatives. Advances in Social Science Methodology, 1:43–70.
Hurvich, C. M. and Tsai, C.-L. (1990). The impact of model selection on inference in linear regression. The American Statistician, 44(3):214–217.
Lovell, M. C. (1983). Data mining. The Review of Economics and Statistics, 65(1):1–12.
Malek, M. H. and Coburn, D. E. B. J. W. (2007). On the inappropriateness of stepwise regression analysis for model building and testing. European Journal of Applied Physiology, 101(2):263–264.
McIntyre, S. H., Montgomery, D. B., Srinivasan, V., and Weitz, B. A. (1983). Evaluating the statistical significance of models developed by stepwise regression. Journal of Marketing Research, 20(1):1–11.
Pope, P. T. and Webster, J. T. (1972). The use of an $F$-statistic in stepwise regression procedures. Technometrics, 14(2):327–340.
Rencher, A. C. and Pun, F. C. (1980). Inflation of R$^2$ in best subset regression. Technometrics, 22(1):49–53.
Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4):1237–1282.
Sribney, B., Harrell, F., and Conroy, R. (2011). Problems with stepwise regression.
Steyerberg, E. W., Eijkemans, M. J., and Habbema, J. D. F. (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of clinical epidemiology, 52(10):935–942.
Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4):525–534.
Wilkinson, L. (1979). Tests of significance in stepwise regression. Psychological Bulletin, 86(1):168–174.
Best Answer
Let's say that your independent variable is $x_i$ and its regression coefficient is $\beta_i$. The p-value for $\beta_i$ is $P(t<| t^* |)+P(t>|t^*|)$ where $t^*=\frac{\beta_i}{\sqrt{(X'X)^{-1}_{ii}\frac{RSS}{n-q}}}$. $RSS$ is the residual sum of squares.
The p-value is large when $|t^*|$ is small, small when $|t^*|$ is large. But when $n$ grows, $RSS/(n-q)$ get smaller and $|t^*|$ larger, so the p-value decreases just because $n$ grows.
This is why "in large samples is more appropriate to choose a size of 1% or less rather than the 'traditional' 5%." (M. Verbeek, A Guide to Modern Econometrics, 3rd edition, §2.5.7, p. 32). If you choose 1%, your coefficient is not statitically significant when $p=0.02$.