multiple regressionp-value
I'm new to the concept of regression and its application .What is the criteria for eliminating variables in a multiple linear regression model whose purpose is prediction ?
My favorite way of showing the results of a basic multiple linear regression is to first fit the model to normalized (continuous) variables. That is, z-transform the $X$s by subtracting the mean and dividing by the standard deviation, then fit the model and estimate the parameters. When the variables are transformed in this way, the estimated coefficients are 'standardized' to have unit $\Delta Y/\Delta sd(X)$. In this way, the distance the coefficients are from zero ranks their relative 'importance' and their CI gives the precision. I think it sums up the relationships rather well and offers a lot more information than the coefficients and p.values on their natural and often disparate numerical scales. An example is below:
EDIT: Another possibility is to use an 'added variable plot' (i.e. plot the partial regressions). This gives another perspective in that it shows the bivariate relations between $Y$ and $X_i$ AFTER THE OTHER VARIABLES ARE ACCOUNTED FOR. For example, the partial regressions of $Y \sim X_1 + X_2 + X_3$ would give bivariate relations between $X_i$ against the residuals of $Y$ after regressing against the other two terms. You would go on to do this for each variable. Function avPlots() from library car gives these plots from a fitted lm object. An example is below:
I would second @EdM's comment (+1) and suggest using a regularised regression approach.
I think that an elastic-net/ridge regression approach should allow you to deal with collinear predictors. Just be careful to normalise your feature matrix $X$ appropriately before using it, otherwise you will risk regularising each feature disproportionately (yes, I mean the $0/1$ columns, you should scale them such that each column has unit variance and mean $0$).
Clearly you would have to cross-validate your results to ensure some notion of stability. Let me also note that instability is not a huge problem because it actually suggests that there is not obvious solution/inferential result and simply interpreting the GLM procedure as "ground truth" is incoherent.
Best Answer
First, there is no one criterion. Predictive modeling is not itself a decision tree, it is a science, a trade, and an artform. If you go looking for hard rules, they will often lead you astray.
Given that, neither of the criteria you state are particularly relevant for predictive modeling.
Much better is a disciplined regularization strategy (ridge, or some combination of ridge with LASSO), a broadly specified model (use splines to capture non-linearities, include reasonably hypothesized interactions), and a steady dose of cross validation.
No. Again, p-values are not tools that are intended to assess the predictive power of a model or variable. They have little to nothing to say about that issue. To assess the predictive power of a model, you need to use cross validation or a help out test data set. Make predictions, compare them to the truth (on data that the model has not yet seen).
Assessing the predictive power of a variable is much more subtle, and there is no general consensus on what that even means, or what a solution would actually measure.
If you would like to control the complexity of a model for purposes of maximizing predictive power, then you need to use regularization. You can search this site for "ridge regression" and "lasso regression" to get started learning about this technique.