Solved – Multiple regression with small data sets

regressionsmall-sample

I have a dataset of project case studies for a new type of research method for Government agencies to support decision making activities. My task is to develop an estimation method based on past experience for future projects for estimation purposes.

My dataset is limited to 50 cases. I have 30+ (potential) predictors recorded and one response variable (i.e. hours taken to complete the project).

Not all predictors are significant, using step-wise selection techniques I'm expecting number of prediction variables is likely to be in the 5-10 variable range. Although I'm struggling to get a predictor set using the standard appraoches in tools like PASW (SPSS).

I'm well aware of all the material talking about rules of thumb for sample sizes and predictor variable to case ratios. My dilemma is that it's taken close to 10 years to collect 50 cases as it is, so it's about as good as it will get.

My question is what should I do to get the most out of this small sample set?

That is any good references for dealing with small smaple sets? Changes in p-value significance? Changes to step-wise selection approaches? Use of transforms such as centre-ing or log?

Any advice is appreciated.

Best Answer

As you want to select a few predictors from your data set, I would suggest a simple linear regression with $L_1$ penalty or using the LASSO (penalized linear regression). Your case is suited for regression with LASSO penalty as your sample size, $n = 50$, and the number of predictors, $p=30$. Changing the tuning parameter will select the number of predictors you want to choose.

If you can give details about the distribution of your variables, I can be more specific.

I don't use SPSS, but this can be done easily in R using the glmnet function in the package of the same name. If you look in the manual, it contains a generic example (very first one, for gaussian case) which will solve your problem. I am sure, similar solution must exist in SPSS.

Related Solutions

Solved – Backward Stepwise Selection

Set up an exit criterion for the p-value. Any independent variable with a p-value higher than this criterion will be removed. There isn't any golden rule on what to set, for exploratory purpose you may see p > 0.2 being removed. Someone may used 0.05, etc.
Set up your full model. Generally, it's the model that contains all independent variables from which you wish to select the predictive bunch.
Fit the full model.
Check the p-values (or t-statistics). If all p-values are less than the exit criterion, then it's the final model. If any of them exceeds the exit criterion, then the one with the highest p-value (aka lowest t-statistics) will be removed. This is pertinent to your "dropping the variable with the smallest z-score."
Fit the model with the remaining independent variables again, repeat steps 4 and 5 until either no independent variable is left or no independent variable has a p-value larger than the exit criterion.

Here is an example using Stata:

. sysuse auto
. stepwise, pr(.2): reg mpg weight turn headroom foreign price
                      begin with full model
p = 0.9238 >= 0.2000  removing price
p = 0.7047 >= 0.2000  removing headroom
p = 0.2045 >= 0.2000  removing turn

      Source |       SS       df       MS              Number of obs =      74
-------------+------------------------------           F(  2,    71) =   69.75
       Model |   1619.2877     2  809.643849           Prob > F      =  0.0000
    Residual |  824.171761    71   11.608053           R-squared     =  0.6627
-------------+------------------------------           Adj R-squared =  0.6532
       Total |  2443.45946    73  33.4720474           Root MSE      =  3.4071

------------------------------------------------------------------------------
         mpg |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      weight |  -.0065879   .0006371   -10.34   0.000    -.0078583   -.0053175
     foreign |  -1.650029   1.075994    -1.53   0.130      -3.7955    .4954422
       _cons |    41.6797   2.165547    19.25   0.000     37.36172    45.99768
------------------------------------------------------------------------------

The independent variables are weight, turn, headroom, foreign, and price. The dependent variable is mpg. The exit criterion is p > 0.2. The first got removed was price (p = 0.9238), followed by headroom (p = 0.7047) and turn (p = 0.2045). The remaining ones have p < 0.2, so they stay.

Solved – Multiple Linear Regression with small dataset

Yes, as Nick mentions, it is definitely a stretch for using models with assumptions. If you just want to predict, I would suggest that you use nearest neighbor method as a non-parametric method for prediction.

If you have little more points, you can use partial least squares (PLS) if more variables than data points.

As pointed out by others, three data points is too few for developing a regression model. I would suggest to collect at least 20 data points to run a regression model.

Best Answer

Related Solutions

Solved – Backward Stepwise Selection

Solved – Multiple Linear Regression with small dataset

Related Question