Solved – Multiple linear regression with low sample size

regressionsmall-sample

I have a dataset with 12 observations for a particular response variable. In this case the response variable is related with genetic diversity parameters. I want to perform a multiple linear regression using 3 predictors. The assumptions of independence, normality of residuals and homogeneity of variances were not violated. However, I am aware that my sample size is most likely not suitable to perform this analysis (I am following the rule of thumb 10 observations per 1 predictor).

I thought that building a set of nested models comprising all combinations of one or two variables would minimize this issue. However, I am not quite confident regarding the implications in my results of using this procedure because effect sizes and model Akaike weights are no longer comparable, right? Can you please help me with this issue?

Also, do you know other relevant methodological alternatives to apply under the described conditions (i.e. low sample size and 3 predictors) or any relevant references about this subject?

Thank you very much for your time.

Best Answer

If your scientific question is about all three predictors and you have no possibility of collecting more data then you just go ahead. It is not ideal of course as your estimates will be rather imprecise but I imagine you already know that. Doing some form of variable selection may be mis-leading since it tends to be data driven and so may have been unduly affected by some minor quirk in your data-set and so not generalise well to other situations .

Related Solutions

Solved – Is randomization reliable with small samples

You are correct to point out the limitations of randomisation in dealing with unknown confounding variables for very small samples. However, the problem is not that the P-values are not reliable, but that their meaning varies with sample size and with the relationship between the assumptions of the method and the actual properties of the populations.

My take on your results is that the P-values performed quite well until the difference in the subgroup means was so large that any sensible experimenter would know that there was an issue prior to doing the experiment.

The idea that an experiment can be done and analysed without reference to a proper understanding of the nature of the data is mistaken. Before analysing a small dataset you must know enough about the data to be able to confidently defend the assumptions implicit in the analysis. Such knowledge commonly comes from prior studies using the same or similar system, studies that can be formal published works or informal 'preliminary' experiments.

Bootstrap Methodology – How It Addresses Small Sample Size Issues

I remember reading that using the percentile confidence interval for bootstrapping is equivalent to using a Z interval instead of a T interval and using $n$ instead of $n-1$ for the denominator. Unfortunately I don't remember where I read this and could not find a reference in my quick searches. These differences don't matter much when n is large (and the advantages of the bootstrap outweigh these minor problems when $n$ is large), but with small $n$ this can cause problems. Here is some R code to simulate and compare:

simfun <- function(n=5) {
    x <- rnorm(n)
    m.x <- mean(x)
    s.x <- sd(x)
    z <- m.x/(1/sqrt(n))
    t <- m.x/(s.x/sqrt(n))
    b <- replicate(10000, mean(sample(x, replace=TRUE)))
    c( t=abs(t) > qt(0.975,n-1), z=abs(z) > qnorm(0.975),
        z2 = abs(t) > qnorm(0.975), 
        b= (0 < quantile(b, 0.025)) | (0 > quantile(b, 0.975))
     )
}

out <- replicate(10000, simfun())
rowMeans(out)

My results for one run are:

     t      z     z2 b.2.5% 
0.0486 0.0493 0.1199 0.1631

So we can see that using the t-test and the z-test (with the true population standard deviation) both give a type I error rate that is essentially $\alpha$ as designed. The improper z test (dividing by sample standard deviation, but using Z critical value instead of T) rejects the null more than twice as often as it should. Now to the bootstrap, it is rejecting the null 3 times as often as it should (looking if 0, the true mean, is in the interval or not), so for this small sample size the simple bootstrap is not sized properly and therefore does not fix problems (and this is when the data is optimally normal). The improved bootstrap intervals (BCa etc.) will probably do better, but this should raise some concern about using bootstrapping as a panacea for small sample sizes.

Related Question