Solved – Why are small sample sized avoided in meta-analyses

meta-analysissmall-sample

In most meta-analyses that I've read, the authors choose to exclude studies with really small sample sizes (e.g n=10). Why is that?

Speculation

I could speculate that one reason is that since the effect size from a study depends on the means of the groups as well as the pooled standard deviation, like so

$$\frac{\bar{X}_1-\bar{X}_2}{SD_{pooled}}$$

and small samples cannot be assumed to achieve enough of a normal distribution among their results, the pooled standard deviation will be terribly off, thereby possibly messing up the overall effect size.

Further, some authors argue that studies with large sample sizes, among other things, tend to be of higher quality than studies with smaller sample sizes, and that there might be a publication bias going on, with smaller studies having to have positive results to get published. For example, this is the argument laid forward by Nüesch et al., 2010, although small studies are here defined as $n<100$. However, I guess one could extrapolate from results like these to what to do with even smaller studies.

References

Nüesch et al., 2010. Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. BMJ, 2010 July 16, 10.1136/bmj.c3515.

Best Answer

If the meta-analysis uses fixed effects then small studies, other things being equal, will get very little weight and so the substantive conclusions are unlikely to be affected by including or excluding them.

If the meta-analysis uses random effects and there is substantial heterogeneity then the weights tend to become more equal. In that case even a small study may have almost the same weight as a large one. Some people feel this is undesirable because they believe that (a) small studies are of poorer quality (b) those small studies which they can find are not likely to be a random sample of all the small studies there have ever been.

Related Solutions

Meta-Analysis – Estimating Summary Statistics (Mean) from Published Studies

Your goal of obtaining a combined effect estimate is the central problem of meta-analysis, as you probably know. Before going too far, it may help you to have a textbook such as this one at hand to use as a reference and to give you some illustrative examples.

Regarding your question:

As you correctly state later, simply averaging the studies will not give appropriate weight to more accurate studies. So scratch this approach.
This is closer to the typically taken approach in meta-analysis (good intuition). Often a weighted average of the observed effect sizes is calculated where the weights correspond to the inverse standard deviations of the effect size estimates from your studies. This is also the approach used in a fixed-effects regression that meta-analyses often include.
It is not entirely clear what you mean by "reported mean weights" but I suppose it is the observed effect sizes from your studies. As mentioned in the previous bullet, it is indeed common to perform a weighted fixed effects regression, using the inverse standard deviation or variance as weight. Below, I will point you to some options.

A very useful R package for performing meta-analysis is metafor. This will provide you with many tools. Of particular interest might be the following.

If you have the observed effect sizes and their variances, you can use the function rma.uni with method = "FE" to perform a fixed effects meta-analysis.
A fixed effect analysis is not appropriate if there is heterogeneity in the studies. Use the Q-test to assess this. If there is heterogeneity, you should consider using a random effects model if you have enough studies. Set method = "REML" in rma.uni to perform one.
You may also want to examine your data for publication bias. Once you have a model from rma.uni you can create a funnel plot to see if you might have this problem (funnel plots do not prove publication bias is present, but indicate visually that it might be).
You can also use your output from rma.uni to create a forest plot, which may be helpful in visualizing your studies.

Solved – Why does finding small effects in large studies indicate publication bias

The answers here are good, +1 to all. I just wanted to show how this effect might look in funnel plot terms in an extreme case. Below I simulate a small effect as $N(.01, .1)$ and draw samples between 2 and 2000 observations in size.

The grey points in the plot would not be published under a strict $p < .05$ regime. The grey line is a regression of effect size on sample size including the "bad p-value" studies, while the red one excludes these. The black line shows the true effect.

As you can see, under publication bias there is a strong tendency for small studies to overestimate effect sizes and for the larger ones to report effect sizes closer to the truth.

set.seed(20-02-19)

n_studies <- 1000
sample_size <- sample(2:2000, n_studies, replace=T)

studies <- plyr::aaply(sample_size, 1, function(size) {
  dat <- rnorm(size, mean = .01, sd = .1)
  c(effect_size=mean(dat), p_value=t.test(dat)$p.value)
})

studies <- cbind(studies, sample_size=log(sample_size))

include <- studies[, "p_value"] < .05

plot(studies[, "sample_size"], studies[, "effect_size"], 
     xlab = "log(sample size)", ylab="effect size",
     col=ifelse(include, "black", "grey"), pch=20)
lines(lowess(x = studies[, "sample_size"], studies[, "effect_size"]), col="grey", lwd=2)
lines(lowess(x = studies[include, "sample_size"], studies[include, "effect_size"]), col="red", lwd=2)
abline(h=.01)

^{Created on 2019-02-20 by the reprex package (v0.2.1)}