Solved – Choice of box size and weights for a forest plot from random effects meta-analysis

forest-plotmeta-analysis

I am doing a random-effects meta-analysis where the variable is measured in standardized mean differences. The actual measures vary, but are related to drug use in rats and control vs. treatment. I used SMD because the measures vary in scale.

For the forest plot, I am considering how to scale the size of the boxes for each study and for the overall effect.

There doesn't seem to be a standard method – some use sample size (bigger samples = wider boxes), some use inverse of the variance (smaller variance, wider boxes), some use the Der Simonian Laird estimates (which modify inverse variance by adding a constant) and some use "quality" of a study (with quality being defined in various ways).

Is there a current best practice?

What are the pros and cons of each of the above (or any others)?

Best Answer

How to scale the squares in a forest plot?

I'd argue for scaling the size (area) of the squares proportional to the weight that the study contributed to the meta-analysis. By scaling by weight, the area of the square is a direct visual cue of the relative impact a study had on the summary effect. The weight is among other things proportional to the standard error (precision) which, in turn, is usually (but not always!) directly related to the study sample size. This makes the most sense to me because the forest plot is a visual display of a statistical analysis which, in effect, is a weighted mean. Scaling by quality seems problematic to me because quality is difficult to measure objectively and the summary effect is not calculated using "quality-weights" (at least I've never seen it).

This seems to be supported by a number of authors. Steff Lewis and Mike Clarke$^{[1]}$ go into the history of the forest plot and write

The area of each square was proportional to the weight that the individual study contributed to the meta­ analysis.

Michael Borenstein et al.$^{[2]}$ recommend the same when they explain

[...] the point is a box, proportional (in area) to that study’s weight in the analysis.

This is again mirrored in Jonathan Sterne's book$^{[3]}$:

The size of the plotting symbol for the point estimate in each study is proportional to the weight that each trial contributes in the meta-analysis.

Lastly, in The Handbook of Research Synthesis and Meta-Analysis$^{[4]}$ we read

In this plot, each study is represented by a square, bounded on either side by a confidence interval. The location of each square on the horizontal axis represents the effect size for that study. The confidence interval represents the precision with which the effect size has been estimated, and the size of each square is proportional to the weight that will be assigned to the study when computing the combined effect.

A disadvantage of this approach becomes obvious when you want to present the results of a fixed-effects and a random-effects meta-analysis (or just two different analysis methods) in the same forest plot. Different analysis methods likely assign different weights to the studies and so the scaling of the area of the squares becomes ambiguous.

Estimators of between-study variance: Which one to use?

The metafor package for R offers no less than nine different estimators for the amount of heterogeneity:

  • DerSimonian-Laird
  • Hedges
  • Hunter-Schmidt
  • Sidik-Jonkman
  • Maximum-likelihood
  • Restricted maximum-likelihood (REML)
  • Empirical Bayes
  • Paule-Mandel
  • Generalized Q-statistic

Descriptions of these estimators can be found in references $[4, 5, 6, 7, 8]$. The question remains which one to use? Veroniki et al.$^{[7]}$ and Langan et al.$^{[8]}$ recommend the Paule-Mandel estimator or restricted maximum likelihood based on simulation studies. A never publication by Langan et al.$^{[9]}$ made the following recommendations:

See also this question.

References

$[1]$ Lewis Steff, Clarke Mike. Forest plots: trying to see the wood and the trees BMJ. 2001. 322:1479 [link]

$[2]$ Michael Borenstein, Larry V. Hedges, Julian P.T. Higgins, Hannah R. Rothstein. Introduction to Meta-Analysis. Wiley 2009.

$[3]$ Jonathan Sterne. Meta-Analysis: An Updated Collection from the Stata Journal. Stata Press 2009.

$[4]$ Harris Cooper, Larry V. Hedges, Jeffrey C. Valentine (ed). The Handbook of Research Synthesis and Meta-Analysis. 2nd ed. Russell Sage Foundation 2009.

$[5]$ Rebecca DerSimonian, Raghu Kacker. Random-effects model for meta-analysis of clinical trials: An update. Contemp Clin Trials 28. 2007. 105-114. [link]

$[6]$ Wolfgang Viechtbauer, José Antonio López-López. A Comparison of Procedures to Test for Moderators in Mixed-Effects Meta-Regression Models. Psychological Methods 20(3). 2015. 360-374. [link]

$[7]$ Areti Angeliki Veroniki et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Syn Meth 7. 2016. 55-79. [link]

$[8]$ Dean Langan, Julian PT Higgins, Mark Simmonds. Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies. Res Syn Meth 8. 2017. 181-198. [link]

$[9]$ Dean Langan et al. A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Res Syn Meth 10. 2019. 83-98. [link]