Solved – Alternative funnel plot, without using standard error (SE)

funnel-plotmeta-analysispublication-biassample-sizestandard error

Before submission of my meta-analysis I want to make a funnel plot to test for heterogeneity and publication bias. I have the pooled effect size and the effect sizes from each study, that take values from -1 to +1. I have the sample sizes n1, n2 for patients and controls from each study. As I cannot calculate the standard error (SE), I cannot perform Egger's regression. I cannot use SE or precision=1/SE on the vertical axis.

Questions

Can I still make a funnel plot with effect size on the horizontal axon and total sample size n (n=n1+n2) on the vertical axis?
How should such a funnel plot be interpreted?

Some published papers presented such funnel plot with total sample size on the vertical axis (Pubmed PMIDs: 10990474, 10456970). Also, wikipedia funnel plot wiki agree on this. But, most importantly, Mathhias Egger's paper on BMJ 1999 (PubMed PMID: 9451274) shows such a funnel plot, with no SE but only sample size on the vertical axis.

More Questions

Is such a plot acceptable when the standard error is not known?
Is it the same as the classical funnel plot with SE or presicion=1/SE on the vertical axon?
Is its interpretation different?
How should I set the lines to make the equilateral triangle?

Best Answer

Q: Can I still make a funnel plot with effect size on the horizontal axon and total sample size n (n=n1+n2) on the vertical axis?
A: Yes

Q: How should such a funnel plot be interpreted?
A: It is still a funnel plot. However, funnel plots should be interpreted with caution. For example, if you have only 5-10 effect sizes, a funnel plot is useless. Furthermore, although funnel plots are a helpful visualization technique, their interpretation can be misleading. The presence of an asymmetry does not proof the existence of publication bias. Egger et al. (1997: 632f.) mention a number of reasons that can result in funnel plot asymmetries, e.g. true heterogeneity, data irregularities like methodologically poorly designed small studies or fraud. So, funnel plots can be helpful in identifying possible publication bias, however, they should always be combined with a statistical test.

Q: Is such a plot acceptable when the standard error is not known?
A: Yes

Q: Is it the same as the classical funnel plot with SE or presicion=1/SE on the vertical axon?
A: No, the shape of the 'funnel' can be different.

Q: Is its interpretation different?
A: Yes, see above

Q: How should I set the lines to make the equilateral triangle?
A: What do you mean by "lines to make the equilateral triangle"? Do you mean the 95%-CI lines? You will need the standard errors...

You also might be interested in:

Peters, Jaime L., Alex J. Sutton, David R. Jones, Keith R. Abrams, and Lesly Rushton. 2006. Comparison of two methods to detect publication bias in meta-analysis. Journal of the American Medical Association 295, no. 6: 676--80. (see "An Alternative to Egger’s Regression Test")

They propose a statistical test which focuses on sample size instead of standard errors.

By the way, do you know the book "Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments"? It will answer a lot of your questions.

Related Solutions

Data Visualization – Calculating Standard Error for Funnel Plots

Generally a variance for a mean of $n$ iid $X_i$s is ${\rm var}(X_i)/n$. Now since the $X_i$s are Bernoulli with success probability $p$, ${\rm var}(X_i)$ has a special form. It is $p(1-p)$. So each of the $23$ means have variance of the special form $p(1-p)/n$ and the standard error is the square root of that.

Solved – Why does finding small effects in large studies indicate publication bias

The answers here are good, +1 to all. I just wanted to show how this effect might look in funnel plot terms in an extreme case. Below I simulate a small effect as $N(.01, .1)$ and draw samples between 2 and 2000 observations in size.

The grey points in the plot would not be published under a strict $p < .05$ regime. The grey line is a regression of effect size on sample size including the "bad p-value" studies, while the red one excludes these. The black line shows the true effect.

As you can see, under publication bias there is a strong tendency for small studies to overestimate effect sizes and for the larger ones to report effect sizes closer to the truth.

set.seed(20-02-19)

n_studies <- 1000
sample_size <- sample(2:2000, n_studies, replace=T)

studies <- plyr::aaply(sample_size, 1, function(size) {
  dat <- rnorm(size, mean = .01, sd = .1)
  c(effect_size=mean(dat), p_value=t.test(dat)$p.value)
})

studies <- cbind(studies, sample_size=log(sample_size))

include <- studies[, "p_value"] < .05

plot(studies[, "sample_size"], studies[, "effect_size"], 
     xlab = "log(sample size)", ylab="effect size",
     col=ifelse(include, "black", "grey"), pch=20)
lines(lowess(x = studies[, "sample_size"], studies[, "effect_size"]), col="grey", lwd=2)
lines(lowess(x = studies[include, "sample_size"], studies[include, "effect_size"]), col="red", lwd=2)
abline(h=.01)

^{Created on 2019-02-20 by the reprex package (v0.2.1)}