R – How to Test for Contrasts and Interpret Multiple Meta-Regression in Metafor

meta-analysismeta-regressionmetaforr

Lets say I was meta-analysing studies which have assessed the influence of a warm- up on running performance (completely hypothetical). Lets also say that I wanted to find out whether pre-exercise nutrition, the duration of the warm up, and the type of music had any influence.

Nutrition<- c("Meal", "Shake", "Nothing")
Duration<- c("Long", "Short")
Music<- c("Self-selected", "Pre-determined"

For Nutrition, the reference level is "Meal". For duration, the reference level is "Long". For music, the reference level is "Self-selected". I then build the following mixed effects model using the excellent package metafor and assume that these are additive (for simplicities sake).

rma.mv(yi= effect, V= samp_var, slab= Auth, Data= Final_Dat, mods= ~ Nutrition + Duration + Music, method "REML", random= ~ Study/Outcome)

Having gotten this far, let's say I receive the following output from the model.

intercept= 15.50
Nutrition Shake= -2.620
Nutrition Nothing = -5.387
Duration short = - 2.113
Music Pre-determined= 4.2

Now based on my reading, and previous stats knowledge, I understand that the intercept is the estimated effect of having a long warm-up with self-selected music following a proper meal. I also understand that coefficients 2 and 3 represent the difference between that level of the factor and the intercept. Based on this my questions are as follows:

Q1: I assume for coefficients 2& 3, the levels of the other factor(s) are held constant? So, in this case, coefficient 2 would be the estimated average difference in running performance when a shake is consumed before a long warm up with self-selected music. Is this the correct interpretation?

Q2: What is the interpretation of coefficients 4&5? Are these similar to 2&3 in that the other categories are held constant? That is, would the interpretation of coefficient 4 be "the average difference in effect between a short warm-up performed to self-selected music after a meal" or is this just the average difference between a short and long warm up, irrespective of the level of the other categories?

Q3: How do I calculate comparisons for those not explicitly reported in the output. That is, what if I wanted to know whether there was a difference between consuming nothing when a short warm-up is completed with pre-determined music?

I tried reading: https://www.metafor-project.org/doku.php/tips:multiple_factors_interactions. This has some excellent information and examples, but I was wondering if anyone could help extending this to the case where categorical moderators are >2. It seems the general way to do this would be to perform an anova with the coefficients specified in someway, or through the glht function of the multicomp package. However, I'm just looking for a bit more guidance on how this would actually be implemented.

Best Answer

Q1: I assume for coefficients 2& 3, the levels of the other factor(s) are held constant? So, in this case, coefficient 2 would be the estimated average difference in running performance when a shake is consumed before a long warm up with self-selected music. Is this the correct interpretation?

In short, no. Coefficient 2 is the effect of consuming a shake irrespective of the duration or the type of music. It is assumed to be the same unless you fit an interaction.

I think, at least in part, that also answers Q2 and Q3.

Related Solutions

Solved – Metafor package in R: meta regression and scatter plot

Assuming you are trying to model the relationship between year and the log odds of the outcome of interest using a logistic mixed-effects model, then yes, you used the right model.

You may want to rescale wi a bit. Such as:

wi <- 0.5 + 3.0 * (wi - min(wi))/(max(wi) - min(wi))

Something like this should do:

years <- 1998:2014
preds <- predict(model_A, transf = transf.ilogit, newmods = years)
plot(year, transf.ilogit(dat$yi), cex=wi)
lines(years, preds$pred)
lines(years, preds$ci.lb, lty="dashed")
lines(years, preds$ci.ub, lty="dashed")

Solved – Metafor package: Interpreting meta-regression model

Yes, based on what you have shown, I would say that the analysis is sensible. One concern might be the relatively large number of moderator variables (or more specifically, model coefficients) relative to the number of estimates. Right now, you have $105 / 14 = 7.5$ estimates per coefficient (not counting the intercept). Some might want that ratio to be closer to 10 or even 15, but some might also be okay with a ratio of 5. None of these are right or wrong, but the lower the ratio, the more concerned I would be with overfitting.
Indeed, strictly speaking, the PSS version factor fails to be significant at $\alpha = .05$. However, I think you can still discuss this factor -- cautiously. Based on psychometric theory and all else equal, it is to be expected that longer versions would lead to higher reliability, which is indeed what you find here (although the 14-item version does not seem to yield, on average, higher reliability than the 10-item version -- maybe those 4 extra items are not as internally consistent as the rest or maybe there is something else that is different about studies examining the 14-item version that is not captured by all the other moderator variables already included in the model).
It is common practice to examine one moderator at a time. In principle, this is poor practice, since moderator variables are often correlated. So, fitting a model including multiple moderators (as you have done) would be better, as that gets you closer to examining the contribution of a particular moderator variable while controlling for the rest. One reason why this is often not done is that the dataset typically looks like Swiss cheese, with lots of holes (i.e., missing data) in it. After listwise deletion, one then ends up with a (much) smaller dataset (i.e., only the studies with complete information on all moderator variables). Besides the loss of information itself, when this happens, a major concern here is potential bias due to the missingness. Hence, instead, analyses are often conducted one moderator at a time, so that all of the studies providing information on a particular moderator variable can be used. Bias due to missingness may still be an issue here, but maybe less so. But, as mentioned at the beginning, you are then not controlling for other moderator variables, so a "fake" moderator might appear to be relevant simply because it is correlated with a "true" moderator.

There are fancy techniques to deal with missingness (e.g., multiple imputation, full information maximum likelihood estimation), but these methods are poorly developed in the meta-analytic context. Alternatively, you could run the 'full model' analysis and the 'one at a time' analyses and put them side-by-side and hopefully you find some consistency in the conclusions. If so, the discussion section will be easy to write. If not, then good luck ;)

Best Answer

Related Solutions

Solved – Metafor package in R: meta regression and scatter plot

Solved – Metafor package: Interpreting meta-regression model

Related Question