Solved – Metafor package: Interpreting rma model with two (or more) moderators.

interactionmeta-analysismeta-regression

New user of both stackexchange and R here. My questions are about mainly about interpretation, and my dataset is quite large, so I have not included my dataframe. I have two queries (in bold).

Using the "metafor" package, I have a random effects model and I am trying to interpret a model with two categorical moderators (one with 2 levels, one with 5). I am not interested in interactions as there are empty cells preventing this.

I think the model should take this form: (note I have used ML so that I can use the anova function for subsequent model comparisons)

MODEL=rma(yi=Ds,sei=SE,data=VPpooled,mods=~factor(SOA2)+factor(Comparison),method="ML")

This yields the following output:

R^2 (amount of heterogeneity accounted for):            88.89%
Test of Moderators (coefficient(s) 2,3,4,5,6): 
QM(df = 5) = 41.5835, p-val < .0001

Model Results:

                              estimate      se     zval    pval    ci.lb    ci.ub    
intrcpt                             0.1424  0.0438   3.2517  0.0011   0.0566   0.2283  **
factor(SOA2)2                       0.1049  0.0380   2.7619  0.0057   0.0305   0.1794  **
factor(Comparison)Anger            -0.1413  0.0458  -3.0818  0.0021  -0.2311  -0.0514  **
factor(Comparison)Negative words   -0.0307  0.0529  -0.5800  0.5619  -0.1344   0.0730    
factor(Comparison)Disgust          -0.1546  0.0601  -2.5715  0.0101  -0.2724  -0.0368   *
factor(Comparison)IAPS             -0.1103  0.0594  -1.8584  0.0631  -0.2267   0.0060   .

My interpretation of this is that including these two moderators in the model accounts for 89% of the heterogeneity in effects. However, it is unclear to me what the QM test means here. In the context of one moderator, I understand it would indicate that there is a difference between levels of that factor, however, in the context of two moderators I don't understand what differences it is testing. First, would it be possible to provide me with an explanation of what this test means in the context of a two factor model?

Secondly, if my understanding of the output is correct, the intercept represents the first level of factor(Comparison) at the first level of factor(SOA2), which is why these parameters are not explicitly listed in the output.

Therefore, to test whether the factor(SOA) is significant (while controlling for the influence of factor(Comparison) I would need to add a btt argument to the model, where 2 indicates the second level of the SOA factor:

MODEL=rma(yi=Ds,sei=SE,data=VPpooled,mods=~factor(SOA2)+factor(Comparison),method="ML",btt=2)

Yielding:

Test of Moderators (coefficient(s) 2): 
QM(df = 1) = 7.6282, p-val = 0.0057

Indicating that the SOA2 factor is nominally significant controlling for the other factor. Equivalently, adding btt=c(3,4,5,6) would provide the same test of the Comparison factor. Is this the appropriate way of conducting/ interpreting this test?

Please let me know if anything is unclear, or if I have made any factual errors. I appreciate your time.

Best Answer

Regarding the first query/test:

Test of Moderators (coefficient(s) 2,3,4,5,6): 
QM(df = 5) = 41.5835, p-val < .0001

This is simultaneously testing the null hypothesis that coefficients 2 through 6 (i.e., all of the model terms, except the intercept) are simultaneously equal to zero. In essence, this is testing whether these two factors account for any of the heterogeneity in these data.

I recently added an example to the metafor package website regarding the interpretation and use of factors in meta-regression models (see here). It doesn't cover the situation with two factors, but it should still help you to understand better what this omnibus test (wikipedia link) is doing.

Regarding the second query/test:

This is all exactly right.

Note that the SOA factor only has two levels, so there is only one dummy variable needed to represent that factor in the model. So, testing that factor (as you have done) is equivalent to testing just that one dummy variable. Accordingly, the p-value for factor(SOA2)2 (p = 0.0057) is identical to the p-value for:

Test of Moderators (coefficient(s) 2): 
QM(df = 1) = 7.6282, p-val = 0.0057

In fact, the square of the z-value for that dummy is identical to the value above:

> 2.7619^2
[1] 7.628092

The one is just a z-test, the other a chi-square test. But they are testing exactly the same thing, so the p-values are identical.

For a factor with more than two levels (like Comparison), you really need to test all of the corresponding dummy variables simultaneously (coefficient 3 through 6). Note that you don't need to keep refitting the model with different btt values; you can just use:

anova(MODEL, btt=2)
anova(MODEL, btt=3:6)

This will test these different (sets of) coefficients (again, the first test isn't really necessary, but if you want to report the chi-square statistics for both, then this is useful). And take a look at the example on the metafor package website, which also that covers this in some detail.

Related Question