I am conducting a reliability generalization for Perceived Stress Scale. This instrument has 3 version: PSS 4, PSS 10, and PSS 14. Descriptive statistics showed PSS 4 (alpha = .74, 95% CI = .70-.76) have lower alpha coefficient than PSS 10 and PSS 14 (both have identical alpha value; (alpha = .84, 95% CI = .83-.85).
I entered 9 categorical and 3 continuous moderators into the meta-regression with code:
res <- rma(measure="ABT", ai=ai, mi=mi, ni=ni, mods = ~ version + size + translation + student + psychiatry + physical + drug + pregnant + mix + male + age + sd, data=Dataset, digits=3)
res
The result is as below:
Mixed-Effects Model (k = 105; tau^2 estimator: REML)
tau^2 (estimated amount of residual heterogeneity): 0.089 (SE = 0.016)
tau (square root of estimated tau^2 value): 0.298
I^2 (residual heterogeneity / unaccounted variability): 91.35%
H^2 (unaccounted variability / sampling variability): 11.56
R^2 (amount of heterogeneity accounted for): 32.23%
Test for Residual Heterogeneity:
QE(df = 90) = 1416.665, p-val < .001
Test of Moderators (coefficient(s) 2,3,4,5,6,7,8,9,10,11,12,13,14,15):
QM(df = 14) = 56.397, p-val < .001
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt 1.160 0.172 6.743 <.001 0.823 1.497 ***
version[T.PSS 10] 0.267 0.117 2.284 0.022 0.038 0.497 *
version[T.PSS 14] 0.234 0.128 1.833 0.067 -0.016 0.485 .
size[T.201-1000] 0.023 0.075 0.313 0.755 -0.123 0.170
size[T.>1000] 0.261 0.140 1.861 0.063 -0.014 0.537 .
translation[T.Translated] -0.235 0.075 -3.143 0.002 -0.381 -0.088 **
student[T.Student] -0.097 0.097 -0.991 0.322 -0.288 0.094
psychiatry[T.Psychiatric] 0.071 0.178 0.402 0.688 -0.277 0.420
physical[T.Physical problems] 0.068 0.100 0.678 0.498 -0.128 0.263
drug[T.Drug-related] -0.120 0.235 -0.511 0.609 -0.581 0.341
pregnant[T.Pregnant-related] 0.001 0.175 0.004 0.997 -0.342 0.343
mix[T.Mixed sample] -0.053 0.171 -0.307 0.759 -0.388 0.283
male -0.001 0.001 -0.540 0.589 -0.003 0.002
age 0.001 0.003 0.212 0.832 -0.005 0.006
sd 0.071 0.020 3.495 <.001 0.031 0.110 ***
The result showed that the reliability of PSS 10 (b=0.267, p<.05) was statistically different from PSS 4 (reference level). Since this is a 3-level moderator, I further examine the factor with code:
anova(res, btt=2:3)
The result showed:
Test of Moderators (coefficient(s) 2,3):
QM(df = 2) = 5.215, p-val = 0.074
which suggest that the factor as a whole is not a significant predictor.
My question here is:
1. Is the analysis above methodologically sound?
2. How can i interpret this finding? Does it means that PSS version is not a significant predictor of variability in reliability?
3. Or should I perform separate analysis for each moderator? I've lost huge number of cases due to low report rate of male ratio, mean age, and SD (268 alpha estimates were extracted).
Best Answer
Yes, based on what you have shown, I would say that the analysis is sensible. One concern might be the relatively large number of moderator variables (or more specifically, model coefficients) relative to the number of estimates. Right now, you have $105 / 14 = 7.5$ estimates per coefficient (not counting the intercept). Some might want that ratio to be closer to 10 or even 15, but some might also be okay with a ratio of 5. None of these are right or wrong, but the lower the ratio, the more concerned I would be with overfitting.
Indeed, strictly speaking, the PSS version factor fails to be significant at $\alpha = .05$. However, I think you can still discuss this factor -- cautiously. Based on psychometric theory and all else equal, it is to be expected that longer versions would lead to higher reliability, which is indeed what you find here (although the 14-item version does not seem to yield, on average, higher reliability than the 10-item version -- maybe those 4 extra items are not as internally consistent as the rest or maybe there is something else that is different about studies examining the 14-item version that is not captured by all the other moderator variables already included in the model).
It is common practice to examine one moderator at a time. In principle, this is poor practice, since moderator variables are often correlated. So, fitting a model including multiple moderators (as you have done) would be better, as that gets you closer to examining the contribution of a particular moderator variable while controlling for the rest. One reason why this is often not done is that the dataset typically looks like Swiss cheese, with lots of holes (i.e., missing data) in it. After listwise deletion, one then ends up with a (much) smaller dataset (i.e., only the studies with complete information on all moderator variables). Besides the loss of information itself, when this happens, a major concern here is potential bias due to the missingness. Hence, instead, analyses are often conducted one moderator at a time, so that all of the studies providing information on a particular moderator variable can be used. Bias due to missingness may still be an issue here, but maybe less so. But, as mentioned at the beginning, you are then not controlling for other moderator variables, so a "fake" moderator might appear to be relevant simply because it is correlated with a "true" moderator.
There are fancy techniques to deal with missingness (e.g., multiple imputation, full information maximum likelihood estimation), but these methods are poorly developed in the meta-analytic context. Alternatively, you could run the 'full model' analysis and the 'one at a time' analyses and put them side-by-side and hopefully you find some consistency in the conclusions. If so, the discussion section will be easy to write. If not, then good luck ;)