Confidence Intervals – Interpreting Non-Significant P-Values but CIs Not Including 0

bootstrapconfidence intervallavaanmediationp-value

I have just created my first mediation model using sem() with the lavaan package in R. I am using a bootstrapping with 5000 resamples and BCA to calculate the confidence intervals at a 0.9 level.

However, now I face difficulties in interpreting my results, as I get non-significant p-values (>.1) for the indirect effects, while the confidence intervals do not include 0. How can or should I interpret such results? Are the CI sufficient to talk about a indirect effect?

I have already researched the web and also this forum but haven't found anything that helps me with the given situation. The only thing I found was this post: https://stats.stackexchange.com/a/302949/353584

My Code:

mediation_model <- '
#Regressions Meditation BC -> Aff -> Exp
v_13 ~ a1 * BC 
v_36 ~ a2 * v_13 + b1 * BC + ExpMan
#Regressions Meditation Exp -> Disco -> Rating
v_3 ~ a3 * v_36 + e1 * BC + ExpMan + v_30 + PerfMan + e2 * v_13
v_6 ~ a4 * v_36 + a5 * v_3 + e3 * BC + ExpMan + v_30 + PerfMan + e4 * v_13 + a6 * v_5
#Regressions Mediation Exp -> Disco -> Sat
v_5 ~ b2 * v_36 +  a7 * v_3 + e5 * BC + ExpMan + v_30 + PerfMan + e6 * v_13 
ACME1 := a1 * a2
ACME2 := a3 * a5
ACME3 := a3 * a7
ACME4 := b2 * a6
indESR := b2 * a6
indEDSR := a3 * a7 * a6
BCind1 := a1*a2*a3*a7*a6
BCind2 := a1*a2*b2*a6
BCind3 := a1*a2*a4
BCind4 := a1*a2*a3*a5
BCind5 := b1*a3*a7*a6
'
fit <- sem(model = mediation_model, data = data_all, se = "boot", bootstrap = 5000)

summary(fit, fit.measures = T, modindices = F)
parameterEstimates(fit, boot.ci.type = "bca.simple", level = 0.9)

Results look like:

       lhs op            rhs   label    est    se      z pvalue ci.lower ci.upper
41   ACME1 :=          a1*a2   ACME1  0.092 0.058  1.578  0.115    0.004    0.198
42   ACME2 :=          a3*a5   ACME2 -0.135 0.043 -3.168  0.002   -0.225   -0.079
43   ACME3 :=          a3*a7   ACME3 -0.311 0.092 -3.378  0.001   -0.480   -0.175
44   ACME4 :=          b2*a6   ACME4  0.069 0.043  1.588  0.112    0.008    0.148
47  BCind1 := a1*a2*a3*a7*a6  BCind1 -0.008 0.007 -1.283  0.199   -0.024   -0.001

Best Answer

You should only expect confidence intervals and p-values to align in tests of significance when the same method is used to compute both. In this case, the p-value is computed using a Z-statistic which is computed as the estimate divided by the bootstrap standard error. The confidence interval doesn't use the standard error at all; it is an adjustment to the percentile bootstrap.

I would not trust the p-values because they are computed assuming the Z-statistic comes from a standard normal distribution, which for most mediation-related quantities is not true in finite samples. The bootstrap confidence intervals are the most accurate because they make few assumptions about the sampling distribution of the quantity of interest. I would not even report p-values in your paper and rely instead on the bootstrap confidence intervals. See for example Hayes and Scharkow (2013), who recommend this method (though the Z-test reported by lavaan is not one of the methods compared).

Related Question