This is for anyone who is interested, after reading pp. 37-43 in Flexible Imputation of Missing Data by Stef van Buuren. If we call the adjusted degrees of freedom nu
m <- nrow(mat)
lambda <- (betweenVar + (betweenVar/m))/totVar
n <- nrow(nhimp$data)
k <- length(coef(lm(chl~bmi,data = complete(nhimp,1))))
nu_old <- (m-1)/lambda^2
nu_com <- n-k
nu_obs <- (nu_com+1)/(nu_com+3)*nu_com*(1-lambda)
(nu_BR <- (nu_old*nu_obs)/(nu_old+nu_obs))
# [1] 15.68225
nu_BR, the Barnard_Rubin adjusted degrees of freedom, matches up with the degrees of freedom for the bmi variable yielded from the the summary(pool(fit)) call above: 15.68225. So we can pass this value into degrees of freedom argument in the pt() function in order to obtain the two-tailed p-value for the imputed model.
pt(q = pooledMean / pooledSE, df = nu_BR, lower.tail = FALSE) * 2
# [1] 0.2126945
And this manually calculated p-value now matches the p-value from the mice function output.
This is explained in Stuart (2008) and in the cobalt vignette. The problem is that when comparing balance before and after matching, the SMD will be affected not only by changes in balance but also by changes in the standard deviation of the covariate when the standard deviation of the matched sample is used as the standardization factor after matching. This muddles two things together when we only care about one. Holding the standard deviation constant prevents this, isolating the effect of matching on balance alone.
Consider the following example. Let's say the mean of a covariate X (e.g., age) in the treated group is 44 and the mean in the control group is 46, and the pooled standard deviation is 9. Let's say that after matching, the control group mean is now 45 and the pooled standard deviation is now 4. Was there better balance before matching or after matching?
It should be clear that the covariate means are closer together, which indicates an improvement in balance and therefore a reduction in bias. Which method of computing the SMD reflects this?
Prior to matching, the SMD is (46-44)/9 = .22. By the standard criterion of SMDs less than .1, this would be considered imbalanced.
Using the formula for the SMD that uses the standard deviation in the unmatched sample, the matched SMD is (45-44)/9 = .11, indicating better balance.
Using the formula for the SMD that uses the standard deviation in the matched sample, the matched SMD is (45-44)/4 = .25, indicating that balance got worse after matching!
Remember that the bias of the effect estimate is a function of the mean differences, and standardizing them to produce the SMD is just a way to simplify balance statistics for users. It's all arbitrary anyway, but at least using the unmatched standard deviation correctly isolates changes in balance from changes in variability, the latter of which is not related to bias.
Best Answer
The difference is that the "normalized difference" in the cited article is an error and the standardized mean difference as described in all other articles on assessing balance and the
cobaltdocumentation is correct. The sum of two variances is not a valid way to represent the variability of the distribution of the covariates. This is likely a typo or a misunderstanding by the authors.The authors say (p. 24)
Take a look at the normalized difference in Imbens and Rubin (2015, Ch 15.2, p. 339):
This definition of the normalized difference is correct and consistent with that used in
cobalt.