I'm doing a study in which I have to analyze group differences (MS patients vs. healthy controls) on three outcomes of neuropsychological tests "while accounting for differences that might be attributable to depression and fatigue". I've got a measure for both depression and fatigue, which both vary significantly between my two groups (as expected). As a consequence, ANCOVA should be wrong as we're talking about natural (not random) differences between groups (see Miller and Chapman (2001): Misunderstanding analysis of covariance for reference).
I've read a lot about this problem and it's difficult to decide what to do. I would prefer acknowledging the differences in depression and fatigue as characteristic to the MS population, and not use them as a covariate (otherwise I'm trying to find differences between a depression- and fatigue-less MS population which deviates from reality) but other papers are saying stuff like this:
"tests to evaluate depression and fatigue must be performed, since those symptoms have a recognized impact in cognitive abilities (Kinsinger et al., 2010)"
"Depression or fatigue must be discriminated from cognitive dysfunctions. Up to 90% of MS patients suffer from fatigue, a subjective lack of energy, which can reduce cognitive performance; on the other hand, cognitive deficits can produce exhaustion (Engel et al., 2007). Fatigue might affect performance over time in tasks that require sustained mental effort, specially in cognitive tasks of working memory and visual vigilance (Krupp and Elkins, 2000)."
As literature strongly suggests to use depression and fatigue as a covariate, I'm willing to do so. But then, the problem is that there is no clear alternative for ANCOVA when there are significant differences in covariates between groups. And unlike multiple published (!) studies I don't want to wrongly use ANCOVA.
I was thinking that multiple regression (using the covariate as a predictor) could be an alternative but this has exactly the same outcomes as the ANCOVA, making me think that I'm just doing exactly the same.
Could somebody offer some advice?
Best Answer
Regarding ANCOVA in general, indeed it seems that classic ANCOVA is used to remove noise from the analysis that was induced by random differences between groups. Essentially other variables are contributing variation to your treatment effects; ANCOVA can improve power my removing this contaminating variation. Including variables that are related to both the independent and dependent variables is thus a departure from classic ANCOVA.
As you found, ANOVA/ANCOVA and regression are equivalent (at least when contrasts are coded properly). Multiple regression, and ANCOVA when the primary dependent variable is categorical, is routinely used to remove confounding effects from non-randomised data. You cannot randomly give patients MS (let alone MS without the comorbidities of exhaustion), so you, like many of us, are hoping to derive causal models from observational data. Causal in the sense that you want to know the effect of MS as if it were randomly assigned.
The specific issue with your data is that fatigue and exhaustion (your covariates) are not confounders but instead are mediators — they are a consequence of MS. One generally does not include mediators as covariates because they "steal" the association between your treatment (MS) and outcome (performance). By "steal" I mean that they reflect an indirect path by which MS would effect performance.
If you want to know whether MS effects performance, then you would not include them. However, if you want to know whether MS has an effect on performance independent of or over-and-above the effect of MS on fatigue, then it would make sense to include them. What the experts in your field are perhaps describing is that they don't want people reporting a whole bunch of effects of MS that are simply due to differences in fatigue and exhaustion because that is uninteresting. The extreme example would be that it is uninformative to report that MS-patients perform poorly in Olympic lifting compared to controls - what would that tell us about the disease?
What Miller and Chapman describe, though, is pertinent: if MS has such consistent and profound effects on fatigue, can one even ask if there are effects that act outside of fatigue? I would imagine that this is only the case if the performance distributions from controls to MS patients don't even overlap. In which case more comparable controls would be needed.
You may be stuck with the analytical conventions of you field, but looking into mediation analysis might be a good way forward.