I am writing a manuscript using an experimental design which predicts interactions between 1 continuous variable and multiple dichotomous variables, all predicting a continuous variable. As is traditional in experimental design, I have used ANCOVAs to analyze the data, but am concerned about the inability to specifically test the interaction between the covariate and the dichotomous variables.
Also, while I am expecting only a small amount of variance to be explained, there is only one significant finding among the multiple predictors and interactions. I suspect the non-significant findings may be due to the small amount of explained variance in the DV, which when spread across multiple predictors in the model is insufficient to distinguish between them.
A colleague has suggested that the best option to deal with both concerns is to do four 2 or 3 stage hierarchical regressions with dummy variables, i.e. each regression would comprise:
stage 1: one dummy variable
stage 2: add the continuous variable and the interaction between the two.
After running these analyses, any significant predictors and interactions could be combined into a single hierarchical regression.
I am unable to find a precedent for this unusual procedure but interestingly, it reveals a number of significant results, albeit explaining a very small amount of variance.
Is this a reasonable procedure to follow, given the requirements of my variables?
Best Answer
ANCOVA and multiple regression are mathematically identical. In matrix algebra terms, both are $Y + XB + e$. If you don't have much variance explained in ANCOVA, you won't in regression.
The main problem I can see with your colleague's approach is that of inflating type I error by running multiple tests.
Statistical significance is, generally, of less importance than many people think it is. From a scientific/substantive point of view, if the effect sizes are small they are usually not interesting (although there are exceptions.... If you could reduce the number of, say, plane crashes by a small amount, airlines would probably be interested).