Based on data from a scenario-based experiment, I am running a $2\times2\times2$ ANCOVA with one continuous covariate (sample size 320). Without including the covariate, the ANOVA model and two of the main effects are significant on the 5% level. However my $R^2$ is only around 3%.
After including my covariate, the significance of my main effects and the ANCOVA model increases heavily. Moreover the $R^2$ of the model increases to around 25%.
The assumptions for including the covariate are met. How should/can I interpret such a result? Does it mean that the manipulated variables do only have a minor effect on the dependent variable and explain much less than the covariate? If so, is the experiment actually useless as the manipulated variables barely explain variance of the dependent variable?
Best Answer
When you run the ANOVA you get a unique prediction for each cell. So if you have 320 observations you have about 40 per cell. Since each of them gets the same prediction there is bound to be a lot of residual variability unaccounted for. When you add a covariate which has many values there will be many different predictions which can potentially be closer to the actual values leading to the phenomenon you observed.