I have the following analysis: I ran a MANOVA with two DVs (both empathy scales) and an IV (gender). Now the $p$-value (.03) tells me that there is a significant difference between men and women. However, I do not know which is higher on which variable. When I then look at the ANOVA table (given with SPSS) output, it tells me that there is no significant difference (all $p$-values larger than .05). How do I interpret this? Had men and women score differently or not?
Also, my supervisor asked me to look at the means, however they are only different by maybe one point, so I cannot see there which one should be larger (and the MANOVA only tells me the joint effect of the two DVs together).
When I ran a t-test to have a look at the individual means, they were also non-significant.
I am a bit lost what to do with this and would highly appreciate some help in regards how to put this interpretation into words.
Best Answer
This answer does not completely address your question about MANOVA and the distinction between univariate and multivariate tests but note that p-values are not all there is to statistical analysis. Looking at the means/plotting them is in any case a good idea.
Thus, it's perfectly possible for a difference to be significantly different from 0 and still very small in some sense. I don't know if “one point” is a lot on your empathy scales but what the test tells you is merely that it would be unlikely to observe it if the rating from men and women were exactly the same. Even if all the tests you performed would reject the null hypothesis, you would still need something else (plots, descriptive statistics, standardized or unstandardized indices of effect size, confidence intervals around these effect sizes, etc.) to fully understand the magnitude of the effect because it's simply not what the test itself is about.
Furthermore, one important point is that “the difference between significant and non-significant is not itself significant”. The logic of hypothesis testing requires you to select an error level (often 5%) and stick to it but there is nothing magical about this threshold. Informally, p = .03 is not very strong evidence of a difference and p = .06 is not evidence that the difference is exactly 0. This point is particularly important when comparing different studies or different groups but there is nothing really surprising about two tests producing slightly higher or lower p-values even with the same data and no need to read too much into it.