I would love to compare 9 multinomial (4 point likert) and 2 binomial (yes/no) measures across two independent groups but I am not sure what test is most appropriate. Treating the group as the independent variable and the likerts as the dependent variables seems to very much lend itself to a MANOVA with post-hocs but is this a valid measure with likert and binomial data?
Solved – Comparing multiple likert items across two groups – is the MANOVA appropriate
group-differenceslikertmanova
Related Solutions
Mixed ANOVA
I think a 2 (between subjects; alcohol group: low, high) by 3 (within subjects; condition; reward, neutral, punishment) mixed ANOVA would probably be a good way to analyse the data.
- The main effect of group would tell you overall whether the groups differ in mean performance averaged across the three conditions
- The main effect of condition would tell you whether overall performance varied across conditions.
- The group by condition interaction would tell you whether the effect of condition varied across groups.
Potential follow up tests
You could potentially perform follow up tests to further decompose the two main effects and interaction effects. You may wish to make performance of follow up tests conditional on the outcome of the initial main effect and interaction significance tests. Here are a few ideas:
- If the interaction effect is significant, perform analysis of simple effects, or interaction contrasts.
- If interaction effect is not significant or if performing analysis of simple effect of condition, and the condition effect is significant, then do some form of contrast or pair wise comparison of means to decompose effect of condition.
Alternatively, if you don't need statistical significance tests for all the little pairwise comparisons, you could just present a table or graph of the means (and some measure of error or variation) and comment on what you think the significance tests for the initial main and interaction tests mean. This is less rigorous, but sometimes sufficient.
Comments on the MANOVA Idea
I'm not a fan of the idea of running a MANOVA with group as IV and the three conditions as DVs. You wouldn't be testing the effect of condition or the interaction effect. If you truly don't care about such things, then the MANOVA would be a reasonable option; or you could simply create a composite variable (i.e., mean of the three conditions) and do an independent groups t-test on the composite.
First, we'll need to know whether you are interested in the response to each Likert question or to a sum of Likert questions; if the latter, it matters how many questions and what the distribution of the scale looks like.
Either way, you will have to account for the nonindependence of the data, because the same people are answering the questions multiple times. Repeated measures ANOVA is one solution to this, but it makes unrealistic assumptions including sphericity, and would only be usable for the scale score, and only if the scores ranged fairly widely so that you could pretend they were continuous.
A better option is a mixed model. If you treat the scores as continuous data, then this would be a linear mixed model; if you treat them as ordinal (as you would have to do if you were interested in each question) then you would need a nonlinear mixed model.
Unfortunately, these models are not simple to implement. If you currently know only about t-tests, then you may need to hire a consultant to help.
Best Answer
Strictly, the means and other summary statistics obtained from binary or ordinal data do not have normally distributed errors and are not homoscedastic. Strictly, ANOVA is not an appropriate tool to analyze such data.
There is a controversy in the literature on whether the violations of the assumptions of the ANOVA matter in practice and whether ANOVA should be used to analyze data from Likert scales. Some authors (Vigderhous, 1977; Knapp, 1990; Kuzon, Urbanchek & McCabe, 1996; Jakobsson, 2004; Jamieson, 2004) claim that non-parametric methods should be used instead of ANOVA. Other researchers (Pell, 2005; Carifio & Perla, 2008; Norman, 2010) argue that ANOVA is robust against the violations and since the choice of non-parametric methods results in a loss of power, ANOVA is preferable.
As O'Connell (2006) points out in the introduction to her book, there are methods that allow to map Likert scale data to an (latent) interval scale without throwing away the information provided by the data. In particular, I recommend the Ordered categorical regression as described by Gelman & Hill (2006) in chapters 6.5 and 15.2 of their book.
Literature
Carifio, J., & Perla, R. (2008). Resolving the 50‐year debate around using and misusing Likert scales. Medical education, 42(12), 1150-1152.
Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge university press.
Jakobsson, U. (2004). Statistical presentation and analysis of ordinal data in nursing research. Scandinavian Journal of Caring Sciences, 18(4), 437-440.
Jamieson, S. (2004). Likert scales: how to (ab) use them. Medical education, 38(12), 1217-1218.
Knapp, T. R. (1990). Treating ordinal scales as interval scales: an attempt to resolve the controversy. Nursing research, 39(2), 121-123.
Kuzon Jr, W. M., Urbanchek, M. G., & McCabe, S. (1996). The seven deadly sins of statistical analysis. Annals of plastic surgery, 37(3), 265-272.
Norman, G. (2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in health sciences education, 15(5), 625-632.
O'Connell, A. A. (2006). Logistic regression models for ordinal response variables (No. 146). Sage.
Pell, G. (2005). Use and misuse of Likert scales. Medical Education, 39(9), 970-970.
Vigderhous, G. (1977). The level of measurement and “permissible” statistical analysis in social research. Pacific Sociological Review, 20(1), 61-72.