I am trying to analyze data based on a sample size of around 50 and would appreciate advice on which statistical test would be most appropriate. Participants, on a single questionnaire, were asked to complete two different sections. Variable 1: rating Likert items on a scale from 1-5 (strongly agree to disagree). Variable 2: rating Likert items of a validated tool on a scale of 1-9, which is averaged to obtain a single number.
My research question is: Do those who 'agree' with the statement in the first section significantly differ in terms of their scores on the variable 2, compared to those who 'disagree'? (ex. Do participants tend to 'agree' with "I like to read books for pleasure." have higher IQs (variable 2) than those who 'disagree'?)
I was advised by a colleague to run a one-way ANOVA with variable 1 (categorical) as IV and variable 2 (continuous) as DV.
Question 1:
My independent variable is the three groups: 'agree', 'neutral', and 'disagree.' (I combined strongly agree with agree because of low frequencies and likewise for strongly disagree and disagree.) However, there are many more participants in the 'agree' category than in the 'disagree' category. The distribution is, on average, around 44 agrees, 3 neutrals, and 3 disagrees. Do the (very) unequal sample sizes prevent me from performing ANOVA, or any other tests? Which test would be most appropriate for this analysis?
Question 2:
I would like to run analyses on all of the Likert items, which is more than 10 statements. (ex. Seeing how IQs correlate with agreement to many statements including “I like to read books for pleasure,” “I watch a lot of TV,” “I exercise more than 30 minutes each week,” etc.) I was planning to do a Bonferroni correction to adjust for the multiple tests. Would this be correct, and help make my analyses more robust?
Note: The continuous data are not normally distributed (there is negative skewness), but I seem to be able to correct the skewness using logarithmic transformation.
Any advice or suggestions for papers to read would be appreciated. Thank you!
Best Answer
A way to improve upon ANOVA with an ordinal predictor is to use dummy codes in penalized regression. Penalized regression takes advantage of the ordering among the response categories in Likert scale data: it reduces overfitting by smoothing differences in slope coefficients for dummy variables corresponding to adjacent ranks. See Gertheiss and Tutz (2009) for an overview of penalized regression for ordinal predictors. I've discussed penalized regression here a few times before:
Unfortunately, this approach probably won't do well for categories with very few observations, and I don't know of any that would. Inferential power is necessarily limited with samples so unbalanced as to have very few observations in groups of interest. Correcting for familywise error would raise the bar even further out of reach, even if that's where it belongs. Whether it is depends on whether you mean to test one big hypothesis several times on separate but related measures, or whether you want to evaluate each hypothesis test separately; familywise error adjustment isn't necessary for the latter.
If you can't collect more data, might as well give the test a shot, but give some thought to the degree of evidence you want to see. You probably won't have enough power to distinguish small differences from the null hypothesis with p < .05, so using the Neyman–Pearson framework for dichotomizing p values interpretively is probably unrealistic (more so than usual, that is). There are less polarized ways of understanding p values – one might also call them more equivocal ways, but that's probably more appropriate with relatively weak evidence anyway. For more on interpreting p values, see for instance:
The recommendation to focus on effect size estimation and confidence intervals may help here too, because it is in essence a recommendation to focus on what you can know from your data, even if it's not "enough to reject" a null hypothesis. Plotting your results may help give you a sense of what's really going on too. Don't feel that confirmatory hypothesis testing is your only option unless you have good reason to; you may be able to get some good ideas of hypotheses to test further by just exploring your data, even if you can't really conclude anything very firmly from what you have.
One last option to consider is treating your Likert scale data as continuous. This is a big assumption (that you're effectively already making with regard to your Variable 2), so keep it in mind when interpreting anything you do based on that...but it would allow you to compute correlations between each item's ratings and your Variable 2. In this case especially, you'd not want to collapse the
dis/agreeandstrongly dis/agreecategories. Also bear in mind that a t-test of a correlation assumes bivariate normality, so you might want to consider alternatives for any hypothesis tests on those effect size estimates as well.Reference
Gertheiss, J., & Tutz, G. (2009). Penalized regression with ordinal predictors. International Statistical Review, 77(3), 345–365. Retrieved from http://epub.ub.uni-muenchen.de/2100/1/tr015.pdf.