Statistical Significance – Analyzing Changes Over Time on a 5-Point Likert Item

likertstatistical significance

Context:

I have two data sets from the same questionnaire run over two years. Each question is measured using a 5-Likert scale.

Q1: Coding scheme

At the moment, I have coded my responses on a [0, 1] interval, with 0 meaning "most negative response", 1 meaning "most positive response", and other responses spaced evenly between.

  • What is the "best" coding scheme to use for the Likert scale?

I realise that this might be a bit subjective.

Q2: Significance across years

  • What is the best way to determine whether there is statistically significant change across the two years?

That is, looking at the results for question 1 for each year, how do I tell if the difference between the 2011 result and the 2010 result is statistically significant? I've got a vague recollection of the Student's t-test being of use here, but I'm not sure.

Best Answer

1. Coding scheme

In terms of assessing statistical significance using a t-test, it is the relative distances between the scale points that matters. Thus, (0, 0.25, 0.5, 0.75, 1) is equivalent to (1, 2, 3, 4, 5). From my experience an equal distance coding scheme, such as those mentioned previously are the most common, and seem reasonable for Likert items. If you explore optimal scaling, you might be able to derive an alternative coding scheme.

2. Statistical test

The question of how to assess group differences on a Likert item has already been answered here.

The first issue is whether you can link observations across the two time points. It sounds like you had a different sample. This leads to a few options:

  • Independent groups t-test: this is a simple option; it also does test for differences in group means; purists will argue that the p-value may be not entirely accurate; however, depending on your purposes, it may be adequate.
  • Bootstrapped test of differences in group means: If you still want to test differences between group means but are uncomfortable with the discrete nature of dependent variable, then you could use a bootstrap to generate confidence intervals from which you could draw inferences about changes in group means.
  • Mann-Whitney U test (among other non-parametric tests): Such a test does not assume normality, but it is also testing a different hypothesis.
Related Question