Online anonymous Likert survey was sent containing 13 items, the survey was sent before and after an intervention. I would like to analyze the % respondents who responded agree/strongly agree to the items both pre and post-intervention.
The total $n$ is small and do not match up as pre-intervention $n=14$, post-intervention $n=18$. I have been struggling to find the correct analysis … I've come across McNemar vs. Wilcoxon vs paired t-test.
Best Answer
Because you can’t match the pre answers with the post answers, there’s no way to pair the responses, and so there’s no way to use tests designed for paired responses (such as those you mention: McNemar, paired-sign rank, and paired t-test).
Probably the best approach is to use a test of association designed for independent samples. Because you are measuring the same individuals in the two groups, this may be a violation of the independence assumption of the test. But I don’t know any way to adjust for this. This violation should be noted in the methods or results.
This approach will probably have lower power than if you had recorded the identity of the respondents in order to pair the results.
Because you are condensing the responses of a single Likert item into two categories (SA/A vs. N/D/SD), this becomes a nominal variable with two levels * ** .
This leaves you with a 2 x 2 contingency table. Appropriate analyses are chi-square test of association, Fisher’s exact test, or G-test of association. Please be sure you understand the assumptions and interpretation of the test you choose to use.
I also strongly suggest you report a measure of effect size. For a 2 x 2 table, the most common measure of effect size is phi. This statistic is fairly common, and may be more meaningful to the reader than the p-value in this case.
Because your observations aren't paired, you can include all observations. The unequal sample sizes won't be problematic.
** You can think of it as either an ordinal variable with two categories or a nominal variable with two categories. It won’t matter.
*** I probably recommend against condensing your data in this manner. If you leave your responses as an ordinal variable (SD, D, N, A, SA), appropriate tests might include Cochran-Armitage test, Mann-Whitney test, or Kendall correlation.