Solved – Analysing pre-and-post intervention study with anonymous responses

hypothesis testinglikertmodelingpaired-datat-test

The study I am analysing is a pre-and-post intervention questionnaire of students' views before and after studying a module. The questionnaire was distributed in three different geographical locations, where the same instructor was delivering the same module at different times within one year. The scale used is a 5-point Likert scale (which I can also convert into a binary agree/disagree variable if needed).

The surveys are anonymous, so I can't match the sample. However, in some samples, all the students answered both the pre-and-post questionnaires (same $N$) and in others this was different (different $N$).

Overall I have 343 respondents for the pre-intervention questionnaire, and 378 respondents for the post-intervention questionnaire. However in two of the three locations, I have equal pre and post intervention responses, but they are anonymous: so they are not independent, but not matched. (This is a weakness of the data, and will be reported in the results).

My questions are:
a) Given the sample size, would running an unpaired t-test on the full sample be appropriate? Or should I be going for a non-parametric test?
b) Can the subsamples where there are equal respondents (i.e. all students answered both pre-and-post questionnaires) be analysed as non-independent non-matched samples?

Best Answer

a) Given the sample size, would running an unpaired t-test on the full sample be appropriate? Or should I be going for a non-parametric test?

The sample size is not typically considered as "small" but the concern that the variable of interest is a 5-point Likert scale remains. If an overwhelming amount of responses center on 1 of 5, the residual may not be normal, biasing the significant test. Not knowing the actual spread of the data, I'd say non-parametric test may be more appropriate.

b) Can the subsamples where there are equal respondents (i.e. all students answered both pre-and-post questionnaires) be analysed as non-independent non-matched samples?

If you have collect an array of demographics you may consider some post hoc matching, but otherwise to my knowledge there is no analysis that can adjust for the feature of pair-samples without any identifier.

I'd recommend just analyze them as if they are independent sample. This approach has one drawback: it pools the within person variations into the overall variation, usually biasing up the p-val. But if your independent tests already show a significant difference, paired tests should find the same result.

A benefit is that you can use that for all sites, the numbers do not need to be equal before and after the intervention. The fact that they have the same numbers do not guarantee they are the same people in both days, anyway. (Unless you know the attendance on both days were 100% or you know who were absent and their name.)

Related Question