Statistical Tests – McNemar’s Test vs. T-test for Measuring Significance in Matched Pre-post Test Results

categorical datarepeated measurest-test

I have a question on selecting the appropriate statistical test as below:

I have a group of people attending a training course, in order to measure whether the training course is effective in improving attendees' knowledge. We did the following:

  1. attendees will complete a pre-test (TRUE or FALSE only) before the training
  2. attendees attending the training
  3. attendees will complete a post-test (TRUE or FALSE only) after training

Both pre and post tests are having exactly the same questions, the test is tagged with barcode so that we are able to trace which pre and post tests are a pair.

By further breaking down the research question, it can be divided into two parts:

1. whether the training is effective in general

2. whether the training can improve answering of a particular question (item)

For question 1, I think paired t-test will be appropriate: we mark all the test, each attendee will have a pre-test score and a post-test score, we take mean of the pre-test and post-test score respectively, and then go for paired t-test.

For question 2, at first I am thinking of doing a McNemar’s test: as the answer for each question in pre-test and post-test can be matched. I planned to do something like this. But then I have received some alternative method which I'm not sure whether it is appropriate or not:

  1. count the number of attendees answered question X correctly in pre-test and divide it with total number of attendees answer question X, so it becomes a percentage
  2. do the same thing in post-test, so we have two percentages here
  3. perform "paired" t-test on it

When I first saw it, I wonder how I can "pair" each question, as the categorical data is already converted into continuous data. Perhaps the proposer accidentally add the word "pair" and he actually means "t-test" only. But I still wonder why discard the information based on pairing.

A few questions here:

  1. for question 1, is using paired t-test appropriate?
  2. for question 2, is McNemar's test appropriate?
  3. for question 2, is "t-test" still appropriate? is there any reason why "t-test" is preferred?

Best Answer

  1. Yes, for research question 1 t-test is appropriate, provided that the difference post-score minus pre-score is about normally distributed; if not, consider using Wilcoxon signed rank test.
  2. Yes, for research question 2 McNemar is the right choice.
  3. No. T-test is unusual to apply for proportions. When they do it for all that, they apply Fisher angular transformation of proportions before doing t-test; this generally is regarded out-of-date approach. Notice also that with those 2 proportions in hands you cannot do a paired test, it will be independent-samples test.
Related Question