I'm working on a study where I am comparing how students did on three different exams. By the end of each exam, as always, students were assigned a final grade based on the percentage of questions they got right (80%, 83%, 73%, etc.) Each exam, however, had a different number of questions.
If I want to compare how the class did overall across the three exams, can I use a Repeated measures ANOVA, as someone (above me) has suggested?
I've read different things, including many saying it's not appropriate to use Repeated measures ANOVA with percentages. But I can't see how using the raw data would work, since, as I stated, each test had a different number of questions. Wouldn't that make it "comparing DIFFERENT variables," whereas a comparison of the percentages would actually make it a comparable measure of the SAME variable?
Or should I ignore what's been suggested, as used a different measurement (I had used separate paired-sample t-tests, but was told I should do Repeated Measures ANOVA instead, as it'd allow me to compare all three samples at once)?
While I've tried running a Repeated measures ANOVA on the raw data, the profile plot, for example, shows a mathematically correct outcome, obviously, but one that does not reflect the outcome (eg On Exam 1, students got, on average, 3.45 of the 7 questions right; on Exam 2, they got 2.60 of the 5 questions right, and on exam 3, they got 2.14 of the 3 questions right. The plot, then, would show 3.45 – 2.60 – 2.14 -> therefore, a continuous downslope. However, if you look at the mean percentage, it's actually 49.3% – 52.0% – 71.3% -> therefore, continuously going up). Again, does that matter?
Advice as always would be welcomed! Thanks!
Best Answer
One way to look at it is that "percentage" is actually exp(log(count) - log(total)). If you do a mixed-effects glm of Class vs. Correct Answers with a poisson distribution, offset for log of Total Questions (since poisson link is log), and random intercepts for "student", this could handle it.