I'm having an argument with a colleague (neither of us have a strong statistical / research methods background) on whether we should randomize questionnaire layout.
The survey in question is a feedback forms on teacher performance which are distributed for high school pupils. So one pupil gets up to 20 surveys (on each teacher he has contact with) which have identical, multiple-choice questions. Most of the questions also have identical scale. The pupil fills the surveys in online and does not necessarily need to do everything at once (but he definitely can). We do not have control how / when they choose to do it.
My colleague argues that by randomizing the order of questions we reduce the possibility that pupils will fill in the questionnaires without thinking too much – e.g. if one fills 5 forms at once, the evaluations of teachers will be very similar.
In my opinion, though, we would actually gain from making sure that the survey questions are grouped together based on the area of question (e.g. teacher communication skills, usage of IT, etc) – this would ensure that pupils provide consistent answers.
I guess there's a point in both approaches – but maybe there are some best practices how to deal with that?
Oh, and if somebody could dig up some references, that would be ultra-cool. My colleague likes facts 🙂
Best Answer
The argument for randomising the order of the questions is due to the fact that the answers to later questions can be biased by the presentation of earlier questions. Randomising the question order means that the influence is no longer subject to this ordering bias.
With repeated administration of the same survey, the ordering effect is only removed for the first time the person completes the questionnaire. I would randomise for the first survey, and then not worry about it for the following surveys.
This concern has been around for decades, I imagine the main paper came out in the 1950s or 1960s. But here are some references to look at:
Any good questionnaire design book will cover this off.