Questionnaires often contain questions which are similar but worded differently. Usually these questions are located in different parts of the questionnaire, intended to weed out participants who don't take the questionnaire seriously, but I've seen examples where related questions are placed together in a block, as well.
What is the purpose of these questions and, more importantly, how are they analysed?
For example, if the purpose of the survey is to find correlations between different variables on a Likert scale, then how can one go about analysing the data?
Assuming that simple linear regression is appropriate, for example, does one simply try out different 'versions' of the regressor (i.e. the different versions of the same question) and see which one results in the smallest p-value (for $H_0$: $\beta_1$ = 0, $H_A$: $\beta_1$ > 0)?
Or is there some other way to go about this?
Edit: To clarify, I am NOT asking how to conduct hypothesis tests on Likert scales or when I can assume them to be interval rather than ordinal, etc., nor am I asking for how to deal with multiple variables. My question was specifically asking about how similar questions on a questionnaire are dealt with, e.g. if Question 1 is Are you satisfied with our services?, Question 2 is Are you happy about our services?, and Question 3 is How good do you think our services are?
Edit 2: In fact, I don't mind if I get a answer about using logistic regression on true-or-false questions or something. My focus is on repeated questions, not on linear regression or the Likert scale.
Edit 3: Imagine I have four questions like these:
- Do you like our services? No 1 2 3 4 5 Yes
- Are you satisfied with our services? No 1 2 3 4 5 Yes
- Do you think our services are good enough? No 1 2 3 4 5 Yes
- Will you come to our store again? No 1 2 3 4 5 Yes
and I want to know the relationship between how people like our services and whether they'll be back. (Assume also that I'm sure simple linear regression is applicable. I've used each of 1-3 as regressor, found the least-square estimates of $b_0$ and $b_1$ each time, ensured the error is normal with the chi-squared goodness-of-fit test, ensured the relationship is linear with the F-test for lack of fit, dealt with outliers, etc.)
How should this relationship be analysis? If the responses to the first three questions are distributed very differently (say, significantly different mean or variance through Student's t- and F-tests), what is to be done about it? What if $H_0$ can be rejected for one or two questions but not the others – what can be done about this? Thanks.
Best Answer
When there are multiple questions getting at the same issue in a questionnaire, these are typically aded together to get a combined score. The individual questions are Likert items, with their combination providing a Likert scale. This helps get around several types of problems that can arise from depending too heavily on a single response to a single item. So the task of a designer of a questionnaire is more typically to evaluate how well a set of items all address the same issue, rather than to find a single "best" item as this question might suggest. I suppose you could consider this a case of making multicollinearity work for you. Cronbach's alpha is a standard way to evaluate whether a set of items is measuring the same thing, to evaluate their internal consistency.
As Michael Chernick has noted in another answer, any plan to choose a "best" item based on p-values for relations to your outcome variable is fraught with difficulty. In addition to the multiple-testing statistical issues, it is likely that the "best" item found in one sample of answers to a set of items will be different from that found in another sample. Try repeating your item-selection protocol on multiple bootstrap samples of your data to illustrate that problem.
This page is one of several useful threads on this site about design, validation, and interpretation of Likert items and scales; a search for Likert item shows many more. I understand the motivation to try to use a smaller number of questions, but that can pose problems as discussed on this page.
If you have several items that you want to analyze as individual predictors rather than in a combined Likert scale, logistic regression as suggested in the question would be a possible approach, but with some cautions. The responses to these items are likely to have substantial multicollinearity, which can pose problems for any type of regression. Ridge regression is often a useful way to combine multicollinear predictors without running risks of overfitting, in logistic, linear, or Cox regression models. As with other regressions, you could incorporate transformations of the predictor variables if needed to get around issues arising from the distributions of the predictors or, more precisely, of their relations to outcome.