I am doing peer review on a paper submission for a journal, and I am somewhat out of my depth.
The authors are developing a psychometric scale. They did a pilot study with 45 items. They measured Cronbach alpha and found that it is 0.89 overall.
Then they found that six items have Cronbach's alpha if item deleted of 0.89 or very similar (between 0.89 and 0.893). They show a table with all six. They then write
A number of modifications were made according to what Table 2 suggested
And go on to explain how the wording of each item was changed to make it clearer. Obviously, they didn't want to have items whose Cronbach alpha if deleted is the same as the overall Cronbach alpha.
Why are they doing this? Is this a common technique in constructing psychometric scales? Are there guidelines for doing it?
They also don't cite any literature about the method they followed to develop their scale. It does look sound overall (pilot study, exploratory factor analysis, then refining the questionnaire based on the results), but I'm not sure whether the decisions they make about details (such as the above change of six items) are justified. Is the knowledge how to make a scale so common that there is no need to cite sources?
Best Answer
Cronbach's $\alpha$ tends to increase with the number of questions. If after deleting an item your Cronbach's $\alpha$ is the same, then you haven't gained any reliability by having that item. All other things being equal we tend to prefer shorter questionnaires... so we drop the item. One possible issue here is that the "Cronbach's $\alpha$ if deleted" will tend to all change any time an item is dropped. So perhaps after dropping one item a different decision would be made as to which items to drop. Regardless, these sorts of approaches to Cronbach's $\alpha$ are sufficiently common in my field that I wouldn't expect for them to be cited (or for them to take the ardious approach I just described); nor would I expect anybody to cite the source of the Cronbach $\alpha$. However, these standards do vary depending on area.
Ultimately, I think your duty as a reviewer in this case would be to let the editor know which part of the manuscript you feel goes outside of your expertise and focus on providing comments on the parts you can provide insight into. Alternatively, if you haven't been sitting on this for too long you might request that the editor assign this paper to another reviewer. Regardless, those issues are probably left to a different SE site.