In the Pilot Study, I am trying to confirm reliability of my scales:
I'm performing Cronbach’s alpha test on 5 items that represent one construct of my research model (IV). I'm receiving α = .710 which sounds good.
However, from Item-Total Statistics I can see, that if item 5 was eliminated from questionnaire, Cronbach’s alpha will be .850. I do perform Cronbach’s alpha test again, without item 5, and receiving α = .850. Again, I am looking at the Item-Total Statistics table and can see, that if I eliminated item 3, Cronbach’s alpha will be even greater, say .890. Of course, I might eliminate another item of my questionnaire (item 3) in order to receive a greater alpha but, is it necessary?
For example, in case if I had 10 constructs, each represented by 5 items, and needed to eliminate 2 items for each construct then my questionnaire instead of 50 items will be made of 30 items.
As we can see, after the first test I am already receiving Cronbach’s alpha = 0.710 which is good enough to confirm internal consistency associated with the scores. What if, after I collected data from much larger sample (then one used for Pilot study – 20 ppl), removed in Pilot study questions could produce different results? Wouldn't it be wiser to not remove the items, run survey and collect data and then, after run Cronbach’s alpha test again (on larger sample), see if any items (responses) need to be excluded from the further study?
Thanks!
Best Answer
First, there's nothing magic about 0.7 as a suitable value for alpha. Papers often cite Nunnally and Bernstein (on Nunnally's earlier edition), but it does not say that 0.7 is somehow OK, it's more complex and more nuanced than that.
Second, alpha has problems - for alpha to be an unbiased estimate of reliability, you have to make a bunch of assumptions, which are rarely satisfied.
Third, if you want high reliability in your instrument, just keep asking the same question. You aim for validity, and reliability is a point along the way but if you focus purely on reliability, you lose validity. For example, depression has a range of symptoms (feeling sad, lack of emotional response, lack of motivation, loss of appetite), and therefore to assess depression, you need to assess all of those symptoms. If your items were "I feel sad", "I feel down", "I feel gloomy", "I don't feel very happy", you will have a very high alpha, but you will not have a good measure of depression.
Finally, reliability is not a property of a test or instrument. Reliability is a property of a test in a sample. A different sample will give you different reliability.