I administered a customer service satisfaction survey, and one of the questions is a check-all-that-apply asking respondents to check the boxes for the services that they liked most (medication assistance, linkage to medical doctor, housing assistance, therapy, and additional funds). I wanted to determine not only if there were significant differences in the proportions of "yes" to "no" response counts as a function of age group (55-60, 61-65, 66-70, 71+) for each of the above services, but also to see if there were any significant decreasing or increasing trends in proportion of "yes" to "no" responses as a function of increasing age for each category. I received insight into this at this post. Here is the link to my post to get clarity on tests to run for these research questions.
I conducted chi Square tests to investigate if any significant differences in popularity of the above services, and separately Cochran-Armitage tests (linear-by-linear association test for trends) to answer the question about whether there are any trends for each of the categories.
Now I wonder if anyone can offer insight into the following finding for the "medication assistance" choice. I'm using an alpha of .05 as the threshold for rejecting the null hypothesis, and the linear-by-linear association finding shows that the result is significant (p=.035) while the associated Pearson Chi-Square is not (p=.213). How can there be both a significant trend of decreasing proportion of popularity of medication assistance, but at the same time a non-significant difference in proportions of "yes" to "no" responses as a function of age group? Isn't the finding of a significant trend dependent on the significant differences between the proportions of "Yes" to "no" responses of each level of age group?
tables of yes and no responses and findings for medication assistance choice
Best Answer
One thing to understand about p-values is that they are dependent on sample size. As a simple experiment, multiply each of your counts by 2, and then re-run the tests. The p-values will decrease, despite the fact that the distribution of the counts is exactly the same as your original data. This is one reason why experimenters are cautioned about putting too much meaning into p-values.
Another thing to keep in mind is that if you fail to reject the null hypothesis, that is, if p >= 0.05, the conclusion is not that you accept the null hypothesis, but that you don't have good evidence to reject the null hypothesis. This may simply be because you don't have sufficient power to reach a small enough p-value. If you had the same distribution of values, but just twice the observations, you may have had sufficient power to come to a different conclusion.
Considering these tests and your results, basically the fact is that the Cochran-Armitage test has greater power than a chi-square test of association if there is a trend in the ordinal categories. That's the advantage of this test in these situations.