I think have a misconception about statistical significance.
For example, I have two treatments A and B. Mean effect of A is -105 with 95%CI:(-168,-42) and for B -88 with 95%CI: (-146, -29). Which of the results is more statistically significant then?
When can we conclude in general that result is more statistically significant based on p-values or confidence intervals?
However I've read here that
A result does not become "more" statistically significant if the p value is "a lot smaller" than the alpha value, as opposed to being simply "slightly smaller".
So is it correct to compare statistical significance in such a way?
I'm very confused, so I appreciate any hints.
Best Answer
Of course you're confused. The web page you're reading is confusing. The opening sentence is incorrect and the opening paragraph is riddled with problems primarily focusing on a magnitude of significance. In fact, the only paragraph on the page that seems satisfactory to me is the one from which your quote is taken.
Nevertheless, there is a consistency. If significance were to have an amount it's not the p-value, it's the alpha value. And, the meaning of alpha is explained there well. Therefore, according to the site, one can infer that the amount of significance can only change due to lowering the alpha value before the study is run and the discovered p-values or CI's make absolutely no difference. The latter part of that is correct. But, generally it's advised that there just really isn't such a thing as more or less significant. I don't know anyone personally who would endorse that website's interpretation of the term "more significant".
It appears, from your question, that you want to say one of your tests is more significant than the other. You cannot do that, even following the referenced website, because you calculated 95% CI's for both and therefore used the same alpha values (alpha is used to determine the % of the CI). You can't change alpha afterwards. Furthermore, it's quite unlikely that you even want to say "more significant" for treatment A. You would probably like to say that it had a larger effect. In that case, what you would do would be to compare the effects of treatment A and treatment B directly.
The test comparing treatment A and B would fail to show they were different. That's evident in the large overlap in your confidence intervals. I'm suspecting they're fairly wide CI's as well. I say "suspect" given that the width is substantially greater than the magnitude relative to 0. I'd need to know much more about the data to make a confident qualitative statement that they are wide because it depends on several factors. However, given the apparent width, the best thing you can say is that the CI's of A and B largely overlap and it cannot be determined from these small samples whether the observed differences are simply due to variable samples because those CI's are also rather wide.