Solved – What does it mean to have a small p value and an OR=1

logisticmachine learningodds-ratiop-valueregression

As the questions says, in a logistic regression I have small p value for a variable indicating statistical significance (p = 0.001), however the odds ratio at a 95% confidence interval is 0.999 (both upper and lower bounds of the confidence interval are also 0.999). How to interpret that? I can reject my null hypothesis, but the variable has the same effect on both classes? How can I have a small p value if that is the case?

Here is some sample output:

                      OR        2.5 %    97.5 %
(Intercept)           1.4315133 0.9037277 2.2814700
variable 1            0.9999999 0.9999998 0.9999999
variable 2            1.3925532 1.2386589 1.5640352

                   Pr(>|z|)    
(Intercept)        0.128767    
variable 1         0.00108 **
variable 2         0.00000002594048139 ***

It is unclear to me how to interpret the results for variable 1.

Best Answer

P-values are not perfectly connected to the magnitude of an effect. For one, if you have an extremely large sample size, it is perfectly possible to have an extremely tiny effect size paired with what most would consider a small p-value.

For example, a simple correlation of 0.01 when paired with a sample size of 100K will have a one-tailed p-value of 0.00078259.

http://www.danielsoper.com/statcalc3/calc.aspx?id=44 (try it out yourself)

P-values are not an indication of effect size in and of themselves and say nothing about the magnitude of the relationship.

At some point, the binary decision process of traditional significance testing loses its utility, and you might be at that point with your data.

In terms of interpretation, the most straightforward interpretation is that you have a very, very small effect made significant on the back of an extremely large sample size.

Related Question