R – Resolving Conflict Between P-Value and Confidence Interval in Gamma Model

confidence intervalgamma distributionp-valuerstatistical significance

I am trying to fit the Gamma model with link = log in R using the glm function.

Below is the summary of the model:

The p-value for level 2 of modact_3 < 0.05, but the confidence interval for this level includes 0 (The null hypothesis is that the coefficient = 0), which should not includes 0 since the null is rejected (p-value < 0.05).

Could anyone help to explain this conflict? Thanks a lot!

Best Answer

See how you get the message "waiting for profiling to be done..." when you run confint? That is because confint returns profile likelihood confidence intervals, which inverts the likelihood ratio test statistic to find regions where the LRT would fail to be rejected. The p values in summary come from a Wald test. The two methods are almost the same, but can yield small differences. If you want Wald confidence intervals, use confint.default.

I don't see any mention of normality in ?confint, so I can only speculate as to what that post you linked to was referring to (you're better off to comment on that answer directly). The Wald test makes an assumption that the log likelihood is locally quadratic, meaning the likelihood is approximately normal. We make use of this assumption a lot (for instance the asymptotic sampling distribution of the coefficients is approximately normal). So the normality is not about the conditional mean, but rather the likelihood.

You're fine using either method. I would suggest using confint since the confidence intervals are derived from the likelihood function directly as opposed to some approximation which may or may not be appropriate based on the size of data you have. Don't base your decision on one method rejecting the null and not the other, because The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant.

Related Solutions

Solved – Calculating confidence interval of return period

You can get confidence intervals for the CDF. So if the quantity of interest is a function of the CDF, you could use those intervals. Here's an example:

x = gamrnd(100,1.3,100,1);
d = fitdist(x,'gamma')
grid = linspace(50,200)';
[y,ylo,yhi] = cdf(d,grid);
plot(grid,y,'b-',grid,ylo,'b:',grid,yhi,'b:')

Statistical Significance – Resolving Discrepancies Between P-Value and Confidence Interval in a Binomial Test

The problem with tests of binomial proportion is that the tests used are generally approximate (since the exact "Clopper-Pearson" test is ridiculously conservative). Therefore, it's not clear that the procedure used to get the CI is the same as that used to test the hypothesis. Theoretically, either approach should lead to the same conclusion if you are using just one CI and one test.

You have a border case. Either statistic is telling you that your observation is not all that common under the null hypothesis. Remember: there is nothing special about 5% significance...its a cultural artifact from Ronald Fisher in the 1930's. Its a guideline.

For what it's worth, I'd conclude that its unlikely that the true success probability is as high as 0.75.

Per @John

In a strict hypothesis testing situation, you're stuck with your 0.05, so you would reject the null hypothesis under that criteria. However, I wouldn't run to the press quite yet ;-)...hypothesis tests can really destroy any nuance in inference.

Best Answer

Related Solutions

Solved – Calculating confidence interval of return period

Statistical Significance – Resolving Discrepancies Between P-Value and Confidence Interval in a Binomial Test

Related Question