Statistical Inference – Is There Any Reference Justifying P<0.10 as a Cutoff for Likelihood-Ratio Tests?

association-measureinferenceregressionstatistical-power

I am conducting log-likelihood-ratio tests to examine the goodness of fit of two competing logistic regression models. The test is the traditional log-likelihood-ratio test, in which I compare the ratio of the two log-likelihoods to identify the "best" model between two nested models.

I have used 10% as a cutoff (pre-specified threshold); when P < 0.10, there is evidence that the more complex model fits the data "better" than the null model without the additional covariate. I haven't used a P <0.05 because my dataset is relatively small.

Is there any empirical evidence/recommendation that using a P<0.10 is a good thing when the statistical power is low? Any references/suggestions are remarkably welcome.

Best Answer

No, there is no evidence that this is a good approach. Building models based on whether some significance test has $p \leq 0.05$ or $\leq 0.1$ associated with it is problematic. For a start, it invalidates naive post-selection inference (i.e. just using the "selected" model and treating coefficients and hypothesis test results from this model, as if it had been prespecified). It is also not a particularly good approach for building a good predictive model, either. The particular p-value threshold for this is kind of irrelevant for this question, but even the common usage of 0.05 is pretty arbitrary in the first place (and some have argued that often lower thresholds would be desirable).