Solved – Is a p-value of 0.04993 enough to reject null hypothesis

hypothesis testingp-valuestatistical significance

In a Wilcoxon signed-ranks statistical significance test, we came across some data that produces a $p$-value of $0.04993$. With a threshold of $p < 0.05$, is this result enough to reject the null hypothesis, or is it safer to say the test was inconclusive, since if we round the p-value to 3 decimal places it becomes $0.050$?

Best Answer

There are two issues here:

1) If you're doing a formal hypothesis test (and if you're going as far as quoting a p-value in my book you already are), what is the formal rejection rule?

When comparing test statistics to critical values, the critical value is in the rejection region. While this formality doesn't matter much when everything is continuous, it does matter when the distribution of the test statistic is discrete.

Correspondingly, when comparing p-values and significance levels, the rule is:

          Reject if $p\leq\alpha$

Please note that, even if you rounded your p-value up to 0.05, indeed even if the $p$ value was exactly 0.05, formally, you should still reject.

2) In terms of 'what is our p-value telling us', then assuming you can even interpret a p-value as 'evidence against the null' (let's say that opinion on that is somewhat divided), 0.0499 and 0.0501 are not really saying different things about the data (effect sizes would tend to be almost identical).

My suggestion would be to (1) formally reject the null, and perhaps point out that even if it were exactly 0.05 it should still be rejected; (2) note that there's nothing particularly special about $\alpha = 0.05$ and it's very close to that borderline -- even a slightly smaller significance threshold would not lead to rejection.