I am currently reading the paper
Duncan J Murdoch, Yu-Ling Tsai & James Adcock (2008) P-Values are Random Variables, The American Statistician, 62:3, 242-245, DOI: 10.1198/000313008X332421
In this paper, the authors argue that a p-value is itself a random test statistic. Moreover, given some test statistic $T$ that takes in an i.i.d. sample $\mathbf{X}$ and outputs a real number, a p-value is the probability integral transform of $T(\mathbf{X})$. That is, if $T(\mathbf{X})$ has a cumulative distribution function $F_{\tau}$, then the corresponding p-value is $F_{\tau}(T(\mathbf{X}))$. We can then decide whether to reject a null hypothesis based on this p-value. For example, a decision rule could be to reject the null hypothesis if the corresponding p-value is less than 0.05.
However, because $F_{\tau}$ is monotone increasing, I am not sure why we need to compute a p-value in the first place to decide whether or not to reject a null hypothesis. Can't the value of the test statistic $T(\mathbf{X})$ be used to decide? For example, if the decision rule is to reject the null hypothesis when the p-value is less than 0.05, then, if the inverse of $F_{\tau}$ exists, we can obtain the threshold value for $T(\mathbf{X})$ beyond which the null hypothesis is rejected. Furthermore, we should be able to compute type I and II error rates just using this threshold value for $T(\mathbf{X})$.
Best Answer
I think you are almost constructing p-values in the question.
You are correct, you can just set a threshold using $t=T(X)$, but as you point out, you'd like to calculate the error rate associated with that t-value. In order to do so you need to know the null distribution of your test statistic, which is $F$. So in order to find the t-value that has a 5% false positive under the null, you need to find $t$ such that $F(t)=0.05$, i.e. $t=F^{-1}(0.05)$
In many (most?) cases inverting the cdf is harder than evaluating it, hence it is much easier to calculate $p=F(T)$ and then check if $p<0.05$ than to check if $t<F^{-1}(0.05)$