Does any definitive work exist, e.g. a review, a book chapter, or a book, on the advantages and disadvantages of existing approaches? Is there a consensus on which approach is the right one?
I have come across three approaches so far:
-
We test by a statistical test whether a given assumption of a statistical test is met. The problem with this is that the error of checking the assumption actually adds to the error of the test we originally wanted to perform. So it's misleading to report later the result of the latter test as, e.g. $p<0.05$ because, in fact, the test procedure's error rate is more than $5\%$. Is this accumulation of errors calculable?
-
Later I was taught that the previous approach is heresy. Instead, we tested by Q-Q plots the probably most common assumption: normality. But it is also based on the sample on which we will later apply the test we want to perform. Is this relevant? Plus, it seems like all that’s happening is that we replace the statistical error with a subjective error because the figure needs to be assessed. This subjective error is certainly not calculable. It might be decreased by randomly generating as many data points as there are in our sample from a normal distribution parametrized based on the sample mean and SD, and comparing the Q-Q plots.
-
We don't use parametric tests. Well, we use them only if we know a priori that their assumptions are met. When we don't know that for sure, we don't check anything, we use non-parametric tests.
Best Answer
In my opinion semiparametric regression models provide the best default approach. They are invariant to Y-transformations and contain common nonparametric tests as special cases, but allow for much more than that. An overview is here.
As you rightly noted, assessing assumptions (whether by p-values or graphs) can make unknown changes to tests' operating characteristics.