Why would exploratory data analysis be important to undertake before null-hypothesis tests?
Solved – Exploratory data analysis vs null hypothesis testing
exploratory-data-analysishypothesis testing
Related Solutions
Answer to question 1: This occurs because the $p$-value becomes arbitrarily small as the sample size increases in frequentist tests for difference (i.e. tests with a null hypothesis of no difference/some form of equality) when a true difference exactly equal to zero, as opposed to arbitraily close to zero, is not realistic (see Nick Stauner's comment to the OP). The $p$-value becomes arbitrarily small because the error of frequentist test statistics generally decreases with sample size, with the upshot that all differences are significant to an arbitrary level with a large enough sample size. Cosma Shalizi has written eruditely about this.
Answer to question 2: Within a frequentist hypothesis testing framework, one can guard against this by not making inference solely about detecting difference. For example, one can combine inferences about difference and equivalence so that one is not favoring (or conflating!) the burden of proof on evidence of effect versus evidence of absence of effect. Evidence of absence of an effect comes from, for example:
- two one-sided tests for equivalence (TOST),
- uniformly most powerful tests for equivalence, and
- the confidence interval approach to equivalence (i.e. if the $1-2\alpha$%CI of the test statistic is within the a priori-defined range of equivalence/relevance, then one concludes equivalence at the $\alpha$ level of significance).
What these approaches all share is an a priori decision about what effect size constitutes a relevant difference and a null hypothesis framed in terms of a difference at least as large as what is considered relevant.
Combined inference from tests for difference and tests for equivalence thus protects against the bias you describe when sample sizes are large in this way (two-by-two table showing the four possibilities resulting from combined tests for difference—positivist null hypothesis, $\text{H}_{0}^{+}$—and equivalence—negativist null hypothesis, $\text{H}_{0}^{-}$):
Notice the upper left quadrant: an overpowered test is one where yes you reject the null hypothesis of no difference, but you also reject the null hypothesis of relevant difference, so yes there's a difference, but you have a priori decided you do not care about it because it is too small.
Answer to question 3: See answer to 2.
Best Answer
It is often necessary to know a little about the system being explored before sensible hypotheses come to mind and it is very useful to know about the variation and noise in an assay prior to designing an experiment. Exploratory experiments and analyses are good for that. Don't be too quick to decide that a dataset is definitive.
Of course, you should know that hypotheses that are suggested by the data in exploratory analyses will have a high chance of giving you a spurious 'significant' result if you test them using the same data, so ideally the exploratory analyses lead to the design and running of new experiments to specifically test hypotheses.