So I understand that saying that a p-value is the probability of a particular result coming about by chance is incorrect as per community consensus, the American Statistical Association, etc.
My internal understanding of the p-value is that it is the probability of getting a test statistic of a certain value or greater (depending on tailedness of test) in a hypothetical sampling distribution of said statistic where the distribution parameters are set to reflect the null hypothesis, which is often specified to be a non-relationship.
However, the problem I am facing is that to me, this does not conflict with the idea of "chance." To me, the spread of that hypothetical distribution is not necessarily just subject to the whims of the world, but could also be a function of observable variations in other variables. All I see that hypothetical distribution as representing is the possible spread of the test statistic, without making any claims about why the spread occurs in any particular way. So to me, "the chance of getting a value of X or greater" feels the same as "probability of getting a value of X or greater by chance over repeated random samples."
It occurred to me that the reason why I was having trouble seeing why saying "probability that result happened by chance" is categorically incorrect may be because I don't understand how people are using the word "chance." Or, alternatively, is there a different problem? Am I indeed understanding something about p-values incorrectly?
On a practical note, this is bleeding into my teaching, where I am struggling to find a way to explain the p-value to practice-oriented (public policy) introductory statistics students in a way that is reflective of what is actually happening re: the null hypothesis. I have seen discussions of this elsewhere (e.g., here), but I have not seen a discussion of the word "chance" in this context and what it means to people. I try to avoid saying "by chance," but I'm finding it difficult to find ways to explain it otherwise in a way that doesn't require me also teaching them about probability distributions, some ideas about areas under a curve, what "random" means in the statistical sense, etc.
Best Answer
I'm not sure it's incorrect. It depends on what you mean by something "coming about by chance", which isn't statistically precise terminology. People who have $p$-values described to them as "the probability that the result came about by chance" may think that, for example, if the $p$-value is low, the result is unlikely to have "come about by chance", and hence the estimate of the population effect must be very accurate. In reality, rejecting the null hypothesis doesn't imply your estimate is particularly accurate.
I don't think it's possible to explain a $p$-value in a way that's correct and not misleading without getting into the basics of statistics. You need to describe probability itself, distinguish between samples and populations, explain the notion of a parameter and a parametric family, etc.