In most of the times when effect size is reported, it seems to me that there is a clear inverse proportionality with p-value. I know that effect sizes bring information that is independent from significance, which is obvious when one considers the extreme cases of a very small and very large sample size – when large effects are difficult to found significant, and small effects can be found significant respectively. What would help is to see a plot of effect size as a function of p-value, with N as a curve parameter, but I haven't been able to find such a plot. Can anyone suggest where to find / how to create such a plot, or otherwise shed light on the problem?
Solved – Relationship between effect size and statistical significance
effect-sizestatistical significance
Related Solutions
It's probably easiest to compute $\eta^2_p$ directly from the $t$- or $F$-statistic observed with the contrast. You can do that using the formula $$ \eta^2_p = \frac{F}{F+v_d/v_n}, $$ where $v_d$ is the denominator degrees of freedom for the $F$-ratio and $v_n$ is the numerator degrees of freedom.
When you do a statistical analysis, you get a parameter. Sometimes that's easy "Males were 4 inches taller than women". We know what it means to be male or female, and we know what inches are. Sometimes the parameters are not so interpretable. "A one unit change Log beta-interferon level was associated with a 4 point drop in PTSD score on the PCL" (I made that up), I don't know what a log beta interferon level is, and I don't know what a 4 point drop on the PCL means.
To get around this problem, there are standardized effect sizes, and Cohen gave some rough guidelines, based on his experience, for the sort of effect size you would expect in social and behavioral sciences (I forget, he might have just said behavioral). These effect sizes, like f^2 (which is a transformation of R^2), and r (the correlation) are designed to help interpretation when you don't know what else you're doing.
A correlation of 0.1 is a small correlation. But a correlation of 0.01, in a regression with one predictor, leads to an f^2 of 0.01. But an f^2 of 0.02 is a small effect size. How can this make sense? Because Cohen said "This is the sort of small effect size that I see in multiple regression - an f^2 of 0.02 (which is an R^2 of about 0.02) and this is the sort of small effect size that I see in correlation - an r of 0.1.
You are making the mistake of reifying these interpretations into something more meaningful. A correlation of 0.5 is a large effect size. But if you wrote "The correlation of the scores of the two blood pressure meters was calculated and found to be 0.5, which is large", you would (hopefully) be laughed at. In the context of two devices that try to measure the same thing, a correlation of 0.5 in incredibly low. Similarly, a correlation of 0.1 (which is small) between living near power lines and leukemia would be enormous.
So what should you do? You should interpret your parameters in the context of what you are investigating. And if you have no idea what that means, you can fall back on an effect size. It doesn't really matter which one, it's just a verbal way of quantifying the disease. It's a bit like when the doctor says "Your blood cell count is 19" and you ask what that means, the doctor says "Bad". The doctor won't describe your blood cell count to other doctors as "bad" because they know what it means.
Best Answer
Take a look at the formulae for your favorite hypothesis test and notice that the $p$-value depends not only on the effect size, but also on the sample size(s). Furthermore, different tests use different notions of effect size. These two reasons are why no such plot as you have requested can be created for the general case.