I am trying to do a two samples t-test. The null hypothesis is there is no difference in mean runs scored by two teams, the alternative is there is a difference.
sample 1 size = 104
sample 2 size = 150
mean of sample 1 = 236
mean of sample 2 = 184
significance of the test= 0.05,
Test statistics = 2.46,
P value = 0.014,
cohen's d = 0.31(small)
How to interpret this result? The t-test says that on average team1 makes more runs than the team2 and the result is statistically significant. But the effect size is small, d=0.31. So what should I conclude from this result? Should I conclude that there is a difference between two samples and team1 perform much better than team2 or should I reject the this claim as the effect size is small? What to do in these type of situations and How do you describe the result to others?
Best Answer
The conclusion is exactly what the test says.
There is a small difference between the two groups, and it is unlikely to have arisen by chance. That is, you can have confidence that the one group really is higher on average than the other. However, the difference is $0.31$, which might not be enough for you to care about the difference. It is okay to conclude that a statistically significant difference lacks practical significance. (Perhaps your business requires a difference of at least $0.5$ to be able to profit from the new technique; you don't have enough of an improvement to profit.) Always remember that hypothesis tests are---and should be---sensitive to sample size, with large sample sizes being able to reject small differences that really are there but might be so small that they do not warrant our attention.
As mentioned in the comments, however, $0.31$ does not have to be small. That is up to your knowledge of the subject of the opinions of subject matter experts (physicians, engineers, etc).