Roughly how many times do I need to roll a 6-sided die to feel confident that it's giving "fair" results? What about a 10-sided or 20-sided die?
Note that I will be actually manually rolling physical dice, this isn't just a textbook exercise. I'd like to minimize how long it takes me to perform this experiment with each die 🙂
I know this depends on my expected "confidence level" (95%? 99%?) If I choose a 95% confidence, for example, does that imply that 1 out of 20 fair dice will fail this test? Or that a single fair dice would fail the test 1 out of 20 times? If so, that sounds fairly high.
Are there standard techniques for doing this kind of a test?
Edit: It is beyond the scope of the math-focused question I've asked here, but I've explained more about the overall testing scenario over on the stats site here: https://stats.stackexchange.com/questions/14301/designing-a-test-for-a-psychic-who-says-he-can-influence-dice-rolls/14302#14302
Best Answer
A chi-square test is the first thing that comes to mind: $$ \sum\frac{(\text{observed} - \text{expected})^2}{\text{expected}} $$ If you roll the die $n$ times, the "expected" number of times you would see any particular outcome is $n/6$. If $n$ is large, this has approximately a chi-square distribution with 5 degrees of freedom. You reject the null hypothesis of fairness if the test statistic given above is large.
95% confidence does mean one out of twenty fair dice will fail.
See also this amazing analysis by a physicist of perhaps the most extensive experiment of this kind ever done: http://bayes.wustl.edu/etj/articles/entropy.concentration.pdf
A further refinement of the chi square test would be to note that each outcome of a roll has an opposite face. If one outcome is unusually high, the opposite face should be unusually low. Thus, it is the difference between opposite face frequencies that detect unbalance in the die. You could create a simulation in a simple spreadsheet and find the confidence limits by Monte Carlo.