In an experiment such as tossing an unbiased coin, we know that the probability of getting a 'head' or a 'tail' is 1/2. I understand this to mean if the number of trials is large enough. My question is how is that "large enough" number determined mathematically? Is there a formula for it? I can guess it would have to depend on the sample space, I mean for a dice with 6 faces, that number should be different from the coin of 2 faces. I am sure also that it is at larger or equal the sample space but I want to know the formula for it if such a formula exists.
Solved – the formula for number of trials required given the probability
probability
Related Solutions
This is a computational exercise, so think recursively. The current state of coin flipping is determined by the ordered pair $(N,M)$ with $N\ge M\ge 0$. Let the expected number of flips to reach $N$ consecutive heads be $e(N,M)$:
(1) There is a 50% chance the next flip will be heads, taking you to the state $(N,M+1)$, and a 50% chance the next flip will be tails, taking you to the state $(N,0)$. This costs one flip. Therefore the expectation (recursively) is given by
$$e(N,M) = \frac{1}{2} e(N,M+1) + \frac{1}{2} e(N,0) + 1.$$
(2) Base conditions: you have already stipulated that
$$e(N,0) = 2^{N+1}-2$$
and obviously
$$e(N,N) = 0$$
(no more flips are needed).
Here's the corresponding Mathematica program (including caching of intermediate results to speed up the recursion, which effectively makes it a dynamic programming solution):
e[n_, m_] /; n > m > 0 := e[n, m] = 1 + (e[n, m + 1] + e[n, 0])/2 // Simplify
e[n_, 0] := 2^(n + 1) - 2
e[n_, n_] := 0
The program would look similar in other programming languages that support recursion. Mathematically, we can verify that $e(N,M) = 2^{N+1} - 2^{M+1}$ simply by checking the recursion, because it obviously holds for the base cases:
$$2^{N+1} - 2^{M+1} = 1 + (2^{N+1} - 2^{M+2} + 2^{N+1} - 2)/2,$$
which is true for any $M$ and $N$, QED.
More generally, the same approach will establish that $e(N,M) = \frac{p^{-N} - p^{-M}}{1-p}$ when the coin has probability $p$ of heads. The hard part is working out the base condition $e(N,0)$. That is done by chasing the recursion out $N$ steps until finally $e(N,0)$ is expressed in terms of itself and solving:
$$\eqalign{ e(N,0) &= 1 + p e(N,1) + (1-p) e(n,0) \\ &= 1 + p\left(1 + p e(N,2) + (1-p) e(N,0)\right) + (1-p) e(N,0) \\ \cdots \\ &= 1 + p + p^2 + \cdots + p^{N-1} + (1-p)[1 + p + \cdots + p^{N-1}]e(N,0);\\ e(N,0) &= \frac{1-p^N}{1-p} + (1-p^N)e(N,0); \\ e(N,0) &= \frac{p^{-N}-1}{1-p}. }$$
Could be a Binomial B(n,p) with n very large and p very small. In such cases you can also have a look to the "law of rare events" which is a Poisson(np) simpler to manage than a Binomial. Look at https://en.wikipedia.org/wiki/Poisson_distribution#Law_of_rare_events
Best Answer
By the weak law of large numbers
$$ \lim_{n\to\infty}\Pr\!\left(\,|\bar{X}_n-\mu| > \varepsilon\,\right) = 0 $$
so as your sample grows up to infinity, the empirical mean $\bar{X}_n$ will get closer and closer to the true mean $\mu$. And by the strong law of large numbers, as sample size goes to infinity, the empirical mean will converge almost surely to the true mean
$$ \Pr\!\left( \lim_{n\to\infty}\bar{X}_n = \mu \right) = 1 $$
As you can see, neither of the statements say that there is any finite sample size that will let you achieve this. On another hand, they do not say that there is some specific sample size that is needed to observe some event with some probability: if you throw a fair coin only once, the probability of heads is still $1/2$.
The thing that you could estimate is the standard deviation of your estimate of probability of heads $p=1/2$ after $n$ trials using Wald's formula
$$ \sigma = \sqrt{ p(1-p)/n } = \sqrt{0.25/n}$$
so you could find $n$ large enough for $\sigma$ to be small enough for you to consider the variability of $p$ acceptable. But still, this is your subjective choice of what do you consider as "small enough".