Assume we have an empirical estimator $\hat{p}_{n}$ ($n$ is the amount of data points) of some true probability mass function $p$ defined on the set (finite or countably infinite) of integers. It is known that
$$
\hat{p}_{n} \overset{a.s.}{\to} p.
$$
Is the following uniform in $p$ result true:
$$
\sup_{p}||\hat{p}_{n} – p ||_{\infty} \overset{a.s.}{\to} 0,
$$
where the $\sup$ is taken among all possible probability mass functions.
I think it is Glivenko-Cantelli theorem, just can not see it.
Best Answer
That is very obviously false.
You likely have fooled yourself by your use of $p$ to mean two different things in the same context (you should never do that). First you use $p$ to be the limit of $p_n$, but then you use it as the dummy variable in $\sup_p\|\hat p_n - p\|_\infty$. Change that to $$\sup_q\|\hat p_n - q\|_\infty$$ And now you have $p$ available again to act as the limit of $p_n$. And in fact $$\lim_n \sup_q\|\hat p_n - q\|_\infty \ge \sup_q \|\lim_n \hat p_n - q\|_\infty = \sup_q\|p - q\|_\infty > 0$$