Let $X_1,X_2,\ldots$ be a sequence of independent and identically distributed real-valued random variables, each defined on the probability space $(\Omega,\mathcal{F},P)$ and with distribution $Q(\cdot)=P(X_i \in \cdot)$. For every measurable set $A$ in the Borel $\sigma$-algebra $\mathcal{B}(\mathbb{R})$, let
$$Q_n(A)=\frac{1}{n}\sum_{i=1}^nI\{X_i\in A\}.$$
From the Strong Law of Large Numbers it follows that
$$P\left(\lim_{n\rightarrow +\infty} Q_n(A) = Q(A) \right)=1\textrm{ for every $A\in \mathcal{B}(\mathbb{R})$.}$$
However, the Strong Law of Large Numbers alone is not sufficient to obtain the following stronger result:
$$P\left(\lim_{n\rightarrow +\infty} Q_n(A) = Q(A) \textrm{ for every
set $A\in \mathcal{B}(\mathbb{R})$.}\right)=1,$$
which amount to the strong convergence of the empirical distribution $Q_n(\cdot)$ to the law $Q(\cdot)$ over $\mathcal{B}(\mathbb{R})$.
Question: Under what conditions does the empirical distribution $Q_n(\cdot)$ converge strongly to $Q(\cdot)$ over $\mathcal{B}(\mathbb{R})$? Can it converge strongly at least over some uncountable subset $S\subseteq\mathcal{B}(\mathbb{R})$?
Best Answer
This answer shows that all random variables yield simultaneous convergence over sets $(-\infty, x]$ for all $x\in \mathbb{R}$. However, we get simultaneous convergence over all sets in $\mathcal{B}(\mathbb{R})$ if and only if the random variable is discrete.
Let $(\Omega, \mathcal{F}, P)$ be a probability triple and let $\{X_i\}_{i=1}^{\infty}$ be i.i.d. random variables. Define $X=X_1$.
1) Suppose $X$ has any CDF $F_X(x)$. The Glivenko-Cantelli theorem implies that $$ P\left[\lim_{n\rightarrow\infty}Q_n((-\infty, x])=F_X(x) \quad \forall x \in \mathbb{R}\right]=1$$ This positively answers the question "Can it converge strongly at least over some uncountable subset $S \subseteq \mathcal{B}(\mathbb{R})$?"
2) Suppose $X$ takes values in a finite or countably infinite set $\mathcal{X}$. We want to show $$ P\left[\lim_{n\rightarrow\infty} Q_n(A)= P[X\in A] \quad \forall A \in \mathcal{B}(\mathbb{R})\right] = 1$$ It suffices to show $$ P\left[\lim_{n\rightarrow\infty}Q_n(V) = P[X \in V] \quad \forall V \subseteq \mathcal{X}\right]=1$$
The law of large numbers implies that for any element $x \in \mathcal{X}$ we get: $$ \lim_{n\rightarrow\infty}Q_n(\{x\}) = P[X=x] \quad \mbox{with prob 1}$$ Define the event $$ B = \cap_{x \in \mathcal{X}} \left\{\lim_{n\rightarrow\infty}Q_n(\{x\})= P[X=x]\right\}$$ Since $\mathcal{X}$ is at most a countably infinite set we get $P[B]=1$. It suffices to show $$ B\subseteq \left\{\lim_{n\rightarrow\infty}Q_n(V)= P[X \in V] \quad \forall V \subseteq \mathcal{X}\right\}$$ (In fact, the above inclusion will imply that $B$ is equal to the set on the right-hand-side, since the opposite inclusion clearly holds.) Suppose $B$ holds true. Fix $V \subseteq \mathcal{X}$. We get $$ Q_n(V) = \sum_{x \in V} Q_n(\{x\})$$ Now, because $B$ holds true, we know $Q_n(\{x\})$ values are nonnegative and satisfy $Q_n(\{x\}) \rightarrow P[X=x]$ for all $x \in V$. Also we get $Q_n(V)\leq 1$ for all $n$. So we can dominate the sum by 1 and we get by Lebesgue dominated convergence (applied deterministically to the summation via the counting measure): $$ \lim_{n\rightarrow\infty} Q_n(V) = \sum_{x \in V} \lim_{n\rightarrow\infty} Q_n(\{x\}) = \sum_{x \in V} P[X=x] = P[X \in V]$$ $\Box$
3) This section shows that simultaneous convergence for all sets in $\mathcal{B}(\mathbb{R})$ (surely) fails if $X$ is not a discrete random variable.
Claim:
Let $\mathcal{X}$ be the set of all points $x \in \mathbb{R}$ for which $P[X=x]>0$ and suppose $P[X \in \mathcal{X}]<1$. Define the set $$ B = \left\{\omega \in \Omega: \lim_{n\rightarrow\infty} Q_n(A) = P[X\in A] \quad \forall A \in \mathcal{B}(\mathbb{R})\right\}$$ Then $B$ is the empty set and so $P[B]=0$.
Proof:
We know $\mathcal{X}$ is either finite or countably infinite and so $\mathcal{X} \in \mathcal{B}(\mathbb{R})$. By definition of $\mathcal{X}$ we know that $P[X =v]=0$ for all $v \in \mathcal{X}^c$. Also,
$$P[X \in \mathcal{X}^c] =1-P[X\in \mathcal{X}]>0$$
Define $\theta = P[X \in \mathcal{X}^c]$ (so $\theta >0$).
Define $\mathcal{J}$ as the collection of all sets of the type $\mathcal{X} \cup C$ where $C$ is a finite or countably infinite subset of $\mathcal{X}^c$. Note that every set $J \in \mathcal{J}$ is in $\mathcal{B}(\mathbb{R})$. Letting $J=\mathcal{X} \cup C$ be a particular set in $\mathcal{J}$, we have
\begin{align} P[X \in J] &= P[X \in \mathcal{X} \cup C] \\ &= P[X \in \mathcal{X}] + P[X \in C] \\ &= (1-\theta) + \sum_{v \in C}\underbrace{P[X=v]}_{0}\\ &=1-\theta \\ &<1 \end{align}
Now, for all $\omega \in \Omega$ and all $A\subseteq \mathbb{R}$ define: $$Q_n^{\omega}(A) = \frac{1}{n}\sum_{i=1}^n I\{X_i(\omega) \in A\}$$ For a given outcome $\omega \in \Omega$ we can define the set $D(\omega)\subseteq \mathbb{R}$ by: $$ D(\omega)= \mathcal{X} \cup \{x \in \mathcal{X}^c: x = X_n(\omega) \quad \mbox{for some $n \in \{1, 2, 3, ...\}$}\}$$ Then $D(\omega)$ is just the union of $\mathcal{X}$ with a finite or countably infinite number of elements of $\mathcal{X}^c$. Thus, $$D(\omega) \in \mathcal{J} \quad \forall \omega \in \Omega$$ It is clear by the definition of $D(\omega)$ that $$ I\{X_n(\omega) \in D(\omega)\} = 1 \quad \forall \omega \in \Omega, n \in \{1, 2, ,3 ...\}$$ and so $$ Q_n^{\omega}(D(\omega)) = 1 \quad \forall \omega \in \Omega, n \in \{1, 2, ,3 ...\}$$ Thus $$ \lim_{n\rightarrow\infty} Q_n^{\omega}(D(\omega)) = 1 \quad \forall \omega \in \Omega$$ Hence $$ \left\{\omega \in \Omega : \cap_{J \in \mathcal{J}} \left\{\lim_{n\rightarrow\infty} Q_n^{\omega}(J) = \underbrace{P[X \in J]}_{1-\theta}\right\}\right\} =\phi $$ where $\phi$ is the empty set. This is because the above intersection over all sets in $\mathcal{J}$ includes the set $D(\omega)$, for which we know the limit is $1$ (not $1-\theta$). Since all sets in $\mathcal{J}$ are in $\mathcal{B}(\mathbb{R})$ it holds that $$ \left\{\omega \in \Omega : \cap_{A \in \mathcal{B}(\mathbb{R})} \left\{ \lim_{n\rightarrow\infty} Q_n^{\omega}(A)= P[X\in A]\right\}\right\} = \phi$$ $\Box$