Say we have the empirical c.d.f. of $X_{i}$ to be
$\hat{F}_{n}\left(x\right)=\frac{1}{n}\sum_{1\leq i\leq n}I\left\{ X_{i}\leq x\right\} $. How can we show that this converges in probability to the true c.d.f. that is to show $\forall\varepsilon>0,\quad\lim_{n\to\infty}P\left\{ \left|\hat{F}_{n}\left(x\right)-F\left(x\right)\right|>\varepsilon\right\} =0$? Asymptotic Statistics by van der Vaart says to use the Strong Law of Large Numbers but I don't really get how that works. Is there a simpler way of showing it with the Weak Law of Large Numbers. I've tried showing it with Markov's Inequality as well, but I've gotten stuck there.
[Math] Showing the empirical cdf to be consistent
probability theorystatistics
Best Answer
Let $Y_i = I\{X_i \leq x\}$. Then, the $Y_i$ are i.i.d. with mean $F(x)$. By the strong law of large numbers (you'll have to check some other condition on $Y_i$ beyond finite mean -- see Durret's Probability book or any other intro probability book, but it suffices to note that $E[Y_i^4] \leq 1$, so you can apply the strong law of large numbers), $\hat{F}_n(x) = \frac{Y_1 + Y_2 + \ldots+ Y_n} {n} \to E[Y_i] = F(x)$ almost surely. Note convergence in the almost sure sense implies convergence in probability, so you have $\hat{F}_n(x) \to F(x)$ in probability as well.
If you switch the law of large numbers, you'll get different versions of convergence, but the idea is basically the same: $\hat{F}_n(x) = \frac{Y_1 + Y_2 + \ldots+ Y_n} {n}$ is an average of i.i.d. random variables, which converges in some appropriate sense (depending on which law of large numbers you apply) to $E[Y_1] = F(x)$. You could apply a weak law of large numbers and get convergence in probability directly.
The Glivenko-Cantelli theorem sharpens the pointwise convergence to uniform convergence in the above argument. And you can further improve on that to get the rate of convergence of the empirical cdf as in the Dvoretzky-Kiefer-Wolfowiz inequality.