Suppose, $X_1, \dots, X_n$ are iid random vectors with some distribution $P$ on $\mathbb{R}^d$ and suppose $E_P(|X_1^k|)$ exists for all $k \leq K$ for some $K > 1$. Now, let $\hat P_n$ be the empirical distribution of $X$, constructed by putting weight $1/n$ on each observed $X_i$; let $F_n$ be its cdf.
If $\dim(X) = 1$, then by Glivenko-Cantelli, $\Vert F_n – F\Vert_\infty \rightarrow 0$ almost surely, and so, $E_{\hat P_n}(X^k) \rightarrow E_P(X^k)$, almost surely, for all $k < K$.
My question is, does this generalize to multiple dimensions? I know that the Glivenko-Cantelli result does not generalize directly without additional restrictions on $P$, but does almost sure convergence of moments hold for general $P$? If not, what kind of conditions are sufficient?
What about coordinate product moments? i.e. do we have $E_{\hat P_n}(X_{J}^kX_{J}^{k'}) \rightarrow E_{P}(X_{J}^kX_{J}^{k'})$ almost surely, where now $X_{J}$ represents the the sub-vector $(X^{(j)})_{j \in J}$ for some $J \subset \{1, \dots, \dim(X)\}$
EDIT: Does this simply follow from the Strong Law of Large numbers?
We should have
$$E_{P_n}(X^k) = \sum_{i=1}^n X_i^k \xrightarrow{a.s.} E_{P}(X^k)~,$$
since $\{X_i\}_{i=1}^n$ is an iid sequence from $P$ and $E(|X^{k}|)< \infty$. Is this correct? I think this also directly generalizes to the coordinate product moments, no?
Changed title to reflect this.
Best Answer
While the almost sure uniform convergence for empirical distribution functions is not clear.
The almost sure convergence for moments is pretty straightforward.
Let $\alpha$ be any multi-index in $\mathbb{N}^d$ such that $|\alpha|\le K$
You can see that:
$$\mathbb{E}_{\hat{P}_N}(X^{\alpha}) = \dfrac{1}{N}\left( \sum_{n=1}^N X^{\alpha}_n \right)$$
Clearly, $(X^{\alpha} ,n \ge 1)$ is a sequence of integrable real random variables.
Hence, by the strong law oflarge number, we have: $$ \dfrac{1}{N}\left( \sum_{n=1}^N X^{\alpha}_n \right) \longrightarrow \mathbb{E}_P(X^{\alpha}) \quad \text{a.e}$$
Hence, your conclusion. $\square$