I see some posts that are related to this one, e.g.
Borel Measures: Atoms (Summary)
I have a sort of particular question: I have one professor saying the following is true, while another says it's false. I think it's true. Neither have a justification haha
Suppose we take a Borel probability measure on a compact Euclidean space. Then can we approximate it by uniform atomic measures in the weak*-topology of Borel probability measures on X?
Specifically, if $\mu$ is a Borel probability measure on the compact Euclidean $X$, then does there exist a sequence of (not necessarily distinct!) points $( x_i )$ in X so that $$\frac{1}{n} \sum_{i = 1}^n \delta_{x_i} \xrightarrow{weak^*} \mu$$
where $\delta_{x_i}$ is the Dirac measure at $x_i$? The reason I think it's true is that we know this measure has a density by the Lebesgue Density Theorem.
Best Answer
Here's one way to do it without getting your hands too dirty. It's not constructive.
Let ${\cal A}$ be the union of a countable dense set in $C_0({\mathbb R}^n)$, and indicators of all balls of integer radius centered at the origin (not necessary, but helpful later).
Let $\{X_1,X_2,\dots \}$ be an IID $\mu$-distributed sequence.
Let $\mu_n = \frac{1}{n} \sum_{j\le n} \delta_{X_j}.$
Then $\mu_n$ is a random probability measure. Observe that for any bounded and measurable function,
$$\int f d\mu_n= \frac {1}{n} \sum_{j\le n } f(X_j).$$
Then by LLN
$$ \lim_{n\to\infty} \int f d\mu_n = \int f d\mu ,~\forall f \in {\cal A}\quad (*),$$
a.s. (here is where we use the countability of ${\cal A}$).
Let $\{\bar \mu_n\}$ be a realization of $\{\mu_n\}$ (or equivalently, $\{X_1,\dots\}$) for which $(*)$ holds. It remains to show that $\{\bar \mu_n\}$ converges weakly.
Let $f\in C_b({\mathbb R}^d)$, by a standard procedure, for every $M$, there exists $f_M\in C_0({\mathbb R}^d)$ such that $f_M$ coincides with $f$ on $\{x:|x|<M\}$ and $|f_M |\le |f|$. Fix $\epsilon>0$ and choose an integer $M$ such that $\mu(\{x:|x|>M\})<\epsilon$
There exists a continuous $g_M\in {\cal A}$ such that $\|f_M-g_M\|<\epsilon$.
We then have
$$ \int f d \bar \mu_n = \int g_M d \bar \mu_n + \int (f_M -g_M) d \bar \mu_n + \int (f- f_M) d \bar \mu_n.$$
The first integral on the RHS converges to $\int g_M d\mu$ by construction. The second is bounded in absolute value by $\epsilon$. The third is bounded by $2\|f\|\mu_n (\{x:|x|>M\})$, which by assumption (recall the indicator functions in ${\cal A}$) converges to $2\|f\|\mu(\{x:|x|>M\})=2\|f\|\epsilon$.
Therefore
$$ \limsup_{n\to\infty} |\int f d\bar \mu_n - \int g_M d\mu| \le \epsilon (1+2\|f\|).$$
But $$\begin{align*} |\int g_M d\mu - \int f d \mu| &\le \int |g_M -f_M| d \mu + \int |f_M -f| d\mu\\ & \le \epsilon + 2\|f\|\mu (\{x:|x|>M\})\le \epsilon (1+2\|f\|)\end{align*}.$$
The result follows.