In exploring the dense subsets of $C\left(I_n\right)$, I've been particularly focused on the application of a foundational result:
$\textbf{Stone-Weierstrass Theorem}$: This theorem posits that for any compact set in $\mathbb{R}^n$ and an algebra $\mathcal{A}$ of continuous real-valued functions on this set, if $\mathcal{A}$ separates the points and includes constant functions, then $\mathcal{A}$ is dense in $C(K)$.
Our interest lies in the functions of the form:
$$
G(x)=\sum_{k=1}^M \beta_k \prod_{j=1}^{N_k} \varphi\left(w_{j k}^T x+\theta_{j k}\right) (1)
$$
where $w_{j k} \in \mathbb{R}^n, \beta_j, \theta_{j k} \in \mathbb{R}$, and $x \in I_n$, under the $\textbf{valid property}$ that $\textbf{they form a dense subset in $C\left(I_n\right)$}$ using a nonconstant activation function $\varphi$.
My approach for proving it depends on identifying the set $\mathcal{U}$, given by the finite sums of products of the specified type, as an algebra of real continuous functions on $I_n$ that satisfies the conditions of the theorem: separation of points and inclusion of constant functions.
Despite understanding the theorem's prerequisites, I'm seeking a more rigorous mathematical justification to firmly establish $\mathcal{U}$ 's adherence to these conditions, thus proving its density in $C\left(I_n\right)$. I would greatly appreciate any guidance or detailed mathematical rigor that could illuminate this process."
Best Answer
If I understand correctly, the question is how to prove that the family $$\tag{*} \left\{\sum_{k=1}^M \beta_k \prod_{j=1}^{N_k} \varphi(w_{jk}^T x+\theta_{jk})\ :\ \beta_k, \theta_{jk}\in\mathbb R, w_{jk}\in\mathbb R^n\right\}$$ is an algebra. That is rather obvious for the following reason. Let me rename $$\tag{$\dagger$} f_{w_{jk}, \theta_{jk}}(x):=\varphi(w_{jk}^Tx+\theta_{jk}), $$ and to simplify notation, let me index it by just one parameter: $(f_\alpha)_{\alpha\in A}$. This set $A$ consists of all $w_{jk}$ and all $\theta_{jk}$.
What is the smallest algebra containing all $f_\alpha$? For a set $\mathcal U$ to be an algebra, we need to be able to take linear combinations and pointwise products. So all elements of the form $$\tag{**} \sum_{k=1}^M \beta_k \prod_{j=1}^{N_k} f_{\alpha_{jk}}$$ must be in $\mathcal U$. Conversely, it is clear that the set consisting precisely of these elements is closed under linear combinations and pointwise products. We conclude that $$ \mathcal{U}=\left\{ \sum_{k=1}^M \beta_k \prod_{j=1}^{N_k} f_{\alpha_{jk}}\ :\ M, N_k\in\mathbb N, \beta_k\in\mathbb R\right\}$$ is the smallest algebra containing $(f_\alpha)_{\alpha \in A}$. Which means that our original (*) is the smallest algebra containing the functions of the form $(\dagger)$.
I hope this explains where does this set $(*)$ come from. Of course, it remains to prove that $(*)$ contains constants and separates points. The first condition is obvious, and correspond to $N_k=0$. The second depends on some specific property of $\varphi$. For example, the constant function $\varphi(x)=0$ violates it. I suppose that having a nonconstant $\varphi$ is enough to separate points, but that is for another question.