For a positive definite kernel, there exists a Reproducing Kernel Hilbert Space (RKHS) whose reproducing kernel is the kernel function.
My question is, what characteristic of the kernel function decides the inner product of the RKHS.
For example, for the second order polynomial kernel $k(x, x') = (x^Tx')^2$ where $x, x' \in \mathbb{R}^2$, its feature map is $\phi_x = k(\cdot,x) = [x_1, \sqrt2x_1x_2, x_2]$ and $\langle\cdot,\cdot\rangle$ is computed in the same way as Euclidean space.
However, for Gaussian kernel the feature map is a mapping to infinite dimensional space, and the inner product will be computed as an integral.
For those well-known kernels we know the feature map, hence we can think about the inner product of the corresponding RKHS, but when we came up with a random positive definite kernel, what characteristics of the kernel function decide the inner product and the feature map?
Best Answer
This is a subject that I do not master at all, I just skimmed the Wikipedia. As far as I understand, for any positive definite kernel there is a corresponding reproducing kernel Hilbert space; this is the Moore-Aronszajn theorem and it is the one you implicitly mention in the question. But I think that you are right, when you write that this theorem sheds little light on the nature of that reproducing kernel Hilbert space.
Such information seems to be provided by the Mercer theorem, which is mentioned in the same Wikipedia page. One of its consequences is that the reproducing Hilbert space $H$ corresponding to the kernel $K$ is given by $$ H=\left\{ f\in L^2(X)\ :\ \sum_i \frac{|\langle f|\phi_i\rangle|^2}{\sigma_i}<\infty\right\}, $$ where $\sigma_i$ and $\phi_i$ are eigenvalues and eigenvectors of the integral operator $$\tag{1} f\mapsto \int_X K(\cdot, y)f(y)\, d\mu(y).$$
This is a vague answer and it sweeps some details under the rug; the most important one is the fact that the theorem of Mercer assumes that a measure space $(X, \mu)$ is given a priori (otherwise, (1) makes no sense). But I hope it is useful to you anyway.