I am trying to understand change of variables for a random variable and how it changes the pdf and the normalizing constant.
Let $\mathbf{w}$ be $N$-dimensional normal variable and let $\mathbf{S}$ be covariance matrix. I can diagonalize that matrix: $\mathbf{S} =\mathbf{O \Sigma O^T} $ and change original variable: $\mathbf{ x = O^T w} $. Now I have diagonal covariance matrix that I call $\mathbf{\Sigma_x}$, and probability density function for $x$ is
$$
P_x(\mathbf{x}) = \frac{\exp\{-\frac12 \mathbf{x}^T \mathbf{\Sigma^{-1}_x} \mathbf{x}\}}{\int d\mathbf{x} \exp\{-\frac12 \mathbf{x}^T \mathbf{\Sigma^{-1}_x} \mathbf{x}\}} = (2\pi)^{-N/2} (\det \mathbf{\Sigma_x})^{-\frac12} \exp\{-\frac12 \mathbf{x}^T \mathbf{\Sigma^{-1}_x} \mathbf{x}\}
$$
Now I make the change of variable to $\mathbf{y} = a \mathbf{x}$ and calculate Jacobian. Function for change of variable is $g(x) = a x$ and $g^{-1}(y) = a^{-1} y$ so according to the changing variable method new pdf must be
$$
P_y(\mathbf{y}) = P_x(g^{-1}(y)) \begin{vmatrix} \frac{d g^{-1}(y)}{dy}\end{vmatrix}
$$
where $\begin{vmatrix} \frac{d g^{-1}(y)}{dy}\end{vmatrix}$ is Jacobian. For case of rescaling of the variable this is just $\begin{vmatrix} a^{-1}\end{vmatrix}$. So pdf for $y$ is
$$
P_y(\mathbf{y}) = (2\pi)^{-N/2} [\det (a\ \mathbf{\Sigma_x} \ a)]^{-\frac12} \exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\} \\= a^{-N} (2\pi)^{-N/2} [\det (\mathbf{\Sigma_x})]^{-\frac12} \exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\}
$$
Now I try to calculate this result by making substitution $\mathbf{x}=a^{-1}\mathbf{y}$ in $P_x(\mathbf{x})$, but I get different answer for the integral in normalizing constant.
$$
P_y(\mathbf{y}) = \frac{\exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\}}{\int d\mathbf{y} a^{-N} \exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\}}
$$
The denominator above is normalizing constant and is a gauss integral
$$
\begin{align}
\mathcal{N}&=a^{-N} \int d\mathbf{y}\exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\} = a^{-N} (2\pi)^{N/2} [\det a^2 \mathbf{\Sigma_x}]^{1/2} \\&= a^{-N}a^{N} (2\pi)^{N/2} [\det \mathbf{\Sigma_x}]^{1/2} = (2\pi)^{N/2} [\det \mathbf{\Sigma_x}]^{1/2}
\end{align}
$$
So the $a$ is cancelling out here and new pdf is
$$
P_y(\mathbf{y}) = (2\pi)^{-N/2} [\det (\mathbf{\Sigma_x})]^{-\frac12} \exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\}
$$
I don't understand why this is different from answer I got when using Jacobian formula? I must have made mistakes in the integral. Thank you.
Best Answer
Your expression for $P_y(\mathbf y)$ should be $$ P_y(\mathbf{y}) = \frac{a^{-N} \exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\}}{\int d\mathbf{y} \ a^{-N} \exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\}}. $$ i.e. the factor of $a^{-N}$ should appear in the numerator too.
Remember, the whole point of the normalization factor $\int d\mathbf{y} \ a^{-N} \exp\{-\frac12 \mathbf{y}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{y}\}$ in the denominator is to ensure that $\int d\mathbf y P_y(\mathbf y) = 1$.
Staring at my expression, it's clear that $\int d\mathbf y P_y(\mathbf y) = 1$.
But with your expression in your question, it's clear that you'd get $\int d\mathbf y P_y(\mathbf y) = a^N$, which isn't right.
To address point raised in comments...
It's legitimate to substitute variables inside the integral. There, we're substituting one dummy variable for another dummy variable, which is fine. For example, it's legitimate to write $$ P_x(\mathbf{x}) = \frac{\exp\{-\frac12 \mathbf{x}^T \mathbf{\Sigma^{-1}_x} \mathbf{x}\}}{\int d\mathbf{x'} \ \exp\{-\frac12 \mathbf{x'}^T \mathbf{\Sigma^{-1}_x} \mathbf{x'}\}} = \frac{\exp\{-\frac12 \mathbf{x}^T \mathbf{\Sigma^{-1}_x} \mathbf{x}\}}{\int d\mathbf{u} \ a^{-N} \exp\{-\frac12 \mathbf{u}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{u}\}}. $$ where we've performed a variable substitution $\mathbf {x'} = a^{-1} \mathbf u$ inside the integral.
But to derive $P_y(\mathbf {y})$ from $P_x(\mathbf {x})$, you must use the Jacobian. That is the only legitimate way to get the expression for $P_y(\mathbf {y})$. $$ P_y(\mathbf y) = P_x(\mathbf x) \left| \frac{\partial \mathbf x}{\partial \mathbf y}\right| = a^{-N} P_x(\mathbf x) = \frac{a^{-N}\exp\{-\frac12 \mathbf{x}^T \mathbf{\Sigma^{-1}_x} \mathbf{x}\}}{\int d\mathbf{u} \ a^{-N} \exp\{-\frac12 \mathbf{u}^T \frac{\mathbf{\Sigma^{-1}_x}}{a^2} \mathbf{u}\}}$$
NB The correct way to change variables for the pdf is to use the Jacobian formula, $$ P_y(\mathbf y) = P_x(\mathbf x) \left| \frac{\partial \mathbf x}{\partial \mathbf y}\right| = a^{-N} P_x(\mathbf x) .$$
It is incorrect to say that $$ P_y(\mathbf y) = P_x(\mathbf x) $$
Why is this?
Consider a infinitesimal volume element $\delta \mathbf x$ in $\mathbf x$-space, which maps to an infinitesimal volume element $\delta \mathbf y$ in $\mathbf y$-space. What is the probability that the sample lies in this infinitesimal volume element?
We can calculate this probability in two ways - using the $\mathbf x$-space representation or using the $\mathbf y$-space representation. The two answers must be equal. In other words,
$$ P_x(\mathbf x) \delta \mathbf x = P_y(\mathbf y) \delta \mathbf y.$$
But the volumes of the volume elements are different! They are related by $$ \delta \mathbf x = \left| \frac{\partial \mathbf x}{\partial \mathbf y}\right| \times \delta \mathbf y.$$
Hence it follows that $$ P_x(\mathbf x) \left| \frac{\partial \mathbf x}{\partial \mathbf y}\right| = P_y(\mathbf y).$$