Suppose I have a strictly positive parameter $\sigma$ and I need to estimate it using the random walk Metropolis-Hasting algorithm.
I know that I can do a parameter transform, i.e., $\beta=log(\sigma)$ and estimate $\beta$ instead. It is more convenient for me to assume that the prior for $\beta$ follows a normal distribution, e.g., $\beta ~ N(\alpha,\lambda)$. ($\alpha$ and $\lambda$ are actually parameters from other parts of my model). Then the log likelihood for the prior is,
$logprior= N(\beta,\alpha,\lambda,loglike=true)$
My confusion is whether I should add the Jacobian term or not. Suppose I still need to add the Jacobian term, should I just make the log of the prior look like this?
$logprior= N(\beta,\alpha,\lambda,loglike=true) + \beta$
Is $\beta$ the correct Jacobian term? I deduced it from this answer, but I don't know if I understand it correctly.
Best Answer
The Jacobian is used when you convert back and forth between a distribution for $\sigma$ and a distribution for $\beta$. This is true of the conversion whether you are talking about the prior or posterior distribution. You don't need to add it when you are dealing with a posterior computation for a given parameter that is not changing. So, if you are making an inference about $\beta$, you can derive its posterior distribution without any use of the Jacobian, as:
$$\pi(\beta|\mathbf{x}) \propto L_\mathbf{x}(\beta) \cdot \pi(\beta).$$
Similarly, if you are making an inference about $\sigma$, you can derive its posterior distribution without any use of the Jacobian, as:
$$\pi'(\sigma|\mathbf{x}) \propto L_\mathbf{x}'(\sigma) \cdot \pi'(\sigma).$$
(I'm using the primes here to reflect the fact that these sets of functions are different from each other, since they are for different paramters.) Since $\beta = \log \sigma$ these functions are related by:
$$\begin{align} L_\mathbf{x}'(\sigma) &\overset{\sigma}{\propto} L_\mathbf{x}(\log \sigma), \\[12pt] \pi'(\sigma) &\overset{\sigma}{\propto} \pi(\log \sigma) \cdot \frac{1}{\sigma}, \\[12pt] \pi'(\sigma|\mathbf{x}) &\overset{\sigma}{\propto} \pi'(\sigma|\mathbf{x}) \cdot \frac{1}{\sigma}, \\[12pt] L_\mathbf{x}(\beta) &\overset{\beta}{\propto} L_\mathbf{x}'(e^\beta), \\[12pt] \pi(\beta) &\overset{\beta}{\propto} \pi'(\exp \beta) \cdot e^\beta, \\[12pt] \pi(\beta|\mathbf{x}) &\overset{\beta}{\propto} \pi'(e^\beta|\mathbf{x}) \cdot e^\beta. \\[12pt] \end{align}$$
As you can see, the Jacobian enters into these relationships when we convert either the prior or posterior distribution for one parameter to the other.