I asked a similar question yesterday,
but I organized my question here a little and further asked my second question.
Suppose $X$ is a continuous random variable with the pdf $f_x$, and $Y=g(X)$.
If $g$ is a bijection, then via the change of variable method,
the pdf of $Y$ is $f_y = f_x/|J|$, where $|J|$ is the Jacobian of $g$.
Therefore,
$$h(Y)=h(X)+\int f_x \log |J| dx$$.
From Wikipedia, if $dim(X)=dim(Y)$, there exists an upper bound on $h(Y)$ for a general map $g$:
$$h(Y)\leq h(X)+\int f_x \log |J| dx \tag{1}$$
My first question is how to derive the inequality in Eq.(1).
Second, is there a general approach to derive $h(Y)$ or an upper bound on it if $dim(X)\neq dim(Y)$.
I'd appreciate any help and suggestions!
Best Answer
If $g$ maps between two sets that have the same dimension, the inequality may be strict. The inequality appears in Papoulis' book on Probability, Random Variables, and Stochastic Processes - an exact value, albeit depending on the function $g$ in a complicated way, can be found in Corollary 1 in this paper:
$$h(Y) = h(X)+\int f_x\log |J|dx - H(X|Y). $$
If $g$ maps between sets of different dimension, then the Jacobian matrix is not square and computing its determinant does not make sense. I'm not sure if there is an easy way to derive the differential entropy of $Y$ from the differential entropy of $X$ in this case...