The joint density function of $X$ and $Y$ is given by $f(x,y) = xe^{-x(y+1)}$, $x > 0, y > 0$.
(a) Find the conditional density of $X$, given $Y = y$, and that
of $Y$, given $X = x$.
(b) Find the density function of $Z = XY$.
MY SOLUTION
In the first place, we should determine the marginal distributions:
\begin{align*}
f_{X}(x) = \int_{0}^{\infty}f_{X,Y}(x,y)\mathrm{d}y = \int_{0}^{\infty}xe^{-x(y+1)}\mathrm{d}y = xe^{-x}
\end{align*}
where $x > 0$. Analogously, we have
\begin{align*}
f_{Y}(y) = \int_{0}^{\infty}f_{X,Y}(x,y)\mathrm{d}x = \int_{0}^{\infty}xe^{-x(y+1)}\mathrm{d}x = \frac{1}{(y+1)^{2}}
\end{align*}
where $y > 0 $.
(a) Based on the previous results, it comes
\begin{cases}
f_{X|Y}(x|y) = \displaystyle\frac{f_{X,Y}(x,y)}{f_{Y}(y)} = x(y+1)^{2}e^{-x(y+1)}\\\\
f_{Y|X}(y|x) = \displaystyle\frac{f_{X,Y}(x,y)}{f_{X}(x)} = e^{-xy}
\end{cases}
(b) Finally, we have
\begin{align*}
F_{Z}(z) & = \textbf{P}(XY \leq z) = \int_{0}^{\infty}\int_{0}^{z/x}f_{X,Y}(x,y)\mathrm{d}y\mathrm{d}x = \int_{0}^{\infty}\int_{0}^{z/x}xe^{-x(y+1)}\mathrm{d}y\mathrm{d}x\\\\
& = \int_{0}^{\infty}(e^{-x} – e^{-x-z})\mathrm{d}x = 1 – e^{-z} \Rightarrow f_{Z}(z) = \frac{\mathrm{d}}{\mathrm{d}z}(1-e^{-z}) = e^{-z}
\end{align*}
where $z > 0$.
Best Answer
The joint density $f(x,y)$ factors in the following ways:
$$f(x,y)=xe^{-xy}\mathbf1_{y>0}\,e^{-x}\mathbf1_{x>0}\tag{1}$$
$$f(x,y)=(1+y)^2xe^{-(1+y)x}\mathbf1_{x>0}\,\frac{1}{(1+y)^2}\mathbf1_{y>0}\tag{2}$$
From $(1)$ it follows that the conditional density of $Y\mid X$ is
$$f_{Y\mid X=x}(y)=xe^{-xy}\mathbf1_{y>0}$$
, and the marginal density of $X$ is $$f_X(x)=e^{-x}\mathbf1_{x>0}$$
From $(2)$, your answer for the conditional and marginal densities is correct.
So your answer for $f_X(x)$, though seemingly okay as it integrates to unity, is not correct. As a result, $f_{Y\mid X=x}(y)$ (the variable is $y$ here because it is a function of $y$ for a given $x$, not $(x,y)$ as you have written) is also out of line. The rest looks okay.