Let $Y$ be a continuous random variable with PDF:
$$f_{Y}(y)=\begin{cases}
3 y^{2} &, & 0<y<1 \\
0 &,& \textrm { else }
\end{cases}$$
Also $X$ is a discrete random variable such that:
$$P\left(X=x \mid Y=y\right)~=~y(1-y)^{x-1}\:,x \in \mathbb{N}$$
Find the joint distribution of $X$ and $Y$.
I am confused in this question.
As I know:
$$P\left(X=x \mid Y=y\right)~=~\frac{P\left(X=x, Y=y\right)}{P(Y=y)}$$
But how to find $P(Y=y)$?
Best Answer
The joint distribution of $X$ and $Y$ is neither jointly discrete nor jointly continuous, but rather a mixed one. To properly describe such distribution, you will need a new type of "density".
Before getting to the problem, let me give a brief review of PMF (probability mass function) and PDF (probability density function).
Suppose there exists a non-negative function $p(x)$ that satisfies the following property: $$ \mathbf{P}(X \in A) = \sum_{x \in A} p(x) $$ for any "reasonable" subset $A$ of $\mathbb{R}$. We remind that such $p$ need not always exists. If this is the case, however, then $X$ is called discrete, and such $p$ is called the PMF of $X$.
Suppose there exists a non-negative function $f(x)$ that satisfies the following property: $$ \mathbf{P}(X \in A) = \int_A f(x) \, \mathrm{d}x $$ for any "reasonable" subset $A$ of $\mathbb{R}$. Again, such $f$ need not always exists. If it does, however, then $X$ is called continuous, and such $f$ is called the PDF of $X$.
Joint PMFs and joint PDFs of two RVs $X$ and $Y$ are defined likewise, adapting the above definitions. For example, $(X, Y)$ is said to be jointly continuous if there exists a non-negative function $f(x, y)$ such that $$ \mathbf{P}((X, Y) \in A) = \int_{A} f(x, y) \, \mathrm{d}x $$ for any "reasonable" subset $A$ of $\mathbb{R}^2$. If this is the case, then $f$ is called the joint PDF of $X$.
Now let us come back to your question. In your setting, $Y$ is continuous and $X$ is discrete with a specified conditional distribution given $Y$. So, using the law of total probability, we get
\begin{align*} \mathbf{P}(X \in A, Y \in B) &= \int_B \mathbf{P}(X \in A \mid Y = y) f_Y(y) \, \mathrm{d}y \\ &= \int_B \left( \sum_{x \in A} \mathbf{P}(X =x \mid Y = y) \right) f_Y(y) \, \mathrm{d}y \\ &= \sum_{x \in A} \int_B \mathbf{P}(X =x \mid Y = y) f_Y(y) \, \mathrm{d}y. \end{align*}
So, the joint distribution of $X$ and $Y$ can be described by the function $q_{X,Y}$ below, which is kind of an intermediate of PMF and PDF:
\begin{align*} q_{X,Y}(x, y) &= \mathbf{P}(X = x \mid Y = y) f_Y(y) \\[0.25em] &= \begin{cases} 3y^3(1-y)^{x-1}, & x \in \mathbb{N}, \ 0 < y < 1, \\ 0, & \text{otherwise}, \end{cases} \end{align*}
and the probabilities for $(X, Y)$ can be recovered by the formula
$$ \mathbf{P}((X, Y) \in A \times B) = \sum_{x \in A} \int_B q_{X,Y}(x,y) \, \mathrm{d}y. $$
Addendum. I am not sure if there is a commonly used term for something like $q_{X,Y}$. However, if you have a chance to revisit probability theory in measure-theoretic viewpoint in a future, you will recognize both PMFs and PDFs as the Radon–Nikodym derivatives with respective to the counting measure and Lebesgue measure, respectively. This way, the theory of PMF/PDF can be unified into a single, general setting. Then $q_{X,Y}$ in this problem is nothing but the Radon–Nikodym derivative of the distribution of $(X, Y)$ with respect to the product of the counting and Lebesgue measure.