I have problems calculating the posterior mode and posterior mean of a posterior density. For example the following exercise:
The random variable $X$ has distribution (for $\theta \in[0,1]$):
$$P(X = 1) = \theta, P(X=2) = 1-\theta.$$
The random variables $X_1, X_2, X_3$ are i.i.d. with the same distribution as $X$. The following observations have been made: $x_1 = 1, x_2 = 2, x_3 =2.$
a) Choose a uniform prior density for $\Theta$ and give the posterior density of $\Theta$ based on the data.
b) Give the posterior mean and mode estimate for $\theta$.
c) Do the same as b), only with the following prior density: $f_{\Theta}(\theta) \propto \theta^3(1-\theta)^51_{[0,1]}(\theta)$.
My thoughts:
a)
$f_{\Theta}(\theta) = 1$ for $\theta \in [0,1]$
$f_{X\mid \Theta}(x_1,x_2,x_3 \mid \theta) = P(X_1 = 1, X_2 = 2, X_3 = 3\mid \theta) = \theta(1-\theta)^2 $
So $f_{\Theta \mid X}(\theta \mid x_1, x_2, x_3) \propto f_{\Theta}(\theta)f_{X\mid \Theta}(x_1,x_2,x_3 \mid \theta) = \theta(1-\theta)^2$
b)
Posterior mode:
$logf_{\Theta \mid X} = log(\theta(1-\theta)^2) = log(\theta) + 2log(1-\theta)$
$\frac{d}{d\theta}logf_{\Theta \mid X} = \frac{1}{\theta}-\frac{2}{1-\theta} = 0 \Longleftrightarrow \theta = \frac{1}{3}$
This answer is correct according to my solutions book.
Posterior mean:
I think I have to calculate $\int_0^1\theta f_{X\mid \Theta}(x_1,x_2,x_3 \mid \theta)d\theta$.
So $E(\Theta) = \int_0^1 \theta^2(1-\theta)^2d\theta = .. = \frac{1}{30}$
The answer should be $\frac{2}{5}$ though…
c)
$f_{\Theta \mid X}(\theta \mid x_1, x_2, x_3) \propto f_{\Theta}(\theta)f_{X\mid \Theta}(x_1,x_2,x_3 \mid \theta) = \theta^3(1-\theta)^5\theta(1-\theta)^2 = \theta^4 (1-\theta)^7$
When I calculate the mode in the same way as b) I get $\frac{4}{11}$, though it should be $\frac{4}{13}$. As for the mean, I don't even know how to solve that integral…
Sorry for the long text. I would appreciate it a lot if someone could explain what I'm doing wrong and how it's supposed to be done.
Best Answer
So the disagreement between your answer and the solutions for the first choice of prior follow from the fact that you have, correctly, calculated the posterior only up to the normalising constant, so we need to find that $$ \int_0^1 \theta(1-\theta)^2d\theta=\frac{1}{12} $$ so that we now have $$ f_{\Theta}(\theta)=12\theta(1-\theta)^2, \qquad\theta \in [0,1]. $$ So for part $b)$ we now have, multiplying your result by the proper normalising constant, that the result is $12 \times \frac{1}{30} = \frac{2}{5}$.
Repeating for part (c) for the mode when you don't need to find the normalising constant it should just be a case of $$ 0 = \frac{\partial \log f_{\Theta|X}(\theta)}{\partial \theta} = \frac{4}{\theta}-\frac{7}{1-\theta} \implies \theta = \frac{4}{11}. $$ which agrees with your work.
To find the mean it helps to identify the posterior with a Beta distribution, that is $$ \begin{align*} \int_0^{1}\theta^{4}(1-\theta)^{7}d\theta&=B(5,8) \underbrace{\int_0^1 \frac{\theta^4(1-\theta)^7}{B(5,8)}d\theta}_{=1} \end{align*} $$ where $B(\alpha, \beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}$ and so the appropriate form of the posterior is $$ \begin{align*} f_{\Theta|X}(\theta) =\frac{1}{B(5,8)}\theta^4(1-\theta)^7, \qquad \mathbf{E}_{\Theta | X}\left[ \Theta \right] = \frac{5}{5 + 8} = \frac{5}{13}. \end{align*} $$ So check that with your solutions and see if I have made any obvious mistakes? If you haven't already seen the Beta function then it would have been possible, but tedious, to get the result by expanding $(1-\theta)^7$ using the Binomial theorem.