Consider a binary classification with one binary output $y$ and two binary features $x_1$ and $x_2$. The Naive Bayes classifier assumes the following distribution for a pair:
$$
p(y,x_1, x_2) = p(x_1|y) p(x_2|y) p(y)
$$
Let the values of the probabilities be
\begin{align*}
p(y = 0) &= 0.5\\
p(x_1 = 1|y = 0) &= 0.9 \\ p(x_2 = 1|y = 0) &= 0.5
\end{align*}
\begin{align*}
p(y = 1) &= 0.5\\
p(x_1 = 1|y = 1) &= 0.2 \\ p(x_2 = 1|y = 1) &= 0.5
\end{align*}
Assumption: the features $x_i$ are assumed to be conditionally independent given the target $y$. For example
$$
p(x_1|y,x_2) = p(x_1|y)
$$
Compute the posterior values:
$$
p(y = 1|x_1 = 1, x_2 = 1) \quad \text{and} \quad p(y = 0|x_1 = 1, x_2 = 1)
$$
How would you classify an exmple with $x_1 = 1$ and $x_2 = 1$
\newline
\newline
\textbf{solution}: Using Bayes rule we have:
\begin{align*}
p(y = 1|x_1 = 1, x_2 = 1) &= \frac{p(x_1 = 1, x_2 = 1|y = 1)p(y = 1)}{p(x_1 = 1, x_2 = 1)}\\
&\propto p(x_1 = 1, x_2 = 1|y = 1)p(y = 1)\\
&= 0.2\cdot 0.5 \cdot 0.5\\
&= 0.05\\
&\text{}\\
&\text{and}\\
&\text{}\\
p(y = 0|x_1 = 1, x_2 = 1) &= \frac{p(x_1 = 1, x_2 = 1|y = 0)p(y = 1)}{p(x_1 = 1, x_2 = 1)}\\
&\propto p(x_1 = 1, x_2 = 1|y = 0)p(y = 0)\\
&= 0.9\cdot 0.5 \cdot 0.5\\
&= 0.225
\end{align*}
Therefore we get
$$
p(y = 1|x_1 = 1, x_2 = 1) = 0.2/1.1 \approx 0.18
\quad \textbf{(1)}$$
The example should be classified as $y = 0$
Here is what I don't understand:
how do you get both $0.2$ and $1.1$ in (1) with the given information to be able to compute the result of (1)?
Best Answer
I do not know where those numbers come from but here is how I would do the computation. $$ P(x_1=1,x_2=1|y=1)=P(x_1=1|y=1)P(x_2=1|y=1)=0.2\cdot 0.5=0.1 $$ (by conditional independence). Same with $y=0$: $$ P(x_1=1,x_2=1|y=0)=P(x_1=1|y=0)P(x_2=1|y=0)=0.9\cdot 0.5=0.45 $$ Next,to be able to use Bayes rule, we need the total probability $$ P(x_1=1,x_2=1)=P(x_1=1,x_2=1|y=1)P(y=1)+P(x_1=1,x_2=1|y=0)P(y=0)=0.1\cdot0.5+0.45\cdot 0.5=0.275 $$ Using Bayes rule now, $$ P(y=1|x_1=1,x_2=1)=\frac{P(x_1=1,x_2=1|y=1)P(y=1)}{P(x_1=1,x_2=1)}=\frac{0.1\cdot 0.5}{0.275}=0.18. $$