Solved – Decision boundaries and Gaussian density functions

bayesianmachine learningself-study

This is for my hw, and if anyone can solve the first part of the question it will be great.
Here is the question:

Assume a two-class problem with equal a priori class probabilities and Gaussian class-conditional densities as follows:

$$p(x\mid w_1) = {\cal N}\left(\begin{bmatrix} 0 \\ 0 \end{bmatrix},\begin{bmatrix} a & c \\ c & b \end{bmatrix}\right)\quad\text{and}\quad p(x\mid w_2) =
{\cal N}\left(\begin{bmatrix} d \\ e \end{bmatrix},\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\right)$$

where $ab-c^2=1$.

  1. Find the equation of the decision boundary between these two classes
    in terms of the given parameters, after choosing a logarithmic
    discriminant function.
  2. Determine the constraints on the values of a, b, c, d and e, such
    that the resulting discriminant function results with a linear
    decision boundary.

Best Answer

I will use $(X,Y)$ for the observation. Given $w = w_1$, we have that the variances of $X$ and $Y$ are $a$ and $b$ respectively, while the covariance is $c$. Thus, the correlation coefficient $\rho = \frac{c}{\sqrt{ab}}$ and so $1-\rho^2 = 1 - \frac{c^2}{ab} = \frac{1}{ab}$. The two conditional (joint) densities are joint normal densities given by $$\begin{align} f_1(x,y) &= \frac{1}{2\pi \sqrt{ab(1-\rho^2)}}\exp\left[-\frac{1}{2(1-\rho^2)}\left(\frac{x^2}{a} - 2\rho\frac{xy}{\sqrt{ab}} + \frac{y^2}{b}\right)\right]\\ &= \frac{1}{2\pi}\exp\left[-\frac{ab}{2}\left(\frac{x^2}{a} - 2\frac{c}{\sqrt{ab}}\frac{xy}{\sqrt{ab}} + \frac{y^2}{b}\right)\right]\\ &= \frac{1}{2\pi}\exp\left[-\frac{1}{2}\left(bx^2 - 2cxy + ay^2\right)\right],\\ f_2(x,y) &= \frac{1}{2\pi}\exp\left[-\frac{1}{2}\left((x-d)^2+(y-e)^2\right)\right]\\ \end{align}$$ The decision boundary is the set of all $(x,y)$ for which $f_1(x,y) = f_2(x,y)$, and so the decision boundary is the conic section specified by $$bx^2 -2cxy +ay^2 - \left((x-d)^2 + (y-e)^2\right) = 0.$$ Edit: As pointed out in the comment by whuber, this conic section can be a hyperbola as well as an ellipse or parabola (including as a special case a straight line). The discriminant $c^2-(a-1)(b-1) = (a+b)-2$ (since we are given that $ab-c^2=1$) can be positive, negative, or zero depending on the variances $a$ and $b$. I suspect that this will be work out to be either an ellipse or a parabola (but not a hyperbola) depending on the parameters $a,b,c,d,e$, including, as a special case of parabola, a straight line. A specific example of a straight-line decision boundary is when $c = 0$ and $a = b = 1$ so that the only difference between the two conditional distributions is the means: $X$ and $Y$ are conditionally independent unit-variance normal random variables under either hypothesis. In this instance, the decision boundary is the perpendicular bisector of the straight line segment with end-points $(0,0)$ and $(d,e)$.

What puzzles me, though, is a hyperbola as a decision boundary since a hyperbola partitions the plane into three regions (two of which are congruent). The joint density surfaces are a (possibly) flattened bell and a circular bell, and so one of these bells subsumes the other in two non-contiguous regions of the plane: I just can't visualize it. Perhaps someone will create a nice illustration....

Related Question