I am trying to find a solution to the decision boundary in QDA. The question was already asked and answered for linear discriminant analysis (LDA), and the solution provided by amoeba to compute this using the "standard Gaussian way" worked well. However, I am applying the same technique for a 2 class, 2 feature QDA and am having trouble. Would someone be able to check my work and let me know if this approach is correct?
I start-off with the discriminant equation,
$\delta_c = -\frac{1}{2}\log{|\mathbf{\Sigma_c}|}-\frac{1}{2}{\mathbf{(x-\mu_c)'\Sigma^{-1}_l(x – \mu_c)}}+\log{p_c}$
Where $\delta_c$ is the discriminant score for some observation $[x,y]$ belonging to class $c$ which could be 0 or 1 in this problem.
The decision boundary between $c=0$ and $c=1$ is the set of poins $\{\boldsymbol{\vec{x}}, \boldsymbol{\vec{y}}\}$ that satisfy the criteria $\delta_0$ equal to $\delta_1$. To find this set of points, I start with:
$$\delta_0=\delta_1$$
Substitute the discriminant equation for both $\delta_0$ and $\delta_1$
$$-\frac{1}{2}\log{|\mathbf{\Sigma_0}|}-\frac{1}{2}{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x – \mu_0)}}+\log{p_0} = -\frac{1}{2}\log{|\mathbf{\Sigma_1}|}-\frac{1}{2}{\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x – \mu_1)}}+\log{p_1}$$
After some algebra obtain:
$${\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x – \mu_1)}}-{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x – \mu_0)}}=C$$
Where $C = \log{|\mathbf{\Sigma_0}|}-\log{|\mathbf{\Sigma_1}|}+2\log{p_1}-2\log{p_0}$
In the next part I am attempting to solve for $\boldsymbol{\vec{y}}$ given some input $\boldsymbol{\vec{x}}$.
to find $\boldsymbol{\vec{y}}$ I re-wrote the previous equation as the following:
$$\begin{bmatrix} x_1 & y_1 \\ \end{bmatrix} \begin{bmatrix} a & b \\ c & d \\ \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \\ \end{bmatrix} – \begin{bmatrix} x_0 & y_0 \\ \end{bmatrix} \begin{bmatrix} p & q \\ r & s \\ \end{bmatrix} \begin{bmatrix} x_0 \\ y_0 \\ \end{bmatrix} = C$$
Where (for now):
$$x_0 = x-\mu_{00}$$
$$y_0 = y-\mu_{01}$$
$$x_1 = x-\mu_{10}$$
$$y_1 = y-\mu_{11}$$
After a lot of algebra I eventually found the solution to be:
$$(d-s)y^2+(-2d\mu_{11}+2s\mu_{01}+bx-b\mu_{10}+cx-c\mu_{10}-qx+q\mu_{00}-rx+r\mu_{00})y = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2+b\mu_{11}x+c\mu_{11}x-q\mu_{01}x-r\mu_{01}x+d\mu_{11}^2-s\mu_{01}^2-b\mu_{10}\mu_{11}-c\mu_{10}\mu_{11}+q\mu_{01}\mu_{00}+r\mu_{01}\mu_{00}$$
Apply the quadratic formula to solve for $y$:
$u = d-s$
$v = -2d\mu_{11}+2s\mu_{01}+bx-b\mu_{10}+cx-c\mu_{10}-qx+q\mu_{00}-rx+r\mu_{00}$
$w = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2+b\mu_{11}x+c\mu_{11}x-q\mu_{01}x-r\mu_{01}x+d\mu_{11}^2-s\mu_{01}^2-b\mu_{10}\mu_{11}-c\mu_{10}\mu_{11}+q\mu_{01}\mu_{00}+r\mu_{01}\mu_{00}$
$$y = \frac{-v\pm\sqrt{v^2-4uw}}{2u}$$
When I solve for $\boldsymbol{\vec{y}}$ I obtain this decision boundary:
To check my work I calculated the difference of two bi-variate Gaussians, based on the same data, across the entire plot area to obtain:
There are errors in my work, and I was wondering is the group would be kind enough to review this question and provide some input on what went wrong?
Pardon my notation, I am relatively new to this math.
(Note: I previously posted this question on the Cross-Validated subgroup, but have not received any responses.)
Best Answer
The discriminant equation: $$ (x-\mu_1)'\Sigma^{-1}_1(x-\mu_1) - (x-\mu_2)'\Sigma^{-1}_2(x-\mu_2) = C $$ should be reduced to an equation of conic section, i.e., in the form of: $$ Ax^2+Bxy+Cy^2=D $$ instead of using uni-variate quadratic formula: $$ x=\frac{-b\pm \sqrt{b^2-4ac}}{2a} $$ which will give incorrectly the two curve lines (green and blue lines).
Green line: $$ x=\frac{-b+ \sqrt{b^2-4ac}}{2a} $$ Blue line: $$ x=\frac{-b- \sqrt{b^2-4ac}}{2a} $$