I am trying to find a solution to the decision boundary in QDA. The question was already asked and answered for LDA, and the solution provided by amoeba to compute this using the "standard Gaussian way" worked well. However, I am applying the same technique for a 2 class, 2 feature QDA and am having trouble. Would someone be able to check my work and let me know if this approach is correct?
I start-off with the discriminant equation,
$\delta_l = -\frac{1}{2}\log{|\mathbf{\Sigma_i}|}-\frac{1}{2}{\mathbf{(x-\mu_i)'\Sigma^{-1}_i(x – \mu_i)}}+\log{p_i}$
Where $\delta_l$ is the discriminant score for some observation $\mathbf{x}$ belonging to class $l$ which could be 0 or 1 in this 2 class problem.
The decision boundary between $l=0$ and $l=1$ is the vector $\boldsymbol{\vec{x}}$ that satisfies the criteria $\delta_0$ equal to $\delta_1$. I approach this in the following way:
$$\delta_0=\delta_1$$
Substitute the discriminant equation for both $\delta_0$ and $\delta_1$
$$-\frac{1}{2}\log{|\mathbf{\Sigma_0}|}-\frac{1}{2}{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x – \mu_0)}}+\log{p_0} = -\frac{1}{2}\log{|\mathbf{\Sigma_1}|}-\frac{1}{2}{\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x – \mu_1)}}+\log{p_1}$$
Algebraic manipulations…
$$\frac{1}{2}{\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x – \mu_1)}}-\frac{1}{2}{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x – \mu_0)}} = \frac{1}{2}\log{|\mathbf{\Sigma_0}|}-\frac{1}{2}\log{|\mathbf{\Sigma_1}|}+\log{p_1}-\log{p_0}$$
$$\frac{1}{2}({\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x – \mu_1)}}-{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x – \mu_0)}}) = \frac{1}{2}\log{|\mathbf{\Sigma_0}|}-\frac{1}{2}\log{|\mathbf{\Sigma_1}|}+\log{p_1}-\log{p_0}$$
$${\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x – \mu_1)}}-{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x – \mu_0)}} = \log{|\mathbf{\Sigma_0}|}-\log{|\mathbf{\Sigma_1}|}+2\log{p_1}-2\log{p_0}$$
The right side of the above equation is a constant that we can assign to the variable $C$ as follows:
$C = \log{|\mathbf{\Sigma_0}|}-\log{|\mathbf{\Sigma_1}|}+2\log{p_1}-2\log{p_0}$
at this point we have:
$${\mathbf{(x-\mu_1)'\Sigma^{-1}_1(x – \mu_1)}}-{\mathbf{(x-\mu_0)'\Sigma^{-1}_0(x – \mu_0)}}=C$$
Next I am trying to solve for the value of y (e.g., feature 2) given some input value of x (feature 1).
To simplify the manipulations, I have temporarily assigned the following variables as:
$$x_0 = x-\mu_{00}$$
$$y_0 = y-\mu_{01}$$
$$x_1 = x-\mu_{10}$$
$$y_1 = y-\mu_{11}$$
$$\begin{bmatrix} x_1 & y_1 \\ \end{bmatrix} \begin{bmatrix} a & b \\ c & d \\ \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \\ \end{bmatrix} – \begin{bmatrix} x_0 & y_0 \\ \end{bmatrix} \begin{bmatrix} p & q \\ r & s \\ \end{bmatrix} \begin{bmatrix} x_0 \\ y_0 \\ \end{bmatrix} = C$$
$$x_1(ax_1+by_1) + y_1(cx_1+dy_1)-x_0(px_0+qy_0)-y_0(rx_0+sy_0) = C$$
$$ax^2_1+bx_1y_1+cx_1y_1+dy^2_1-px^2_0-qx_0y_0-rx_0y_0-sy^2_0 = C$$
$$bx_1y_1+cx_1y_1+dy^2_1-qx_0y_0-rx_0y_0-sy^2_0 = C-ax^2_1+px^2_0$$
$$dy^2_1-sy^2_0+bx_1y_1+cx_1y_1-qx_0y_0-rx_0y_0 = C-ax^2_1+px^2_0$$
$$dy^2_1-sy^2_0+x_1y_1(b+c)+x_0y_0(-q-r) = C-ax^2_1+px^2_0$$
substituting for $x_0, y_0, x_1, y_1$ we now have the following:
$$d(y-\mu_{11})^2-s( y-\mu_{01})^2+(x-\mu_{10})(y-\mu_{11})(b+c)+(x-\mu_{00})(y-\mu_{01})(-q-r) = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2$$
then I calculated the squares and reduced the terms to the following result:
$$(d-s)y^2+(-2d\mu_{11}+2s\mu_{01}+bx-b\mu_{10}+cx-c\mu_{10}-qx+q\mu_{00}-rx+r\mu_{00})y = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2+b\mu_{11}x+c\mu_{11}x-q\mu_{01}x-r\mu_{01}x+d\mu_{11}^2-s\mu_{01}^2-b\mu_{10}\mu_{11}-c\mu_{10}\mu_{11}+q\mu_{01}\mu_{00}+r\mu_{01}\mu_{00}$$
Finally, I can apply the quadratic formula to solve for $y$ where
$u = d-s$
$v = -2d\mu_{11}+2s\mu_{01}+bx-b\mu_{10}+cx-c\mu_{10}-qx+q\mu_{00}-rx+r\mu_{00}$
$w = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2+b\mu_{11}x+c\mu_{11}x-q\mu_{01}x-r\mu_{01}x+d\mu_{11}^2-s\mu_{01}^2-b\mu_{10}\mu_{11}-c\mu_{10}\mu_{11}+q\mu_{01}\mu_{00}+r\mu_{01}\mu_{00}$
The quadratic formula with these variables would be the following:
$$y = \frac{-v\pm\sqrt{v^2-4uw}}{2u}$$
After attempting to check this solution on a simple data set I obtain poor results. I cannot figure out if it's the approach to the solution or if something is wrong in my code.
Thanks for your time and help!
Best Answer
If you look at the calculations, you will see there are a few bugs in this.
Correct value of w comes out to be : $$w = C-a(x-\mu_{10})^2+p(x-\mu_{00})^2+b\mu_{11}x+c\mu_{11}x-q\mu_{01}x-r\mu_{01}x-d\mu_{11}^2+s\mu_{01}^2-b\mu_{10}\mu_{11}-c\mu_{10}\mu_{11}+q\mu_{01}\mu_{00}+r\mu_{01}\mu_{00} $$
After then the value of y comes out to be: $$ y = \frac{-v\pm\sqrt{v^2+4uw}}{2u} $$
After making these two changes, you will get the correct quadratic boundary.