classificationoptimization
The fisher linear classifier for two classes is a classifier with this discriminant function:
$h(x) = V^{T}X + v_0$
where
$V = \left[ \frac{1}{2}\Sigma_1 + \frac{1}{2}\Sigma_2\right]^{-1}(M_2-M_1)$
and $M_1$, $M_2$ are means and $\Sigma_1$,$\Sigma_2$ are covariances of the classes.
$V$ can be calculated easily but the fisher criterion cannot give us the optimum $v_0$.
Then, how we can find the optimum $v_0$ analytically if we are using fisher classifier?
Following the paper you linked to (Mika et al., 1999), we have to find the $\mathbf{w}$ which maximizes the so called generalized Rayleigh quotient,
$$\frac{\mathbf{w}^\top \mathbf{S}_B \mathbf{w}}{\mathbf{w}^\top \mathbf{S}_W \mathbf{w}},$$
where for means $\mathbf{m}_1, \mathbf{m}_2$ and covariances $\mathbf{C}_1, \mathbf{C}_2$,
\begin{align} \mathbf{S}_B &= (\mathbf{m}_1 - \mathbf{m}_2)(\mathbf{m}_1 - \mathbf{m}_2)^\top, & \mathbf{S}_W &= \mathbf{C}_1 + \mathbf{C}_2. \end{align}
The solution can be found by solving the generalized eigenvalue problem \begin{align} \mathbf{S}_B\mathbf{w} = \lambda \mathbf{S}_W\mathbf{w}, \end{align} by first computing the eigenvalues $\lambda$ by solving \begin{align} \det(\mathbf{S}_B - \lambda \mathbf{S}_W) = 0 \end{align} and then solving for the eigenvector $\mathbf{w}$. In your case, $$\mathbf{S}_B - \lambda \mathbf{S}_W = \begin{pmatrix}16 - 3\lambda & 16 \\ 16 & 16 - 2\lambda\end{pmatrix}.$$ The determinant of this 2x2 matrix can be computed by hand.
The eigenvector with the largest eigenvalue maximizes the Rayleigh quotient. Instead of doing the calculations by hand, I solved the generalized eigenvalue problem in Python using scipy.linalg.eig and got
$$w_1 \approx 0.5547, w_2 \approx 0.8321,$$
which is different from the solution you found in your book. Below I plotted the optimal hyperplane of the weight vector I found (black) and the hyerplane of the weight vector found in your book (red).
$\hskip1in$
The discriminative value of a feature is based on its statistical distance between classes.
I have calculated the mean and variance for each feature and each class
Using your feature $i$ class $j$ estimated mean $\hat{\mu}_{i,j}$ and estimated variance $\hat{\sigma}_{i,j}^2$, one approach would be to compute the symmetric KL divergence for each feature for two classes you compare. The largest distance between feature distributions is the best discriminative feature for that pair.
KL divergence for two normal distributions is easy to compute.
Best Answer
When $X$ is normally distributed with known mean $M_1$ and covariance $\Sigma_1$ or with mean $M_2$ and covariance $\Sigma_2$, as indicated in comments to the question, then $V^{\ '}X$ is normally distributed either with mean $\mu_1 = V^{\ '} M_1$ and covariance $\sigma_1^2 = V^{\ '} \Sigma_1 V$ or with mean $\mu_2 = V^{\ '} M_2$ and covariance $\sigma_2^2 = V^{\ '} \Sigma_2 V$; $\mu_2 \gt \mu_1$. We might then care to optimize the chance of correct classification. This can be done provided we stipulate a prior distribution for the two classes. Letting $\pi_1$ be the chance of class 1 and $\pi_2$ the chance of class 2 and $\phi$ the standard normal pdf, then the posterior probabilities of the classes are equal (and therefore $x$ is at the threshold) when
$$f(x) = \pi_1 \phi(\frac{x - \mu_1}{\sigma_1}) - \pi_2 \phi(\frac{x - \mu_2}{\sigma_2}) = 0.$$
There will be at most one zero of $f$ between $x = \mu_1$ and $x = \mu_2$. (When the zeros lie outside this interval we might question the utility of this classifier.) Assuming one exists and choosing $v_0$ to be the negative of this zero gives a linear classifier $X \to V^{\ '}X + v_0$ that, when negative, indicates class 1 is more likely than class 2 and, when positive, indicates class 2 is more likely than class 1.
A simple case arises when the two classes are taken to be equally likely, $\pi_1 = \pi_2 = 1/2,$ for then it is clear from the symmetry and unimodality of $\phi$ that $v_0 = -(\mu_1 + \mu_2)/2$. Note, though, that in general it is not the case that the zero equals $\pi_1 \mu_1 + \pi_2 \mu_2$ (although that might be a good starting guess in a systematic search for the zero).