We know that the equation $ax^2+by^2+cz^2+2hxy+2fyz+2gzx=0$ represents a pair of planes if and only if $\begin{vmatrix}
a& h& g\\
h&b& f\\
g& f&c
\end{vmatrix}=0.$ I wanted to prove this result. Let the two planes be of the form $l_1x+m_1y+n_1z=0$ and $l_2x+m_2y+n_2z=0.$ Then we have $$ax^2+by^2+cz^2+2hxy+2fyz+2gzx=(l_1x+m_1y+n_1z)(l_2x+m_2y+n_2z)=0.$$ On comparing the coefficients of $x, y$ and $z$ both sides we get \begin{align*}
l_1l_2&=a\\
m_1m_2&=b\\
n_1n_2&=c\\
l_1m_2+l_2m_1&=2h\\
l_1n_2+l_2n_1&=2f\\
m_1n_2+m_2n_1&=2g.
\end{align*}
Now we have six equations in six variables. How to show that these equations have a non-trivial solution if and only if
$\begin{vmatrix}
a& h& g\\
h&b& f\\
g& f&c
\end{vmatrix}=0?$
Best Answer
I put material on solving $P^T M P = D$ diagonal when $M$ is real symmetric at reference for linear algebra books that teach reverse Hermite method for symmetric matrices
The above is just an algorithm for repeated completing the square.
As far as how to do it, we are given a symmetric real matrix, call it $M.$ We may construct a real matrix $P$ with $\det P = \pm 1$ so that $P^T M P = D$ is diagonal. With determinant $0$ this means as many as two nonzero entries in $D.$ If we take $Q = P^{-1}$ we get $Q^T D Q = M.$ If we name your quadratic form $R(x,y,z)$ we have reached $$ \color{blue}{ R(x,y,z) = d_1 (a_1x+b_1 y + c_1z)^2 + d_2 (a_2x+b_2 y + c_2z)^2}$$
When $d_1, d_2$ are both positive or both negative, both linear expressions must be zero and the zero set is a single (real) line. When $d_1 d_2 < 0$ we see that $R(x,y,z)$ factors over the reals. One may write $$ A V^2 - B W^2 = (V \sqrt A + W \sqrt B)(V \sqrt A - W \sqrt B) $$