Here are two ways to answer the question.
- 1) A matrix oriented arguing:
It is clear that the following condition must hold:
$$\tag{0}m:=a b - h^2 \neq 0$$
(otherwise, the LHS of the given equation would become a constant).
The equation of the conical section can thus be divided by $m$, and therefore be written
$$\tag{1}U:=ax^2+2hxy+by^2+2gx+2fy+c=0 \ \ \text{where} \ \ c:=\frac{a f^2 + b g^2 - 2f g h}{a b - h^2}$$
or, in a matrix form:
$U=X^TMX=0$ with $X^T=(x \ y \ 1)$ and:
$$\tag{2} M=\pmatrix{a& h& g\\h& b& f\\g& f& c}.$$
Expanding the determinant of $M$ along its third column gives $det(M)=0$.
Therefore $rank(M)\leq 2$.
But condition (0) expresses the fact that the upper left minor of $M$ is nonzero. Thus $rank(M)\geq 2$. Finally:
$$\tag{3} rank(M)=2.$$
Thus the conical section is decomposable into 2 straight lines (see for example (Decomposition of a degenerate conic)), but these straight lines might have complex coefficients!
It appears that a condition of reality is
$$\tag{3}h^2 > ab$$
(were you aware of it?) as we are going to see it in the second way.
- 2) An explicit way, by decomposition of $U$.
Under condition (3), (1) can be written
$$\tag{4}U=\left(s x + \frac{h y + g}{s}\right)^2-\left(t y + \frac{u}{t}\right)^2=0$$
by setting $s=\sqrt{a}, t=\dfrac{\sqrt{h^2 - a b}}{s}, u=\dfrac{g h - a f}{a}.$
Being a difference of two squares, (4) can be decomposed into two pairs of first degree equations which are the equations of the straight lines.
In general, the equation $\ ax^2+2hxy+by^2+2gx+2fy+c=0\ $ defines a conic, of which two intersecting straight lines is one (degenerate) special case. One standard way to determine what form of conic the equation represents is to diagonalise the matrix,
$$
A=\pmatrix{a&h\\h&b}\
$$
by finding its eigenvalues and egenvectors. Let $\ \lambda_1, \lambda_2\ $ be the eigenvalues, and $\ \boldsymbol{e}_1, \boldsymbol{e}_2\ $ the corresponding normalised eigenvectors, which can be chosen so that $\ \boldsymbol{e}_1=\pmatrix{\cos\theta\\-\sin\theta}\ $ and $\ \boldsymbol{e}_2=\pmatrix{\sin\theta\\\cos\theta}\ $ for some angle $\ \theta\ $. If $\ \Theta\ $ is the matrix with columns $\ \boldsymbol{e}_1\ $ and $\ \boldsymbol{e}_2\ $, then $\ \Theta\ $ is a rotation matrix, with $\ \Theta^{-1} = \Theta^\top\ $, and
$$
\Theta^\top A\Theta = \pmatrix{\lambda_1 & 0\\0&\lambda_2}\ .
$$
Now, the equation we are interested in can be written as
\begin{eqnarray}
0&=& \boldsymbol{x}^\top A\boldsymbol{x} + 2\boldsymbol{g}^\top\boldsymbol{x} + c\\
&=& \left(\Theta^\top \boldsymbol{x}\right)^\top\Theta^\top A\Theta\left(\Theta^\top\boldsymbol{x}\right) + 2\left(\Theta^\top\boldsymbol{g}\right)^\top \Theta^\top\boldsymbol{x}+c\\
&=&\boldsymbol{x}'^\top\Lambda\boldsymbol{x}'+2\boldsymbol{g}'^\top\boldsymbol{x}'+c\ ,
\end{eqnarray}
where $\ \boldsymbol{x}=\pmatrix{x\\y}\ $, $\ \boldsymbol{x}'=\Theta^\top\boldsymbol{x}\\$, $\ \Lambda = \pmatrix{\lambda_1 & 0\\0&\lambda_2}\ $, $\ \boldsymbol{g}=\pmatrix{g\\h}\ $, and $\ \boldsymbol{g}'=\Theta^\top\boldsymbol{g}\ $. The entries of $\ \boldsymbol{x}'\ $, $\ x_1'=x\cos\theta+y\sin\theta\ $, and $\ x_2'=-x\sin\theta+y\cos\theta\ $, are the coordinates of a point $P$ with respect to a set of axes that have been rotated clockwise through an angle $\ \theta\ $relative to the original axes, where $\ x\ $ and $ y\ $ are the coordinates of $P$ with respect to those original axes. It follows from above that the equation the the conic with respect to the new axes is
\begin{eqnarray}
0 &=& \lambda_1 x_1'^2 + \lambda_2x_2'^2 +2g_1'x_1'+ 2g_2'x_2' + c\\
&=& \lambda_1\left(x_1' +\frac{g_1'}{\lambda_1}\right)^2 + \lambda_2\left(x_2' +\frac{g_2'}{\lambda_2}\right)^2 +c - \frac{g_1'^2}{\lambda_1}-\frac{g_2'^2}{\lambda_2}\ .
\end{eqnarray}
This is the equation of two intersecting straight lines if and only if $\ \lambda_1\ne0\ $, $\ \lambda_2\ne0\ $, $\ \lambda_1\ $ and $\ \lambda_2\ $ are of opposite sign, and $\ c - \frac{g_1'^2}{\lambda_1}-\frac{g_2'^2}{\lambda_2}=0\ $. If this is the case, suppose, without loss of generality, that $\ \lambda_1>0\ $ and $\ \lambda_2<0\ $. Then the above equation becomes
\begin{eqnarray}
0 &=& \left(\sqrt{\lambda_1}x_1' + \frac{g_1'}{\sqrt{\lambda_1}}\right)^2- \left(\sqrt{-\lambda_2}x_2' - \frac{g_2'}{\sqrt{-\lambda_2}}\right)^2\\
&=& \left(\sqrt{\lambda_1}x_1' + \sqrt{-\lambda_2}x_2'+ \frac{g_1'}{\sqrt{\lambda_1}}-\frac{g_2'}{\sqrt{-\lambda_2}}\right)\\
&&\ \ \ \cdot \left(\sqrt{\lambda_1}x_1' - \sqrt{-\lambda_2}x_2'+ \frac{g_1'}{\sqrt{\lambda_1}}+\frac{g_2'}{\sqrt{-\lambda_2}}\right)\ ,
\end{eqnarray}
and the equations of the two straight lines in the new coordinates are
\begin{eqnarray}
\sqrt{\lambda_1}x_1' + \sqrt{-\lambda_2}x_2'+ \frac{g_1'}{\sqrt{\lambda_1}}-\frac{g_2'}{\sqrt{-\lambda_2}}&=&0\ \ \mbox{, and}\\
\sqrt{\lambda_1}x_1' - \sqrt{-\lambda_2}x_2'+ \frac{g_1'}{\sqrt{\lambda_1}}+\frac{g_2'}{\sqrt{-\lambda_2}} &=&0\ .
\end{eqnarray}
Finally, substituting $\ x_1'=x\cos\theta+y\sin\theta\ $, and $\ x_2'=-x\sin\theta+y\cos\theta\ $ in these equations, we get the equations of the lines in the original coordinates:
\begin{eqnarray}
x\left(\sqrt{\lambda_1}\cos\theta - \sqrt{-\lambda_2}\sin\theta\right)&+ &y\left(\sqrt{\lambda_1}\sin\theta +\sqrt{-\lambda_2}\cos\theta\right)\\
&+& \frac{g_1'}{\sqrt{\lambda_1}}-\frac{g_2'}{\sqrt{-\lambda_2}}&=&0
\end{eqnarray}
and
\begin{eqnarray}
x\left(\sqrt{\lambda_1}\cos\theta + \sqrt{-\lambda_2}\sin\theta\right)&+ &y\left(\sqrt{\lambda_1}\sin\theta -\sqrt{-\lambda_2}\cos\theta\right)\\
&+& \frac{g_1'}{\sqrt{\lambda_1}}+\frac{g_2'}{\sqrt{-\lambda_2}}&=&0\ .
\end{eqnarray}
Best Answer
Step $1:$ Calculate the solution $(x_0,y_0)$ for the system $$ l_jx+m_jy+n_j=0,\qquad j=1,2 $$ Using Crammer's rule we get $$ x_0^2=\left({ m_1n_2-m_2n_1\over l_1m_2-l_2m_1}\right)^2={( m_1n_2+m_2n_1)^2-4n_1n_2m_1m_2\over(l_1m_2+l_2m_1)^2-4l_1l_2m_1m_2}={f^2-bc\over h^2-ab } $$ Similarly calculate $$y_0^2={g^2-ac\over h^2-ab}$$.
Step $2:$ Calculate $x_0^2+y_0^2$.