The theory of Conic Sections stems from ancient times. It is an example of pure mathematics, which has found applications only many centuries after it
has been developed, e.g. with the laws of planet motion as discovered
by Johannes Keppler. But, interesting as it is, we shall leave aside
history and immediately come to core - or rather
cone - business.
The problem is to intersect a
circular cone with a plane and determine the curves of intersection, like shown
in the above picture. This could be done in the way the old Greek mathematicians
did it. But we prefer to take a path that requires less ingenuity and we shall
employ the means of modern analytical geometry instead. With such an approach,
though, one should be prepared for tedious algebra when working out
the details.
Analysis
A circular cone is characterized by the fact that the angle $\phi$ between
the cone axis and its surface is a constant. Let the unit vector $\vec{a}$ be
the direction of the cone axis and let $\vec{p}$ point to the top vertex of the
cone. An arbitrary point at the surface of the cone is pinpointed by $\vec{r}$.
Then the following is an equation of the cone surface:
$$
(\vec{a}\cdot\vec{r}-\vec{p}) = |\vec{a}||\vec{r}-\vec{p}|\cos(\phi)
$$
Square both sides:
$$
(\vec{a}\cdot\vec{r}-\vec{p})^2 =
(\vec{a}\cdot\vec{a})(\vec{r}-\vec{p}\cdot\vec{r}-\vec{p})\cos^2(\phi)
$$
And work out:
$$
(\vec{a}\cdot\vec{r})^2 - 2(\vec{a}\cdot\vec{p})(\vec{a}\cdot\vec{r})
+ (\vec{a}\cdot\vec{p})^2 = \cos^2(\phi) \left\{
(\vec{r}\cdot\vec{r}) - 2(\vec{p}\cdot\vec{r}) + (\vec{p}\cdot\vec{p})\right\}
$$
The unit vector $\vec{a}$ can be written as:
$$
\vec{a} = \left[
\cos(\alpha)\cos(\gamma),\cos(\alpha)\sin(\gamma),\sin(\alpha)\right]
$$
Where $\alpha$ is the angle between the cone axis and the XY-plane and $\gamma$
is an angle that indicates how the conic section is rotated in the plane.
The vector of the top of the cone can be written in its coordinates as:
$$
\vec{p} = (p,q,h)
$$
Where $h$ is the height of the cone above the XY plane and $(p,q)$ indicates
how the conic section is translated in the plane. Last but not least, the vector
pointing to the cone surface is written as:
$$
\vec{r} = (x,y,z)
$$
Where the intersections with the XY plane are found for $z = 0$. Let's do just
that and work out the above:
$$
\begin{cases}
(\vec{a}\cdot\vec{r}) &=&
\cos(\alpha)\cos(\gamma)\,x + \cos(\alpha)\sin(\gamma)\,y \\
(\vec{a}\cdot\vec{p}) &=&
\cos(\alpha)\cos(\gamma)\,p + \cos(\alpha)\sin(\gamma)\,q + \sin(\alpha)\,h \\
(\vec{r}\cdot\vec{r}) &=& x^2 + y^2 \\
(\vec{p}\cdot\vec{r}) &=& p\,x + q\,y \\
(\vec{p}\cdot\vec{p}) &=& p^2 + q^2 + h^2
\end{cases}
$$
Collecting powers of $x$ and $y$ results in:
$$
A\,x^2 + B\,xy + C\,y^2 + D\,x + E\,y + F = 0
$$
Where:
$$
\begin{cases}
A &=& \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \\
B &=& - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \\
C &=& \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma) \\
D &=& 2 \left\{ \cos(\alpha)\cos(\gamma) (\vec{a}\cdot\vec{p})
- \cos^2(\phi)\,p \right\} \\
E &=& 2 \left\{ \cos(\alpha)\sin(\gamma) (\vec{a}\cdot\vec{p})
- \cos^2(\phi)\,q \right\} \\
F &=& (\vec{p}\cdot\vec{p}) \cos^2(\phi) - (\vec{a}\cdot\vec{p})^2
\end{cases}
$$
Meaning
The first three coefficients of the conic section equation are:
$$
\begin{cases}
A &=& \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \\
B &=& - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \\
C &=& \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma)
\end{cases}
$$
All kind of conics can still be produced if the angles $\phi$ and $\alpha$ are
limited to sensible values:
$$
\begin{cases}
&&0 < \phi < 90^o \quad \Longrightarrow \quad 0 < \cos(\phi) < 1 \\
&&0 \le \alpha \le 90^o \quad \Longrightarrow \quad 0 \le \cos(\alpha) \le 1
\end{cases}
$$
Generality is not affected by these choices. Moreover it is seen from the picture
below that the form of the conic section is determined
by the angles $\phi$ and $\alpha$ and nothing else. Therefore the ratio of the
two angles will be defined here as the excentricity ($\epsilon$) of the
conic section:
$$
\epsilon = \frac{\cos(\alpha)}{\cos(\phi)}
$$
The following relationships exist between the excentricity and the form of a
conic section, as is clear from the picture:
$$
\begin{cases}
\mbox{Circle : }& \alpha = 90^o &\quad \Longleftrightarrow \quad \epsilon = 0 \\
\mbox{Ellipse : }& \alpha > \phi &\quad \Longleftrightarrow \quad \epsilon < 1 \\
\mbox{Parabola : }& \alpha = \phi &\quad \Longleftrightarrow \quad \epsilon = 1 \\
\mbox{Hyperbola : }& \alpha < \phi &\quad \Longleftrightarrow \quad \epsilon > 1
\end{cases}
$$
So far so good. The coefficients $(A,B,C)$ can be combined into
some interesting quantities which are only dependent upon form,
that is: the angles $\phi$ and $\alpha$.
It is remarked in the first place that $(A,B,C)$ are independent of the vector
$\vec{p} = (p,q,h)$ and thus independent of translation and scaling. If we seek
to eliminate any dependence upon the angle of rotation $\gamma$, then we find:
$$
A + C = 2 \cos^2(\phi) - \cos^2(\alpha) = \cos^2(\phi) (2 - \epsilon^2)
$$
This quantity is known (for some good reasons) as the trace of the conic
section. Instead of eliminating the angle of rotation, we could also try
to calculate it.
$$
A - C = - \cos^2(\alpha)\left[\cos^2(\gamma)-\sin^2(\gamma)\right]
= - \cos^2(\alpha)\cos(2\gamma)
$$
There is a striking resemblance with:
$$
B = - \cos^2(\alpha)\sin(2\gamma)
$$
We thus find:
$$
\frac{B}{A - C} = \frac{\sin(2\gamma)}{\cos(2\gamma)} \quad \Longrightarrow \quad
\tan{2\gamma} = \frac{B}{A - C}
$$
Herewith - in principle - the angle of rotation $\gamma$ can be reconstructed
from the conic section equation; provided that $A \neq C$.
Let's proceed with
another quantity that is independent of any rotation.
$$
B^2 - 4 A C = \left[ - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \right]^2
$$ $$
- 4 \left[ \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \right]
\left[ \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma) \right]
$$
This quantity is known (also for some good reasons) as the determinant or
discriminant of the conic section. Work out:
$$
= 4 \cos^4(\alpha)\cos^2(\gamma)\sin^2(\gamma) - 4 \cos^4(\phi)
- 4 \cos^4(\alpha)\cos^2(\gamma)\sin^2(\gamma) \\
+ 4 \cos^2(\phi)\cos^2(\alpha)\left[\cos^2(\gamma)+\sin^2(\gamma)\right] \\
= - 4\cos^4(\phi) + 4\cos^2(\phi)\cos^2(\alpha) \\
\quad \Longrightarrow \quad B^2 - 4 A C
= 4\cos^2(\phi)\left[\cos^2(\alpha) - \cos^2(\phi)\right]
= \left[2\cos^2(\phi)\right]^2(\epsilon^2 - 1)
$$
The following relationships exist between the discriminant and the form of a
conic section, as is clear from the above:
$$
\begin{cases}
&\mbox{Ellipse}& \quad \Longleftrightarrow \quad \epsilon < 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) < 0 \\
&\mbox{Parabola}& \quad \Longleftrightarrow \quad \epsilon = 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) = 0 \\
&\mbox{Hyperbola}& \quad \Longleftrightarrow \quad \epsilon > 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) > 0
\end{cases}
$$
BONUS on excentricity.
$$
(A + C)^2 = 4\cos^2(\phi)\left[\cos^2(\phi) - \cos^2(\alpha)\right]
+ \cos^4(\alpha)
$$
Upon addition this gives:
$$
(B^2 - 4 A C) + (A + C)^2 = \cos^4(\alpha) \quad \Longrightarrow \\
\cos(\alpha) = \sqrt{\sqrt{B^2 + (A-C)^2}}
$$
About the angle $\phi$ between the cone axis and its surface:
$$
A + C = 2\cos^2(\phi) - \sqrt{B^2 + (A-C)^2} \quad \Longrightarrow \quad
\cos(\phi) = \sqrt{\frac{(A+C) + \sqrt{B^2 + (A-C)^2}}{2}}
$$
Herewith the excentricity $\epsilon$ can be expressed into the coefficients of
the conic section equation $(A,B,C)$:
$$
\epsilon = \sqrt{\frac{2\sqrt{B^2 + (A-C)^2}}
{(A+C) + \sqrt{B^2 + (A-C)^2}}}
$$
Many other expressions can be derived, especially for the coefficients $D,E,F$.
But I think that deriving them here will disturb a good balance between interesting and cumbersome.
EDIT. Explanation of
$$
\vec{a} = \left[
\cos(\alpha)\cos(\gamma),\cos(\alpha)\sin(\gamma),\sin(\alpha)\right]
$$
See picture. Overline denotes " length of " in:
$$
\overline{OZ}=\overline{OA}\, \sin(\alpha) \quad ; \quad
\overline{OD}=\overline{OA}\, \cos(\alpha) \\
\overline{OX}=\overline{OD}\, \cos(\gamma) \quad ; \quad
\overline{OY}=\overline{OD}\, \sin(\gamma)
$$
With $\overline{OA}=1$ .
A more general proof:
Let Q and R be the points at which lines through $P=(x_1,y_1)$ touch a non degenerate conic $S(x,y) \equiv Ax^2+2Bxy+Cy^2+2Dx+2Ey+F=0$. In other words, lines PR and PQ are the tangents to this conic at points Q and R, and RQ is the chord of contact.
Let $PR(x,y)=0$, $PQ(x,y)=0$, $RQ(x,y)=0$ be the equation of these lines.
As RQ is polar of P in relation to this conic,
$$RQ(x,y)\equiv (Ax+By+D)x_1+(Bx+Cy+E)y_1+(Dx+Ey+F)=0$$
On the other hand, the equation $\lambda(PR(x,y).PQ(x,y))+\mu(RQ(x,y))^2=0$ represents all conics which are touched by lines PR and PQ at points R and Q. Therefore, for especific values of $\lambda$ and $\mu $ (none of which can be equal to zero, because otherwise S would be a degenerate conic):
$$S(x,y)\equiv \lambda(PR(x,y).PQ(x,y))+\mu(RQ(x,y))^2=0$$
Then,
$$S(x_1,y_1)=\lambda(PR(x_1,y_1).PQ(x_1,y_1))+\mu(RQ(x_1,y_1))^2,$$
$$S(x_1,y_1)=\mu(RQ(x_1,y_1))^2$$
Besides that,
$$RQ(x_1,y_1)=(Ax_1+By_1+D)x_1+(Bx_1+Cy_1+E)y_1+(Dx_1+Ey_1+F),$$
$$RQ(x_1,y_1)=S(x_1,y_1)$$
Thus
$$S(x_1,y_1)=\mu(S(x_1,y_1))^2,$$
$$\mu=\frac {1}{S(x_1,y_1)}$$
Therefore
$$S(x_1,y_1).S(x,y)\equiv S(x_1,y_1)\lambda(PR(x,y).PQ(x,y))+(RQ(x,y))^2,$$
$$S(x_1,y_1)\lambda(PR(x,y).PQ(x,y))\equiv S(x_1,y_1).S(x,y)-(RQ(x,y))^2$$
Finally, equating left and right members of this identity to zero, we get that the equation of tangents PR and PQ to conic S can be represented by equation
$$S(x_1,y_1).S(x,y)-(RQ(x,y))^2=0$$
Best Answer
Take $ax^2+2hxy+by^2=0$. Multiply by $4a$ $(a\neq 0)$ and complete the square to obtain the equivalent equation $$(2ax+hy)^2+(4ab-h^2)y^2=0$$
If $4ab-h^2\gt 0$ then both terms on the left-hand side are non-negative, and must therefore be zero. If $4ab-h^2=0$ you get the single line $2ax+hy=0$ (the two lines coincide to give a degenerate case). And if $4ab-h^2\lt 0$ you get two lines by factoring the left-hand side as the difference of two squares.