The theory of Conic Sections stems from ancient times. It is an example of pure mathematics, which has found applications only many centuries after it
has been developed, e.g. with the laws of planet motion as discovered
by Johannes Keppler. But, interesting as it is, we shall leave aside
history and immediately come to core - or rather
cone - business.
The problem is to intersect a
circular cone with a plane and determine the curves of intersection, like shown
in the above picture. This could be done in the way the old Greek mathematicians
did it. But we prefer to take a path that requires less ingenuity and we shall
employ the means of modern analytical geometry instead. With such an approach,
though, one should be prepared for tedious algebra when working out
the details.
Analysis
A circular cone is characterized by the fact that the angle $\phi$ between
the cone axis and its surface is a constant. Let the unit vector $\vec{a}$ be
the direction of the cone axis and let $\vec{p}$ point to the top vertex of the
cone. An arbitrary point at the surface of the cone is pinpointed by $\vec{r}$.
Then the following is an equation of the cone surface:
$$
(\vec{a}\cdot\vec{r}-\vec{p}) = |\vec{a}||\vec{r}-\vec{p}|\cos(\phi)
$$
Square both sides:
$$
(\vec{a}\cdot\vec{r}-\vec{p})^2 =
(\vec{a}\cdot\vec{a})(\vec{r}-\vec{p}\cdot\vec{r}-\vec{p})\cos^2(\phi)
$$
And work out:
$$
(\vec{a}\cdot\vec{r})^2 - 2(\vec{a}\cdot\vec{p})(\vec{a}\cdot\vec{r})
+ (\vec{a}\cdot\vec{p})^2 = \cos^2(\phi) \left\{
(\vec{r}\cdot\vec{r}) - 2(\vec{p}\cdot\vec{r}) + (\vec{p}\cdot\vec{p})\right\}
$$
The unit vector $\vec{a}$ can be written as:
$$
\vec{a} = \left[
\cos(\alpha)\cos(\gamma),\cos(\alpha)\sin(\gamma),\sin(\alpha)\right]
$$
Where $\alpha$ is the angle between the cone axis and the XY-plane and $\gamma$
is an angle that indicates how the conic section is rotated in the plane.
The vector of the top of the cone can be written in its coordinates as:
$$
\vec{p} = (p,q,h)
$$
Where $h$ is the height of the cone above the XY plane and $(p,q)$ indicates
how the conic section is translated in the plane. Last but not least, the vector
pointing to the cone surface is written as:
$$
\vec{r} = (x,y,z)
$$
Where the intersections with the XY plane are found for $z = 0$. Let's do just
that and work out the above:
$$
\begin{cases}
(\vec{a}\cdot\vec{r}) &=&
\cos(\alpha)\cos(\gamma)\,x + \cos(\alpha)\sin(\gamma)\,y \\
(\vec{a}\cdot\vec{p}) &=&
\cos(\alpha)\cos(\gamma)\,p + \cos(\alpha)\sin(\gamma)\,q + \sin(\alpha)\,h \\
(\vec{r}\cdot\vec{r}) &=& x^2 + y^2 \\
(\vec{p}\cdot\vec{r}) &=& p\,x + q\,y \\
(\vec{p}\cdot\vec{p}) &=& p^2 + q^2 + h^2
\end{cases}
$$
Collecting powers of $x$ and $y$ results in:
$$
A\,x^2 + B\,xy + C\,y^2 + D\,x + E\,y + F = 0
$$
Where:
$$
\begin{cases}
A &=& \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \\
B &=& - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \\
C &=& \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma) \\
D &=& 2 \left\{ \cos(\alpha)\cos(\gamma) (\vec{a}\cdot\vec{p})
- \cos^2(\phi)\,p \right\} \\
E &=& 2 \left\{ \cos(\alpha)\sin(\gamma) (\vec{a}\cdot\vec{p})
- \cos^2(\phi)\,q \right\} \\
F &=& (\vec{p}\cdot\vec{p}) \cos^2(\phi) - (\vec{a}\cdot\vec{p})^2
\end{cases}
$$
Meaning
The first three coefficients of the conic section equation are:
$$
\begin{cases}
A &=& \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \\
B &=& - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \\
C &=& \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma)
\end{cases}
$$
All kind of conics can still be produced if the angles $\phi$ and $\alpha$ are
limited to sensible values:
$$
\begin{cases}
&&0 < \phi < 90^o \quad \Longrightarrow \quad 0 < \cos(\phi) < 1 \\
&&0 \le \alpha \le 90^o \quad \Longrightarrow \quad 0 \le \cos(\alpha) \le 1
\end{cases}
$$
Generality is not affected by these choices. Moreover it is seen from the picture
below that the form of the conic section is determined
by the angles $\phi$ and $\alpha$ and nothing else. Therefore the ratio of the
two angles will be defined here as the excentricity ($\epsilon$) of the
conic section:
$$
\epsilon = \frac{\cos(\alpha)}{\cos(\phi)}
$$
The following relationships exist between the excentricity and the form of a
conic section, as is clear from the picture:
$$
\begin{cases}
\mbox{Circle : }& \alpha = 90^o &\quad \Longleftrightarrow \quad \epsilon = 0 \\
\mbox{Ellipse : }& \alpha > \phi &\quad \Longleftrightarrow \quad \epsilon < 1 \\
\mbox{Parabola : }& \alpha = \phi &\quad \Longleftrightarrow \quad \epsilon = 1 \\
\mbox{Hyperbola : }& \alpha < \phi &\quad \Longleftrightarrow \quad \epsilon > 1
\end{cases}
$$
So far so good. The coefficients $(A,B,C)$ can be combined into
some interesting quantities which are only dependent upon form,
that is: the angles $\phi$ and $\alpha$.
It is remarked in the first place that $(A,B,C)$ are independent of the vector
$\vec{p} = (p,q,h)$ and thus independent of translation and scaling. If we seek
to eliminate any dependence upon the angle of rotation $\gamma$, then we find:
$$
A + C = 2 \cos^2(\phi) - \cos^2(\alpha) = \cos^2(\phi) (2 - \epsilon^2)
$$
This quantity is known (for some good reasons) as the trace of the conic
section. Instead of eliminating the angle of rotation, we could also try
to calculate it.
$$
A - C = - \cos^2(\alpha)\left[\cos^2(\gamma)-\sin^2(\gamma)\right]
= - \cos^2(\alpha)\cos(2\gamma)
$$
There is a striking resemblance with:
$$
B = - \cos^2(\alpha)\sin(2\gamma)
$$
We thus find:
$$
\frac{B}{A - C} = \frac{\sin(2\gamma)}{\cos(2\gamma)} \quad \Longrightarrow \quad
\tan{2\gamma} = \frac{B}{A - C}
$$
Herewith - in principle - the angle of rotation $\gamma$ can be reconstructed
from the conic section equation; provided that $A \neq C$.
Let's proceed with
another quantity that is independent of any rotation.
$$
B^2 - 4 A C = \left[ - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \right]^2
$$ $$
- 4 \left[ \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \right]
\left[ \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma) \right]
$$
This quantity is known (also for some good reasons) as the determinant or
discriminant of the conic section. Work out:
$$
= 4 \cos^4(\alpha)\cos^2(\gamma)\sin^2(\gamma) - 4 \cos^4(\phi)
- 4 \cos^4(\alpha)\cos^2(\gamma)\sin^2(\gamma) \\
+ 4 \cos^2(\phi)\cos^2(\alpha)\left[\cos^2(\gamma)+\sin^2(\gamma)\right] \\
= - 4\cos^4(\phi) + 4\cos^2(\phi)\cos^2(\alpha) \\
\quad \Longrightarrow \quad B^2 - 4 A C
= 4\cos^2(\phi)\left[\cos^2(\alpha) - \cos^2(\phi)\right]
= \left[2\cos^2(\phi)\right]^2(\epsilon^2 - 1)
$$
The following relationships exist between the discriminant and the form of a
conic section, as is clear from the above:
$$
\begin{cases}
&\mbox{Ellipse}& \quad \Longleftrightarrow \quad \epsilon < 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) < 0 \\
&\mbox{Parabola}& \quad \Longleftrightarrow \quad \epsilon = 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) = 0 \\
&\mbox{Hyperbola}& \quad \Longleftrightarrow \quad \epsilon > 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) > 0
\end{cases}
$$
BONUS on excentricity.
$$
(A + C)^2 = 4\cos^2(\phi)\left[\cos^2(\phi) - \cos^2(\alpha)\right]
+ \cos^4(\alpha)
$$
Upon addition this gives:
$$
(B^2 - 4 A C) + (A + C)^2 = \cos^4(\alpha) \quad \Longrightarrow \\
\cos(\alpha) = \sqrt{\sqrt{B^2 + (A-C)^2}}
$$
About the angle $\phi$ between the cone axis and its surface:
$$
A + C = 2\cos^2(\phi) - \sqrt{B^2 + (A-C)^2} \quad \Longrightarrow \quad
\cos(\phi) = \sqrt{\frac{(A+C) + \sqrt{B^2 + (A-C)^2}}{2}}
$$
Herewith the excentricity $\epsilon$ can be expressed into the coefficients of
the conic section equation $(A,B,C)$:
$$
\epsilon = \sqrt{\frac{2\sqrt{B^2 + (A-C)^2}}
{(A+C) + \sqrt{B^2 + (A-C)^2}}}
$$
Many other expressions can be derived, especially for the coefficients $D,E,F$.
But I think that deriving them here will disturb a good balance between interesting and cumbersome.
EDIT. Explanation of
$$
\vec{a} = \left[
\cos(\alpha)\cos(\gamma),\cos(\alpha)\sin(\gamma),\sin(\alpha)\right]
$$
See picture. Overline denotes " length of " in:
$$
\overline{OZ}=\overline{OA}\, \sin(\alpha) \quad ; \quad
\overline{OD}=\overline{OA}\, \cos(\alpha) \\
\overline{OX}=\overline{OD}\, \cos(\gamma) \quad ; \quad
\overline{OY}=\overline{OD}\, \sin(\gamma)
$$
With $\overline{OA}=1$ .
Over on the Wikipedia page on conic sections you can see that we can write a general conic section as:
$$Ax^2+Bxy+Cy^2+Dx+Ey+F=0$$
And this can be rewritten as an equation:
$$X^TMX + c=0$$
Where
$$X = \left[ \begin{matrix}x\\y\\\end{matrix} \right]$$
$$M = \left[ \begin{matrix}A&B/2\\B/2&C\\\end{matrix} \right]$$
and $c$ contains the linear terms.
When we do a general linear transformation ($X\to TX$ with $T$ invertible) of the plane (rotation, shear, or scale) this equation transforms to:
$$(TX)^TMTX + c'=0$$
$$XT^TMTX + c'=0$$
So $M\to T^TMT$ is the transformation of the coefficients of the conic section.
Then, since the sign of the discriminant $B^2-4AC = -4\det M$ determines the type of conic section, and the transformation $\det M \to (\det T)^2\det M$ preserves the sign, all linear and affine transformations of the plane map conics to conics of the same type (ellipses to ellipses, parabolas to parabolas, and hyperbolas to hyperbolas). But only orthogonal transformations preserve circles, of course.
Best Answer
Write the general equation using a $2 \times 2$ matrix $Q$ and a $2 \times 1$ vector $v$ and a constant $c$. Starting with
$ a x^2 + 2 h xy +b y^2 + 2 g x + 2 f y + c = 0 $
by defining the vector $r = [x, y]^T$, this becomes
$r^T Q r + v^T r + c = 0 $
where
$Q = \begin{bmatrix} a && h \\ h && b \end{bmatrix} $
$v = \begin{bmatrix} 2 g \\ 2 f \end{bmatrix} $
The condition to have an ellipse or an empty set is that the eigenvalues of $Q$ have the same sign. The eigenvalues of $Q$ are the roots of this quadratic equation
$ (\lambda - a)(\lambda - b) - h^2 = 0 $
And they will have the same sign if $ a b - h^2 \gt 0$, i.e. $h^2 \lt a b $