The theory of Conic Sections stems from ancient times. It is an example of pure mathematics, which has found applications only many centuries after it
has been developed, e.g. with the laws of planet motion as discovered
by Johannes Keppler. But, interesting as it is, we shall leave aside
history and immediately come to core - or rather
cone - business.
The problem is to intersect a
circular cone with a plane and determine the curves of intersection, like shown
in the above picture. This could be done in the way the old Greek mathematicians
did it. But we prefer to take a path that requires less ingenuity and we shall
employ the means of modern analytical geometry instead. With such an approach,
though, one should be prepared for tedious algebra when working out
the details.
Analysis
A circular cone is characterized by the fact that the angle $\phi$ between
the cone axis and its surface is a constant. Let the unit vector $\vec{a}$ be
the direction of the cone axis and let $\vec{p}$ point to the top vertex of the
cone. An arbitrary point at the surface of the cone is pinpointed by $\vec{r}$.
Then the following is an equation of the cone surface:
$$
(\vec{a}\cdot\vec{r}-\vec{p}) = |\vec{a}||\vec{r}-\vec{p}|\cos(\phi)
$$
Square both sides:
$$
(\vec{a}\cdot\vec{r}-\vec{p})^2 =
(\vec{a}\cdot\vec{a})(\vec{r}-\vec{p}\cdot\vec{r}-\vec{p})\cos^2(\phi)
$$
And work out:
$$
(\vec{a}\cdot\vec{r})^2 - 2(\vec{a}\cdot\vec{p})(\vec{a}\cdot\vec{r})
+ (\vec{a}\cdot\vec{p})^2 = \cos^2(\phi) \left\{
(\vec{r}\cdot\vec{r}) - 2(\vec{p}\cdot\vec{r}) + (\vec{p}\cdot\vec{p})\right\}
$$
The unit vector $\vec{a}$ can be written as:
$$
\vec{a} = \left[
\cos(\alpha)\cos(\gamma),\cos(\alpha)\sin(\gamma),\sin(\alpha)\right]
$$
Where $\alpha$ is the angle between the cone axis and the XY-plane and $\gamma$
is an angle that indicates how the conic section is rotated in the plane.
The vector of the top of the cone can be written in its coordinates as:
$$
\vec{p} = (p,q,h)
$$
Where $h$ is the height of the cone above the XY plane and $(p,q)$ indicates
how the conic section is translated in the plane. Last but not least, the vector
pointing to the cone surface is written as:
$$
\vec{r} = (x,y,z)
$$
Where the intersections with the XY plane are found for $z = 0$. Let's do just
that and work out the above:
$$
\begin{cases}
(\vec{a}\cdot\vec{r}) &=&
\cos(\alpha)\cos(\gamma)\,x + \cos(\alpha)\sin(\gamma)\,y \\
(\vec{a}\cdot\vec{p}) &=&
\cos(\alpha)\cos(\gamma)\,p + \cos(\alpha)\sin(\gamma)\,q + \sin(\alpha)\,h \\
(\vec{r}\cdot\vec{r}) &=& x^2 + y^2 \\
(\vec{p}\cdot\vec{r}) &=& p\,x + q\,y \\
(\vec{p}\cdot\vec{p}) &=& p^2 + q^2 + h^2
\end{cases}
$$
Collecting powers of $x$ and $y$ results in:
$$
A\,x^2 + B\,xy + C\,y^2 + D\,x + E\,y + F = 0
$$
Where:
$$
\begin{cases}
A &=& \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \\
B &=& - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \\
C &=& \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma) \\
D &=& 2 \left\{ \cos(\alpha)\cos(\gamma) (\vec{a}\cdot\vec{p})
- \cos^2(\phi)\,p \right\} \\
E &=& 2 \left\{ \cos(\alpha)\sin(\gamma) (\vec{a}\cdot\vec{p})
- \cos^2(\phi)\,q \right\} \\
F &=& (\vec{p}\cdot\vec{p}) \cos^2(\phi) - (\vec{a}\cdot\vec{p})^2
\end{cases}
$$
Meaning
The first three coefficients of the conic section equation are:
$$
\begin{cases}
A &=& \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \\
B &=& - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \\
C &=& \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma)
\end{cases}
$$
All kind of conics can still be produced if the angles $\phi$ and $\alpha$ are
limited to sensible values:
$$
\begin{cases}
&&0 < \phi < 90^o \quad \Longrightarrow \quad 0 < \cos(\phi) < 1 \\
&&0 \le \alpha \le 90^o \quad \Longrightarrow \quad 0 \le \cos(\alpha) \le 1
\end{cases}
$$
Generality is not affected by these choices. Moreover it is seen from the picture
below that the form of the conic section is determined
by the angles $\phi$ and $\alpha$ and nothing else. Therefore the ratio of the
two angles will be defined here as the excentricity ($\epsilon$) of the
conic section:
$$
\epsilon = \frac{\cos(\alpha)}{\cos(\phi)}
$$
The following relationships exist between the excentricity and the form of a
conic section, as is clear from the picture:
$$
\begin{cases}
\mbox{Circle : }& \alpha = 90^o &\quad \Longleftrightarrow \quad \epsilon = 0 \\
\mbox{Ellipse : }& \alpha > \phi &\quad \Longleftrightarrow \quad \epsilon < 1 \\
\mbox{Parabola : }& \alpha = \phi &\quad \Longleftrightarrow \quad \epsilon = 1 \\
\mbox{Hyperbola : }& \alpha < \phi &\quad \Longleftrightarrow \quad \epsilon > 1
\end{cases}
$$
So far so good. The coefficients $(A,B,C)$ can be combined into
some interesting quantities which are only dependent upon form,
that is: the angles $\phi$ and $\alpha$.
It is remarked in the first place that $(A,B,C)$ are independent of the vector
$\vec{p} = (p,q,h)$ and thus independent of translation and scaling. If we seek
to eliminate any dependence upon the angle of rotation $\gamma$, then we find:
$$
A + C = 2 \cos^2(\phi) - \cos^2(\alpha) = \cos^2(\phi) (2 - \epsilon^2)
$$
This quantity is known (for some good reasons) as the trace of the conic
section. Instead of eliminating the angle of rotation, we could also try
to calculate it.
$$
A - C = - \cos^2(\alpha)\left[\cos^2(\gamma)-\sin^2(\gamma)\right]
= - \cos^2(\alpha)\cos(2\gamma)
$$
There is a striking resemblance with:
$$
B = - \cos^2(\alpha)\sin(2\gamma)
$$
We thus find:
$$
\frac{B}{A - C} = \frac{\sin(2\gamma)}{\cos(2\gamma)} \quad \Longrightarrow \quad
\tan{2\gamma} = \frac{B}{A - C}
$$
Herewith - in principle - the angle of rotation $\gamma$ can be reconstructed
from the conic section equation; provided that $A \neq C$.
Let's proceed with
another quantity that is independent of any rotation.
$$
B^2 - 4 A C = \left[ - 2 \cos^2(\alpha)\cos(\gamma)\sin(\gamma) \right]^2
$$ $$
- 4 \left[ \cos^2(\phi) - \cos^2(\alpha)\cos^2(\gamma) \right]
\left[ \cos^2(\phi) - \cos^2(\alpha)\sin^2(\gamma) \right]
$$
This quantity is known (also for some good reasons) as the determinant or
discriminant of the conic section. Work out:
$$
= 4 \cos^4(\alpha)\cos^2(\gamma)\sin^2(\gamma) - 4 \cos^4(\phi)
- 4 \cos^4(\alpha)\cos^2(\gamma)\sin^2(\gamma) \\
+ 4 \cos^2(\phi)\cos^2(\alpha)\left[\cos^2(\gamma)+\sin^2(\gamma)\right] \\
= - 4\cos^4(\phi) + 4\cos^2(\phi)\cos^2(\alpha) \\
\quad \Longrightarrow \quad B^2 - 4 A C
= 4\cos^2(\phi)\left[\cos^2(\alpha) - \cos^2(\phi)\right]
= \left[2\cos^2(\phi)\right]^2(\epsilon^2 - 1)
$$
The following relationships exist between the discriminant and the form of a
conic section, as is clear from the above:
$$
\begin{cases}
&\mbox{Ellipse}& \quad \Longleftrightarrow \quad \epsilon < 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) < 0 \\
&\mbox{Parabola}& \quad \Longleftrightarrow \quad \epsilon = 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) = 0 \\
&\mbox{Hyperbola}& \quad \Longleftrightarrow \quad \epsilon > 1 \quad \Longleftrightarrow \quad (B^2 - 4 A C) > 0
\end{cases}
$$
BONUS on excentricity.
$$
(A + C)^2 = 4\cos^2(\phi)\left[\cos^2(\phi) - \cos^2(\alpha)\right]
+ \cos^4(\alpha)
$$
Upon addition this gives:
$$
(B^2 - 4 A C) + (A + C)^2 = \cos^4(\alpha) \quad \Longrightarrow \\
\cos(\alpha) = \sqrt{\sqrt{B^2 + (A-C)^2}}
$$
About the angle $\phi$ between the cone axis and its surface:
$$
A + C = 2\cos^2(\phi) - \sqrt{B^2 + (A-C)^2} \quad \Longrightarrow \quad
\cos(\phi) = \sqrt{\frac{(A+C) + \sqrt{B^2 + (A-C)^2}}{2}}
$$
Herewith the excentricity $\epsilon$ can be expressed into the coefficients of
the conic section equation $(A,B,C)$:
$$
\epsilon = \sqrt{\frac{2\sqrt{B^2 + (A-C)^2}}
{(A+C) + \sqrt{B^2 + (A-C)^2}}}
$$
Many other expressions can be derived, especially for the coefficients $D,E,F$.
But I think that deriving them here will disturb a good balance between interesting and cumbersome.
EDIT. Explanation of
$$
\vec{a} = \left[
\cos(\alpha)\cos(\gamma),\cos(\alpha)\sin(\gamma),\sin(\alpha)\right]
$$
See picture. Overline denotes " length of " in:
$$
\overline{OZ}=\overline{OA}\, \sin(\alpha) \quad ; \quad
\overline{OD}=\overline{OA}\, \cos(\alpha) \\
\overline{OX}=\overline{OD}\, \cos(\gamma) \quad ; \quad
\overline{OY}=\overline{OD}\, \sin(\gamma)
$$
With $\overline{OA}=1$ .
The "complete the square" method allows you to center the conic, i.e. make the linear terms vanish (by a translation, $ax^2+bx+c\to a'u^2+c'$).
To deal with "obliqueness", you need to let the cross term $xy$ vanish, by means of a rotation.
Let $x=cu-sv,y=su+cv$, which expresses a rotation around the origin ($c,s$ denote the cosine and sine of the angle), and let the centered equation be in the form
$$Ax^2+2Bxy+Cy^2=1.$$
Then substituting,
$$(Ac^2+2Bcs+Cs^2)u^2+(-2Acs+B(c^2-s^2)+2Ccs)uv+(As^2-2Bcs+Cc^2)v^2=1.$$
By a suitable choice of the angle, you can achieve
$$-2Acs+B(c^2-s^2)+2Ccs=0$$
and the equation reduces to
$$A'u^2+C'v^2=1.$$
Depending on the signs of $A',C'$, you get an ellipse or an hyperbola. By a further rescaling of the variables, you can obtain the "canonical" forms (circle and equilateral hyperbola)
$$p^2\pm q^2=1.$$
To find the suitable angle, use the "double angle" formulas and rewrite
$$(C-A)\sin(2\theta)+B\cos(2\theta)=0,$$ which is easy to solve.
Best Answer
Clearly, if $h^2-ab>0$, $y$ is unbounded and the curve is open (hyperbola). It negative, it is bounded and closed (ellipse). If zero, $y$ is unbounded only on one side.