The locus
After doing some computations in the spirit (and building on the code) of my vertex anser, I found the locus of $P$ to be a quadric
$$ a\,x^2 + b\,y^2 + c\,xy + d\,x + e\,y + f = 0 $$
with the following parameters:
\begin{align*}
a&=4\left(-A_y\,B_y + B_y^2 + A_y\,C_y - B_y\,C_y\right)\\
b&=4\left(-A_x\,B_x + B_x^2 + A_x\,C_x - B_x\,C_x\right)\\
c&=4\left(A_y\,B_x + A_x\,B_y - 2\,B_x\,B_y - A_y\,C_x + B_y\,C_x - A_x\,C_y + B_x\,C_y\right)\\
d&=4\left(A_y\,B_x\,B_y - A_x\,B_y^2 + A_y\,B_y\,C_x - B_y^2\,C_x - 2\,A_y\,B_x\,C_y + A_x\,B_y\,C_y + B_x\,B_y\,C_y\right)\\
e&=4\left(-A_y\,B_x^2 + A_x\,B_x\,B_y + A_y\,B_x\,C_x - 2\,A_x\,B_y\,C_x + B_x\,B_y\,C_x + A_x\,B_x\,C_y - B_x^2\,C_y\right)\\
f&=A_y^2\,B_x^2 - 2\,A_x\,A_y\,B_x\,B_y + A_x^2\,B_y^2 - 2\,A_y^2\,B_x\,C_x + 2\,A_x\,A_y\,B_y\,C_x \\&\quad{}
- 2\,A_y\,B_x\,B_y\,C_x + 2\,A_x\,B_y^2\,C_x + A_y^2\,C_x^2 - 2\,A_y\,B_y\,C_x^2 + B_y^2\,C_x^2 \\&\quad{}
+ 2\,A_x\,A_y\,B_x\,C_y + 2\,A_y\,B_x^2\,C_y - 2\,A_x^2\,B_y\,C_y - 2\,A_x\,B_x\,B_y\,C_y \\&\quad{}
- 2\,A_x\,A_y\,C_x\,C_y + 2\,A_y\,B_x\,C_x\,C_y + 2\,A_x\,B_y\,C_x\,C_y - 2\,B_x\,B_y\,C_x\,C_y \\&\quad{}
+ A_x^2\,C_y^2 - 2\,A_x\,B_x\,C_y^2 + B_x^2\,C_y^2
\end{align*}
Assumption confirmed
Using this formulation, I could verify your assumption: the conic is indeed a hyperbola, with the lines $AB$ and $CB$ as asymptotes. The way to check this is by ensuring that the lines are tangents, and that they touch the conic at infinity.
Special location
In a comment below, you mention that you might assume $B_x=B_y=0$. With that your conic will become
\begin{align*}
a&=4\,A_y\,C_y\\
b&=4\,A_x\,C_x\\
c&=-4\left(A_y\,C_x + A_x\,C_y\right)\\
d&=0\\
e&=0\\
f&=A_y^2\,C_x^2 - 2\,A_x\,A_y\,C_x\,C_y + A_x^2\,C_y^2
\end{align*}
The formula is indeed a lot easier, so it might be a good starting point for a geometric description of the curve. For example, it is readily apparent that this hyperbola will be symmetric around the origin. But we already knew $B$ to be the center due to the asymptotes. You can even take this one step further and, via an affine transformation, consider only the case where
\begin{align*}
A &= \begin{pmatrix}1\\0\end{pmatrix} &
B &= \begin{pmatrix}0\\0\end{pmatrix} &
C &= \begin{pmatrix}0\\1\end{pmatrix}
\end{align*}
You end up with the hyperbola
$$4xy=1$$
From this you can e.g. conclude that the midpoint between $A$ and $C$ will also lie on that hyperbola. It will of course correspond to a Bézier curve which passes through $B$ for some $t<0$ or $t>1$. You can take that midpoint and reflect it in $B$ to obtain a point on the segment of the hyperbola you get for $0\le t\le 1$. This construction is invariant under affine transformations, so it still holds for the general case. Two asymptotes plus one point on the hyperbola amounts to five real degrees of freedom, so this should be enough to uniquely define your hyperbola.
Summary
To sum it up: the locus of the apex $P$ is the unique hyperbola with asymptotes $AB$ and $CB$ which passes through the midpoint between $A$ and $C$. The portion of it which corresponds to parabolas where $B$ lies between $A$ and $C$, i.e. is obtained in the Bézier curve for $0<t<1$, is the component of the hyperbola which does not contain that midpoint. It does contain the point obtained by reflecting that midpoint in $B$.
The code
Here is the sage code I used to obtain this representation:
# Define multivariate polynomial ring and points
PR1.<A_x, A_y, B_x, B_y, C_x, C_y, P_x, P_y, t> = QQ[]
A = vector(PR1, [A_x, A_y, 1])
B = vector(PR1, [B_x, B_y, 1])
C = vector(PR1, [C_x, C_y, 1])
P = vector(PR1, [P_x, P_y, 1])
# Quadratic Bézier curve parametrized by t
Bt = (1-t)^2*A + 2*(1-t)*t*P + t^2*C
r1 = (Bt[0] - B_x).resultant(Bt[1] - B_y, t) # eliminate t
# Obtain coefficients for coordinates of P
c1 = vector(PR1, flatten([list(i.polynomial(i.parent()(P_y)))
for i in r1.polynomial(P_x)]))
f, e, b, d, c, a = c1
# Print result
fmt1 = [str(i/4).replace('*','\\,')
for i in [a, b, c, d, e]] + [str(f)]
fmt2 = [i + '&=4\\left(' + j + '\\right)'
for i, j in zip('abcdef', fmt1)]
fmt2[-1] = 'f&='+fmt1[-1]
print('\\\\\n'.join(fmt2))
# Check whether lines AB and CB are asymptotes of the Hyperbola
Hyperbola = Matrix([
[2*a, c, d],
[c, 2*b, e],
[d, e, 2*f]])
def onConic(p, c=Hyperbola):
return (p.row()*c*p.column())[0,0].is_zero()
asymptotes = [B.cross_product(p) for p in [A, C]]
infLine = vector(QQ, [0,0,1])
# asymptotes are tangents to the hyperbola:
assert(all(onConic(i, Hyperbola.adjoint()) for i in asymptotes))
# asymptotes touch the hyperbola at infinity:
assert(all(onConic(i.cross_product(infLine)) for i in asymptotes))
# The midpoint between A and C is on the hyperbola:
assert(onConic(A+C))
Originally I had more complicated code which did not rely on the interpretation as a Bézier curve. The result was the same, though.
Generalizing to rational case
Can we generalize to rational quadratic Bezier curves (i.e. to conic section curves that are not necessarily parabolas)?
A parabola has four real degrees of freedom. If you choose a non-rational Bézier curve, you have two real degrees of freedom, but with these you not only specify a parabola but also select a start point and an end point on that parabola, so the degrees of freedom match. A conic in general has five real degrees of freedom. So if you want it to pass through three given points, that still leaves a two-parameter family of corresponding conics. Therefore your locus will not be a single curve, but either the whole plane or some portion of it.
You can define a conic using five points through which it should pass. In addition to your $A,B,C$ you might use two more points, which you move close to the end points $A$ and $C$. By making them arbitrary close (i.e. computing some limit), you can use these control points to exactly and arbitrarily determine the direction of the tangents in $A$ and $C$. Therefore you can choose any point $P$ in the plance, and find a conic through $A,B,C$ which will have $P$ as its apex. In this sense, the whole plane will be your locus.
I'm not completely sure whether a rational Bézier curve can be defined in such a way that it passes through infinity, but I believe that to be the case. If not, then there might be cases where the resulting conic would be a hyperbola, and $A,B,C$ are not all three in the same of its components. This might result in a restriction to part of the plane.
Best Answer
What your teacher was referring to is the fact that:
$$x=\dfrac{a t^2+bt+c}{gt^2+ht+k}, y=\dfrac{d t^2+et+f}{gt^2+ht+k}, \ \ \ 0 \le t \le 1 \tag{1}$$
(with real coefficients $a,b,c,d,e,f,g,h,k$)
is in general a conic curve under the form of a rational quadratic Bezier curve.
This property comes from the vast domain of projective geometry.
The classification is easy: either the denominator
can be zero twice (which happens when its discriminant $\Delta=h^2-4gk > 0$) it means that there are two points at infinity: you have a hyperbola (a hyperbola has 4 points at infinity, but in projective geometry opposite points are considered as identical).
can be zero once ($\Delta=0$): parabola.
can never be zero ($\Delta<0$): ellipse.
In order to retrieve the driving points $A,B,C$, just decompose the numerators onto the basis $(1-t)^2, 2t(1-t), t^2$, which means finding $x_A, x_B,...$ such that:
$$\begin{cases}a t^2+bt+c&=&(1-t)^2 x_A + 2t(1-t)x_B+t^2x_C\\d t^2+et+f&=&(1-t)^2 y_A + 2t(1-t)y_B+t^2y_C\end{cases}$$
Remark: Any bijective change of parametrization $T=f(t)$ in (1) such that
$$f(0)=0 \ \ \text{and} \ \ f(1)=1\tag{2}$$
is possible. Taking a "Möbius" transformation
$$T=\frac{t}{rt+(1-r)}$$
complying with (2) preserves the form of equations (1) and allows, for an adequate choice of parameter $r$, to obtain (slightly) simpler expressions.
Recall: Equations (1) can be written under the form:
$$\underbrace{\begin{bmatrix} X\\ Y\\ Z\end{bmatrix}}_{C'}=\underbrace{\begin{bmatrix} a & b & c\\ d & e & f\\ g & h & k \end{bmatrix}}_H \underbrace{\begin{bmatrix} t^2\\ t\\ 1 \end{bmatrix}}_C \tag{2}$$
followed by :
$$x=X/Z \ \ \text{and} \ \ y=Y/Z \tag{3}$$
Equation (2) expresses that one takes the image of "standard" parabola with parametric description given by vector $C$ (with so-called "homogeneous coordinates" : please note the third coordinate equal to $1$) to which an "homography matrix" $H$ is applied. Equations (3) are the classical homogenizing ratios by the third coordinate. See Fig. below.
Knowing that the image by a homography (either under the form (1) or (2)+(3)) of a conic curve is a conic curve.
See as well here.