The equation $$ax^2+2hxy+by^2=c$$ represents a conic rotated about the axis (depending on $a,b,h,c$) with centre $(0,0)$.
You can always rotate the axis to remove the $xy$ term. This makes it easy to find the focus, length of major axis, etc.
To remove $xy$ term without shifting origin, lets rotate the axis by an angle $\theta$.
Let $X,Y$ represent the new coordinate system. Then,
$$x=X\cos\theta-Y\sin\theta$$
$$y=X\sin\theta+Y\cos\theta$$
Substituting and letting coefficient of $XY=0$, you get $$\tan2\theta=\frac{2h}{a-b}$$
For example, lets find $e$ of the following equation (which represents an ellipse): $$4x^2+2y^2+2\sqrt{3}xy=1$$
Raotate the axis by $$\tan2\theta=\frac{2h}{a-b}=\sqrt{3}$$
$$\theta=\frac{\pi}{6}$$
Now, substitute $$x=X\frac{\sqrt{3}}{2}-Y\frac{1}{2}$$
$$y=Y\frac{\sqrt{3}}{2}+X\frac{1}{2}$$
You will get $$5X^2+Y^2=5$$
Now you can find eccentricity using the formula you specified. (Rotating the ellipse or hyperbola will not change the eccentricity and the length of major\minor axis.
This is the graph for the two equations:
(Black represents the new ellipse)
Now, to prove these definitions are equivalent, we’re going to take the first definition and make an equation out of it. We will then manipulate it a little. Then, we’ll do the same for the second definition and we’ll get the second equation into the form of the first equation. Once we’ve done that, we will prove that the eccentricities are equal for both definitions of the eccentricity.
For the first definition, let’s say the focus is $(a, b)$ and the directrix is $Ax+By+C=0$ (this is not the same $A$ and $C$ as discussed in the first definition). The distance to the focus is:
$$\sqrt{(x-a)^2+(y-b)^2}$$
The distance to the line is:
$$\frac{\lvert Ax+By+C \rvert}{\sqrt{A^2+B^2}}$$
However, by dividing $Ax+By+C=0$ by some factor, we can make $\sqrt{A^2+B^2}$ equal to 1, so we can get rid of the denominator.
The ratio of the distance to the point to the distance to the line is the eccentricity, which we will call $e$, so we have:
$$\frac{\sqrt{(x-a)^2+(y-b)^2}}{\lvert Ax+By+C \rvert}=e$$
Now, take the first equation and multiply both sides by the denominator to get rid of the fraction:
$$\sqrt{(x-a)^2+(y-b)^2}=e\lvert Ax+By+C \rvert$$
Square both sides to simplify things:
$$(x-a)^2+(y-b)^2=(eAx+eBy+eC)^2$$
Note that since $\sqrt{A^2+B^2}=1$, we have that:
$$e=e\sqrt{A^2+B^2}=\sqrt{(eA)^2+(eB)^2}$$
Thus, the eccentricity is the square root of the sum of the squares of the coefficients of $x$ and $y$ on the right side. This will become relevant later.
For the second definition, let’s say one focus is $(a, b)$ and the other is $(c, d)$. The sum/difference of the distances is $D$, so we have:
$$\sqrt{(x-a)^2+(y-b)^2}\pm\sqrt{(x-c)^2+(y-d)^2}=D$$
Again, $+$ is for ellipses, $-$ is for hyperbola.
Also, since $F$ is the distances between the foci, we know that:
$$F=\sqrt{(a-c)^2+(b-d)^2}$$
Square both sides:
$$F^2=(a-c)^2+(b-d)^2$$
Divide both sides by $D^2$:
$$\left(\frac F D\right)^2=\left(\frac{a-c}{D}\right)^2+\left(\frac{b-d}{D}\right)^2$$
Take the square root of both sides and substitute $e=\frac F D$:
$$e=\sqrt{\left(\frac{a-c}{D}\right)^2+\left(\frac{b-d}{D}\right)^2}$$
This will come up again later on.
Now, back to where we were before this tangent. Square both sides:
$$\left(x-a\right)^2+\left(y-b\right)^2+\left(x-c\right)^2+\left(y-d\right)^2\pm2\sqrt{\left(\left(x-a\right)^2+\left(y-b\right)^2\right)\left(\left(x-c\right)^2+\left(y-d\right)^2\right)}=D^2$$
Get the radicand alone on the right side:
$$\left(x-a\right)^2+\left(y-b\right)^2+\left(x-c\right)^2+\left(y-d\right)^2-D^2=\mp2\sqrt{\left(\left(x-a\right)^2+\left(y-b\right)^2\right)\left(\left(x-c\right)^2+\left(y-d\right)^2\right)}$$
Square both sides:
$$\left(\left(x-a\right)^2+\left(y-b\right)^2\right)^2+\left(\left(x-c\right)^2+\left(y-d\right)^2\right)^2+2\left(\left(x-a\right)^2+\left(y-b\right)^2\right)\left(\left(x-c\right)^2+\left(y-d\right)^2\right)+D^4-2D^2\left(\left(x-a\right)^2+\left(y-b\right)^2+\left(x-c\right)^2+\left(y-d\right)^2\right)=4\left(\left(x-a\right)^2+\left(y-b\right)^2\right)\left(\left(x-c\right)^2+\left(y-d\right)^2\right)$$
Note that the $\mp$ has been removed by squaring. Now, subtract both sides by the expression on the right side:
$$\left(\left(x-a\right)^2+\left(y-b\right)^2\right)^2+\left(\left(x-c\right)^2+\left(y-d\right)^2\right)^2-2\left(\left(x-a\right)^2+\left(y-b\right)^2\right)\left(\left(x-c\right)^2+\left(y-d\right)^2\right)+D^4-2D^2\left(\left(x-a\right)^2+\left(y-b\right)^2+\left(x-c\right)^2+\left(y-d\right)^2\right)=0$$
Use the $a^2+b^2-2ab=(b-a)^2$ identity to simplify and rearrange the terms so it looks more like a quadratic in terms of $D^2$:
$$D^4-2D^2\left(\left(x-a\right)^2+\left(y-b\right)^2+\left(x-c\right)^2+\left(y-d\right)^2\right)+\left(\left(x-c\right)^2+\left(y-d\right)^2-\left(x-a\right)^2-\left(y-b\right)^2\right)^2=0$$
Fix the $D^2$ term so that it has the $(x-c)^2+(y-d)^2-(x-a)^2-(y-b)^2$ that’s in the constant term:
$$D^4-2D^2\left(\left(x-c\right)^2+\left(y-d\right)^2-\left(x-a\right)^2-\left(y-b\right)^2\right)-2D^2\left(2\left(x-a\right)^2+2\left(y-b\right)^2\right)+\left(\left(x-c\right)^2+\left(y-d\right)^2-\left(x-a\right)^2-\left(y-b\right)^2\right)^2=0$$
Add both sides by the new term that is not part of the quadratic:
$$D^4-2D^2\left(\left(x-c\right)^2+\left(y-d\right)^2-\left(x-a\right)^2-\left(y-b\right)^2\right)+\left(\left(x-c\right)^2+\left(y-d\right)^2-\left(x-a\right)^2-\left(y-b\right)^2\right)^2=2D^2\left(2\left(x-a\right)^2+2\left(y-b\right)^2\right)$$
Use the $a^2-2ab+b^2=(a-b)^2$ identity to simplify the left side. Also, factor out the $2$ from the right side:
$$\left(\left(x-c\right)^2+\left(y-d\right)^2-\left(x-a\right)^2-\left(y-b\right)^2-D^2\right)^2=4D^2\left(\left(x-a\right)^2+\left(y-b\right)^2\right)$$
Simplify the left side more by expanding the quadratics and subtracting like terms. Also, write $(2D)^2$ on the right side:
$$\left(2\left(a-c\right)x+c^2-a^2+2\left(b-d\right)y+d^2-b^2-D^2\right)^2=\left(2D\right)^2\left(\left(x-a\right)^2+\left(y-b\right)^2\right)$$
Now, divide both sides by $(2D)^2$:
$$\left(\frac{a-c}{D}x+\frac{b-d}{D}y+\frac{c^2-a^2+d^2-b^2-D^2}{2D}\right)^2=\left(x-a\right)^2+\left(y-b\right)^2$$
Switch both sides of the equation:
$$\left(x-a\right)^2+\left(y-b\right)^2=\left(\frac{a-c}{D}x+\frac{b-d}{D}y+\frac{c^2-a^2+d^2-b^2-D^2}{2D}\right)^2$$
Now, we have the equation in the same form as the first equation. However, we still need to check that the definition of eccentricity from the first equation is equal to the definition of eccentricity from this equation. As we said above from the first equation, $e$ is the square root of the sum of the squares of the coefficients of $x$ and $y$ on the right side, which means that here, according to the first equation, the eccentricity is:
$$e=\sqrt{\left(\frac{a-c}{D}\right)^2+\left(\frac{b-d}{D}\right)^2}$$
However, this is the exact same eccentricity we derived for the second equation, so the eccentricities are equal.
Thus, we are done and both definitions are equivalent with equivalent eccentricities for both ellipses and hyperbolas.
Best Answer
One of your roots corresponds to a hyperbola, which is what you want. The second root corresponds to an ellipse. The ellipse in the second root has an eccentricity equal to $1$ divided by the eccentricity of the hyperbola you want. A key difference between the two conic sections is that a hyperbola has its directrices between the foci, while for an ellipse it's the other way around.
To orient yourself, find the value of $c$ such that $x-3y+c=0$ at the given focus. This, of course, is $-15$. The given directrix has $c=-4$, which is greater. So now, if you try the root $c=-256/9<-15$ for the other directrix, you have the focus between the directrices which corresponds to that ellipse. To get the hyperbola you must select $c>-4$ for the other directrix, putting the given directrix ($c=-4$) between the focus ($c=-15$) and the directrix you're trying to find. Therefore select $c=-14/9$.