A point moves so that the distance between the feet of the perpendiculars drawn from it to the lines $ax^2+2hxy+by^2=0$ is a constant $c$. Prove that the equation of its locus is
$$4(x^2+y^2)(h^2-ab)=c^2(4h^2+(a-b)^2).$$
What I have done: Suppose $P$ is any point on the locus, $A$ and $B$ be the feet of the perpendiculars, and $O$ is the origin. Then I want to show that
$$OP=\frac{AB}{\sin AOB}=\frac{c}{\sin \theta}$$
where $\theta$ is the angle between the two lines.
Note that the arc $PAO$ is a semi-circle, since $\angle PAO=90$. Similarly the arc $PBO$ is a semi-circle since $\angle PBO=90$. Also $\angle APB=\angle AOB$, as it is the angle of the same arc.
Now, $\tan \theta =\frac{2\sqrt{h^2-ab}}{a+b}$. From here I can find $\sin \theta$ and $OP=\sqrt{x^2+y^2}$. Putting all these in,
$$OP=\frac{c}{\sin \theta}$$
will give me the equation of the locus. The problem is how to show that
$$OP=\frac{AB}{\sin AOB}.$$
Best Answer
Please note that $OAPB$ is cyclic quadrilateral and $OP$ is the diameter of the circle. Also both lines pass through origin.
So if $(x, y)$ is the coordinates of $P$, radius of the circle is $ \displaystyle r = \frac{\sqrt{x^2 + y^2}}{2}$.
If angle between two lines is $\theta$, the angle subtended by chord $AB$ on the center is $2\theta$. We then have,
$ \displaystyle AB = 2 r \sin \theta \implies \sin\theta = \frac{c}{ \sqrt{x^2+y^2}}$ (as $AB = c$)
Now if the equations of lines are $l_1: a_1 x + b_1 y = 0$ and $l_2: a_2 x + b_2 y = 0$
Multiplying both equations and equating to $ax^2 + 2hxy+b y^2 = 0$,
$a_1a_2 = a, \ b_1b_2 = b, \ a_1b_2 + a_2b_1 = 2h \tag1$
Also $ \displaystyle m_1 = - \frac{a_1}{b_1}, m_2 = - \frac{a_2}{b_2}$
So, $ \displaystyle \tan \theta = \left|\frac{m_2 - m_1}{1 + m_1 m_2}\right| = \left|\frac{a_1 b_2 - a_2b_1}{a_1a_2 + b_1b_2}\right| = \left|\frac{ \sqrt{(a_1 b_2 + a_2b_1)^2 - 4 a_1 a_2 b_1 b_2}}{a_1a_2 + b_1b_2}\right|$
With help of equation $(1)$, $ \displaystyle \tan \theta = \left|\frac{ \sqrt{4h^2 - 4 ab}}{a + b}\right|$
Using $ \displaystyle \sin^2 \theta = \frac{\tan^2\theta}{1+\tan^2\theta} \ $, $ \ \displaystyle \frac{c^2}{x^2+y^2} = \frac{4h^2 - 4ab}{4h^2 + (a-b)^2}$
EDIT: seeing Jean Marie's nice diagrams, I realize adding an example to go with my answer may help visualize the situation better. I do not have such nice tools with me so my diagram is basic Desmos :)
Before we get to the example, using the formula we derived, it is easy to see that the locus is a circle centered at the origin. Now why origin? Simply because the question uses pair of lines intersecting at the origin. In other words, if the lines intersected at a different point, that point will be the center of the circle.
Let's take pair of lines given by $x^2 - 5xy + 4y^2 = 0$. So we have $a = 1, b = 4, h = - \frac{5}{2}$.
The lines are $x - y = 0, x - 4y = 0$.
So using the formula we derived earlier, the locus of the point in this example is,
$ \displaystyle x^2 + y^2 = \frac{34 c^2}{9}$. See the Desmos diagram for $c = 1$. From any point on this circle, if we draw perpendicular to the given pair of lines, the distance between the feet of both perpendicular will be $1$.