What we will show: If we square the heights of the flagpoles, then the line requested in the problem is the line where the plane containing the three new flagpole tops meets the ground.
Caveat: This solution is clearly not the one intended by the problem author, and it does not proceed via the big equation given in the problem. It does not even use any particular coordinate system.
Part 1: The soldier marches in a circle
Let's start by confirming that the soldiers march in circles.
We will assume all flagpoles have unequal heights.
If flagpole $A$ is twice as high as flagpole $B$, then the soldier $S$ must be twice as far from $A$ as from $B$ for the flagpoles to appear to be the same height (meaning that the angles of elevation to the tops of $A$ and $B$ are the same).
More generally,
if flagpole $A$ has height $a$ and flagpole B has height $b$,
and we write $\overline{SA}$ for the distance from $S$ to $A$,
then we have
$\overline{SA}\,/\,a = \overline{SB}\,/\,b$,
Squaring this we get
$ \overline{SA}^2 \,/\, a^2 = \overline{SB}^2 \,/\, b^2 $.
We can think of $ \overline{SA}^2 $ as a function of $S$,
which, if we plot it, is a paraboloid centered at $A$.
Recall some trivial paraboloid facts:
Paraboloids are quadratics with the same coefficient for $x^2$ as for $y^2$, so if we add or subtract two paraboloids, we get another paraboloid.
(Or a plane, if the resulting quadratic coefficient is zero.)
Also, multiplying a paraboloid by a constant yields another paraboloid.
If the quadratic coefficient is negative, the paraboloid curves down instead of up.
(If you are familiar with mass points,
paraboloids (ignoring their constant term) are just like mass points,
regarding addition and multiplication.)
If we rewrite our equation as
$\frac{1}{a^2} \overline{SA}^2 - \frac{1}{b^2} \overline{SB}^2 = 0$,
we see that the left hand side is a paraboloid. (Not a plane, since $a≠b$.)
The desired locus of points is those points $S$ for which this paraboloid equals zero. Since paraboloids are circularly symmetric, every level plane of a paraboloid is a circle, so this confirms that the soldier marches in a circle.
Part 2: Where is the center of the circle?
By symmetry,
when we add two paraboloids $G$ and $H$,
the center of $G+H$ must lie on the line connecting the centers of $G$ and $H$.
Where on this line will it lie?
If we look at the paraboloid's height along this line,
it is the sum of the contribution from $G$ (which is a parabola)
and the contribution from $H$ (another parabola),
so the sum is also a parabola.
The center of $G+H$ lies at the extremal point of this parabola.
If we look at the line $AB$ so $B$ is to the right of $A$,
we can consider the slope of the summed parabola
$\frac{1}{a^2} \overline{SA}^2 - \frac{1}{b^2} \overline{SB}^2$,
so as to find its extremal point by finding where the slope is zero.
At point $A$, the contribution from $A$'s parabola has slope 0,
so the slope of the sum is determined by $B$'s parabola,
whose slope there is $\frac{2 \overline{AB}}{b^2}$.
Similary, at $B$, the slope is $\frac{2 \overline{AB}}{a^2}$.
We can linearly extrapolate from these two slopes to find the extremum,
since the slope of a parabola varies linearly.
The extremum yields the center of the circle where the soldier is marching.
(Again, if you are familiar with mass points,
this diagram just shows regular 1-D mass point addition
of a negative mass $\frac{-b^2}{2 \overline{AB}}$ at $A$ and a positive mass $\frac{a^2}{2 \overline{AB}}$ at $B$.)
This figure gives the impression of being yet another diagram of two flagpoles!
Multiplying the y-axis by
$\frac{a^2 b^2}{2 \overline{AB}}$
makes the height over $A$ be $a^2$,
and makes the height over $B$ be $b^2$,
while leaving the circle center (the x-intercept) unchanged.
Part 3: Collinearity of the circle centers
This follows immediately.
If we square the heights of the three flagpoles, then each circle center lies at the point $S$ from which the new flagpole tops appear (to the soldier) to coincide, as in the diagram above.
So the line requested in the problem is the line where the plane containing the three new flagpole tops meets the ground.
Best Answer
Area is proportional to the square of linear dimension. So the area of the full cone is $k(3^2)$ for some $k$. The area of the second largest segment is $k(2^2) - k(1^2)$, so the ratio is $3:9 = 1:3$. You are right, and the book is wrong.