I think you overworked here. The condition is
$$|AP|^2=9|BP|^2\iff (x+3a)^2+y^2=9\left[(x-a)^2+y^2\right]\iff$$
$$\iff 8x^2-24ax+8y^2=0\iff \left(x-\frac{3a}2\right)^2+y^2=\frac{9a^2}4$$
and we get that the points$\;P=(x,y)\;$ are the locus of circle with center $\;\left(\frac{3a}2\,,\,0\right)\;$ and radius $\;\frac{3a}2\;$ , and since radius = absolute value of $\;x$ - coordinate of circle, the circle is tangent to the $\;y$ - axis
Let $\;Q=(x,y)\;$ , then its distance from the $\;y$ - axis is $\;|x|\;$ , and the length of any of its two tangets to the above circle is
$$\sqrt{\left(x-\frac{3a}2\right)^2+y^2\;-\;\frac{9a^2}4}$$
and thus we get the equation:
$$x^2=\left(x-\frac{3a}2\right)^2+y^2\;-\;\frac{9a^2}4\iff$$
$$\iff -3ax+y^2=0\;\;(*)$$
Now, let us check what is the locus of all point whose distance to $\;4x+3a=0\iff x=-\frac{3a}4\;$ equals its distance to $\;\left(\frac{3a}4\,,\,0\right)\;$:
$$\left(x-\frac{3a}4\right)^2+y^2=\frac{(4x+3a)^2}{16}\iff x^2-\frac{3a}2x+\frac{9a^2}{16}+y^2=x^2+\frac{3a}2x+\frac{9a^2}{16}\iff$$
$$\iff -3ax+y^2=0\;\;(*)$$
and thus both conditions are identical
Perhaps the examiner intended the students to notice the square is determined by a $(3, 4, 5)$ triangle, because $3 + 5 = 4 + 4$ (!):
![A circle and square](https://i.stack.imgur.com/a5W4P.png)
Consequently, as several others have noted,
$$
\frac{\text{perimeter of the circle}}{\text{perimeter of the square}}
= \frac{5 \cdot 2\pi}{4 \cdot 8} = \frac{\pi}{3.2}
< 1.
$$
For an approach less dependent on inspiration, taking the origin of the coordinate system at the center of the circle seems easier than placing the origin at the center of the square. Without loss of generality, assume the circle has unit radius:
![Expressing the dimensions of the square](https://i.stack.imgur.com/hLgFX.png)
Equating the lengths of the horizontal and vertical sides of the square in this diagram, we read off
$$
x + 1 = 2y\quad\text{(or $x = 2y - 1$).}
$$
Invoking the Pythagorean theorem and substituting the preceding line,
\begin{align*}
0 &= x^{2} + y^{2} - 1 \\
&= (2y - 1)^{2} + y^{2} - 1 \\
&= 5y^{2} - 4y \\
&= y(5y - 4).
\end{align*}
Clearly $y \neq 0$, so $y = 4/5$, $x = 3/5$, and we notice the Examiner's Favorite Triangle.
Best Answer
Without loss of generality, let the distance between $A$ and $B$ equal 3 (this is for convenience in calculations) and suppose that $\overline{AB}$ is horizontal (this just makes it easier to talk about where things are).
Now, the point $P_1$ that is on $\overline{AB}$ a distance of 2 from $A$ and 1 from $B$ is in the locus, as are points $P_2$ and $P_3$ that are directly above and below $B$, a distance of $\sqrt{3}$ from $B$ and $2\sqrt{3}$ from $A$. Using these three points, we can determine that if the locus were a circle, its center would have to be at a point we'll call $C$, on $\overrightarrow{AB}$ a distance of 4 from $A$ and 1 from $B$ (1 unit past $B$ from $A$).
Next, consider a point $P$ in the locus with $PA=2x$, $PB=x$, and $PC=y$. Apply Stewart's Theorem to get $$\begin{align} 3y^2+(2x)^2&=4(x^2+3) \\ 3y^2+4x^2&=4x^2+12 \\ y^2=4 \\ y=2. \end{align}$$ So, all possible points $P$ in the locus are a distance of 2 from $C$, which means that the locus is a circle of radius 2, centered at $C$.
n.b. This suggests that if $A=(a,0)$ and $B=(-a,0)$, $C=(-2a,0)$ and the radius is $2a$, which would give an equation of $(x+2a)^2+y^2=4a^2$, which is different from what you said you got.