First note that the quadrilaterial $PAOB$ is a kite, this follows from the fact that the tangent from one point to a circle are from a same length, and the $AO=OB=r$.
From the fact that $\angle AOB = 60^{\circ}$ and using $AO=OB$ we get that the $\triangle AOB$ is equilaterial. This leads to conclusion $AB=r=1$, which is the smaller diagonal from the kite. Now we need to find the other one.
Let $D$ be the point when both diagonals intersect, and from the propertiy of the kite we know that they are normal to each other. So we can split the diagonal $PO$ into $PO = OD + PD$. We know that the $OD$ is the height of the equilaterial triangle so we have:
$$OD = \frac{r\sqrt{3}}{2} = \frac{\sqrt{3}}{2}$$
We know that the angle between the tangent line to a circle and the radius at the touching point is $90^{\circ}$, so $\angle PAO = 90^{\circ}$. We can denote $\angle PAO$ as sum of two angles: $\angle PAO = \angle OAB + \angle BAP = 60^{\circ} + \angle BAP$, because $\angle OAB$ is an angle in an equilaterial triangle. So from this we have:
$$\angle BAP = 30^{\circ}$$
Also we know that $AD = \frac r2 = \frac 12$. Now in the right triangle $PAD$ we have:
$$\tan \angle BAP = \frac{PD}{AD}$$
$$\tan \angle 30^{\circ} = \frac{PD}{\frac 12}$$
$$ \frac {1}{\sqrt{3}} = \frac{PD}{\frac 12}$$
$$ PD = \frac{1}{2\sqrt{3}}$$
Now we have: $PO = OD + PD$, i.e.:
$$PO = \frac{\sqrt{3}}{2} + \frac{1}{2\sqrt{3}}$$
$$PO = \frac{3}{2\sqrt{3}} + \frac{1}{2\sqrt{3}}$$
$$PO = \frac{4}{2\sqrt{3}}$$
$$PO = \frac{2}{\sqrt{3}} \approx 1.154$$
So the distance from $O$ to $P$ is constant, that means that $P$ can be any point that's $\frac{2}{\sqrt{3}}$ units away from $(0,0)$, i.e. $P$ lies on a circle with this equation:
$$x^2 + y^2 =\left(\frac{2}{\sqrt{3}}\right)^2$$
$$x^2 + y^2 = \frac 43$$
So the locus of the point $P$ is:
$$x^2 + y^2 = \frac 43$$
The locus is a straight line (dotted line in the figure below).
Let $C(1,1)$ be the center of the given circle. Let $E$, $F$ be the contact points of the tangents issued from $P$. Due to the symmetry with respect to line $PC$ of the two tangents, $PEF$ is an isosceles triangle; therefore, the center $G$ of the circumscribed circle, which is situated on the altitude issued from $P$ in triangle $PEF$ is aligned with $P$ and $C$.
Moreover, $G$ is the midpoint of segment PC for the following reason: radius $EC$ being orthogonal to tangent $PE$ in contact point $P$, $\widehat{PEC}$ is a right angle, thus $PC$ is a diameter of the circumscribed circle; therefore $G$ is the midpoint of $PC$.
The locus of such a midpoint is the homothetic line from the original line in the homothety $\frak{H}$$(C,1/2)$, thus a parallel line to the original line.
Remark: The equation of this line is necessarily of the form $\ell x + my +n_1=0$ because they share the same normal vector $(\ell,m)$ with the original line. It remains to find $n_1$. This can be done by using the distance formula from point $C$ to the two straight lines (https://en.wikipedia.org/wiki/Distance_from_a_point_to_a_line).
Best Answer
None of a), b), c) or d) (as written) is the answer.
By using the dotted $3-4-5$ triangle in the picture below, you can capture a convenient point on the locus, then test it in the various equations. The point $E$ in this picture is at $(9/5,0)$, which happens to be on $20(x^2 - y^2)- 36x + 45y = 0$.
If you compute the point $F$ using a similar strategy, you'll find that this point is $(0,-9/4)$ is not on $a)$. However, both points lie on $20(x^2 + y^2)- 36x + 45y = 0$. I guess you typoed $a)$.
In the graph, the smaller circle is my suggested fix for a).
It's not very tough to work out what transformations like this look like. Take a look at this diagram, viewed as being in $\Bbb C$, where $A$ is the origin:
Since $\triangle ACZ$ and $\triangle ACE$ are similar, $|AE|=|AC|\frac{|AC|}{|AZ|}$. In our situation, we may as well start confusing segment lengths for complex norms and write $|AZ|=|Z|$ and $|AC|=3$, so $|AE|=\frac{9}{|Z|}$. To get $E$, we can normalize $Z$ and multiply by the length $|AE|$ to get $\frac{9}{|Z|}\frac{Z}{|Z|}=\frac{9Z}{|Z|^2}$.
The map $Z\mapsto \frac{9Z}{|Z|^2}$ is a Möbius transformation of the complex plane, and as such it will carry the given line onto a circle contained inside the circle radius $3$. It is also its own inverse function.
At the very worst, you can deduce the equation of the circle from the two points given above along with $(0,0)$.
But let's do better and deduce the equation! Let $m$ be the Möbius transformation above. Then if $(x,y)$ lie in the locus, $m(x+iy)$ lies on the given line, and therefore $m(x+iy)$ has to satisfy $4x-5iy=20$. In terms of $\Bbb R^2$, the transformation maps $(x,y)$ to $(\frac{9x}{x^2+y^2},\frac{9y}{x^2+y^2})$. To satisfy $4x-5y=20$, we have:
$$4(\frac{9x}{x^2+y^2})-5(\frac{9y}{x^2+y^2})=20$$
Rewriting:
$$4(9x)-5(9y)=20(x^2+y^2)\\ 36x-45y=20(x^2+y^2)\\ 0=20(x^2+y^2)-36x+45y$$
In standard form: $(x-\frac{9}{10})^2+(y+\frac{9}{8})^2 =\frac{3321}{1600}$