First note that the quadrilaterial $PAOB$ is a kite, this follows from the fact that the tangent from one point to a circle are from a same length, and the $AO=OB=r$.
From the fact that $\angle AOB = 60^{\circ}$ and using $AO=OB$ we get that the $\triangle AOB$ is equilaterial. This leads to conclusion $AB=r=1$, which is the smaller diagonal from the kite. Now we need to find the other one.
Let $D$ be the point when both diagonals intersect, and from the propertiy of the kite we know that they are normal to each other. So we can split the diagonal $PO$ into $PO = OD + PD$. We know that the $OD$ is the height of the equilaterial triangle so we have:
$$OD = \frac{r\sqrt{3}}{2} = \frac{\sqrt{3}}{2}$$
We know that the angle between the tangent line to a circle and the radius at the touching point is $90^{\circ}$, so $\angle PAO = 90^{\circ}$. We can denote $\angle PAO$ as sum of two angles: $\angle PAO = \angle OAB + \angle BAP = 60^{\circ} + \angle BAP$, because $\angle OAB$ is an angle in an equilaterial triangle. So from this we have:
$$\angle BAP = 30^{\circ}$$
Also we know that $AD = \frac r2 = \frac 12$. Now in the right triangle $PAD$ we have:
$$\tan \angle BAP = \frac{PD}{AD}$$
$$\tan \angle 30^{\circ} = \frac{PD}{\frac 12}$$
$$ \frac {1}{\sqrt{3}} = \frac{PD}{\frac 12}$$
$$ PD = \frac{1}{2\sqrt{3}}$$
Now we have: $PO = OD + PD$, i.e.:
$$PO = \frac{\sqrt{3}}{2} + \frac{1}{2\sqrt{3}}$$
$$PO = \frac{3}{2\sqrt{3}} + \frac{1}{2\sqrt{3}}$$
$$PO = \frac{4}{2\sqrt{3}}$$
$$PO = \frac{2}{\sqrt{3}} \approx 1.154$$
So the distance from $O$ to $P$ is constant, that means that $P$ can be any point that's $\frac{2}{\sqrt{3}}$ units away from $(0,0)$, i.e. $P$ lies on a circle with this equation:
$$x^2 + y^2 =\left(\frac{2}{\sqrt{3}}\right)^2$$
$$x^2 + y^2 = \frac 43$$
So the locus of the point $P$ is:
$$x^2 + y^2 = \frac 43$$
Given Circle
The fixed circle described in your problem is actually
$$(x-2)^2+(y-\frac{\sqrt{5}}{2})^2=\frac{9}{4}$$
This is because its tangent circle is $(x-1)^2+y^2=9$ is centered in $(1,0)$ which lies on the given circle, which means that its diameter is equal to the radius of the tangent circle, so it is $\frac{3}{2}$.
The $x$ coordinate $a$ of the center is 2 because $(1,0)$ and $(3,0)$ lie on the circle.
The $y$ coordinate $b$ of the center can be found by plugging $(1,0)$ into the equation $(x-2)^2+(y-b)^2=\frac{9}{4}$.
Circumcenter
The circumcenter of $\triangle PAB$ is the midpoint of $PO$ where $O=(a,b)$ is the center of the given circle. This is because $\angle PAO=\frac{\pi}{2}$.
Locus
For the point $P=(x,y=cos(x))$, the circumcenter is $C=(\frac{x+a}{2},\frac{y+b}{2})$ so its locus is a modified cosine line:
$$ Y = \frac{\cos(2X-a)+b}{2}$$
where $a=2$ and $b=\frac{\sqrt{5}}{2}$.
Best Answer
As @user0 and @dan_fulea mentioned in their comments above, the circumcircle of $PQR$ passes through the origin, and therefore, the circumcenter will be the midpoint of $OP$. Therefore, if the circumcenter is $C$, then
$C = \dfrac{1}{2} P $
Since $P$ traces a line, then $C$ also traces a line. In particular the line that $C$ traces will be parallel to the $P$ line.
Since $P$ is on the line $ 5x + 4y = 20 $, then $C$ will be on the line
$ 5 x + 4 y = 10 $