Let's look at a simpler problem. Suppose you have the situation depicted in the figure below:
Then, given the angle $\alpha$, the coordinates of the point $C''$ are:
$$
C''_x = r\cos\alpha
\qquad\mbox{and}\qquad
C''_y = r\sin\alpha
$$
where $r$ is the radius of the circle.
Now let's look at a slightly more complicated problem, depicted below:
This is very similar to the situation above. In fact,
$$
C'_x = r\cos(\alpha+\beta)
\qquad\mbox{and}\qquad
C'_y = r\sin(\alpha+\beta)
$$
By using the trigonometric relations $\sin(\alpha+\beta) = \sin\alpha\cos\beta + \sin\beta\cos\alpha$ and $\cos(\alpha+\beta) = \cos\alpha\cos\beta - \sin\alpha\sin\beta$, we can write the above as follows:
$$
C'_x = r\cos\alpha\cos\beta - r\sin\alpha\sin\beta
\qquad\mbox{and}\qquad
C'_y = r\sin\alpha\cos\beta + r\sin\beta\cos\alpha
$$
But, wait... By looking at the previous situation and replacing $C''$ with $B'$ and $\alpha$ with $\beta$, we see that
$$
B'_x = r\cos\beta
\qquad\mbox{and}\qquad
B'_y = r\sin\beta
$$
Therefore, we can write
$$
C'_x = B'_x\cos\alpha - B'_y\sin\alpha
\qquad\mbox{and}\qquad
C'_y = B'_x\sin\alpha + B'_y\cos\alpha
$$
But what you want is this, instead:
Well, we can just move everything rigidly by the vector $-\vec{OA}$ so that $A$ is now the origin of the coordinate system and we get the situation just above. This amounts to subtracting $A$ from both $B$ and $C$ to get $B'$ and $C'$ in the above, and we find
$$
C_x - A_x = (B_x-A_x)\cos\alpha - (B_y-A_y)\sin\alpha
$$
$$
C_y - A_y = (B_x-A_x)\sin\alpha + (B_y-A_y)\cos\alpha
$$
Then, finally,
$$
C_x = A_x + (B_x-A_x)\cos\alpha - (B_y-A_y)\sin\alpha
$$
$$
C_y = A_y + (B_x-A_x)\sin\alpha + (B_y-A_y)\cos\alpha
$$
Hint:
using elemental geometrical properties of equilateral triangles you can see that:
$$
PC=PA=PB=\frac{10\sqrt{3}}{3}
$$
and the distance of $P$ from the sides is half this value.
Can you do from this adding or subtracting such values from the coordinates of $P$ to find the vertices?
Best Answer
To begin with we may assume the circle $C$, center $(0,0),$ has radius $1$ (divide all coordinates by the circle radius, then remultiply at the end). Rewrite each of $A,B$ in polar coordinates as $(r_a,\theta_a),\ (r_b,\theta_b).$ For each of $A,B$ there are then two points on $C$ where the tangents from $A$ or $B$ meet $C$, say they are at $t_a,t_b$ in the sense that their coordinates are e.g. $P(t_a)=(\cos(t_a),\sin(t_a))$ for $A$ and similarly for $B.$ The values for these $t$ are $$t_a=\theta_a \pm \cos^{-1}(1/r_a) \tag{1}$$ and similarly for $t_b$, as may be seen from a right triangle with one vertex at $A$, another at $P(t_a)$ and the third at the origin $O=(0,0).$ [right angle is at $P(t_a).$] The $\pm$ choice reflects which of the two tangents from $A$ one wants to use.
Once we have made choices for which tangent to use at each point, which amounts to choosing the sign in $(1),$ we have specific values for the angles $t_a$ and $t_b.$ Then the polar coordinates for the intersection point of the two tangents has its angle as $(t_a+t_b)/2$ and its radius as $\sec[(t_a-t_b)/2].$