Let's approach the problem via a simplified case, where $(x_c,y_c)$ is at origin, and $(x_1, y_1)$ is on the $x$ axis at $(a, 0)$:
Obviously, we can calculate $a$ from the original coordinates,
$$a = \sqrt{\left(x_1 - x_c\right)^2 + \left(y_1 - y_c\right)^2}$$
We have three unknowns, $x$, $y$, and $b$ (the distance from origin to $(x,y)$), and three equations:
$$\begin{cases}
x = b \cos \theta \\
y = b \sin \theta \\
(x - a)^2 + y^2 = c^2
\end{cases}$$
There is a pair of solutions:
$$\begin{cases}
x = a \left(\cos\theta\right)^2 \pm \cos\theta \sqrt{c^2 - a^2 \left(\sin\theta\right)^2} \\
y = a \sin\theta \cos\theta \pm \sin\theta \sqrt{c^2 - a^2 \left(\sin\theta\right)^2} \\
b = a \cos\theta \pm \sqrt{c^2 - a^2\left(\sin\theta\right)^2}
\end{cases}$$
Pick either the upper or the lower signs for all three, but note that only the triplet for which $b \ge 0$ is actually valid. Indeed, for my illustration above, I've shown the "-" solution; the "+" solution would have $(x,y)$ somewhere near $(a,c)$, making the lower right corner angle somewhat near ninety degrees.
However, now that we do know (possibly two valid values of) $b$, we can go back to looking at the situation in the original coordinates.
Numerical solutions, using atan2()
:
The simplest way is to use the atan2()
function available in most programming languages. ($\operatorname{atan2}(y,x) = \tan(y/x)$, except the former also takes account the quadrant, too.)
With it, in original coordinates,
$$\begin{cases}
b = a \cos\theta \pm \sqrt{c^2 - a^2\left(\sin\theta\right)^2}, & b \ge 0 \\
\theta_0 = \operatorname{atan2}(y_1 - y_c, x_1 - x_c) \\
x_3 = x_c + b \cos \left ( \theta_0 + \theta \right ) \\
y_3 = y_c + b \sin \left ( \theta_0 + \theta \right )
\end{cases}$$
If you want positive $\theta$ to be clockwise, use $(\theta_0 - \theta)$ instead in the formulas for $x_3$ and $y_3$, above.
Symbolic solutions, via coordinate system transformation:
A two-dimensional rotation matrix is defined as
$$\mathbf{R} = \left[\begin{matrix}\cos\varphi & -\sin\varphi \\ \sin\varphi & \cos\varphi\end{matrix}\right] = \left[\begin{matrix}C&-S\\S&C\end{matrix}\right]$$
and rotating a point $\vec{p} = (x, y)$ by matrix $\mathbf{R}$ is
$$\mathbf{R} \vec{p} = \left[\begin{matrix}C&-S\\S&C\end{matrix}\right]\left[\begin{matrix}x\\y\end{matrix}\right]$$
i.e.
$$\begin{cases}
x' = C x - S y \\
y' = S x + C y
\end{cases}$$
In this particular case, we need to rotate our simplified case solutions using a matrix which rotates point $(a,0)$ to $(x_1-x_c, y_1-y_c)$:
$$\begin{cases}
x_1 - x_c = C a \\
y_1 - y_c = S a
\end{cases} \iff \begin{cases}
C = \frac{x_1 - x_c}{a} \\
S = \frac{y_1 - y_c}{a}
\end{cases}$$
Applying the above rotation to our simplified case results, and a translation to move $(x_c, y_c)$ back to its proper place from origin, we get:
$$\begin{cases}
b = a \cos\theta \pm \sqrt{c^2 - a^2\left(\sin\theta\right)^2}, & b \ge 0 \\
x = b \cos\theta \\
y = b \sin\theta \\
C = \frac{x_1 - x_c}{a} \\
S = \frac{y_1 - y_c}{a} \\
x_3 = x_c + C x - S y \\
y_3 = y_c + S x + C y
\end{cases}$$
or equivalently
$$\begin{cases}
b = a \cos\theta \pm \sqrt{c^2 - a^2\left(\sin\theta\right)^2}, & b \ge 0 \\
x_3 = x_c + \frac{b}{a}(x_1 - x_c)\cos\theta - \frac{b}{a}(y_1 - y_c)\sin\theta \\
y_3 = y_c + \frac{b}{a}(y_1 - y_c)\cos\theta + \frac{b}{a}(x_1 - x_c)\sin\theta
\end{cases}$$
or equivalently
$$\begin{cases}
z = \frac{b}{a} = \cos\theta \pm \sqrt{\frac{c^2}{a^2} - \left(\sin\theta\right)^2}, & z \ge 0 \\
x_3 = x_c + z(x_1 - x_c)\cos\theta - z(y_1 - y_c)\sin\theta \\
y_3 = y_c + z(y_1 - y_c)\cos\theta + z(x_1 - x_c)\sin\theta
\end{cases}$$
.
Addendum added to react to the comment question of the original poster
The computations can be simplified by temporarily assuming that one endpoint of the hypotenuse has Cartesian coordinates $(0,0)$ and the other endpoint has Cartesian coordinates
$\displaystyle (R,0) ~: R = \sqrt{A^2 + B^2} = \sqrt{(x_1 - x_0)^2 + (y_1 - y_0)^2}.$
Then, at this stage, all that is missing is the Cartesian Coordinate of the $3$rd vertex. The easy way is to construe the line segment going from the origin to the 3rd vertex as a vector of length $R\cos(\alpha)$ and direction $(\alpha)$
Here, I am making the standard assumption that the positive directions of angles are in a counter-clockwise direction.
If you then take scratch paper, and draw the corresponding diagram, you will see that the Cartesian coordinates of the $3$rd vertex are therefore
$$\left[R\cos^2(\alpha), ~~R\cos(\alpha)\sin(\alpha)\right]. \tag1 $$
All that remains it to rotate and then shift the diagram, and compute the resulting affect on the Cartesian coordinates of the $3$rd vertex, which is currently shown in (1) above.
Under the initial temporary assumption, the slope of the line $\left(\overline{AB}\right)$ is $0$.
The actual slope needs to be
$$M = \frac{y_1 - y_0}{x1 - x_0}.$$
So,
If $M > 0$, then let $\theta = \arctan(M).$
If $M < 0$, then let $\theta = (\pi) - \arctan(-M) \implies $
$\pi/2 < \theta < \pi.$
An easy way of then performing the rotation is to repeat the procedure that was initially used to compute the Cartesian coordinates shown in (1) above. Instead of construing the pertinent leg of the triangle to be a vector of magnitude $R\cos(\alpha)$ and direction $(\alpha)$, simply construe the pertinent leg of the triangle to be a vector of magnitude $R\cos(\alpha)$ and direction $(\alpha + \theta)$.
Then, the rotational affect is to alter the Cartesian coordinates, shown in (1) above. Instead, they become
$$(P,Q) = \left[R\cos(\alpha)\cos(\alpha + \theta), ~~R\cos(\alpha)\sin(\alpha + \theta)\right]. \tag2 $$
So, in (2) above, $(P,Q)$ represents the Cartesian coordinate of the $3$rd vertex, after the rotational adjustment, but before the shift adjustment.
The shift adjustment is straightforward. Assuming that $(0,0)$ is shifted to $(x_0, y_0)$, you have that
$$(P,Q) ~~\text{is shifted to}~~ (P + x_0, Q + y_0).$$
Addendum
The OP (i.e. original poster) indicated a flaw in my response. What happens if $(x_1 - x_0)$ equals $0$. In that case, the attempt to compute of $M$ will fail, so the strategy used to determine $(\theta)$ will fail.
That situation represents a vertical hypotenuse, which represents a rotation of $(\pi/2)$. So, in that case, set $(\theta)$ to $(\pi/2)$.
Best Answer
Use the fact that the perpendicular bisectors of segments $p_1 p'_1 $ and $p_2 p'_2$ pass through the center of rotation $p_0$. Therefore, if
$n_1 = p'_1 - p_1 $
$n_2 = p'_2 - p_2 $
and
$m_1 = \dfrac{1}{2} (p'_1 + p_1 ) $
$m_2 = \dfrac{1}{2} (p'_2 + p_2 ) $
and
$p_0 = (x, y) $
then
$n_1^T (p_0 - m_1) = 0 $
$n_2^T (p_0 - m_2 ) = 0 $
So that by matrix inversion, we get the center of rotation $p_0$
$p_0 = \begin{bmatrix} n_1^T \\ n_2^T \end{bmatrix}^{-1} \begin{bmatrix} n_1^T m_1 \\n_2^T m_2 \end{bmatrix} $
To get the angle of rotation, find the angle between the vector $p'_1 - p_0$ and $p_1 - p_0$. This angle is equal to the angle between the vectors $p'_2 - p_0$ and $p_2 - p_0$.