It can also be proved using geometry only.
1] Through $M$, draw $MX \parallel BC$ cutting $AC$ at $X$. From that, we have
- (1.1) $\beta = \beta’$; and
- (1.2) $AX = XC$.
2] Through $X$, draw $XYZ \parallel AB$ cutting $CH$ at $Y$ and $CM$ at $Z$. From that, we have
- (2.1) $XY$ is the perpendicular bisector of $CH$; and
- (2.2) $Z$ is the midpoint of $CM$.
(2.1) + (1.1) implies $\alpha’ = \alpha = \beta = \beta’$. Since $H$ and $M$ are distinct ($\triangle ABC$ is not isosceles), the inscribed angle theorem means $XHMC$ is a cyclic quadrilateral. This also gives us that $\angle MXC = \angle MHC = 90^\circ$.
Then since $MX \parallel BC$, also $\angle ACB = 90^\circ$ as required.
The proof given below is done in two steps. In the first step, we prove that the circumcircles of the three equilateral triangles drawn outwardly on sides of an arbitrary $ABC$ have a common point. The second step consists of the arguments which prove that the centroids of the aforementioned equilateral triangles are the vertices of another equilateral triangle.
In our proof, we use the fact that the centroid of an equilateral triangle coincides with the center of its circumcircle.
In $\mathrm{Fig. 1}$, we have drawn the circumcircles of the equilateral triangles $BPC$ and $CQA$.
They intersect each other at $C$ and $F$ and, therefore, $CF$ is the common chord of the two shown circumcircles.
To facilitates the proof, we need to join $BF$ and $AF$. Since $BPCF$ and $AFCQ$ are cyclic quadrilaterals, we have,
$$\measuredangle CFB = \measuredangle AFC = 120^o.$$
Therefore, $\measuredangle BFA = 120^o$, which means that $ARBF$ is also a cyclic quadrilateral, because two of its opposite angles, namely $\measuredangle BFA$ and $\measuredangle ARB$, are supplementary.
Consequently, the circumcircles of the three equilateral triangles drawn outwardly on sides of $ABC$ have $F$ as their common point.
$\mathrm{Fig. 2}$ shows the triangle obtained by joining the centroids $O_A$, $O_B$, and $O_C$.
Our objective is to prove that each of the angles of this triangle is equal to $60^o$.
Now, draw the two common chords $BF$ and $CF$ of the circumcircle pairs {$O_A,\space O_C$} and {$O_A,\space O_B$} respectively.
The sides of the triangle in question $O_A O_C$ and $O_A O_B$ are perpendicular to the two common chords $BF$ and $CF$ respectively.
The lines $O_A O_B$ and $CF$ meets at $P$, while the lines $O_A O_C$ and $BF$ intersect each other at $Q$.
We have already shown that $\measuredangle CFB=120^o$. Now, consider the quadrilateral $O_A PFQ$.
Its opposite angles $\measuredangle O_A PF$ and $\measuredangle FQO_A$ are supplementary.
Therefore, $O_A PFQ$ is a cyclic quadrilateral.
Consequently, its other pair of opposite angles $\measuredangle QO_A P$ and $\measuredangle PFQ$ are also supplementary.
Therefore,
$$\measuredangle QO_A P + \measuredangle PFQ = 180^o,$$
which means, that
$$\measuredangle QO_A P = 180^o - \measuredangle PFQ = 180^o - 120^o = 60^o.$$
Similarly, we can show that any of the other two angles of the triangle $O_A O_B O_C$ is equal to $60^o$,
which is necessary and sufficient to conclude that it is indeed an equilateral triangle.
Best Answer
Following your work, we have
$$r = \frac{\sin\frac{\pi}{3}}{\sin\left(\frac{\pi}{3} - \phi\right)} = \frac{\sin\left(\frac{\pi}{3} + \phi\right)}{\sin\frac{\pi}{3}}$$
The last two terms give $\sin\left(\frac{\pi}{3} - \phi\right)\sin\left(\frac{\pi}{3} + \phi\right) = \frac{3}{4}$. We have
$$\begin{align*} LHS &= \left(\frac{\sqrt{3}}{2}\cos\phi - \frac{1}{2}\sin\phi\right)\left(\frac{\sqrt{3}}{2}\cos\phi + \frac{1}{2}\sin\phi\right) \\ &= \frac{3}{4}\cos^2\phi - \frac{1}{4}\sin^2\phi \end{align*}$$
Setting equality, we have
$$\cos^2\phi - \frac{1}{3}\sin^2\phi = 1$$
From this, it is clear that we need $\sin^2\phi = 0$, so there is no non-degenerate triangle satisfying your requirements (assuming your sine law equations are correct :)