Given $\triangle ABC$, and a scale factor $r \lt 1 $, I want to find the necessary rotation (center and angle) such that the rotated/scaled version of the triangle has its vertices lying on the sides of $\triangle ABC$.
To clarify what I want. Suppose the vertices $A,B,C$ are in counter-clockwise order. I want to scale these points (homothety) about a certain point $P$ by a scale factor $r < 1$, and then rotate them by an angle $\theta$ about the SAME point $P$, such that the images of homothety $A',B',C'$ simultaneously land (after rotation) on $AB, BC, CA$ respectively, or simultaneously land on $AC, BC, AB$ respectively.
How to do this ? Are there known results on this ?
Best Answer
Here's a discussion of the case where each of the triangle's vertices is moved to its "opposite" side.
Define points $A'$, $B'$, $C'$ on the side-lines of $\triangle ABC$ by these Ceva-esque ratios:
$$\alpha := \frac{|A'C|}{|BA'|} \qquad \beta := \frac{|B'A|}{|CB'|} \qquad \gamma := \frac{|C'B|}{|BC'|} \tag1$$
Also, define $\theta$ as the (signed, non-obtuse) angle made by lines $A'B'$ and $AB$ (and by $B'C'$ and $BC$, and by $C'A'$ and $CA$). A little angle-chasing tells us that the remaining angles in the triangles surrounding $\triangle A'B'C'$ have the form $A\pm\theta$, $B\pm\theta$, $C\pm\theta$, as shown:
Therefore, by the Law of Sines, and the presumed similarity of $\triangle ABC$ and $\triangle A'B'C'$, we have
$$\alpha =\frac{\frac{c'}{\sin C}\sin(A-\theta)}{\frac{b'}{\sin B}\sin(A+\theta)}=\frac{\sin(A-\theta)}{\sin(A+\theta)} \qquad \beta= \frac{\sin(B-\theta)}{\sin(B+\theta)}\qquad \gamma=\frac{\sin(C-\theta)}{\sin(C+\theta)}\tag{2}$$
Further, since $a = |BA'|+|A'C|$, we find $$\begin{align} a &= \frac{c'}{\sin C}\sin(A-\theta)+\frac{b'}{\sin B}\sin(A+\theta) \tag3\\[4pt] &= \frac{a'}{\sin A}(\sin A\cos\theta-\cos A\sin\theta) +\frac{a'}{\sin A}(\sin A\cos\theta+\cos A\sin\theta) \tag4 \end{align}$$ $$ \to\qquad\frac{a}{a'} \left(=\frac{b}{b'}=\frac{c}{c'}\right) = 2\cos\theta \tag5 $$
From $\theta$ we easily get the position and size of $\triangle A'B'C'$.
Sanity check: When $AB\parallel A'B'$, so that $\theta=0^\circ$, we have $\alpha=\beta=\gamma=1$ and $a/a'=2$. Here, $\triangle A'B'C'$ is the well-known midpoint (or medial) triangle of $\triangle ABC$. In the context of the problem, it is the image of $\triangle ABC$ after rotation by $180^\circ$ and dilation by $1/2$, about the triangles' common centroid. A nice start.
Now, while our triangles are similar, as desired, it's not at all obvious that (apart from the nice case, $\theta=0^\circ$) the former is the image of rotating-and-scaling the latter in a particular ($\theta$-dependent) center. And yet, it is.
As the ostensible center (call it $P_\theta$) would need to be a fixed point proportionally-located within each triangle, we can equate generic (barycentric) weighted averages of the vertices: $$\frac{uA+vB+wC}{u+v+w} = \frac{uA'+vB'+wC'}{u+v+w} \tag6$$ Solving, we find these barycentric coordinates of $P_\theta$: $$\begin{align} u:v:w &\;=\; 4 - \left(1 + \frac{\tan\theta}{\tan B}\right) \left(1 - \frac{\tan\theta}{\tan C} \right)\\ &\quad: 4 - \left(1 + \frac{\tan\theta}{\tan C}\right) \left(1 - \frac{\tan\theta}{\tan A} \right) \\ &\quad: 4 - \left(1 + \frac{\tan\theta}{\tan A}\right) \left(1 - \frac{\tan\theta}{\tan B} \right) \end{align} \tag7$$ The transformation center lacks the necessary symmetry to be a proper "triangle center" in the Kimberling sense. (For instance, we typically can't simply swap $B$ and $C$ in the $u$ component.) Of course, for $\theta=0$, we re-reconfirm the nice case, with $P_0$ being $\triangle ABC$'s centroid.
Also, something interesting happens as $\theta$ approaches a right angle: the $u:v:w$ components in $(4)$ become infinite, but as a proportion, they approach a limiting case of $\tan A:\tan B:\tan C$ (simply multiply-through by $\tan A\tan B\tan C/\tan^2\theta$). These are the barycentric coordinates of $\triangle ABC$'s orthocenter!
Further, and this seems worth highlighting:
The radius (say, $p$) of this circle is given by $$p^2 = r^2 - \frac19(a^2+b^2+c^2) \tag8$$ The center of the circle is a proper triangle center; namely (or, rather, numberly), Kimberling's $X(381)$ ... helpfully identified in the Encyclopedia of Triangle Centers as "MIDPOINT OF X(2) [centroid] AND X(4) [orthocenter]". Jokes aside, the ETC proves its worth again by documenting that $X(381)$ is the center of the "orthocentroidal circle", giving a previously-unknown-to-me name to the locus of $P_\theta$, in turn providing additional leads for research. Indeed, I readily got to $X(381)$ from ETC's mentions in the centroid ($X(2)$) and orthocenter ($X(4)$) entries that each was the reflection of the other in that point. So ... Thanks (again), Clark!
BTW: The linked Wikipedia article doesn't mention how this circle relates to centers of rotation-and-scaling of inscribed similar triangles; indeed, the counterpart MathWorld article asserts "The circle does not pass through any notable centers other than [the centroid and orthocenter]." (Granted, $P_\theta$ isn't a triangle center, but it seems pretty notable.) You'd think that the fact that every point on the circle is one of these $P_\theta$s would be worth mentioning. Perhaps we've stumbled onto a new result in triangle lore?
Anyway, here's an animation:
Points $G$ and $H$ are the original triangle's centroid and orthocenter, respectively.
I'll note in closing that $\theta$ is somewhat directly related to the location of $P_\theta$ on the orthocentroidal circle; writing $M$ for that circle's center, $$\tan\theta \;=\; 3\tan\tfrac12\angle P_\theta MG \tag9$$ There's probably a clever geometric justification for the $3$, but I don't see it ... yet.
(Then again, the first pass at this answer —check the Edit History— overlooked some rather obvious geometry in establishing $(2)$ and $(5)$, so I'll probably be embarrassed to have missed this relation, too!)
The case of where each of the triangle's vertices is moved to an adjacent side of the triangle is considered in Section 2.5 "Center of the Spiral Symilarity for similar triangles" of Art of Problem Solving's "Spiral Similarity" page. There, it is noted that all such transformations have a common center, a Brocard Point of the triangles (whether the "First" or "Second" of these points depends upon how $\triangle A'B'C'$ is situated within $\triangle ABC$).
For completeness, I'll sketch how my approach above can be used to analyze this case.
Let $A_{-}$, $B_{-}$, $C_{-}$ lie on the side-lines $CA$, $AB$, $BC$. (The "$-$" is meant to evoke the that the triangle has been rotated in the opposite to the orientation $A\to B\to C$.) Let $\theta$ be the (signed) angle made by vectors $\overrightarrow{AB}$ and $\overrightarrow{A_{-}B_{-}}$ (and by $\overrightarrow{BC}$ and $\overrightarrow{B_{-}C_{-}}$, and by $\overrightarrow{CA}$ and $\overrightarrow{C_{-}A_{-}}$).
The Ceva-esque ratios for these points are a bit less elegant than before:
$$\alpha := \frac{|C_{-}C|}{|BC_{-}|} = \frac{\sin^2B \sin\theta}{\sin C\sin A \sin(B+\theta)} \qquad \beta = \cdots \qquad \gamma = \cdots \tag{10}$$
as is the scale factor $$\frac{a}{a_{-}} = \frac{b}{b_{-}} = \frac{c}{c_{-}} = \frac{\sin A \cos(A-\theta) + \sin B \cos(B-\theta) + \sin C \cos(C-\theta)}{2\sin A \sin B \sin C} \tag{11}$$ but the barycentric coordinates of the center of similarity, $P_{-}$, seem to more-than-make-up for this: $$u:v:w \;=\; \csc^2 C : \csc^2 A : \csc^2 B \tag{12}$$ Not only are the coordinates far simpler, they're independent of $\theta$; each spiral symmetry uses this center. (Here, $P_{-}$ is the second Brocard Point; the companion $\triangle A_{+}B_{+}C_{+}$ would have $P_{+}$ as its first Brocard Point.)
Here's an animation:
Note that the angle of rotation exactly matches $\theta$; no half-turn adjustment needed.
Addendum. These spiral symmetries only work for "directly"-similar inscribed triangles, because rotation-and-scaling cannot reverse the orientation of $\triangle A'B'C'$ relative to $\triangle ABC$. You need a linear reflection for that.
... like ... for instance ... reflection in the Euler line of $\triangle ABC$, which conveniently passes through the orthocenter and centroid, and hence bisects the orthocentroidal circle that proved useful earlier.
Well, as it happens ...
With appropriately-related angles and scale factors (details of which I'll omit for now), one can rotate-and-scale $\triangle ABC$ in some point on the orthocentroidal circle, and reflect in the Euler line, to get an inscribed, indirectly-similar $\triangle A'B'C'$.
Note: With such indirect-similarity, exactly one vertex travels to the opposite side-line of the triangle.