$R$ is a reflection in the line $y=2x$ with matrix $\begin{pmatrix}-0.6&0.8\\ 0.8&0.6\end{pmatrix}$
$S$ is a $90$ degree rotation anticlockwise about the origin.
$T$ is a stretch, scale factor $3$, parallel to the y-axis.
Find the invariant lines, if any, of:
$S^{-1}RS$ and $TR$
So I started by calculating what $T$ and $S$ are:
$S$ = $\begin{pmatrix}0&-1\\ 1&0\end{pmatrix}$ and $T$ = $\begin{pmatrix}1&0\\ 0&3\end{pmatrix}$
Then $S^{-1}RS = \begin{pmatrix}0.6&-0.8\\ -0.8&-0.6\end{pmatrix}$ and
$u'\:=\:0.6u-0.8v,\:v'\:=\:-0.8u-0.6v$
I don't know what to do from here however. I'm not told whether the line passes through the origin so I don't want to assume the line has equation $y=mx$.
Best Answer
The invariant linear subspaces of $\Bbb{R}^2$ for a linear transformation represented by a matrix are the ones that involve the matrix's eigenvectors. In the case of $S^{-1}RS$ the eigenvectors are: $$v_1 = (1, 2) \qquad v_2 = (-2, 1)$$ This means that the lines $y=mx$ with $m= 2, -1/2$ are surely invariant under this linear transformation. The eigenvalue relative to $v_2$ is $1$, which means that any invariant line translated by a multiple of $v_2$ is again going to be invariant.
In short, this matrix represents a reflection along the axis $y=-1/2 x$.
Another way you can view this is using the actual effects of the matrices. A point on the line $y=-1/2 x$ is orthogonal to the original matrix's axis of reflection. Applying $S$ will rotate it $90$ degrees counterclockwise, bringing it on the line $y=2x$. Applying the reflection will not move the point, so that applying $S^{-1}$ is going to bring it right back where it started.
The eigenvalues for the second transformation are... Odd. So, let's think of the solution in terms of the effects. First of all, if a line does not go through the origin, it will necessarily move after the two actions: the reflection will change the sign of the $y$-intercept, and the stretch will alter its value but keep the sign; so we may rule those out.
Let $x$ be the angle our line forms with the $x$-axis. The action of the first sends this angle to: $$x \to 2 \text{arctan}(2) - x $$ As you can easily check. The action of the second sends any vertical length, say $l$, to $3l$. So we expect the value $\tan \theta$ to be mapped on $3 \tan \theta$. Since we are looking for the angle that goes back to being the same after both these transformations, we are looking for the solutions to the equation: $$ \tan x = 3 \tan (2h - x)$$ Where of course $h=\text{arctan}2$.
One of those solutions is pretty close to $1.3129$; you may check that the line $y=\tan (1.3129)x $ almost does not move after being reflected and stretched.