The exercise asks us to determine whether the given orthogonal matrix represents a rotation or a reflection. If it is a rotation, give the angle of rotation; if it is a reflection, give the line of reflection.
$$
A = \begin{bmatrix}
-\frac{3}{5} & -\frac{4}{5}\\[0.3em]
-\frac{4}{5} & \frac{3}{5}\\[0.3em]
\end{bmatrix}
$$
I know you can check whether it is a reflection or rotation by calculating the determinant. So for example for the matrix above
$$det(A) = -\frac{3}{5}\cdot\frac{3}{5}-(-\frac{4}{5})\cdot(-\frac{4}{5}) = -1$$
And so that means that matrix $A$ corresponds to a reflection in $R^2$, but how do you get the line of reflection from this?
$$
B = \begin{bmatrix}
-\frac{1}{2} & \frac{\sqrt{3}}{2}\\[0.3em]
-\frac{\sqrt{3}}{2} & -\frac{1}{2}\\[0.3em]
\end{bmatrix}
$$
And for a rotation, so for example matrix $B$ given above, can you simply say that it corresponds to the rotation matrix $R$
$$
R = \begin{bmatrix}
cos(\theta) & -sin(\theta)\\[0.3em]
sin(\theta) & cos(\theta)\\[0.3em]
\end{bmatrix}
$$
So this gives
$$cos(\theta) = -\frac{1}{2}, \theta = cos^{-1}(-\frac{1}{2}) = 120^{\circ}\\
sin(\theta) = -\frac{\sqrt{3}}{2}, \theta = sin^{-1}(-\frac{\sqrt{3}}{2}) = -60^{\circ}
$$
Which means that matrix $B$ corresponds to a counterclockwise rotation of $120^{\circ}$, right?
Best Answer
Here is a general guideline for $2 \times 2$ orthogonal matrices.
They have one of the two forms
$$\text{Either} \ \ R = \begin{bmatrix} a &-b\\[0.3em] b & \ \ \ a\\[0.3em] \end{bmatrix} \ \ \ \ \text{or} \ \ \ \ S = \begin{bmatrix} a & \ \ \ b\\[0.3em] b & -a\\[0.3em] \end{bmatrix}$$
with norm $1$ column vectors (thus $a^2+b^2=1$), the first case with $\det(A)=a^2+b^2=1$, the second with $\det(A)=-(a^2+b^2)=-1$.
More precisely, they have the form (you have cited the first one, the second one is less known...): $$R_{\theta} = \begin{bmatrix} \cos(\theta) & -\sin(\theta)\\[0.3em] \sin(\theta) & \ \ \ \cos(\theta)\\[0.3em] \end{bmatrix} \ \ \ \ \ \ \text{or} \ \ \ \ \ \ S_{\alpha}=\begin{bmatrix} \cos(2 \alpha) & \ \ \ \sin(2 \alpha)\\[0.3em] \sin(2 \alpha) & -\cos(2 \alpha)\\[0.3em] \end{bmatrix} $$
where $\theta$ is the rotation angle, of course, and $\alpha$ is the polar angle of the axis or symmetry i.e., the angle of one of its directing vectors with the x-axis.
Thus, for your question, once you have recognized that a matrix is a symmetry matrix, it suffices to pick the upper left coefficient $ \cos(2 \alpha)$ and identify the possible $\alpha$s, with a disambiguation brought by the knowledge of $ \sin(2 \alpha)$.