My goal is to prove that a matrix in $\text{SO}(3)- \{I\}$ is completely determined by its angle of rotation and its axis.
Suppose $A\in \text{SO}(3)$ and let $f$ be the unique linear operator associated to. Then if $A$ is not the identity there's a unique one dimensional subspace $a$ of $\mathbb{R}^3$ such that $v \in a \implies Av=v$.
$a$ is defined to be the axis of rotation.
Now since $a$ is unique given $A\neq I$ so is $a^\perp$, and both are $f$-invariant subspaces. From here how do I define the angle of rotation $\theta$?
Here's a possible approach.
Define the restriction $f'$ of $f$: $f':a^\perp \rightarrow a^\perp$. Since $f$ is associated to an orthogonal matrix it is an isometry, and so is $f'$. Being $f'$ the restriction of a linear map it is also linear. Furthermore $\det(f)=\det(f')=1$ because of the following reasoning:
pick a unitary $v\in a$, and complete $v$ to $\{v,w,u\}$, an orthonormal basis of $\mathbb{R}^3$. The matrix $P$ of change of basis is orthogonal, and $B=P^{-1}AP$, the matrix of $f$ w.r.t. $\{v,w,u\}$, is still orthogonal. It will be of the form $\begin{pmatrix}
1 & 0\\
0 & M
\end{pmatrix}$. Now $B^TB=I$, so $M^TM=I$, and $\det(f')=\det(M)=\det(B)=\det(A)=\det(f)=1$.
So $f'$ is a rotation on the plane, hence it is associated with a unique angle, and so is $A$, if an orientation of the plane $a^\perp$ is given.
Indeed, $M\in \text{SO}(2)$, so $M=\begin{pmatrix}
\cos(\theta) & \sin(\theta)\\
-\sin(\theta) & \cos(\theta)
\end{pmatrix}$. This determines a unique angle $\theta$. How can I be sure this angle doesn't depend on the choice of $\{v,w,u\}$? For sure its magnitude is 'fixed', since $2\cos(\theta)=\text{tr}(M)=-1+\text{tr}(B)=-1+\text{tr}(A)$, so it is invariant under change of basis.
What about the sign? If I chose as a basis $\{v,u,w\}$ instead of $\{v,w,u\}$ then $M$ would still be in $\text{SO}(2)$ by the above result, its trace would be the same, so the only option would be $\begin{pmatrix}
\cos(\theta) & -\sin(\theta)\\
\sin(\theta) & \cos(\theta)
\end{pmatrix}$, corresponding to $-\theta$.
Will define the angle in $(-\pi, \pi]$ solve the problem?
Is it false that a matrix in $\text{SO}(3)- \{I\}$ is completely determined by its angle of rotation and its axis? I could easily prove my statement If I was given the axis with an orientation…
Best Answer
It's true that an axis and angle (together with an orientation of 3-space) determine a unique rotation matrix; Rodrigues' formula provides the mapping from $S^2 \times \Bbb R \to SO(3)$.
It's not true that an element of $SO(3)$ (or, equivalently, a rotation of 3-space, together with an orientation of 3-space) provide a unique axis and angle, at least not if we want the axis to be in $S^2$. But you've at least partly resolved this by saying that the axis is in $\Bbb RP^2$, which is a good choice: a 90 degree rotation about the $z$-axis is also a rotation about the $-z$ axis (so to speak), but in $RP^2$, these are the same "axis".
The problem is that the 90 degree rotation about $z$ is a $-90$ degree rotation about $-z$. So which angle do you use? You've bumped up against this problem, and have seen that it's a problem, but let me ramble on a bit. Let's name a function based on Rodrigues' formula: $$ F: S^2 \times \Bbb R \to SO(3) $$ takes an axis-angle pair to a rotation matrix. There's an action of $G = \Bbb Z / 2 \Bbb Z = \{e,a\}$ on the domain, namely $$ e(v, \theta) = (-v, \theta) $$ The quotient of the domain by this action is $\Bbb RP^2 \times \Bbb R$, and you're hoping that $F$ has an inverse when we look at this "quotient domain."
The problem is that $F$ doesn't "pass to the quotient", i.e., it's not true that $$ F( a \cdot (v, \theta)) = F(v, \theta), $$ in general. So hoping for an inverse in this quotient domain is hopeless.
However there is another action of $G$ on the domain, namely $$ a \cdot (v, \theta) = (-v, -\theta), $$ and with this action, $F$ does pass to the quotient: $$ F(v, \theta) = F(-v, -\theta) $$ Furthermore, if we remove $I$ from the codomain, and restrict the domain to be $$ S^2 \times ( [-\pi, 0) \cup (0, \pi] ), $$ then $F$, after passing to the quotient, is actually $1-1$ and onto, so it has an inverse. That means that you can associate, to any rotation, an element of the quotient, i.e. a pair of the form $$ \{ (v, \theta), (-v, \theta) \} $$ where $v \in S^2$.
Now you could say "OK, great. I'm gonna select from that pair the one for which the rotation angle is positive" and that will actually work...but it won't give a continuous function, which may turn out to be important to you.
I'm not sure whether this addresses your question, but I hope it's at least of some value.