I've been trying for the past couple of days to write an algorithm to convert a rotation matrix to a rotation about an axis. After dealing with a couple other issues I found that the problem was converting a matrix when the angle is 180 degrees or pi radians. The problem is that I'm not even certain if I can, and yet I need to do so in order to convert a transformation to a form I can use.
I know that the cosine of the angle is the sum of the components of the diagonal of the matrix plus $1$ and divided by $2$. I also know that the angle itself can be chosen as either so long as further calculations use that same angle.
For example one matrix I had was
$\begin{bmatrix}0 & 0 & -1\\0 & -1 & 0\\-1 & 0 & 0\end{bmatrix}$
According to the formula for the angle this matrix has a rotation angle of 180. Now of course this is just an example but I cannot mentally conceive what sort of vector the rotation will rotate around. I can imagine components of the reflection but I cannot conceive how I would get the rotation vector or how I would visualize it.
Note the assumption here is that the input matrix is a valid rotation matrix with a rotation angle of 180 degrees. Other cases have already been handled appropriately.
Edit:
The comments gave an interesting algorithm. Just take the original matrix $M$ and any row of the following matrix gives an axis of rotation for 180 degree rotation:
$$M + M^t + 2I = T$$
However consider the following:
$\begin{bmatrix}-1 & 0 & 0\\0 & 0 & -1\\0 & -1 & 0\end{bmatrix}$
The first row of $T$ for that matrix is all $0$'s, and so the algorithm fails.
Best Answer
The axis of rotation has the property that
$R v = v$
So you have to solve for the eigenvector that corresponds to the eigenvalue $1$.
In this case $v=(-1,0,1)$. This is in the xz plane. It is in the second quadrant of that plane.
For the algorithm, do this in general. Output the eigenvector with eigenvalue 1 for any given $R$. That gives the axis of rotation for that $R$.