Let $0 \neq S \in \mathfrak{so}(3)$ be a non-zero skew-symmetric matrix, and let $R \in SO(3)$. The claim is that, indeed, $SR$ is skew-symmetric if and only if $R = I_3$ or $R$ is the rotation by $\pi$ about the axis $\ker(S)$; since $SI = S$ is necessarily skew-symmetric, we only need to consider the case where $R \neq I_3$. Observe that
$$
(SR)^T+SR = R^TS^T+SR = -R^TS+SR =-R^T(S-RSR),
$$
so that $SR$ is skew-symmetric if and only if $S=RSR$.
Before continuing, let me check that $\ker(S)$ is indeed $1$-dimensional. Observe that, since $S$ is skew-symmetric, that $\ker(S)$ and $\ker(S)^\perp$ are both $S$-invariant. Now, since $S \neq 0$, fix a unit vector $0 \neq x \in \ker(S)^\perp$. Then, since $S$ is skew-symmetric, $x$ is orthogonal to $0 \neq Sx \in \ker(S)^\perp$, so that $\ker(S)^\perp$ is at least two-dimensional and hence $\ker(S)$ is at most $1$-dimensional. However, $S$ cannot be invertible, for $\operatorname{set}(S) = \operatorname{det}(S^T) = \operatorname{det}(-S) = -\operatorname{det}(S)$, so that, indeed, $\operatorname{det}(S) = 0$. Thus, $\ker(S)$ is, in fact, $1$-dimensional.
For future convenience, observe that if $0 \neq x \in \ker(S)^\perp$ is a unit vector, then for $y := \tfrac{1}{\|Sx\|}Sx$ and $0 \neq z \in \ker(S)$ any unit vector, $\{x,y\}$ is an orthonormal basis for $\ker(S)^\perp$, $\ker(S) = \mathbb{R}z$, and $\beta = \{x,y,z\}$ is an orthonormal basis for $\mathbb{R}^3$. Indeed, let’s see how $S$ looks in terms of this basis. On the one hand, we already have that $Sz = 0$ and that $Sx = \alpha y$ for $\alpha = \|Sx\|$. On the other hand, by the same argument as above, $y$ is orthogonal to $0 \neq Sy \in \ker(S)^\perp$, so that
$$
Sy = \langle x, Sy \rangle = -\langle Sx, y \rangle = -\langle \alpha y, y \rangle = -\alpha.
$$
Thus,
$$
Sx = \alpha y, \quad Sy = -\alpha x, \quad Sz = 0,
$$
for $\alpha := \|Sx\| \neq 0$.
Suppose, on the one hand, that $SR$ is skew-symmetric, and hence that $S=RSR$. We first need to show that $\ker(S) = \ker(R-I_3)$, so assume otherwise. Let $z \in \ker(S)$ be a unit vector. Fix a unit vector $x \in \ker(R-I_3)$, so that $\ker(R-I_3) = \mathbb{R}x$. Then,
$$
z = z_0 + cx
$$
for $z_0 \in \ker(R-I_3)^\perp$ and $c = \langle x,z\rangle$; by assumption, $z_0 \neq 0$. Then
$$
0 = Sz = RSRz = RSR(z_0 + cx) = RS(Rz_0 + cx),
$$
so that $Rz_0 + cx \in \ker(S)$, i.e., $Rz_0 + cx = az$ for some $a \in \mathbb{R}$. But then,
$$
Rz_0 + cx = az = az_0 + acx,
$$
so that $Rz_0 = az_0$ and $cx = acx$. Since $z_0$ is non-zero, it is therefore a real eigenvector for $R$, forcing $a = -1$ and hence $c = 0$. Thus, $R$ is the rotation by $\pi$ about the axis $\ker(R-I_3) = \mathbb{R}x$, and $\ker(S) \subset \ker(R-I_3)^\perp = \ker(R+I_3)$.
But now, starting with the unit vectors $x \in \ker(R-I_3) \subset \ker(S)^\perp$ and $z \in \ker(S)$, we can construct, as above, an orthonormal basis $\{x,y,z\}$ of $\mathbb{R}^3$, where $y = \tfrac{1}{\|Sx\|}Sx$, such that
$$
Sx = \alpha y, \quad Sy = -\alpha x, \quad Sz = 0
$$
for $\alpha = \|Sx\| \neq 0$. But then, in particular, since $y \in \ker(R-I_3)^\perp = \ker(R+I_3)$,
$$
-\alpha x = Sy = RSRy = -RSy = \alpha R x = \alpha x,
$$
contradicting $\alpha \neq 0$.
So, as we have seen, $\ker(S) = \ker(R-I_3)$, so again, choose a unit vector $x \in \ker(S)^\perp = \ker(R-I_3)^\perp$ and construct an orthonormal basis $\{x,y,z\}$ of $\mathbb{R}^3$ such that
$$
Sx = \alpha y, \quad Sy = -\alpha x, \quad Sz = 0,
$$
for $\alpha = \|Sx\| \neq 0$. Since $S$ is skew-symmetric, $\ker(S)^\perp = \ker(R-I_3)^\perp$ is $R$-invariant, so that, in particular, $Rx \in \ker(S)^\perp$. But then, since $SR$ is skew-symmetric,
$$
0 = \langle x, SR x \rangle = -\langle Sx, Rx \rangle = -\alpha \langle y, Rx \rangle,
$$
so that $Rx \in \ker(S)^\perp$ is orthogonal to $y$, and hence $Rx = cx$ for some $c \in \mathbb{R}$, i.e., $x \in \ker(S)^\perp = \ker(R-I_3)^\perp$ is a non-zero real eigenvector for $R$. Thus, necessarily, $c = -1$, so that, indeed, $R$ is the rotation by $\pi$ about the axis $\ker(R-I_3) = \ker(S)$.
Suppose, on the other hand, that $R$ is the rotation by $\pi$ about the axis $\ker(S) = \ker(R-I_3)$. Let $z \in \mathbb{R}^3$, so that $z = z_0 + z_1$ for $z_0 \in \ker(S) = \ker(R-I_3)$ and $z_1 \in \ker(S)^\perp = \ker(R+I_3)$. Then, since $S$ is skew-symmetric, $\ker(S)^\perp = \ker(R+I_3)$ is $S$-invariant, and hence
$$
RSRv = RSR(v_0 + v_1) = RS(v_0-v_1) = -RSv_1 = Sv_1 = Sv_0 + Sv_1 = Sv,
$$
as required.
Best Answer
Here's an approach which is relatively light on computation.
As in the comments, we observe that $A^2 = -(x^2 + y^2 + z^2)I$. It follows that any rational function of $A$ can be written in the form $p I + q A$ for some $p,q$. In other words, there necessarily exist numbers $p,q$ such that $$ (M-I)^{-1} = ([\cos(a) - 1]I + \sin(a)A)^{-1} = p I + q A $$ Now, expand the product $(M-I)(pI + qA),$ and solve for the $p$ and $q$ that cause this product to equal $I$. In particular, we have $$ (M - I)(pI + qA) = ([\cos(a) - 1]I + \sin(a)A)(pI + qA)\\ = p(\cos(a) - 1)I + (q[\cos(a) - 1] + p\sin(a))A + \sin(a)qA^2\\ = \left[p(\cos(a) - 1) - \sin(a)(x^2 + y^2 + z^2)q \right]I + \left[q(\cos(a) - 1) + p\sin(a)\right]A. $$ Thus, it suffices to find $p,q$ that solve the system of equations $$ \begin{array}{ccccccc} (\cos(a) - 1) &p &- &\sin(a)(x^2 + y^2 + z^2)&q &= &1\\ \sin(a) &p &+ &(\cos(a) - 1)&q &= &0. \end{array} $$ Solving this system yields $$ p = \frac{1 - \cos(a)}{D}, \quad q = \frac{\sin(a)}{D} $$ where $D$ is the determinant of the coefficient matrix of this system, that is $$ D = \sin^2(a)(x^2 + y^2 + z^2) + (\cos(a) - 1)^2. $$ I see no general way of simplifying the resulting expression.