Others have raised some good points, and a definite answer really depends what kind of a linear transformation do we want call a rotation or a reflection.
For me a reflection (may be I should call it a simple reflection?) is a reflection with respect to a subspace of codimension 1. So in $\mathbf{R}^n$ you get these by fixing a subspace $H$ of dimension $n-1$. The reflection $s_H$ w.r.t. $H$ keeps the vectors of $H$ fixed (pointwise) and multiplies a vector perpendicular to $H$ by $-1$. If $\vec{n}\perp H$, $\vec{n}\neq0$, then $s_H$ is given by the formula
$$\vec{x}\mapsto\vec{x}-2\,\frac{\langle \vec{x},\vec{n}\rangle}{\|\vec{n}\|^2}\,\vec{n}.$$
The reflection $s_H$ has eigenvalue $1$ with multiplicity $n-1$ and eigenvalue $-1$ with multiplicity $1$ with respective eigenspaces $H$ and $\mathbf{R}\vec{n}$. Thus its determinant is $-1$. Therefore geometrically it reverses orientation (or handedness, if you prefer that term), and is not a rigid body motion in the sense that in order to apply that transformation to a rigid 3D body, you need to break it into atoms (caveat: I don't know if this is the standard definition of a rigid body motion?). It does preserve lengths and angles between vectors.
Rotations (by which I, too, mean simply an orthogonal transformations with $\det=1$) have more variation. If $A$ is a rotation matrix, then Adam's calculation proving that the lengths are preserved, tells us that the eigenvalues must have absolute value $=1$ (his calculation goes through for a complex vectors and the Hermitian inner product). Therefore the complex eigenvalues are on the unit circle and come in complex conjugate pairs. If $\lambda=e^{i\varphi}$ is a non-real eigenvalue, and $\vec{v}$ is a corresponding eigenvector (in $\mathbf{C}^n$), then the vector $\vec{v}^*$ gotten by componentwise complex conjugation is an eigenvector of $A$ belonging to eigenvalue $\lambda^*=e^{-i\varphi}$.
Consider the set $V_1$ of vectors of the form $z\vec{v}+z^*\vec{v}^*$. By the eigenvalue property this set is stable under $A$:
$$A(z\vec{v}+z^*\vec{v}^*)=(\lambda z)\vec{v}+(\lambda z)^*\vec{v}^*.$$ Its components are also stable under complex conjugation, so $V_1\subseteq\mathbf{R}^n$. It is obviously a 2-dimensional subspace, IOW a plane. It is easy to guess and not difficult to prove that the restriction of the transformation $A$ onto the subspace $V_1$ is a rotation by the angle $\varphi_1=\pm\varphi$. Note that we cannot determine the sign of the rotation (clockwise/ccw), because we don't have a preferred handedness on the subspace $V$.
The preservation of angles (see Adam's answer) shows that $A$ then maps the $n-2$ dimensional subspace $V^\perp$ also to itself. Furthermore, the determinant of $A$ restricted to $V_1$ is equal to one, so the same holds for $V_1^\perp$. Thus we can apply induction and keep on splitting off 2-dimensional summands $V_i,i=2,3\ldots,$ such that on each summand $A$ acts as a rotation by some angle $\varphi_i$ (usually distinct from the preceding ones). We can keep doing this until only real eigenvalues remain, and end with the situation:
$$
\mathbf{R}^n=V_1\oplus V_2\oplus\cdots V_m \oplus U,
$$
where the 2D-subspaces $V_i$ are orthogonal to each other, $A$ rotates a vector in $V_i$ by the angle $\varphi_i$, and $A$ restricted to $U$ has only real eigenvalues.
Counting the determinant will then show that the multiplicity of $-1$ as an eigenvalue of $A$ restricted to $U$ will always be even. As a consequence of that we can also split that eigenspace into sums of 2-dimensional planes, where $A$ acts as rotation by 180 degrees (or multiplication by $-1$). After that there remains the eigenspace belonging to eigenvalue $+1$. The multiplicity of that eigenvalue is congruent to $n$ modulo $2$, so if $n$ is odd, then $\lambda=+1$ will necessarily be an eigenvalue. This is the ultimate reason, why a rotation in 3D-space must have an axis = eigenspace belonging to eigenvalue $+1$.
From this we see:
- As Henning pointed out, we can continuously bring any rotation back to the identity mapping simply by continuously scaling all the rotation angles $\varphi_i,i=1,\ldots,m$ continuously to zero. The same can be done on those summands of $U$, where $A$ acts as rotation by 180 degrees.
- If we want to define rotation in such a way that the set of rotations contains the elementary rotations described by Henning, and also insist that the set of rotations is closed under composition, then the set must consist of all orthogonal transformations with $\det=1$. As a corollary to this rotations preserve handedness. This point is moot, if we defined a rotation by simply requiring the matrix $A$ to be orthogonal and have $\det=1$, but it does show the equivalence of two alternative definitions.
- If $A$ is an orthogonal matrix with $\det=-1$, then composing $A$ with a reflection w.r.t. to any subspace $H$ of codimension one gives a rotation in the sense of this (admittedly semi-private) definition of a rotation.
This is not a full answer in the sense that I can't give you an 'authoritative' definition of an $n$D-rotation. That is to some extent a matter of taste, and some might want to only include the simple rotations from Henning's answer that only "move" points of a 2D-subspace and keep its orthogonal complement pointwise fixed. Hopefully I managed to paint a coherent picture, though.
I believe the problem you are having is that if you multiply the matrices in the order $R_X\cdot R_Y\cdot R_Z$, then you are rotating first around the global $Z$ axis, then the global $Y$ axis, then the global $X$ axis.
To get the rotations applied to the local (body) axes instead, you simply have to reverse the order of multiplication.
This might seem like magic, so I'll give a short explanation. After applying the $Z$-axis rotation, the body reference frame is not the same as the global reference frame: It has been rotated by $R_Z$. So at this point, if $A$ is a matrix denoting any transformation according to the body frame, then in the global frame that same transformation has the matrix representation
$$
R_Z\cdot A\cdot R_Z^{-1}
$$
(This will take a vector expressed in the global basis, then $R_Z^{-1}$ is the same vector expressed in the body basis. We apply $A$, then translate back to the global basis with $R_Z$.)
So first applying $R_Z$, then after that applying $R_Y$ in the new local frame has matrix representation
$$
\left(R_Z\cdot R_Y\cdot R_Z^{-1}\right)\cdot R_Z = R_Z\cdot R_Y
$$
If you want to apply a rotation about the body $X$-axis after this again, then similarily, the global matrix representation of this transformation is
$$
(R_Z\cdot R_Y)\cdot R_X\cdot (R_Z\cdot R_Y)^{-1}
$$
and if we apply this after our initial rotation of $R_Z\cdot R_Y$, then the result becomes
$$
R_Z\cdot R_Y\cdot R_X
$$
Best Answer
The slightly counterintuitive bit is that the metric comes from the (normed) vector space structure of the matrix algebra, but when you're dealing with $SO_3$ you're focusing on the complementary structure (the bilinear operation of multiplication). Still, the Frobenius norm gives us an extrinsic metric on $SO_3$ when we consider it embedded in matrix land: $d(R_1, R_2) = ||R_1 - R_2||_F$. $SO_3$ is not a metric vector space, but a subset of a metric is still a metric space.
Let's look at another related example.
Consider the complex numbers, $\mathbb{C}$, which form an algebra over the reals under the operations of multiplication and addition. We have a standard norm in the complex numbers, $$|a + bi| = \sqrt{a^2 + b^2}$$ which induces a metric in which the distance between $u = a+bi$ and $v = c + di$ is $d(u,v) = |u-v|$. Now this metric gives a perfectly good way to measure the distance between any two complex numbers. This metric still provides a consistent way to conceive of distance between elements if we restrict our attention to the unit circle (those complex numbers $z$ such that $|z| = 1$, which form a group under multiplication). However, you might have the objection that, considering the unit circle as embedded in $\mathbb{C}$, this measure of distance is extrinsic. If that's an issue for you, you can establish an intrinsic metric by taking the distance of two points on the unit circle to be the minimum length among all paths between the points restricted to the unit circle.
Your matrix case is much the same. True you are considering $SO_3$ which is a special subset of matrices that forms a group under multiplication, but that does not invalidate the Frobenius norm on all matrices as a measure of distance.