I don't understand how you arrived at your rotation matrix, so I'll just go by your description of the transformed system and the image, which I think I understand.
If we choose $\phi=0$, $O'$ lies on the $x$-axis, so the $z'$-axis should be the negative $x$-axis, the $y'$-axis should be the $z$-axis, and the $x'$-axis should correspondingly be the negative $y$-axis. That corresponds to the transformation matrix
$$\left(\begin{array}{ccc}0&-1&0\\0&0&1\\-1&0&0\end{array}\right)\;.$$
This is not what your expression for the rotation matrix yields for $\phi=0$. Since the $y'$-axis always points along the $z$-axis and only the $x'$-axis and the $z'$-axis are rotated into each other if we change $\phi$, we can immediately infer the general form of the rotation matrix as
$$\left(\begin{array}{ccc}\sin\phi&-\cos\phi&0\\0&0&1\\-\cos\phi&-\sin\phi&0\end{array}\right)\;,$$
where only the sign of the sine was still to be determined and can be fixed by considering $\phi=\pi/2$.
Here is a "small" addition to the answer by @rschwieb:
Imagine you have the following rotation matrix:
$$
\left[
\begin{array}{ccc}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{array}
\right]
$$
At first one might think this is just another identity matrix. Well, yes and no. This matrix can represent a rotation around all three axes in 3D Euclidean space with...zero degrees. This means that no rotation has taken place around any of the axes.
As we know $\cos(0) = 1$ and $\sin(0) = 0$.
Each column of a rotation matrix represents one of the axes of the space it is applied in so if we have 2D space the default rotation matrix (that is - no rotation has happened) is
$$
\left[
\begin{array}{cc}
1 & 0\\
0 & 1
\end{array}
\right]
$$
Each column in a rotation matrix represents the state of the respective axis so we have here the following:
$$
\left[
\begin{array}{c|c}
1 & 0\\
0 & 1
\end{array}
\right]
$$
First column represents the x axis and the second one - the y axis. For the 3D case we have:
$$
\left[
\begin{array}{c|c|c}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{array}
\right]
$$
Here we are using the canonical base for each space that is we are using the unit vectors to represent each of the 2 or 3 axes.
Usually I am a fan of explaining such things in 2D however in 3D it is much easier to see what is happening. Whenever we want to rotate around an axis, we are basically saying "The axis we are rotating around is the anchor and will NOT change. The other two axes however will".
If we start with the "no rotation has taken place" state
$$
\left[
\begin{array}{c|c|c}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{array}
\right]
$$
and want to rotate around - let's say - the x axis we will do
$$
\left[
\begin{array}{c|c|c}
1 & 0 & 0\\
0 & \cos(\theta) & -\sin(\theta)\\
0 & \sin(\theta) & \cos(\theta)
\end{array}
\right] .
\left[
\begin{array}{c|c|c}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{array}
\right] =
\left[
\begin{array}{c|c|c}
1 & 0 & 0\\
0 & \cos(\theta) & -\sin(\theta)\\
0 & \sin(\theta) & \cos(\theta)
\end{array}
\right]
$$
What this means is:
- The state of the x axis remains unchanged - we've started with a state of no rotation so the x axis will retain its original state - the unit vector $\left[\begin{array}{c}1\\0\\0\end{array}\right]$
- The state of the y and z axis has changed - instead of the original $\left[\begin{array}{c}0\\1\\0\end{array}\right]$ (for y) and $\left[\begin{array}{c}0\\0\\1\end{array}\right]$ (for z) we now have $\left[\begin{array}{c}0\\\cos(\theta)\\\sin(\theta)\end{array}\right]$ (for the new orientation of y) and $\left[\begin{array}{c}0\\-\sin(\theta)\\\cos(\theta)\end{array}\right]$ (for the new orientation of z).
We can continue applying rotations around this and that axis and each time this will happen - the axis we are rotating around remains as it was in the previous step and the rest of the axes change accordingly.
Now when it comes to 2D we have
$$
R(\theta) = \left[
\begin{array}{c|c}
\cos(\theta) & -\sin(\theta)\\
\sin(\theta) & \cos(\theta)
\end{array}
\right]
$$
for counterclockwise rotation and
$$
R(-\theta) = \left[
\begin{array}{c|c}
\cos(\theta) & \sin(\theta)\\
-\sin(\theta) & \cos(\theta)
\end{array}
\right]
$$
for clockwise rotation. Notice that both column vectors are different. This is because in 2D none of the two axes remains idle and both need to change in order to create a rotation. This is why also the 3D version has two of the three axes change simultaneously - because it is just a derivative from its 2D version.
When it comes to rotating clock- or counterclockwise you can always use the left or right hand rule:
- Use your right or left hand to determine the axes:
- See which way is clock- and which way is counterclockwise. In the image below the four finger tips that go straight into your palm always point along the direction of rotation (right hand rule):
Once you pick one of the two hands stick with it and use it until the end of the specific task otherwise the results will probably end up screwed up. Notice also that this rule can also be applied to 2D. Just remove (but not cut off) the finger that points along the z axis (or whichever dimension of the three you don't need) and do your thing.
A couple of must knows things:
Matrix multiplication is generally not commutative - what this means is that $A.B \ne B.A$
Rotation order is determined by the multiplication order (due to 1)) - there are a LOT of rotation conventions (RPY (roll,pitch and yaw), Euler angles etc.) so it is important to know which one you are using. If you are not certain pick one and stick with it (better have one consistent error than 10 different errors that you cannot follow up on) (see here for some compact information on this topic)
Inverse of a rotation matrix rotates in the opposite direction - if for example $R_{x,90}$ is a rotation around the x axis with +90 degrees the inverse will do $R_{x,-90}$. On top of that rotation matrices are awesome because $A^{-1} = A^t$ that is the inverse is the same as the transpose
Best Answer
To begin with, in the second equation you shouldn't think of the vector $\mathbf{i}$ as $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$; as you say, that wouldn't make sense. That equation is just a symbolic way of writing the vector equations $\mathbf{i}' = (\cos\theta) \mathbf{i} + (\sin\theta) \mathbf{j}$ and $\mathbf{j}' = (-\sin\theta) \mathbf{i} + (\cos\theta) \mathbf{j}$, which say that the new basis $(\mathbf{i}',\mathbf{j}')$ is rotated by an angle $\theta$ (counterclockwise) relative to the old basis $(\mathbf{i},\mathbf{j})$.
As you'll see below, it's better to write it like this instead: $$ \begin{bmatrix} \mathbf{i}' & \mathbf{j}' \end{bmatrix} = \begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix} \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}. $$
Now for any vector $\mathbf{u}$, we can look at its coordinates with respect to the old basis, $\mathbf{u} = x \mathbf{i} + y \mathbf{j}$, or with respect to the new basis, $\mathbf{u} = x' \mathbf{i}' + y' \mathbf{j}'$. (Note: No prime on $\mathbf{u}$ in the second formula, since it's the same geometrical vector as in the first formula.) On matrix form, this can symbolically be written as $$ \mathbf{u} = \begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} \mathbf{i}' & \mathbf{j}' \end{bmatrix} \begin{bmatrix} x' \\ y' \end{bmatrix}. $$ Here we replace the primed basis with the expression given by the formula above; this results in $$ \begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix} \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix} x' \\ y' \end{bmatrix}, $$ hence (since coordinates with respect to a basis are unique) $$ \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix} x' \\ y' \end{bmatrix}, $$ which is the relation between the coordinates of the vector $\mathbf{u}$ in the old and in the new basis, just as your textbook says.
As you yourself can testify, your book is also correct in its statement "there can be confusion"! ;-)