To begin with, in the second equation you shouldn't think of the vector $\mathbf{i}$ as $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$; as you say, that wouldn't make sense. That equation is just a symbolic way of writing the vector equations $\mathbf{i}' = (\cos\theta) \mathbf{i} + (\sin\theta) \mathbf{j}$ and $\mathbf{j}' = (-\sin\theta) \mathbf{i} + (\cos\theta) \mathbf{j}$,
which say that the new basis $(\mathbf{i}',\mathbf{j}')$ is rotated by an angle $\theta$ (counterclockwise) relative to the old basis $(\mathbf{i},\mathbf{j})$.
As you'll see below, it's better to write it like this instead:
$$
\begin{bmatrix} \mathbf{i}' & \mathbf{j}' \end{bmatrix} =
\begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix}
\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}.
$$
Now for any vector $\mathbf{u}$, we can look at its coordinates with respect to the old basis, $\mathbf{u} = x \mathbf{i} + y \mathbf{j}$, or with respect to the new basis,
$\mathbf{u} = x' \mathbf{i}' + y' \mathbf{j}'$.
(Note: No prime on $\mathbf{u}$ in the second formula, since it's the same geometrical vector as in the first formula.)
On matrix form, this can symbolically be written as
$$
\mathbf{u} =
\begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix}
\begin{bmatrix} x \\ y \end{bmatrix}
=
\begin{bmatrix} \mathbf{i}' & \mathbf{j}' \end{bmatrix}
\begin{bmatrix} x' \\ y' \end{bmatrix}.
$$
Here we replace the primed basis with the expression given by the formula above; this results in
$$
\begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix}
\begin{bmatrix} x \\ y \end{bmatrix}
=
\begin{bmatrix} \mathbf{i} & \mathbf{j} \end{bmatrix}
\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}
\begin{bmatrix} x' \\ y' \end{bmatrix},
$$
hence (since coordinates with respect to a basis are unique)
$$
\begin{bmatrix} x \\ y \end{bmatrix}
=
\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}
\begin{bmatrix} x' \\ y' \end{bmatrix},
$$
which is the relation between the coordinates of the vector $\mathbf{u}$ in the old and in the new basis, just as your textbook says.
As you yourself can testify, your book is also correct in its statement "there can be confusion"! ;-)
The matrix $$
\begin{bmatrix}
-1 & 0\\
0 & 1
\end{bmatrix}$$ is not the matrix of a rotation, but rather a reflection about the $y$-axis. The rotation through 180 degrees is given by the matrix
$$
\begin{bmatrix}
-1 & 0\\
0 & -1
\end{bmatrix}.$$ The second matrix
$$
\begin{bmatrix}
0 & -1\\
1 & 0
\end{bmatrix}$$ is indeed the matrix of a rotation through 90 degrees in the positive (counterclockwise) direction, with center of rotation at the origin $(0,0)$. So it's not surprising that your red and green points moved while the blue point (origin) remained fixed!
In general, to compute the matrix of a linear transformation $T$ (such as a reflection or rotation fixing the origin), you compute its effect on the basis vectors---the matrix of $T$ has columns $T(e_1)$ and $T(e_2)$ (or, in higher dimensions, $T(e_1),...,T(e_n)$). Using a little geometry and the definitions of sine and cosine, a rotation by $\theta$ degrees fixing the origin maps the vectors
$$
\begin{bmatrix}
1 \\
0
\end{bmatrix} \quad \text{and} \quad \begin{bmatrix}
0 \\
1
\end{bmatrix}$$ to the vectors
$$
\begin{bmatrix}
\mathrm{cos}(\theta) \\
\mathrm{sin}(\theta)
\end{bmatrix} \quad \text{and} \quad \begin{bmatrix}
-\mathrm{sin}(\theta) \\
\mathrm{cos}(\theta)
\end{bmatrix}.$$ So the formula you give in the question is off by a little (and doesn't match the correct matrix you gave for rotation through 90 degrees).
Best Answer
This is a manifestation of the handedness of coordinate systems. Only $(\mathbf{i},\mathbf{j},\mathbf{k})$, $(\mathbf{j},\mathbf{k},\mathbf{i})$ and $(\mathbf{k},\mathbf{i},\mathbf{j})$ are right handed systems. This plays a role because we need the handedness of the system to determine whether a rotation is clockwise/counterclockwise. In other words, the $xyz$-coordinates should be viewed in a cyclic order: $x$ followed by $y$ followed by $z$ followed by $x$ et cetera.
Think of these matrices as follows. The first coordinate is that of the axis of rotation, then the other two in the order indicated above. The apparent difference between the last rotation matrix and the others reflects this. You are looking at the matrix as if the $x$ coordinate were still the first coordinate after the axis of rotation (there $y$) even though we really should stick to $y$ followed by $z$ followed $x$.
Another way of seeing this is to expand the matrices to $5\times5$ matrices by replicating $x$ and $y$ rows/columns, in the order $x,y,z,x,y$. Like this: $$ \left(\begin{array}{rrrrr} \cos\theta&-\sin\theta&0&\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta&0&\sin\theta&\cos\theta\\ 0&0&1&0&0\\ \cos\theta&-\sin\theta&0&\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta&0&\sin\theta&\cos\theta \end{array}\right) $$ Do you see the diagonal $3\times3$ blocks here now that the coordinates follow the right handed ordering? $$ \left(\begin{array}{rrrrr} \color{red}{\cos\theta}&\color{red}{-\sin\theta}&\color{red}0&\cos\theta&-\sin\theta\\ \color{red}{\sin\theta}&\color{red}{\cos\theta}&\color{red}{0}&\sin\theta&\cos\theta\\ \color{red}0&\color{red}0&\color{red}1&0&0\\ \cos\theta&-\sin\theta&0&\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta&0&\sin\theta&\cos\theta \end{array}\right), $$ $$ \left(\begin{array}{rrrrr} {\cos\theta}&{-\sin\theta}&0&\cos\theta&-\sin\theta\\ {\sin\theta}&\color{red}{\cos\theta}&\color{red}{0}&\color{red}{\sin\theta}&\cos\theta\\ 0&\color{red}0&\color{red}1&\color{red}0&0\\ \cos\theta&\color{red}{-\sin\theta}&\color{red}0&\color{red}{\cos\theta}&-\sin\theta\\ \sin\theta&\cos\theta&0&\sin\theta&\cos\theta \end{array}\right) $$ and $$ \left(\begin{array}{rrrrr} \cos\theta&-\sin\theta&0&\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta&0&\sin\theta&\cos\theta\\ 0&0&\color{red}1&\color{red}0&\color{red}0\\ \cos\theta&-\sin\theta&\color{red}0&\color{red}{\cos\theta}&\color{red}{-\sin\theta}\\ \sin\theta&\cos\theta&\color{red}0&\color{red}{\sin\theta}& \color{red}{\cos\theta} \end{array}\right) $$ This way we see that the three matrices really are the same when we keep track of the cyclic order of the coordinates.