The multiplicity of a root $r$ of a polynomial $p(x)$ is the number of times that $x-r$ appears as a factor when $p(x)$ is fully factored into linear factors. You have the equation $\lambda^2(\lambda-1)=0$, which is fully factored into the linear factors $\lambda$, $\lambda$, and $\lambda-1$. Thus, $0$ is the root of multiplicity $2$, and $1$ is the root of multiplicity $1$.
Now you want to find the eigenvectors. For a given eigenvalue $\lambda$, these are the vectors $v$ such that $Av=\lambda v$ or, equivalently, $(A-\lambda I)v=\vec 0$. For $\lambda=0$ this equation becomes
$$\begin{bmatrix}1&-1&1\\-1&1&-1\\-1&1&-1\end{bmatrix}\begin{bmatrix}x\\y\\z\end{bmatrix}=\begin{bmatrix}0\\0\\0\end{bmatrix}\;,$$
a matrix equation that you solve by your favorite method. I’d just row-reduce the augmented matrix to
$$\left[\begin{array}{rrr|r}1&-1&1&0\\0&0&0&0\\0&0&0&0\end{array}\right]$$
and choose a basis for the two-dimensional solution space.
Then do the same thing for $\lambda=1$; your starting point should be the equation
$$\begin{bmatrix}0&-1&1\\-1&0&-1\\-1&1&-2\end{bmatrix}\begin{bmatrix}x\\y\\z\end{bmatrix}=\begin{bmatrix}0\\0\\0\end{bmatrix}\;.$$
This will actually be a little easier, since the solution space will be one-dimensional.
Solving the System of linear Equations
So it seems you want to solve the system of equations
$$
Ax =
\begin{pmatrix}-13&40&-48\\-8&23&-24\\0&0&3\end{pmatrix}
\begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix}
=
\begin{pmatrix} x'_1 \\ x'_2 \\ x'_3 \end{pmatrix}
= x'
$$
Where the $x'$ is given and the $x_j$ are looked for.
You would normally solve this by calculating the inverse of $A$, since then
$$
x = A^{-1} x'
$$
But let's say you really want to solve this by using the eigenvalues and eigenvectors. You have already determined both of these, providing the change of base matrix $S$ which contains the eigenvectors
$$
S =
\begin{pmatrix}
2 & 5 & -3 \\
1 & 2 & 0 \\
0 & 0 & 1
\end{pmatrix}
$$
which accomplishes
$$
S^{-1} A S = D =
\begin{pmatrix}
7 & 0 & 0 \\
0 & 3 & 0 \\
0 & 0 & 3
\end{pmatrix}
$$
We can now solve for $x$. Using the above,
\begin{align*}
x &= A^{-1} x'
\\&= (SDS^{-1})^{-1} x'
\\&= SD^{-1}S^{-1} x'
\\&=
\begin{pmatrix}
2 & 5 & -3 \\
1 & 2 & 0 \\
0 & 0 & 1
\end{pmatrix}
\begin{pmatrix}
\frac{1}{7} & 0 & 0 \\
0 & \frac{1}{3} & 0 \\
0 & 0 & \frac{1}{7}
\end{pmatrix}
S^{-1} x'
\end{align*}
As you can see, we do not yet have the solution, since we don't know what $S^{-1}$ looks like. So what's left to do is: calculate the inverse of $S$, multiply everything and be done.
(So whatever method you use, you have to calculate the inverse of some matrix)
Getting the Eigenvectors
To determine the last two eigenvectors, you got the equation
$$
x_1 - 2.5 x_2 + 3 x_3 = 0
$$
whose solutions $(x_1,x_2,x_3)$ you want to find. What the equation tells us is, that if we are given two of the variables, the third one (for example $x_1$) will be determined.
$$
x_1 = 2.5 x_2 - 3 x_3
$$
Thus the solutions can be written as
$$
( 2.5 x_2 - 3 x_3 , x_2 , x_3 )
\equiv
\begin{pmatrix}
2.5 x_2 - 3 x_3 \\
x_2 \\
x_3
\end{pmatrix}
=
x_2
\begin{pmatrix}
2.5 \\
1 \\
0
\end{pmatrix}
+
x_3
\begin{pmatrix}
- 3 \\
0 \\
1
\end{pmatrix}
~~~~~~~ \text{with} ~~ x_2 , x_3 \in \mathbb{R}
$$
Any solution of this form is an eigenvector. Since you need two eigenvectors, you take two different solutions. For example the solutions with
$$
(x_2,x_3) = (2,0) ~~~\text{or}~~~ (0,1)
$$
Which yield the eigenvectors that the computer gave you.
Best Answer
Given:
$$\tag 1 x'(t) = A x(t) = \begin{bmatrix} 1 & 1 \\ -1 & 3 \end{bmatrix}x(t)$$
Find the solution to the system in $(1)$.
For the matrix A, we get the eigenvalues / eigenvectors:
From this, we can write the solution:
$$X(t) = \begin{bmatrix}x(t)\\ y(t)\end{bmatrix} = e^{2t}\left(c_1 \begin{bmatrix}1\\1\end{bmatrix} + c_2\left(\begin{bmatrix}-1\\0\end{bmatrix} + t\begin{bmatrix}1\\1\end{bmatrix}\right)\right) = e^{2t}\begin{bmatrix}c_1 + c_2(t-1)\\c_1 + c_2 t\end{bmatrix}$$
This result matches your book.
Do you see where you went astray? Be careful when writing that second solution because we have a repeated eigenvalue.
Update
We need to find a generalized eigenvector, so we have $[A - 2I]v_2 = v_1$, and when we do RREF, we end up with:
$$\begin{bmatrix}1 & -1 \\0 & 0 \end{bmatrix}v_2 = \begin{bmatrix}-1\\0\end{bmatrix}$$
From this, we have:
$$a - b = -1 \rightarrow a = -1 + b \rightarrow ~\mbox{let}~ b = 0,~\mbox{so}~ a = -1$$
This results in the eigenvector:
$$v_2 = (-1,0)$$
Do you see your issue?
Update 2
To verify the result:
If both of those match, it is a correct result.