The solution to the vector differential equation $\mathbf y'(x)=A\mathbf y(x)$ is, not surprisingly, $e^{xA}\mathbf C$, where $\mathbf C$ is a vector of constants determined by boundary conditions. The exponential of a matrix is defined via a power series, but in practice one doesn’t use that to compute it.
If $A$ is diagonalizable, it can be decomposed as $B\Lambda B^{-1}$, where $B=\begin{bmatrix}\mathbf b_1,\cdots,\mathbf b_n\end{bmatrix}$ with eigenvectors of $A$ as its columns and $\Lambda=\operatorname{diag}(\lambda_1,\dots,\lambda_n)$ with $\lambda_j$ the eigenvalues corresponding to $\mathbf b_j$. Just as $A^k=B\Lambda^kB^{-1}=B\operatorname{diag}(\lambda_1^k,\dots,\lambda_n^k)B^{-1}$, so, too $e^{xA}=Be^{x\Lambda}B^{-1}=B\operatorname{diag}(e^{\lambda_1x},\dots,e^{\lambda_nx})B^{-1}$. So, the solution to the equation is $$e^{xA}\mathbf C=Be^{x\Lambda}B^{-1}\mathbf C.$$ Since $B$ has full rank, $B^{-1}\mathbf C$ is also a vector of arbitrary constants, so we can expand the above expression as $$C_1e^{\lambda_1x}\mathbf b_1+\cdots+C_ne^{\lambda_nx}\mathbf b^n.\tag1$$
In this problem, $$A=\begin{bmatrix}-2&1&-2\\1&-2&2\\3&-3&5\end{bmatrix}.$$ Its eigenvalues can be found to be $3$, $-1$ and $-1$. We compute the kernel of $3I-A$ via row-reduction: $$\begin{bmatrix}5&-1&2\\-1&5&-2\\-3&3&-2\end{bmatrix}\to\begin{bmatrix}1&0&\frac13\\0&1&-\frac13\\0&0&0\end{bmatrix}$$ so an eigenvector of $3$ is $(-1,1,3)^T$. Moving to $A+I$, $$\begin{bmatrix}-1&1&-2\\1&-1&2\\3&-3&6\end{bmatrix}\to\begin{bmatrix}1&-1&2\\0&0&0\\0&0&0\end{bmatrix}$$ from which we get $(1,1,0)^T$ and $(-2,0,1)^T$ as linearly independent eigenvectors. Plugging these values into (1), the solution to the equation is therefore $$C_1e^{3x}\begin{bmatrix}-1\\1\\3\end{bmatrix}+C_2e^{-x}\begin{bmatrix}1\\1\\0\end{bmatrix}+C_3e^{-x}\begin{bmatrix}-2\\0\\1\end{bmatrix}=\begin{bmatrix}-C_1e^{3x}+(C_2-2C_3)e^{-x}\\C_1e^{3x}+C_2e^{-x}\\3C_1e^{3x}+C_3e^{-x}\end{bmatrix}.$$
Setting $t=0$ in the solution proposed by the author of the OP gives $x_1 (0) = c_1 - c_2$, which is in contradiction with the definition of $c = x (0)$. Equation $(1)$ is correct. If $y = P^{-1} x$, then we have
$$y_1 (t) = e^{-t} y_1 (0) ,\quad y_2 (t) = e^{2t} y_2 (0),$$
and in particular
$$x_1 (t) = y_1 (t) - y_2 (t) . $$
Now, it remains to express $y (0)$ in terms of $x (0) = c$, i.e.
$$y_1 (0) = c_1 + c_2 ,\quad y_2 (0) = c_2,$$
which gives the result of the book.
Best Answer
The equation $y'' - 4y' + 4y = 0$ has solutions of the form $c_1e^{2x}+c_2xe^{2x}$. Combining this with the fact that $y'' - 4y' + 4y = e^x$ has $e^x$ as a particular solution, the general solution is $y=e^x+c_1e^{2x}+c_2xe^{2x}$.