Your work is fine. However, it is very ad hoc. If you are interested in a more algorithmic way of computing rational canonical forms, read on.
There are actually two things that have the right to be called the "Rational Canonical Form", the one obtained through invariant factors (what most people consider the rational canonical form) and the one obtained from elementary divisors (more akin to the Jordan form). I'll describe how to compute the elementary divisors and then get the invariant factors from them.
Let $T: V \to V$ be a linear transformation.Suppose that $p(x)$ is an irreducible factor of the minimal polynomial of $T$ of degree $d$. We will construct the "dot" diagram corresponding to $p(x)$ which will tell us the elementary divisors corresponding to $p(x)$.
Compute the following numbers until you get $0$:
\begin{align*}
r_1 &= \frac{1}{d}\left( dim(V) - rank(p(T))\right)\\
r_k &= \frac{1}{d}\left( rank(p(T)^{k-1}) - rank(p(T)^k)\right) \text{ for } k \geq 2
\end{align*}
Now construct a diagram with $r_k$ dots in the $k^{th}$ row. The elementary divisors are $p(x)^{s_i}$ where $s_i$ is the number of dots in the $i^{th}$ column of the dot diagram.
Example: Say we are in a real vector space. Suppose $p(x) = x^2+1$ and we compute the $r$ numbers to be $r_1 = 3, r_2 = 3, r_3 = 1, r_4 = 1, r_5 = 0$.
The dot diagram would be:
\begin{array}{lll}
\cdot & \cdot & \cdot \\
\cdot & \cdot & \cdot \\
\cdot \\
\cdot \\
\end{array}
The elementary divisors corresponding to $x^2+1$ would then be $(x^2+1)^4, (x^2+1)^2, (x^2+1)^2$.
To obtain the rational canonical form via elementary divisors, do this for each irreducible factor of the minimal polynomial, and then populate a matrix with the companion matrices of elementary divisors along the diagonal.
To obtain the invariant factors from the elementary divisors: In an array, line up the elementary divisors from greatest degree to least degree with each row corresponding to an irreducible factor of the minimal polynomial. The invariant factors are the products along the columns of this array. The biggest invariant factor is always the minimal polynomial.
For example: Suppose the elementary divisors corresponding to $x^2+1$ are as above, and the elementary divisors corresponding to $x-2$ are: $(x-2)^3, (x-2)$. And that these are the only irreducible factors of the the minimal polynomial.
Then the invariant factors are $$(x^2+1)^4(x-2)^3, (x^2+1)^2(x-2), (x^2+1)^2.$$
To obtain the rational canonical form via invariant factors, populate a matrix with companion matrices of the invariant factors along the diagonal.
In your problem, the only irreducible factor of the minimal polynomial is $(t-1)$. Let's compute its dot diagram.
$$ r_1 = \frac{1}{1}(3 - rank(A - I)) = (3 - 1) = 2 \\
r_2 = \frac{1}{1}(rank(A-I) - rank((A-I)^2)) = (1 - 0) = 1 \\
r_3 = \frac{1}{1}(rank((A-I)^2) - rank((A-I)^3) = (0 -0) = 0$$
So the dot diagram is
\begin{array}{ll}
\cdot & \cdot \\
\cdot \\
\end{array}
Therefore the elementary divisors are $(t-1)^2, (t-1)$. In this case, since there is only one irreducible factor of the minimal polynomial, the invariant factors are the same as the elementary divisors. Anyway, we get that the rational canonical form (via elementary divisors and invariant factors) is:
$$\begin{pmatrix}
0 & -1 & 0 \\
1 & 2 & 0 \\
0 & 0 & 1
\end{pmatrix}$$
just as you had.
HINT
Note that a not trivial solution to $V_1x=0$ is $x=(1,0)$.
Anyway as noticed by David we need to consider the matrix associated to the given transformation $T(M)=AM$, that is with reference to the basis $E_1$, $E_2$, $E_3$, $E_4$
$E_1=\begin{pmatrix}1 & 0\\ 0 & 0\end{pmatrix}$
$E_2=\begin{pmatrix}0 & 1\\ 0 & 0\end{pmatrix}$
$E_3=\begin{pmatrix}0 & 0\\ 1 & 0\end{pmatrix}$
$E_4=\begin{pmatrix}0 & 0\\ 0 & 1\end{pmatrix}$
$$T_A=\begin{pmatrix}1 & 0 &3&0\\ 0 & 1 &0&3\\ 0 & 0 &6&0\\ 0 & 0 &0&6 \end{pmatrix}$$
Best Answer
This is a bit of a brute-force diagonalisation approach - I do feel as though there should be a simpler method, but this at least gives you an answer.
We can reduce this to solving an equation of the form $X^2=C$ by completing the square, as we would do in the real case. Note that (dropping the now-implicit identity matrix for scalars) $$\left(P + \frac{1}{2}\right)^2 = P^2 + P + \frac{1}{4} \implies P^2 + P = \left(P + \frac{1}{2}\right)^2 - \frac{1}{4}$$ Writing $Q$ for the RHS of your original equation, we have $$P^2 + P + 3 = Q \iff \left(P + \frac{1}{2}\right)^2 = Q - \frac{11}{4}$$ We now seek a matrix $X$ such that $X^2 = Q'$ where $Q' = Q - 11/4$. One way to do this is to diagonalise $Q'$ as $$Q' = V^{-1} D V$$ where $D$ is diagonal, since then we find that taking $$X = V^{-1} \sqrt{D} V \implies X^2 = Q'$$ and $\sqrt{D}$ is easy to calculate since we just take the square root of the diagonal entries. Indeed, such a decomposition is possible, with $$V = \begin{pmatrix} -1 & 0 & 0 \\ 0 & 0 & 1 \\ 1 & 1 & 0 \end{pmatrix}, \quad D = \frac{1}{4} \begin{pmatrix} 1 & 0 & 0 \\ 0 & 13 & 0 \\ 0 & 0 & 13 \end{pmatrix}$$ Computing $X$, as indicated above, one can show that $$ X = \frac{1}{2} \begin{pmatrix} 1 & 0 & 0 \\ \sqrt{13}-1 & \sqrt{13} & 0 \\ 0 & 0 & \sqrt{13} \end{pmatrix}$$ and hence $$P = \frac{1}{2} \begin{pmatrix} 0 & 0 & 0 \\ \sqrt{13}-1 & \sqrt{13}-1 & 0 \\ 0 & 0 & \sqrt{13}-1 \end{pmatrix} = \frac{\sqrt{13}-1}{2} \begin{pmatrix} 0 & 0 & 0 \\ 1 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix} $$