When you say, "Which has the following matrix", have a think about what you mean by this. You've taken a basis of polynomials (not a linear transformation) and somehow turned it into a matrix of numbers. What does this matrix mean? Why do you expect the columns to be linearly independent?
What you have done is form the change of basis matrix from the given basis to the standard basis, in the order $(x^2, x, 1)$. Such a matrix must certainly be invertible; change of basis matrices always are, and their inverse is the change of basis matrix from the second basis to the first.
Your issue seems to suggest that you're looking to find a matrix for the transformation, between the two bases. Bear in mind that this is impossible, since the basis is for the kernel and range, both of which are respectively subspaces of the domain and codomain, and the kernel happens to be a strict subspace. You would need full bases for the domain and codomain before you can compute a matrix for the transformation.
But, this isn't what the question wanted. It asked for two bases for the kernel and range, and you have found exactly that.
There are a couple of mistakes. First, the Reduced Row Echelon Form (RREF) of the matrix should be
\begin{equation}
\begin{pmatrix}
1 & 0 & 0 & -2 \\
0 & 1 & 0 & 3 \\
0 & 0 & 1 & 3
\end{pmatrix}
\end{equation}
It should be $-2$ in the corner, not $2$. But surprisingly, the basis for $\ker (\varphi)$ you found is actually correct (you must have made a second sign error in the computation which cancelled out with the first error).
and the dimension of the kernel $=1$. Since it's $\mathbb{R^4} \to \mathbb{R^3}$, the dimension of image has to be $3$.
This is correct.
We have pivots in the first $3$ colums, so we can say that $<(3,0,−1),(2,1,0),(0,−1,0)>$ is the image of $\varphi$. And for bases of $\varphi$, we can take $(3,0,−1),(2,1,0),(0,−1,0)$, as they are linearly independent.
This is an incorrect statement; this I think is more of a terminology mistake rather than a deep conceptual one. The proper statement is $\{(3,0,−1),(2,1,0),(0,−1,0)\}$ forms a basis for the image of $\varphi$. Recall that the image of a linear map is a subspace of $\mathbb{R^3}$, so it can't just consist of $3$ vectors, but a basis for a $3$-dimensional image consists of $3$ vectors.
By the way, for this particular example, there is a much easier way to determine a basis for the image of $\varphi$. You already mentioned that the image has dimension $3$. But notice that the target space $\mathbb{R^3}$ also has dimension $3$. Hence, $\text{image}(\varphi) = \mathbb{R^3}$. So there's a particularly obvious basis: $\{(1,0,0), (0,1,0), (0,0,1) \}$.
Lastly, to compute $[\varphi]_B^C$, the matrix of $\varphi$ with respect to the bases $B$ and $C$, what you have to do is for each vector $v \in B$, compute what $\varphi(v)$ is, and write it as a linear combination of vectors from $C$. The coefficients will then be the entries of the matrix
For example, the first vector in $B$ is $(1,0,0,0)$. So, now we have to evaluate $\varphi$ on this vector:
\begin{align}
\varphi(1,0,0,0) &= (1,0,-1) \\
&= (1,1,1) - (0,1,1) + (0,0,-1)
\end{align}
Notice that the coefficients are $1,-1,1$. So, the first column of $[\varphi]_B^C$ looks like
\begin{pmatrix}
1 & \cdot & \cdot & \cdot\\
-1 & \cdot & \cdot & \cdot \\
1 & \cdot & \cdot & \cdot
\end{pmatrix}
The second vector of $B$ is $(1,1,0,0)$. Now, we compute again:
\begin{align}
\varphi(1,1,0,0) &= (5,1,-1) \\
&= 5 (1,1,1) -4 (0,1,1) + 2(0,0,-1)
\end{align}
So, the first two out of four columns of $[\varphi]_B^C$ look like:
\begin{equation}
\begin{pmatrix}
1 & 5 & \cdot & \cdot\\
-1 & -4 & \cdot & \cdot \\
1 & 2 & \cdot & \cdot
\end{pmatrix}
\end{equation}
I'll leave it to you to figure out what the last two columns are (follow the same process I did).
Best Answer
Yes. The kernel are those vectors $v$ whose coordinates $x$; $y$, and $z$ with respect to the basis $B$ are such that$$\begin{bmatrix}6&9&12\\2&3&5\\0&0&0\end{bmatrix}.\begin{bmatrix}x\\y\\z\end{bmatrix}=\begin{bmatrix}0\\0\\0\end{bmatrix}.$$This is equivalent to$$\left\{\begin{array}{l}6x+9y+12z=0\\2x+3y+5z=0\\0=0.\end{array}\right.$$That is, the kernel is$$\left\{-\frac{3y}2\begin{bmatrix}1\\2\\3\end{bmatrix}+y\begin{bmatrix}1\\3\\4\end{bmatrix}\,\middle|\,y\in\mathbb R\right\}.$$
The image of $F$ is$$\left\langle\begin{bmatrix}6\\2\\0\end{bmatrix},\begin{bmatrix}9\\3\\0\end{bmatrix},\begin{bmatrix}12\\5\\0\end{bmatrix}\right\rangle.$$