A linear operator can be written as a matrix in a given basis.
For example, suppose we have the linear operator, T, from $R^2$ to $R^2$ that maps (x, y) to T(x, y)= (x- y, 2y). Since that is from $R^2$ to $R^2$, in can be written as a 2 by 2 matrix: $\begin{bmatrix}a & b \\ c & d \end{bmatrix}$. If we use the "standard basis" for $R^2$, (1, 0) and (0, 1), then (x, y)= x(1,0)+ y(0, 1) so $\begin{bmatrix} x \\ y\end{bmatrix}$ is the representation in the standard basis. The operation, in matrix form is $\begin{bmatrix}a & b \\ c & d \end{bmatrix}\begin{bmatrix}x \\ y\end{bmatrix}= \begin{bmatrix} ax+ by \\ cx+ dy\end{bmatrix}$.
We want that to be $T(x, y)= \begin{bmatrix} x- y \\ 2y\end{bmatrix}$ so we must have ax+ by= x- y and cx+ dy= 2y. That is two equations for the four unknowns, a, b, c, and d, but remember this must be true for all x and y. In particular, taking x= 1, y= 0 we get a(1)+ b(0)= a= 1- 0, so a= 1, and c(1)+ d(0)= 2(0), so c= 0. Taking x= 0, y= 1, we get a(0)+ b(1)= 0- 1, so b= -1, and c(0)+ d(1)= 2(1) so d= 2. The matrix representing linear operator, T, in this particular basis, is $\begin{bmatrix} 1 & -1 \\ 0 & 2\end{bmatrix}$. This particular choice of "x= 0, y= 1" and "x= 1, y= 0" makes the calculations especially easy since (0, 1) and (1, 0) are the basis vectors.
Notice that $\begin{bmatrix}a & b \\ c & d \end{bmatrix}\begin{bmatrix}1 \\ 0 \end{bmatrix}= \begin{bmatrix}a \\ c \end{bmatrix}$ and $\begin{bmatrix}a & b \\ c & d \end{bmatrix}\begin{bmatrix}0 \\ 1 \end{bmatrix}= \begin{bmatrix}b \\ d \end{bmatrix}$. That is, applying the linear operator to each basis vector in turn, then writing the result as a linear combination of the basis vectors gives us the columns of the matrices as those coefficients.
For another example, let the vector space be the set of all polynomials of degree at most 2 and the linear operator, D, be the differentiation operator. That is, any such "vector" can be written as $P= ax^2+ bx+ c$ and $DP= 2ax+ b$.
If we take $\{x^2,x, 1\}$ as basis $ax^2+ bx+ c$ will be written as $\begin{bmatrix} a \\ b \\ c\end{bmatrix}$. Applying the derivative operator to the first "basis vector", $x^2= 1x^2+ 0x+ 0$, gives $2x= 0x^2+ 2x+ 0$ so the first column of the matrix representation is $\begin{bmatrix}0 \\ 2 \\ 0\end{bmatrix}$. Applying the derivative operator to the second "basis vector", $x= 0x^2+ 1x+ 0$, gives $1= 0x^2+ 0x+ 1$ so the second column of the matrix representation is $\begin{bmatrix}0 \\ 0 \\ 1\end{bmatrix}$. Finally, applying the derivative operator to the third "basis vector", $1= 0x^2+ 0x+ 1$, gives $0= 0x^2+ 0x+ 0$ so the third column of the matrix representation is $\begin{bmatrix}0 \\ 0 \\ 0\end{bmatrix}$. The matrix representing the derivative operator in this basis is $\begin{bmatrix}0 & 0 & 0 \\ 2 & 0 & 0 \\ 0 & 1 & 0\end{bmatrix}$
Please bear with me if you know the initial part of this answer.
How do we write a matrix transformation? Given $(V,B_V)$ and $(W,B_W)$, where $V,W$ are vector spaces with bases $B_V,B_W$ respectively, and a linear transformation $T : V \to W$, how do we write the matrix for $T$, with respect to $B_V$ and $B_W$?
Arrange $B_V,B_W$ in some order.
Find where the elements of $B_V$ go under $T$.
Express the image vectors in terms of the basis $B_W$, and write the coefficients as columns one after the other to get a matrix.
For example, if $T : \mathbb R^2 \to \mathbb R^2$, both with standard bases, is given by $T(x) = -x$, then we find where $(1,0)$ goes : it goes to $(-1,0)$ in the standard basis, and then find where $(0,1)$ goes : it goes to $(0,-1)$ in the standard basis. Therefore, the matrix of $T$ will be : first column $(-1,0)$, second column $(0,-1)$. You can see that you will get minus the identity matrix if you put this together, for the matrix of $T$.
Now, what is a change of basis matrix? Here is a nice interpretation :
The change of basis matrix from basis $A$ to basis $B$ (in a vector space $V$), is the matrix of the identity linear transformation from $(V,A)$ to $(V,B)$.
That is, find where all elements of $A$ go, and express them in basis $B$.
What have you done? For the change of basis matrix from $B$ to $B_1$, this is the matrix of the identity map from $(V,B)$ to $(V,B_1)$. So, express the elements of $B$ in terms of the elements of $B_1$. You have done the opposite. You expressed $B_1$ in terms of $B$, when you wrote own your change of basis matrix.
The same with the third question.
Consequently, we see that the change of basis matrices are the inverses of what you have written, because base conversion needs to go the other way. The right answer for b and c, therefore, is the inverse of the matrices you have written.
Finally, for $d$, of course the given polynomial can be interpreted as an element of $(V,B)$. Now, to change the basis to $B_1$ we want to shift to $(V,B_1)$, so simply multiply the change of basis matrix you have with you, and the vector expressed in $B_1$.
That is, the inverse of the $P_{B,B_1}$ which you got, when multiplied with $[0,3,-1,8]^T$, gives you $[10,-10,-7,-8]$. You can check this is right.
Best Answer
Let $P = \begin{bmatrix} 1 & 2 & 1 \\ 0 & 1 & 2 \\ -3 & -5 & 0\end{bmatrix}$.
The function $t \mapsto t^2$ is represented as $(0,0,1)^T$ in the standard basis, so to find the representation in the basis $\cal B$, compute $P^{-1} (0,0,1)^T$ (note that you do not need to compute the matrix inverse to do this).
Here is the solution:
Here is another approach: Figure out constants $c_1,c_2,c_3$ such that $c_1(1-3t^2) + c_2 (2+t-5t^2) + c_3 (1+2t) = t^2$ for all $t$. Matching coefficients of $t^k$ gives three equations in three unknowns $c_1+ 2 c_2 + c_3 = 0$, $c_2+2 c_3 = 0$, and $-3 c_1-5 c_2 = 1$. Solving these gives the above solution.