Here is how I would do it:
- First remember the change of basis matrix $P_\mathcal B^{\mathcal B'}$ from an
old base $\mathcal B$ of a vector space $ E $ to a new base $\mathcal B\,' $ has as column-vectors the coordinates of the newbase in the oldbase. It is the matrix of the identity map from $ (E, \mathcal B\,') $ to $ (E, \mathcal B) $.
- The change of basis matrix the other way is just the inverse matrix of the previous one: $$P_{\mathcal B'}^{\mathcal B}=\bigl(P_\mathcal B^{\mathcal B'}\bigr)^{-1}$$
- It allows to express the
old coordinates $X$ of a vector from the new coordinates $X'$: $$X=P_\mathcal B^{\mathcal B'}X'$$
- You can compose the change of basis matrix:
$$ P_{\mathcal B}^{\mathcal B''} = P_{\mathcal B}^{\mathcal B’}P_{\mathcal B’}^{\mathcal B''}$$
Denote the canonical base as $\mathrm{Can}$. Determining $\mathcal B$ is the same as determining $P_{\mathrm{Can}}^{\mathcal B}$. By the composition formula, we have:
$$P_{\mathrm{Can}}^{\mathcal B}=P_{\mathrm{Can}}^{\mathcal C}P_{\mathcal C}^{\mathcal B}=P_{\mathrm{Can}}^{\mathcal C}\bigl(P_{\mathcal B }^{\mathcal C}\bigr)^{-1}.$$
Now $ P_{\mathrm{Can}}^{\mathcal C}= \begin{bmatrix} 1 & 0 \\1&2\end{bmatrix} $, and we compute with the pivot method that
$$\bigl(P_{\mathcal B }^{\mathcal C}\bigr)^{-1}= \begin{bmatrix} 1 & 0 \\2&3\vphantom{\tfrac 13}\end{bmatrix}^{-1}= \begin{bmatrix} 1 & 0 \\-\tfrac23& \tfrac 13\end{bmatrix}$$
so that $$P_{\mathrm{Can}}^{\mathcal B}=\begin{bmatrix} 1 & 0 \\-\tfrac 13& \tfrac 23\end{bmatrix}$$
Hence $b_1=\begin{bmatrix} 1 \\-\tfrac 13\end{bmatrix}$, $\quad b_2=\begin{bmatrix} 0 \\\tfrac 23\end{bmatrix}$.
Let
$\mathcal{E}=\left\{e_1,e_2,e_3\right\}$
be our canonical base. With this base, transormation T has representation
$T=\left(
\begin{array}{ccc}
3 & 1 & 0 \\
1 & 0 & 1 \\
1 & 0 & -1 \\
\end{array}
\right)$.
Now we have got a new base:
$\mathcal{F}=\left\{e_1,e_1+e_2,e_1+e_2+e_3\right\}$.
Let
$M_{\mathcal{F}}=\left(
\begin{array}{ccc}
1 & 1 & 1 \\
0 & 1 & 1 \\
0 & 0 & 1 \\
\end{array}
\right)$
be the transition between the two bases.
Then canonical coordinates are transormed in new coordinates
(with respect to base $\mathcal{F}$ ) by inverse matrix, which is
$N_{\mathcal{F}}=\left(
\begin{array}{ccc}
1 & -1 & 0 \\
0 & 1 & -1 \\
0 & 0 & 1 \\
\end{array}
\right)$.
Take
$A=\left\{a_1,a_2,a_3\right\}$
and get new coordinates
$B=N_{\mathcal{F}}.A$.
Then, with $S=T.M_{\mathcal{F}}$
we see:
$T.A=T.M_{\mathcal{F}}.N_{\mathcal{F}}.A=S.B$.
It's not a miracle, only lin. Algebra.
Key is transformation of basis, which implies
transformation of coordinates. That's all.
By the way: Calculating without inverses is not
possible. Your transformation with bases must be
regular. They must be invertible, otherwise it didn't
work.
Let's see. Other basis
$\mathcal{B}=\left\{2 e_1+5 e_3,e_1+e_2+6 e_3,3 e_1+9 e_3\right\}$,
another transition:
$M_{\mathcal{B}}=\left(
\begin{array}{ccc}
2 & 1 & 3 \\
0 & 1 & 0 \\
5 & 6 & 9 \\
\end{array}
\right)$.
The inverse:
$N_{\mathcal{B}}=\left(
\begin{array}{ccc}
3 & 3 & -1 \\
0 & 1 & 0 \\
-\frac{5}{3} & -\frac{7}{3} & \frac{2}{3} \\
\end{array}
\right)$.
Old transformation T
$T=\left(
\begin{array}{ccc}
3 & 1 & 0 \\
1 & 0 & 1 \\
1 & 0 & -1 \\
\end{array}
\right)$.
Transformed T:
$S=T.M_{\mathcal{B}}=\left(
\begin{array}{ccc}
6 & 4 & 9 \\
7 & 7 & 12 \\
-3 & -5 & -6 \\
\end{array}
\right)$
Transformed A:
$B=N_{\mathcal{B}}.A$.
$T.A=T.M_{\mathcal{B}}.N_{\mathcal{B}}.A=S.B$
Like before.
Best Answer
The change of basis matrix from $B$ to $S$ is
$$P = \begin{pmatrix} 1 & r & r^2 \\ 0 & 1 & 2r \\ 0 & 0 & 1\end{pmatrix},$$
so the change of basis matrix from $S$ to $B$ is
$$P^{-1} = \begin{pmatrix} 1 & -r & r^2 \\ 0 & 1 & -2r \\ 0 & 0 & 1\end{pmatrix}.$$
Hence the $B$-coordinates of $p(x) = a_0 + a_1 x + a_2x^2$ are
$$ P^{-1}\begin{pmatrix} a_0 \\ a_1 \\ a_2 \end{pmatrix} = \begin{pmatrix} a_0 - a_1r + a_2 r^2 \\ a_1 - 2a_2 r \\ a_2 \end{pmatrix}.$$