NOTE Given a finite dimensional vector space $\Bbb V$ and a basis $B=\{v_1,\dots,v_n\}$ of $\Bbb V$, the coordinates of $v$ in base $B$ are the unique $n$ scalars $\{a_1,\dots,a_n\}$ such that $v=\sum_{k=1}^n a_kv_k$, and we note this by writing $(v)_B=(a_1,\dots,a_n)$.
All you need is to find what $T$ maps the basis elements to. Why? Because any vector in $P_2$ can be written as a linear combination of $1$, $t$ and $t^2$, whence if you know what $T(1)$, $T(t)$ and $T(t^2)$ are, you will find what any $T(a_0 +a_1 t +a_2 t^2)=a_0T(1)+a_1 T(t)+a_2 T(t^2)$ is. So, let us call $B=\{1,t,t^2\}$. Then
$$T(1)=3\cdot 0 +7\cdot 1=7=(7,0,0)$$
$$T(t)=3\cdot 1 +7\cdot t=(3,7,0)$$
$$T(t^2)=6\cdot t +7\cdot t^2=(0,6,7)$$
(Here I'm abusing the notation a bit. Formally, we should enclose the two first terms of the equations with $(-)_B$ )
Now note that our transformation matrix simply takes a vector in coordinates of base $B$, and maps it to another vector in coordinates of base $B$. Thus, if $|T|_{B,B}$ is our matrix from base $B$ to base $B$, we must have
$$|T|_{B,B} (P)_B=(T(P))_B$$
where we wrote $P=P(t)$ to avoid too much parenthesis.
But let's observe that if $(P)_B=(a_0,a_1,a_2)$ then $a_0T(1)+a_1 T(t)+a_2 T(t^2)=a_0(7,0,0)+a_1 (3,7,0)+a_2(0,6,7)$ is the matrix product
$$\left(\begin{matrix}7&3&0\\0&7&6\\0&0&7\end{matrix}\right)\left(\begin{matrix}a_0 \\a_1 \\a_2 \end{matrix}\right)$$
And $|T|_{B,B}=\left(\begin{matrix}7&3&0\\0&7&6\\0&0&7\end{matrix}\right)$ is precisely the matrix we're after. It has the property that for each vector of $P_2$
$$|T|_{B, B}(P)_B=(T(P))_B$$
(well, actually
$$(|T|_{B,B} (P)_B^t)^t=(T(P))_B$$
but that looks just clumsy, doesn't it?)
Best Answer
First you change coordinates to where your basis is given by $\begin{pmatrix} 1 \\ 2\end{pmatrix}$ and $\begin{pmatrix} 2 \\ -1\end{pmatrix}$. This is done by multiplying by $$\begin{pmatrix} 1 & 2 \\ 2 & -1\end{pmatrix}^{-1} = \frac{1}{5}\begin{pmatrix} 1 & 2 \\ 2 & -1\end{pmatrix}.$$ Next, you reflect the second coordinate. This is done by multiplying by $$ \begin{pmatrix} 1 & 0 \\ 0 & -1\end{pmatrix}.$$ Finally, you change your coordinates back by multiplying by $$\begin{pmatrix} 1 & 2 \\ 2 & -1\end{pmatrix}.$$ Thus, your transformation is multiplication by $$ \frac{1}{5}\begin{pmatrix} 1 & 2 \\ 2 & -1\end{pmatrix}\begin{pmatrix} 1 & 0 \\ 0 & -1\end{pmatrix}\begin{pmatrix} 1 & 2 \\ 2 & -1\end{pmatrix} = \frac{1}{5}\begin{pmatrix} 1 & 2 \\ 2 & -1\end{pmatrix}\begin{pmatrix} 1 & 2 \\ -2 & 1\end{pmatrix} = \frac{1}{5}\begin{pmatrix}-3 & 4 \\ 4 & 3 \end{pmatrix}.$$
Now try to do the second reflection yourself.