Recall what the matrix $[\, T\,]_B$ is:
If you write a vector $x\in\Bbb R^2$ in terms of the basis $B=\{\,b_1,b_2\,\}$
$$
x=\alpha_1 b_1+\alpha_2 b_2,
$$
then if you multiply $[\, T\,]_B$ by the coordinate vector $x_B=\bigl[{\alpha_1\atop\alpha_2}\bigr]$, you get the coordinate vector of $T(x)$ with respect to $B$.
That is
$$\tag{1}
[T(x)]_B = [\,T\,]_B x_B.
$$
Now the columns of the matrix $[\,T\,]_E$ where $E=\{e_1,e_2\}$ are the vectors $T(e_1)$ and $T(e_2)$. To find these vectors, we can use the matrix $[\, T\,]_B$.
There are three steps involved here. Considering the vector $e_2$, we have to
- Find the coordinates of $e_2$ with respect to the basis $B$.
- Find the coordinates of $T(e_2)$ with respect to the basis $B$.
- Find $T(e_2)$ expressed in the standard basis.
Step 1: For $e_2=(0,1)$, we first find the coordinates of $e_2$ in terms of the basis $B$. Towards this end, we have to solve the system
$$
\Bigl[{0\atop1}\Bigr]= \alpha_1 \Bigl[{-1\atop-3}\Bigr] +\alpha_2\Bigl[{-3\atop-10}\Bigr].
$$
Doing so gives:
$$\alpha_1=3,\quad \alpha_2=-1$$ The coordinate vector of $e_2$ with respect to $B$ is
$\bigl[{3\atop-1}\bigr]$.
Note we could have done this differently: the coordinate vector $\bigl[{\alpha_1\atop\alpha_2}\bigr]$ of $x$ with respect to $B$ satisfies $[b_1 \ b_2] \bigl[{\alpha_1\atop\alpha_2}\bigr] =x$; so $\bigl[{\alpha_1\atop\alpha_2} \bigr]=[b_1 \ b_2]^{-1} x$.
Thus we could have found $[b_1\ b_2]^{-1}$ and just multiplied this by $e_2$. This is actually preferable, since we can use the inverse when considering $e_1$ later.
Step 2:
Using $(1)$ now, the coordinate vector of $T(e_2)$ with respect to $B$ is
$$
[\,T\,]_B (e_2)_B =
\Bigl[ \matrix{4&4\cr 4&5 }\Bigr]\Bigl[{3\atop -1} \Bigr] =
\Bigl[{8\atop 7} \Bigr].
$$
Step 3:
But note that $T(e_2)$ is not the vector $\bigl[{8\atop 7} \bigr]$; this vector gives the coordinates of $T(e_2)$ with respect to the basis $B$. In general, if $x_B$ is the coordinate vector of $x$ with respect to $B$, then $x=[b_1\ b_2] x_B$;
so $$
T(e_2)=[b_1\ b_2]\Bigl[{8\atop 7} \Bigr] =8\, b_1+7\, b_2= 8\Bigl[{-1\atop -3} \Bigr] +7 \Bigl[{-3\atop -10} \Bigr]
=\Bigl[{-29\atop -94} \Bigr].
$$
Thus, the second column of $[\,T\,]_E$ is $\bigl[{-29\atop -94} \bigr]$.
To find the first column
of $[\,T\,]_E$, apply the same procedure to the vector $e_1$. The first step here would be to write $e_1$ in terms of the basis $B$. To do that, you need to solve the system
$$
\Bigl[{1\atop0}\Bigr]= \alpha_1 \Bigl[{-1\atop-3}\Bigr] +\alpha_2\Bigl[{-3\atop-10}\Bigr].
$$
or compute:
$$\Bigl[{\alpha_1\atop\alpha_2} \Bigr]=[b_1 \ b_2]^{-1} e_1.$$
In general, given $x$ written in terms of the standard basis, the coordinates of $x$ with respect to $B$ are given by $ P^{-1} x$ where $P=[b_1\ b_2]$. Then the coordinates of $T(x)$ with respect to $B$ are $[\,T\,]_B P^{-1} x$. Then we have that $T(x)$ expressed in terms of the usual basis is $P [\,T\,]_B P^{-1} x$. So,
$$
[\,T\,]_E = P[\,T\,]_B P^{-1}.
$$
All those four cases are really just 1 case. Given vector space $U$ with basis $B = \{u_1, \dots, u_n\}$, define the coordinate isomorphism $J_B \colon \mathbb{F}^n \to U$ by
$$J_Be_j = u_j \text{ for each } j \in \{1, \dots, n\}.$$
Suppose $T \colon V \to W$ is linear, $B_1 = \{v_1, \dots, v_n\}$ is a basis of $V$, and $B_2 = \{w_1, \dots, w_m\}$ is a basis of $W$. Then the matrix representation of $T$ with respect to these bases is
$$M_{B_1}^{B_2}(T) = J_{B_2}^{-1}TJ_{B_1}.$$
So $M_{B_1}^{B_2}(T)$ takes in $B_1$-coordinates of a vector $v \in V$ and returns $B_2$-coordinates of $Tv$. Thus the $j$-th column of $M_{B_1}^{B_2}(T)$ is the $B_2$-coordinates of $Tv_j$.
Best Answer
You're given a basis which I'll label as $\vec{v}_1 = \cos(t)$ and $\vec{v}_2 = \sin(t)$. This lets you interpret columns of numbers as vectors. The column vector $\begin{bmatrix} a\\b\end{bmatrix}$ literally means the vector $a\cos(t) + b\sin(t)$.
Writing the matrix $A$ relative to this basis is done by computing $A\vec{v}_1$ and expressing it in terms of $\vec{v}_1$ and $\vec{v}_2$ and then doing the same for $A\vec{v}_2$.
In matrix language, $T\vec{v}_1$ corresponds to $A\begin{bmatrix}1\\0\end{bmatrix}$. On the other hand, $T\vec{v}_1 = T\cos(t) = \cos(t)''+7\cos(t)' +4\cos(t) = 3\cos(t) -7\sin(t) = \begin{bmatrix}3\\-7\end{bmatrix}$
So, the matrix $A$ should have the property that $A\begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix} 3\\-7\end{bmatrix}$. This forces $A$ to have the form $A = \begin{bmatrix} 3 & *\\-7 & *\end{bmatrix}$.
Can you take it from here?