Let $\mathbf{x} = (x_{1}, x_{2}, x_{3})^T$ be the coordinates of a point in the $e$-basis and let $\mathbf{y} = (y_{1}, y_{2}, y_{3})^T$ be the coordinates of the same point in the $f$-basis.

It is the same point, so we require the following condition.
$$
x_{1} \mathbf{e}_{1} +
x_{2} \mathbf{e}_{2} +
x_{3} \mathbf{e}_{3}
=
y_{1} \mathbf{f}_{1} +
y_{2} \mathbf{f}_{2} +
y_{3} \mathbf{f}_{3}
$$

The question gives the way of writing the $f$-basis vectors in terms of the $e$-basis vectors:
$$
\begin{aligned}
\mathbf{f}_1 &= \mathbf{e}_1 + \mathbf{e}_2 \\
\mathbf{f}_2 &= \mathbf{e}_2 \\
\mathbf{f}_3 &= \mathbf{e}_1 - \mathbf{e}_3
\end{aligned}
$$

We can substutite these formulas into the equation for the coordinates above

$$
x_{1} \mathbf{e}_{1} +
x_{2} \mathbf{e}_{2} +
x_{3} \mathbf{e}_{3}
=
y_{1} (\mathbf{e}_{1} + \mathbf{e}_{2}) +
y_{2} \mathbf{e}_{2} +
y_{3} (\mathbf{e}_{1} - \mathbf{e}_{3})
$$

$$
x_{1} \mathbf{e}_{1} +
x_{2} \mathbf{e}_{2} +
x_{3} \mathbf{e}_{3}
=
y_{1} \mathbf{e}_{1} + y_{1} \mathbf{e}_{2} +
y_{2} \mathbf{e}_{2} +
y_{3} \mathbf{e}_{1} - y_{3} \mathbf{e}_{3}
$$

$$
x_{1} \mathbf{e}_{1} +
x_{2} \mathbf{e}_{2} +
x_{3} \mathbf{e}_{3}
=
(y_{1} + y_{3}) \mathbf{e}_{1}
+
(y_{1} + y_{2} ) \mathbf{e}_{2}
- y_{3} \mathbf{e}_{3}
$$

Now $\mathbf{e}_1$, $\mathbf{e}_2$ and $\mathbf{e}_3$ are three linearly independent vectors so that the components of each vector can be equated on both sides of the above. I.e. we can write:

$$
\begin{aligned}
x_1 &= y_1 + y_3 \\
x_2 & = y_1 + y_2 \\
x_3 &= -y_3
\end{aligned}
$$

The following is exactly the same as the above with slightly different spacing
$$
\begin{aligned}
x_1 &= y_1 & &+y_3 \\
x_2 & = y_1 &+ y_2 & \\
x_3 &= & &-y_3
\end{aligned}
$$

Writing the equations that relate the coordinates in this way, we can see how the set can be written as a single matrix equation:

$$
\begin{pmatrix}
x_{1} \\ x_{2} \\ x_{3}
\end{pmatrix}
=
\begin{pmatrix}
1 & 0 & 1 \\
1 & 1 & 0 \\
0 & 0 & -1
\end{pmatrix}
\begin{pmatrix}
y_{1} \\ y_{2} \\ y_{3}
\end{pmatrix}
\Rightarrow
\mathbf{x}
=
\begin{pmatrix}
1 & 0 & 1 \\
1 & 1 & 0 \\
0 & 0 & -1
\end{pmatrix}
\mathbf{y}$$

This shows how we can use a matrix to convert coordinates in the $f$-basis to coordinates in the $e$-basis, i.e. $\mathbf{x} = T \mathbf{y}$, i.e. it represents the matrix $T$ in the formula

$$
A' = T^{-1} A T
$$

where $A$ is the transformation that is applied to coordinates in the $e$-basis. The above formula is applied

Having found $T$, we can find its inverse (by hand or with some software):
$$
T^{-1}
=
\begin{pmatrix}
1& -1& 0 \\
0& 1& 0 \\
1& -1& -1
\end{pmatrix}
$$

and finally, we can calculate $A'$

$$
A'
=
\begin{pmatrix}
-3& -2& -2 \\
3& 3& -1 \\
-7& -1& -6
\end{pmatrix}
$$
which is the matrix of the transformation that is applied to coordinates in the $f$-basis.

This next part goes into how the formula relating the two transformation matrices in the different bases is derived.

If we write $\mathbf{u}$
for the result of applying $A$ to the $e$-basis vector $\mathbf{x}$ and if we write
$\mathbf{v}$ for the result of applying $A'$ to the $f$-basis vector $\mathbf{y}$.

$$
\begin{aligned}
\mathbf{u} &= A \mathbf{x} \\
\mathbf{v} &= A' \mathbf{y} \\
\end{aligned}
$$

The vectors
$\mathbf{x}$ and $\mathbf{y}$
correspond to the same point in the two different bases and so do the pair of vectors
$\mathbf{u}$ and $\mathbf{v}$. In other words they can be written:
$$
\begin{aligned}
\mathbf{x} &= T \mathbf{y} \\
\mathbf{u} &= T \mathbf{v} \\
\end{aligned}
$$

This means we can write the following
$$
\begin{aligned}
\mathbf{u} &= T \mathbf{v} \\
A \mathbf{x} &= T \mathbf{v} \\
A \mathbf{x} &= T A' \mathbf{y} \\
A T\mathbf{y} &= T A' \mathbf{y} \\
T^{-1} A T\mathbf{y} &= A' \mathbf{y} \\
\end{aligned}
$$
As this works for all $\mathbf{y}$ we can conclude that $T^{-1} A T = A' $.

There are a couple ways to view a dot product as a linear map by changing your view slightly.

The map $\langle \cdot,\cdot \rangle : V\times V\to F$ is not linear, it is what we call *bilinear*, which means that it is linear in each variable. I.e. a map $B : V\times V' \to W$ is *bilinear* if for all fixed $v\in V$, and all fixed $v'\in V'$ the maps $u'\mapsto B(v, u')$ and $u\mapsto B(u, v')$ are linear. (Though these maps are equal in the case of a dot product, since it is symmetric, so you just need to check that one is linear. Symmetric meaning that $\langle v, w\rangle= \langle w, v\rangle$)

The other view is perhaps a little more faithful to the idea of viewing the dot product as a linear map, but essentially equivalent. Though perhaps a little more abstract (though such judgments are inherently subjective).

The idea is that we can take a bilinear map $B: V\times V' \to W$ and turn it into a linear map $\tilde{B}: V\to \newcommand\Hom{\operatorname{Hom}}\Hom_F(V',W)$. Where $\Hom_F(V',W)$ denotes the vector space of $F$-linear maps from $V'$ to $W$. We define $\tilde{B}(v) = v'\mapsto B(v,v')$. Then one can use the defining property of bilinear maps given above to show that $\tilde{B}$ is linear, and for any $v\in V$, $\tilde{B}(v)$ is a linear map from $V'$ to $W$. This process is called currying. Then $\tilde{B}$ is basically the same as $B$, since we can recover $B$ from $\tilde{B}$ from the fact that $B(v,v')=(\tilde{B}v)v'$ (sorry for changing notation to parenthesis-less function application, I just think it's much more readable here).

Thus one can curry the dot product to get a linear map, call it $D$ from $V$ to $\Hom_F(V,F)$. In general, $\Hom_F(V,F)$ is a vector space called *$V$-dual*, often written $V^*$, so we can say $D$ is a linear map from $V$ to $V^*$. I.e., we can view the dot product as being equivalent to a particular nice linear map from $V$ to $V^*$.

## Best Answer

Yes, $T$ is a matrix of some kind. You can tell that it is $2\times 2$ since both its inputs and outputs are vectors of length $2$. My recommendation for solving this is the following. Suppose $T$ has matrix representation

$$ T \;\; =\;\; \left [ \begin{array}{cc} a & b \\ c & d \\ \end{array} \right ]. $$

Start with the first equation which you can write as

$$ \left [ \begin{array}{cc} a & b \\ c & d \\ \end{array} \right ] \left [ \begin{array}{c} 1 \\ -1\\ \end{array} \right ] \;\; =\;\; \left [ \begin{array}{c} 0 \\ 3\\ \end{array} \right ]. $$

You can find the values of $a,b,c,d$ by working through these equations.