Let $\mathbf{x} = (x_{1}, x_{2}, x_{3})^T$ be the coordinates of a point in the $e$-basis and let $\mathbf{y} = (y_{1}, y_{2}, y_{3})^T$ be the coordinates of the same point in the $f$-basis.
It is the same point, so we require the following condition.
$$
x_{1} \mathbf{e}_{1} +
x_{2} \mathbf{e}_{2} +
x_{3} \mathbf{e}_{3}
=
y_{1} \mathbf{f}_{1} +
y_{2} \mathbf{f}_{2} +
y_{3} \mathbf{f}_{3}
$$
The question gives the way of writing the $f$-basis vectors in terms of the $e$-basis vectors:
$$
\begin{aligned}
\mathbf{f}_1 &= \mathbf{e}_1 + \mathbf{e}_2 \\
\mathbf{f}_2 &= \mathbf{e}_2 \\
\mathbf{f}_3 &= \mathbf{e}_1 - \mathbf{e}_3
\end{aligned}
$$
We can substutite these formulas into the equation for the coordinates above
$$
x_{1} \mathbf{e}_{1} +
x_{2} \mathbf{e}_{2} +
x_{3} \mathbf{e}_{3}
=
y_{1} (\mathbf{e}_{1} + \mathbf{e}_{2}) +
y_{2} \mathbf{e}_{2} +
y_{3} (\mathbf{e}_{1} - \mathbf{e}_{3})
$$
$$
x_{1} \mathbf{e}_{1} +
x_{2} \mathbf{e}_{2} +
x_{3} \mathbf{e}_{3}
=
y_{1} \mathbf{e}_{1} + y_{1} \mathbf{e}_{2} +
y_{2} \mathbf{e}_{2} +
y_{3} \mathbf{e}_{1} - y_{3} \mathbf{e}_{3}
$$
$$
x_{1} \mathbf{e}_{1} +
x_{2} \mathbf{e}_{2} +
x_{3} \mathbf{e}_{3}
=
(y_{1} + y_{3}) \mathbf{e}_{1}
+
(y_{1} + y_{2} ) \mathbf{e}_{2}
- y_{3} \mathbf{e}_{3}
$$
Now $\mathbf{e}_1$, $\mathbf{e}_2$ and $\mathbf{e}_3$ are three linearly independent vectors so that the components of each vector can be equated on both sides of the above. I.e. we can write:
$$
\begin{aligned}
x_1 &= y_1 + y_3 \\
x_2 & = y_1 + y_2 \\
x_3 &= -y_3
\end{aligned}
$$
The following is exactly the same as the above with slightly different spacing
$$
\begin{aligned}
x_1 &= y_1 & &+y_3 \\
x_2 & = y_1 &+ y_2 & \\
x_3 &= & &-y_3
\end{aligned}
$$
Writing the equations that relate the coordinates in this way, we can see how the set can be written as a single matrix equation:
$$
\begin{pmatrix}
x_{1} \\ x_{2} \\ x_{3}
\end{pmatrix}
=
\begin{pmatrix}
1 & 0 & 1 \\
1 & 1 & 0 \\
0 & 0 & -1
\end{pmatrix}
\begin{pmatrix}
y_{1} \\ y_{2} \\ y_{3}
\end{pmatrix}
\Rightarrow
\mathbf{x}
=
\begin{pmatrix}
1 & 0 & 1 \\
1 & 1 & 0 \\
0 & 0 & -1
\end{pmatrix}
\mathbf{y}$$
This shows how we can use a matrix to convert coordinates in the $f$-basis to coordinates in the $e$-basis, i.e. $\mathbf{x} = T \mathbf{y}$, i.e. it represents the matrix $T$ in the formula
$$
A' = T^{-1} A T
$$
where $A$ is the transformation that is applied to coordinates in the $e$-basis. The above formula is applied
Having found $T$, we can find its inverse (by hand or with some software):
$$
T^{-1}
=
\begin{pmatrix}
1& -1& 0 \\
0& 1& 0 \\
1& -1& -1
\end{pmatrix}
$$
and finally, we can calculate $A'$
$$
A'
=
\begin{pmatrix}
-3& -2& -2 \\
3& 3& -1 \\
-7& -1& -6
\end{pmatrix}
$$
which is the matrix of the transformation that is applied to coordinates in the $f$-basis.
This next part goes into how the formula relating the two transformation matrices in the different bases is derived.
If we write $\mathbf{u}$
for the result of applying $A$ to the $e$-basis vector $\mathbf{x}$ and if we write
$\mathbf{v}$ for the result of applying $A'$ to the $f$-basis vector $\mathbf{y}$.
$$
\begin{aligned}
\mathbf{u} &= A \mathbf{x} \\
\mathbf{v} &= A' \mathbf{y} \\
\end{aligned}
$$
The vectors
$\mathbf{x}$ and $\mathbf{y}$
correspond to the same point in the two different bases and so do the pair of vectors
$\mathbf{u}$ and $\mathbf{v}$. In other words they can be written:
$$
\begin{aligned}
\mathbf{x} &= T \mathbf{y} \\
\mathbf{u} &= T \mathbf{v} \\
\end{aligned}
$$
This means we can write the following
$$
\begin{aligned}
\mathbf{u} &= T \mathbf{v} \\
A \mathbf{x} &= T \mathbf{v} \\
A \mathbf{x} &= T A' \mathbf{y} \\
A T\mathbf{y} &= T A' \mathbf{y} \\
T^{-1} A T\mathbf{y} &= A' \mathbf{y} \\
\end{aligned}
$$
As this works for all $\mathbf{y}$ we can conclude that $T^{-1} A T = A' $.
First of all, notice that $M_{2x2} \cong R^4 $ with $ \begin{bmatrix}a&b\cr c&d\end{bmatrix} \to (a,b,c,d)$.
Your first step is the right way to go, but I would be careful with the notation. Let
$\alpha$ be the standard basis auf $\mathbb{R}^3$, then
$$T(\begin{bmatrix}2&0\cr 0&0\end{bmatrix})=(1,0,1)_B = (2,5,10)_\alpha =(10,-3,-5)_{B'}$$
$$T(\begin{bmatrix}0&3\cr 0&0\end{bmatrix})=(2,1,1)_B = (4,8,9)_\alpha =(9,-4,-1)_{B'}$$
$$T(\begin{bmatrix}0&0\cr 5&0\end{bmatrix})=(0,1,-1)_B =(2,5,12)_\alpha =(12,-3,-7)_{B'}$$
$$T(\begin{bmatrix}0&0\cr 0&6\end{bmatrix})=(1,0,-1)_B =(0,-3,-8)_\alpha =(-8,3,5)_{B'}$$
Thus, we have
$$[T]_{B',A} = \begin{bmatrix}10&9&12&-8\cr -3&-4&-3&3\cr -5&-1&-7&5\end{bmatrix}$$
Now we are almost done. The new basis $A'$ is good to handle because every new basis vector is a scalar multiple of a vector of $A$. For example the first basis vector of $A'$ is half of its counterpart of $A$. (So $(2,0,0,0)_{A'} = (1,0,0,0)_A$ ). So if you map $(1,0,0,0)_{A'}$ with our wanted matrix, it will have the same result as if we map $(\frac{1}{2},0,0,0)_A$ into $[T]_{B',A}$ (because they both map into the vector space generated by $B'$)
Thus, the first column of $[T]_{B',A'}$ is $\begin{pmatrix}5\cr-\frac{3}{2}\cr-\frac{5}{2}\end{pmatrix}$. You can get the other columns with the same method. The solution is:
$$[T]_{B',A'} = = \begin{bmatrix}5&9\frac{4}{3}&12\frac{2}{5}&-8\frac{7}{6}\cr -\frac{3}{2}&-4\frac{4}{3}&-3\frac{2}{5}&3\frac{7}{6}\cr \frac{5}{2}&-1\frac{4}{3}&-7\frac{2}{5}&5\frac{7}{6}\end{bmatrix}$$
Best Answer
Notice that your ordered basis is in fact an orthogonal basis under the standard inner product. In particular, that means we have $$v = \frac{\langle v,\ f_1\rangle}{2} f_1 + \frac{\langle v,\ f_2\rangle}{2} f_2 + \frac{\langle v,\ f_3\rangle}{2} f_3 + \frac{\langle v,\ f_4\rangle}{2} f_4$$ This gives you $$v = f_1 - 3f_2 -2f_3$$ so your coordinates are $$[v]_F = \begin{pmatrix}1\\-3\\-2\\0\end{pmatrix}$$