Basically the way to think about your matrix product is as a series of transformations.
First you transform your vector from the new basis to the standard basis. Notice that $H$ transforms vectors from the new basis to their representations wrt the standard basis. This is exactly what we want. So the first matrix (the one that'll go on the right -- and thus will be multiplied by the vector on the right first) will be $H$.
Now that your vector is in the standard basis, you want to transform your vector with $T$. So that'll be your middle matrix in this product.
Finally, you'd like to transform your vector from it's coordinates with respect to the standard basis to the new basis. This is just the inverse of $H$: i.e. $H^{-1}$.
So your matrix wrt to the new basis is just $T' = H^{-1}TH$.
P.S. I don't see how solving for $x$ in $Hx=T$, which is presumably a column vector, has anything to do with getting the matrix which represents the same linear transformation as $T$ wrt your new basis.
Let
$\mathcal{E}=\left\{e_1,e_2,e_3\right\}$
be our canonical base. With this base, transormation T has representation
$T=\left(
\begin{array}{ccc}
3 & 1 & 0 \\
1 & 0 & 1 \\
1 & 0 & -1 \\
\end{array}
\right)$.
Now we have got a new base:
$\mathcal{F}=\left\{e_1,e_1+e_2,e_1+e_2+e_3\right\}$.
Let
$M_{\mathcal{F}}=\left(
\begin{array}{ccc}
1 & 1 & 1 \\
0 & 1 & 1 \\
0 & 0 & 1 \\
\end{array}
\right)$
be the transition between the two bases.
Then canonical coordinates are transormed in new coordinates
(with respect to base $\mathcal{F}$ ) by inverse matrix, which is
$N_{\mathcal{F}}=\left(
\begin{array}{ccc}
1 & -1 & 0 \\
0 & 1 & -1 \\
0 & 0 & 1 \\
\end{array}
\right)$.
Take
$A=\left\{a_1,a_2,a_3\right\}$
and get new coordinates
$B=N_{\mathcal{F}}.A$.
Then, with $S=T.M_{\mathcal{F}}$
we see:
$T.A=T.M_{\mathcal{F}}.N_{\mathcal{F}}.A=S.B$.
It's not a miracle, only lin. Algebra.
Key is transformation of basis, which implies
transformation of coordinates. That's all.
By the way: Calculating without inverses is not
possible. Your transformation with bases must be
regular. They must be invertible, otherwise it didn't
work.
Let's see. Other basis
$\mathcal{B}=\left\{2 e_1+5 e_3,e_1+e_2+6 e_3,3 e_1+9 e_3\right\}$,
another transition:
$M_{\mathcal{B}}=\left(
\begin{array}{ccc}
2 & 1 & 3 \\
0 & 1 & 0 \\
5 & 6 & 9 \\
\end{array}
\right)$.
The inverse:
$N_{\mathcal{B}}=\left(
\begin{array}{ccc}
3 & 3 & -1 \\
0 & 1 & 0 \\
-\frac{5}{3} & -\frac{7}{3} & \frac{2}{3} \\
\end{array}
\right)$.
Old transformation T
$T=\left(
\begin{array}{ccc}
3 & 1 & 0 \\
1 & 0 & 1 \\
1 & 0 & -1 \\
\end{array}
\right)$.
Transformed T:
$S=T.M_{\mathcal{B}}=\left(
\begin{array}{ccc}
6 & 4 & 9 \\
7 & 7 & 12 \\
-3 & -5 & -6 \\
\end{array}
\right)$
Transformed A:
$B=N_{\mathcal{B}}.A$.
$T.A=T.M_{\mathcal{B}}.N_{\mathcal{B}}.A=S.B$
Like before.
Best Answer
What you need to do is form the matrix $B = (\vec{b}_1|\vec{b}_2)$, where $\vec{b}_i$ is the $i$th column of B, and note that this matrix converts vectors from the standard basis into the basis $\mathcal{B}$, while the inverse $B^{-1}$ will convert vectors in the basis $\mathcal{B}$ into the standard basis. Thus if you have a vector already in the basis $\mathcal{B}$, you can convert it to standard basis by multiplying by $B^{-1}$, multiply it by $A$, and finally convert back to $\mathcal{B}$ by multiplying by $B$, so your overall matrix is $BAB^{-1}$.