NOTE Given a finite dimensional vector space $\Bbb V$ and a basis $B=\{v_1,\dots,v_n\}$ of $\Bbb V$, the coordinates of $v$ in base $B$ are the unique $n$ scalars $\{a_1,\dots,a_n\}$ such that $v=\sum_{k=1}^n a_kv_k$, and we note this by writing $(v)_B=(a_1,\dots,a_n)$.
All you need is to find what $T$ maps the basis elements to. Why? Because any vector in $P_2$ can be written as a linear combination of $1$, $t$ and $t^2$, whence if you know what $T(1)$, $T(t)$ and $T(t^2)$ are, you will find what any $T(a_0 +a_1 t +a_2 t^2)=a_0T(1)+a_1 T(t)+a_2 T(t^2)$ is. So, let us call $B=\{1,t,t^2\}$. Then
$$T(1)=3\cdot 0 +7\cdot 1=7=(7,0,0)$$
$$T(t)=3\cdot 1 +7\cdot t=(3,7,0)$$
$$T(t^2)=6\cdot t +7\cdot t^2=(0,6,7)$$
(Here I'm abusing the notation a bit. Formally, we should enclose the two first terms of the equations with $(-)_B$ )
Now note that our transformation matrix simply takes a vector in coordinates of base $B$, and maps it to another vector in coordinates of base $B$. Thus, if $|T|_{B,B}$ is our matrix from base $B$ to base $B$, we must have
$$|T|_{B,B} (P)_B=(T(P))_B$$
where we wrote $P=P(t)$ to avoid too much parenthesis.
But let's observe that if $(P)_B=(a_0,a_1,a_2)$ then $a_0T(1)+a_1 T(t)+a_2 T(t^2)=a_0(7,0,0)+a_1 (3,7,0)+a_2(0,6,7)$ is the matrix product
$$\left(\begin{matrix}7&3&0\\0&7&6\\0&0&7\end{matrix}\right)\left(\begin{matrix}a_0 \\a_1 \\a_2 \end{matrix}\right)$$
And $|T|_{B,B}=\left(\begin{matrix}7&3&0\\0&7&6\\0&0&7\end{matrix}\right)$ is precisely the matrix we're after. It has the property that for each vector of $P_2$
$$|T|_{B, B}(P)_B=(T(P))_B$$
(well, actually
$$(|T|_{B,B} (P)_B^t)^t=(T(P))_B$$
but that looks just clumsy, doesn't it?)
There is a mistake in the fourth formula, the one you are trying to
understand (that is apparent from the last formula where that mistake
disappears). Precisely, I mean it should be written $$\hat{f} \left( x \right) =
\frac{\hat{f}_{\text{trans}} \left(
T_{\hat{\alpha}, \hat{M}, \hat{c}} \left( x \right) \right)}{\left| \left(
T^{- 1}_{\hat{\alpha}, \hat{M}, \hat{c}} \right)' \left( x
\right) \right|}$$ and not $$\hat{f} \left( x \right) =
\frac{\hat{f}_{\text{}} \left(
T_{\hat{\alpha}, \hat{M}, \hat{c}} \left( x \right) \right)}{\left| \left(
T^{- 1}_{\hat{\alpha}, \hat{M}, \hat{c}} \right)' \left(
T_{\hat{\alpha}, \hat{M}, \hat{c}} \left( x \right) \right) \right|}$$
The notation in these formulas are clumsy and not very intuitive, but I will explain how that formula is derived and where the mistake occurs.
The relation between the two random variables $Y$ and $X$ is given by $Y = T
\left( X \right)$ (I will denote by $T$ the function $T_{\hat{\alpha},
\hat{M}, \hat{c}}$ to simplify notation). The transformation $T$ is the
cumulative distribution function of an absolutely continuous random variable
and thus is strictly montonically increasing with unique inverse $T^{- 1}$.
Let $t \left( x \right) = T' \left( x \right) = \frac{\partial T \left( x
\right)}{\partial x}$ be the density corresponding to $T$. Denote the by $f_X$
and $f_Y$ the densities of $X$ and $Y$. The relation between the two
densities is
$$ f_X \left( x \right) = f_Y \left( T \left( x \right) \right) t \left(
\left. x \right) \right. $$
$$ f_Y \left( y \right) = f_X \left( T^{- 1} \left( y \right) \right)
\frac{1}{t \left( T^{- 1} \left( y \right) \right)} $$
This is clear because the jacobian of the transformation $X = T^{- 1} \left( y
\right)$ is $t \left( x \right)$. (and not $t \left( T \left( x \right)
\right)$ as is assumed in the fourth formula). The term $\left|
\frac{1}{\left( T^{- 1} \right)' T \left( x \right)} \right|$ appears mistakenly in the
fourth formula because
$$ t \left( T \left( x \right) \right) = \left| t \left( T \left( x \right)
\right) \right| = \left| T' \left( T \left( x \right) \right) \right| =
\left| \frac{1}{\left( T^{- 1} \right)' T \left( x \right)} \right| $$
(Notice that the derivative of the inverse function is the inverse of the
derivative of the original function. Also there is no need for the absolute
values here because densities are positive. To quickly check the error notice that $T:(0,\infty)\rightarrow (0,1)$ and $t:(0,\infty)\rightarrow (0,\infty)$). In the last formula, the last
term is $t \left( x \right)$ and not the incorrect one $t \left( T \left( x \right) \right)$.
Best Answer
There are lots of ways to do this. Here's one way.
Let \begin{align*} \vec v_1 &= (1,2,3,4) & \vec v_2&=(0,1,0,1) \end{align*} Our strategy will be to first extend $\{\vec v_1,\vec v_2\}$ to a basis $\{\vec v_1,\vec v_2,\vec v_3,\vec v_4\}$ of $\Bbb R^4$. Once we have done this, we can define $T:\Bbb R^4\to\Bbb R^3$ by $$ T(\vec v_k) = \begin{cases} \vec 0 & k=1,2 \\ \vec u & k=3 \\ \vec v & k=4 \end{cases} $$ where $\vec u,\vec v\in\Bbb R^3$ are arbitrary nonzero vectors. Of course by "define $T$" we mean to define $T$ on the basis $\{\vec v_1,\vec v_2,\vec v_3,\vec v_4\}$ and extend linearly.
One of the advantages to using this strategy is we avoid matrices and get practice in working with linear maps.
Is this enough to get you started or is this confusing?