So, the exercise is incorrect as stated, as the nice example in the question shows. They probably meant to say that the matrix is invertible in the subalgebra of upper triangular matrices if and only if the diagonal entries are invertible. This is the version given on page 16 in a set of lecture notes by Matthes and Szymański based primarily on the same book. They also give a counterexample to the original statement.
Part 1): Show that the product of two upper triangular matrices is upper triangular.
Let $A, B \in M_n(\mathbb{R})$ be upper triangular matrices (with respect to the standard basis). Then we can express them as
$$\left( \begin{array}{ccc}
a_{11} & a_{12} & \cdots &a_{1n} \\
0 & a_{22} & \cdots & a_{2n} \\
\vdots & 0 & \cdots & \vdots \\
0 & \cdots & 0 & a_{nn}\end{array} \right) \text{ and }\left( \begin{array}{ccc}
b_{11} & b_{12} & \cdots &b_{1n} \\
0 & b_{22} & \cdots & b_{2n} \\
\vdots & 0 & \cdots & \vdots \\
0 & \cdots & 0 & b_{nn}\end{array} \right)$$
Hence their product is clearly
$$\left( \begin{array}{ccc}
a_{11} & a_{12} & \cdots &a_{1n} \\
0 & a_{22} & \cdots & a_{2n} \\
\vdots & 0 & \cdots & \vdots \\
0 & \cdots & 0 & a_{nn}\end{array} \right)\left( \begin{array}{ccc}
b_{11} & b_{12} & \cdots &b_{1n} \\
0 & b_{22} & \cdots & b_{2n} \\
\vdots & 0 & \cdots & \vdots \\
0 & \cdots & 0 & b_{nn}\end{array} \right) = \left( \begin{array}{ccc}
a_{11}b_{11} & a_{11}b_{12}+a_{12}b_{22} & \cdots &\sum_{i=1}^n a_{1i}b_{in} \\
0 & a_{22}b_{22} & \cdots & \sum_{i=2}^n a_{2i}b_{in} \\
\vdots & 0 & \cdots & \vdots \\
0 & \cdots & 0 & a_{nn}b_{nn}\end{array} \right)$$
I think I have given you enough information to describe the general pattern of entry $c_{jk}$ of the product matrix.
Part 2): To do this, show that triangular matrices are closed under linear combinations. In particular, show that (1) the zero matrix is technically upper triangular, (2) that any triangular matrix times a scalar is upper triangular, and (3) that the sum of any two upper triangular matrices is itself upper triangular. These should be very straightforward proofs. See here for a specific case of the dimension argument.
Part 3): If $M(\phi, B)$ is the matrix representation of $\phi$ and it is upper triangular, then this means that $\phi(e_k)$ is the first column and has all zero entries except for the first $k$. Stop and think about what this means in terms of the subspace of $E$ that contains $\phi(e_k)$. I am being intentionally vague because this is an important concept that you should reach on your own, but I can be more explicit and supply a reference need be.
Part 4): By reversible do you mean invertible? If so, just think about rearranging the basis vectors after inverting the map.
Best Answer
Hint: One approach applies the following observation: if $A$ is block upper-triangular with $$ A = \pmatrix{B & C\\0 & D} $$ Then $$ A^TA = \pmatrix{B^TB & B^TC\\C^TB & C^TC + D^TD}\\ AA^T = \pmatrix{BB^T + CC^T & CD^T\\D^TC & DD^T} $$ By taking the trace of each side, show that $$ C^TC + D^TD = DD^T \implies C = 0 $$