From Humphreys' Introduction to Lie Algebras and Representation Theory:
We conclude this subsection by mentioning several other subalgebras of $gl(n,F)$ which play an important subsidiary role for us. Let $t(n,F)$ be the set of upper triangular matrices $(a_{ij})$, $a_{ij}=0$ if $i>j$. Let $n(n,F)$ be the strictly upper triangular matrices ($a_{ij}=0$ if $i\geq j$). Finally, let $o(n,F)$ be the set of all diagonal matrices. It is trivial to check that each of these is closed under the bracket. Notice also that $t(n,F)=o(n,F)+n(n,F)$ (vector space direct sum), with $[o(n,F),n(n,F)]=n(n,F)$, hence $[t(n,F),t(n,F)]=n(n,F)$. (If $H,K$ are subalgebras of $L$, $[H K]$ denotes the subspace of $L$ spanned by commutators $[xy]$, $x\in H, y\in K$.)
I am confused about why $[t(n,F),t(n,F)]=n(n,F)$. If we have two upper triangular matrices $X,Y$, then $XY$ is also upper triangular, and so $XY-YX$ is upper triangular, so it should stay in $t(n,F)$. But I don't see why $XY-YX$ should be strictly upper triangular.
Best Answer
Two things make it easy to see that the commutator of upper triangular matrices is a strictly(1) upper triangular matrix:
The multiplication of diagonal matrices amounts to multiplication of corresponding diagonal elements, and since/if multiplication in $F$ is commutative, that entails the commutativity of multiplication in $o(n,F)$.
To see that the product of a strictly upper triangular matrix with an upper triangular matrix is strictly upper triangular (I assume it already known that the product is upper triangular), consider the generic diagonal element of the product,
$$c_{ii} = \sum_{k=1}^n a_{ik} b_{ki} = \sum_{k=1}^i a_{ik}b_{ki} = \sum_{k=i}^i a_{ik}b_{ki} = a_{ii}b_{ii};$$
the first reduction of the range $k$ traverses is because $B$ is upper triangular, hence $b_{ki} = 0$ for $k > i$, the second because $A$ is upper triangular, hence $a_{ik} = 0$ for $k < i$, and that leaves only $a_{ii}b_{ii}$. One of the factors is $0$ since one of $A$ and $B$ is strictly upper triangular by assumption.
Thus, when we have two upper triangular matrices $X$ and $Y$, write them as $X = D_X + S_X,\; Y = D_Y + S_Y$ with diagonal $D_\ast$ and strictly upper triangular $S_\ast$, to find
$$\begin{align} [X,Y] &= [D_X+S_X,Y] = [D_X,Y] + [S_X,Y]\\ &= [D_X, D_Y + S_Y] + [S_X,D_Y + S_Y]\\ &= [D_X,D_Y] + [D_X,S_Y] + [S_X,D_Y] + [S_X,S_Y]\\ &= [D_X,S_Y] + [S_X,D_Y] + [S_X,S_Y] \end{align}$$
and all products that remain involve at least one strictly upper triangular matrix, hence are strictly upper triangular.
(1) If the ring of scalars is commutative.