From a Masters Qual. Practice Exam:
Let $G \leq GL(3, \mathbb{F}_3)$, be the group of invertible $3 × 3$
upper triangular matrices over the field with $3$ elements (i.e.
entries below the diagonal are zero). Find the order of $G$ and show
that $G$ is not a simple group i.e. show that $G$ has a proper
non-trivial normal subgroup.
If we didn't have "invertible" condition, this would give us $3^6$ matrices, but I'm not sure how to quickly weed out which are and which aren't invertible.
Intuitively, I think the normal subgroup should be the diagonal matrices (I could be wrong), but I'm not sure how to prove that without some really gross matrix multiplication.
Best Answer
In order that an upper triangular matrix
$$\begin{pmatrix} a & b & c \\ 0 & d& e \\ 0 &0 & f\end{pmatrix}$$
be invertible, it is necessarily and sufficient that the determinant $adf$ be nonzero. This means that $b,c,e$ can be anything you want, while $a, d, f$ cannot be zero. This means there are
$$3 \cdot 3 \cdot 3 \cdot 2 \cdot 2 \cdot 2 = 216 $$
such matrices.
The subgroup $D$ of diagonal invertible matrices is as far from being normal in $G$ as you can get (if $x \in G$, but not in $D$, then $xDx^{-1} \neq D$).
However, if you multiply two upper triangular matrices $x, y \in G$, notice that the entries on the diagonal of $xy$ are obtained by multiplying the corresponding entries on the diagonal of $x$ and $y$. This implies that
$$N = \{ \begin{pmatrix} 1 & b & c \\ 0 & 1& e \\ 0 &0 & 1\end{pmatrix} : b, c, e \in \mathbb{F}_3 \}$$ is a normal subgroup of $G$.
Actually, $G$ is the semidirect product of $N$ and $D$.