In page 4 of this paper, it has been mentioned that the collection
$$B=\left\{\begin{pmatrix} a & b \\0 & c\end{pmatrix}~:~a,c \in (\mathbb{Z}/4 \mathbb{Z})^{\times},~ b \equiv 0~(\text{mod}~2) \right\} \leq \text{GL}(2, \mathbb{Z}/4 \mathbb{Z})$$ is abelian.
But my calculation is not saying this. For, take two upper triangular matrices $M,N \in B$ with $$M=\begin{pmatrix} a_1 & b_1 \\ 0 & c_1 \end{pmatrix},~N=\begin{pmatrix} a_2 & b_2 \\ 0 & c_2 \end{pmatrix}.$$ Then we have $$MN=\begin{pmatrix} a_1a_2 & a_1b_2+b_1c_2 \\ 0 & c_1c_2 \end{pmatrix},~NM=\begin{pmatrix}a_2a_1 & a_2b_1+b_2c_1 \\ 0 & c_2c_1 \end{pmatrix}.$$
In order $B$ to be abelian, we need $a_1b_2+b_1c_2=a_2b_1+b_2c_1.$
But this is not true. We can say only $a_1b_2+b_1c_2 \equiv a_2b_1+b_2c_1$ (mod $2$).
So can we say $MN=NM$ ?
In general, when is the set $S$ of all upper triangular matrices in $\text{GL}(2, \mathbb{Z}/p \mathbb{Z})$ is abelian for $p$=prime ?
Ofcourse when $S$ contains only diagonal matrices then it is abelian.
What are the other possibilities ?
Best Answer
Let $R$ be a commutative ring, and let $B$ be an abelian group fitting in the chain of inclusions $$\{\pmatrix{a&0\\0&c} : a,c \in R^\times\}=:T \subseteq B \subseteq U:= \{\pmatrix{a&b\\0&c} : a,c \in R^\times, b \in R\}.$$
Conjugation with $\pmatrix{a&0\\0&c}\in T$ maps $\pmatrix{r&b\\0&s}$ to $\pmatrix{r&ac^{-1}b\\0&s}$, so for $B$ to be abelian we need $R^\times \cdot b =b$ for all $b$ such that some $\pmatrix{r&b\\0&s} \in B$.
Hence, for $T \subsetneq B$, all elements $u-1$ for $u \in R^\times$ have to be zero-divisors in $R$. (Indeed, as soon a some element $\pmatrix{1&b\\0&1} \in B$, one similarly shows that $b \cdot (a-c) = 0$ for all units $a,c$, i.e. if $b \neq 0$, each pairwise difference of units must be a zero-divisor in $R$.)
And, as soon as there exists $1 \neq a \in R^\times$, then the full set $U$ of upper triangulars cannot be abelian (take $b=1$ in the above), in fact all elements $b$ appearing in some upper right corner have to be zero-divisors as well.
In particular, let $n \ge 2$. If $R$ contains $\mathbb Z/n$ or $\mathbb Z$, a strict inclusion $T \subsetneq B$ is impossible unless $n$ is even.
If, on the other hand, $n=2m$, and $R=\mathbb Z/n$, then one can check directly that $B=\{\pmatrix{r&b\\0&s}: r,s \in R^\times, b \in m\cdot(\mathbb Z/n) \}$ is indeed abelian. Yours is the case $m=2$.
For $R= \mathbb Z/n$, the full set $U$ is abelian if and only if $n=2$.