Let $A$ and $B$ be $n \times n$ complex matrices and
let $[A,B] = AB – BA$.
Question: If $A , B$ and $[A,B]$ are all nilpotent matrices,
is it necessarily true that $\operatorname{trace}(AB) = 0$?
If,in fact, $[A,B] = 0$, then we can take $A$ and $B$ to be strictly upper triangular matrices so that the answer would be yes in this very special case.
Best Answer
The answer is no.
Take $A=\left(\begin{array}{cccc} 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 \end{array}\right)$, $B=XAX^{-1} = \left(\begin{array}{cccc} -1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ -1 & 1 & -1 & 1\end{array}\right)$, where we chose $X=\left(\begin{array}{cccc} 1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & 0 & 1 & 1 \end{array}\right)$.
Then $[A,B]= \left(\begin{array}{cccc} 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ -1 & 1 & -1 & 1 \\ 0 & 1 & -1 & 1 \end{array}\right)$ is nilpotent, but we have $AB=\left(\begin{array}{cccc} 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ -1 & 1 & -1 & 1 \\ 0 & 0 & 0 & 0 \end{array}\right)$ and $\operatorname{trace}(AB)=-1\not=0$.