Proving that determinant is zero

Question

Let $A,B \in M_3(\mathbb{C})$ such that $(AB)^2 = A^2B^2$ and $(BA)^2 = B^2A^2$. Prove that $\det(AB-BA) = 0$.

Answer 1

Hints. Let $C=AB-BA$. The given conditions imply that $ACB=0$ and $BCA=0$. We don't need both of them. One --- say, $ACB=0$ --- is enough:

• If at least one of $A$ or $B$ has rank $\le1$, argue that $\operatorname{rank}(C)=\operatorname{rank}(AB-BA)\le2$.
• If $\operatorname{rank}(A),\operatorname{rank}(B)\ge2$, then $\operatorname{rank}(CB)\ge2$ when $C$ is non-singular. Now consider Sylvester's rank inequality $ \operatorname{rank}(A)+\operatorname{rank}(CB)-3\le\operatorname{rank}(ACB)$.