Let $A,B$ be two $3 \times 3$ matrices with complex entries such that $$(A-B)^2=O_3$$
Prove that $$\det(AB-BA)=0$$
I tried to prove this with ranks. I denoted $X=A-B$ and thus $X^2=O_3$ which means that $\det X=0$ and $\operatorname{rank}X \leq 2$. Then, I wrote $AB-BA=(X+B)B-B(X+B)=XB-BX$ and finally I used $\operatorname{rank}(M \pm N) \leq \operatorname{rank}M+\operatorname{rank}N$ and Frobenius's inequality in order to get $$\operatorname{rank}(XB-BX) \leq \operatorname{rank}BX+\operatorname{rank}XB \leq \operatorname{rank}X+\operatorname{rank}BXB$$
and if we knew that $\operatorname{rank}BXB=0$, the problem would be solved. However, I don't quite know if the latter is true.
Best Answer
As pointed above by @Lord Shark the Unknown (whose comment struck me, pointing the right way) we have from Sylvester's inequality: $$0=\operatorname{rank}O_3=\operatorname{rank}(X^2) \geq \operatorname{rank}X+\operatorname{rank}X-3 \Rightarrow \operatorname{rank}X \leq 1.$$ Thus, $$\operatorname{rank}(XB-BX) \leq \operatorname{rank}(XB)+\operatorname{rank}(BX) \leq \operatorname{rank}X+\operatorname{rank}X \leq 2$$ and so $\det(AB-BA)=0$.