Let $A, B \in M_3(\mathbb{C})$ so that $$A^2+B^2=2AB.$$
Prove that$$\det(A+B) ^2=8\det(A^2+B^2).$$
My work: Let $A=X+Y$, $B=X-Y$ with $X, Y \in M_3(\mathbb{C})$. The condition rewrites as $-2Y^2=[X,Y]$. Since $Y$ commutes with $Y^2$ it will also commute with $[X, Y] $, so according to Jacobson's lemma $[X, Y] $ is nilpotent. I am not sure if this helps, but using my notations the conclusion is equivalent to $\det(X^2)=\det(X^2+Y^2)$ and we also have $\det Y=0$.
Best Answer
You are almost there. You've obtained that $2Y^2=[Y,X]$ and that $Y$ is nilpotent. Hence \begin{aligned} 0=2Y^3 &=Y[Y,X]\\ &=YYX-YXY\\ &=YYX-([Y,X]+XY)Y\\ &=YYX-(2Y^2+XY)Y\\ &=YYX-XYY. \end{aligned} Thus $Y^2$ commutes with $X$. Hence $X^2$ and $Y^2$ are simultaneously triangulable.
The hypothesis $\det(A+B)^2=8\det(A^2+B^2)$ can be rewritten as $\det(2X)^2 = 8\det(2(X^2+Y^2))$. Pulling out the constants, it is equivalent to $\det(X^2) = \det(X^2+Y^2)$. The result now follows because $X^2$ and $Y^2$ are simultaneously triangulable and $Y^2$ is nilpotent.