Question: $A$ and $B$ are positive self-adjoint matrix such that $AB$ is self-adjoint then prove that $AB$ is positive.
My first try: Here I can see that the same question was there but it has a little amount of typo. What I have to prove is that $(x, ABx)>0$ for all $x>0$.
Now, $(x,ABx)=(Ax,Bx)=(BAx,x)$ now we also have that $AB=BA$ because $AB=(AB)^*=B^*A^*=BA$ because $A,B,AB$ are all self-adjoint and we can also see that $(BAx,Ax)>0$ but I can't get rid of $A$ in the 2nd co-ordinate.
My second try: Then I tried using this result that "if $A$ and $B$ are positive self-adjoint matrix then there exist a basis $x_1,\cdots, x_n$ of $X$ satisfies an eqn of the form $$Ax_j=\lambda_j Bx_j.$$
where $\lambda_j$ is real."
Now $A,B$ are positive $\Rightarrow \lambda_j>0$ using Generalised Raleigh quotient then $$(x_j,ABx_j)=(Ax_j,Bx_j)=(\lambda_j Bx_j,Bx_j)=\lambda_j||Bx_j||^2>0.$$
Hence $AB$ is positive.
Here I am confused that what is the place where I used $AB$ is self-adjoint! Please help. You can also help me with different proofs.
Best Answer
Let us start with the positive operators $A$, $B$ as in the OP. Then $AB$ self-adjoint implies $$ AB = (AB)^*=B^*A^*=BA\ , $$ so that $AB$ are commuting. Using continuous functional calculus w.r.t. the function $f(x)=\sqrt x$ defined on $[0,\infty)$ (and this domain contains the spectrum of $A$, and the spectrum of $B$) we obtain square roots $S=f(A)$, $T=f(B)$, i.e. $A=S^2$, and $T=B^2$.
Explicitly, for $A$ only, let $p_n$ be a sequence of polynomials such that $p_n\to f$ on a compact interval (for instance $[0,\|A\|]$) contaning the spectrum of $A$. Such a sequence is insured by Stone-Weierstraß. Then $$p_n(A)\to f(A)=:S\ .$$ Here, $f(A)$ is defined as $\lim p_n(A)$, the limit exists, Cauchy sequence. (Functional calculus of bounded operators shows this does not depend on $(p_n)$, but we do not need this.) We denote by $S$ this value.
Then $$ \begin{aligned} S^2 &=(f(A))^2=(\lim p_n(A))^2=\lim (p_n(A))^2 =\lim p_n^2(A) \\ &=(\lim p_n^2)(A) =(\lim p_n)^2(A) \\ &=f^2(A)=\operatorname{id}(A)=A\ . \\[3mm] SB &=(\lim p_n(A))B=\lim p_n(A)B\\ &=\lim B p_n(A)=B(\lim p_n(A))=BS\ . \\[3mm] (S^*x, y) &=(x,Sy)=(x,(\lim p_n(A))y)=(x,\lim p_n(A)y)=\lim (x,p_n(A)y)\\ &=\lim (p_n(A)x,y)=(\lim p_n(A)x,y)=((\lim p_n(A))x,y)\\ &=(Sx,y)\ ,\qquad\text{ for all $x,y$ in the given Hilbert space.} \end{aligned} $$ (We have used $AB=BA$. These properties are basic properties of the functional calculus. Starting from $AB=BA$ we get to $f(A)B=Bf(A)$, so $SB=BS$. Similarly, starting with $SB=BS$ we obtain $Sf(B)=f(B)S$. So $ST=TS$ the two operators $S,T$ also commute.)
Let now $x$ be $\ne 0$. We have in a row: $$ (ABx,x) =(SSTTx,x)=(TSSTx,x)=(STx,STx)=\|STx\|^2>0\ . $$ (We have $(STx,STx)=\|STx\|^2\ge 0$, and in case of an equality, from $(STx,STx)=0$ we get first $Tx=0$, since $S>0$, then from $(Tx,Tx)=0$ also $x=0$, since $T>0$. Contradiction, since we started with an $x\ne 0$.)
Alternatively, we could have intoduced only $T$, and have the same argument with $SS$ replaced by $A$, e.g. $(ABx,x)=(ATTx,x)=(TATx,x)=(ATx,Tx)>0$ since $Tx\ne 0$ since $(Tx,Tx)=(TTx,x)=(Bx,x)>0$.
Note: In case of finitely dimensional spaces, things are simple. The two commuting self-adjoint operators can be diagonalized simultaneously w.r.t. some ortonormal basis, and if $A=\operatorname{diag}(a_1,\dots,a_n)>0$, then $S=\sqrt A:= \operatorname{diag}(\sqrt{a_1},\dots,\sqrt{a_n})>0$ is the explicit square root of $A$ (which is positive), it is diagonal w.r.t. the same basis, et caetera.
Note: See also