How can I prove tr$\left(\sqrt{\sqrt A B \sqrt A}\right)\leq 1$, where both $A$ and $B$ are positive semidefinite,and $\text{tr}(A)=\text{tr}(B)=1$.
The background of $\text{tr}\left(\sqrt{\sqrt A B \sqrt A}\right)$ is a concept in quantum information called the fidelity of quantum states. For two quantum state $\rho$ and $\sigma$,where $\text{tr}(\rho)=\text{tr}(\sigma)=1,\rho ,\sigma\succeq 0$,$F(\rho,\sigma)=\text{tr}\left(\sqrt{\sqrt \rho \sigma \sqrt \rho}\right)$ is defined as their fidelity,and is used to denote their distance in the Hilbert space.See Quantum computation and quantum information Chapter 9 for more details.
Now back to the problem. It is not hard under some special cases,like $AB=BA$ so they can have the same set of eigenvector. What about some general cases?
Also these two problems are relevant: Why is $\text{tr}(\sqrt{\sqrt A B \sqrt A})=\text{tr}(\sqrt{A B }) $ for positive semidefinite matrices $A,B$ and Prove that if A and B are positive semidefinite, then 0≤tr(AB)≤tr(A)tr(B).
But I think there are some flaws in the answer under the post Why is $\text{tr}(\sqrt{\sqrt A B \sqrt A})=\text{tr}(\sqrt{A B }) $ for positive semidefinite matrices $A,B$ because $\text{det}(A)$ can be 0, so $\sqrt{(A)}^{-1}$ may not exist.
Best Answer
For a positive semidefinite matrix $C$ the absolute value $|C|$ is defined as $|C|=(C^*C)^{1/2}.$ Moreover by the polar decomposition there is a unitary matrix $U$ such that $C=U|C|$ and consequently $U^*C=|C|.$ In our case for $C=B^{1/2}A^{1/2} $ we have $$(A^{1/2}BA^{1/2})^{1/2}=|B^{1/2}A^{1/2}|$$ Thus $$U^*B^{1/2}A^{1/2}=|B^{1/2}A^{1/2}|$$ for a unitary matrix $U.$ Let $\{e_n\}_{n=1}^d$ be an orthonormal basis and $f_n=Ue_n.$ Then $\{f_n\}_{n=1}^d$ is an orthonormal basis and
$$ \displaylines{{\rm tr}\, |B^{1/2}A^{1/2}|={\rm tr}\, [U^*B^{1/2}A^{1/2}]=\sum_{n=1}^d \langle U^*B^{1/2}A^{1/2}e_n,e_n\rangle \\ =\sum_{n=1}^d \langle A^{1/2}e_n,B^{1/2}Ue_n\rangle = \sum_{n=1}^d \langle A^{1/2}e_n,B^{1/2}f_n\rangle}$$ Thus by the Cauchy-Schwarz inequality applied twice we get $$\displaylines{{\rm tr}\, |B^{1/2}A^{1/2}|\le \sum_{n=1}^d \| A^{1/2}e_n\|\|B^{1/2}f_n\| \\ \le\left (\sum_{n=1}^d\| A^{1/2}e_n\|^2\right )^{1/2} \left (\sum_{n=1}^d\| B^{1/2}f_n\|^2\right )^{1/2} \\ =({\rm tr}\,A)^{1/2}({\rm tr}\,B)^{1/2}= 1}$$