So far:
The commutativity of the matrices gives me simultaneous diagonalization, the fact that they're symmetric tells me that they're simultaneously diagonizable by a orthogonal matrix. The fact that they're positive definite means that their eigenvalues are all positive, so that means that the eigenvalues of AB are also positive definite.
Am I going in the right direction? What am I missing here? How do I formally prove this?
Thanks for your time.
Best Answer
First, check that $AB$ also is symmetric. In the condition that $x^T AB x >0$ for all non-zero $x$, it is enough to check that for, for instance, a basis consisting of eigenvectors for $B$ (or for $A$). So let $x_1, \dots, x_n$ be a basis consisting of eigenvectors for $B$ with corresponding eigenvalues $\mu_1, \dots, \mu_n$. Then calculate $$ x_i^T AB x_i = x_i^T A \mu_i x_i = \mu_i x_i^T A x_i > 0 $$ for all $i$, and that is enough.
EDIT
What is given above is not quite enough, we have not really used the condition that $AB = BA$. From that condition follows that $A$ and $B$ have the same eigensystems. This is easy to see: Let $x_1, \dots, x_n$ be eigenvectors of $A$ with distinct eigenvalues $\lambda_1, \dots, \lambda_n$. Calculate $$ BAx_i = \lambda_i Bx_i = AB x_i $$ so we can see that the $Bx_i$ also are eigenvalues of $A$ with the same eigenvalue as $x_i$! Since we assumed the eigenvalues are distinct, we can conclude that $Bx_i = \mu_i x_i$: $x_i$ is also an eigenvector of $B$ with a different eigenvalue $\mu_i$. So $A$ and $B$ have the same eigensystems. If the original matrices have eigenvalues with multiplicity larger than one, we approximate them by matrices with distinct eigenvalues, and get the same result by taking limit. Now we can repeat the calculation above: $$ x_i^T AB x_i = \mu_i x_i^T x_i \lambda_i > 0 $$