Given two real symmetric matrices $A$ and $B$, are the eigenvectors of the generalized eigenvalue problem $Ax = \lambda B x$, $x^T B x = 1$ orthogonal?
Put another way, if we denote by $P$ the matrix with the eigenvectors as columns, and $D$ the diagonal matrix that contains the eigenvalues, then $A = B P D P^{-1}$. Does $A$ and $B$ symmetric implies $P^{-1} = P^T$?
What happens if only one matrix of $A$ and $B$ is symmetric but not the other?
What happens if $A$, or $B$, or both are positive-definite as well?
Best Answer
Example: consider $2\times 2$ matrices $$ A=\begin{bmatrix}1&\epsilon\\\epsilon&-1\end{bmatrix}, B=\begin{bmatrix}1&\epsilon\\\epsilon&1\end{bmatrix} $$ where $0<\lvert\epsilon\rvert<1$. Then $$ B^{-1}A=\frac1{1-\epsilon^2}\begin{bmatrix} 1&-\epsilon\\-\epsilon&1 \end{bmatrix}\begin{bmatrix}1&\epsilon\\\epsilon&-1\end{bmatrix}=\frac1{1-\epsilon^2} \begin{bmatrix}1-\epsilon^2&2\epsilon\\0&-1-\epsilon^2\end{bmatrix} $$ which does not have an orthogonal eigenbasis.