Let $A\in\mathbb{R}^{n\times n}$ denote some symmetric, and $B\in\mathbb{R}^{n\times n}$ some
positive-definite matrix. The generalized eigenvalue problem, $[A, B]$,
corresponds to a scalar-vector pair, $(\lambda, u)$, satisfying
$$Au=\lambda Bu.$$
What is the property of generalized eigenvectors $u$, e.g., are they mutually orthogonal? Are they somehow related to eigenvectors of $A$ (or $B$ )?
Best Answer
If $B$ is invertible, then you can rewrite this equation as $B^{-1}Au=\lambda u$, so you get an ordinary eigenvector equation, and thus you get all the properties of normal eigenvectors.
Moreover, if $A$ is invertible, you can rewite the equation as $\lambda^{-1} u = A^{-1}Bu$ to again get an ordinary eigenvalue problem (however note that in this case, the vectors from $B$'s null space cannot be generalized eigenvalues, despite being eigenvalues of $A^{-1}B$, because $0$ has no inverse).
The interesting case of course is is when $B$ is not invertible. Obviously if the intersection of the null spaces of $A$ and $B$ doesn't vanish, every vector from that intersection is a generalized eigenvector to an arbitrary generalized eigenvalue (because then the equation reduces to $0=0$ irrespective of $\lambda$). Moreover, if $v_0$ is in the intersection of both null spaces, and $v$ is a generalised eigenvector with generalized eigenvalue $\lambda$, then $v+v_0$ is, too.
If $v$ is in $A$'s null space, but not in $B$'s, then it is a generalized eigenvector to the generalized eigenvalue $0$. If $v$ is in $B$'s null space but not in $A$'s, then it cannot be a generalized eigenvalue.
I guess for vectors $v\perp \operatorname{Ker}(B)$ the problem can again be reduced to an ordinary eigenvalue problem, but I don't currently see how.