I have a question about the simultaneous iteration. I am currently working for an exam and I do not understand this step (taken from Numerical Linear Algebra from Trefethen/Bau):
For the power iteration it holds, that for an arbitrary starting vector $v^{(0)}$ with $||v^{(0)}|| = 1$ that $A^kv \rightarrow q_j$ for $k \rightarrow \infty$, where $q_j$ is the eigenvector corresponding to the maximum eigenvalue (in absolute value).
So far, so good.
Now the book states the following: For a set of n linearly independent vectors $v_1^{(0)}, …, v_n^{(0)}$, the space $span\{A^kv_1^{(0)}, …, A^kv_n^{(0)}\}$ converges to the space $span\{q_1, …, q_n\}$, i.e. the space spanned by the eigenvectors. And this is something I do not understand. Above it is stated, that for almost any arbitrary starting vector, $v$ would converge to the eigenvector corresponding to the largest eigenvalue. How can than the space spanned by several vectors span the same space as all eigenvectors? I would assume that the all will converge sooner or later to the same vector, thus only spanning a one-dimensional subspace.
Where is my mistake?
Best Answer
The problem is that, for suitable matrices and generic $n$-sets of vectors, $\text{Span} \{A^k v_1, \cdots, A^k v_n\}$ is a $n$-dimension vector space. And a sequence of $n$-dimensional vector spaces won't converge in any reasonable sense to a $1$-dimensional vector space, for $n>1$. In the limit, the vectors $A^k v_1$, ... , $A^k v_n$ each converge (at least in the projective space) to a single vector. But, for as long as you iterate the matrix $A$, these vectors won't be exactly the same, and this small difference is what makes their spanned linear space large. For instance, if we take:
$$A := \left( \matrix{1 & 0 & 0 \\ 0 & 1/2 & 0 \\ 0 & 0 & 1/4}\right),$$
$$v_1 := (1,1,1), \ v_2 := (1,2,3),$$
then:
$$A^k v_1 := (1,2^{-k}, 4^{-k}), \ A^k v_2 := (1, 2 \cdot2^{-k}, 3 \cdot 4^{-k}),$$
and you can check that those two vectors still generate a two-dimensional vector space. Moreover, as $k$ goes to $+ \infty$, this two-dimensional vector space will converge to the space $\{(x,y,0): \ x,y \in \mathbb{R}\}$, which is the vector subspace spanned by the eigenvectors $(1,0,0)$ and $(0,1,0)$ corresponding to the two eigenvalues of maximal modulus.