First of all, I'd like to make clear that I'm talking about the Gram-Schmidt Conjugation process (in the context of conjugate gradient method for optimization), which instead of making a set of n independent vectors orthonormal, makes a set of n independent vectors conjugates – that is: expressable in the form
(where vector u_i and vector u_j are conjugates, or sometimes, "A-orthogonal").
Understandably, this formula is going to be very similar to Gram-Schmidt Orthogonalization, since the process is the same – but instead of projecting (and subtracting) some vector v onto all the previous orthogonal vectors and subtracting, we're projecting on to the all the previous conjugate vectors and subtracting.
So, the question is what this "projection matrix" is. In the paper/explanation I'm working through to understand Conjugate Gradient, there is two pages dedicated to the explanation of Gram-Schmidt Conjugation Process (pages 31-32 in pdf) and there is a proof of what the "projection" matrix is, but I don't understand it. I recommend you look at it in the paper but here is a screenshot:
I get equation (36), but kind of get lost after that. I get why they're premultiplying, since the whole definition of conjugate is that they abide to that property of u_i(T) * A*u_j = 0, but I really don't get the new subscript "j" which I haven't seen before and the removal of the summation.
If someone could explain it step by step that would be really good,
Thank you!
Best Answer
The goal is to find $\beta_{ij}$ such that $d_{(i)}^TAd_{(j)}=0$.
We transpose equation $(36)$ and premultiply it to $Ad_{(j)}$
In the very next line, the summation seems to vanish, that is because if $k \ne j$, then $d_{(k)}^TAd_{(j)}=0$. Hence only one term is left.
After which, we can solve for $\beta_{ij}$.