I've been given the following question:
Consider a function $f$ on $(0,1)$ such that $||f|| < \infty$, where the norm $||f|| = \sqrt{(f,f)}$ is induced by the following inner product: $(g_1, g_2) = \int_0^1 g_1(x)g_2(x) dx $
Let $p_1(x) = c_0 + c_1x$ denote the Least Square Approx to f.
Determine the system of linear equations for $c_0, c_1$
I've been given the answer:
$M(c_0, c_1)^T = b$,
with $M = $\begin{bmatrix}
1&1/2\\
1/2&1/3
\end{bmatrix}
and $b = $\begin{bmatrix}
(f,1)\\
(f,x)
\end{bmatrix}
But, I don't understand how to get there.
Thanks in advance for any help.
Best Answer
Here is the way I like to think of the Least Square Method. Let $V$ be a Hilbert space, and let $W$ be a finite-dimensional space, of dimension less than or equal to that of $W$ (in applications usually much smaller). Let $A$ be an injective linear map from $W$ into $V$. Thus $W$ can be identified using $A$ with its image $A(W) \subseteq V$, and can thus be thought of as a subspace of $V$.
Let $b \in V$. Note that $b$ need not be in $A(W)$, so that the equation $Ay=b$ may not have any solution $y \in W$. We think of $b-Ay \in V$ as an error vector. It is precisely the norm squared of this vector that we want to minimize (norm with respect to the inner product in $V$).
Geometrically, the norm squared of the error vector is minimal iff the error vector is orthogonal to $A(W)$. Thus our condition is:
$(b-Ay,Aw) = 0$ for all $w \in W$.
Hense $(A^T(b-Ay),w) = 0$ for all $w \in W$. But the inner product restricted to $W$ is non-degenerate, so the previous condition is equivalent to:
$A^T(b-Ay) = 0$ or, $\quad$ $A^TAy = A^Tb$ (these are called the normal equations)
In the case you are interested in, $A = (1 \,\, x)$, $y = (c_0 \,\, c_1)^T$ and $b = (f)$.
Hence $A^TA = \left( \begin{array}{cc} (1,1) & (1,x) \\ (x,1) & (x,x) \end{array} \right)$. And $A^T b = ((f,1)\,\, (f,x))^T$.