Given a real $m \times n$ matrix $A$ and $b \in \mathbb{R}^m$, let $f(x)=\|Ax-b\|^2$ for any $x \in \mathbb{R}^n$.
Find Frechet derivative of $f(x)=\|Ax-b\|^2$ at any $x^*$?
Actually I am wondering how to use the following to find the Frechet derivative, i.e., $J$:
$$\lim_{h \rightarrow 0} \frac{|f(x+h)-f(x)-Jh|}{\|h\|} =0$$
Another question is what the difference between what we would get from Frechet derivative and the gradient $\nabla f(x)=2A^T(Ax-b)$?
Please explain your reasons in detail, especially, what is the difference between gradient of $f$ and Frechet derivative. Also, explain when they might be identical.
Best Answer
If $\mathrm{H}$ is a Hilbert space, then every continuous linear function $u:\mathrm{H} \to \mathbf{R}$ can be represented by means of scalar product with respect to a unique vector, here denoted as $x_u:$
$$u(y) = (y \mid x_u).$$
Hence, if $f:\mathrm{H} \to \mathbf{R}$ is a differentiable function, then its derivative $u = f'(a)$ at $a$ is a continuous linear function. The vector $x_u$ is denoted $\nabla f(a)$ in this case. And we have the fundamental relation:
$$f'(a) \cdot h = (\nabla f(a) \mid h).$$
In regards to your particular $f,$ we can write $f(x) = (Ax -b \mid Ax - b)$ and by the products and chain rules, $$f'(x) \cdot h = (Ax - b \mid Ah) + (Ah \mid Ax - b) = 2(Ah \mid Ax - b).$$
If $\mathrm{H} = \mathbf{R}^d,$ and we are dealing with the standard Euclidean inner product, we can write further $f'(x) \cdot h = (2A^\intercal (Ax - b) \mid h),$ this signifies $\nabla f(x) = 2A^\intercal (Ax - b).$ Q.E.D.