I ma trying to prove that if $A$ is an $n\times n$ matrix, then $f:\mathbb{R}^n\to\mathbb{R},\ f(\vec{x})=A\vec{x}\cdot\vec{x}=\vec{x}^T A\vec{x}$ is differentiable and $Df(\vec{a})\vec{h}=A\vec{a}\cdot\vec{h}+A\vec{h}\cdot\vec{a}.$
(NOTE: computation edited according to Snaw's comment below)
Now, since a function $f:\mathbb{R}^n\to\mathbb{R}^m$ is differentiable if $$
\lim\limits_{\vec{h}\to\vec{0}}\frac{\vec{f}(\vec{x}+\vec{h})-\vec{f}(\vec{x})-Df(\vec{a})\vec{h}}{||\vec{h}||}=\vec{0}
$$
we have that $f$ is differentiable at $\vec{a}\in\mathbb{R}^n$ if $$\lim\limits_{\vec{h}\to\vec{0}}\frac{(\vec{a}+\vec{h})^TA(\vec{a}+\vec{h})-\vec{a}^TA\vec{a}-A\vec{a}\cdot\vec{h}-A\vec{h}\cdot\vec{a}}{||\vec{h}||}\\
=\lim\limits_{\vec{h}\to\vec{0}}\frac{\vec{a}^TA\vec{h}+\vec{h}^TA\vec{a}+\vec{h}^TA\vec{h}-\vec{h}^TA\vec{a}-\vec{a}^TA\vec{h}}{||\vec{h}||}=\lim\limits_{\vec{h}\to\vec{0}}\frac{\vec{h}^TA\vec{h}}{||\vec{h}||}
$$
but at this point I am not sure if I can claim that, as $\vec{h}\to\vec{0}$, the limit is $\vec{0}$, since it seems to me that I would encounter an indeterminate form of the type $\vec{0}/0$.
For example, if $A\in\mathbb{M}_{2\times 2}$ we would have $$\lim\limits_{(h_1,h_2)\to (0,0)}\frac{h_1^2 A_{11}+(A_{12}+A_{21})h_1h_2+h_2^2 A_{22}}{\sqrt{h_1^2+h_2^2}}.$$
How could I justify this claim?
Thanks
Best Answer
The limit $$\lim_{h\to 0} \frac{h^T A h}{||h||}$$ intuitively equals $0$ because in the denominator we roughly have $h$ as a linear term and in the numerator it exists as a quadratic term. For instance, for $A\in \Bbb M _{2\times 2}$ we have as you've noted $$\lim\limits_{(h_1,h_2)\to (0,0)}\frac{h_1^2 A_{11}+(A_{12}+A_{21})h_1h_2+h_2^2 A_{22}}{\sqrt{h_1^2+h_2^2}}$$ and while this is of the indeterminate form $\frac{0}{0}$, the numerator tends to $0$ faster because of the quadratic terms. All of this of course requires justification and is nothing more than intuition so far. To prove this rigorously use the squeeze theorem and the triangle inequality: $$0\leq\left|\frac{h_1^2 A_{11}+(A_{12}+A_{21})h_1h_2+h_2^2 A_{22}}{\sqrt{h_1^2+h_2^2}}\right|\leq \frac{h_1^2 | A_{11}|}{\sqrt{h_1^2+h_2^2}} + \frac{|h_1h_2| | A_{12} + A_{21}|}{\sqrt{h_1^2+h_2^2}} + \frac{h_2^2 | A_{22}|}{\sqrt{h_1^2+h_2^2}}$$ For the first term we get $$0\leq \frac{h_1^2 | A_{11}|}{\sqrt{h_1^2+h_2^2}} \leq \frac{h_1^2 | A_{11}|}{\sqrt{h_1^2}} = \frac{h_1^2 | A_{11}|}{|h_1|} = |h_1|\cdot | A_{11}|\to 0$$ and the same calculation yields that the other two terms tend to $0$ as well.
For the general case of $A\in\Bbb M _{n\times n}$ a similar calculation works: in the denominator we get $\sqrt{h_1^2+\ldots+h_n^2}$ which is greater than $\sqrt{h_i^2}=|h_i|$ for each $1\leq i\leq n$, and by using the squeeze theorem and the triangle inequality again we get that each term in the numerator contains the product $|h_i|\cdot|h_j|$ for some $i,j$ (possibly $i=j$), so that over all we are left with the sum of a finite number of terms all tending to $0$.