I would appreciate if someone could share some intuition as to the geometric meaning of differentiability condition of functions defined on $\mathbb{R}^n$.
Such a function say $f:\mathbb{R}^n\rightarrow \mathbb{R}$ is differentiable if there is a matrix (vector in this case) $A$ such that
$$\lim\limits_{\mathbf{h} \to 0} \frac{|f(\mathbf{x}+\mathbf{h})-f(\mathbf{x})-A\cdot \mathbf{h}|}{|\mathbf{h}|}=0.$$
$A$ is the vector of partial derivatives of $f$, but the above condition is stronger than the mere existence. I would like to know the geometric nature of what more is required.
Specifically, with respect to the formula for computing directional derivatives,
$$D_{\mathbf{v}} f=A\cdot \mathbf{v}$$
Rudin gives a nice example of a non differentiable function with all directional derivatives exist but the above equation is false.
So to be more precise I ask
1) Is the existence of all directional derivatives and the equation
$$D_{\mathbf{v}} f=A\cdot \mathbf{v}$$ equivalent to differentiablity.
2)For which $X$ is "all directional derivatives exist" +X, equivalent to differentiablilty.
3) Does the existence of directional derivatives in $n$ independent directions imply all directional derivatives exist.
Best Answer
You can think of the derivative (matrix) as providing a linear approximation to the function around a given point. The question then arises: can the function be well-approximated by a linear map around that point? (Recall from linear algebra that linear functions can be expressed in terms of matrices.)
If so, then in neighborhoods close to the point $\mathbf{x}$, the difference between the function and its linear approximations should be incredibly small $-$ in particular, it should be even smaller than the difference $|\mathbf{h}|$ between the new and old inputs. Put another way, the approximation should get "infinitely" good, relative to the change in input, as the change in input goes to $0$ (in magnitude). In symbols: $$\lim\limits_{\mathbf{h} \to 0}\frac{\left|f(\mathbf{x}+\mathbf{h})-\Big(f(\mathbf{x})+A\big((\mathbf{x}+\mathbf{h})-\mathbf{x}\big)\Big)\right|}{|\mathbf{h}|}=\lim\limits_{\mathbf{h} \to 0} \frac{|f(\mathbf{x}+\mathbf{h})-f(\mathbf{x})-A\cdot \mathbf{h}|}{|\mathbf{h}|}=0.$$
In many dimensions, there are situations in which direction derivatives can exist, but the function does not admit a nice "tangent plane" at the (pathological) point in question. In one dimension, the condition reduces to $$\begin{align}\lim\limits_{\mathbf{h} \to 0} \frac{|f(\mathbf{x}+\mathbf{h})-f(\mathbf{x})-A\cdot \mathbf{h}|}{|\mathbf{h}|}&=\lim_{h\to 0}\frac{|f(x+h)-f(x)-f'(x)h|}{|h|}\\&=\lim_{h\to 0}\left(\frac{f(x+h)-f(x)}{h}-f'(x)\right)=0\end{align}$$ or $$\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}=f'(x)$$ (I have dropped magnitude/absolute value signs, which is valid here but not important anyway.) So in one dimension, a derivative exists exactly when the function is differentiable. In higher dimensions, this is not the case; however, if all partial derivatives exist and are continuous in some neighborhood of the point, then we can say that the function is indeed differentiable.
As to your other three questions, it looks like @davin has addressed them nicely.