This question has been bothering me for quite a while and I still haven't found a satisfying answer anywhere on the internet or in any of my books (which may not be that advanced, mind you…). Since I couldn't find a similar question here on MSE, it probably is rather obvious if you have a sufficiently advanced level, which I do not (I'm still undergraduate and my knowledge of tensors is really small!). Out of simplicity, I'll take a specific example for my question. Here it is:
Let $f : \mathbb{R}^2 \rightarrow \mathbb{R}$ ; $(x,y) \rightarrow f(x,y)$ be a continous function with two variables $x$ and $y$.
The "equivalent" of the first derivative known from single-variable functions would be the gradient:
$\nabla f = \left(\begin{array}{cccc}
f_x \\
f_y
\end{array}\right)$
The second derivative is the Hessian-Matrix: $H_f(x,y) = \left(\begin{array}{cccc}
f_{xx} & f_{xy} \\
f_{yx} & f_{yy}
\end{array}\right)$
So what does the third-order derivative of this function (or in general…) look like? I've been able to work out that it is a tensor with order $3$. How does one notate such a geometric object?
Concerning the properties: the gradient is a vector field depicting the "slope" of the function in any point using a vector (i.e.: an orientation and a magnitude). The Hessian-Matrix shows us the rate of change of the gradient (correct use of words?). So by this analogy, the third derivative measures the rate of change of the second derivative. But how?
Feel free to change the tags and edit my answer, if you think it necessary. I'm looking forward to any answers!
Thanks in advance, SDV
Best Answer
I have written a somewhat eccentric course on this, which can be found here:
http://ximera.osu.edu/course/kisonecat/m2o2c2/course/activity/week1/
A short answer to your question, though, is that if $f:\mathbb{R}^n \to \mathbb{R}$, then $D^{k+1} f$ is a locally a $(k+1)$-linear function with the property that
$$ D^{k}f\big|_{p+v_{k+1}}(v_1,v_2,...,v_k) \approx D^{k}f\big|_{p}(v_1,v_2,...,v_k) +D^{k+1}\big|_{p}(v_1,v_2,...,v_k,v_{k+1}) $$
In other words, the $(k+1)$ derivative measures changes in the $k$ derivative.
To write things out in a basis, in Einstein notation, we have
$$D^k f = f_{i_1i_2...i_k} dx^{i_1} \otimes dx^{i_2} \otimes ... \otimes dx^{i_k}$$
where $f_{i_1i_2...i_k}$ is the higher partial derivative of $f$ with respect to $x_{i_1}$ then $x_{i_2}$, etc.
I should note that the multivariable Taylor's theorem becomes especially easy to write down using this formalism:
$$ f(x+h) = f(x)+Df\big|_x (h)+\frac{1}{2!} D^2 f\big|_x (h,h)+\frac{1}{3!} D^3 f\big|_x (h,h,h)+... $$
This may also illuminate the presence of $\frac{1}{k!}$ in Taylor's theorem: it arises from the $k!$ permutations of the arguments.