Finding the Gradient of a Scalar Field
I understand that you can find the gradient of a scalar field, in an arbitrary number of dimensions like so :
$$grad(f) = \vec{\nabla}f = \left<\frac{\partial f}{\partial x_{1}}, \frac{\partial f}{\partial x_{1}},…,\frac{\partial f}{\partial x_{n}}\right> = \begin{bmatrix}
\frac{\partial f}{\partial x_{1}} \\
\frac{\partial f}{\partial x_{2}} \\
… \\
\frac{\partial f}{\partial x_{n}}
\end{bmatrix}$$
where $f$ is a scalar function, $f: \mathbb{R^n} \to \mathbb{R}$. And as you can see this generalizes consistently to higher-dimensional scalar fields. All of this is straight out of Multivariable Calculus.
Finding the Gradient of a Vector Field
Furthermore finding the gradient of a Vector Field, is given by a Tensor i.e. given $f$ to be a vector function, $f : \mathbb{R^m} \to \mathbb{R^n}$ :
$$grad(\vec{f}) = \displaystyle
\nabla \vec{f} = T =
\begin{bmatrix}
\frac{\partial f_1}{\partial x_1} &
\frac{\partial f_1}{\partial x_2} &
… &
\frac{\partial f_1}{\partial x_m} \\
\frac{\partial f_2}{\partial x_1} &
\frac{\partial f_2}{\partial x_2} &
… &
\frac{\partial f_2}{\partial x_m} \\
… &
… &
… &
… \\
\frac{\partial f_n}{\partial x_1} &
\frac{\partial f_n}{\partial x_2} &
… &
\frac{\partial f_n}{\partial x_m} \\
\end{bmatrix}$$
with $T$ denoting the tensor (a $n\ $x$\ n$ matrix of partial derivatives of $\vec{f}$'s scalar components, i.e. rank-$0$ tensor components, correct me if what I said in these brackets is wrong) which tells us how the vector field changes in any direction.
How to find the Gradient of a Tensor Field?
But how do you find the gradient of a Tensor Field? I understand that to answer this question we may need to generalize the concept of a tensor field a bit further.
If I understand correctly, a scalar is a tensor of rank-$0$, a vector is a tensor of rank-$1$. Is it then fine to generalize scalar fields as tensor fields of rank-$0$, and vector fields as tensor fields of rank-$1$? If so then it means we've been finding the gradient of tensor fields (albeit of rank-0 being scalar fields) in our Multivariable courses all along, we just didn't know it.
By extending the logic behind the leap between taking the gradient of a Scalar Field, to taking the gradient of a Vector Field, is it then correct to say that :
The gradient of a Tensor field of rank-$n$ is a Tensor field of rank-($n+1$) ?
Best Answer
A tensor is just a multilinear, scalar-valued function. If I write $V \multimap W$, for the collection of linear functions from a vector space $V$ to a vector space $W$, then, over the reals, a rank-$n$ tensor is just $V^{\otimes n}\multimap\mathbb R$ where $V^{\otimes n}$ means the $n$-fold tensor product, e.g. $V^{\otimes 2} = V\otimes V$. A tensor field is just a smoothly indexed family of tensors.
For simplicity, I'll just talk about the manifold $\mathbb{R}^n$, but anywhere I explicitly write out $\mathbb{R}^n$ (as opposed to $V$), you could just as well use a submanifold of $\mathbb{R}^n$, e.g. a 1-dimensional curve in $\mathbb{R}^n$. A rank-$k$ tensor field on $\mathbb{R}^n$ is a (suitably smooth) function $\tau : \mathbb{R}^n \to (V^{\otimes k} \multimap \mathbb R)$ where $V$ is itself $\mathbb{R}^n$. Now say we write the directional derivative of $\tau$ in some direction $v \in V$ at $x \in \mathbb{R}^n$ as $D(\tau)(x; v)$. The result itself would be a rank-$k$ tensor, i.e. $D(\tau)(x; v) : V^{\otimes k} \multimap \mathbb R$. So what is the type of $D(\tau)$ itself. Well we know the type of $\tau$ and we know the type of $V$ and we know $D(\tau)(x; v)$ is linear in $v$ and non-linear in $x$. So we have $$D(\tau) : \mathbb{R}^n \to (V \multimap (V^{\otimes k}\multimap\mathbb{R}))$$ but it is easy to show that $(V \multimap (V^{\otimes k}\multimap\mathbb{R})) \cong (V\otimes V^{\otimes k} \multimap \mathbb R) = (V^{\otimes k+1}\multimap \mathbb R)$. Which is to say $D(\tau)$ as a function of $x$ alone — essentially currying $D(\tau)$ — is a rank-$(k+1)$ tensor field.
So to answer your question, you find the gradient of a tensor field by viewing the directional derivative as a linear function of the direction. When you have a basis, as you do for $\mathbb{R}^n$, this linear function can be represented by a vector of partial derivatives (which are directional derivatives along the basis directions). A bit more concretely, $$D(\tau)(x)_i = \frac{\partial\tau}{\partial x_i}(x)$$
And yes, even high school, single variable calculus was doing this for the $n = 1$ case. It's just that $\mathbb{R}^{\otimes k} \cong \mathbb{R}$ for any $k$, and $(\mathbb{R}\multimap\mathbb{R}) \cong \mathbb{R}$.