This is a quick question! I keep seeing gradient vectors explained as such:
"If you imagine standing at a point $(x_0,y_0…)$ in the input space
$f$, the vector $\triangledown f$$(x_0,y_0…)$ tells you which
direction you should travel to increase the value of $f$ most rapidly"
(one example of where I got this explanation of a gradient
vector).
Now the big problem I have with this is, it feels like it does not really make sense in the context of what the math is doing (unless I am misunderstanding gradient vectors). As in, the gradient vector is a combination of all the individual effects ("work" could be a way to say it) each variable in the multivariable-function contributes towards the rate-of-change of the output for that function. So, this means that the gradient vector is not representing "the direction you should travel to increase the output of $f$$(x_0,y_0…)$ most rapidly" because there is no "choice", the gradient vector just models (is a representation of) the intrinsic property of how the output of a function will change, with or without specific input.
[BAD ANALOGY AHEAD]: It would be like someone asking me where they need to go to get to a coffee shop, and I point in a direction of the store. The direction I point is determined by the location of the store and is not different depending on where the person I am talking to is standing.
One other explanation of the gradient vector that often comes as a result of the previous explanation is:
"The gradient vector encapsulates the combined effects of all the partial derivatives, representing the direction of steepest ascent."
The "representing the direction of steepest ascent", implies that the gradient vector has other possible ascents that are "less steep", but this is not true, right?! Since the gradient vector as a whole is just the modeling of the rate-of-change of the output of an input that is intrinsic to a function, there is no other "option" for ascending, except the one option we have which is modeled by the gradient vector.
[ANOTHER BAD ANALOGY AHEAD]: It's like saying, "our function is $f(x) = 2x$ so $f'(x) = 2$ gives us our direction of steepest ascent!" But that isn't true, 2 is the ONLY ascent, its not like there are other "less steep" ascents that are possible for this function.
I think a better (ROUGH) explanation of the gradient vector would be something like:
The gradient vector combines all the variable's individual contributions to rate-of-change in a multivariable function to model the direction of the rate-of-change for an output of the function.
That is super rough, but I think it gets the jist. Am I missing something? Or are these explanations leading me astray?
Best Answer
The standard motivation for this is in terms of directional derivatives.
Define the derivative of a real-valued function $f$ (say on $\mathbb{R}^n$) in the direction of a vector $u$ as
$$\nabla_u f = u \cdot \nabla f.$$
It’s easy to see that this quantity is maximized among vectors of unit norm by $v = \nabla f/|\nabla f|$, since this vector is parallel to $\nabla f$ at each point. This yields
$$ \nabla_v f = |\nabla f|, $$
illustrating why the gradient is called the direction of maximal increase.
EDIT: Your misunderstanding seems to come from the fact that you think partial derivatives cannot be represented as directional derivatives. This is false. Let $e_i$ denote the $i^{th}$ standard basis vector, then
$$ \nabla_{e_i}f = e_i\cdot \nabla f = \frac{\partial f}{\partial x_i}. $$
Moreover, $\nabla f = \sum_i \frac{\partial f}{\partial x_i} e_i$, so the gradient really is a linear combination of basis vectors.