[Math] Gradient is NOT the direction that points to the minimum or maximum

Question

I understand that the gradient is the direction of steepest descent (ref: Why is gradient the direction of steepest ascent? and Gradient of a function as the direction of steepest ascent/descent).

However, I am not able to visualize it.

The Blue arrow is the one pointing towards minima/maxima. The gradient (black arrow) is not and that's why we have this zig-zag motion.

Then how come gradient is the direction of steepest descent/ascent?

I have a related question: Why does gradient ascent/descent have zig-zag motion?

Answer 1

Although the gradient vector is defined at every point, it is really a local concept.

At any given point, it tells you the direction in which the function changes with the greatest rate. If you think of the function as height, then it gives the direction in which the ground is steepest.

As soon as you move an inch, the ground changes and the steepest direction changes.

Instead of the black zig-zag, you need an integral curve of the gradient vector field.