I am able to prove that $x$ must be a stationary point, i.e., $\nabla f(x) = 0$. However I feel the proof given in my lecture is wrong, the key part of the proof uses the Taylor series expansion
$$f(x) \leq f(x + td) = f(x) + \frac{t^2}{2}d^T\nabla^2f(x)d + o(t^2),$$
where $d$ can be arbitraty direction vector.
The problem is the above expansion only works when all third order and below partial derivatives of $f$ exist and are continuous, not just twice, according to Taylor's theorem. I believe the result is still correct, but how to prove it rigorously?
Update: turns out for the Taylor series expansion in the above form, $f$ only needs to be in $C^2$, hence the proof is actually correct.
Best Answer
Thanks for all the comments, here is a proof for my own question.
For a twice continuously differentiable function $f$, if we use Peano form of the remainder, its Taylor series expansion can be written as $$f(x + td) = f(x) + \nabla f(x)^Tdt + \frac{t^2}{2}d^T\nabla^2f(x)d + o(t^2)$$ for any $d \in \mathbb{R}^n$ and $|t|$ small enough (how small it needs to be depends on $d$). My confusion originated from I used other forms of the remainder, which requires more conditions.
If $x$ is a local minimiser, then we know $\nabla f(x) = 0$ and $$f(x + td) - f(x) = \frac{t^2}{2}d^T\nabla^2f(x)d + o(t^2) \geq 0.$$ Divide both sides by $t^2$ and let $t \rightarrow 0^+$, we have $$\lim_{t \rightarrow 0^+}\frac{f(x + td) - f(x)}{t^2} = \frac{1}{2}d^T\nabla^2f(x)d \geq 0$$ for all $d$ and hence $\nabla^2f(x)$ is positive semi-definite.