For the function $f:\mathbb{R}\to\mathbb{R}$, I can write the Taylor expansion
$$f(x+h) = f(x) + f'(x)h + \frac{1}{2!}f''(x)h^2 + O(h^3)$$
where the remainder is $o(h^2)$ as well.
I'm confused with what is the remainder written with Big oh for the second-order expansion in the multivariate case $f:\mathbb{R}^n\to\mathbb{R}$, i.e.
$$f(\mathbf{x} + \mathbf{h}) = f(\mathbf{x}) + Df(\mathbf{x})\mathbf{h} + \frac{1}{2!}\mathbf{h}'D^2f(\mathbf{x})\mathbf{h} + O(?)$$
Should it be something like $O\left(\sum_{k,l,m}h_kh_lh_m\right)$? I know that with little oh it is simply $o(\|\mathbf{h}\|^2)$.
Best Answer
By Cauchy-Schwarz $(\sum_{k,l,m}h_kh_lh_m)^2\leq n^6\sum_{k}h_k^2 \sum_{l}h_l^2 \sum_{m}h_m^2 = n^6\|\mathbf{h}\|^6$.
Also, $\|\mathbf{h}\|^6 \leq (\sum_{k,l,m}h_kh_lh_m)^2$.
Hence $\mathcal{O}(\sum_{k,l,m}h_kh_lh_m)=\mathcal{O}(\|\mathbf{h}\|^3)$, as you might expect.
I am not making any effort to explain why the error term might be of either of those forms; I am hoping you had some intuitive understanding which brought you to your error term.