First Part: (First-order derivative)
Assuming $f$ is an infinitely differential function everywhere, the Taylor series of $f(x + h)$ at $x$ is
\begin{align}\tag{1}
f(x + h) = f(x) + hf'(x) + \frac{1}{2}h^2f''(\xi)
\end{align}
where $\xi$ is a number between $x$ and $x+h$.
After rearrangment of terms in (1), we can write
$$
f'(x) = \frac{f(x+h) – f(x)}{h} – \frac{1}{2}hf''(\xi).
$$
Now, we define a finite difference approximation of $f'(x)$ by
$$
f'_h(x) = \frac{f(x+h) – f(x)}{h},
$$ and we express
$$
f'(x) = f'_h(x) + E_1
$$
where approximation error $E_1$ satisfy
\begin{align}
|E_1| &= |- 0.5 hf''(\xi)| \\
&\leq Ch
\end{align}
assuming $|- 0.5f''(\xi)| \leq C$. Now, using the definition of Big-O notation, we can say
\begin{align}\tag{2}
f'(x) = f'_h(x) + O(h)
\end{align}
This is a very standard result. However, I have a question for clarification.
Question 1:
It seems that the constant $C$ can be based on the local behavior of function between $x$ and $x+h$. Can I say that $C$ depends on $h$? Moreover, can I comment on the behavior of $C$ as $h \to 0$?
Second Part: (Square of the first-order derivative)
Using (2), the square of $f'(x)$ can be expressed as
$$
\Big(f'(x)\Big)^2 = \Big(f'_h(x) + O(h)\Big)^2
= \Big(f'_h(x)\Big)^2 + 2f'_h(x)O(h) + O(h^2) = \Big(f'_h(x)\Big)^2 + E_2
$$
where the approximation error $E_2$ is
$$
E_2 = 2f'_h(x)O(h) + O(h^2).
$$
It seems that, $E_2$ depends on the local approximation quantity $f'_h(x)$.
Question 2:
How can we estimate the leading order term for the error $E_2$?
Best Answer
1.
You don't necessarily change $C$ as $h \to 0$. The constant $C$ is usually take to be a uniform bound of $f'$ in a fixed interval and, although changing this constant according to $h$ can give you better error bounds, it changes nothing with respect to convergence.
2.
If $f'$ is bounded in some interval $[a,b]$ containing $x, x+h$, you can use Lagrange's theorem to get $|f'_h(x)|\leq \|f'\|_{\infty}$ and, therefore, the term $f'_h(x) O(h)$ is in fact $O(h)$. I think this also answers the question about the leading term.