This thought came to me yesterday when I was doing a simple first moment approximation of a function of a discrete random variable, using the standard Taylor expansion. For a discrete random variable, strictly speaking we should be working with finite difference, so why would using the Taylor expansion which assumes a continuous variable work? Are there convergence results showing that this method works just as well for discrete vs. continuous r.v.? or is there actually a difference in how well the approximation works depending on whether the r.v. is discrete vs. continuous?
As requested, here is a specific example to illustrate the question.
Consider discrete random variable $X$ with known mean $\mu$ and known variance $\sigma^2$, and we'd like to approximate $\mathbb{E}[g(X)] = \mathbb{E}[\exp(-X)]$.
A standard approach is to use Taylor expansion for the function $g$ evaluated at $\mu$ to approximate $\exp(-X)$,
$$\mathbb{E}[g(X)] = \mathbb{E}[\exp(-X)] \approx g\left(\mu\right)+\frac{g^{\prime \prime}\left(\mu\right)}{2} \sigma^{2} = \exp(-\mu) + \frac{\exp(-\mu)}{2}\sigma^2$$
I can see how this is perfectly reasonable if $X$ is a continuous r.v., say defined on $[0, \infty)$. But it seems there is a technical issue with taking derivative if $X$ is discrete, say $X$ is defined only on non-negative integers, i.e. $X \in \{0, 1, 2, …\}$, as the usual derivative can only be applied on continuous domain.
Best Answer
The appropriateness of the use of the Taylor series in this case (and in every case where it should be used) hinges on the function, not on the random variable.
To clarify the issue, consider the following statement:
$e^{0.5} \approx 1 + \frac{0.5}{1!} + \frac{0.5^2}{2!} + \frac{0.5^3}{3!}$
This statement holds because $e^x$ is a nice function, not because there's anything special about the number $0.5$. Note that no other numbers need to be in consideration at all for the statement to make sense.
Random variables work in the same way. For a given $\omega \in \Omega$, the quantity $X(\omega)$ is a fixed number, and the Taylor approximation holds just like it did for the example of $0.5$ above. You can make this claim simultaneously across many values of $\omega$ to get an approximate equality involving the random variable, and then you can apply the linearity of expectation to get the desired result.