If
$x \sim N(\mu,\sigma^2)$
then by first principles,
$E(e^x) = e^{\mu + \sigma^2 / 2}$
I am trying to figure out where the "Delta method" is wrong here:
If
$(x-\mu) \sim N(0,\sigma^2)$
then
$(f(x)-f(\mu)) \sim N(0,\sigma^2 * f'(\mu)^2)$
Delta method using $f(x) = e^x$ will imply
$(e^x-e^\mu) \sim N(0,\sigma^2 * (e^\mu)^2 )$
which implies
$E(e^x) = e^\mu$
Best Answer
To see what is happening, let's go back to the first principles of the Delta method. The Delta method is based upon the Taylor expansion of a function $f(x)$ around the value $f(\mu)$, where $\mu$ is the mean of the function:
$$f(x) = f(\mu) + (x-\mu)f'(\mu) + {1\over 2}(x-\mu)^2f''(\mu) + \dots$$
Taking the expectation and noting that $\mathrm{E}(x-\mu) = 0$ by definition of $\mu$ gives us:
$$\mathrm{E}f(x) = f(\mu) + {1\over 2}\mathrm{E}(x-\mu)^2f''(\mu) + \dots$$
Dropping all but the first two terms of the expansion and substituting $\sigma^2 = \mathrm{E}(x-\mu)^2$ results in:
$$\mathrm{E}f(x) \approx f(\mu) + {1\over 2}\sigma^2f''(\mu)$$
Now for this specific case. Setting $f(x) = e^x$ and substituting gives:
$$\mathrm{E}e^x \approx e^{\mu} + {1\over 2}\sigma^2e^{\mu}$$
Rearranging terms slightly:
$$\mathrm{E}e^x \approx e^{\mu}\left(1 + {1\over 2}\sigma^2\right)$$
Now, let's consider the Maclaurin expansion (Taylor expansion around zero) of $e^x$:
$$e^x = 1 + x + {x^2\over 2} + {x^3\over 6} + \dots$$
If we substitute ${1\over 2}\sigma^2$ for $x$ in the above, we can readily see that the term $\left(1 + {1\over 2}\sigma^2\right)$ which multiplies $e^{\mu}$ above corresponds to the first two terms of the Maclaurin expansion of $\exp\{{1\over 2}\sigma^2\}$. If we were to use the full series, which we can do because we know it converges, we would get:
$$\mathrm{E}e^x = e^{\mu}e^{{1\over 2}\sigma^2} = e^{\mu + {1\over 2}\sigma^2}$$
which is, as you've observed, correct.
So what has happened here is that we have truncated an infinite series as part of our application of the Delta method, and the difference between the exact result and the approximate result of the Delta method is due to that truncation.