This seems like a relatively simple equation, but I have not really found an explanation that works for me. In my probability class, we were simply given that the kth moment of a random variable X is $E[X^k]$. Are we supposed to be squaring the probability as well when we are computing the second moment? I.e. computing E[X] means computing $\sum_i iP(X=i)$. So then is $E[X^2]$ equal to $\sum_i (iP(X=i))^2$? Or is it $\sum_i i^2P(X=i^2)$ or $\sum_i i^2P(X=i)$? From reading around the internet, I assume it is the last equation, but I do not really understand the intuition behind this.
Here's what I understand: Moments are used to determine how much data points are spread out. So the Expectation of {5, 5, 5, 5, 5} is the same as the expectation of {3, 4, 5, 6, 7}, but they have different second moment-expectations. I understand the concept, but not much more.
Clarification, intuition, or links to visual representations would be much appreciated
Edit: Additionally, if someone could provide an explanation for this simple fact, it would help: "$E[X_i]^2 = E[X_i] = \frac{1}{2}$ where $X_i$ is 1 if you toss a coin and get heads, and 0 if you get tails." What is $E[X^2]$ for n coin tosses?
Best Answer
It is indeed the last one, as you suspect.
Here's what's happening: for any discrete random variable $X$, the expected value $\mathbb E[X]$ is $\sum i \mathbb P(X = i)$, as you noted. To take a particular example, consider the variable $X$ that can be $1$ with probability $1/2$, can be $2$ with probability $1/4$, and can be $5$ with probability $1/4$. I think you're already comfortable with the idea that for this variable, $$\mathbb E[X] = 1 \cdot \frac 1 2 + 2 \cdot \frac 1 4 + 5 \cdot \frac 1 4 = \frac 9 4.$$
Now, let's consider another variable, $Y = X^2$. (That is, this variable is nothing more than the square of $X$, but I'm going to call it $Y$.) Just like you already know, we can find $\mathbb E[Y]$ as $\sum i \cdot \mathbb P(Y = i)$. But let's think carefully about what $Y$ can be here:
So, we must have $$\mathbb E[Y] = 1 \cdot \frac 1 2 + 4 \cdot \frac 1 4 + 25 \cdot \frac 1 4 = \frac{31}{4}$$ for the same reasons as above. Note that this turns out to be nothing more than just the last answer you pitched above, $\sum i^2 \mathbb P(X = i)$.
You're asking for intuition about why this formula works; my best attempt at that is outlined above. If you want to compute $\mathbb E[f(X)]$ for some function $f(X)$, then your task is to imagine $f(X)$ as some new random variable of its own right. Its distribution is "shaped" similarly to that of $X$ -- that is, it will possess the same probabilities, but just different values. (Specifically, it will be the values that $X$ can be, pushed through the function $f$.) Under that perspective, the logic for the right formula is hopefully clear.
For coin tosses, I think what you meant to say was
where the exponent is on the inside of the expectation. Here's what's going on there; we usually think of a coin flip as a variable $X_i$ that will be either $1$ or $0$, each with probability $1/2$. Under that scenario, note that $X_i^2 = X_i$, since $1^2 = 1$ and $0^2 = 0$. Thus, $$\mathbb E[X_i^2] = 1^2 \cdot \frac 1 2 + 0^2 \cdot \frac 1 2 = \frac 1 2$$ which is the same calculation as we'd get for $\mathbb E[X_i]$.