I am studying Probability and Monte Carlo methods, and it feels that the more I study the less I truly understand the theory. I guess I just confuse myself now. So the expected value of a random variable X is:
$E[X] = \int x_i p(x_i)\:dx$
where X is is random variable and p(x) is the pdf of X. This sort of makes sense to me especially of you think in terms of discrete random variable. If an outcome is more likely to be drawn than others, then the pdf for this outcome $x_i$ accounts for this.I have a problem when this apply to a function. So we have:
$E[f(x)] = \int f(x_i) p(x_i) \: dx$
My understanding here is that f can be any sort of function we like. $x_i$ is a possible value from the random variable X and $p(x_i)$ is the probability distribution. Because $x_i$ is random $f(x_i)$ is random. So f(x) is a function of the random variable X.
Now what I don't understand with this: doesn't the result of the expected value E[f(x)] depends on the choice of the pdf? For example of you have a simple function such as f(x) = x (imagine the integration domain is [0,1]), if you choose a uniform pdf or a gaussian distribution, wouldn't the result of E[f(x)] be different?
I am sure I am just mixing in my mind "simple" concepts. I am probably not thinking this problem the right way. If anybody could straighten that up for me, it would be great.
EDIT
Thank you for your answers. So it seems my question wasn't clear enough and complete. From the answer you gave me, I understand that:
- X is distributed according to a give distribution (in other words, X and its PDF are interdependent).
- therefore indeed changing X, implies that the distribution has changed as well
- and yes, E[f(X)] is likely to be different for different Xs.
I think my confusion comes partly from the fact this PDF plays a role in the computation of the Monte Carlo integral and particular the general form in which the integrand is divided by the PDF:
$E[f(X)] \approx { 1 \over N } \sum_{i=0}^{N-1} { f(x_i) \over p(x_i) }.$
where $x_i$ is a sequence of random numbers drawn from a random variable X with distribution p(X). We are trying to approximate:
$F = \int f(x)\:dx.$
We know the result of this MC integral converges in probability to the expected value E[f(X)]. So according to the three points I listed above, wouldn't changing X and its PDF give a different result for E[f(x)] when MC integration is used?
When you measure the area "under" (a way of interpreting what an integral is) the curve, that area is constant. So if we get a different E[f(x)] for something that should always be the same, what I am missing?
In other words, in a lot of books $E[f(X)] = \int f(X) p(X) \: dx$ is presented as "the formula" to calculate the expected value of f. This seems like misleading to me. Should it be more accurate to say "the expected value of f given the particular random variable X with PDF p(X)"?, knowing that if we change X we will get a different E[f(X)].
Best Answer
Let me add a little bit to what's already been said.
In your first example, $E[X] = \int x_i p(x_i)\:dx$, you are looking for the expected value of $X$. Where did the $p$ come from? The original problem would have been to find $E[X]$, given that $X$ has the distribution $p$. Often, $p$ is not explicitly stated when asking the question about the expectation. However it probably was stated as the pdf when $X$ was first brought into the problem.
The same thing is true of your second example. In that case, in order to even ask the question about the expectation of $f(x)$, you would have to have been given the distribution of $x$.
By the way, in this case it would generally be written $f(X)$, with the $X$ in upper case to indicate it's a Random Variable. Generally, the lower case letter is then used, as you did use it, in your integral.
Also, with a continuous Random Variable you wouldn't use the subscript $i$.