It is my first time learning probability theory, and if I understand correctly, the following is the meaning/motivation behind the definition of a random variable:
" A function's output is uniquely determined by its input. Random variable $f$ is a function defined on a sample space of a random phenomenon. Only after a realisation of the random phenomenon, we can know what is $x$ and hence $f(x).$ Additionaly, if the function is measurable, knowing the distribution of the underlying random phenomenon(on the sample space $\Omega$), one can understand the probability distribution of $f(\Omega).$ Thus, the so-called measurable functions are used to model this and are referred to as random variables.
Now let $f$ and $g$ two measurable functions defined on a measurable sample space $(\Omega,\mu).$ We say that $f$ and $g$ are independent random variables if the elements of sigma algebra generated by them are mutually independent.
What is the motivation behind this definition. ?
I understand that by independence of two events, we mean that occurrence of one does not change the chance of occurrence of others. Also, I am looking for some explicit real-world examples of a few functions that are all defined on the same sample space $\Omega$ to better understand the following:
- Random variables are independent but not identically distributed.
- Identically distributed but not independent.
Practical ones are appreciated. Thanks in advance.
Best Answer
Motivation
Fix probability space $(\Omega, \mathcal F, \mathbb P)$.
Let's see how the definition of independence works in practice. Let $ f, g: \Omega \to \mathbb R$ be measurable functions (random variables). Denote $\mathcal A = \sigma(f) = \{ f^{-1}(E)| E\in \operatorname{Borel}(\mathbb R)\}$ and $\mathcal B = \sigma(g)$. $\sigma$-algebras $\mathcal A$ and $\mathcal B$ are independent, if $$ \forall A \in \mathcal A, B \in \mathcal B \quad \mathbb P (A\cap B) = \mathbb P(A)\mathbb P(B). $$ But elements of $\mathcal A$ are of the form $f^{-1}(E)$, for some Borel sets $E$. By convention, we write them $\{f \in E \}$. So our definition of independent variables is $$ \forall E_1, E_2 \in \operatorname{Borel}(\mathbb R) \quad \mathbb P (f\in E_1, g\in E_2) = \mathbb P(f\in E_1)\mathbb P(g\in E_2). $$ where again we use some notation conventions (ommiting $\{$, $\}$ and replacing $\cap$ with $,$).
This shows what the definition is saying: any event that we could come up with which regards $f$, is independent with any event regarding $g$. This is fairly strong condition, which comes in handy in many situations.
Examples
1. Independent but not identically distributed.
Very silly example could be $f(\omega) = 1$, and $g(\omega) = 0$ for all $\omega \in \Omega$. Those functions are measurable in any probability space, and all constant variables are independent (because $\sigma(f) = \sigma(g) = \{\Omega, \emptyset\}$). They obviously have different distributions.
Something more useful could be for example $\Omega = \{0, 1, 2, 3\}$, with $$ f(x) = \begin{cases} 0 : \quad \text{when } x \leq 1\\ 1 : \quad \text{when } x > 1 \end{cases}, \quad g(x) = (x\!\!\!\! \mod 2) + 7 $$ where we deal with classical probability. Here again, distributions are quite similar (only shifted by $7$), but not the same.
Product space
In general, given 2 random variables defined on different probability spaces we can "produce probability space in which they are independent". Given $(\Omega_1, \mathcal F_1, \mathbb P_1), (\Omega_2, \mathcal F_2, \mathbb P_2), f:\Omega_1 \to \mathbb R$ and $g:\Omega_2 \to \mathbb R$ we define $\tilde f, \tilde g$ on product space $\Omega_1 \times \Omega_2$ as $$ \tilde f(\omega_1, \omega_2) = f(\omega_1), \quad \tilde g(\omega_1, \omega_2) = g(\omega_2). $$ It is quite easy to show, that $\tilde f$ has the same distribution as $f$, similarly for $g$. On top of that $\tilde f$ and $\tilde g$ are independent. So you could take any 2 variables in some model (different distribution or not) and make model in which they are independent.
2. Identically distributed but not independent.
Take any symmetrical distributed variable $f$. Define $g = -f$. Then $f$ and $g$ are not independent, but since distribution of $f$ was symmetrical, we have $g \overset D = f$.
Down-to-earth version of it can be for example $f(x) = 2x-1$ on $\Omega = [0, 1]$ with Lebesgue measure as probability. $f$ has uniform distribution $U([-1, 1])$. Then $g(x) = 1 - 2x$ has the same uniform distribution. Knowing value of $f$ gives immediately value of $g$, so they are not independent (you could check that generated $\sigma$-algebras are the same, which is contradiction to $\sigma(f) \cap \sigma(g) = \{\Omega, \emptyset\}$ for any independent $f, g$).