Consider a die with probability space $(\Omega = \{one,two,three,four,five,six\}, \mu)$ such that $\forall \omega \in \Omega, \mu(\omega) = 1/6$. Consider also a coin with probability space $(\Lambda = \{H, T\}, \nu)$ such that $\nu(H)=\nu(T) = 1/2$.
Let us define the random variables $D: \Omega \to \{1, \dots, 6\}$ and $C: \Lambda \to \{0, 1\}$, and define $E = 1 + \sum_{i = 1}^{5} C_i$ where $C_i$ are independent copies of $C$. Note that $E: \Lambda^5\to \{1, \dots 6\}$, the same output space as $D$.
We can define total variation distance between measures on a same sample space, but here $\mu$ and $\nu$ are not on the same space. Still it seems to me that it makes sense to define the total variation between the random variables $D$ and $E$ in the following sense:
$$\| D – E \|_{VAR} = \sup_{A \subseteq \{1,\dots, 6\}} |Pr(D \in A) – Pr(E \in A)|$$
and
$$\| D – E \|_{VAR} = \sum_{k = 1}^{6} |Pr(D = k) – Pr(E = k)|$$
Is there a problem with this definition ? Is that standard ? What did I really do here ?
Best Answer
If we start with probability spaces $(\Omega,\mathcal A,P)$ and $(\Omega',\mathcal A',Q)$ and define random variables $X:\Omega\to\mathbb R$ and $Y:\Omega'\to\mathbb R$ then actually the new measurable space $(\mathbb R,\mathcal B(\mathbb R))$ appears with on it the two probability measures $P_X$ and $Q_Y$.
Here $P_X$ is prescribed by $B\mapsto P(X\in B)$ and $Q_Y$ is prescribed by $B\mapsto Q(Y\in B)$.
This situation invites us to take a look at: $$\delta(P_X,Q_Y)=\sup_{B\in\mathcal B(\mathbb R)}|P_X(B)-Q_X(B)|=\sup_{B\in\mathcal B(\mathbb R)}|P(X\in B)-Q(Y\in B)|$$
So actually not the total variation of pair $(P,Q)$ is measured (in your question denoted as $\mu$ and $\nu$) but the total variation of the distributions of $X$ and $Y$.