Suppose there are two random variables, $X$ and $Y$. Each of the random variables subjects to its distribution respectively. Now, I have two approaches of sampling.
Approach 1
Sample $X_1, …, X_N$ independently and form the set $\{X_j\}^N:=\mathcal{X}$. In the same way we sample and form the set $\{Y_j\}^N:=\mathcal{Y}$. Then, the final samples would be all $NM$ elements in $\mathcal{X} \times \mathcal{Y}$.
Approach 2
Sample $(X_1, Y_1), …, (X_{NM}, Y_{NM})$ independently.
Is it possible to say something about the distribution of two approaches? Which approach would 'cover' the sample space more reasonably?
Sorry for asking the question in a somehow ambiguous way. How I came up with this question is that I was trying to do some numerical experiments that involves two random variables, and I was trying to figure out which way of sampling would make more sense. Many thanks in advance.
Best Answer
Edit: Correction to my answer. In the second case the samples are not iid.
We can resolve the ambiguity by stating the problem as:
Suppose that $(X_i,Y_i) \sim P(X,Y)$ where $P(X,Y)$ is the joint probability distribution.
Now consider the following approaches:
Clearly in the first case we will get $N \times M$ independent and identically distributed (iid) pairs.
How about the second case? In this case the samples are not iid. It actually doesn't depend on $P(X,Y)$; the samples will be dependent.
Proof: We consider the case $P(X,Y)=P(X)P(Y)$. If the pairs are not iid here, then they are not iid for $P(X,Y)$ either; so it is sufficient to show that the pairs are dependent for this case. Next, let $X_1$ and $X_2$ be iid samples from $P(X)$. Let $Y_1$ be a sample from $P(Y)$. Define the pairs $A:=(X_1,Y_1)$ and $B:=(X_2,Y_1)$. If the pairs are independent then $P(A|B) = P(A)$ holds. $P(A) = P(X_1) P(Y_1)$. On the other hand $P(A|B) = P((X,Y)=(X_1,Y_1) | Y=Y_1)$ which then gives $P(A|B) = P(X_1)$. Clearly $P(A) \neq P(A|B)$ and $A$ and $B$ are not iid. $\blacksquare$