Solved – Test if two normally distributed random variables have the same mean

normal distributionp-valuerandom variable

We have two independent random variables which follow normal distributions $X_1\sim \mathcal N(\mu_1,\sigma_1)$, $X_2\sim \mathcal N(\mu_2,\sigma_2)$.

From the context, we have that $\mu_1\leq\mu_2$. It is also reasonable to assume that $\sigma_1=\sigma_2$, or $\sigma_1\geq\sigma_2$.

I want a test/procedure to determine if $\mu_1=\mu_2$. I know it is not possible to make such a test that always returns the correct answer, but I want a test where at least the probability of a false positive or a false negative can be determined, and perhaps reduced by incrementing the number of observed samples of $X_1$ and $X_2$.

Moreover, drawing values from $X_2$ is very time consuming, so we want to minimize the amount of observations of $X_2$ that we use in this test (observing $X_1$ is simple).

Context of the question: I am trying to see if a randomized optimization algorithm makes any difference or not. $X_1$ are the scores of random solutions (easy to generate) while $X_2$ are the scores of solutions outputted by the algorithm (it is expensive to run the algorithm). You want to minimize the number of times that you have to run the algorithm to perform this test (preferably only once, and only if the first sample does not give you enough information, more times).

Best Answer

The test

if (a) $X_1 \sim N(\mu_1, \sigma_1)$ and (b) $X_1 \sim N(\mu_2, \sigma_2)$ and (c) $X_1$ and $X_2$ are independent then if you draw a sample of size $N_1$ from the distribution of $X_1$ and a sample of size $N_2$ from the distribution of $X_2$, then the arithemtic average of the sample from $X_i$ is $\bar{x}_i$ and is distributed $\bar{X}_i \sim N(\mu_i, \frac{\sigma_i}{\sqrt{N_i}})$. If the variables are independent then $\bar{X}_1-\bar{X}_2 \sim N(\mu_1 - \mu_2, \sqrt{\frac{\sigma_1^2}{N_1}+\frac{\sigma_2^2}{N_2}})$.

You seem to have some a priori knowledge about the 'truth' namely that $\mu_1 \le \mu_2$ so if you want to find evidence that $\mu_1 < \mu_2$ then (see What follows if we fail to reject the null hypothesis?) your $H_1$ should be $H_1: \mu_1 < \mu_2$ and in order to 'demonstrate' this, one assumes the opposite, but as you say that $\mu_1 \le \mu_2$ the opposite is $H_0: \mu_1 = \mu_2$.

So you have a one-sided test $H_0: \mu_1 - \mu_2 = 0$ against $H_1: \mu_1 - \mu_2 < 0$.

By the above, if $H_0$ is true, then $\mu_1-\mu_2=0$ and therefore $\bar{X}_1-\bar{X}_2 \sim N(0, \sqrt{\frac{\sigma_1^2}{N_1}+\frac{\sigma_2^2}{N_2}})$. With this you can define a left tail, one sided test and using the Neyman-Pearson lemma it can be shown tbe be the unformly most powerfull test. If you known $\sigma_i$ then you can use the normal distribution to define the critical region in the left tail, if you don't know them then you estimate them from the samples and then you have to define the critical region using the t-distribution.

To define the test you have to (1) define a significance level $\alpha$ (2) draw a sample of size $N_1$ from $X_1$ and of size $N_2$ from $X_2$, then (3) compute $\bar{x}_i$ for each of these samples and then compute the p-value of $\bar{x}_1 - \bar{x}_2$ knowing that $\bar{X}_1-\bar{X}_2 \sim N(0, \sqrt{\frac{\sigma_1^2}{N_1}+\frac{\sigma_2^2}{N_2}})$. If the obtained p-value is below $\alpha$ then the test concludes that there is evidence in favour of $H_1$. One can also compute the $\alpha$-quantile of the distribution under $H_0$, : $q_{\alpha}^0$ and $H_0$ will be rejected when $\bar{x}_1-\bar{x}_2 \le q_{\alpha}^0$.

The probability of a false positive is equal to the significance level $\alpha$ that you choose, the probability of a false negative can only be computed for a given value of $\mu_1 - \mu_2$.

The sample sizes

If you have such a value for $\mu_1 - \mu_2$ e.g. the difference is -0.05, then you can compute the type II error for this value:

The type II error is the probability that $H_0$ is accepted when it is false, to compute it, let us assume that it is false and that $\mu_1-\mu_2 = -0.05$. In that case $\bar{X}_1-\bar{X}_2 \sim N(-0.05, \sqrt{\frac{\sigma_1^2}{N_1}+\frac{\sigma_2^2}{N_2}})$ (the mean has changed).

Now we have to compute the probability that $H_0$ is accepted when the latter is true. This is the probability of observing $q_\alpha^0$ under the above distribution with mean -0.05.

it will be a function of $N_1$ and $N_2$. You can then fix a value for $N_2$ (the one that is difficult to sample) and find $N_1$ by fixing a type II error that is acceptable for you.

Related Solutions

Solved – Joint distribution of sum of independent normals

Not entirely clear to me from reading the comments if the OP has solved this but there is no answer so I will write one.

The distribution of each $Y_i$ will be normal with given means and variances:

$\mu_0+\mu_1$ and $\sigma_0^2+\sigma^2_1$ for $Y_0$ and

$\mu_1+\mu_2$ and $\sigma_1^2+\sigma^2_2$ for $Y_1$. Now finally we need to determine if there is a correlation between $Y_0$ and $Y_1$. To do this we can calculate

$$\mathbb{C}ov(Y_0,Y_1)=\mathbb{C}ov(X_0+X_1,X_1+X_2) =\mathbb{C}ov(X_1,X_1) =\mathbb{V}ar(X_1) =\sigma_1^2. $$ Now you can turn this into a correlation by dividing by the square roots of the variances

$$\rho = \frac{\sigma_1^2}{\sqrt{(\sigma_0^2+\sigma^2_1)(\sigma_1^2+\sigma^2_2)} }.$$

Now we know that the sum of two normal random variables is normally distributed so that both $Y_0$ and $Y_1$ have normal distributions with the stated means and variances and the correlation is given by $\rho$ above. So the joint density of $Y_0, Y_1$ is

$$ f(y_0,y_1) = N\left(\vec{\mu} = \begin{bmatrix} \mu_0+\mu_1 \\ \mu_1+\mu_2 \\ \end{bmatrix}, \Sigma = \begin{bmatrix} \sigma^2_0+\sigma^2_1 &\sigma_1^2 \\ \sigma_1^2 & \sigma^2_1+\sigma^2_2 \\ \end{bmatrix} \right). $$

Solved – When is the distribution of product of two normal distributed variables near normal distribution

(this answer uses parts of @whuber's comment)

Let $X,Y$ be two independent normals. We can write the product as $$ XY = \frac14 \left( (X+Y)^2 - (X-Y)^2 \right) $$ will have the distribution of the difference (scaled) of two noncentral chisquare random variables (central if both have zero means). Note that if the variances are equal, the two terms will be independent. Since chisquare distribution is a case of gamma, Generic sum of Gamma random variables is relevant. I will give a very special case of this, taken from the encyclopedic reference https://www.amazon.com/Probability-Distributions-Involving-Gaussian-Variables/dp/0387346570

When $X$ and $Y$ are independent, zero-mean with possibly different variances the density function of the product $Z=XY$ is given by $$ f(z)= \frac1{\pi \sigma_1 \sigma_2} K_0(\frac{|z|}{\sigma_1 \sigma_2}) $$ where $K_0$ is the modified Bessel function of the second kind.

This can be written in R as

    dprodnorm  <-  function(x, sigma1=1, sigma2=1) {
       (1/(pi*sigma1*sigma2)) * besselK(abs(x)/(sigma1*sigma2),  0)
    }
    ### Numerical check:
    integrate( function(x) dprodnorm(x), lower=-Inf,  upper=Inf)
    0.9999999 with absolute error < 3e-06

Let us plot this, together with some simulations:

    set.seed(7*11*13)  
    Z  <-  rnorm(10000) * rnorm(10000)
    
    hist(Z, prob=TRUE, nclass="scott", ylim=c(0, 1.5), 
            main="histogram and density of product of independent 
                  normals")
    plot( function(x) dprodnorm(x),  from=-5,  to=5,  n=1001,  
          col="red", add=TRUE, lwd=3)
    ### Change to nclass="fd" gives a closer fit

The plot shows quite clearly that the distribution is not close to normal.

The stated reference do also give more involved cases (non-zero means ...) but then expressions for density functions becomes so complicated that they only gives characteristic function, which still are reasonably simple, and can be inverted to get densities.