Solved – The impact of non-normality (contaminated normal) on type 1 error and power when sigma is known

confidence intervalstatistical-powertype-i-and-ii-errors

When sigma is unknown, contaminated normal distributions tend to have (1) the issue of the actual Type 1 error rate being lower than the nominal level and (2) the issue of reduced power. This pattern has been confirmed by data simulations.

The question then is: does knowing the population variance help with (2) keeping the type 1 error at the nominal level and (2) having good power?

Thank you in advance for your help!

To just add some info:

By power, I simply mean correctly rejecting the null hypothesis of mu=mu0 when this h0 is actually false. I thought that contaminated normal distributions tend to cause wider CIs, which can make correct rejections more difficult.

I did some data simulations to check contaminated normal distributions. Say I repeatedly sample from a contaminated normal distribution with mu=0 and sigma=3 for 1000 times. Then I create 90% confidence intervals based on these samples and check how many of these 1000 CIs contain the population mean. As it turned out (I tried this several times), the percentage of CIs containing the population mean is approximately 90% (that is when sigma is known). So in this case, it seems that the the actual type 1 error rate is comparable to the nominal level of 0.1.

I also tried this assuming that sigma is unknown (so using t-statistics, rather than z), I consistently got more than 90% CIs that contained the population mean, so in this case, the actual type 1 error rate seems to be lower than the nominal level of .1.

But I thought the type 1 error rate and power would suffer even when sigma is known (e.g., the actual type 1 error can be lower the nominal level). But my simulations (assuming that I did it correctly) at least did not show this. So I want to check with you guys to see what's going on. Thanks 🙂

Best Answer

Are you testing 2 populations for the same mean, or one population against a theoretical mean?

Whatever ..... a contaminated normal will tend to inflate the variance estimate (and standard deviation) and thus reduce the value of the t-statistic (by inflating the denominator). It can wreck havoc with the numerator as well, but typically, the inflated denominator will pull the t-statistic towards 0 and inflate the type 1 error.

If $\sigma$ is known, then your test statistic will be a $z$ score, based on the sample mean. The mean is non-robust in this situation, so spurious large values could inflate that score and increase the type-1 error.

I am assuming here that the mean and variance in question are those of the "true" distribution, absent contamination. But concretely, in the real world, what does that actually mean? Suppose I have 2 industrial processes A and B that make widgets. Suppose the diameter of widget A is less than that of widget B (which I want, for some reason), but 10% of the A widgets are way off because of "contamination". Which process should I prefer? And knowing the extent of the contamination - which in a sense is what knowing the variance amounts to in this case - is not going to help very much.

Rather than hope your type 1 and type 2 errors don't lead you astray, would it not be better to cull the outliers, use a robust estimator, or ensure the contamination does not occur in the first place?

Just sayin'

Related Solutions

Solved – Known alpha and type I error rate simulation

If you simulate a lot more tests it gets closer to 5% - I think this is a case of a slower convergence to the "true" value than you seem to expect:

    set.seed(1)
    n=100000 # testing 100,000 times
    t1err=0
    for (i in 1:n){
      x=rnorm(100, 0, 1)
      if (((t.test(x, mu=0))$p.value)<=0.05) (t1err=t1err+1) 
    }
    cat("Type I error rate in percentage is", (t1err/n)*100,"%")

gives 4.894%

Standard Error – Understanding Standard Error When Sigma is Unknown

Suppose you want to estimate the mean $\mu$ of a normal population using the mean $\bar X$ of a random sample $X_1, X_2, \dots, X_n$ of size $n$ from the population.

The term 'standard error' usually refers to the the standard deviation of an estimator. In the current situation the standard error of $\bar X$ is $\sigma/\sqrt{n},$ where $\sigma$ is the standard deviation of the population.

However, if $\sigma$ is unknown and estimated by the sample standard deviation $S= \sqrt{\frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2},$ then $T = \frac{\bar X = \mu}{S/\sqrt{n}}$ has Student's t distribution with $\nu = n-1$ degrees of freedom, $\mathsf{T}(\nu).$

This fact is used to make a 95% confidence for (estimating) $\mu$ of the form $\bar X \pm t^*\frac{S}{\sqrt{n}},$ where $t^*$ cuts probability $0.025= 2.5\%$ from the upper tail of the distribution $\mathsf{T}(\nu).$ [By the symmetry of the t distribution $-t^*$ cuts 2.5% of the probability from the lower tail of the distribution, leaving 95% in the middle.]

In this situation the estimated standard deviation of $\bar X$ is $S/\sqrt{n}.$ Usually the word estimated is left out because as soon as you see $S$ you know it's an estimate of $\sigma.$

You might say you have two standard errors of the sample mean $\bar X,$ the theoretical one $\sigma/\sqrt{n},$ and the estimated one, $S/\sqrt{n}.$ However, if $\sigma$ is unknown, only the (estimated) standard error $S/\sqrt{n}$ is of practical use---in making a confidence interval as above or in testing a hypothesis about $\mu.$

Note: There are situations in which $\mu$ is unknown and $\sigma$ is known. Then a 95% confidence interval for $\mu$ is $\bar X \pm 1.96 \frac{\sigma}{\sqrt{n}},$ where 1.96 cuts probability 2.5% from the upper tail of the standard normal distribution. (This confidence interval is based on the fact that $Z = \frac{\bar X - \mu}{\sigma/\sqrt{n}}$ has a standard normal distribution.) Notice that the theoretical standard error $\sigma/\sqrt{n}$ is used when $\sigma$ is known.

Best Answer

Related Solutions

Solved – Known alpha and type I error rate simulation

Standard Error – Understanding Standard Error When Sigma is Unknown

Related Question