Solved – Known alpha and type I error rate simulation

Question

I am not sure why running the following code in R does not result in a type I error rate of exactly 5% given alpha = 0.05 when set.seed is set to 1. Is there a reason why running the simulation below gives a type I error rate of 4% when it should be exactly 5% using set.seed(1)? Sometimes when you run the simulation, you get a p-value <0.05 in 3.2% of the simulations, is this due to rnorm drawing a random normal number each time?

set.seed(1)
n=1000 # testing 1,000 times
t1err=0
for (i in 1:n){
  x=rnorm(100, 0, 1)
  if (((t.test(x, mu=0))$p.value)<=0.05) (t1err=t1err+1) 
}
cat("Type I error rate in percentage is", (t1err/n)*100,"%")

---

Why is it that seed = 123, p = 3.2% sometimes with n = 1000? This seems far off from the stated true value of 5%? Is this due to how R draws random normal numbers each time using rnorm regardless of set.seed? Supposedly the function of set.seed is to make the simulations more reproducible each time.

Answer 1

If you simulate a lot more tests it gets closer to 5% - I think this is a case of a slower convergence to the "true" value than you seem to expect:

    set.seed(1)
    n=100000 # testing 100,000 times
    t1err=0
    for (i in 1:n){
      x=rnorm(100, 0, 1)
      if (((t.test(x, mu=0))$p.value)<=0.05) (t1err=t1err+1) 
    }
    cat("Type I error rate in percentage is", (t1err/n)*100,"%")

gives 4.894%