Equivalence Test – How to Perform Equivalence Test for Binomial Data?

equivalencehypothesis testingstatistical significance

I have two binomial (success=1 and failure=0) datasets and intend to apply a statistical test to show whether they are statistically equivalence or not. First dataset (x) includes around 164 data points in which 58 of them are "1" and the rest are "0". The second dataset/group (y) consists of 280 data points in which 113 of the are "1" and the rest are "0". I applied the following test in r:

  equivalence::tost(x,y,paired = FALSE, epsilon=0.15,conf.level = 0.95, alpha = 0.05)

And I got the following results:

   data:  x and y
   df = 348.24
   sample estimates:
   mean of x mean of y 
   0.3536585 0.4035714 

   Epsilon: 0.15 
   95 percent two one-sided confidence interval (TOST interval):
   -0.12840509  0.02857931
   Null hypothesis of statistical difference is: rejected 
   TOST p-value: 0.01809221

Regardless of the epsilon value (which is somehow high! and shows the difference between the mean of two population), I think this function is for another type of data, Not binomial data. Does anyone know, Is this the right function to test the equivalency and if not, is there any other test which is appropriate for Binomial data?

I also applied the following test as @user2974951 suggested:

 prop.test(c(58, 113), c(164, 280),alternative = 't')

And I got these results which I can not interpret:

 data:  c(58, 113) out of c(164, 280)
 X-squared = 0.88749, df = 1, p-value = 0.3462
 alternative hypothesis: two.sided
 95 percent confidence interval:
 -0.1477880  0.0479622
 sample estimates:
  prop 1    prop 2 
0.3536585 0.4035714

Best Answer

To do an equivalence test, you need some equivalence margins on some appropriate scale. Then you use some method that gives you a valid confidence interval at the desired level (i.e. to perform a test at level $\alpha$ you need a two-sided level $1-\alpha$ confidence interval) and see whether it lies completely within the equivalence boundaries.

E.g. you might want to work with odds-ratios and might think that anything within a factor of 0.8 to 1.25 (equal delta in both directions on the log-odds-ratio scale) is not a meaningful difference. You could then use logistic regression to get and estimated odds ratio (0.809 for group 1 vs. 2) and to get 95% asymptotic Wald confidence intervals (0.542 to 1.206). Since the confidence interval is not completely within 0.8 to 1.25 (i.e. the lower end falls out of it), you would not have shown equivalence (i.e. you cannot reject the null hypothesis of non-equivalence). With other equivalence margins, you might have equivalence.

Often we can also derive a p-value for the decision, but if all you want is "reject null hypothesis" vs. "null hypothesis not rejected", then all you need is confidence interval. When asymptotics apply, you can use the normal approximation to get such a p-value (essentially looking at what level confidence interval would just overlap with the margin, which is easy to do, if the CI is formed via estimated log-odds ratio +- SE * appropriate percentile of the normal distribution such as 1.96), while with more sparse data this can be a bit difficult.

You could of course also decide that you are not interested in an odds ratio and want a risk difference or a risk ratio.

example <- data.frame(group = c(rep(1,164), rep(2,280)),
                      outcome = c(rep(1,58), rep(0,164-58), rep(1,113), rep(0,280-113)))

glmfit1 <- glm(data = example,
               formula = outcome ~ factor(group),
               family = "binomial")

summary(glmfit1)

# Get estimate and standard error
# use negative coefficient because of choice of reference group
estimate <- -summary(glmfit1)$coefficients[2,1] 
se <- summary(glmfit1)$coefficients[2,2]

# Wald confidence interval on the odds-ratio scale, you get a test decision simply 
# by comparing your NI margin to the CI limits (if the CI limits extend
# beyond it, you cannot reject the null hypothesis of non-equivalence).
# Here we are assuming that you want 95% CIs corresponding to a 5% level test
wald_ci <- c(exp( estimate + se* qnorm(0.975) ),
             exp( estimate - se* qnorm(0.975) ))

# If you really need a p-value, then you can do the following:

# Let us assume this is your equivalence margin:
nimargin <- log(2) 
# p-value using large sample normality assumption (corresponding to Wald confidence limits)
pvalue <- min(1, 2*(1-pnorm( (abs(nimargin) - abs(estimate))/se, 0, 1)))

The example code above gives you a p-value for equivalence of 0.0183, if your NI margin is log(2) on the logit-scale. So for that NI margin you could reject the null hypothesis of non-equivalence and conclude that within those limits the groups are equivalent. On the other hand for a NI margin of 1.25, you would get a p-value from the equivalence test of 0.9579, so you could not reject that null hypothesis.

Related Question