Chi-Squared-Test – How to Perform Chi-Square Independence Test for A/B Split Testing?

ab-testchi-squared-test

I have the results of an A/B split test of a web page.

Page  Impressions   Clicks    CTR       
A     56,000        10        0.018%    
B     78,000        21        0.027%

I'd like to test the ctr for statistically significant difference. Null hypothesis – there is no difference in CTR between pages A & B.

How would I go about doing this on a more theoretical level? As in what should be rows, column, setting expected values?

Best Answer

Here's one way to lay out the chi-squared test -- you don't necessarily need all this detail. The test doesn't care which way around you have rows and columns, so I'll do it your way around.

Comparing proportions via Pearson's Chi-squared test of independence
at the 5% level      #(for this example, at least - you choose your own level)

Null hypothesis - there is no difference in CTR between pages A & B.

Observed:
   Clicks   Nonclicks     Impressions 
A     10       55990           56000
B     21       77979           78000

Tot   31      133969          134000

Expected:
   Clicks   Nonclicks     Impressions 
A   12.96    55987.04          56000
B   18.04    77981.96          78000

Tot   31      133969          134000

(Entries in the body of the Expected table are row.total*column.total/overall.total)

Contribution to chi-square = (Observed - Expected)^2/Expected

       Clicks   Nonclicks   
 A     0.6741   0.0001560
 B     0.4840   0.0001120

Chi-square = sum((Observed - Expected)^2/Expected)
           = 1.158368

 df = 1
 p-value = 0.2818  

 At the 5% level we do not reject H0 - there's no evidence of a difference in
 click through rate.

(You need tables or a program to find the p-value.)

Related Question