So, working in web analytics, it's common to use a chi-square test for A/B tests to test statistical significance. I often use a calculator like this: http://www.usereffect.com/split-test-calculator , so if each variation gets 1000 impressions, and A has 100 conversions and B has 200 conversions, I know that, with a 99.9% confidence interval (assuming randomized samples) that B is the winning variation.
But occasionally, I have values that create percentages over 100%, and seem to throw these calculations off.
So, if I'm testing two donation pages, and I want to use the $ values for conversions, how would I go about that? My goal is to make a statistically informed decision about which page is performing better, in terms of its ability to raise more money.
If I have 1000 impressions of each variation, and one variation raises $2000 off of 200 contributions, and the othe other raises $2500 off of 100 contributions, what calculation can I use to know if the difference is meaningful?
Best Answer
I assume you have data on the amount of each donation, not just the two totals for each variation. You don't want to use Pearson's chi-square test; that's only appropriate for counts or proportions, not for amounts. You want an independent (unpaired) two-sample test for equality of location. The classic such 'parametric' test is Student's t-test. As it's quite plausible that the variances differ as well as the means, so I'd suggest you use Welch's t-test, which is a slight variation that allows for unequal variances.
You might also consider 'non-parametric' ('distribution-free') tests, of which the obvious one is the Mann–Whitney U test, but as your outcome of interest is clearly the difference in mean donation and your sample size is reasonably large, you're almost certainly better off sticking with parametric tests such as the above t-tests that use the mean as the location parameter and so directly address the null hypothesis of equality of means of the two samples.
By the way, don't forget to include the people who saw either variation of the donation page but didn't donate anything as donations of value zero.