Solved – Measuring Statistically Significant Lift in Weekly Sales (Using T-Test)

confidence intervalstatistical significancet-test

I am trying to measure a statistically significant lift in sales and want to confirm appropriate methodology

Test Design

  1. Created "Matched Markets" of 20 stores (10 in control vs. 10 in test)

Source Data (represented as total sales during test period):

Control Test

 100   185
 125   180
 135   170
 130   172
 140   190
 170   177
 165   165
 160   197
 162   185
 134   196

Methodology

I am running a one-tailed t-test since I do not know the SD of the population and we are hypothesizing that Test is greater than control. Based on this, I am calculating a p-value of .00003 with the result is significant at p < .05.

I want to confirm a t-test is appropriate here and I am not violating any assumptions around populations (since the stores in test vs. control are in different states) or anything else.

I also wanted to confirm whether you are ALSO able to declare significance since the average of the test group (181.7) is greater than the Confidence Interval of control at 95% (calculated to be (142.1 ± 14)

Best Answer

Overview

Your instinct to use a t-test is ok, but maybe a non-parametric equivalent would be better. More importantly, your data and experimental design might not match the conclusions you want to draw.

Paired vs Independent

From what you've said, it seems like the stores are independent, but I'm not sure if you've run a paired or independent t-test. If you have 20 stores that you measured once, then you'll need to run an independent sample t-test.

Assumptions for a t-test

It looks like your data are continuous, but your sample size is too low to really conclude that your data are normally distributed. To play it safe, I'd advocate the use of a non-parametric version of a t-test.

Non-parametric t-tests

For an independent t-test equivalent, check out the Mann-Whitney U test, which is available in R, python, SPSS and JASP. This is the most conservative analysis you could do under these circumstances. When I ran your numbers, I found a p<0.001, which looks good!

Two Tailed vs One Tailed

Although you could theoretically run a one-tailed t-test, it's generally thought that these are not conservative enough (and in fact that p-values themselves don't do enough to eliminate alpha error). My advice is to use a two-tailed assumption. My p-value above was using a two-tailed assumption, so you'll be just fine.

Interpretation

From what you've presented, it's hard to conclude that there was a statistically significant "lift in sales". You've said that the stores are in separate states (which is why you needed to run an independent t-test), but how were the stores doing before you started?

Instead of total sales, you should take the difference between sales during the 'test' period and subtract the baseline from them. Then run an independent t-test on the "difference" between baseline and test periods.

Related Question