I am having trouble calculating the statistical significance of a split test based on revenue. Most online calculators only look at the number of visitors and the number of conversions, outputting a p-value (and thus measuring the significance). I want to take this to the next level and measure if a test is significant based on revenue, not just the number of purchases.
Here is an example calculator using the "traditional" method: http://visualwebsiteoptimizer.com/ab-split-significance-calculator/
Here' an example to try to illustrate this a bit better:
Pretend you have baseball card store, and you are testing a new version of your website. Version A is the control, Version B is the variation.
With normal split testing, you would measure the number of people who visit both websites, and the number of purchases.
But what if you had more data? Instead of just tracking wether or not a purchase was made, what if you knew how much each customer spent, and what version of the website they saw? Over the course of a month you track purchases from customers depending on the variation they saw. At the end of the month, you want to know – with 95%+ confidence – if the new version of the website resulted in more revenue, not just the number of sales.
Is this possible to calculate in excel? Here's some sample data to use:
Version A
2000 visitors, 500 sales: $
123
115
103
177
188
120
114
189
134
80
126
142
144
108
141
139
114
119
181
82
154
114
128
166
163
116
112
180
122
159
109
185
126
178
40
161
101
175
184
150
180
174
159
132
166
154
125
151
126
157
111
210
152
151
130
107
118
105
113
162
118
163
142
124
132
157
128
101
106
167
147
148
158
160
156
168
177
149
155
830
113
100
136
152
176
171
129
158
189
188
156
187
178
78
130
130
171
163
120
164
128
168
145
110
186
189
149
140
142
134
145
189
187
161
164
129
104
167
166
186
142
169
137
60
126
157
174
183
176
166
136
190
170
109
141
125
187
105
178
147
113
137
141
106
60
167
153
155
174
141
117
154
140
118
118
177
187
149
159
151
109
111
115
149
182
138
168
115
171
167
123
178
104
120
121
126
176
154
182
164
139
106
162
139
138
138
166
137
152
182
172
106
120
162
107
117
103
114
162
168
160
122
161
160
138
159
124
179
124
112
138
170
146
137
153
122
163
167
156
150
186
127
153
187
141
187
143
172
184
104
121
104
176
145
102
156
143
167
139
137
125
101
109
183
100
110
127
106
119
179
131
116
166
169
108
141
189
143
109
107
124
108
146
147
158
155
183
140
101
100
111
107
151
177
130
147
186
156
117
162
187
139
131
177
147
146
111
112
142
107
141
164
157
159
169
148
190
165
172
181
190
111
175
190
103
127
132
173
101
178
101
161
160
144
123
189
121
182
147
141
166
144
180
126
161
141
116
152
130
164
147
163
177
126
163
167
135
164
116
104
169
126
175
179
185
137
130
139
131
139
147
123
124
172
130
100
189
111
186
105
106
147
136
122
171
165
151
123
126
132
169
106
158
120
165
144
120
182
131
174
107
127
165
105
139
127
161
103
114
143
184
101
105
185
153
148
142
162
187
167
154
149
158
135
180
179
182
139
104
159
164
149
168
187
131
107
175
111
189
162
133
131
180
101
141
153
139
171
175
124
119
102
130
152
121
159
122
124
123
188
104
130
184
109
126
114
188
100
139
179
187
104
165
169
152
169
127
152
165
177
103
187
167
157
141
150
120
180
101
129
104
157
104
161
120
162
139
110
182
150
105
170
121
160
133
151
116
190
153
189
101
171
184
179
153
188
173
141
164
100
Version B
2,200 visitors, 510 sales: $
144
143
159
194
90
158
149
145
197
192
139
132
467
185
164
159
166
190
159
170
168
132
157
134
151
183
183
180
135
320
128
185
141
184
181
126
126
167
134
120
163
138
345
150
199
156
173
194
183
186
360
165
183
149
143
156
167
174
177
430
258
175
143
183
122
145
184
1029
170
183
126
146
165
193
157
154
199
192
145
199
129
156
157
173
125
127
182
164
159
182
147
126
160
184
193
133
154
127
183
125
189
151
192
182
178
169
189
132
157
142
310
147
158
164
196
155
135
167
146
167
123
172
198
129
135
176
131
120
124
166
157
280
149
165
146
158
128
199
147
198
149
153
157
154
133
195
198
385
192
195
123
158
199
146
163
183
164
173
130
187
168
123
196
120
176
179
197
150
173
161
150
187
193
195
192
164
130
194
180
124
142
180
152
125
186
181
166
181
187
159
127
125
130
179
134
143
148
154
139
161
162
161
147
133
160
143
126
127
130
160
133
156
168
194
164
120
130
168
188
127
141
127
171
150
162
189
163
177
160
145
197
176
134
142
124
137
166
197
188
191
193
178
121
162
134
175
199
146
157
147
127
151
159
140
125
163
197
187
165
155
137
163
139
136
181
151
148
176
178
153
162
151
164
199
147
161
148
185
132
141
157
134
158
145
150
127
140
165
138
163
182
176
173
173
184
196
174
121
137
121
159
145
139
145
191
152
178
163
171
198
175
145
192
126
159
165
128
161
129
134
192
136
175
132
176
144
159
147
131
138
188
134
132
193
183
166
176
170
136
146
139
151
197
173
161
121
194
150
179
120
174
196
139
137
140
148
161
195
157
174
197
193
156
156
183
168
192
146
170
120
145
151
134
173
182
172
167
134
145
132
182
178
158
156
168
170
167
195
159
181
177
126
152
142
134
131
176
139
190
162
195
139
193
172
169
153
184
172
167
154
147
124
141
156
169
146
142
168
147
163
143
154
175
179
175
127
157
129
190
189
175
159
146
183
183
164
154
167
162
156
145
151
181
121
140
145
158
183
185
166
122
136
189
121
191
136
166
192
134
195
153
168
146
133
144
123
162
181
125
181
196
182
165
196
146
168
138
146
144
179
152
149
134
195
158
135
129
189
177
142
137
199
144
193
170
122
184
194
130
146
213
394
128
394
231
495
201
120
90
50
Best Answer
For revenue it is a bit difficult. There is a similar question to yours here https://stats.stackexchange.com/questions/72242/bootstrapping-do-i-need-to-remove-outliers-first
If you have large enough samples the CLT lets you just use the mean, $\mu$ and the standard error on the mean, $\sigma_\mu = \sigma / \sqrt{N}$. You can then calculate your 95% CL through the comparison of the two Gaussians formed by $\mu$ and $\sigma_\mu$ for each scenario in your split test.
This is a very stripped down answer and there are a lot of special cases to be considered. However, this is a fast way to get your answer in excel.