Before addressing this, it's important to acknowledge that the statistical malpractice of "removing outliers" has been wrongly promulgated in much of the applied statistical pedagogy. Traditionally, outliers are defined as high leverage, high influence observations. One can and should identify such observations in the analysis of data, but those conditions alone do not warrant removing those observations. A "true outlier" is a high leverage/high influence observation that's inconsistent with replications of the experimental design. To deem an observation as such requires specialized knowledge of that population and the science behind the "data generating mechanism". The most important aspect is that you should be able to identify potential outliers apriori.
As for the bootstrapping aspect of things, the bootstrap is meant to simulate independent, repeated draws from the sampling population. If you prespecify exclusion criteria in your analysis plan, you should still leave excluded values in the referent bootstrap sampling distribution. This is because you will account for the loss of power due to applying exclusions after sampling your data. However, if there are no prespecified exclusion criteria and outliers are removed using post hoc adjudication, as I'm obviously rallying against, removing these values will propagate the same errors in inference that are caused by removing outliers.
Consider a study on wealth and happiness in an unstratified simple random sample of 100 people. If we took the statement, "1% of the population holds 90% of the world's wealth" literally, then we would observe, on average, one very highly influential value. Suppose further that, beyond affording a basic quality of life, there was no excess happiness attributable to larger income (nonconstant linear trend). So this individual is also high leverage.
The least squares regression coefficient fit on unadulterated data estimates a population averaged first order trend in these data. It is heavily attenuated by our 1 individual in the sample whose happiness is consistent with those near median income levels. If we remove this individual, the least squares regression slope is much larger, but the variance of the regressor is reduced, hence inference about the association is approximately the same. The difficulty with doing this is that I did not prespecify conditions in which individuals would be excluded. If another researcher replicated this study design, they would sample an average of one high income, moderately happy individual, and obtain results that were inconsistent with my "trimmed" results.
If we were apriori interested in the moderate income happiness association, then we should have prespecified that we would, e.g. "compare individuals earning less than $100,000 annual household income". So removing the outlier causes us to estimate an association we cannot describe, hence the p-values are meaningless.
On the other hand, miscalibrated medical equipment and facetious self-reported survey lies can be removed. The more accurately that exclusion criteria can be described before the actual analysis takes place, the more valid and consistent the results that such an analysis will produce.
As I understand you have a 2X2 experimental design ( Factor1 - website(levels: a,b), factor2 - visitors group (levels: a,b)) and the dependent variable 'Revenue'.
I would consider ANOVA being careful to all assumptions behind. The way you measure/code dependent variable 'Revenue' has different implications.
This movie from Andy Field might be useful.
Regards,
Marius
Best Answer
Bootstrapping for a mean does not usually make sense for due to the CLT. Just use the mean and standard error on the mean. This will either give the same result as your bootstrap or your bootstrap would have given a poor result.
If you truly know you want to compare the mean (and it is not clear that you do Better estimator of expected sum than mean) then you want to test if the two samples are likely to have come from the same population. This would be a Welch t-test with the mean, $\mu$, and standard error on the mean, $\sigma_\mu$.
The problem becomes more complicated if there is a different associated risk with each choice coming from other factors (eg implementation cost). If the t-test says there is no significant difference then you clearly want to take the variant with lower risk. However, if the variant with higher risk has a statistically significant impact on mean revenue then it is a judgment call.