Whilst I understand why median can be a better metric than mean in certain circumstances (e.g. you're creating/marketing a product, you want to use the median as it represents the biggest age group within your population and therefore real potential buyers, whereas mean age might be a number which doesn't correspond to anybody, or a very few people) I am wondering whether median is a good metric when it comes to measuring website performance.
Let's look at the below data representing page load time (in sec, the smaller the better):
Sample 1 and Sample 2 are identical, except for C4, meaning the website is only faster for 1 person. Yet, the median changes significantly between the 2 samples (nearly 40% improvement) whereas the mean is much more stable (less than 10% difference).
Accordingly my take on this is that mean is more accurate are representing the page load time.
Do you agree? / Is my observation biased because of the particular samples I have chosen or would mean be more representative than median no matter the samples?
Best Answer
You performed your analysis with a sample of only 10 data items. I recommend you try it again, but using a sample which contains a few 100 data items. Or try it again with another sample of 10 data items, but find a class of problem where more than just a single data item differs from one sample iteration to the next. What you will likely find (this is the usual reason why people often think that median makes a more reliable estimator than the mean) is that the outliers (i.e., the 8's and 9's in your sample) often vary wildly between samples.
Suppose you had one user who had a value of 25 for his website speed on one day, and a more "typical" value of 4 or 5 on some other day. That kind of thing happens all the time in real life. Think about your commute to work, as another example—most days it probably takes almost exactly the same amount of time every day, but every once in a while there is a giant traffic jam because of an accident on the road, or the train is delayed for a large and random amount of time because another train ahead of you experienced a mechanical problem or someone accidentally fell onto the tracks a few stops in front of you, and the whole system grinds to a halt during rush hour while the authorities deal with the problem.
Outliers in distributions like these, which represent a process that has gone "wrong" in some unpredictable way on one particular occasion, tend to be large and variable in size and frequency, and they usually have a much larger effect on the calculation of the mean than the median. Thus, although you've detected a counterexample that works in this particular case, in general, the median usually offers a more robust estimator (in the sense of being more stable from one sample to the next) than the mean.