Solved – Two-tailed tests… I’m just not convinced. What’s the point

hypothesis testinginferencestatistical significancetwo-tailed-test

The following excerpt is from the entry, What are the differences between one-tailed and two-tailed tests?, on UCLA's statistics help site.

… consider the consequences of missing an effect in the other direction. Imagine you have developed a new drug that you believe is an improvement over an existing drug. You wish to maximize your ability to detect the improvement, so you opt for a one-tailed test. In doing so, you fail to test for the possibility that the new drug is less effective than the existing drug.

After learning the absolute basics of hypothesis testing and getting to the part about one vs two tailed tests… I understand the basic math and increased detection ability of one tailed tests, etc… But I just can't wrap around my head around one thing… What's the point? I'm really failing to understand why you should split your alpha between the two extremes when your is sample result can only be in one or the other, or neither.

Take the example scenario from the quoted text above. How could you possibly "fail to test" for a result in the opposite direction? You have your sample mean. You have your population mean. Simple arithmetic tells you which is higher. What is there to test, or fail to test, in the opposite direction? What's stopping you just starting from scratch with the opposite hypothesis if you clearly see that the sample mean is way off in the other direction?

Another quote from the same page:

Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how "close" to significant the two-tailed test was.

I assume this also applies to switching the polarity of your one-tailed test. But how is this "doctored" result any less valid than if you had simply chosen the correct one-tailed test in the first place?

Clearly I am missing a big part of the picture here. It all just seems too arbitrary. Which it is, I guess, in the sense that what denotes "statistically significant" – 95%, 99%, 99.9%… Is arbitrary to begin with.

Best Answer

Think of the data as the tip of the iceberg – all you can see above the water is the tip of the iceberg but in reality you are interested in learning something about the entire iceberg.

Statisticians, data scientists and others working with data are careful to not let what they see above the water line influence and bias their assessment of what's hidden below the water line. For this reason, in a hypothesis testing situation, they tend to formulate their null and alternative hypotheses before they see the tip of the iceberg, based on their expectations (or lack thereof) of what might happen if they could view the iceberg in its entirety.

Looking at the data to formulate your hypotheses is a poor practice and should be avoided – it's like putting the cart before the horse. Recall that the data come from a single sample selected (hopefully using a random selection mechanism) from the target population/universe of interest. The sample has its own idiosyncracies, which may or may not be reflective of the underlying population. Why would you want your hypotheses to reflect a narrow slice of the population instead of the entire population?

Another way to think about this is that, every time you select a sample from your target population (using a random selection mechanism), the sample will yield different data. If you use the data (which you shouldn't!!!) to guide your specification of the null and alternative hypotheses, your hypotheses will be all over the map, essentially driven by the idiosyncratic features of each sample. Of course, in practice we only draw one sample, but it would be a very disquieting thought to know that if someone else performed the same study with a different sample of the same size, they would have to change their hypotheses to reflect the realities of their sample.

One of my graduate school professors used to have a very wise saying: "We don't care about the sample, except that it tells us something about the population". We want to formulate our hypotheses to learn something about the target population, not about the one sample we happened to select from that population.

My way of explaining calculating p-values & the tails

I start with a explaining that if we believe that we have a fair coin we know it should end up tails 50 % of the flips on average ($=H_0$). Now if you wonder what the probability of getting only 2 tails out of 10 flips with this fair coin you can calculate that probability as I've done in the bar graph. From the graph you can see that the probability of getting 8 out of 10 flips with a fair coin is about about $\approx 4.4\%$.

Since we would question the fairness of the coin if we got 9 or 10 tails we have to include these possibilities, the tail of the test. By adding the values we get that the probability now is a little more than $\approx 5.5\%$ of getting 2 tails or less.

Now if we would get only 2 heads, ie 8 heads (the other tail), we would probably be just as willing to question the fairness of the coin. This means that you end up with a probability of $5.4...\%+5.4...\% \approx 10.9\%$ for a two-tailed test.

Since we in medicine usually are interested in studying failures we need to include the opposite side of the probability even if our intent is to do good and to introduce a beneficial treatment.

My flipping coins graph

Reflections slightly out of topic

This simple example also shows how dependent we are on the null hypothesis to calculate the p-value. I also like to point out the resemblance between the binomial curve and the bell curve. When changing into 200 flips you get a natural way of explaining why the probability of getting exactly 100 flips starts to lack relevance. The defining intervals of interest is a natural transition to probability density/mass function functions and their cumulative counterparts.

In my class I recommend them the Khan academy statistics videos and I also use some of his explanations for certain concepts. They also get to flip coins where we look into the randomness of the coin flipping - the thing that I try to show is that randomness is more random than what we usually believe inspired by this Radiolab episode.

The code

I usually have one graph/slide, the R-code that I used to create the graph:

library(graphics)

binom_plot_function <- function(x_max, my_title = FALSE, my_prob = .5, edges = 0, 
                                col=c("green", "gold", "red")){
  barplot(
    dbinom(0:x_max, x_max, my_prob)*100, 
    col=c(rep(col[1], edges), rep(col[2], x_max-2*edges+1), rep(col[3], edges)),
    #names=0:x_max,
    ylab="Probability %",
    xlab="Number of tails", names.arg=0:x_max)
  if (my_title != FALSE ){
    title(main=my_title)
  }
}

binom_plot_function(10, paste("Flipping coins", 10, "times"), edges=0, col=c("#449944", "gold", "#994444"))
binom_plot_function(10, edges=3, col=c(rgb(200/255, 0, 0), "gold", "gold"))
binom_plot_function(10, edges=3, col=c(rgb(200/255, 0, 0), "gold", rgb(200/255, 100/255, 100/255)))

Solved – Interpreting one- and two-tailed tests

You don't choose a one-tailed test based on near-significance in a two-tailed test.

You don't choose the direction of a one-tailed test based on directional information from the data.

Or at the least, if you do those things, you must also double the resulting p-value.

A one tailed test - if you do one at all - must be based on prior considerations, in place before you know what is in the data. If this is not the case, the significance levels (and p-values) are meaningless.

Best Answer

Related Solutions

Hypothesis Testing – Understanding Two-Tailed Tests in Depth

My way of explaining calculating p-values & the tails

Reflections slightly out of topic

The code

Solved – Interpreting one- and two-tailed tests

Related Question