I have a large sample of values, bounded in $[0, 1]$, divided into two conditions A and B, and I want to test the significance of one condition A having higher value than condition B.
For the details, these values are bootstrap scores for a given relationship (A or B) of species taxa in many different phylogenetic gene trees (i.e. A = taxon1 is sister to taxon2 VS B = taxon1 is sister to taxon3). I would like to set aside the fact that actual phylogenetic tests exist for this kind of question and come back to the case of a t-test for proportion values.
I feel that a t-test (Welch) for independent samples would not be valid, even if the sample is very large, because values are proportions (note: I don't have the original counts).
Indeed, on this forum many people suggested (here, here or here) not using T-test but something more appropriate like a generalized linear model, which was my intuition as well.
However, I could not answer: which assumption of either the Central Limit Theorem or of the Welch's T-test is violated by my dataset?
First, I reasoned by translating the T-test problem into the (I believe equivalent) linear regression problem, in which we would clearly violate the assumption of identically normally distributed residuals. My problem is that I still don't find what is violated in the T-test formulation. How are the assumptions related between these two formulations?
Indeed, it seems that with the Central Limit Theorem, the mean of many percentages would follow a normal distribution, thus the condition that the mean is normally distributed would hold; so the violated assumption has to be another one in the T-test. I saw that the $T$ variable
$$ T = \frac{Z}{\sqrt{U/k}} $$
requires that $Z$ and $U$ be independent to follow the Student distribution. Is it the assumption that is violated by a dataset of proportion values?
Best Answer
The answer is that, contrary to popular belief, the Welch (not Welsh) T-test kind of assumes equal variance of populations. Now, before you go crazy and think you were lied to, let me explain. A T-test only holds when the following are true:
Welch's T-test applies a correction to the regular t-test that makes violations of the first assumption practically unimportant when condition 1 is fulfilled. However, the Welch t-test, in theory, is still not exact unless condition 1 is fulfilled or the sample size is very large.
In theory, the Welch and regular T-tests have their assumption of equal variances violated when dealing with a binomial distribution, because a binomial distribution has a standard error equal to: $$\frac{\sqrt{p (1 - p)}}{\sqrt n}$$
Notice that this variance depends on $p$. Therefore, any change in $p$ technically invalidates any t-test, so a t-test can't be used.
But what if $n$ is really big? Well then, the variance hardly changes, because the variance is close to 0, and going from "Basically 0" to "Basically 0" is a change of, well, basically 0. So the fact that the variance changes a little is fine, because Welch's t-test is insensitive to small differences in variance (unlike the regular t-test).