Suppose I have a mixture of finitely many Gaussians with known weights, means, and standard deviations. The means are not equal. The mean and standard deviation of the mixture can be calculated, of course, since the moments are weighted averages of the moments of the components. The mixture is not a normal distribution, but how far from normal is it?
The above image shows the probability densities for a Gaussian mixture with component means separated by $2$ standard deviations (of the components) and a single Gaussian with the same mean and variance.
Here the means are separated by $1$ standard deviation and it is harder to separate the mixture from the Gaussian by eye.
Motivation: I disagree with some lazy people about some actual distributions they haven't measured which they assume are close to normal because that would be nice. I'm lazy, too. I don't want to measure the distributions, either. I want to be able to say their assumptions are inconsistent, because they are saying that a finite mixture of Gaussians with different means is a Gaussian which isn't right. I don't just want to say that the asymptotic shape of the tail is wrong because these are just approximations which are only supposed to be reasonably accurate within a few standard deviations of the mean. I'd like to be able to say that if the components are well-approximated by normal distributions then the mixture isn't, and I'd like to be able to quantify this.
I don't know the right distance from normality to use: supremum of differences between CDFs, $L^1$ distance, earth-mover's distance, KL divergence, etc. I'd be happy to get bounds in terms of any of these, or other measures. I would be happy to know the distance to the Gaussian with the same mean and standard deviation as the mixture, or the minimum distance with any Gaussian. If it helps, you may restrict to the case that the mixture is of $2$ Gaussians so that the smaller weight is greater than $1/4$.
Best Answer
KL divergence would be natural because you have a natural base distribution, the single Gaussian, from which your mixture diverges. On the the other hand, the KL divergence (or its symmetrical 'distance' form) between two Gaussian mixtures, of which your problem is a special case, seems to be intractable in general. Hershey and Olson (2007) looks like a reasonable summary of the available approximations, including variational methods that may possibly offer easier bounds.
However, if you want to have an argument about the ill-effects of assuming something is Gaussian when it's really a mixture then it's best to have a good idea about the consequences you're actually interested in - something more specific than simply 'being wrong' (this is @Michael-Chernick's point). For example, the consequences for a test, or an interval, or somesuch. Two obvious effects of the mixture are overdispersion, which is pretty much guaranteed, and multimodality, which will confuse maximizers.