Does anyone know if the Gumbel can occur as a limit distribution for such a sum?
When we have $n$ exponential distributed variables $X_i \sim Exp(\gamma = i)$, (with expectation $1/i$ and variance $1/i^2$) then the sum
$$S = \sum_{i=1}^n (X_i - 1/i)$$
approaches a Gumbel distribution.
There is a connection between this sum and the maximum order statistic.
We can see this sum as the waiting time for filling $n$ bins when the filling of the bins is a Poisson process.
- Approach with the sum. The waiting time between the filling of bins bin is exponential distributed. For waiting until one bin is filled, since all bins are empty the rate is $n$. The waiting time for a second bin to be filled is when $n-1$ bins are empty and the rate will be $n-1$, and so on...
- Approach with the maximum. We can consider the waiting times for filling each individual bin. The waiting time to fill all bins is equal to the maximum of the individual waiting times.
The distribution of the maximum of exponential distributed variables approaches a Gumbel distribution. Therefore the expression in terms of a sum, which has an equal distribution, will also approach the Gumbel distribution.
See also Intuition about the coupon collector problem approaching a Gumbel distribution on Cross Validated.
This is of course not general.
If we use $X_i = N(\mu = 1/i, \sigma^2 = 1/i^2)$ then a (properly scaled) sum will approach a normal distribution.
That is a trivial example but there are more cases that will converge to a normal distribution. The relevant condition that needs to be fulfilled is the Lyapunov condition.
That the covariance matrix (say $\Sigma$) is degenerate just means that the corresponding random vector $X$ (say in $\mathbb{R}^k$) lies almost surely in a proper linear subspace $V$ of $\mathbb{R}^k$, say of dimension $r<k$. Replace the random vector $X$ by the $r$-tuple (say $Y$) of the coordinates of $X$ in an arbitrary basis $B$ of $V$. Then the covariance matrix of $Y$ will be nonsingular, and so, the multivariate CLT will be applicable -- to iid copies $Y^{(1)},\dots,Y^{(n)}$ of the vector $Y$ in $\mathbb{R}^r$.
The subspace $V$ can be described as the orthogonal complement to $\mathbb{R}^k$ of the null space of $\Sigma$. For the basis $B$ you can take any set of orthonormal eigenvectors (say $e_1,\dots,e_r$) of $\Sigma$ corresponding to the nonzero eigenvalues (say $\lambda_1,\dots,\lambda_r$) of $\Sigma$. Then $X=\sum_{j=1}^r Y_je_j$ and $Y=(Y_1,\dots,Y_r)$, with $Y_j=\langle e_j,X\rangle$, where $\langle\cdot,\cdot\rangle$ denotes the inner product in $\mathbb{R}^k$. The (nonsingular) covariance matrix of $Y$ will then be the diagonal matrix with $\lambda_1,\dots,\lambda_r$ on its diagonal.
Best Answer
The comments list many reasons why the Gaussian distribution is special, but is it "the most fundamental" among all distributions, as suggested in the OP? I would like to argue that (1) conservation laws are among the most fundamental laws of Nature, and (2) a quantity that obeys a conservation law will naturally follow an exponential -- rather than a Gaussian distribution.
Consider energy: two systems with energies $E_1$ and $E_2$ have total energy $E_1+E_2$, and if they are sufficiently large they will be independent, so the probability distribution must factorize: $P(E_1+E_2)=P(E_1)P(E_2)$, with the exponential distribution $P(E)\propto e^{-\beta E}$ as the unique normalizable solution (assuming $E$ is bounded from below).
This is the Gibbs measure. The Hammersley–Clifford theorem states that any probability measure which satisfies a Markov property is a Gibbs measure for an appropriate choice of (locally defined) energy function. The Gibbs measure is the fundamental measure of statistical physics, but it also applies widely outside of physics.
Economics is one such example. The statistical mechanics of money explains the exponential distribution of money as a direct consequence of the fact that money is conserved in general (there are exceptions). Wealth, in contrast, is not conserved (stocks may rise or fall), hence the non-exponential wealth distribution (Pareto distribution).