This is a heuristic question that I think was once asked by Serge Lang. The gaussian: $e^{-x^2}$ appears as the fixed point to the Fourier transform, in the punchline to the central limit theorem, as the solution to the heat equation, in a very nice proof of the Atiyah-Singer index theorem etc. Is this an artifact of the techniques (such as the Fourier Transform) that people like use to deal with certain problems or is this the tip of some deeper platonic iceberg?
Probability – Why is the Gaussian So Pervasive in Mathematics?
gm.general-mathematicspr.probability
Related Solutions
The bivariate distribution formed by two independent normalized Gaussians is rotationally symmetric (think about the usual argument for evaluating the probability integral). The quotient of two random variables $X$ and $Y$ is the tangent of the angle between $(0,0)$ and $(X,Y)$ with the $x$-axis. If one has a rotationally symmetric distribution for $X$ and $Y$ (with no point mass at the origin) then $Y/X$ is a tangent of a uniformly distributed angle. This is the Cauchy distribution.
Added Your example with the Brownian motion states in effect that if $P$ is the first point that the motion hits the $x$-axis then the angle between the line from $P$ to the starting point and the $y$-axis is uniformly distributed between $-\pi$ and $\pi$. I can't see any reason why this should be so, but perhaps someone (unlike me) who actually knows something about Brownian motion might know why.
Let
\begin{equation}
\mu_X=\tfrac12\,\mu_{aZ}+\tfrac12\,\mu_{bZ},
\end{equation}
where $\mu_U$ denotes the probability distribution of a random vector $U$, $Z\sim N(0,I_n)$,
and $a,b$ are constants such that
\begin{equation}
0<a<1<b\quad\text{and}\quad \tfrac12\,a^2+\tfrac12\,b^2=1.
\end{equation}
Then $EXX^T=I_n$. Also, for any unit vector $u$ and real $s>0$ \begin{equation} E\exp\{\left<X,u\right>^2/s^2\}=\frac1{2\sqrt{1-2a^2/s^2}}+\frac1{2\sqrt{1-2b^2/s^2}}<2 \end{equation} if $s$ is large enough (depending only on $a,b$), so that, by the definition of $\|\cdot\|_{\psi_2}\|$, we have $\|\left<X,u\right>\|_{\psi_2}\le s$. For instance, here we can take $a=1/5,b=7/5,s=3$.
On the other hand, for \begin{equation} t:=(b-1)\sqrt{n}/2, \end{equation} \begin{multline} 2\,Ee^{(\|X\|-\sqrt n)^2/t^2}>Ee^{(\|bZ\|-\sqrt n)^2/t^2} >Ee^{(\|bZ\|-\sqrt n)^2/t^2}1_{\|Z\|^2>n} \\ >e^{(b\sqrt n-\sqrt n)^2/t^2}\,P(\|Z\|^2>n)=e^4\,P(\|Z\|^2>n)\to e^4/2>4, \end{multline} because, by the central limit theorem, $P(\|Z\|^2>n)\to1/2$. So, for all large enough $n$, \begin{equation} \|\|X\|-\sqrt n\|_{\psi_2}\ge t=(b-1)\sqrt{n}/2\to\infty, \end{equation} as desired.
Best Answer
Quadratic (or bilinear) forms appear naturally throughout mathematics, for instance via inner product structures, or via dualisation of a linear transformation, or via Taylor expansion around the linearisation of a nonlinear operator. The Laplace-Beltrami operator and similar second-order operators can be viewed as differential quadratic forms, for instance.
A Gaussian is basically the multiplicative or exponentiated version of a quadratic form, so it is quite natural that it comes up in multiplicative contexts, especially on spaces (such as Euclidean space) in which a natural bilinear or quadratic structure is already present.
Perhaps the one minor miracle, though, is that the Fourier transform of a Gaussian is again a Gaussian, although once one realises that the Fourier kernel is also an exponentiated bilinear form, this is not so surprising. But it does amplify the previous paragraph: thanks to Fourier duality, Gaussians not only come up in the context of spatial multiplication, but also frequency multiplication (e.g. convolutions, and hence CLT, or heat kernels).
One can also take an adelic viewpoint. When studying non-archimedean fields such as the p-adics $Q_p$, compact subgroups such as $Z_p$ play a pivotal role. On the reals, it seems the natural analogue of these compact subgroups are the Gaussians (cf. Tate's thesis). One can sort of justify the existence and central role of Gaussians on the grounds that the real number system "needs" something like the compact subgroups that its non-archimedean siblings enjoy, though this doesn't fully explain why Gaussians would then be exponentiated quadratic in nature.