On page 50, line 2 of the High Dimensional Probability book (HDP-book.pdf), the author asserts that the expected angle between 2 random directions is $\pi/4$, however, I keep ending up with $\pi/2$. Here is how I proceed:
The cumulative density function (CDF) of the angle $\theta$ is $$F_{\theta}(\alpha) = \Pr[\theta \leq \alpha] = 2 \cdot \frac{\alpha}{2\pi} = \frac{\alpha}{\pi}, \,\, \alpha \in [0,\pi]$$ The factor of $2$ is because two distinct arrangements of random vectors have the same angle between them (e.g., vectors oriented at $60^{\circ}$ and $300^{\circ}$ w.r.t each other have angle $60^{\circ}$ between them).
Hence, the probability density function (PDF) $f_{\theta}$ is $$f_{\theta}(\alpha) = \frac{1}{\pi}, \,\, \alpha \in [0, \pi] $$ Using this, the expected angle becomes $$\mathbb{E}[\theta] = \int_{0}^{\pi}\alpha f_{\theta}(\alpha) \, d\alpha = \frac{\pi}{2}$$
This also seems to agree with an existing post (Distribution of angle between random vectors). Could there be a different interpretation of the statement in the book, which could lead to $\mathbb{E}[\theta]=\pi/4$?
Best Answer
The interpretation in the book differs from yours in several ways.
The book starts by taking an expected value of the square of an inner product, not an expected angle. $$ \mathbb E \langle X,Y \rangle^2 = n. \tag1$$ The book then (implicitly) treats that expected value as the likely value of the square of the inner product of two vectors: $$ \langle X,Y \rangle^2 \asymp n. \tag2$$ The book then takes the square root on both sides: $$ \lvert \langle X,Y \rangle \rvert \asymp \sqrt n. \tag3$$ It then supposes that normalizing the vectors shrinks the result by a factor of $n$: $$ \lvert \langle \hat X,\hat Y \rangle \rvert \asymp \frac{1}{\sqrt n}. \tag4$$ As a footnote in the book points out, this is a non-rigorous argument.
Notice that the left-hand side of Equation $(4)$ is never negative, so if we consider it to be the cosine of the angle between the vectors, that angle is never greater than $\pi/2.$ You might say that this is not actually measuring the angle between the vectors themselves but rather the smaller of the two angles between the lines in which those vectors lie.
Actually, there are a whole bunch of hand-wavy steps here, but it turns out that if you measure "the angle" between two vectors by taking the supplement of the actual angle whenever the actual angle is obtuse, then indeed the "average" angle in two dimensions is $\pi/4$.
I find this a strange way of measuring an angle; but if you suppose that we designate the "north pole" on the unit sphere as whatever point $X$ points toward, and let $P$ be the point on the unit sphere that $Y$ points to, then what we are measuring here is not the distance of $P$ from the north pole but rather the complement of the distance of $P$ from the equator. And you can reasonably say that if you take the unit sphere embedded in a Euclidean space of $n$ dimensions, the typical distance from the equator of a random point is $\pi/2 - \pi/4 = \pi/4$ in two dimensions and approximately $\pi/2 - \pi/2 = 0$ when the number of dimensions is very large.