I promised Sean a detailed answer, so here it is.
As José has already mentioned, it is only $G_2$ (of the five exceptional Lie groups) which can arise as the holonomy group of a Riemannian manifold. Berger's classification in the 1950's could not rule it out, and neither could he rule out the Lie group $\mathrm{Spin}(7)$, but these were generally believed to not possibly be able to exist. However, in the early 1980's Robert Bryant succeeded in proving the existence of local examples (on open balls in Euclidean spaces). Then in the late 1980's Bryant and Simon Salamon found the first complete, non-compact examples of such manifolds, on total spaces of certain vector bundles, using symmetry (cohomogeneity one) methods. (Since then there are many examples of non-compact cohomogeneity one $G_2$ manifolds found by physicists.) Finally, in 1994 Dominic Joyce stunned the mathematical community by proving the existence of hundreds of compact examples. His proof is non-constructive, using hard analysis involving the existence and uniqueness of solutions to a non-linear elliptic equation, much as Yau's solution of the Calabi conjecture gives a non-constructive proof of the existence and uniqueness of Calabi-Yau metrics (holonomy $\mathrm{SU}(n)$ metrics) on Kahler manifolds satisfying certain conditions. (In 2000 Alexei Kovalev found a new construction of compact $G_2$ manifolds that produced several hundred more non-explicit examples. These are the only two known compact constructions to date.) It is exactly this similarity to Calabi-Yau manifolds (and to Kahler manifolds in general) that I will explain.
When it comes to Riemannian holonomy, the aspect of the group $G_2$ which is important is not really that it is one of the five exceptional Lie groups, but rather that it is the automorphism group of the octonions $\mathbb O$, an $8$-dimensional non-associative real division algebra. The octonions come equipped with a positive definite inner product, and the span of the identity element $1$ is called the real octonions while its orthogonal complement is called the imaginary octonions $\mathrm{Im} \mathbb O \cong \mathbb R^7$. This is entirely analogous to the quaternions $\mathbb H$, except that the non-associativity introduces some new complications. In fact the analogy allows us to define a cross product on $\mathbb R^7$ in the same way, as follows. Let $u, v \in \mathbb R^7 \cong \mathrm{Im} \mathbb O$ and define $u \times v = \mathrm{Im}(uv)$, where $uv$ denotes the octonion product. (In fact the real part of $uv$ is equal to $-\langle u, v \rangle$, just as it is for quaternions.) This cross product satisfies the following relations:
\begin{equation}
u \times v = - v \times u, \qquad \qquad \langle u \times v , u \rangle = 0, \qquad \qquad {|| u\times v||}^2 = {|| u \wedge v ||}^2,
\end{equation}
exactly like the cross product on $\mathbb R^3 \cong \mathrm{Im} \mathbb H$. However, there is a difference, unlike the cross product in $\mathbb R^3$, the following expression is not zero:
\begin{equation}
u \times (v \times w) + \langle u, v \rangle w - \langle u, w \rangle v
\end{equation}
but is instead a measure of the failure of the associativity $(uv)w - u(vw)$, up to a factor. Note that on $\mathbb R^7$ there can be defined a $3$-form (totally skew-symmetric trilinear form) using the cross product as follows: $\varphi(u,v,w) = \langle u \times v, w \rangle$, which is called the associative $3$-form for reasons that we won't get into here.
Digression: In fact one can show that only on $\mathbb R^3$ and $\mathbb R^7$ can one construct such a cross product, and this is intimately related to the fact that only the spheres $S^2$ and $S^6$ can admit almost complex structures. But I digress...
Getting back to $G_2$ geometry: a $7$-dimensional smooth manifold $M$ is said to admit a $G_2$-structure if there is a reduction of the structure group of its frame bundle from $\mathrm{GL}(7, \mathbb R)$ to the group $G_2$ which can actually be viewed naturally as a subgroup of $\mathrm{SO}(7)$. For those familiar with $G$-structures, this tells you that a $G_2$-structure determines a Riemannian metric and an orientation. In fact, one can show on a manifold with $G_2$-structure, there exists a non-degenerate $3$-form $\varphi$ for which, given a point $p$ on $M$, there exists local coordinates near $p$ such that, in those coordinates, at the point $p$, the form $\varphi$ is exactly the associative $3$-form on $\mathbb R^7$ discussed above. Now one can show that there is a way to canonically determine both a metric and an orientation in a highly non-linear way from this $3$-form $\varphi$. Then one can define a cross product $\times$ by essentially using the metric to ``raise an index'' on $\varphi$. In summary, a manifold $(M, \varphi)$ with $G_2$-structure comes equipped with a metric, cross product, $3$-form, and orientation, which satisfy
\begin{equation}
\varphi(u,v,w) = \langle u \times v , w \rangle.
\end{equation}
This is exactly analogous to the data of an almost Hermitian manifold, which comes with a metric, an almost complex structure $J$, a $2$-form $\omega$, and an orientation, which satisfy
\begin{equation}
\omega(u,v) = \langle Ju , v \rangle.
\end{equation}
Essentially, a manifold admits a $G_2$-structure if one can identify each of its tangent spaces with the imaginary octonions $\mathrm{Im} \mathbb O \cong \mathbb R^7$ in a smoothly varying way, just as an almost Hermitian manifold is one in which we can identify each of its tangent spaces with $\mathbb C^m$ (together with its Euclidean inner product) in a smoothly varying way.
For a manifold to admit a $G_2$-structure, the necessary and sufficient conditions are that it be orientable and spin. (This is equivalent to the vanishing of the first two Stiefel-Whitney classes.) So there are lots of such $7$-manifolds, just as there are lots of almost Hermitian manifolds. But the story does not end there.
Let $(M, \varphi)$ be a manifold with $G_2$-structure. Since it determines a Riemannian metric $g_{\varphi}$, there is an induced Levi-Civita covariant derivative $\nabla$, and one can ask if $\nabla \varphi = 0$? If this is the case, $(M, \varphi)$ is called a $G_2$-manifold, and one can show that the Riemannian holonomy of $g_{\varphi}$ is contained in the group $G_2 \subset \mathrm{SO}(7)$. These are much harder to find, because it involves solving a fully non-linear partial differential equation for the unknown $3$-form $\varphi$. They are in some ways analogous to Kahler manifolds, which are exactly those almost Hermitian manifolds that satisfy $\nabla \omega = 0$, but those are much easier to find. One reason is because the metric $g$ and the almost complex structure $J$ on an almost Hermitian manifold are essentially independent (they just have to satisfy the mild condition of compatibility) whereas in the $G_2$ case, the metric and the cross product are determined non-linearly from $\varphi$. However, the analogy is not perfect, because one can show that when $\nabla \varphi = 0$, the Ricci curvature of $g_{\varphi}$ necessarily vanishes. So $G_2$-manifolds are always Ricci-flat! (This is one reason that physicists are interested in such manifolds---they play a role as ``compactifications'' in $11$-dimensional $M$-theory analogous to the role of Calabi-Yau $3$-folds in $10$-dimensional string theory.) So in some sense $G_2$-manifolds are more like Ricci-flat Kahler manifolds, which are the Calabi-Yau manifolds.
In fact, if we allow the holonomy to be a proper subgroup of $G_2$, there are many examples of $G_2$-manifolds. For example, the flat torus $T^7$, or the product manifolds $T^3 \times CY2$ and $S^1 \times CY3$, where $CYn$ is a Calabi-Yau $n$-fold, have Riemannian holonomy groups properly contained in $G_2$. These are in some sense ``trivial'' examples because they reduce to lower-dimension constructions. The manifolds with full holonomy $G_2$ are also called irreducible $G_2$-manifolds and it is precisely these manifolds that Bryant, Bryant-Salamon, Joyce, and Kovalev constructed.
We are lacking a ``Calabi-type conjecture'' which would give necessary and sufficient conditions for a compact $7$-manifold which admits $G_2$-structures to admit a $G_2$-structure which is parallel ($\nabla \varphi = 0$.) Indeed, we don't even know what the conjecture should be. There are topological obstructions which are known, but we are far from knowing sufficient conditions. In fact, this question is more similar to the following: suppose $M^{2n}$ is a compact, smooth, $2n$-dimensional manifold that admits almost complex structures. What are necessary and sufficient conditions for $M$ to admit Kahler metrics? We certainly know many necessary topological conditions, but (as far as I know, and correct me if I am wrong) we are nowhere near knowing sufficient conditions.
What makes the Calabi conjecture tractable (I almost said easy, of course it is anything but easy) is the fact that we already start with a Kahler manifold (holonomy $\mathrm{U}(m)$ metric) and want to reduce the holonomy by only $1$ dimension, to $\mathrm{SU}(m)$. Then the $\partial \bar \partial$-lemma in Kahler geometry allows us to reduce the Calabi conjecture to a (albeit fully non-linear) elliptic PDE for a single scalar function. Any analogous ``conjecture'' in either the Kahler or the $G_2$ cases would have to involve a system of PDEs, which are much more difficult to deal with.
That's my not-so-short crash course in $G_2$-geometry. I hope some people read all the way to the end of this...
Best Answer
This doesn't directly address your question, but it does give you a way of thinking about torsion in the cohomology of Lie groups in general.
(This is all coming from Borel and Serre's Sur certains sous-groupes des groupes de Lie, which can be found in Commentarii mathematici Helvetici Volume 27, 1953)
As you mentioned above, every compact lie group is rationally a product of odd spheres. But how many odd spheres? Turns out, if G is compact and rank k, then it is rationally a product of k spheres (of various dimensions).
There is an analogous result for torsion. That is, one can define the 2-group of G to be any subgroup which is isomorphic to $(\mathbb{Z}/2\mathbb{Z})^n$ or some n. One defines the 2-rank of a group as the maximal $n$ of any 2-group in G. (On can show that for connected $G$, the 2-rank is bounded by twice the rank, and is thus finite).
Just to point out something that really threw me when I first learned of these - while the rank is an invariant of the algebra (i.e., all Lie groups with the same algebra have the same rank), the 2-rank of a Group is NOT an invariant of the algebra. For example, the 2-rank of SU(2) is 1 (in fact, -Id is the UNIQUE element of SU(2) of order 2), while the 2-rank of SO(3) is 2 (generated by diag(-1,-1,1) and diag(-1,1,-1) ). The 2-rank of O(3) is 3 (generated by diag(-1,1,1), diag(1,-1,1), and diag(1,1,-1) ).
Now, given $T\subseteq G$, the maximal torus, it's clear that simply by taking the maximal 2-group in T, that the 2-rank of G is AT LEAST the rank of G. When is it strictly bigger? Precisely when the group G contains 2-torsion.
The analogous result for p-groups and p-torsion (p any prime) also holds.
In short, to understand the existence of the 5-torsion in $E_{8}$, one need only understand why there is a subgroup isomorphic to $(\mathbb{Z}/5\mathbb{Z})^n\subseteq E_8$ for some $n\geq 9$.