I am chemist and ask for apologies for all my mathematical inabilities when asking this question in advance, but after quite a bit of searching I found that this problem could be "open" or at least hard enough to find addressed in the literature and also advanced enough that it's possibly suitable to be asked here.
I work over a subset of the finite groups called point groups these are all the ((essentially) finite) subgroups of $O(3)$. The "degeneracies" arising in those groups are of importance in Chemistry so I started to investigate them. With "degeneracy" the dimensions of the irreps over $\mathbb R$, (this is of essential importance) occurring for the group under consideration is meant. Whenever a group is represented by an irrep of dimension $n>1$ we speak about $n$-fold degeneracy.
The term degeneracy in this context relates to the fact that the quantum mechanical eigenstates of (symmetric) systems/molecules form such $n$-dimensional sub spaces of the Hilbert space. Since the Hamilton operator is self-adjunct it makes sense to regard representations over $\mathbb R$ instead of over the usually more elegant $\mathbb C$.
My first question is, under which conditions in terms of group elements degeneracy can occur in a group? And the second question is how is the maximal dimension $n_{\max}$ of real irreps, over all irreps of the group, determined by the structure of the group?
My primary observation is that groups that contain exactly one generator $y$ of order $m=3$, like a group $\langle x,y,…| x^2=y^m = 1 = …\rangle $ have $n_{\max}=2$
Then there are just a very few point groups with $n_{\max} > 2$. We call them "high symmetry groups". Basically its the symmetry groups of the tetrahedron, the octahedron and the icosahedron (with 2 or 3 certain subgroups of theirs), where the former two have $n_{\max}=3$, the latter $n_{\max}=5$.
These high symmetry groups all have two generators, the tetrahedral groups one of order $m=3$, the octahedral groups one of order $m=4$ and the icosahedral groups one of order $5$.
So I would assume that there is a connection between the order of the generators and the degeneracies that can occur in a group. What it is exactly remains very obscure to me. So I would be very grateful about any hints also to the literature.
Edit
Since the audience is so fantastically knowledgeable, I can't resist to make a small comment on the motivation of my research in the hope it might ring some bell and give rise to more inspiring comments:
The motivation of my question is, that you can see certain interesting physical properties in the states (of physical systems) if they are degenerate. "States" are some manifestations of irreps where we have direct numerical access to and a good intuition about their visual representations. It happens that there are certain, it seems, deep connections between the angular momentum operator, which is essentially an infinitesimal rotation in the (physical space) $\mathbb R^3$, and the occurrence of degeneracy (at least if it's 2-fold). At the same time we see that these states which are instances of degenerate representations are transformed into each others by (finite) rotations. Such that the question was arising if all such degeneracies are related to rotations, or if there is at least something in the structure of rotations that is general in $d>1$ dimensional representations.
There was recently a result that suggested that there is a "hidden" anti-unitary symmetry (state transforming operator) at the basis of any $2$-fold degeneracy of the form
$$ \mathcal{O} = i \begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix} \mathcal{K}$$ (with $\mathcal{K}$ as complex conjugation operator) that transforms between the two degenerate irreps. Such an operator can be easily constructed from the angular momentum operator, for example, but not only, constructions of operators like {\mathcal{O}} based on spin or time reversal and others are possible if one considers more general "parametrizations" of physical systems then only $\mathbb R^3$. Anyway, as far as symmetries of the real space $\mathbb R^3$ are concerned it seems that rotation (and infintesimal rotation) are crucial and I aim to understand what "crucial" here exactly means.
What I find in particular intriguing in the light of the answer from @QiaochuYuan is the connection with "non-Abelianess", because the defining relation of $J$, the angular momentum or also spin operator is the so called "angular momentum algebra", that is a commutator relation between its components
$$ [J_i,J_j] = i \varepsilon_{ijk} J_k$$
(with the Levi-Civita symbol $\varepsilon$). So this seems to suggest that angular momentum is an essential source of non-commutativity somehow. I like then to understand what types of sources else there might be, for this non-commutativity in quantum mechanical systems, especially if only representations of states in $\mathbf{R}^3$ are considered. One thing which complicates the question is that in most systems rotational symmetry "is broken" but degeneracy can nevertheless occur. Then my question would be, what is the explicit form of the $\mathcal{O}$ operator and if it can be continuously related to angular momentum if one views the "symmetry breaking" as a continuous process.
(Sorry for the post-answer edit, I hope it complies with the MO rules!)
Best Answer
Your question touches on many issues in group representation theory, and I can only give a few general remarks which may point you in interesting directions for further reading.
As to your question regarding the maximal real irreducible representation of a finite group, there is an interesting connection with the Frobenius Schur indicator.
If $\chi$ is a (complex) irreducible character of a finite group $G$, the Frobenius-Schur indicator of $\chi$ is denoted by $\nu(\chi)$ defined to be $0$ if $\chi$ is not real-valued, to be $-1$ if $\chi$ is real-valued, but $\chi$ may NOT be afforded by a representation over $\mathbb{R}$, and to be $1$ if $\chi$ is afforded by a representation over $\mathbb{R}.$ For example, the unique irreducible complex character of degree $2$ of the quaternion group of order $8$ has Frobenius-Schur indicator $-1$, and the unique irreducible character of degree $2$ of the dihedral group of order $8$ ( I mean the one with $8$ elements) has Frobenius-Schur indicator $1$.
The number of solutions of $x^{2}=1 $ in the finite group $G$ is equal to $\sum_{\chi} \nu(\chi) \chi(1)$, where $\chi$ runs over the complex irreducible characters of $G$.
This is especially useful if all irreducible characters $\chi$ of $G$ have $\nu(\chi) = 1$, which is always the case for $G = S_{n}$ (the symmetric group).
The FS-indicator may (in principle at least) be calculated via the formula $\nu(\chi) = \frac{1}{|G|} \sum_{g \in G} \chi(g^{2}).$
In the case of the alternating group of degree $5$, for example, all irreducible characters $\chi$ have $\nu(\chi) = 1$, the irreducible characters have degree $1,3,3,4,5$. Hence we get $\sum_{\chi} \nu(\chi)\chi(1) = 16$, and there are indeed $16$ solutions of $x^{2} = 1$ in $G$ (the identity and fifteen elements of order $2$).
As to the question of what you term degeneracy, there is some ambiguity (related to the Frobenius-Schur indicator and also the Schur index). For example, the quaternion group of order $8$ has a $4$-dimensional representation which is irreducible as a real representation, but which is equivalent to the sum of two equivalent $2$-dimensional complex irreducible representations. An absolutely irreducible real representation is a real irreducible representation which remains irreducible as a complex representation. This is a representation whose character $\chi$ is irreducible as a complex character and has $\nu(\chi) = 1.$
A real irreducible representation which is not absolutely irreducible is one which is not irreducible as a complex representation. Such a representation may afford a character of the form $2\chi$ where $\chi$ is a complex irreducible character with $\nu(\chi) = -1$, or it may afford a character of the form $\chi + \overline{\chi}$, where $\chi$ is a complex irreducible character with $\nu(\chi) = 0$ (ie $\chi$ is not real-valued).
In terms of complex irreducible representations, it is one of the earliest theorems in group theory (due to C. Jordan) that if a finite group $G$ has a complex representation of degree $n$ (irreducible or not), then $G$ has an Abelian normal subgroup whose index is bounded in terms of $n$. This also applies to real irreducible representations.
If we restrict to complex irreducible representations which are primitive (that is, can not be induced from a representation of a proper subgroup), this tells us that if $G$ has a primitive complex irreducible representation of degree $n$, then the number of possibilities for $G/Z(G)$ is bounded in terms of $n$.
On the other hand, the symmetric group $S_{n+1}$ always has a real irreducible representation of degree $n$, and has order $(n+1)!$, yet has no non-identity Abelian normal subgroup if $n >3.$ This is related to the "generic" worst case bound for Jordan's Theorem, and is genuinely an upper bound for that Theorem if $n$ is large enough.
I think that in general, it is very difficult to relate the order of generators of a finite group $G$ with the largest degree of its real (or complex) irreducible representations. For example, there are arbitrarily large finite simple groups $G$ which may be generated by an element of order $2$ and an element of order $3$, and there is therefore no upper bound on the dimensions of the real irreducible representations of finte groups which may be generated by an element of order $2$ and an element of order $3$.
Later edit: Another general fact which is often useful, is a result of N. Ito, which states that if the finite group $G$ has an Abelian normal subgroup $A$, then the degree of any complex irreducible representation of $G$ is a divisor of the index $[G:A].$