There is certainly an abundance of advanced books on Galois representations and automorphic forms. What I'm wondering is more simple: What is the basic connection between modular forms and representation theory?
I have a basic grounding in the complex analytic theory of modular forms (their dimension formulas, how they classify isomorphism classes of elliptic curves, some basic examples of level N modular forms and their relation to torsion points on elliptic curves, series expansions, theta functions, Hecke operators). This is all with an undergraduate background in complex analysis and algebra (Galois theory). I also know a little bit about the basics of algebraic number theory and algebraic geometry, if that helps. More importantly, I have a basic background in the representation theory of finite groups. My question is, then, could one example how modular forms and/or theta functions relate to representations of groups?
I'm asking this in part because I imagine a number of students with similar background as I have would have learned about modular forms and thus might be interested to understand how they relate to representation theory, despite not having an extensive background in more advanced results in algebraic geometry and commutative algebra needed for advanced study in the field.
Here are some ideas which might bear fruit: In analytic number theory, one often sees sums over characters – but characters are also very relevant in representation theory. In particular, Jacobi sums come up in both number theory and representation theory (and quadratic forms then relate to theta functions). Is there a connection here?
In addition, Hecke operators are symmetric-like sums over elements of groups, which would suggest a strong connection to representation theory.
Or is the connection to representations of $\mathrm{SL}_2(\mathbb{Z})$? Quotients of this group appear as Galois groups of extensions of spaces of modular forms, so they might be given representations by acting on these spaces?
The point of listing ideas is to show the kind of intuition I might be looking for. One of my ideas might be fruitful, or they all might have nothing to do with why representation theory connects to modular functions. The point is that I'm looking for basic ideas that someone with an elementary background might be able to understand.
I also added "reference request" because I imagine there might be a text which is at my level and discusses these ideas.
EDIT: The answer of paul garett here actually gives a nice history of how modular forms came to be viewed in terms of representations theory: What is the difference between an automorphic form and a modular form?
Best Answer
Caveat: in order to give you an overview, I've been vague/sloppy in several places.
Well the basic link to representation theory is that modular forms (and automorphic forms) can be viewed as functions in representation spaces of reductive groups. What I mean is the following: take for example a modular form, i.e. a function $f$ on the upper-half plane satisfying certain conditions. Since the upper-half plane is a quotient of $G=\mathrm{GL}(2,\mathbf{R})$, you can pull $f$ back to a function on $G$ (technically you massage it a bit, but this is the main idea) which will be invariant under a discrete subgroup $\Gamma$. Functions that look like this are called automorphic forms on $G$. The space all automorphic forms on $G$ is a representation of $G$ (via the right regular represenation, i.e. $(gf)(x)=f(xg)$). Basically, any irreducible subrepresentation of the space of automorphic forms is what is called an automorphic representation of $G$. So, modular forms can be viewed as certain vectors in certain (generally infinite-dimensional) representations of $G$. In this context, one can define the Hecke algebra of $G$ as the complex-valued $C^\infty$ functions on $G$ with compact support viewed as a ring under convolution. This is a substitute for the group ring that occurs in the representation theory of finite groups, i.e. the (possibly infinite-dimensional) group representations of $G$ should correspond to the (possibly infinite-dimensional) algebra representations of its Hecke algebra. This type of stuff is the basic connection of modular forms to representation theory and it goes back at least to Gelfand–Graev–Piatestkii-Shapiro's Representation theory and automorphic functions. You can replace $G$ with a general reductive group.
To get to more advanced stuff, you need to start viewing modular forms not just as functions on $\mathrm{GL}(2,\mathbf{R})$ but rather on $\mathrm{GL}(2,\mathbf{A})$, where $\mathbf{A}$ are the adeles of $\mathbf{Q}$. This is a "restricted direct product" of $\mathrm{GL}(2,\mathbf{R})$ and $\mathrm{GL}(2,\mathbf{Q}_p)$ for all primes $p$. Again you can define a Hecke algebra. It will break up into a "restricted tensor product" of the local Hecke algebras as $H=\otimes_v^\prime H_v$ where $v$ runs over all primes $p$ and $\infty$ ($\infty$ is the infinite prime and corresponds to $\mathbf{R}$). For a prime $p$, $H_p$ is the space of locally constant compact support complex-valued functions on the double-coset space $K\backslash\mathrm{GL}(2,\mathbf{Q}_p)/K$ where $K$ is the maximal compact subgroup $\mathrm{GL}(2,\mathbf{Z}_p)$. If you take something like the characteristic function of the double coset $KA_pK$ where $A_p$ is the matrix with $p$ and $1$ down the diagonal, and look at how to acts on a modular form you'll see that this is the Hecke operator $T_p$.
Then there's the connection with number theory. This is mostly encompassed under the phrase "Langlands program" and is a significantly more complicated beast than the above stuff. At least part of this started with Langlands classification of the admissible representation of real reductive groups. He noticed that he could phrase the parametrization of the admissible representations say of $\mathrm{GL}(n,\mathbf{R})$ in a way that made sense for $\mathrm{GL}(n,\mathbf{Q}_p)$. This sets up a (conjectural, though known now for $\mathrm{GL}(n)$) correspondence between admissible representations of $\mathrm{GL}(n,\mathbf{Q}_p)$ and certain $n$-dimensional representations of a group that's related to the absolute Galois group of $\mathbf{Q}_p$ (the Weil–Deligne group). This is called the Local Langlands Correspondence. The Global Langlands Correspondence is that a similar kind of relation holds between automorphic representations of $\mathrm{GL}(n,\mathbf{A})$ and $n$-dimensional representations of some group related to Galois group (the conjectural Langlands group). These correspondences should be nice in that things that happen on one side should correspond to things happening on the other. This fits into another part of the Langlands program which is the functoriality conjectures (really the correspondences are special cases). Basically, if you have two reductive groups $G$ and $H$ and a certain type of map from one to the other, then you should be able to transfer automorphic representations from one to the other. From this view point, the algebraic geometry side of the picture enters simply as the source for proving instances of the Langlands conjectures. Pretty much the only way to take an automorphic representation and prove that it has an associated Galois representation is to construct a geometric object whose cohomology has both an action of the Hecke algebra and the Galois group and decompose it into pieces and pick out the one you want.
As for suggestions on what to read, I found Gelbart's book Automorphic forms on adele groups pretty readable. This will get you through some of what I've written in the first two paragraphs for the group $\mathrm{GL}(2)$. The most comprehensive reference is the Corvallis proceedings available freely at ams.org. To get into the Langlands program there's the book an introduction to the Langlands program (google books) you could look at. It's really a vast subject and I didn't learn from any one or few sources. But hopefully what I've written has helped you out a bit. I think I need to go to bed now. G'night.