I would like to study elements of representation theory as I often encounter it when reading texts on harmonic analysis. I was therefore curious if someone could recommend a book for this.
When looking into the books that have "representation theory" as their title, it seems there is a variety of books written by people from different fields. For example, it seems there is almost no overlap in the content of Naimark's "Theory of Group representations", and the books by Serre and Fulton & Harris on representation theory.
I was therefore wondering which "type of representation theory" is used in harmonic analysis, and how it relates to other "types of representation theory" such as the one studied in abstract algebra.
Any advice or explanation is highly appreciated.
Best Answer
As you've seen, "representation theory" is a broad subject, and means different things to different people.
Varadarajan's delightful book "An introduction to harmonic analysis on semi-simple Lie groups" is one of the most readable books on "harmonic analysis" in the repn-theoretic sense, while taking analytic issues seriously. A. Deitmar has a readable book on harmonic analysis, and Echterhoff-Deitmar. Many people have on-line notes about the more-analytic, harmonic-analysis aspects of "repn theory". (E.g., see my various course notes, at http://www.math.umn.edu/~garrett/m/repns/, the latter trying to make a transition from finite-group repn theory to more genuinely analytical issues, and also applications to modular forms and number theory, at http://www.math.umn.edu/~garrett/m/mfms/)
The classics by Serre and Fulton-Harris are almost entirely aimed at finite-dimensional repns of finite groups, or of finite-dimensional Lie algebras. The issues there are very different from the issues of infinite-dimensional repns of not-finite, especially non-abelian, non-compact groups on Hilbert spaces and other topological vector spaces. The latter context was first investigated for reasons coming from physics, where unitary repns on (infinite-dimensional) Hilbert spaces have significance. Bargmann and Wigner did the earliest examples, and then Gelfand-Naimark et al. (Their invocation of asymptotics of special functions appearing as solutions of ODEs was an antecedent of Harish-Chandra's subquotient theorem, and then Casselman's subrepn theorem.)
But by now the utility of repn theory of all sorts in the theory of automorphic forms, number theory is well known. But, equally, the extremes of the "sort" of repn theory are wildly different from each other. Indeed, it has started to seem to me that "repn theory" is not a reasonable name for the vast collection of things that might fall under that label, since it is hilariously non-descriptive.
The courses I've given under the name "repn theory" have invariably displeased at least half the audience on every occasion, by being either "too algebraic" or "too analytic"... or both. Some people want Lie algebras and their finite-dimensional repns, but not Lie groups, no infinite-dimensional repns, and no analysis whatsoever.
One reason that the situation is invariably confusing for beginners is that, for example, Fulton-Harris do not explain very strongly that they are ignoring infinite-dimensional repns. Of course, any author would spend a crazy amount of time if they were obliged to explain in detail all the things they're not doing. (!) Still, it leaves a beginner wondering.
From the opposite end, "harmonic analysis" sometimes (e.g., E. Stein's book of that title) has very little to do with group repns, dealing perhaps exclusively with the abelian groups $\mathbb R^n$ and products of circles. ... or ragged chunks of the group $\mathbb R^n$!?! :) I think that this use of "harmonic analysis" is so extreme that it is not connected to repn theory any more than the rest of mathematics may be.
The contemporary repn theory of finite groups has no analytical issues, naturally, but in effect is always in the "bad" situation, that the characteristic of the field on whose vector spaces the (finite) group acts divides the order of the group. Thus, Maschke's theorem tends to fail, and the homological algebra of the situation is much more complicated than "direct sum". But/and the various non-vanishing (co-)homologies are "new" repn spaces.
There is also a failure of "semi-simplicity" in the "harmonic analysis" version of repn theory, e.g., principal series repns of nice groups like $SL_2(\mathbb R)$ can be reducible, but do not decompose as direct sums. Analogous to the finite-group case, this means that some derived functors will be non-zero, and thus give "new" repn spaces. Borel-Wallach's book pursues this.
Another example is the "oscillator repn", or "Segal-Shale-Weil repn", which began as a Lie algebra repn, then a real Lie group repn, then abstracted considerably by Weil. Discussion of this is made less crazy in the context of various incarnations of the Stone-vonNeumann theorem (roughly asserting the uniqueness of models of quantum thingies) about repns of the Heisenberg group. The finite-field analogues of these things are relatively elementary, since there's no analysis, but/and the real Lie and p-adic cases require some non-trivial analysis to have something more than a mere heuristic. (My notes mentioned above do some of this.)