I am interested in the irreducible unitary representations of the orthogonal groups $O(p,q)$. By $O(p,q)$ I mean the real Lie groups which preserve the quadratic form of signature $(p,q)$ in $\mathbb{R}^n$, $n = p+q$ dimensions. Special cases of interest in physics are the conformal group O(4,2), the deSitter group O(4,1) and the anti-deSitter group O(3,2) in dimension 4 = 3+1 (i.e. Minkowski space). I am only interested in the non-compact groups, the compact cases being well-understood. (So I expect that the irreducible unitary representations are all infinite dimensional.) I am not exclusively interested in Lorentzian signature $n = (n-1) + 1$, nor am I exclusively interested in $n=4$. As a theoretical physicist, I am not familiar with the undoubtedly vast literature on representations of non-compact Lie groups, and I would appreciate a few pointers to the most relevant references; those that review the general setting, but especially which address these groups specifically.
Lie Groups – Unitary Irreps of O(p,q)
lie-groupsrt.representation-theory
Related Solutions
I strongly recommend you read the article "Representations of semisimple Lie groups" by Knapp and Trapa in the park city/ias proceedings "Representation theory of Lie groups". It's a very nice introduction to the problem of describing the "unitary dual" (which is what you are asking about) which focusses on SL(2,R). For example, page 9 says "the irreducible unitary representations that appear in L^2(G) do not nearly exhaust the unitary dual" for general semisimple Lie groups (thus answering you question 2). For more info, you can check out knapp's book "Representation theory of semisimple groups: an overview based on examples". For example, sections II.4 and II.5 describe the unitary duals of SL(2,C) and SL(2,R) respectively. The unitary duals of GL(n,C) and GL(n,R) were described by Vogan. Some other unitary duals are known, but in general, I don't think anything else is known. One approach is via Langlands' parametrization of irreducible admissible representations of reductive groups. This result is known for all groups and unitary representations are admissible, so the problem would be to identify which admissible representations are unitary (the knapp-trapa article talks about this). As for 3), every irreducible unitary representation of a compact group is finite-dimensional, so you don't get any of the infinite dimensional representations you get for SL(2,R). I don't know what you mean by 4).
For a full answer to 1) you can check out link text.
I'm not quite sure if this is the answer that you looking for but anyway he we go. For a locally compact group you are going to generally want to look at strongly continuous representation. By this is mean endow $B(H)$, the bounded operators on a hilbert space $H$ with the topology of point-wise norm convergence. And only consider reps $\pi:G\rightarrow B(H)$ that are continuous with this topology. Now such a rep is unitary if, for every $g\in G$, $\pi(g)$ is a unitary operator.
Now the notion of "occur in" that you mention seems to be the notion of strong containment. We say that $\rho:G\rightarrow B(K)$ is strongly contained in $\pi:G\rightarrow B(H)$ if there is a $G$-equivarient unitary operator from $K$ to a closed subspace of $H$.
So it now seems that you are asking when does the left regular rep ($L^2(G)$) strongly contain all irreducibles. So yes for compact Lie groups this follows from Peter-weyl this is true.
However as soon as you go to something non-compact this might not be true.
In fact, there is a much weaker notion known as weak containment of representation. and it is known that $L^2(G)$ weakly contains all irreducible reps if and only if $G$ is amenable.
Non-compact Lie groups are in general not amenable, (any groups which contains $\mathbb{F}_2$ the free group on 2 generators is non-amenable)
There is much more to be said about this but I think that this should suffice for now
Best Answer
The unitary dual of Spin(n,1) is known for all n (Hirai, 1962, see Math Reviews MR0696689). This gives the unitary dual of the identity component of SO(n,1) (which is a quotient of Spin(n,1)). The unitary dual of any group can be deduced readily from that of its identity component. Also SL(2,C)=Spin(3,1), and SL(2,R)=Spin(2,1).
The unitary dual of Sp(4,R) is known (Nzoukoudi, 1983, MR0736241), and Sp(4,R)=Spin(3,2). The unitary dual of SU(2,2) is known (Knapp/Speh, 1982, MR0645645), and SU(2,2)=Spin(4,2). These give the unitary duals of SO(3,2) and SO(4,2).
Besides this (and the compact groups) I think the entire unitary dual is not known for any O(p,q), although for any fixed group a large part of its unitary dual is known. You might look at some papers by Welleda Baldoni and Tony Knapp from the 1980s or so.
Note: much of the literature applies to groups of "Harish-Chandra's class". This includes all SO(p,q), but O(p,q) only if p+q is odd. So if p+q is even you have to do a little extra work to get from the unitary dual of SO(p,q) to that of O(p,q) (for each irreducible unitary representation of SO(p,q) you have to decide if its induction to O(p,q) has 1 or 2 irreducible summands).
As for www.liegroups.org, we hope to have the complete answer for any group, but that day has not yet arrived.