This is maybe not an entirely mathematical question, but consider it a pedagogical question about representation theory if you want to avoid physics-y questions on MO.
I've been reading Singer's Linearity, Symmetry, and Prediction in the Hydrogen Atom and am trying to come to terms with the main physical (as opposed to mathematical) argument of the text. The argument posits, if I understand it correctly, that a quantum system described by a Hilbert space $H$ on which a group $G$ of symmetries acts by unitary transformations should have the property that its "elementary states" "are" irreducible subrepresentations of the representation of $G$ on $H$. She begins this argument in section 5.1:
Invariant subspaces are the only physically natural subspaces. Recall from Section 4.5 that in a quantum system with symmetry, there is a natural representation $(G, V, \rho)$. Any physically natural object must appear the same to all observers. In particular, if a subspace has physical significance, all equivalent observers must agree on the question of a particular state's membership in that subspace.
and continues it in section 6.3:
We know from numerous experiments that every quantum system has *elementary states*. An elementary state of a quantum system should be **observer-independent**. In other words, any observer should be able (in theory) to recognize that state experimentally, and the observations should all agree. Secondly, an elementary state should be indivisible. That is, one should not be able to think of the elementary state as a superposition of two or more "more elementary" states. If we accept the model that every recognizable state corresponds to a vector subspace of the state space of the system, then we can conclude that elementary states correspond to irreducible representations. The independence of the choice of observer compels the subspace to be invariant under the representation. The indivisible nature of the subspace requires the subspace to be irreducible. So elementary states correspond to irreducible representations. More specifically, if a vector $w$ represents an elementary state, then $w$ should lie in an *irreducible* invariant subspace $W$, that is, a subspace whose only invariant subspaces are itself and $0$. In fact, every vector in $W$ represents a state "indistinguishable" from $w$, as a consequence of Exercise 6.6.
(For people who actually know their quantum, Singer is ignoring the distinction between representations and projective representations until later in the book.)
My first problem with this argument is that Singer never gives a precise definition of "elementary state." My second problem is that I'm not sure what physical principle is at work when she posits that physically natural subspaces and elementary states should be observer-independent (i.e. invariant under the action of $G$). What underlying assumption of quantum mechanics, or whatever, is at work here? Why should a mathematician without significant training in physics find this reasonable? (I have the same question about the identification of elementary particles with irreducible representations of the "symmetry group of the universe," so any comments about this physical argument are also welcome.)
Singer goes on to use this assumption to deduce the number of electrons that fill various electron orbitals, and I won't be able to convince myself that this makes sense until I understand the physical assumption that allows us to use irreducible representations to do this.
Best Answer
Invariant states are not the only meaningful ones. Even in classical mechanics, a baseball traveling 90 mph toward my head is quite meaningful to me, even though it is of no consequence to my fellow mathematician a mile away.
The focus on invariant subspaces comes not from an assumption, but from the way physicists do their work. They want to predict behavior by making calculations. They want to find laws that are universal. They want equations and calculation rules that will be invariant under a change of observers.
Any particular calculation might require a choice of coordinates, but the rules must be invariant under that choice. Once we're talking about one particular baseball trajectory, that trajectory will look different in different coordinate systems; the rules governing baseball flight, however, must look the same in all equivalent coordinate systems.
The natural features of baseballs arise from the equivalence classes of trajectories of baseballs -- equivalence under the group action. Here, if we pretend the earth is flat, gravity is vertical, and air does not resist the baseball, the group is generated by translations and rotations of the plane. Any physically natural, intrinsic property of the baseball itself (such as its mass) or its trajectory (such as the speed of the baseball) must be invariant under the group action. If you don't know a priori what these properties will be, a good way to find them is to pass from individual instances (the baseball heading toward me at 90mph) to the equivalence class generated by individual instances under the group action (the set of all conceivable baseballs traveling at 90mph). Note that the equivalence class is invariant under the group action, and it is exactly this invariance that makes the equivalence class a useful object of the physicists' study.
More generally, if you are studying a physical system with symmetry, it's a good bet that the invariant objects will lead to physically relevant, important quantities. It's more a philosophy than an axiom, but it has worked for centuries.