Orthogonality makes sense without character theory. There's an inner product on the space of representations given by $\dim \operatorname {Hom}(V, W)$. By Schur's lemma the irreps are an orthonormal basis. This is "character orthogonality" but without the characters.
How to recover the usual version from this conceptual version? Notice $$\dim \operatorname{Hom}(V,W) = \dim \operatorname{Hom}(V \otimes W^*, 1)$$ where $1$ is the trivial representation. So in order to make the theory more concrete you want to know how to pick off the trivial part of a representation. This is just given by the image of the projection $\frac1{|G|} \sum_{g\in G} g$.
The dimension of a space is the same as the trace of the projection onto that space, so
$$
\def\H{\rule{0pt}{1.5ex}H}
\dim \operatorname{Hom}(V \otimes W^*, 1) = \operatorname{tr}\left(\frac1{|G|} \sum_{g\in G} {\large \rho}_{\small V \otimes W^*}(g)\right)
= \frac1{|G|} \sum_{g\in G} {\large\chi}_{V}(g)\ {\large\chi}_{W}\left(g^{-1}\right)
\\
$$ using the properties of trace under tensor product and duals.
Invariant states are not the only meaningful ones. Even in classical mechanics, a baseball traveling 90 mph toward my head is quite meaningful to me, even though it is of no consequence to my fellow mathematician a mile away.
The focus on invariant subspaces comes not from an assumption, but from the way physicists do their work. They want to predict behavior by making calculations. They want to find laws that are universal. They want equations and calculation rules that will be invariant under a change of observers.
Any particular calculation might require a choice of coordinates, but the rules must be invariant under that choice. Once we're talking about one particular baseball trajectory, that trajectory will look different in different coordinate systems; the rules governing baseball flight, however, must look the same in all equivalent coordinate systems.
The natural features of baseballs arise from the equivalence classes of trajectories of baseballs -- equivalence under the group action. Here, if we pretend the earth is flat, gravity is vertical, and air does not resist the baseball, the group is generated by translations and rotations of the plane. Any physically natural, intrinsic property of the baseball itself (such as its mass) or its trajectory (such as the speed of the baseball) must be invariant under the group action. If you don't know a priori what these properties will be, a good way to find them is to pass from individual instances (the baseball heading toward me at 90mph) to the equivalence class generated by individual instances under the group action (the set of all conceivable baseballs traveling at 90mph). Note that the equivalence class is invariant under the group action, and it is exactly this invariance that makes the equivalence class a useful object of the physicists' study.
More generally, if you are studying a physical system with symmetry, it's a good bet that the invariant objects will lead to physically relevant, important quantities. It's more a philosophy than an axiom, but it has worked for centuries.
Best Answer
The "chronology" isn't clear to me, and having looked through the literature it seems much more convoluted than it should be. Although it seems like this is basically how things were done since the beginning of quantum mechanics (at least, by the big-names) in some form or another, and was 'partly' formalized in the '30s-'40s with the beginnings of QED, but not really completely carefully formalized until the '60s-'70s with the development of the standard model, and not really mathematically formalized until the more careful development of things in terms of bundles in the '70s-'80s. (These dates are guesses--someone who was a practicing physicist during those periods is more than welcome to correct my timeline!)
Generally speaking, from a 'physics' point of view, the reason particles are labeled according to representations is not too different than how, in normal quantum mechanics, states are labeled by eigenvalues (the wiki article linked to mentions this, but it's not as clear as it could be).
In normal QM, we can have a Hilbert space ('space of states') $\mathcal{H}$, which contains our 'physical states' (by definition). To a physicist, 'states' are really more vaguely defined as 'the things that we get the stuff that we measure from,' and the Hilbert space exists because we want to talk about measurements. The measurements correspond to eigenvalues of operators (why things are 'obviously' like this is a longer historical story...).
So we have a generic state $| \psi \rangle \in \mathcal{H}$, and an operator that corresponds to an observable $\mathcal{O}$. The measured values are
Because the $o_i$ are observable quantities, it's useful to label systems in terms of them.
We can have a list of observables, $\mathcal{O}_j$, (which we usually take to be commuting so we can simultaneously diagonalize), and then we have states $|\psi\rangle$,
So, what we say, is that we can uniquely define our normal QM states by a set of eigenvalues $o_{ij}$.
In other words, the $o_{ij}$ define states, from the physics point of view. Really, this defines a basis where our operators are diagonal. We can--and do!--get states that do not have observables which can be simultaneously diagonalized, this happens in things like neutrino oscillation, and is why they can turn into different types of neutrinos! The emitted neutrinos are emitted in states with eigenvalues which are not diagonal in the operator that's equivalent to the 'particle species' operator. (Note, we could just as well define the 'species' to be what's emitted, and then neutrinos would not oscillate in this basis, but would in others!)
This has to do with representations, because when we talk about particles with spin, for example, we're talking about operators which correspond to 'angular momentum.' We have an operator:
and label eigenvalues by half-integer states which physically correspond to spin. Group theoretically, $L_z$ comes from the lie algebra of the rotation group, because we're talking about angular momentum (or spin) which has associated rotational symmetries.
Upgrading from here to quantum field theory (and specializing that to the standard model) is technically complicated, but is basically the same as what's going on here. The big difference is, we want to talk there about 'quantum fields' instead of states, and have to worry about crazy things like apparently infinite values and infinite dimensional integrals, that confuse the moral of the story.
But the idea is simply, we want to identify things by observables, which correspond to eigenvalues, which correspond to operators, which correspond to lie algebra elements, which have an associated lie group.
So we define states corresponding to things which transform under physically convenient groups as 'particles.'
If you want a more mathematically careful description, that's still got some physical intuition in it, you can check out Gockler and Schuker's "Differential Geometry, Gauge theory, and Gravity," which does things from the bundle point of view, which is slightly different than I described (because it describes classical field theories) but the moral is similar. At first it might seems surprising that the classical structure here is the same, when it seemed to rely on operators and states in Hilbert spaces, but it only technically relied on it, but morally, what's important is actions under symmetry groups. And that is in the classical theory as well. But it's not as physically clear from the beginning from that point of view.