Invariant states are not the only meaningful ones. Even in classical mechanics, a baseball traveling 90 mph toward my head is quite meaningful to me, even though it is of no consequence to my fellow mathematician a mile away.
The focus on invariant subspaces comes not from an assumption, but from the way physicists do their work. They want to predict behavior by making calculations. They want to find laws that are universal. They want equations and calculation rules that will be invariant under a change of observers.
Any particular calculation might require a choice of coordinates, but the rules must be invariant under that choice. Once we're talking about one particular baseball trajectory, that trajectory will look different in different coordinate systems; the rules governing baseball flight, however, must look the same in all equivalent coordinate systems.
The natural features of baseballs arise from the equivalence classes of trajectories of baseballs -- equivalence under the group action. Here, if we pretend the earth is flat, gravity is vertical, and air does not resist the baseball, the group is generated by translations and rotations of the plane. Any physically natural, intrinsic property of the baseball itself (such as its mass) or its trajectory (such as the speed of the baseball) must be invariant under the group action. If you don't know a priori what these properties will be, a good way to find them is to pass from individual instances (the baseball heading toward me at 90mph) to the equivalence class generated by individual instances under the group action (the set of all conceivable baseballs traveling at 90mph). Note that the equivalence class is invariant under the group action, and it is exactly this invariance that makes the equivalence class a useful object of the physicists' study.
More generally, if you are studying a physical system with symmetry, it's a good bet that the invariant objects will lead to physically relevant, important quantities. It's more a philosophy than an axiom, but it has worked for centuries.
The primary reason for studying Lie algebras is the following fundamental fact: the representation theory of a Lie algebra is the same as the representation theory of the corresponding connected, simply connected Lie group.
Of course, the representation theory of a Lie group in general is very complicated. First of all, it might not be connected. Then the Lie algebra cannot tell the difference between the whole group and the connected component of the identity. This connected component is normal, and the quotient is discrete. So to understand the representation theory of the whole group requires, at the very least, knowing the representation theory of that discrete quotient. Even in the compact case, this would require knowing the representation theory of finite groups. For a given finite group, the characters know everything, but of course the classification problem in general is completely intractable.
The other thing that can go wrong is that even a connected group need not be simply connected. Any connected Lie group is a group quotient of a connected, simply connected group with the same Lie algebra, where the kernel is a discrete central subgroup of the connected, simply connected guy. So the representation theory of the quotient is the same as the representations of the simply connected group for which this central discrete acts trivially.
There is a complete classification of connected compact groups. You start with the steps above: classifying the disconnected groups is intractable, but a connected one is a quotient of a connected simply connected one. This simply connected group is compact iff the corresponding Lie algebra is semisimple; otherwise it has some abelian parts. In general, any compact group is a quotient by a central finite subgroup of a direct product: torus times (connected simply connected) semisimple. Torus actions are easy, and the representation theory of semisimples is classified as well. Whether your representation descends to the quotient I'm not entirely sure I know how to read off of the character. When the group is semisimple (no torus part), I do: finite-dimensional representations are determined by their highest weights (which can be read from the character), which all lie in the "weight" lattice; quotients of the simply-connected semisimple correspond exactly to lattices between the weight lattice and the "root" lattice, and you can just check that your character/weights are in the sublattice.
All of this should be explained well in your favorite Lie theory textbook.
Best Answer
I'm not sure which sort of examples of Lie algebras without the corresponding groups you have in mind, but here is a typical example from Physics.
Many physical systems can be described in a hamiltonian formalism. The geometric data is usually a symplectic manifold $(M,\omega)$ and a smooth function $H: M \to \mathbb{R}$ called the hamiltonian. If $f \in C^\infty(M)$ is any smooth function, let $X_f$ denote the vector field such that $i_{X_f}\omega = df$. If $f,g \in C^\infty(M)$ we define their Poisson bracket $$\lbrace f, g\rbrace = X_f(g).$$ It defines a Lie algebra structure on $C^\infty(M)$. (In fact, a Poisson algebra structure once we take the commutative multiplication of functions into account.)
In this context one works with the Lie algebra $C^\infty(M)$ (or particular Lie subalgebras thereof) and not with the corresponding Lie groups, should they even exist.
Symmetries in this context are functions which Poisson commute with the hamiltonian, hence the centraliser of $H$ in $C^\infty(M)$. They define a Lie subalgebra of $C^\infty(M)$.
Added
Another famous example occurs in two-dimensional conformal field theory. For example, the Lie algebra of conformal transformations of the Riemann sphere is infinite-dimensional: any holomorphic or antiholomorphic function defines an infinitesimal conformal transformation. On the other hand, the group of conformal transformations is finite-dimensional and isomorphic to $\mathrm{PSL}(2,\mathbb{C})$.