A result of Borel and Lichnerowicz states that the holonomy group of a connection on a principal $G$-bundle is a Lie subgroup of $G$ (Cartan had earlier asserted this, but apparently without proof).
This restriction, that it be a Lie subgroup, allows for a lot of poorly-behaved subgroups, for example a line with irrational slope on a torus. This subgroup comes from a perfectly fine immersion of the Lie group $\mathbb{R}$, but it's not closed in the induced topology of the torus.
As an example of something that's not a Lie subgroup, let $G= \mathbb{R}$, consider an uncountable set of $\mathbb{Q}$-independent points, none of which are rational, and consider the subgroup they generate. If this were a Lie subgroup it would be the image of an uncountable discrete space (there can't be anything $1$-dimensional, since we left out the rationals), which wouldn't be second countable, hence not a manifold and not a Lie group.
This seems like a pretty contrived example, and I suspect there is more content to 'being a Lie subgroup' than having countably many components. However, I can't seem to pin down something that would illustrate this. Can anyone give me an example of a connected subgroup of a Lie group that is not a Lie subgroup?
Best Answer
As Aaron and Eric have pointed out, Yamabe's theorem is the statement that an arcwise connected subgroup of a Lie group must be a Lie subgroup. There is also a classical theorem by Cartan and Chevalley, also proved in Hochschild's book, that a connected locally compact topological group admitting a continuous homomorphism into a Lie group which is injective in a neighborhood of the identity must be a Lie group. In particular, if $H$ is a subgroup of a Lie group $G$ and $H$ is connected and locally compact with respect to a topology containing the relative topology, then $H$ is a Lie group (note that Prop. 2.11 in the book by Helgason only addresses local compactness in the relative topology).
Edit: Regarding Robert's post, it is certainly important to point out the different existing definitions of Lie subgroups. To make myself clear, I am using the definition of Lie subgroup as in the book by Helgason, ch.2 (or Warner, ch. 3), namely, a Lie subgroup of a Lie group is an abstract subgroup which is an (immersed) submanifold and may have a topology finer than the relative topology. I think this definition is very common and has the advantage of yielding, for a given Lie group, a bijective correspondence between Lie subalgebras of its Lie algebra and its Lie subgroups.