7 years ago I wrote in an unfinished paper a proof of this result; I didn't know then if it was original but it indeed seems to be known to a number of specialists although I'm not aware of a written complete proof. Since I don't want anymore to complete this paper, I reproduce the proof here (not self-contained, relying on [LW] below) as Proposition 5; possibly Uri has a more direct approach. I include closely related Proposition 1, which was my original goal.
Proposition 1
Let $G$ be a compact group (not necessarily Lie). Consider the action by $G$ by conjugation on $\mathcal{S}(G)$ (the space of subgroups of $G$, endowed with the Chabauty topology). Then the orbits of $G_0$ (connected component of 1) coincide with the connected components of $\mathcal{S}(G)$.
As a corollary, for $G$ a compact group, $\mathcal{S}(G)$ is totally disconnected if and only if $G_0$ is central in $G$ (equivalently, $G/Z(G)$ is totally disconnected), a result obtained in [FG]. More generally, it can be checked that for an arbitrary locally compact group $G$, $\mathcal{S}(G)$ is totally disconnected if and only if either totally disconnected, or $G$ is pointwise elliptic with $G_0$ central. ($G$ pointwise elliptic means that every single element generates a cyclic subgroup with compact closure.)
Lemma 2
Let $G$ be a compact Lie group. Let $F$ be a finite group. Then there are finitely many conjugacy classes of homomorphisms $F\to G$.
Proof.
The space $\mathrm{Hom}(F,G)$ can be described as a real algebraic variety, so has finitely many components. By [LW], all homomorphisms in a given component are conjugate under $G_0$.
\end{proof}
Lemma 3
Let $G$ be a compact Lie group. Let $S$ be a connected compact semisimple Lie group. Then there are finitely many conjugacy classes of homomorphisms $S\to G$.
Proof.
The space $\mathrm{Hom}(S,G)$ is canonically identified with the space of Lie algebra homomorphisms $\mathfrak{s}\to\mathfrak{g}$, which is an algebraic variety as well, so has finitely many connected components. So we can again use [LW].
Lemma 4
Let $G$ be a compact Lie group. Let $D$ be a connected compact abelian Lie group. Then there are countably many conjugacy classes of homomorphisms $D\to G$.
Proof.
Any such homomorphism maps into a maximal torus, and all maximal tori are conjugate [Bourbaki, Groupes et Algèbres de Lie, Chap 9, §2.2]. This reduces to the case when $G$ is a torus, but then the result just follows from the fact that $\mathrm{Hom}((\mathbf{R}/\mathbf{Z})^a,(\mathbf{R}/\mathbf{Z})^b)\simeq\mathbf{Z}^{ab}$, which is countable.
Lemma 5
Let $G$ be a compact Lie group. Then there are countably many conjugacy classes of closed connected subgroups in $G$.
Proof.
Since there are finitely many isomorphism types of compact semisimple groups in a given dimension, we can fix this type, so by Lemma 3, it is enough to consider the set of closed connected subgroups whose semisimple part is a given subgroup $S$. Therefore, it is enough to count conjugacy classes of closed abelian connected subgroups in $N(S)/S$, where $N(\cdot)$ denotes the normalizer, and Lemma 4 applies.
Proposition 6
Let $G$ be a compact Lie group. Then there are countably many conjugacy classes of closed subgroups in $G$.
Proof.
By Lemma 5, we can consider those subgroups for which $H_0=L$ is given. So it is enough to count conjugacy classes of finite subgroups in $N(L)/L$, there are countably many because of Lemma 2.
Proof of Proposition 1
We begin with the case when $G$ is a Lie group. Since $G_0$ has finite index in $G$, Proposition 6 implies that there are countably many $G_0$-conjugacy classes of closed subgroups in $G$. Let $C$ be a connected component of $\mathcal{S}(G)$. Clearly, it is invariant under conjugation by $G_0$. We just obtained that the quotient of $C$ by the action of $G_0$ is countable. It is also a compact connected Hausdorff space, so is reduced to a point (it is Hausdorff because the acting group is compact, see [Bourbaki, Topologie Générale, III, §4.1]. This means that $G_0$ acts transitively on each connected component of $\mathcal{S}(G)$, and we are done.
Now let us deal with a general compact group $G$. (It is no longer true in general that the set of connected components of $\mathcal{S}(G)$ is countable). We write $G$ as a projective limit of compact Lie groups $G/K_i$. Clearly, if two elements $H,H'$ of $\mathcal{S}(G)$ are conjugate under $G_0$, then they lie in the same component. Conversely, suppose they belong to the same connected component of $\mathcal{S}(G)$. Then since $K_i$ is compact, the projection map $\mathcal{S}(G)\to\mathcal{S}(G/K_i)$ is continuous. Because of the case of Lie groups, there exists $g_i\in G_0$ such that $(g_iHg_i^{-1}K_i=H'K_i)$ (we use the fact that the inverse image of the unit component of $G/K_i$ is the unit component of $G$). Let $g$ be a limit point of $(g_i)$. If $h\in H$, then $g_ihg_i^{-1}$ can be written as $h'_ik_i$ with $h'_i\in H'$ and $k_i\in K_i$. Necessarily $k_i\to 1$, so $h'_i=g_ihg_i^{-1}k_i^{-1}$ also converges, necessarily to an element $h'$ of $H'$. We have $ghg^{-1}=h'$. This proves that $gHg^{-1}\subset H'$, and the converse inclusion $g^{-1}H'g\subset H'$ is strictly similar.
[FG] S. Fisher, P. Gartside. On the space of subgroups of a compact group II. Topology Appl. 156 (2009) 855-861
[LW] D.H. Lee, T.S. Wu.
On conjugacy of homomorphisms of topological groups.
Illinois J. Math. 13 1969 694-699
in [LW] I refer to Theorem 2.6: if $G,H$ are compact Lie groups, then the connected components of $\mathrm{Hom}(G,H)$ are precisely the $H_0$-orbits (for action by post-conjugation).
Yes, it is true. Let us show that, in a compact Lie group $G$ with $G^0$ nonabelian, finite subgroups that are not contained in any proper subgroup of positive dimension have bounded cardinal.
By contradiction, let finite subgroups $G_n$ have cardinal tending to infinity, and not contained in any proper subgroup of positive dimension.
Let $B$ be a neighborhood of 0 in the Lie algebra of $G$ such that $\exp|_B$ is a homeomorphism onto its image, with inverse denoted by $\log$ (we will let $B$ vary, but can assume it is contained in a single such neighborhood $B_0$ once and for all, hence the function $\log$ will not depend on $B$).
Define $T_n^B=\exp\big(\mathrm{span}\big(\log(G_n\cap\exp(B))\big)\big)$.
The first point is that for each $B\subset B_0$ and for $n$ large enough, $G_n\cap\exp(B)$ consists of commuting elements. This is close to Jordan's theorem and is a particular case of Zassenhaus' theorem (see for instance Theorem 1.3 here, with in mind that connected nilpotent subgroups of compact groups are abelian). We can extract once and for all to suppose that this holds for all $n$.
The next point is that $T_n^B$ is a torus. For an arbitrary commutative subset, it might have been dense in a torus. But the point is that elements of finite subgroups are well-behaved. More precisely: for this point, we can suppose that $G$ is embedded in the unitary group $\mathrm{U}(d)$. Every element of $G$ can be diagonalized with eigenvalues $e^{i\pi k_1/m},\dots,e^{i\pi k_d/m}$. We can arrange $B_0$ so that it avoids elements with eigenvalues $-1$ and define the log accordingly, so assume all $k_i$ to be in $]-m,m[$. Hence the log of such an element will be, in the same diagonal basis, the matrix with eigenvalues $i\pi k_1/m,\dots i\pi k_d/m$, and the exp of its span will be the 1-dimensional torus of diagonal matrices with eigenvalues $\theta^{k_1},\dots,\theta^{k_d}$ for $\theta$ in the unit circle. Now when we consider all $G_n\cap \exp(B)$, we just consider the torus generated by several such (commuting) tori.
Now consider $\liminf_n \dim(T_n^B)$. This is positive (because $G_n$ is infinite, so that $G_n\cap\exp(B)$ is never reduced to $\{1\}$. This liming decreases when $B$ decreases, and hence we can find $B_1$ for which its value, say $c>0$, is minimal. After extracting once again, we an suppose that $\lim_n\dim(T_n^{B_1})=c$. Define $T_n=T_n^{B_1}$. Hence, for every neighborhood $B\subset B_1$, there exists $n_0$ (depending on $B$) such that for all $n\ge n_0$ we have $T_n^B=T_n$ (because $T_n^B\subset T_n$ and these are tori of the same dimension).
This observation allows to deduce that $T_n$ is, for large $n$, normalized by $G_n$. Indeed, for $g_n\in G_n$, we have $$g_nT_n^Bg_n^{-1}=\exp(\mathrm{span}\log(G_n\cap \exp(g_nBg_n^{-1})));$$
since $1$ has a basis of conjugation-invariant neighborhoods, we can choose $B$ such that $\bigcup_g gBg^{-1}$ is contained in $B_1$. We deduce that $g_nT_ng_n^{-1}\subseteq T_n$ for large $n$, hence this is an equality.
Then $G_nT_n$ is a positive-dimensional closed subgroup containing $G_n$. By assumption, it equals $G$, and this is a contradiction since $G^0$ is non-abelian.
Best Answer
A natural condition is that $G$ has finitely many connected components. One can easily reduce this case to the connected group case, and then to the compact group case, as all maximal compact subgroups in a connected Lie group are conjugated. Then the representation variety $\text{Hom}(\Gamma,G)$ is compact and local rigidity, that is vanishing of $H^1(\Gamma,\mathfrak{g})$, guarantees the finiteness of the number of $G$-orbits. Here $\mathfrak{g}$ denotes the Lie algebra of $G$. The fact that $H^1$ vanishes could be deduced from the fact that every isometric action of $\Gamma$ on $\mathfrak{g}$ has a fixed point, by averaging an orbit.