QUESTION: General Cayley's theorem: Let $G$ be a group and $H$ a subgroup of $G$ such that $[G:H]=n$. Then the quotient $G/H_G$ is isomorphic to a subgroup of $S_n$.
ANSWER: In the answer I found, there is this isomorphism $\pi: G/H_G \rightarrow \operatorname{Sym}(\{yH: y \in G\})\cong S_n$. But I really didn't understand why is this a isomorphism, I didn't understand even less why this is isomorphic to $S_n$.
$H_G:=\operatorname{kernel}(\pi)$ – where $H_G$ is called normal core of the isomorphism $\pi$. Here for example, I didn't understand why $H=H_G$. Actually, I didn't undestand how to find the kernel of this isomorphism equal to $\displaystyle\bigcap_{g \in G}gHg^{-1}$.
Best Answer
You neither necessarily have an isomorphism to $S_n$, nor do you necessarily have $H=H_G$.
I am only guessing that the proof you found says the set of symmetries of $\{gH\mid g\in G\}$ is isomorphic to $S_n$, not that the map gives an isomorphism. The notation $$\pi\colon G/\mathrm{core}(H) \to \mathrm{Sym}(\{gH\mid g\in G\})\cong S_n$$ does not say $\pi$ is an isomorphism, it says that $$\mathrm{Sym}(\{gH\mid g\in G\})$$ is isomorphic to $S_n$. And this follows because the set has $n=[G:H]$ elements.
If they had meant to say that $\pi$ is an isomorphism, they would probably have written: $$\pi \colon G/H_G \stackrel{\cong}{\to} \mathrm{Sym}(\{gH\mid g\in G\})\cong S_n$$ which they do not do. So they are not saying that $\pi$ is an isomorphism between $G/H_G$ and $S_n$.
It also does say that $H_G$ equals $H$. But the map factors through $G/H_G$. To see that this is the kernel, note first that the kernel is certainly normal. And if $x$ lies in the kernel, then for every $g\in G$ you have that $x(gH) = gH$.
In particular, $xH=H$, so $x\in H$. This proves the kernel is normal and contained in $H$, hence contained in $H_G$.
Conversely, if $x\in H_G$, then for every $g\in G$ we have $x\in gHg^{-1}$, hence $g^{-1}xg\in H$. Therefore, $g^{-1}xgH = H$, and so $xgH = gH$. That means that $x(gH) = gH$, so the action of $x$ on the cosets of $H$ is trivial, so $\pi(x)$ is the identity; that is, $x\in\mathrm{ker}(\pi)$. This vies the other inclusion and hence equality.