I’m trying to show that if $H, S$ are subgroups with order $p^k, p^n$ inside of $G$ with order $p^n m $ (and $p,m$ relative prime), if $H$ is inside the normalizer of $S$ then $H \leq S$.
I’ve tried to proof this using group actions, but have seemed to show that distinct $p$-Sylow groups are actually the same. I can’t see where I’ve gone wrong though.
Where is my mistake, and how do I fix it and solve the problem?
My method:
Let $H$ act on the left cosets $G/S$ as follows:
$$
h \cdot gS = gh^{-1}S.
$$
Then since $H$ is a p-group, a fixed point theorem says that the number of elements stabilized by the action is equal the number of elements of the set mod $p$:
$$
|G/S|^H \equiv |G/S| \pmod p
$$
Since $S$ is a $p$-Sylow group, $p$ and $|G/S|$ are relatively prime so $|G/S|^H$ cannot be non-empty. Hence let $gS$ be fixed by all of $H$.
Then $gh^{-1}S = gS$, for all $h$, hence $gHS = gS$, so $HS = S$ and in particular $H \leq S$.
But if $H$ is some other $p$-Sylow group (and hence also a $p$-group), I don’t have $H\leq S$, hence my confusion.
Best Answer
I'm going to guess that the $P$ in the first paragraph is supposed to be $H$. The reasons I'm going to guess that are (i) There is no $P$ mentioned until that statement; and (ii) for your action to be well-defined, you need $H$ to normalize $S$. Indeed, if $gS = xS$, then $x^{-1}g\in S$. But then to get $gh^{-1}S = xh^{-1}S$, you need $hx^{-1}gh^{-1}\in S$, but all you know is that it lies in $hSh^{-1}$. So if $H$ normalizes $S$, then this works.
If you have that assumption, then your qualm is actually born from you forgetting about this hypothesis! Note that $S\triangleleft N_G(S)$, since the normalizer is the largest subgroup of $G$ in which $S$ is normal. But that means that $S$ is a normal $p$-Sylow subgroup of $N_G(S)$, and therefore that $S$ is the unique $p$-Sylow subgroup of $N_G(S)$. Hence, you cannot have $H$ be a $p$-Sylow subgroup of $G$ that is both different from $S$ and contained in $N_G(S)$.
I will note that your argument for $H\leq S$ can be simplified. Once you have $gS=gh^{-1}S$, this implies $hg^{-1}g = h\in S$. Hence $H\subseteq S$.