Here $xHx^{-1} \subsetneq H$ means $xHx^{-1}$ as to be a proper subset of $H$.
From Gallian, Contemporary Abstract Algebra:
Normal Subgroup:
A subgroup $H$ of a group $G$ is called a normal subgroup of $G$ if $aH =
Ha$ for all $a$ in $G$. We denote this by $H$ $\triangleleft$ $G$…..
Normal Subgroup Test:
A subgroup $H$ of $G$ is normal in $G$ if and only if $xHx^{-1} \subseteq H$
for all $x$ in $G$.Proof If $H$ is normal in $G$, then for any $x$ $\epsilon$ $G$ and $h
$ $\epsilon$ $H$ there is an $h^{'}$ in
H such that $xh = h^{'}x$. Thus, $xhx^{-1} = h^{'}$, and therefore $xHx^{-1} \subseteq H$.Conversely, if $xHx^{-1} \subseteq H$ for all $x$, then, letting $x = a$, we have
$aHa^{-1} \subseteq H$ or $aH \subseteq Ha$. On the other hand, letting $x = a^{-1}$, we have
$a^{-1}H(a^{-1})^{-1} = a^{-1}Ha \subseteq H$ or $Ha \subseteq aH$.
From above theorem of Normal subgroup test, if
$1.$ $xHx^{-1} \subsetneq H$
or
$2.$ $xHx^{-1}=H$, $H$ will be a normal subgroup.
But from the first definition of normal subgroups given above in the extract, $H$ seems to be normal only if $xH=Hx \implies xHx^{-1}=H$.
Then, if $H$ is a subgroup, and $xHx^{-1} \subsetneq H$, for all $x$ in $G$, is H a normal subgroup? If yes, is it not against the definition of normal subgroup?
Best Answer
No such subgroups exist, since for $x=1$, we always have $xHx^{-1}=H$. So, it is true that $xHx^{-1}\subsetneq H$ for all $x\in G$ implies $H$ is a normal subgroup, but the implication is vacuous.
More strongly, if $xHx^{-1}\subseteq H$ for all $x\in G$, then actually $xHx^{-1}=H$ for all $x\in G$ (the inclusion cannot be strict for even a single $x$). This follows from the normal subgroup test, since as you have observed the definition of a normal subgroup actually gives equality rather than just containment. Or more directly, letting $y=x^{-1}$, we have $x^{-1}Hx=yHy^{-1}\subseteq H$ and then multiplying on the left by $x$ and on the right by $x^{-1}$ gives $H\subseteq xHx^{-1}$. (In other words, the reverse inclusion of $xHx^{-1}\subseteq H$ comes from replacing $x$ with its inverse.)