A teacher shared with me his textbook (writing in progress) about group theory and this exercise is at the end of the first chapter.
The whole problem:
A subgroup $H$ of $G$ is special if for all $x \in G\backslash H$ and $y \in G$ exists only one $a \in H$ such that $y^{-1}xy=a^{-1}xa$. Prove that if $H$ is a special subgroup of $G$, then $H \unlhd G$.
I tried by contradiction, suppose $ H $ is not a normal subgroup of $G$. Then, there is $g \in G$ and $h \in H$ such that $gHg^{-1}\neq H $. Equivalently, there is $g \in G$ and $h \in H$ such that $ghg^{-1} \in G \backslash H $. Since $H$ is a special group, there is only one $a \in H$ such that $y^{-1}(ghg^{-1})y = a^{-1}(ghg^{-1})a$.
What i'm trying to do is to show that there is more than one $a \in H$ that satisfies this property.
Best Answer
You are almost there. You just need to pick $y$ carefully.
Say $h\in H$, $g\in G$, and $x=ghg^{-1}\notin H$.
Then letting $y=g^{-1}$, there exists a unique $a\in H$ such that $yxy^{-1}=axa^{-1}$. Writing this out, that means that $$\begin{align*} h &= (g^{-1}g)h(g^{-1}g)^{-1}\\ &= g^{-1}(ghg^{-1}) (g^{-1})^{-1}\\ &= yxy^{-1}\\ &= axa^{-1}. \end{align*}$$
But then we also have that $$(ha)x(ha)^{-1} = h(axa^{-1})h^{-1} = hhh^{-1}=h=yxy^{-1}.$$ Now bring it home.
The result holds without uniqueness of $a$, though; it is enough that such an $a$ exist. From $h=axa^{-1}$ with $h,a\in H$ we immediately conclude that $x\in H$.