Let $G$ be a group and $H$ a subgroup, then it is standard to say that $H$ is normal in $G$ if and only if for all $x \in G$, $xHx^{-1} \subseteq H$. My question here is, if $G$ and $H$ are additive groups, it is okay notation to write $x + H + x^{-1}$? Or is this something that is not done.
Normal subgroup notation
abstract-algebragroup-theorynormal-subgroupsnotation
Related Solutions
No such subgroups exist, since for $x=1$, we always have $xHx^{-1}=H$. So, it is true that $xHx^{-1}\subsetneq H$ for all $x\in G$ implies $H$ is a normal subgroup, but the implication is vacuous.
More strongly, if $xHx^{-1}\subseteq H$ for all $x\in G$, then actually $xHx^{-1}=H$ for all $x\in G$ (the inclusion cannot be strict for even a single $x$). This follows from the normal subgroup test, since as you have observed the definition of a normal subgroup actually gives equality rather than just containment. Or more directly, letting $y=x^{-1}$, we have $x^{-1}Hx=yHy^{-1}\subseteq H$ and then multiplying on the left by $x$ and on the right by $x^{-1}$ gives $H\subseteq xHx^{-1}$. (In other words, the reverse inclusion of $xHx^{-1}\subseteq H$ comes from replacing $x$ with its inverse.)
Your argument is correct.
The result appears in many books. For example:
J.J. Rotman, Introduction to the Theory of Groups, 4th Edition. The 9th Exercise in the section on normal subgroups (the section only contains the definition), page 31, Exercise 3.33: "If $K\leq H\leq G$ and $K\triangleleft G$, then $K\triangleleft H$."
W.R. Scott's Group Theory, from 1964. Page 29, Exercise 2.1.24(c): "If $H\triangleleft G$ and $H\subset K\subset G$, then $H\triangleleft G$." (Scott uses $\subset$ to denote subgroups that need not be proper, reserving $\lt$ for proper subgroups).
Likely you have not spotted it because the more general version that is most useful is the following:
Let $G$ be a group, and $N\triangleleft G$ a normal subgroup of $G$. Then for any subgroup $H$ of $G$, $N\cap H\triangleleft H$.
In fact, this statement is part of the Second Isomorphism Theorem (which states that $NH/N\cong H/(N\cap H)$). In the special case where $N\leq H\leq G$, you get your result: $N\cap H=N\triangleleft H$.
In this form, it appears in:
Dummit and Foote, Abstract Algebra Second Edition. Exercises on Chapter 3, Quotient Groups and Homomorphisms, has Exercise 24, p. 89.
Lang's Algebra, Revised 3rd Edition, mentions en passant on page 17: "Let $G$ be a group, and let $H,K$ be two subgroups. Assume that $H$ is contained in the normalizer of $K$. Then $H\cap K$ is obviously a normal subgroup of $H$[.]" This yields this result when $K$ is normal and $H$ contains $K$.
Derek Robinson's A Course in the Theory of Groups 2nd Edition, 1.4.4 (on page 19), the Second Isomorphism Theorem: "Let $H$ be a subgroup and $N$ a normal subgroup of a group $G$. Then $N\cap H\triangleleft H$ and $(N\cap H)x\longmapsto Nx$ is an isomorphism from $H/(N\cap H)$ to $NH/N$."
Jacobson's Basic Algebra I, Theorem 1.9 (page 64): "Let $H$ and $K$ be subgroups of $G$, $K$ normal in $G$. Then $HK=\{hk\mid h\in H, k\in K\}$ is a subgroup of $G$ containing $K$, $H\cap K$ is normal in $H$, and the map $hK\mapsto h(K\cap H)$, $h\in H$, is an isomorphism of $HK/K$ with $H/(H\cap K)$.
van der Waerden's Algebra, volume I, Section 7.3, page 147; he uses "divisor" rather than subgroup: "If $N$ is a normal divisor in $G$, $K$ is a subgroup $G$, then the intersection $N\cap K$ is a normal divisor in $K$[.]"
etc. Since this fact is key to one of the Isomorphism Theorems, I dare say that any book on Group Theory that includes the Isomorphism Theorems will necessarily include it, your result being a special case.
Best Answer
As Mark points out in the comments, it would be more appropriate to write $x+H+(-x)$ in such a case, but we typically use the additive notation for abelian groups, in which case every subgroup is normal, so we wouldn't bother to write it.