Suppose $N,H$ are groups. Assuming $\varphi$ is a homomorphism from $H$ to $\textrm{Aut}(N)$, we can construct what's called the semi-direct product $N \rtimes H$, defining the operation on the cartesian product $N \times H$ as $(n,h) (n',h') = (n \varphi_{h} (n'), hh')$.
By a direct check we can see that this truly defines a group which is generally denoted by $N \rtimes H$, and if we identify $N$ with $\{ (n,0) : n \in N \}$ and $H$ with $\{ (0,h) : h \in H \}$, we see that $N$ is normal in $N \rtimes H$, $N \cap H =1$, and $NH = N \rtimes H$.
This gives us a way to describe the situation when $N,H$ are subgroups of $G$, $N$ is normal in $G$, $N \cap H = 1$, and $NH = G$ – here we consider the the action of $H$ on $G$ by conjugation. But thinking about this, I realized that the actual conjugation action is somewhat arbitrarily chosen as either left or right action – conjugation of $n$ by $h$ can be either $n \mapsto h n h^{-1}$ or $n \mapsto h^{-1} n h$. If this map is to be a homomorphism from $H$ to $\textrm{Aut}(N)$ then only the first map works, but I would have expected both of these conjugations to work (in the sense that both can be used to describe some sort of semidirect product of $N$ and $H$).
If both of these conjugations actually work, this could indicate that a homomorphism from $H$ to $\textrm{Aut}(N)^{op}$ could also be used to create a semi-direct product of some sort. Here, by $\textrm{Aut}(N)^{op}$ I mean the group of automorphisms on $N$, where the composition of functions is reversed – i.e. $\varphi : H \rightarrow \textrm{Aut}(N)^{op}$ satisfies $\varphi(ab)=\varphi_{ab}=\varphi_b \circ \varphi_a$, where $\circ$ is the standard composition defined as $f \circ g (x) = f(g(x))$.
In fact, if $\varphi \colon H \rightarrow \textrm{Aut}(N)^{op}$ is such a homomorphism, then defining the operation on the cartesian product $N \times H$ as:
$(n,h)(n',h') = ( \varphi_{h'}(n) n', hh')$
seems to work properly – it is a group $G$ that contains $N$ as a normal subgroup, $N \cap H =1 $ and $NH = G$. Also notice the first coordinate is the same as the opposite group of $N \rtimes H$ as constructed by the first method mentioned in this post (first paragraph of this post).
Assuming I haven't made a mistake checking the properties above, why doesn't the semidirect product definition not contain this method of constructing it? Maybe these semi-direct products defined using $\varphi:H \rightarrow \textrm{Aut}(N)^{op}$ are all isomorphic to some other semi-direct product constructed by some homomorphism $\psi:H \rightarrow \textrm{Aut}(N)$, and so they don't describe anything new.
I would appreciate any input on this issue or reference to a book that considers this issue.
Best Answer
$\renewcommand{\phi}{\varphi}$$\DeclareMathOperator{\Aut}{Aut}$Let $N, H$ be groups, and $\Aut(N)$ the automorphism group of $N$. Let us decide how we can compose on $\Aut(N)$, let us say we do it left-to-right, that is, if $\phi, \psi \in \Aut(N)$, then in the composition $\phi \circ \psi$ it is $\phi$ that acts first. In $\Aut(N)^{op}$ composition will be right-to-left, then.
Consider a homomorphism $f : H \to \Aut(N)$, and construct the semidirect product $H \ltimes N$ with multiplication $$ (h_{1}, n_{1}) (h_{2}, n_{2}) = (h_{1} h_{2}, n_{1}^{f(h_{2})} n_{2}). $$ (I write automorphisms as exponents.)
With the identifications you have provided, in this group we have $$ h^{-1} n h = n^{f(h)}. $$
Now consider the map $f'$ given by $f'(h) = f(h^{-1})$. It is easily seen that $$ f' : H \to \Aut(N)^{op} $$ is a homomorphism. Moreover, it is easily seen that every homomorphism $H \to \Aut(N)^{op}$ is of the form $f'$, for some homomorphism $f : H \to \Aut(N)$.
Now consider the previous semidirect product, and consider in it, with your identifications, $$ h n h^{-1} = (h, 1) (n, 1) (h^{-1}, 1) = (h, n) (h^{-1}, 1) = (1, n^{f(h^{-1})}) = n^{f'(h)}. $$ So $f, f'$ give the same semidirect product, it is just that you read the action of $f(h)$ as a conjugation $n \mapsto h^{-1} n h$ (that is, a right action, as it corresponds to left-to-right composition in $\Aut(G)$), and that of $f'(h)$ as $n \mapsto h n h^{-1}$ (that is, a left action, as it corresponds to right-to-left composition in $\Aut(G)^{op}$)).