Edit, 9/25/20: The suggestion I made at the end works.
Proposition: Let $G$ be a group of order $n$ (which may be infinite). Then $\text{Inn}(G)$ is precisely the kernel of the action of $\text{Aut}(G)$ acting on the set $\text{Hom}_{\text{HGrp}}(F_n, G)$ of (simultaneous) conjugacy classes of $n$-tuples of elements of $G$.
Proof. Suppose $\varphi \in \text{Aut}(G)$ acts trivially. Consider its action on the $n$-tuple given by every element of $G$. Fixing this $n$-tuple means fixing it up to conjugacy, which means there is some $g \in G$ such that $\varphi(h) = ghg^{-1}$ for all $h \in G$, which says precisely that $\varphi \in \text{Inn}(G)$. On the other hand, every element of $\text{Inn}(G)$ clearly acts trivially. $\Box$
Of course we can do much better than considering every element of $G$; it suffices to consider a generating set. But this construction is at least "canonical."
Here's an approach that maybe will seem like it doesn't tell you anything new but I'll extract something slightly more concrete out of it, which generalizes the suggestion to look at conjugacy classes. $\text{Out}(G)$ occurs naturally as the automorphism group of $G$ in a category we might call the homotopy category of groups $\text{HGrp}$. This category can be defined concretely as follows:
- objects are groups $G$, and
- morphisms $f : G \to H$ are conjugacy classes of homomorphisms, where two homomorphisms $f_1, f_2 : G \to H$ are identified (homotopic) iff there exists $h \in H$ such that $h f_1 = f_2 h$.
For example:
- $\text{Hom}_{\text{HGrp}}(\mathbb{Z}, G)$ is the set of conjugacy classes of $G$
- $\text{Hom}_{\text{HGrp}}(G, S_n)$ is the set of isomorphism classes of actions of $G$ on a set of size $n$
- $\text{Hom}_{\text{HGrp}}(G, GL_n(\mathbb{F}_q))$ is the set of isomorphism classes of actions of $G$ on $\mathbb{F}_q^n$
and so forth.
Now we can prove the more general fact that composition in this category is well-defined (that is, that the homotopy class of a composition of morphisms only depends on the homotopy class of each morphism), which implies in particular that the automorphism group $\text{Aut}_{\text{HGrp}}(G)$ of $G$ in this category is really a group, and of course this group is $\text{Out}(G)$.
So far this is just a slight extension and repackaging of the proof via conjugating by an inner automorphism, but the point is that this construction tells you what conjugating by an inner automorphism means. The homotopy category of groups has a second description, as follows:
- objects are Eilenberg-MacLane spaces $K(G, 1) \cong BG$, and
- morphisms $f : BG \to BH$ are homotopy classes of homotopy equivalences.
We get the ordinary category of groups if we instead insist that Eilenberg-MacLane spaces have basepoints and our morphisms and homotopies preserve basepoints. So the passing to conjugacy classes has to do with the extra freedom we get from throwing out basepoints. Here the incarnation of conjugacy classes $\text{Hom}(\mathbb{Z}, G)$ is the set of free homotopy classes of loops $S^1 \to BG$.
Anyway, all this suggests the following generalization of looking at conjugacy classes: we can look at the entire representable functor
$$\text{Hom}_{\text{HGrp}}(-, G) : \text{HGrp}^{op} \to \text{Set}.$$
By the Yoneda lemma, the automorphism group of this functor is precisely $\text{Aut}_{\text{HGrp}}(G) \cong \text{Out}(G)$. What this says is that an outer automorphism of $G$ is the same thing as a choice, for each group $H$, of an automorphism (of sets) of $\text{Hom}_{\text{HGrp}}(H, G)$, which is natural in $H$. We can furthermore hope that it's possible to restrict attention to a smaller collection of groups $H$; for example (and I haven't thought about this at all) maybe it's possible to restrict to the free groups $H = F_n$, which means looking at $\text{Hom}_{\text{HGrp}}(F_n, G)$, the set of conjugacy classes of $n$ elements of $G$ (under simultaneous conjugacy).
I don't know if this is appropriate as an answer or if this should rather be an extended comment.
A generic action $\phi$ with $\phi_g \in {\rm Sym}(H) \setminus {\rm Aut}(H)$ for $g\neq 1$ that is not in any way compatible with the group structure of $H$ is just an action of $G$ on $H$ regarded as a set.
Suppose $H$ is finite for simplicity, actions of $G$ on (finite) sets can be classified in terms of transitive actions. More explicitly you can describe $H$ as a disjoint union of orbits $H = {\rm Orb}_{h_1} \sqcup \cdots \sqcup {\rm Orb}_{h_n}$ for some $h_1, \dots, h_n \in H$ and ${\rm Orb}_h = \{\phi_g(h) | g \in G\}$.
By the orbit stabilizer theorem each orbit ${\rm Orb}_{h_i}$ is in bijection with the coset space $G/{\rm Stab}_{h_i}$ (where ${\rm Stab}_{h_i} = \{g \in G | \phi_g(h) = h\}$ is the stabilizer subgroup of $h_i$) and the original action $\phi_g$ can be described in terms of left multiplication of cosets.
Notice that in all of this the fact that $H$ is a group does not matter at all!
When $\phi$ is a group homomorphism $G \to {\rm Aut}(H)$ the situation is more interesting, for example you can use this action to construct a new group $H \rtimes_\phi G$ called 'semidirect product' of $G$ and $H$. Its elements are pairs $(g, h) \in G\times H$ just as in the normal direct product, but the multiplication is given by $$(h_1, g_1)(h_2, g_2) = (h_1 \phi_{g_1}(h_2), g_1g_2).$$
Semidirect products are very interesting and crop up in many places! The fact that $\phi_g$ is an automorphism of $H$ and not just a bijective function is crucial for this construction to work.
I hope this gives you a clearer picture about group acting on sets vs groups acting on groups.
Best Answer
Normally, if $H$ is a normal subgroup of $G$, then $$G/H=\{gH\mid g\in G\}\quad \text{and}\quad H\setminus G=\{Hg\mid g\in G\}.$$