In spite of being the "action nr. 1" (Cayley's) of a group $G$ on "another" group ($G$ itself) definitely not by automorphisms, why is the usual meaning of a $G$-action on a group $H$ being a homomorphism $ϕ\colon G→\operatorname{Aut}(H)$? Said differently, why not caring about actions $\phi$ such that $\phi_g\in\operatorname{Sym}(H)\setminus\operatorname{Aut}(H)$ for some $g\in G\setminus\{e\}$?
$G$, $H$ groups. Why not caring about $G$-actions on $H$ s.t. $\phi_g\in{\rm{Sym}}(H)\setminus{\rm{Aut}}(H)$ for some $g\in G\setminus\{e\}$
group-actionsgroup-theory
Best Answer
I don't know if this is appropriate as an answer or if this should rather be an extended comment.
A generic action $\phi$ with $\phi_g \in {\rm Sym}(H) \setminus {\rm Aut}(H)$ for $g\neq 1$ that is not in any way compatible with the group structure of $H$ is just an action of $G$ on $H$ regarded as a set.
Suppose $H$ is finite for simplicity, actions of $G$ on (finite) sets can be classified in terms of transitive actions. More explicitly you can describe $H$ as a disjoint union of orbits $H = {\rm Orb}_{h_1} \sqcup \cdots \sqcup {\rm Orb}_{h_n}$ for some $h_1, \dots, h_n \in H$ and ${\rm Orb}_h = \{\phi_g(h) | g \in G\}$.
By the orbit stabilizer theorem each orbit ${\rm Orb}_{h_i}$ is in bijection with the coset space $G/{\rm Stab}_{h_i}$ (where ${\rm Stab}_{h_i} = \{g \in G | \phi_g(h) = h\}$ is the stabilizer subgroup of $h_i$) and the original action $\phi_g$ can be described in terms of left multiplication of cosets.
Notice that in all of this the fact that $H$ is a group does not matter at all!
When $\phi$ is a group homomorphism $G \to {\rm Aut}(H)$ the situation is more interesting, for example you can use this action to construct a new group $H \rtimes_\phi G$ called 'semidirect product' of $G$ and $H$. Its elements are pairs $(g, h) \in G\times H$ just as in the normal direct product, but the multiplication is given by $$(h_1, g_1)(h_2, g_2) = (h_1 \phi_{g_1}(h_2), g_1g_2).$$
Semidirect products are very interesting and crop up in many places! The fact that $\phi_g$ is an automorphism of $H$ and not just a bijective function is crucial for this construction to work. I hope this gives you a clearer picture about group acting on sets vs groups acting on groups.