This isn't a complete answer, but I thought it would be useful to write it out in more detail than would fit in the space provided.
First, I'd like to observe that both of the equivalent definitions of group action admit generalisations:
The definition using a homomorphism $G \to \text{Aut}(X)$ generalises straightforwardly to the case when $G$ is a (discrete) group and $X$ a vector space, giving rise to the notion of linear representations of groups.
The definition using a map $G \times X \to X$ generalises straightforwardly to the case when $G$ is a topological group and $X$ a topological space, giving rise to the notion of continuous group actions on topological spaces.
Your question seems to be asking whether we can find a definition which would encompass both branches of generalisations. It's not an unreasonable question — after all, both of the examples above converge when we have a continuous linear representation of a topological group.
Let's try to formulate categorically the equivalence of the two definitions in $\textbf{Set}$, and see how things go when we try to change to a more general category. Firstly, we note that a small group is equivalently
- a category $\mathcal{G}$ enriched over $\textbf{Set}$ with one object and all arrows invertible, and
- a set $G$ equipped with maps $m : G \times G \to G$, $e : 1 \to G$, $i : G \to G$ satisfying the group axioms.
The connection between the two definitions is expressed by the equation $\mathcal{G}(*, *) = G$, where $*$ is the unique object in $\mathcal{G}$. Using the first definition, a group action of $G$ is simply any functor $\mathscr{F} : \mathcal{G} \to \textbf{Set}$. Focusing on the hom-sets, we see that we have a monoid homomorphism $G \to \textbf{Set}(X, X)$, where $X = \mathscr{F}(*)$, and since the domain is a group, the codomain must lie in $\text{Aut}(X) \subseteq \textbf{Set}(X, X)$. Thus we have the first definition of group action.
Now, we recall that $\textbf{Set}$ is a cartesian closed category (indeed, a topos), so in particular we have the exponential objects $Y^X$, which are defined by following the universal property: $\text{Hom}(Z \times X, Y) \cong \text{Hom}(Z, Y^X)$ naturally in $Z$ and $Y$. Thus, we may identify the map $G \to \textbf{Set}(X, X)$ with a map $G \times X \to X$, and translating the homomorphism axioms through this identification gives the second definition of group action.
Consider a group object $G$ in a cartesian monoidal category $(\mathcal{C}, \times, 1)$. This may be viewed as a category $\mathcal{G}$ enriched over $\mathcal{C}$. Then, an action of $G$ on an object $X$ in another category $\mathcal{D}$ enriched over $\mathcal{C}$ is simply a $\mathcal{C}$-enriched functor $\mathcal{G} \to \mathcal{D}$. If $\mathcal{D}$ is such that there is a $\mathcal{C}$-enriched "forgetful" functor $U : \mathcal{D} \to \mathcal{C}$ and $\mathcal{C}$ is a cartesian closed category, we may do the same trick as before and obtain an arrow $G \times X \to X$ in $\mathcal{C}$.
In particular, if $\mathcal{C}$ is a cartesian closed category, it is enriched over itself. Indeed, consider the counit $\epsilon_{Z,X} : Z^X \times X \to Z$ of the product-exponential adjunction. If we compose with $\text{id} \times \epsilon_{X,Y} : Z^X \times X^Y \times Y \to Z^X \times X$, we get a map $Z^X \times X^Y \times Y \to Z$, which we may take transpose to obtain a map $Z^X \times X^Y \to Z^Y$, which is the composition of arrows. Thus we may recover the notion of continuous group actions at least in the case where both the group and the space being acted on are compactly-generated Hausdorff spaces...
Best Answer
As the discussion at the linked nLab page notes, the word action can have various meanings and related applications in category theory. The phrase action of a group is more precise, but still susceptible of two distinct but equivalent framings.
We are first given a group $G$ and a category $C$ that has an object $x$. What is defined is an action of group $G$ on such an object $x$. It is said to be "a representation of $G$ on $x$, that is a group homomorphism $\rho:G \to \operatorname{Aut}(x)$". Note that this allows the trivial homomorphism, one that sends all elements of $G$ to the identity arrow on $x$.
Also the category $C$ might as well be reduced to just the one object $x$ in this context, because $\operatorname{Aut}(x)$ depends only on certain arrows from $x$ to itself.
The "more sophisticated" approach restates this idea by "treat[ing] the group $G$ as a category denoted $\mathbf B G$ with one object, say $*$." Here the arrows of category $\mathbf B G$ are the elements of $G$, the composition of arrows is given by group multiplication, and the identity arrow on $\mathbf B G$ corresponds to the identity element of $G$. This construction allows us to concisely define an action of group $G$ as a functor $\rho:\mathbf B G \to C$.
Verifying the equivalence of these two approaches is straightforward once the claim is established that a group essentially amounts to a one-object category in which every arrow has an inverse, as previously shown on Math.SE.
We are able to uniquely identify the object $x$ in category $C$ as the image of object $*$ in category $\mathbf B G$, and likewise the identity arrow on $x$ must correspond to the identity arrow on $*$. Because the arrows of $\mathbf B G$ are the elements of group $G$, the functor $\rho$ sends arrows of $\mathbf B G$ to arrows from $x$ to $x$ in category $C$, and because the group elements are invertible, these arrow images have inverse arrows in $C$, also from $x$ to $x$, and thus these arrow images belong to $\operatorname{Aut}(x)$.
It's hard to be sure I'm not overlooking an important point that causes doubts here about equivalence of the two approaches. A rigorous step-by-step demonstration strikes me as overkill, but if more detail is needed I'd be glad to supply them.