In Sheaves in Geometry and Logic the authors write:
In the category $BG = G$-Sets of representations of $G$ [Example (iv)
of §1] an object is an action $X \times G\to X$ of the fixed group $G$ on some
set $X$, and a subobject is just a subset $S\subset X$ closed under this action
… The complement of $S$ in $X$
is thus also invariant under this action, so we can still use the ordinary
characteristic function $\phi_S : X\to 2$ of $S$, where the subobject classifier is
the usual map ${\rm true}: 1\to 2$, with $G$ acting trivially on both sets $1$ and $2$.
What has subobjects being closed under complement to do with the choice of subobject classifier?
After that, they discuss the case of $M$-sets for a monoid $M$, and remark that in this case subobjects aren't closed under complement, so one has to choose another subobject classifier.
To be clear: I understand the these choices of subobject classifier satisfy the definition of subobject classifier, and that in the case of $M$-sets, the subobject classifier can't be given by ${\rm true}:1\to 2$ (since the subobject classifier is unique up to isomorphism), but I just don't understand the explanation with subobjects being closed under complement.
Best Answer
The reason why the characteristic function $\phi_S:X \to 2$ is actually a morphism of $G$-sets is that the complement is also $G$-invariant, I think this should become clear if you write out what it means for the characteristic function to be $G$-equivariant.