Apologies in advance that this is a somewhat vague question. I attempt to make precise below what would constitute an answer for me.
Let $G$ be a topological group and $X$ a topological space on which $G$ acts as a set. I have in my mind two different definitions of the statement "$G$ acts continuously on $X$," and I would like to know when they are equivalent. The real definition, per Wikipedia, is the assertion that the action map
$$\alpha: G\times X \rightarrow X$$
given by
$$(g,x)\mapsto gx$$
is a continuous map. This definition is very concise and conceptually slim. But the following alternative definition feels natural to me too, probably because it consists of the information I actually want to know about the action:
-
For any $g\in G$, the map $\rho_g:X\rightarrow X$ given by $x\mapsto gx$ is a homeomorphism, thus there is a natural group homomorphism $\rho: G\rightarrow \operatorname{Homeo}(X)$ given by $g\mapsto \rho_g$, where $\operatorname{Homeo}(X)$ is the group of homeomorphisms of $X$.
-
Endow $\operatorname{Homeo}(X)$ with the compact-open topology. Then $\rho:G\rightarrow \operatorname{Homeo}(X)$ is a continuous map of topological spaces.
I assume that these definitions are not equivalent at the generality of my setup, since if $X$ is a totally general topological space, I know that weird things can happen. For example, my impression is that $\operatorname{Homeo}(X)$ needn't be a topological group. (While I am unaware of a specific example, there is a theorem due to Richard Arens [Theorem 4 here] stating that if $X$ is locally compact, locally connected hausdorff, then $\operatorname{Homeo}(X)$ with the compact-open topology is a topological group; so presumably this fails in greater generality. Of course, it is possible in principle that the definitions are equivalent even when $\operatorname{Homeo}(X)$ is not a topological group.)
On the other hand, the real definition does imply the alternative definition in full generality (see appendix below), and it seems likely to me that under some "ordinary circumstances" they will coincide. My question is:
Under what topological conditions on $G$ and $X$ do the two above definitions of a continuous group action coincide with each other?
This question is admittedly vague. I would be happy with any set of conditions sufficient for equivalence that are in the language of general (i.e., point-set) topology. For example, "this is true if $G$ and $X$ are locally connected, locally compact hausdorff [or whatever]" would be a satisfying answer, whereas, "this is true if $G$ is a Lie group and $X$ is a manifold" is narrower than what I am looking for.
Appendix: proof that the standard definition implies the alternative one:
Suppose that the action map $\alpha:G\times X\rightarrow X$ is continuous. A "cross-section embedding" $X\hookrightarrow G\times X$ given, for a fixed $g$, by $x\mapsto (g,x)$, is a continuous map. (An open set in $G\times X$ is a union of sets $V\times U$, with $V\subset G$ and $U\subset X$ open, and the pullback in $X$ under this "cross-section embedding" will be the union of just those $U$'s whose corresponding $V$ contains $g$. This is a union of open sets, therefore open.) Therefore, fixing $g$, the composite map
$$ X\hookrightarrow G\times X \xrightarrow{\alpha} X $$
given by
$$ x\mapsto (g,x) \mapsto gx$$
is continuous. This is the map $\rho_g$. It is inverse to $\rho_{g^{-1}}$, which is continuous for the same reason. Thus $\rho_g$ is a homeomorphism for every $g$, in full generality.
Let $K,U\subset X$ be compact and open, respectively, and let $[K,U]\subset \operatorname{Homeo}(X)$ be the set of homeomorphisms $f:X\rightarrow X$ satisfying $f(K)\subset U$. Then $\rho^{-1}([K,U])$ consists of those $g\in G$ such that $\rho_g(K)\subset U$. I would like to know that $\rho^{-1}([K,U])$ is open, since the $[K,U]$'s form a subbase for the topology of $\operatorname{Homeo}(X)$.
Fix an arbitrary $g\in \rho^{-1}([K,U])$. Let $k\in K$ be arbitrary. Then $gk = \rho_g(k)=\alpha(g,k)\in U$, i.e., $(g,k)\in\alpha^{-1}(U)$. Since $\alpha$ is continuous and $U$ is open, $\alpha^{-1}(U)\subset G\times X$ is open, thus $(g,k)$ is contained in a basis open set of the product topology on $G\times X$ that is entirely contained in $\alpha^{-1}(U)$. Let it be $V_{gk}\times U_{gk}$, where $V_{gk}\subset G$ and $U_{gk}\subset X$ are open, and $g\in V_{gk}$ while $k\in U_{gk}$. Keeping $g$ fixed while allowing $k$ to vary across $K$, we get an open cover $\{U_{gk}\}|_{k\in K}$ of $K$. Since $K$ is compact, this cover has a finite subcover $U_{gk_1},\dots,U_{gk_s}$. Let $V=\bigcap_1^s V_{gk_j}$. Because this intersection is finite, $V$ is an open subset of $G$. Since $V\times U_{gk_j} \subset V_{gk_j}\times U_{gk_j}\subset \alpha^{-1}(U)$ for each $j=1,\dots,s$, we have
$$ V\times K \subset V\times \bigcup_1^s U_{gk_j} = \bigcup_1^s V\times U_{gk_j} \subset \alpha^{-1}(U).$$
The first containment is because by construction, $U_{gk_1},\dots,U_{gk_s}$ is a cover of $K$. Translating the containment $V\times K\subset \alpha^{-1}(U)$ in terms of $\rho$, this says that $V\subset\rho^{-1}([K,U])$. Now clearly $g\in V$ since $g\in V_{gk_j}$ for each $j$ (in fact, $g\in V_{gk}$ for each $k$). Thus we have identified an open set of $G$ containing $g$ and contained in $\rho^{-1}([K,U])$. Since $g$ was arbitrary, it follows that $\rho^{-1}([K,U])$ is open. Therefore, $\rho$ is continuous.
This proof that $\rho$ is continuous is a direct adaptation of this proof given by Olivier Begassat in the special case that $X=G$ and the action is regular.
Best Answer
If $X$ is locally compact hausdorff, then the two definitions are equivalent.
Local connectedness appears not to be needed for this, nor do we appear to need any topological constraint on $G$ beyond the setup.
We follow the notational conventions set up in the OP.
Theorem: Assume $X$ is locally compact hausdorff. Then the following are equivalent:
Proof: It was already proven in the OP (without needing either the local compactness or the hausdorffness assumptions) that if $\alpha$ is continuous, then $\rho$ is continuous (and its image lands in $\operatorname{Homeo}(X)$). So, we suppose each $\rho_g$ is continuous, and $\rho$ is continuous, and $X$ is locally compact hausdorff, and aim to establish $\alpha$ is continuous.
Let $U\subset X$ be open, and consider $\alpha^{-1}(U)\subset G\times X$. We have to show it is open. Take an arbitrary $(g,x)\in \alpha^{-1}(U)$; it will suffice to find an open set of $G\times X$ containing $(g,x)$ and contained in $\alpha^{-1}(U)$.
Since by assumption $\rho_g$ is continuous, the set $\rho_g^{-1}(U)\subset X$ is open. Note that it contains $x$, because $(g,x)\in\alpha^{-1}(U)$, or equivalently, $\rho_g(x)\in U$.
Since $X$ is locally compact hausdorff, $x$ has a local base of compact neighborhoods. In particular, there is a compact set $K$ contained in the open set $\rho_g^{-1}(U)$ and containing an open set $U_x$ that in turn contains $x$:
$$x\in U_x\subset K\subset \rho_g^{-1}(U) \subset X.$$
The set $[K,U]\subset \operatorname{Homeo}(X)$ is open, by definition of the topology on $\operatorname{Homeo}(X)$. By supposition, $\rho$ is continuous; therefore $\rho^{-1}([K,U])\subset G$ is open; call it $V$.
Since $K\subset \rho_g^{-1}(U)$, i.e., $\rho_g(K)\subset U$, we have that $g\in \rho^{-1}([K,U])$; in other words, $g\in V$. Thus $(g,x)$ is contained in $V\times U_x$. Meanwhile, by definition of $V$, for any $h\in V$ we have $\rho_h(K)\subset U$, and in particular, $\rho_h(U_x)\subset U$ since $U_x\subset K$. It follows that the entirety of $V$ sends $U_x$ into $U$; in other words, $V\times U_x\subset \alpha^{-1}(U)$.
Now $V\times U_x\subset G\times X$ is open by definition of the product topology, since $V\subset G$ and $U_x\subset X$ are open. Thus we have identified an open set $V\times U_x$ containing $(g,x)$ and contained in $\alpha^{-1}(U)$. We conclude $\alpha$ is continuous.