Below locally compact spaces are assumed to be Hausdorff. The following is essentially a distillate of results from Bourbaki's Topologie Générale, Chapitres II and III.
Definition. A continuous function $f: X \to Y$ is called proper if $f$ maps closed sets to closed sets and $f^{-1}(K)$ is compact for all compact $K \subset Y$.
Remark. If $X$ is Hausdorff and $Y$ is locally compact then a continuous function $f: X \to Y$ is proper if and only if $f^{-1}(K)$ is compact for all compact $K \subset Y$. Moreover, $X$ must be locally compact.
To see this, cover $Y$ with open and relatively compact sets $U_{\alpha}$. Then $f^{-1}(U_{\alpha})$ is an open covering of $X$ by relatively compact sets, hence $X$ is locally compact. If $F \subset X$ is closed then $f(F)$ is closed. Indeed, if $(y_{n}) \subset f(F)$ is a net converging to $y$, then we may assume that all $y_{n}$ are in a compact neighborhood $K$ of $y$. Pick a pre-image $x_{n}$ of each $y_{n} \in f^{-1}(K)$, which is compact by assumption. If $x_{i} \to x \in f^{-1}(K)$ is a convergent subnet of $(x_{n})$ then $(f(x_{i}))$ is a subnet of $(y_{n})$, hence $f(x) = y$ by continuity and thus $y \in f(F)$.
Remark. In the definition of properness it would suffice to require that $f$ is closed and $f^{-1}(y)$ is compact for all $y \in Y$, but the definition above is good enough for the present purposes.
Definition. Let $G$ be a topological group acting continuously on a topological space $X$. The action is called proper if the map $\rho: G \times X \to X \times X$ given by $(g,x) \mapsto (x,gx)$ is proper.
Proposition. If $G$ acts properly on $X$ then $X/G$ is Hausdorff. In particular, each orbit $Gx$ is closed. The stabilizer $G_{x}$ of each point is compact and the map $G/G_{x} \to Gx$ is a homeomorphism. Moreover, if $G$ is Hausdorff then so is $X$.
Proof. Indeed, the orbit equivalence relation is the image of $\rho$, hence it is closed. Since the projection $X \to X/G$ is open, this implies that $X/G$ is Hausdorff. Since the pre-image of the point $[x]$ in $X/G$ is its orbit $Gx$, we see that orbits are closed. The stabilizer $G_{x}$ of a point $x$ is the projection of $\rho^{-1}(x,x)$ to $G$, hence it is compact. The map $G/G_{x} \to Gx$ is proper and $1$-to-$1$, hence a homeomorphism. Finally, if $G$ is Hausdorff, then $\{e\} \times X \subset G \times X$ is closed and therefore the diagonal $\Delta_{X} = \rho(\{e\} \times X)$ of $X \times X$ is closed, hence $X$ is Hausdorff.
Exercise. Let $G$ be a Hausdorff topological group acting properly on a locally compact space $X$. Then $G$ and $X/G$ are both locally compact. If $X$ is compact Hausdorff then so are $G$ and $X/G$.
Replace finite by compact in Type A and Type B. Then we have the following implications for a continuous action:
Proper $\Longrightarrow$ Type A, the converse holds if both $G$ and $X$ are locally compact.
Type A $\Longrightarrow$ Type B.
Let $K \subset X$ be compact. Then $K \times K \subset X \times X$ is compact. Thus, if the action is of type A, then $\rho^{-1}(K \times K) = \{(g,x) \in G \times X\,:\,(x,gx) \in K \times K\} \subset G \times X$ is compact. The projection of this set to $G$ is compact and consists precisely of the $g \in G$ for which $K \cap gK \neq \emptyset$.
Type B $\Longrightarrow$ Type A if $X$ is Hausdorff.
We have to show that $\rho^{-1}(L)$ is compact for every compact $L \subset X \times X$. Let $K$ be the union of the two projections of $L$. Then $(g,x) \in \rho^{-1}(K \times K)$ is equivalent to $x \in K \cap gK$. Since $\rho^{-1}(K \times K)$ is compact and $\rho^{-1}(L)$ is a closed subset of $\rho^{-1}(K \times K)$, we have that $\rho^{-1}(L)$ is compact.
Corollary. If $G$ and $X$ are locally compact, properness, Type A and Type B are all equivalent.
Let me now show that in the locally compact setting properness is equivalent to a refinement of Type C:
Proposition. Let $G$ and $X$ be locally compact and assume that $G$ acts continuously on $X$. The following are equivalent:
- The action is proper.
- For all $x,y \in X$ there are open neighborhoods $U_{x}, U_{y} \subset X$ of $x$ and $y$ such that $C = \{g \in G\,:\,gU_x \cap U_{y} \neq \emptyset \}$ is relatively compact.
Proof. $1.$ implies $2.$ Let
$K_{x}$ and $K_{y}$
be compact neighborhoods of $x$ and $y$. Then the set $\rho^{-1}(K_{x} \times K_{y})$ is compact and its projection to $G$ contains $C$ and is compact. Now let $U_{x}$ and $U_{y}$ be the interiors of $K_{x}$ and $K_{y}$.
$2.$ implies $1$. Let $K \subset X \times X$ be compact. We want to show that $\rho^{-1}(K)$ is compact as well. Let $(g_{n},x_{n})$ be a universal net in $\rho^{-1}(K)$. Then $(x_{n},g_{n}x_{n})$ is a universal net in $K$ and hence converges to some $(x,y) \in K$. Let $U_{x}, U_{y}$ and $C$ be as in $2.$. Then $(x_{n},g_{n}x_{n}) \in U_{x} \times U_{y}$ eventually and thus also $(g_{n}) \subset C$ eventually. Since $(g_{n})$ is universal and $C$ is relatively compact, $(g_{n})$ converges to some $g \in G$. Hence $(g_{n},x_{n})$ converges to $(g,x) \in \rho^{-1}(K)$.
Example.
To see that Type C is weaker than properness, consider $A = \begin{pmatrix} 2 & 0 \\ 0 & 2^{-1} \end{pmatrix}$ and the action of $\mathbb{Z}$ on $\mathbb{R}^{2} \smallsetminus \{0\}$ given by $n \cdot x = A^{n} x$. For instance for $x = \begin{pmatrix} 1 \\ 0 \end{pmatrix}$ and $y = \begin{pmatrix} 0 \\ 1 \end{pmatrix}$ and all neighborhoods $U_{x} \ni x$ and $U_{y} \ni y$ the set $\{n \in \mathbb{Z}\,:\, U_{x} \cap n \cdot U_{y} \neq \emptyset \}$ is infinite. Thus this action isn't proper. On the other hand, it is easy to see that it is of Type C.
Remark.
The previous example shows that properness of an action is not a local property.
Exercise.
If the action of a locally compact group $G$ on a locally compact space $X$ is of type C and $X/G$ is Hausdorff then it is proper.
To finish this discussion, it is evident that an action of type C is also of type E, hence type E is also weaker than properness. Finally, a trivial action is of type D, hence this property has nothing to do with properness.
Here are some references:
I've followed Bourbaki, Topologie Générale, Ch. III, in terminology, and the proofs I've given are variants of Bourbaki's. I happen to like Koszul's Lectures on groups of transformations. If you're looking for a more pedestrian approach, you can find the most important facts in Lee's Introduction to topological manifolds.
If you have any category that admits finite products, you have a notion of $G$-pseudotorsor, namely an object $X$ equipped with a map $act: G \times X \to X$, such that
- $act \circ (id_G, act) = act \circ (m_G,id_X): G \times G \times X \to X$ and $act \circ (e_G, id_X) = id_X: X \to X$ (i.e., it is an action).
- $(act, p_2): G \times X \to X \times X$ is an isomorphism.
An initial object in your category is a pseudotorsor under any group object. I would say that the notion of pseudotorsor is a natural one, but I don't think the name "torsor" should be used. (See EGA IV 16.5.15)
The notion of torsor requires your category to admit a notion of objects being locally isomorphic - in particular, if $X$ is a $G$-torsor, it should be locally isomorphic to $G$, in a $G$-equivariant way. There are equivalent definitions, where you demand the local existence of a section, or you demand that the object be nonempty (non-initial) over all nonempty opens (non-initial objects), and although these conditions are useful in a bootstrapping sense, I don't think they look as natural (Warning: the non-emptiness condition is not global, and this makes a difference over non-connected bases). The isomorphism condition gives the category of torsors a groupoid structure (and base change yields a category fibered in groupoids). The existence of an initial pseudotorsor ensures that the category whose objects are $G$-pseudotorsors and whose morphisms are $G$-equivariant morphisms is not a groupoid, so you have to make a choice of which nice property you need when choosing between pseudotorsors and torsors.
When you work with $G$ as a set-theoretic group, or even as a group object in sets over some other set, the notion of local isomorphism collapses, and you demand that $X$ is in fact $G$-equivariantly isomorphic to $G$. The language of torsors in sets is useful when you want to make sure your constructions are canonical, but I think torsors become more interesting when you can glue locally trivial objects in nontrivial ways. Basic examples include the orientation double cover of a manifold as a $\mathbb{Z}/2\mathbb{Z}$-torsor, whose global trivializations are in bijection with choices of orientation, and the spectrum of a Galois extension of fields as a torsor under the Galois group, where trivializations exist étale-locally, but not Zariski-locally.
It looks like you also want a notion of pseudotorsor with a faithful $G$-action. If I'm not mistaken, you can define faithfulness by finality of the equalizer of $G \times X \overset{act}{\underset{p_2}{\rightrightarrows}} X$. This is equivalent to being a torsor if and only if $G$ is nontrivial.
Best Answer
(This answer started out as a comment, but got too long, and perhaps there is something that may be gleaned that will help the OP)
The problem with this question for me is that you are assuming that such a thing can be done with just an abstract category. This is not possible. The data involved is not just a category but at least a pair of categories $D$, $C$ with a fully faithful inclusion $D\to C$ and a class of maps $E$ in $C$ that 'behave like proper maps'. Then one can specialise to the case of $D = Set$, $C = Mfld$ and $E =$ proper maps.
If you are working in a category where the objects behave like spaces (e.g. an extensive category), then perhaps some ideas can be brought over. For instance, there is a definition of a compact object in a category, but this doesn't agree with the usual definition of compact for the category of topological spaces (whether it does for manifolds is an interesting question that Urs Schreiber is trying to answer at present, at least in the setting of manifolds embedded in $Sh(Mfld)$). One could try to define a proper map in a finitely complete category as a map $f:A \to B$ such that $f^*:CptSub(B) \to CptSub(A)$ preserves compact objects (here $CptSub(A)$ is the subset of the set of subobjects consisting of the compact objects), but I don't know if this class of maps satisfies the conditions that would make it 'behave like proper maps' - it depends on how you define this.
The real question for you is why do you want to do this? What categories are you thinking of applying this abstract description to? If some sorts of categories of spaces, like schemes, algebraic spaces, generalised topological spaces of various sorts, generalised manifolds of various sorts, or even toposes, then there are probably already definitions that will achieve what you want. If you really are set on using an arbitrary category, or using categories of things which aren't like spaces (e.g. an abelian category), then it is hard to see what you are trying to achieve.
I hope this helps.