I am interested in orbits of the action of a group scheme on a scheme and I'm particularly interested in the following special case: Let $k$ be an algebraically closed field, let $G$ be an affine algebraic group over $k$ and let $X$ be an affine $k$-variety. Then it's a basic fact that orbits $Gx$ are locally closed subsets of $X$. Let us now pass to schemes, i.e., $G$ is an affine group $k$-scheme acting on an affine $k$-scheme $X$. How are orbits defined in this setting? Are these simply the set-theoretic orbits? Is the orbit of $G$ in a closed point of $X$ the same as above? (I'm confused because I think in this case the orbit consists entirely of closed points which seems wrong). Is it a locally closed subset and thus canonically a subscheme of $X$? If so, what is the relation between $G(k)x$ and $(Gx)(k)$ for a closed point $x$?
All this should be well known but I couldn't find references. In Mumford's GIT there is a very general definition of orbits (as scheme-theoretic images of a morphism obtained by the action, see Definition 0.4) but I don't know what this gives in my special basic setting above. If someone can deduce it from there, this would be awesome.
Best Answer
Presumably you meant to assume the schemes are finite type over $k$. To work naturally with orbit questions for such schemes one just has to bring in appropriate use of flatness to adapt intuition and experience from the traditional smooth setting over algebraically closed fields (e.g., one uses the robust theory of quotients modulo possibly non-smooth closed subgroup schemes).
For the orbit of a $k$-point $x \in X$ everything is fine: one has the orbit morphism $G \rightarrow X$, and its constructible image is locally closed because this can be checked over $\overline{k}$ (where we can pass to underlying reduced schemes). Likewise, for such $x$ one has the (possibly not smooth) stabilizer scheme $G_x$ inside $G$ over $k$ and a well-defined monomorphism $j_x:G/G_x \rightarrow X$. This is a locally closed immersion when $G$ is smooth (so in such cases we denote its smooth locally closed image with reduced structure as $Gx$, onto which the orbit map from $G$ is faithfully flat). Indeed, by descent theory we can extend the ground field to be algebraically closed, and then if $Z$ is the locally closed orbit with reduced structure we see that the orbit map factors through $Z$ and as such $G \rightarrow Z$ must be generically flat and hence faithfully flat, whence $j_x$ is an isomorphism onto $Z$ by descent theory.
Mere surjectivity of an orbit map (without flatness) is not a good notion of "homogeneous space".
In general with $x \in X(k)$ there is an absolutely huge gulf between $(Gx)(k)$ and $G(k)x$ (e.g., think about a surjective homomorphism between smooth affine groups!). In effect, say assuming $G$ is smooth so that the orbit is $G/G_x$ as shown above, you're asking about the existence of $k$-points in fibers of $\pi:G \rightarrow G/G_x$ over $k$-points of the target. Since $\pi$ is a $G_x$-torsor for the fppf topology, the obstructions lie in the fppf cohomology set ${\rm{H}}^1(k, G_x)$ (almost by definition), and in favorable cases one can analyze that by using knowledge of the structure of $G_x$.
Note however that considering the "orbit" of a point that is not a $k$-point doesn't quite make sense and is asking for trouble unless one is absolutely clear about definitions (and usually it is not what you want to ever consider -- it is technically usually much better to simply extend the ground field to work with orbits through rational points, and to use descent theory to go back down if you wish). However, there is a kind of substitute. You could contemplate $G$-stable non-empty locally closed subschemes $Z$ of $X$ such that the $G(\overline{k})$-action on $Z(\overline{k})$ is transitive and the surjective orbit map $G_{\overline{k}} \rightarrow Z_{\overline{k}}$ through some (equivalently, any) $z \in Z(\overline{k})$ is flat or equivalently identifies $Z_{\overline{k}}$ with $G_{\overline{k}}/(G_{\overline{k}})_z$. (If $G$ is smooth and $Z$ is geometrically reduced then this is equivalent to transitivity of the action on $\overline{k}$-points, for reasons explained above.) In effect, such $Z$ could be regarded as (non-empty) homogeneous spaces for $G$ inside $X$, and as an "orbit" for $G$ through any of closed point of $Z$ (it is easily checked that such a $Z$ is uniquely determined by any single closed point $x$ that it contains, but using the notation $Gx$ for such a $Z$ is going to lead to mistakes if $x \not\in X(k)$, so don't do that). If $Z(k)$ is empty then one should tread carefully to call such $Z$ an "orbit"; better to just say "homogeneous space".
Of course, for this viewpoint of homogeneous spaces inside $X$ (possibly without $k$-points) to be useful, say assuming $G$ is smooth to avoid some confusion, you want to prove an existence property: for any closed point $x \in X$ is there such a (necessarily smooth) $Z$ through $x$, or in other words does the orbit of $G_{\overline{k}}$ through a $\overline{k}$-point over $x$ descend to a locally closed subscheme of $X$?
The existence of such $Z$ fails most of the time when $G$ is not connected and $k$ is not algebraically closed. For example, let $G = \mu_n$ acting on $X = {\rm{GL}}_1$ by scaling with ${\rm{char}}(k) \nmid n$ and suppose $x \in X$ is a closed point such that $[k(x):k]$ does not divide $n$. In such cases, there is no way for $x$ to lie in a $k$-finite closed subscheme of $X$ of order dividing $n$, so there is no meaningful notion of $Gx$ over $k$. Even if $k$ is separably closed but not algebraically closed (say with characteristic $p > 0$) you're going to have problems when $G$ is finite etale since such a $Z$ would have to also be finite etale and hence cannot be found containing $x$ such that $k(x)$ is not separable over $k$ (so you get more counterexamples when $X = \mathbf{A}^1_k$ on which $G = \mathbf{Z}/(p)$ acts by translation, even when $[k(x):k] = p$ so one cannot use the more elementary degree obstruction).
In view of those basic counterexamples, it is an interesting exercise with Galois descent to prove that such a $Z$ does exist when $k(x)$ is separable over $k$ provided that the smooth $G$ is also connected. (The fun part is to figure out where you use connectedness.) If $k(x)$ is not separable over $k$ (and $G$ is smooth and connected) then this can fail. For instance, if $k$ is imperfect with characteristic $p$ and $t \in k - k^p$ then $G := \{v^p = u - tu^p\}$ is a smooth connected 1-dimensional $k$-subgroup of $\mathbf{G}_a^2$ and its Zariski closure $X$ in $\mathbf{P}^2_k$ has 1 point $x$ on the line at $\infty$, with $k(x) = k(t)$, so the $G$-action on $\mathbf{P}^2_k$ via $(u,v).[a,b,c] = [a + uc, b + vc, c]$ restricts to an action on $X$ in which $X - \{x\} = G$ with the translation action. There is no reasonable sense in which the $G$-stable complement $\{x\}$ of $X - \{x\}$ in $X$ can be regarded as a homogeneous space for $G$ since it is not $k$-smooth (and in particular, there is no $Z$ as above in this case).