Let $M$ be a differentiable manifold, and $G$ a Lie group acting smoothly on $M$. Under which condition – if any – is the set of fixed points of the action a submanifold of $M$?
My thoughts so far: if we could determine this set as the preimage of some non-critical value of a diffeomorphism, we would be done. In the easy case $M=\mathbb R^n$, one can consider the function $g\cdot x – x$, with $g\in G$ fixed and $x\in M$ varying, and then pick the preimage of zero. In any case, this gives a larger set than those of all the fixed points: an intersection would be needed, and then who knows what may be happening?
I'm particularly interested in compact Lie groups, and I tried looking for counterexamples, but I can't find any. I have the feeling it shouldn't be too hard to find a set of fixed points self-intersecting in some point (I'm thinking of sth like the axes in $\mathbb R^2$), but I probably shouldn't be trusting feelings.
References, hints, generic intuitions are very welcome!
Best Answer
Let $x \in M$ be a fixed point of a $G$-action, $G$ acting properly on $M$ (meaning the map $G \times M \to M \times M$, $(g,m) \mapsto (m,gm)$, is proper). We'll use the following tubular neighborhood theorem:
If $M$ is a smooth proper $G$-manifold without boundary, and $N$ is a closed $G$-invariant submanifold, then there is an open neighborhood $U \supset N$ which is $G$-equivariantly diffeomorphic to the total space of a $G$-vector bundle over $N$ (that is, a vector bundle over $N$ such that the action of $G$ on $N$ extends to a fiberwise linear action of $G$ on the bundle.) See proposition 4.5 here.
Now, let's kill the fly. Since $x$ is a fixed point, it's a $G$-invariant submanifold. A $G$-vector bundle over it is the same thing as a vector space with a linear action of $G$. The fixed point set of this action is then necessarily a vector subspace. Transporting this back to the neighborhood of $x$, we see that locally near $x$, the fixed point set forms a submanifold, as desired.
(Indeed, the set of points $x$ such that the isotropy group of the action at $x$ contains some fixed closed subgroup $H \subset G$ forms a submanifold by the exact same argument; of course, the example above corresponds to the case $H=G$.)