Let $A$ be a commutative unitary ring and $A[t_1, t_2, …, t_n]$ the polynomial ring over $A$ in $n$ variables. The symmetric group of degree $n$ acts on $A[t_1, t_2, …, t_n]$ in a natural way and the ring of invariants under this action is well understood. If we restrict the action to an arbitrary subgroup $G$ of the symmetric group of degree $n$, what are known results for the ring of invariants under this action? Can we classify them in some sense?
The ring of invariants under the action of an arbitrary subgroup of the symmetric group
abstract-algebracombinatoricsgroup-theory
Best Answer
Let $G\subset S_n$ be the permutation group in question. Let $A[\mathbf{t}]:=A[t_1,\dots,t_n]$ as a shorthand.
General results:
(a) The invariant ring $A[\mathbf{t}]^G$ is generated by the invariants of degree at most $\operatorname{max}(n,n(n-1)/2)$, a result usually attributed to Manfred Gobel (see here), although it was actually anticipated by Leopold Kronecker (see section 12 of his paper Grundzuge einer arithmetischen theorie der algebraischen grossen, Crelle, Journal fur die reine und angewandte Mathematik 92:1-122, 1881, reprinted in Werke, vol. 2, 237–387).
(b) If the coefficient ring $A$ is a field of characteristic not dividing the group order $|G|$, then $A[\mathbf{t}]^G$ is free as a module over the subring generated by any homogeneous system of parameters (equivalently, $A[\mathbf{t}]^G$ is Cohen-Macaulay). This result is not specific to permutation groups -- it is a consequence of the Hochster-Eagon theorem. (Though again it happens that Kronecker proved it in the case of a permutation group and a field of characteristic 0.) Then any homogeneous system of parameters for $A[\mathbf{t}]^G$ is called a set of primary invariants, and a module basis over the subring they generate is a set of secondary invariants. There are algorithms based on Grobner bases to compute primary and secondary invariants, again not specific to permutation groups; see the book by Derksen and Kemper. However, in the case of permutation groups, the elementary symmetric polynomials provide a uniform choice for the primary invariants, and there is a method due to Nicolas Borie that aims for more effective computability of the secondary invariants (see here).
(c) There is also a method due to Garsia and Stanton that produces secondary invariants from a shelling of a certain cell complex (specifically, the quotient of the barycentric subdivision of the boundary of an $(n-1)$-simplex by the $G$ action on the simplex's vertices), when such exists (see here). When this shelling exists, the assumption that $A$ be a field of characteristic not dividing $|G|$ becomes superfluous, i.e. the secondary invariants produced by the method give a module basis for $A[\mathbf{t}]^G$ over the subring generated by the elementary symmetric polynomials, entirely regardless of $A$. It is not an easy problem to find the shelling in general, but has been done in specific cases (the original paper by Garsia and Stanton handles the Young subgroups $Y\subset S_n$ [i.e., direct products of smaller symmetric groups acting on disjoint sets of indices], work of Vic Reiner handles alternating subgroups $Y^+\subset S_n$ of Young subgroups $Y$, and diagonally embedded Young subgroups $Y \hookrightarrow Y\times Y \hookrightarrow S_n\times S_n\subset S_{2n}$, and work of Patricia Hersh handles the wreath product $S_2\wr S_n\subset S_{2n}$). There is a detailed development of Garsia and Stanton's shelling result in my thesis, sections 2.5 and 2.8, along with a discussion of its connection to Gobel's work (see last paragraph) and some speculation about generalizations.
(d) From (b) you can see that $A[\mathbf{t}]^G$ has a nice structure of free-module-over-polynomial-subring when $A$ is a field of characteristic not dividing $|G|$, but from (c) you can see that sometimes this nice structure still exists even when $A$ doesn't satisfy this (e.g. perhaps it is $\mathbb{Z}$, or else a field whose characteristic does divide $|G|$). There is a characterization, due to myself and Sophie Marques, of which groups $G\subset S_n$ have the property that this structure in $A[\mathbf{t}]^G$ exists regardless of $A$. It turns out to be the groups generated by transpositions, double transpositions, and 3-cycles.
(Our paper is framed in the language of Cohen-Macaulay rings and is focused on the situation that $A$ is a field. To see that my claim about "any $A$" in the previous paragraph follows, one shows that if for a given $G$, the described structure obtains for $A$ any field, then it also obtains with $A=\mathbb{Z}$ -- this is supposedly well-known, but "just in case", it is written down carefully in section 2.4.1 of my thesis -- and then one notes that a free module basis of $\mathbb{Z}[\mathbf{t}]^G$ over the subring generated by the elementary symmetric polynomials will also be a free module basis of $A[\mathbf{t}]^G$, just by base changing to $A$. See this MSE question for why the base change doesn't mess anything up.)
(e) As lisyarus stated, the special case of $G=A_n$ is well-understood: the invariant ring is generated by the elementary symmetric polynomials and the Vandermonde determinant. Actually this requires the hypothesis that $2$ is a unit in $A$, as you note in comments. If $2$ is not a unit in $A$, one can still generate the invariant ring with the elementary symmetric polynomials and the sum of the positive terms in the Vandermonde determinant (or, the sum of the negative terms). Certain other cases, e.g. $D_4$, also have explicit descriptions coming from Galois theory. The classical material usually assumes $A$ is a field, but see sections 5.4 and 5.5 in Owen Biesel's thesis for $A_n$ and $D_4$; Biesel is working over general $A$.
I am not sure what you're hoping for in terms of a classification theorem.
(This was really probably more of an MO than an MSE question in the end.)