I have asked a question about the first part of the paragraph in question here, and after thinking about it for a while I determined that the resulting action was indeed a group action. But I came back to this for use in another example, and I am confused again.
The action is by $GL_n(k)$ ($k$ a field) on the polynomial ring $S=k[x_1,\dots,x_n]$, and it is given by $g(f)(x_1,\dots,x_n)=f(g^{-1}(x_1),\dots,g^{-1}(x_n))$, where we view $GL_n(k)$ as the general linear group of span$\{x_1,\dots,x_n\}$.
The problem is, I seem to get contradictory results depending on which order I apply the group action in the case $g(h(f))$. If I interpret $g(h(f))$ as $g$ acting on $h(f)$, then I get $$g(h(f))(x_1,\dots,x_n)=(h(f))(g^{-1}(x_1),\dots,g^{-1}(x_n))=f(h^{-1}g^{-1}(x_1),\dots,h^{-1}g^{-1}(x_n)),$$ where the last equality is from taking $h(f)$ but first replacing the $x_i$'s in $f$ by $g^{-1}(x_i)$. This is the correct result.
I tried to go about this by having $h$ act on $f$ first, and then have $g$ act on that polynomial, but this does not appear to give the correct result. Here is an example of it failing:
Let $S=k[x_1,x_2]$ and $f=x_1$. Also let $g^{-1}$ and $h^{-1}$ be represented by $(a_{ij})$ and $(b_{ij})$ respectively. Then $h(f)(x_1,x_2)=h^{-1}(x_1)=b_{11}x_1+b_{21}x_2$. Then shouldn't $g$ act on $h(f)=b_{11}x_1+b_{21}x_2$ by replacing each $x_i$ by $g^{-1}(x_i)$ (this is the definition…)? Then we get $g(h(f))=b_{11}(a_{11}x_1+a_{21}x_2)+b_{21}(a_{12}x_1+a_{22}x_2)$, which is seen to be the opposite of what would happen if we interpreted this as above.
So, what am I doing wrong when I evaluate it like that? Also, is there a way to evaluate it "outwards" like I tried to do, but actually achieve the desired result?
I reckon that somehow the notation $g(h(f))(x_1,\dots,x_n)$ as opposed to $g(h(f)(x_1,\dots,x_n))$ has something to do with it, but I am not sure.
Best Answer
The problem is that the notation $g^{-1}(x_i)$ looks unambiguous but it's actually not. There are two things you might interpret this to mean and they give two different results.
As usual, things are cleaner if we first don't allow ourselves to work in coordinates. Let $V$ be a finite-dimensional vector space over $k$. There are two polynomial rings you can construct out of $V$, and $GL(V)$ acts on both of them:
Now we pick a basis $v_1, \dots v_n \in V$. This induces a dual basis $f_1, \dots f_n \in V^{\ast}$ defined by $f_i(v_j) = \delta_{ij}$, and these bases allow us to identify our symmetric algebras as polynomial algebras
$$S(V) \cong k[v_1, \dots v_n]$$ $$S(V^{\ast}) \cong k[f_1, \dots f_n].$$
If we now use the basis $v_i$ to identify $GL(V)$ with $GL_n(k)$, then $g \in GL_n(k)$ acts on a sum $\sum c_i v_i \in V$ via matrix multiplication of $g$ on the column vector with entries $c_i$ in the usual way. It also acts on a sum $\sum r_i f_i \in V^{\ast}$, but now via multiplication by the matrix $(g^T)^{-1}$ (note the transpose); said another way, it acts via the action of $g^{-1}$ on the row vector with entries $r_i$.
So, explicitly, if $g^{-1} = \left[ \begin{array}{cc} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array} \right]$, then there are two ways to interpret the meaning of $g^{-1}(x_1)$, depending on whether you're thinking of $x_1$ as $v_1$ or as $f_1$:
If you choose the covariant interpretation ($x_1 = v_1$), which is implied by "general linear group of $\text{span}(x_1, \dots x_n)$," then you don't need to put an inverse in the definition of the action, but you do need to be careful about how you define it. Safest to say the coordinate-invariant thing.