I'm having trouble solving Exercise 5.19 from Isaacs' book.
The condition of the problem is:
Let $G$ be doubly transitive on $\Omega$ and let $H \subseteq G$ with $|G:H|<|\Omega|$. Show that $H$ is transitive on $\Omega$.
I know various theorems and properties related to transitive and doubly transitive actions described in the fifth chapter of this book. However, I don't understand how to apply them.
I would be very glad of any help with this problem!
Best Answer
Let $G$ act $2$-transitively on the set $\Omega$, $H \lt G$, with $|G:H| \lt \#\Omega$. Let $\chi$ be the corresponding permutation character of $G$. Observe that since $H$ is a subgroup of $G$, the restriction $\chi_H$ is the permutation character of $H$ acting on $\Omega$.
By $(5.17)$ Corollary in Isaacs' CTFG we can write $$\chi=1_G+\psi$$ with $\psi \in Irr(G)$ and $\psi \neq 1_G$. It follows that $\psi(1)=\#\Omega -1$, we will need this later.
Now, by the equation above $\chi_H=1_H+\psi_H$. From this it follows that $[\chi_H,1_H]=1 + [\psi_H,1_H]$. By Frobenius reciprocity, $[\psi_H,1_H]=[\psi, 1_H^G]$. Now assume that indeed $\psi$ is an irreducible constituent of $1_H^G$. Observe that $1_G$ is also an irreducible constituent of $1_H^G$ (different from $\psi$) with multiplicity $1$. Hence $$1_H^G=a\psi + 1_G+ \Delta,$$ with $a$ a non-negative integer and $\Delta$ a character, with $[\Delta,\psi]=0$ or $\Delta \equiv 0$. Then $$1_H^G(1)=|G:H|=a\psi(1)+ 1 +\Delta(1) \geq \psi(1)+1=\#\Omega$$ contradicting $|G:H| \lt \#\Omega$. We conclude that $a=[\psi_H,1_H]=0$ and we can now apply $(5.15)$ Corollary leading to $[\chi_H,1_H]=1$, that is, $H$ acts transitively on $\Omega$. $\square$
Note (added December 4th 2022) If $H \unlhd G$, then you can lift the restriction on the index: in this case either $H$ acts trivially on $\Omega$ (that is $\omega^h=\omega$ for all $h \in H$ which is equivalent to $\chi_H=\#\Omega \cdot 1_H$) or $H$ is transitive on $\Omega$.
For again as above we have $\chi_H=1_H + \psi_H$, with $\psi \in Irr(G)$ and $\psi \neq 1_G$. Now assume for the moment that $1_H$ is an irreducible constituent of $\psi_H$. Then by Clifford Theory (see Chapter $6$ in CTFG, notably $(6.10)$ ff.) and the fact that $1_H$ is $G$-invariant, we get $\psi_H=\psi(1)1_H$. So $\chi_H=1_H+\psi_H=(1+\psi(1)1_H=\#\Omega \cdot 1_H$. This means that $H$ acts trivially on $\Omega$.
If $[\psi_H,1_H]=0$, then it follows that $[\chi_H,1_H]=1$ and we can apply $(5.15)$ Corollary, giving $H$ is acting transitively on $\Omega$.