I want to prove that the only finite subgroups of $GL_2(\mathbb Q)$ are $C_1, C_2, C_3, C_4, C_6, V_4, D_6, D_8,$ and $D_{12}$.
First, we determine all possible finite orders of elements. Now, an element of order $n$ will have a minimal polynomial that divides $x^n-1$, so its (complex) roots will be distinct, and so the matrix will be diagonalizable over $\mathbb C$. This implies that both eigenvalues are $n$th roots of unity, and that at least one is primitive, and so the minimal polynomial will be the $n$th cyclotomic polynomial. But since the minimal polynomial can only have degree $1$ or $2$, since we're dealing with $2\times 2$ matrices, the only possible orders are those $n$ for which $\phi(n)=1$ or $2$, so the only possible orders are $1, 2, 3, 4$, and $6$. Thus, if $G$ is a finite subgroup of $GL_2(\mathbb Q)$, then $|G|=2^a3^b$.
Now, since $G$ contains a Sylow-$3$ of order $3^b$, and any $3$-group contains subgroups of every possible order, once we show that $C_3\times C_3$ is not a subgroup of $GL_2(\mathbb Q)$, then we can conclude that $b=0$ or $1$, since we already saw that $C_9$ cannot be a subgroup.
WLOG, let $g=\begin{bmatrix} 0&-1\\1&-1\end{bmatrix}$, which is the Rational Canonical Form for the minimal polynomial $x^2+x+1$, and so has order $3$. We seek to show that there is no matrix $h$ such that $h$ has order $3$, commutes with $g$, and isn't a power of $g$. So assume $h=\begin{bmatrix} a&b\\c&d\end{bmatrix}$. Then $gh=\begin{bmatrix} -c&-d\\a-c&b-d\end{bmatrix}$ and $hg=\begin{bmatrix} b&-(a+b)\\d&-(c+d)\end{bmatrix}$. Equating these, we get that $c=-b, d=a+b$, so $h=\begin{bmatrix} a&b\\-b&a+b\end{bmatrix}$. For $h$ to have order $3$, the trace must be $-1$, and the determinant must be $1$, just like in the Rational Canonical Form, so $2a+b=1$, and $a^2+ab+b^2=1$. The solutions are $a=-1, b=1$ and $a=0, b=-1$. The latter gives $h=g$ and the former gives $h=g^2$, and so there can be no subgroup isomorphic to $C_3\times C_3$, and thus $9$ does not divide the order of the group.
Now, the next step would be to restrict the exponent of $2$, but I'm not quite sure how to do this. One part of the problem tells me to show that the only noncyclic abelian subgroup is $V_4$, the Klein-$4$ group. So if we let $g=\begin{bmatrix} 0&1\\1&0\end{bmatrix}$ be the RCF of $x+1$, then the only order $2$ matrices that commute with it are $-g$ and $-g^2=-I$, which form a Klein-$4$ group. This means that $C_2^3$ is not a subgroup. Also, if we get $g=\begin{bmatrix} 0&-1\\1&0\end{bmatrix}$ be the RCF of $x^2+1$, then the only order $2$ element that commutes with it is $-I=g^2$, so there is also no subgroup isomorphic to $C_4\times C_2$. In a previous exercise, I showed that $Q_8$ is not a subgroup of $GL_2(\mathbb R)$, and thus isn't a subgroup over $\mathbb Q$ either.
But this doesn't seem to prevent subgroups of order $16$, since $D_8$ is in fact a subgroup. Checking all $14$ groups of order $16$, it seems that all of them have an order $8$ subgroup besides $D_8$, but is there a cleaner way to rule out groups of order $16$ without classifying them, since I'm apparently supposed to conclude that the order of $G$ divides $24=2^3\cdot 3$ from the fact that the Klein-$4$ group is the only noncyclic abelian subgroup.
So now, if I assume as known that $24$ divides the order of $G$, then the possible orders for $G$ are $1, 2, 3, 4, 6, 8, 12$, and $24$. I can find $C_1, C_2, C_3, C_4, C_6, V_4, D_6, D_8,$ and $D_{12}$. This takes care of all the orders except $12$ and $24$. I can show that $C_{12}$ and $C_{6}\times C_2$ are not subgroups, but I'm not sure how to rule out the nonabelian groups of order 12, $A_4$ and $C_3\rtimes C_4$, or the groups of order $24$.
So in summary, I'm a bit stuck in ruling out the nonabelian groups of order $12, 16,$ and $24$. Is there a more elegant way to do this than to look at the classifications of these groups and find subgroups which I have already shown to be impossible?
Best Answer
One approach is the following: Let $M = M_{2}(\mathbb{Q}).$ Note that if $x$ is an element of order $4$ or $3$ in $G,$ then the subalgebra $C_{M}(x)$ is a division ring by Schur's Lemma, as $x$ acts irreducibly. Hence $C_{G}(x)$ is cyclic. In particular, Sylow $3$-subgroup of $G$ is cyclic, of order $3$ as you have shown already. Let $S$ be a Sylow $2$-subgroup of $G,$ and let $A$ be a maximal Abelian normal subgroup of $S$. If $A$ has exponent $2,$ then $A$ is elementary Abelian, and hence has order at most $4.$ Otherwise, $A$ contains an element $x$ of order $4,$ and then $A$ is cyclic as $C_{G}(x)$ is cyclic. Also, as you have shown, $|A| \leq 4$ in that case. Now as $A$ is maximal Abelian normal in $S,$ we have $C_{S}(A) = A,$ and $S/A$ is isomorphic to a subgroup of ${\rm Aut}(A).$ If $A$ is a Klein $4$-group, then ${\rm Aut}(A) \cong S_{3}$ and if $A$ is cyclic of order $,$ then ${\rm Aut}(A)$ has order $2.$ Hence in either case, $|S| \leq 8.$ Since you have already ruled out a quaternion group of order $8,$ the only possibility if $S$ is non-Abelian of order $8$ is that it is dihedral ( and we already know that if $|S| = 8,$ it is non-Abelian).
By the way, you can't rule out all non-Abelian group of order $12,$ since a dihedral group with $12$ elements does occur.
I don't know how much group theory you know, but I would finish as follows: let $F = F(G),$ the Fitting subgroup of $G.$ If $F$ contains an element of order $3$ or then $C_{G}(F) = F$ is cyclic of order $3$ or $6$ (for $F$ contains a central element of order $3$ in that case). Then $G/F$ has order dividing $2$. I won't give all details, but if $|G/F| =2,$ that leads to $G$ dihedral with $6$ or $12$ elements ($G=F$ leads to $G$ cyclic of order $3$ or $6$ in this case).
There remains the case that $F$ is a $2$-group. If $F$ is Abelian , we have $F$ cyclic of order $4$ or a Klein $4$-group. The first possibility leads to $G = F.$ The second also leads to $G = F,$ eg by Clifford's theorem. there can be no element of order $3$ in $G$ in that case. If $F$ is non-Abelian of order $8,$ then we must also have $G = F$, because a dihedral group of order $8$ has no automorphism of order $3.$