Find the smallest nontrivial conjugacy class in $S_n$.
For small $n$, the answer can be found by counting the permutations of each possible cycle type. The result is:
$$\begin{array}{ccc}
n & \text{smallest nontrivial class(es)} & \text{size} \\
\hline
1 & \text{none} & \\
2 & \text{transpositions} & 1 \\
3 & 3\text{-cycles} & 2 \\
4 & \text{double transpositions} & 3 \\
5 & \text{transpositions} & 10 \\
6 & \text{transpositions, triple transpositions} & 15
\end{array}$$
For $n\geq 7$, I think the unique smallest nontrivial conjugacy class in $S_n$ is given by the transpositions.
The conjugacy classes in $S_n$ are given by the sets of permutations of the same cycle type. For the cycle type $\alpha = 1^{a_1} 2^{a_2} \ldots$, the size of the respective conjugacy class is known to be
$$N_\alpha = \frac{n!}{\prod_i (i^{a_i} a_i!)}.$$
In particular, the number of transpositions in $S_n$ is $\binom{n}{2}$.
In this way, it remains to show the following:
Let $n \geq 7$ and $\alpha$ be a partition of $n$ distinct from $1^n$ and $2^1 1^{n-2}$. Show that $N_\alpha > \binom{n}{2}$.
So far I can only come up with a lengthy and technical case-by-case study.
I'm looking for a more elegant proof of the statement.
Best Answer
I don't think there is a very conceptual approach giving a meaning to the comparison of conjugacy class sizes, but I think there is a way to make the argument quite short. The first thing to do is to cancel the term $a_1!$ for $i=1$ (for the points remaining fixed) in the denominator against a part of the numerator $n!$, which leaves the numerator equal to $n(n-1)\ldots(a_1+1)$ (the number of factors is the sum of the sizes of the cycles for the conjugacy class, in the group-theoretic sense where cycles cannot have length$~1$).
A first easy simplification rids us the case of mixed cycle structures, classes with is more than one different cycle length${}\geq2$. For such cycle types we can map permutations in the class to another non-trivial cycle type by simply omitting the cycle(s) of the longest length$~l$ from the decomposition. This map commutes with any conjugation by a permutations (which simple relabels elements in the cycles) and is therefore surjective to a single new conjugacy class; moreover it is not injective since there is always more than one way to reconstruct the length$~l$ cycle(s). Therefore such classes are never the smallest non-trivial ones.
The next step is to show that we need only deal with the case of transpositions, possibly flying in formation. More precisely (and with three exceptions that do not occur for $n\geq7$), the class with $m$ disjoint $k$-cycles with $k>2$ is larger than the one with $m$ disjoint $2$-cycles. We have $a_k=m$ and factor our formula as $$ \frac{n(n-1)\ldots(n-mk+1)}{k^m}\times\frac1{m!}, $$ where the second factor is identical to the one for $k=2$. In the first factor replacing $k$ by $2$ means dropping the last $m(k-2)$ factors from the numerator and dividing the denominator by $(k/2)^m$. Using that a product of $k-2$ distinct positive integers can only be${}\leq k/2$ if $k\leq4$, we see that this decreases the first factor except when $a_1=0$, and $(m,k)\in\{(1,3),(1,4),(2,3)\}$, which happens only for $n=3,4,6$ (it explains that $3$-cycles are least numerous for $n=3$, that for $n=4$ there are as many $4$-cycles as $2$-cycles, namely $6$, and that for $n=6$ there are fewer ($40$) pairs of $3$-cycles then pairs of transpositions $(45)$).
We have reduced to the case of $m$ disjoint transpositions, which we shall compare with isolated transpositions ($m=1$). For fixed $m$, increasing $n$ leaves the denominator unchanged and increases all factors in the numerator by$~1$. This gives more relative increase for $m>1$ than for $m=1$, so if the class with $m$ disjoint transpositions is to be less numerous than that with one transposition, it must already be the case for the minimal possible value of $n$, which is $n=2m$. The formula for the number of products of $m$ disjoint transpositions when $n=2m$ is the product $1\times 3\times\cdots\times(2m-1)$ (which some people, defying the ambiguity, write as $(2m-1)!!$) which is${}\leq\binom{2m}2$ only for $m=2,3$; this explains these cases for $n=4,6$. For $n\geq7$ this does not happen any more.