Let $H$ and $N$ be groups, where $N$ is abelian. Suppose that $H$ acts on $N$ by automorphisms, and consider the semidirect product $G=H\ltimes N$. I am busy classifying all irreducible representations of $G$, as in Section 8.2 by Serre. Since $H$ acts on $N$, $H$ also acts on the character group $\widehat{N}$ by $(h\cdot\chi)(n)=\chi(\phi_{h^{-1}}(n))$. Let $(\chi_i)_{1\leq i\leq k}$ be representatives of the orbits of the $H$-action. For each $i$, let $H_i$ denote the stabilizer subgroup of $\chi_i$ and define $G_i=H_i\ltimes N$ as a normal subgroup of $G$. The function $\chi_i^\circ:G_i\to\mathbb{C}$ given by $\chi_i^\circ(h,n)=\chi_i(n)$ is a linear representation of $G_i$ since $H_i$ stabilizes $\chi_i$.
Now, let $\rho$ be any irreudcible representation of $H_i$, and define $\tilde{\rho}$ to be the irreducible representation of $G_i$ given by $\rho\circ pr_{H_i}$, where $Pr_{H_i}$ is the natural projection on $H_i$. Then define the representation $\theta_{i,\rho}$ of $G$ by $\theta_{i,\rho}=\text{Ind}_{G_i}^G(\tilde{\rho}\otimes\chi_i^\circ)$. By Mackey's irreducibility criterion, each $\theta_{i,\rho}$ is irreducible. Also, $$
\sum_{i=1}^k\sum_{\rho\in S_i}\dim(\theta_{i,\rho})^2=|G|,
$$
where $S_i$ is the set of isomorphism classes of irreducible representations of $H_i$. To know that I have really classified all irreducible representations of $G$, I need to prove that if $\theta_{i,\rho}\cong\theta_{i',\rho'}$, then $i=i'$ and $\rho\cong\rho'$. There is a proof of this in Serre (Proposition 25 $ii$), but I don't understand what is going on. Can anyone explain the proof in more detail?
I also tried to come up with my own proof, but it seems to break down. I tried to use character theory. First, I followed Serre's argument to show that $i=i'$. To show that $\rho$ and $\rho'$ are isomorphic, we show that their characters agree. Let $\chi_{\theta_{i,\rho}}$ and $\chi_{\theta_{i,\rho'}}$ denote the characters of $\theta_{i,\rho}$ and $\theta_{i,\rho'}$. Since these are isomorphic, we know that $\chi_{\theta_{i,\rho}}=\chi_{\theta_{i,\rho'}}.$ Since $G_i$ is a normal subgroup of $G$, we have that $$
\chi_{\theta_{i,\rho}}(h',n')=\frac{1}{|G_i|}\sum_{(h,n)\in G}\chi_i^\circ\big((h,n)^{-1}(h',n')(h,n)\big)\chi_{\tilde{\rho}}\big((h,n)^{-1}(h',n')(h,n)\big)=\frac{1}{|G_i|}\sum_{(h,n)\in G}\chi_i^\circ\big((h,n)^{-1}(h',n')(h,n)\big)\chi_{\rho}(hh'h^{-1})$$
and
$$
\chi_{\theta_{i,\rho'}}(h',n')=\frac{1}{|G_i|}\sum_{(h,n)\in G}\chi_i^\circ\big((h,n)^{-1}(h',n')(h,n)\big)\chi_{\tilde{\rho}'}\big((h,n)^{-1}(h',n')(h,n)\big)=\frac{1}{|G_i|}\sum_{(h,n)\in G}\chi_i^\circ\big((h,n)^{-1}(h',n')(h,n)\big)\chi_{\rho'}(hh'h^{-1})
$$
I am stuck on what to do next, since I can't simply simplify $\chi_\rho(hh'h^{-1})=\chi_\rho(h')$. Thanks in advance for the time
Best Answer
In this answer I will try to explain in details the proof of Proposition 25 in Serre's book. For my convenience in writing this, I will use Serre's notation so my $A$ is your $N$, i.e. $G=H\ltimes A$ and $G_i=H_i\ltimes A$ where $H_i=\{h\in H:h\chi_i=\chi_i\}$ subgroup of $H$.
As the statement said, we consider character $\chi$ of $\theta_{i,\rho}$ for $a\in A$. Recall the character for induced representation for $s\in G$, $H$ subgroup of $G$, $f:H\to \mathbf{C}^{\times}$, is
$$\text{Ind}_H^G(f)(s)=f'(s)=\frac{1}{|H|}\sum_{t\in G, t^{-1}st\in H}f(t^{-1}st)$$ Hence, \begin{align*} \chi(a) & = \frac{1}{|G_i|}\sum_{t\in G, t^{-1}at\in G_i} \chi_{\chi_i\otimes \tilde{\rho}}(t^{-1}at), \\ & = \frac{1}{|G_i|}\sum_{t\in G} \chi_i(t^{-1}at)\chi_{\tilde{\rho}}(t^{-1}at), \; (A \text{ normal so } t^{-1}at\in A\subset G_i \; \forall t\in G), \\ & = \frac{\text{dim }V}{|G_i|}\sum_{t\in G}\chi_i(t^{-1}at), \; (t^{-1}at\in A \Rightarrow \tilde{\rho}(t^{-1}at)=\rho(1)=\text{id}_V), \\ & = \frac{\text{dim }V}{|G_i|} \sum_{a'\in A,h\in H}\chi_i((a'h)^{-1}a(a'h)),\\ & = \frac{\text{dim }V}{|G_i|} \sum_{a'\in A,h\in H} \chi_i(h^{-1}ah), \; (A \text{ abelian}) \\ & = \frac{|A|\text{dim }V}{|G_i|} \sum_{h\in H} (h\chi_i)(a). \end{align*} Note that $h\chi_i\in X=\text{Hom}(A,\mathbf{C}^{\times})$ is an irreducible representation of $A$ so the above implies that the restriction of $\theta_{i,\rho}$ to $A$ is the direct sum of representations corresponding to $h\chi_i$. Note that $h\chi_i$'s lie in the orbit $H\chi_i\subset X$, which is disjoint with $H\chi_{i'}$ for $i\ne i'$. Hence, if $i\ne i'$, the restriction of $\theta_{i,\rho}$ and $\theta_{i',\rho'}$ to $A$ are not isomorphic.
Recall from $\S 7.1$ of Serre's book, $W$ can be identified with $W=\mathbf{C}[G]\otimes_{\mathbf{C}[G_i]}V$ where $V$ is the representation space of $\chi_i\otimes \tilde{\rho}$. In particular, this is a $\mathbf{C}[G]$-module where $g$ acts on $g'\otimes v$ by $(gg')\otimes v$. Also note that, $(gg_i)\otimes v$ and $g\otimes (\chi_i\otimes \tilde{\rho})(g_i)v$ are considered the same in $W$ for $g_i\in G_i$.
Now, we identify $W_i$. For $a\in A, g\otimes v\in W$, we have $\theta_{i,\rho}(a)(g\otimes v)= (ag)\otimes v = g \otimes (g^{-1}ag)v$. Since $g^{-1}ag\in A$ so $(\chi_i\otimes \tilde{\rho})(g^{-1}ag)v=\chi_i(g^{-1}ag)v$. Thus, $\theta_{i,\rho}(a)(g\otimes v)= \chi_i(g^{-1}ag) (g\otimes v)$. Thus, in order for $\theta_{i,\rho}(a)(g\otimes v)=\chi_i(a)(g\otimes v)$ for all $a\in A$, we must have $\chi_i(a)=(g\chi_i)(a)$ for all $a\in A$. This follows $g\in H_i$. Thus, $W_i=\mathbf{C}[H_i] \otimes_{\mathbf{C}[G_i]} V$.
With this, it is obvious that $W_i$ is stable under $H_i$. Furthermore, observe $W_i$ is spanned by $1\otimes v_j$ where $v_j$'s basis of $V$, $1$ identity in $H_i$ (this holds since $h\otimes v=1\otimes \rho(h)v$ for all $h\in H_i,v\in V$). Hence, one can easily construct isomorphism representation of $H_i$ in $W$ to $\rho$, as desired.