We let $G$ be a finite group.
If $\chi$ is a complex character of $G$, we define $\overline{\chi}:G \to \mathbb{C}$ by $\overline{\chi}(g)=\overline{\chi(g)}$ for all $g \in G$, and define $\chi^{(2)}:G \to \mathbb{C}$ by $\chi^{(2)}(g) = \chi(g^2)$. We write $\chi_{S}$ and $\chi_{A}$ for the symmetric and alternating part of $\chi$. We note that $\chi_{S}$ and $\chi_{A}$ are characters of $G$ with $\chi^2=\chi_{S} + \chi_{A}$ and $\chi^{(2)}=\chi_{S} – \chi_{A}$. We write
$\nu(\chi):= \frac{1}{|G|}\displaystyle\sum_{g \in G}\chi(g^2)$
for the Frobenius Schur Indicator.
First, let $\chi_{1}$ be the trivial character of $G$, i.e. $\chi_{1}(g)=1$ for all $g \in G$. We want to show that $\langle \chi , \overline{\chi} \rangle= \langle \chi_{S},\chi_{1}\rangle + \langle \chi_{A}, \chi_{1} \rangle$.
We have:
\begin{split}
\langle \chi , \overline{\chi} \rangle &= \frac{1}{|G|}
\displaystyle\sum_{g \in G} \chi(g)\overline{\overline{\chi(g)}}\\ &= \frac{1}{|G|}
\displaystyle\sum_{g \in G} \chi(g)\chi(g)1\\ &= \frac{1}{|G|}\displaystyle\sum_{g \in G} (\chi_{S}+\chi_{A})(g)1 \\ &=\langle \chi_{S}+\chi_{A}, 1 \rangle \\ &= \langle \chi_{S},1 \rangle
+\langle \chi_{A} , 1 \rangle \\ &= \langle \chi_{S},\chi_{1} \rangle
+\langle \chi_{A} , \chi_{1} \rangle
\end{split}
Is this correct?
Next, we let $\chi$ be irreducible. We want to show that $\nu(\chi) \in \{-1,1\}$ if $\chi$ is real-valued, and that $\nu(\chi)=0$ otherwise. Let us start from the 'otherwise' case first. We have:
\begin{split}
\nu(\chi) &:=\frac{1}{|G|} \displaystyle\sum_{g \in G} \chi(g^2)\\
&= \frac{1}{|G|} \displaystyle\sum_{g \in G} (\chi_{S}-\chi_{A})(g) \\
&= \langle \chi_{S},\chi_{1} \rangle – \langle \chi_{A},\chi_{1} \rangle \\
&= \langle \chi , \overline{\chi} \rangle – 2\langle \chi_{A} , \chi_{1} \rangle
\end{split}
and I get stuck here. I think, for the 'otherwise' case, $\langle \chi, \overline{\chi} \rangle = 0$, because we assumed that $\chi$ is irreducible and so it follows that $\overline{\chi}$ is also irreducible and we also know that the irreducible characters form an orthonormal basis (but is it for an arbitrary field?) and so it follows(?). For the real case we'd have that $\langle \chi , \overline{\chi} \rangle = \langle \chi , \chi \rangle =1 $ from irreducibility of $\chi$, but then again, I am still not sure how to deal with $1- 2\langle \chi_{A} , \chi_{1} \rangle$…I'd very much appreciate some help.
Best Answer
I think everything you've done is correct. There are actually two formulas that you can derive for $\nu(\chi)$ (not that it matters): $$ \nu(\chi) = \langle \chi, \bar\chi \rangle - 2\langle \chi_A, \chi_1 \rangle = 2\langle \chi_S, \chi_1 \rangle - \langle \chi, \bar\chi \rangle .$$ And you're right that, since the character is irreducible, $\langle \chi, \bar\chi \rangle$ is $1$ if the character is real, or $0$ if the character is complex.
Now, let's throw in a few more facts.
If $\chi$ is the character for a representation $\rho$ of $G$ on a vector space $V$, then $\chi_S$ is the character for the representation that $\rho$ induces on ${\rm Sym}^2(V)$, and $\chi_A$ is the character for the representation that $\rho$ induces on $\wedge^2 V$.
$\langle\chi_S, \chi_1 \rangle$ is the dimension of the $G$-invariant subspace in ${\rm Sym}^2(V)$, and $\langle \chi_A, \chi_1 \rangle$ is the dimension of the $G$-invariant subspace in $\wedge^2V$.
We have the decomposition ${\rm Hom}(V, V^\star) \cong {\rm Sym}^2(V) \oplus \wedge^2 V$, where ${\rm Hom}(V, V^\star)$ is the vector space of linear maps from $V$ to $V^\star$. Our representation $\rho$ induces a representation on ${\rm Hom}(V, V^\star)$, which reduces to the representations mentioned earlier when restricted to ${\rm Sym}^2(V)$ and $\wedge^2 V$.
Since $\rho$ is irreducible, Schur's lemma tells us the number of $G$-invariant linear maps from $V$ to $V^\star$ is either equal to $1$ (if the representation $\rho$ on $V$ and its dual representation on $V^\star$ are equivalent), or equal to $0$ (if $\rho$ and its dual are inequivalent).
The character of the dual representation is precisely $\bar\chi$.
This should be enough information for you to piece together a proof. Feel free to leave a comment if this is insufficient.