Reading chapter 2.4 of linear representations of finite groups by Serre, can someone create an example for me to understand what's going on?
Let the irreducible characters be denoted by $\chi_i$ and their degrees are $n_i$. Let the order of $G$ be $|G|$ and let $V$ be a vector space of dimension $|G|$ with a basis $\left\lbrace e_t \right\rbrace$ for $t \in G$. Define the linear map of $\rho_s$ from $V$ into $V$ as $\rho_s(e_t) = e_{st}$, which is called the regular representation R. Its degree is equal to the order of $G$.
Proposition 5
The character $r_G$ of the regular representation $R$ is given by
\begin{align*}
r_G(1) &= |G| \\
r_G(s) &= 0 \qquad \text{if } s\neq 1
\end{align*}Proof
If $s=1$, we have
$$
Tr(\rho_s)=Tr(1)=\dim(R)=| G|
$$
If $s\neq 1$, then $st\neq t$ and $\rho_s(e_t) = e_{st} \neq e_t$ for all $t$ so all diagonal terms of the matrix for $\rho_s$ are zero.Corollary 6
Every irreducible representation $W_i$ is contained in the regular representation with multiplicity equal to its degree $n_i$.
Proof
From theorem 2.3.3, this number equates to $(r_G|\chi_i)$
$$
(r_G|\chi_i)=\frac{1}{\,|G|} \sum_{s \in G} r_G(s)\chi_i(s)^* = \chi_i(1)^* = n_i
$$Remark:
Hence we notice, there is only a finite number of irreducible representations.Corollary 7
(i) The degrees $n_i$ satisfy $\sum_{i=1} n_i^2 =|G|$
(ii) If $s \in G$ is different from $1$, we have $\sum_{i=1} n_i \chi_i(s) =0$
Proof
By corollary 2.4.2, we have $r_G(s) = \sum n_i \chi_i(s)$ for all $s \in G$. For $(i)$ take $s=1$ and for $(ii)$ take $s\neq1$.
\end{proof}Corollary 8
The matrix entries of the unitary irreducible representations form an orthogonal basis for the set of all functions on $G$.Proof
We know that the matrix entries are all orthogonal as functions. There are $\sum n_i^2=|G|$ of them, and this is the dimension of the vector space of all functions.
Best Answer
As @Dane suggested, lets consider the group $G = S_3$ the symmetric group on $3$ letters $1,2,3$. We will consider propositions $5,6$ here. Let $\mathsf{k}$ be some field. Consider the six dimensional $\mathsf{k}$-vector space $V$ with basis $e_0$, $e_{(12)}$, $e_{(23)}$, $e_{(13)}$, $e_{(123)}$, $e_{(132)}$. All we have done here is indexed the basis by the elements of $S_3$. We then define a representation $\rho$ of $G$ by $\rho(g)(e_{h}) =e_{gh}$. So for example
$$ \rho((12))e_{(23)} = e_{(132)}, \quad \rho((13))e_{(23)} = e_{(123)}, \quad \rho((12))e_{(12)} = e_0. $$
Let's explicitly calculate the matrix of $\rho(12)$ with respect to the ordered basis $e_0$, $e_{(12)}$, $e_{(23)}$, $e_{(13)}$, $e_{(123)}$, $e_{(132)}$. Note that
$$ \begin{array}{c | c c } g & e_0 & e_{(12)} & e_{(23)} & e_{(13)} & e_{(123)} & e_{(132)} \\ \hline \rho((12))e_{g} & e_{(12)} & e_0 & e_{(132)} & e_{(123)} & e_{(13)} & e_{(23)} \end{array} $$
and so
$$ \rho((12)) = \begin{pmatrix} 0 & 1 & 0 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0\end{pmatrix} $$
so note that $\chi((12)) = 0$, where $\chi = \operatorname{trace} \circ \rho$. This is what is called $r_G$ in your post. You might notice that
$$ \chi(g) = \left| \left\{ h \in S_3 \mid gh = h \right\} \right| = \left|\operatorname{Fix}_{g}(S_3) \right|. $$
Hence indeed
$$ \chi(g) = \left\{ \begin{array}{ll} \left| G \right| = 6, & \text{if} \ g = e_0, \\ 0, & \text{otherwise} \end{array} \right. $$
Now we have a representation of $S_3$ given by the sign representation. That is $\rho_{\operatorname{sign}} : S_3 \to \mathsf{k}^{\times}$ such that
$$ \rho_{\operatorname{sign}}(g) = \left\{ \begin{array}{ll} 1, & \text{if} \ g \ \text{is a product of an even number of transpositions,} \\ -1, & \text{if} \ g \ \text{is a product of an odd number of transpositions.} \end{array} \right. $$
Now we will find this a subrepresentation of $V$. Consider the vector
$$ v_{\operatorname{sign}} = e_0 - e_{(12)} - e_{(23)} - e_{(13)} + e_{(123)} + e_{(132)} \in V. $$
and define $V_{\operatorname{sign}} = \mathsf{k} \cdot v_{\operatorname{sign}}$, then I claim that this is a subrepresentation. But it is easy to check that $S_3$ acts on $v_{\operatorname{sign}}$ by $\pm 1$ so this is indeed a subrepresentation and whats more its easy to check that this is indeed the sign representation. It is also easy to check that this is the unique subrepresentation of $V$ isomorphic to the sign representation.