In the representation theory of $p$-adic groups, intertwining operators of induced representations are commonly defined by integration. For an example, let $G = \operatorname{GL}_2(\mathbb R)$, $T$ the group of diagonal matrices, $U$ the group of upper triangular matrices with $1$s on the diagonal, and $\chi_1, \chi_2$ quasicharacters of $\mathbb R^{\ast}$. If we let $I(\chi_1,\chi_2) = \operatorname{Ind}_{TU}^G\chi_1 \otimes \chi_2$ be the space of continuous functions $f: G \rightarrow \mathbb C$ which satisfy $$f( \begin{pmatrix} t_1 \\ & t_2 \end{pmatrix} u g) = \chi_1(t_1)\chi_2(t_2)|\frac{t_1}{t_2}|^{1/2}f(g)$$
for all $t_i \in \mathbb R^{\ast}, u \in U, g \in G$, then $I(\chi_1,\chi_2)$ is a representation of $G$, with $G$ acting by right translation. We can define an intertwining operator $A: I(\chi_1,\chi_2) \rightarrow I(\chi_2,\chi_1)$ by an integral
$$A(f)(g) = \int\limits_{\mathbb R} f(\begin{pmatrix} 0& 1 \\ 1 & 0 \end{pmatrix} \begin{pmatrix} 1 & x \\ & 1 \end{pmatrix}g)dx.$$
The integral converges for certain choices of $\chi_i$.
How natural is it to define intertwining operators by integrals in this way? For intertwining operators of induced representations for finite groups, do they naturally as integrals (sums)?
Best Answer
Yes, this is natural and is a part of Mackey theory. Suppose $G$ is a finite group and $(\sigma,B)$ and $(\sigma',B')$ are two representations of subgroups $B,B'$ of $G$. Then consider an operator valued function $$A: (B',\sigma') \backslash G / (B,\sigma) \to \mathrm{Hom}(\sigma, \sigma'),$$ by which we mean an operator valued function transforming under $B' \times B$ by the rule $$A(b'xb)=\sigma'(b')A(x)\sigma(b):V(\sigma) \to V(\sigma').$$
Then, by an averaging argument we obtain an intertwiner $$A \mapsto \mathcal{I}(A,\sigma,\sigma'): \mathrm{Ind}_B^G(\sigma)\to \mathrm{Ind}_{B'}^G(\sigma')$$
Where if $\varphi \in \mathrm{Ind}_B^G(\sigma)$, $$(\mathcal{I}(A,\sigma,\sigma')(\varphi))(g) := \sum_{B \backslash G} A(gu^{-1})\varphi(u).$$
Then you can check that $\mathrm{Maps}((B',\sigma') \backslash G / (B,\sigma),\mathrm{Hom}(\sigma, \sigma'))$ is in bijection with $\mathrm{Int}(\mathrm{Ind}_B^G(\sigma),\mathrm{Ind}_{B'}^G(\sigma'))$. Notes that discuss Mackey's intertwining number theorem or something of that sort should explain this. The left hand side of this bijection start to look like the usual vector space that decomposes into a space of $\mathrm{Hom}(\sigma,\sigma')$-valued functions supported on some disjoint union of double cosets $B'xB$. The subspace of functions $A$ with support strictly in $B'xB$ are characterized by the intertwining property $$ (*) \ A(b'x)=A(xx^{-1}b'x)=A(x)\sigma(x^{-1}b'x) = \sigma'(b')A(x)$$ whenever $b' \in B' \cap xBx^{-1}$. Therefore if $A$ is supported on $B'xB$, the intertwiner can be rewritten
\begin{align} \mathcal{I}(A,\sigma,\sigma')(\varphi)(g) &= \sum_{u \in B \backslash G} A(u^{-1})\varphi(ug) \\ &= \sum_{u \in B \backslash Bx^{-1}B'} A(u^{-1})\varphi(ug).\\ &= \sum_{b' \in x^{-1}(B' \cap xBx^{-1}) \backslash B'} \sigma'(b')A(x)\varphi(x^{-1}b'g) \end{align}
where we use the isomorphisms $B \backslash Bx^{-1}B' \cong x^{-1}(B' \cap xBx^{-1}) \backslash B'$. This is the finite group intertwining intergral.
Applying this to the case when $G$ is reductive, $B$ and $B'$ are Borels (or parabolics) with representations $\sigma, \sigma'$ then $(*)$ means that we need $A(x)$ to be an intertwiner between $x \cdot \sigma_{|B' \cap xBx^{-1}}$ and $\sigma'_{|B' \cap xBx^{-1}}$, and if we assume as is usual that $B = x^{-1}B'x$ and $x\cdot \sigma \cong \sigma'$ then $(B' \cap xBx^{-1}) \backslash B' \cong U' \cap xUx^{-1} \backslash U'$ and proceeding formally
$$ \mathcal{I}(A,\sigma,\sigma')(\varphi)(g) = \int_{U' \cap xUx^{-1} \backslash U'} A(x) \varphi(x^{-1}ug)du $$
which should specialize to usual intertwining integrals.