Let $\rho: G \to GL(V)$ and $\sigma: G \to GL(W)$ be representations. The dual representation of $\rho$, namely $\rho ^* $, is defined as follows:
$\rho^*(g)=\ ^t\rho(g^{-1})$.
The reason for this definition is that we want the two representations
of $G$ with respect to the natural pairing between $V^* $ and $V$ satisfy the following condition: $$ \langle \rho ^* (g)(v^* ),\rho(g)(v) \rangle=\langle v^* ,v \rangle . \ \ \ \ \ (I)$$
The representation $\text{Hom}(V,W)$ is defined as follows: $$(g\phi)(v)=g(\phi(g^{-1}v))=\sigma_g(\phi(\rho_{g^{-1}}(v)));\ \ \ \phi \in \text{Hom}(V,W).$$
Now I want to see the reason behind this definition much the same way there is a reason for the definition of dual representations. Therefore, I did as follows.
$\text{Hom}(V,W) \cong V^* \otimes W $. I identify $ \phi \in \text{Hom}(V,W) $ as $\phi \otimes 1 \in V^* \otimes W$ and $(v^* \otimes w)(u)=v^* (u)w $. Hence, $$(g\phi)(v)=\left[ (\rho^* \otimes \sigma)_g(\phi \otimes 1)\right](v) =\rho_g^* (\phi)(v) \sigma_g(1)=$$ $$(\text{because of (I)})\ \phi(\rho_{g^-1}(v))\sigma_g(1)= $$ $$\sigma_g(\phi(\rho_{g^-1}(v)))=$$ $$(g\phi)(v).$$
Is this argument rigorous enough? or does it require more consideration?
Best Answer
Here's a more simple-minded motivation for the definition of the $G$-action on linear maps $\phi:V\to W$. Think of a function like $\phi$ as "officially" a set of ordered pairs (its graph); if $\phi(v)=w$, then $\langle v,w\rangle\in\phi$. Now, to apply some $g\in G$ to $\phi$, just apply it to both components of all the ordered pairs in $\phi$. In other words, if $\langle v,w\rangle\in\phi$, then $\langle g(v),g(w)\rangle\in g(\phi)$. In yet other words, if $\phi(v)=w$ then $g(\phi)(g(v))=g(w)$. Apply this with $v$ replaced by $g^{-1}(v)$ to get the definition of $g(\phi)$ that you quoted.