I've been thinking about this question already for a long time and I've just encountered it again in the following lemma:
$$f(x) = g(Ax + b) \implies \nabla f = A^T \nabla g(Ax + b) $$
This lemma makes intuitive sense if you think of it as taking the $x$ to the space $Ax$, calculating the gradient and then taking the result back to the original space. But why is "taking the result back" realised as $A^T$ and not $A^{-1}$?
By doing the calculations you get $A^T$, no doubt, but I always expect an inverse. In general, when should I expect a transpose and when an inverse? Where are they similar and where do they differ?
Best Answer
We usually see matrices as linear transformations. The inverse of $A$, when it exists, means simply "reversing" what $A$ does as a function. The transpose originates in a different point of view.
So we have vector spaces $X,Y$, and $A:X\to Y$ is linear. For many reasons, we often look at the linear functionals on the space; that way we get the dual $$ X^*=\{f:X\to\mathbb R:\ f\ \text{ is linear}\}, $$ and correspondingly $Y^*$. Now the map $A$ induces a natural map $A^*:Y^*\to X^*$, by $$ (A^*g)(x)=g(Ax). $$ In the particular case where $X=\mathbb R^n$, $Y=\mathbb R^m$, one can check that $X^*=X$ and $Y^*=Y$, in the sense that all linear functionals $f:\mathbb R^n\to\mathbb R$ are of the form $f(x)=y^Tx$ for some fixed $y\in\mathbb R^n$. In this situation $A$ is an $m\times n$ matrix, and the matrix of $A^*$ is the transpose of $A$.