Consider, for example, the map $f: \mathbb{R}^{n \times n} \rightarrow \mathbb{R}^{n \times n}, f(A) = A^2.$ Then its differential is $df(A)(T) = AT+TA$. I would like a reference that states what this differential means and then how to obtain such results, but not necessarily in a completely rigorous way. I also understand that differentials can be defined and manipulated in the usual way for functionals (e.g. for the Lagrangian, leading to the Euler-Lagrange equations) and I'd like to see this done without developing the whole machinery of variational calculus.
In short, I'm looking for a clear treatment of differentials of operator-valued functions. I've tried looking up books on matrix calculus, calculus on normed vector spaces and variational calculus but haven't found anything suitable (the closest option was Cartan's Differential Calculus, but I'd like something more concrete). Where do people learn this sort of thing?
Best Answer
Just compute the directional derivative, as you would in ordinary calculus. $df(A)(T) = \lim\limits_{h\to 0} \dfrac{f(A+hT)-f(A)}h$. Just do the matrix computation: \begin{align*} \frac{f(A+hT)-f(A)}h &= \frac{(A+hT)^2-A^2}h = \frac{h(AT+TA) + h^2T^2}h \\ &= (AT+TA) + hT^2 \to AT+TA \quad\text{as}\quad h\to 0. \end{align*} The point is that it's nothing different from calculus in Euclidean space, since the space of matrices is naturally a finite-dimensional Euclidean space.
Aside from other texts mentioned, Dieudonné's Treatise on Analysis is a standard reference. Differential Calculus in normed spaces appears in Volume 1.