Let $A$ be a real-valued matrix and $$f(A) := \lVert A \rVert_F = \sqrt{A:A} = \sqrt{\mbox{tr}(A^TA)}$$ be its Frobenius norm. What is $\frac{\partial^2f}{\partial A^2}$?
For the squared Frobenius norm, $f^2$, I found that $\frac{\partial{f^2}}{\partial A}=2A$ and $\frac{\partial^2 f^2}{\partial A^2}=2 I\otimes I$. For the case of interest I was able to compute $\frac{\partial f}{\partial A}=A/\lVert A\rVert_F$, however I am not able to deal with the square root in the denominator for the second derivative and would appreciate any advice.
Best Answer
The matrix inner product $\,A:B={\rm Tr}(A^TB)$ can be used to write the Frobenius norm as $$\eqalign{ f^2 &= \|A\|_F^2 = A:A \cr }$$ Find the differential and gradient of this expression. $$\eqalign{ 2f\,df &= 2A:dA \cr df &= \frac{A}{f}:dA \cr G=\frac{\partial f}{\partial A} &= \frac{A}{f} \cr }$$ This matrix gradient can be vectorized. $$\eqalign{ g &= {\rm vec}(G) = \frac{{\rm vec}(A)}{f} = \frac{a}{f} \cr }$$ Note the alternate expressions for the differential of the norm. $$\eqalign{ df &= G:dA \cr df &= g^Tda \cr }$$ Now find the Hessian. $$\eqalign{ dg &= \frac{da}{f} - \frac{a}{f^2}\,df \cr &= \frac{da}{f} - \frac{a}{f^2}\,(g^Tda) \cr &= \frac{1}{f}\Big(I-gg^T\Big)\,da \cr H=\frac{\partial g}{\partial a} &= \frac{1}{f}\Big(I-gg^T\Big) \,\,= \frac{\partial^2f}{\partial a\,\partial a^T} \cr }$$ The Hessian can also be written entirely in terms of $a$ $$\eqalign{ H &= \frac{1}{f^3}\Big(f^2I-aa^T\Big) }$$ Without vectorization, the Hessian is a 4th order tensor, which can be written in component form as $$\eqalign{ H_{ijkl} &= \frac{\partial^2f}{\partial A_{ij}\,\partial A_{kl}} \cr &= \frac{1}{f^3}\Big(f^2\delta_{ik}\delta_{jl}-A_{ij}A_{kl}\Big) }$$ Note the permuted order of the indices on the $\delta$ symbols.