I am given a symmetric $n\times n$-matrix $A$ and a vector $v$, and now I have to compute the gradient and the Hessian of
$$f(x) = (x^TAx)^2-(x^TAv)^2.$$
I guess that the gradient is
$$\nabla f (x) = 4x^TAxx^TA – 2x^TAvv^TA$$
and the Hessian is
$$\nabla^2 f(x) = 12Axx^TA-2Avv^TA.$$
However, when I do the calculation by hand (i.e. first doing the matrix vector multiplications and then taking the derivatives), the results for the Hessian do not match. Are the derivatives above correct? And if not, how do I compute them correctly?
Best Answer
Note that not all $x$ in the first term in the formula for $\nabla f$ have equal standing. Two of the are multiplying one matrix $A$ from both sides, but the third is multiplying the other matrix $A$ that isn't acting on any vector.
The correct formula for $\nabla^2f$ is $$\nabla^2f(x) = 8Ax x^TA + 4(x^TAx) A - 2Avv^TA $$