I was following Stephen Boyd's convex optimisation course and came across the following slide:
Can somebody explain to me how the gradient was calculated for the quadratic and least-squares objective. Is there a general method to find the gradient of a matrix?
Best Answer
$f$ is an normal real valued function. If you want you can write it componentwise as
$$f(x) = {1\over 2}\sum_j\sum_k p_{jk}x_jx_k + \sum_j q_jx_j + r$$
Now the first double sum contains the $x_jx_k$ term twice if $j\ne k$ and if $j=k$ it becomes an $x_j^2$ term, so the derivate with respect to $x_j$ becomes:
$$f'_j(x) = \sum p_{jk}x_k + q_j$$
Which in matrix notation becomes
$$\nabla f(x) = Px + q$$