I have the following two functions of a symmetric $k\times k$ positive-definite covariance matrix $\boldsymbol\Sigma = \begin{pmatrix}s_{11} & \cdots & s_{1k}\\ \vdots & \ddots & \vdots \\ s_{k1} & \cdots & s_{kk}\end{pmatrix}$:
$$f_1(\boldsymbol\Sigma) = \ln |\boldsymbol\Sigma|$$
$$f_2(\boldsymbol\Sigma) = -\frac12(\textbf{x}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\textbf{x}-\boldsymbol\mu).$$
My question is: what is the derivatives of these two functions w.r.t. matrix elements $s_{ij}$? That is, how to calculate:
$$\frac{\partial f_1}{\partial s_{ij}}\;\;\;\;\;\;\text{and}\;\;\;\;\;\;\frac{\partial f_2}{\partial s_{ij}}?$$
I'd be happy also simply to an answer which redirects to a book/source where these questions are addressed. Thank you!
Best Answer
For ease of typing, define the variables $$w=(x-\mu),\qquad S=\Sigma$$ We'll also need the following results for an arbitrary matrix ($A$) and function ($f$) $$\eqalign{ dA^{-1} &= -A^{-1}dA\,A^{-1} \\ d\,{\rm Tr}\Big(f(A)\Big) &= {\rm Tr}\Big(df(A)\Big) = {\rm Tr}\Big(f'(A)\cdot dA\Big) \\ \log\Big(\det(A)\Big) &= {\rm Tr}\Big(\log(A)\Big) + 2n\pi i\\ }$$ Applying this to the first function $$\eqalign{ f_1 &= \log\Big(\det(S)\Big) \\ df_1 &= d\log\Big(\det(S)\Big) \\ &= d\,{\rm Tr}\Big(\log(S)\Big) \\ &= {\rm Tr}\Big(S^{-1}\,dS\Big) \\ \frac{\partial f_1}{\partial S} &= S^{-1} \quad\implies\quad \frac{\partial f_1}{\partial s_{ij}} &= \big(S^{-1}\big)_{ij} \\ }$$ and to the second function $$\eqalign{ f_2 &= -\tfrac 12 w^TS^{-1}w \\ df_2 &= -\tfrac 12 w^T\color{red}{dS^{-1}}w \\ &= +\tfrac 12 w^T\color{red}{S^{-1}dS\,S^{-1}}w \\ &= \tfrac 12\,{\rm Tr}\Big(S^{-1}ww^TS^{-1}dS\Big) \\ \frac{\partial f_2}{\partial S} &= \tfrac 12 S^{-1}ww^TS^{-1} \quad\implies\quad \frac{\partial f_2}{\partial s_{ij}} &= \tfrac 12\big(S^{-1}ww^TS^{-1}\big)_{ij} \\ }$$