Let X be a vector $\mathbb{R}^{n\times1}$ and M be a constant matrix $\mathbb{R}^{n\times n}$, and given the function $f(X)$ how could i find the derivative of $f(X)$ with respect to $X$? In the expression diag($X$) represents the diagonal matrix of $X$.
$$f(X)=\text{diag}(X)M^T\text{diag}^{-1}(MX)$$
Thank you in advance.
Best Answer
For typing convenience, define the variables $$\eqalign{ \def\bx{\boxtimes} \def\LR#1{\left(#1\right)} \def\qiq{\quad\implies\quad} \def\A{A^{-1}} \def\o{{\tt1}} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\bbR#1{{\mathbb R}^{#1}} \def\m#1{\left[\begin{array}{r}#1\end{array}\right]} \def\D{\operatorname{Diag}} \def\v{\operatorname{vec}} X &= {\rm Diag}(x) \\ A &= {\rm Diag}(Mx) = A^T \\ F &= XM^T\A \\ }$$ and the symbol $\bx$ for the columnar Khatri-Rao product $$\eqalign{ A &= \m{a_1&a_2&\ldots&a_n} \;&\in\bbR{m\times n} \\ B^T &= \m{b_1&\,b_2&\ldots&\,b_n} \;&\in\bbR{p\times n} \\ C &= \m{c_1&\,c_2&\ldots&\,c_n} \;&\in\bbR{mp\times n} \\ C &= \LR{B^T\bx A} \;\iff\; c_k &= \LR{b_k\otimes a_k} \\ }$$ where $\otimes$ is the Kronecker product. The one remarkable property of this product is its ability to vectorize products involving a diagonal matrix $$\eqalign{ &\v\!\big(A\,\D(x)\;B\,\big) \;=\; (B^T\bx A)\,x \\ }$$ Use the above notation to calculate the gradient. $$\eqalign{ dF &= dX\,M^T\A + XM^T\,d\A \\ &= I_n\,\D(dx)\;M^T\A - XM^T\A\,\D(M\,dx)\;\A \\ \\ df &= \v(dF) \\ &= \LR{\A M\bx I_n}dx - \LR{\A\bx XM^T\A}M\,dx \\ \\ \grad{f}{x} &= \LR{\A M\bx I_n} - \LR{\A\bx F}M \\ \\ }$$