I am trying to solve the derivative:
$$\frac{\partial Tr\,[AX'(X \oslash B)]}{\partial X},$$
where $\oslash$ is the symbol used for the Hadamard division (or element-wise division) and A is a square matrix. I think that the derivative should be:
$$(XA + XA') \oslash B,$$
but I couldn't find a reference to double check it and I recently started doing similar calculations. Any hints?
Best Answer
Let's use a colon to denote the trace/Frobenius product, i.e. $$\eqalign{A:B={\rm Tr}(A^TB)}$$ Properties of the trace give rise to rules for rearranging terms in a Frobenius product, such as $$\eqalign{ A:B &= B:A \cr&= A^T\!:\!B^T \cr A:BC &= B^TA:C \cr&= AC^T:B \cr }$$ Also note that the Hadamard product commutes with itself and the Frobenius product $$\eqalign{ A\odot B &= B\odot A \cr A:(B\odot C) &= (A\odot C):B \cr&= (B\odot A):C \cr }$$ Instead of $B$ let's use its Hadamard inverse, and while we're at it, let's define a few other matrices. $$\eqalign{ C &= 1\oslash B \cr Y &= X^TA \cr Z &= X\odot C = X\oslash B \cr }$$ to write the cost function. Then find its differential and gradient. $$\eqalign{ \phi &= XA^T:X\odot C \cr&= Y:Z \cr d\phi &= Z:dY + Y:dZ \cr &= Z:dX\,A^T + Y:dX\odot C \cr &= ZA:dX + Y\odot C:dX \cr &= (ZA + Y\odot C):dX \cr \frac{\partial\phi}{\partial X} &= ZA + Y\odot C \cr &= (X\oslash B)A + (XA^T)\oslash B \cr }$$