I know the title is a mouth full, and there have been many similar (and probably more complicated) questions/answers on this site, but I'm stuck on this specific problem.
I am working with the following function:
$$f(x) = \frac{1}{2}||xx^{T} \circ Y||_{2}^{2}$$
where $x \in \mathbb{R}^{n}$, $Y \in \mathbb{R}^{n \times n}$ and $\circ$ denotes the hadamard (element wise) product. I would like to compute the following gradient:
$$\frac{\partial f(x)}{\partial x} = \frac{\partial }{\partial x} \frac{1}{2}||xx^{T} \circ Y||_{2}^{2}$$
The furthest I've gotten is w.r.t. $xx^{T}$, following this:
$$\frac{\partial f(x)}{\partial xx^{T}} = xx^{T} \circ Y$$
but I do not know how to actually compute this w.r.t. $x$. I am pretty inexperienced with this kind of thing, so any input would be really helpful.
Thank you!
Best Answer
For ease of typing, define the matrices $$\eqalign{ W &= xx^T\circ Y \quad\implies\quad dW = \left(dx\,x^T+x\,dx^T\right)\circ Y \\ Z &= Y\circ Y \\ }$$ and use a colon as a product notation for the trace, i.e. $$\eqalign{ A:B &= {\rm Tr}(AB^T) \;=\; \sum_{i=1}^m \sum_{j=1}^n A_{ij} B_{ij} \\ A:A &= \big\|A\big\|^2_F \\ }$$ Write the function in terms of these new variables. Then calculate the differential and gradient. $$\eqalign{ f &= \tfrac 12\,W:W \\ df &= W:dW \\ &= (xx^T\circ Y):(dx\,x^T+x\,dx^T)\circ Y \\ &= (xx^T\circ Y\circ Y):(dx\,x^T+x\,dx^T) \\ &= xx^T\circ\left(Z+Z^T\right):dx\,x^T \\ &= \left(xx^T\circ\left(Z+Z^T\right)\right)x:dx \\ \frac{\partial f}{\partial x} &= \left(xx^T\circ\left(Z+Z^T\right)\right)x \\ }$$ If $\;Y^T=Y\,$ then the gradient can be simplified to $$\eqalign{ \frac{\partial f}{\partial x} &= 2\left(xx^T\circ Y\circ Y\right)x \\ }$$