Let $\mathbf{x}$, $\mathbf{y}$ and $\mathbf{z}$ are $n$-dimensional column vector, and
$$f = \mathbf{x}\circ \mathbf{y} \circ\mathbf{z}$$
Here $\circ$ is the element-wise Hadamard product. Then how to compute the gradient
$\frac{\partial f }{\partial\mathbf{x}}$, $\frac{\partial f }{\partial\mathbf{y}}$, and $\frac{\partial f }{\partial\mathbf{z}}$?
My solution is
$$\frac{\partial f }{\partial\mathbf{x}} = \mathbf{y} \circ\mathbf{z}, \frac{\partial f }{\partial\mathbf{y}} = \mathbf{x} \circ\mathbf{z},\frac{\partial f }{\partial\mathbf{z}} = \mathbf{x} \circ\mathbf{y}$$
Is it correct?
thanks.
Best Answer
You can replace the Hadamard product of a vector by a matrix product with the diagonal matrix formed from that vector, e.g. let $$\eqalign{ X &= {\rm Diag}(x), \quad Y &= {\rm Diag}(y), \quad Z &= {\rm Diag}(z) \\ }$$ then the function can be written in a number of ways $$\eqalign{ f &= x\circ y\circ z \\&= YZx = ZXy = XYz \\ }$$ The differential of the function is $$\eqalign{ df &= YZ\,dx + ZX\,dy + XY\,dz \\ }$$ yielding the gradients $$\eqalign{ \frac{\partial f}{\partial x} &= YZ, \quad \frac{\partial f}{\partial y} &= ZX, \quad \frac{\partial f}{\partial z} &= XY \\ }$$ Since diagonal matrices commute, the order of multiplication can be reversed.