Calculate gradient of Kronecker product.

kronecker productvector analysis

Assume we have a matrix $M\in \mathbb R^{m\times n}$ and two vectors $z\in \mathbb R^{n}$ and $p\in\mathbb R^{k}$. Then, define $\mathbb 1_{i\times j}$ is $i$-by-$j$ vector whose elements are all 1. $\otimes$ means the Kronecker operator.

How to calculate the gradient of this expression with respect to $z$?
$$
Mz \otimes \mathbb 1_{r\times 1} – \mathbb 1_{s\times1}\otimes p
$$

where $s = mr/k$.

Best Answer

Using the well-known Kronecker-vec formula $${\rm vec}(ABC) = (C^T\otimes A)\,{\rm vec}(B)$$ the outer product of two vectors can be expanded in several equivalent ways $$\eqalign{ {\rm vec}(ab^T) &= {\rm vec}(ab^TI_b) = {\rm vec}(I_aab^T) \\ b\otimes a &= (I_b\otimes a)\,b = (b\otimes I_a)\,a \\ }$$ Apply this result to the current the function and calculate the gradient. $$\eqalign{ y &= Mz\otimes{\tt1}_r - {\tt1}_s\otimes p \\ &= (I_m\otimes{\tt1}_r)(Mz) - {\tt1}_s\otimes p \\ dy &= (I_m\otimes{\tt1}_r)M\,dz \\ \frac{\partial y}{\partial z} &= (I_m\otimes{\tt1}_r)M \\ }$$

Related Question