Consider $ J = ||\mathbf{G} – ( \mathbf{B} \otimes \mathbf{X} )||_F^2 $, where $\mathbf{G}$ and $\mathbf{B}$ are complex matrices, and $||.||_F$ is the Frobenius norm. Find the derivative with respect to $ \mathbf{X} $
Note: My question is related to this post: Derivative of a trace with second order Kronecker product. I would like a solution that does not involve SVD decomposition. Any help/hints on how to solve this problem are welcomed.
Best Answer
Let $\langle \cdot , \cdot \rangle$ denote the Frobenius inner product $\langle A,B \rangle = A:B = \operatorname{Tr}(AB^T)$. One approach is to expand $J(X + H)$, then extract the total derivative. In this case, we have $$ J(X + H) = \langle G - (B \otimes (X + H)), G - (B \otimes (X+ H)) \rangle\\ = \langle (G - (B \otimes X)) - B \otimes H, (G - (B \otimes X)) - B \otimes H \rangle\\ = J(X) - 2 \operatorname{Re} \langle G - (B \otimes X),B \otimes H \rangle + o(H). $$ So, the derivative of $J$ with respect to $X$ is $$ J'(X)(H) = - 2 \operatorname{Re} \langle G - (B \otimes X),B \otimes H \rangle. $$ The trick, however, is to extract the matrix form of this derivative. For the numerator-layout derivative, we're looking for a matrix $\frac{\partial J}{\partial X} = M$ (that depends on $X$) for which $\langle M,H \rangle = J'(X)(H)$.
The post you linked explains how this matrix can be found using SVD. For another approach, we can use the fact that $M_{ij} = J'(X)(E_{ij})$, where $E_{ij}$ denotes the matrix with a $1$ as the $E_{ij}$ entry and zeros elsewhere. We therefore have $$ M_{ij} = - 2 \operatorname{Re} \langle G - (B \otimes X),B \otimes E_{ij} \rangle. $$ We can make a bit more sense out of this if we break $G$ into a sum. If $X$ has size $m \times n$, then we can write $$ G = \sum_{i=1}^m \sum_{j=1}^n G_{ij} \otimes E_{ij}, $$ where each $G_{ij}$ has the same size as $B$ (note that $G_{ij}$ is actually a submatrix of $G$). With that, we have $$ M_{ij} = - 2 \operatorname{Re} \left\langle \sum_{i=1}^m \sum_{j=1}^n G_{ij} \otimes E_{ij} - (B \otimes X),B \otimes E_{ij} \right\rangle\\ = - 2 \sum_{i=1}^m \sum_{j=1}^n\operatorname{Re} \left\langle G_{ij} \otimes E_{ij} - (B \otimes X),B \otimes E_{ij} \right\rangle\\ = - 2 \sum_{i=1}^m \sum_{j=1}^n \operatorname{Re}[\langle G_{ij},B \rangle - \langle B,B\rangle x_{ij}]. $$