Let us assume that
\begin{equation}
f(X)=\mbox{tr}\left(XCX^TXCX^T\right),
\end{equation}
in which $C\in\mathbb{R}^{r\times r}$ is a symmetric matrix, and $X \in \mathbb{R}^{r'\times r}$. From the Page 13 of the book The Matrix Cookbook, we know that
\begin{equation}
\frac{\partial f}{\partial X}=4CXX^TCX.
\end{equation}
Now, let us consider the following function:
\begin{equation}
g(X)=\mbox{tr}\left(BXCX^TB^TBXCX^TB^T\right),
\end{equation}
where $B\in\mathbb{R}^{m\times r'}$. What is the derivative of $g$ with respect to $X$ (i.e., $\frac{\partial g}{\partial X}$)?
Best Answer
First of all, the gradient of $f$ is wrong and it should be $$ \frac{\partial f}{\partial \mathbf{X}} = 4(\mathbf{X}\mathbf{C}\mathbf{X}^T)\mathbf{X}\mathbf{C} $$ Now let $\mathbf{Y}=\mathbf{BX}$, so that your new function writes $g(\mathbf{X})=f(\mathbf{Y})$.
Using this fact, it follows (using the same notations as Greg) that the differential of $g$ writes \begin{eqnarray*} dg &=&4(\mathbf{Y}\mathbf{C}\mathbf{Y}^T)\mathbf{Y}\mathbf{C}:d\mathbf{Y} \\ &=&4\mathbf{B}^T(\mathbf{Y}\mathbf{C}\mathbf{Y}^T)\mathbf{Y}\mathbf{C}:d\mathbf{X} \end{eqnarray*} so the gradient of $g$ is $4\mathbf{B}^T(\mathbf{Y}\mathbf{C}\mathbf{Y}^T)\mathbf{Y}\mathbf{C}$