I am trying to differentiate the Kronecker product of a vector with respect to itself:
$$ \frac{\partial}{\partial \pmb{\delta}} (\pmb{\delta} \otimes \pmb{\delta} ) $$
That is part of this larger differentiation:
$$
\begin{equation}
\label{eq1}
\tag{1}
\frac{\partial}{\partial \pmb{\delta}} \left(\pmb{\delta}^\intercal \mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \right)
\end{equation}
$$
Where:
-$\pmb{\delta}_{K \times 1}$ is a vector and $\pmb{\delta}^{\intercal}_{1\times K}$ is its transpose.
-$ \mathbf{A}_{K \times K^2}$ is constant matrix that is independent from $\pmb{\delta}$ ($ \partial \mathbf{A}/ \partial \pmb{\delta}=0) $.
Using definitions that can be found in Matrix Calculus textbooks, I have simplified it as follows:
$$
\begin{split}
\frac{\partial}{\partial \pmb{\delta}} (\pmb{\delta} \otimes \pmb{\delta} )
&=\frac{\partial \pmb{\delta}}{\partial \pmb{\delta}} \otimes \pmb{\delta}+ [\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][\frac{\partial \pmb{\delta}}{\partial \pmb{\delta}} \otimes \pmb{\delta}][\mathbf{I}_1 \otimes \mathbf{I}_{1\times 1} ]\\
&=vec(\mathbf{I}_K) \otimes \pmb{\delta}+[\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}]\\
&=[\mathbf{I}_{K^3}+\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}]
\end{split}
$$
Where:
- $\mathbf{I}_n$ is $n \times n$ unit matrix
- $vec(\mathbf{B})$ is the classic vectorization operator.
- $\mathbf{U}_{r \times s}$ is the permutation matrix as it is defined in classic matrix calculus textbooks (e.g. here is shown with $T_{m,n}$).
Can the above equation get simplified further? My goal is to collect whatever form of $ \pmb{\delta}$ that appears as a result of equation \ref{eq1} to the right hand side of the term. To be more precise the furthest I got from \ref{eq1} is here:
$$
\begin{equation}
\label{eq2}
\tag{2}
\begin{split}
\frac{\partial}{\partial \pmb{\delta}} \left(\pmb{\delta}^\intercal \mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \right)&
=\mathbf{A} (\pmb{\delta} \otimes \pmb{\delta} ) \\
&+
[\mathbf{I}_K \otimes (\pmb{\delta}^\intercal \mathbf{A})][\mathbf{I}_{K^3}+\mathbf{I}_K \otimes \mathbf{U_{K^2 \times K^2}}][vec(\mathbf{I}_K) \otimes \pmb{\delta}]
\end{split}
\end{equation}
$$
and I am trying to collect the $\pmb{\delta}$ in the second term of the right hand side of above equation on the right hand side like the first term.
Best Answer
For ease of typing, let's use $\,x={\pmb\delta}\,$ as the independent variable.
We'll also use the fact that the Kronecker product of two vectors can be expanded in two ways: $$a\otimes b = (I_a\otimes b)\,a = (a\otimes I_b)\,b$$ where $I_a$ is the identity matrix whose dimensions are compatible with the $a$ vector, while $I_b$ is compatible with the $b$ vector.
The differential and Jacobian of first function can be calculated as $$\eqalign{ y &= x\otimes x \cr dy &= dx\otimes x + x\otimes dx = \big(I\otimes x + x\otimes I\big)\,dx \cr J = \frac{\partial y}{\partial x} &= I\otimes x + x\otimes I \cr }$$ Using the previous result $\big(dy=J\,dx\big),\,$ your second function can be easily dispatched $$\eqalign{ \phi &= x^TAy = y^TA^Tx \cr d\phi &= x^TA\,dy + y^TA^T\,dx \cr &= \big(x^TAJ + y^TA^T\big)\,dx \cr &= \big(Ay + J^TA^Tx\big)^T\,dx \cr \frac{\partial\phi}{\partial x} &= Ay + J^TA^Tx \cr &= A(x\otimes x) + \big(I\otimes x^T + x^T\otimes I\big)A^Tx \cr }$$ The first term is the same between you, Frank, and me.
My second term appears to be the transpose of Frank's and different from yours.