[Math] How to take the derivative of a matrix with respect to itself

Question

Could someone please explain how to take the derivative of matrix with respect to itself?

$$\frac{\partial \textbf{X}}{\partial \textbf{X}}$$

where $\textbf{X}$ is an M x N matrix

Answer 1

$\def\p#1#2{\frac{\partial #1}{\partial #2}}$A matrix/matrix gradient produces a 4th order tensor, which is easily evaluated in index notation $$\eqalign{ {\cal E} &= \p{X}{X} \quad\implies\quad {\cal E}_{ijk\ell} &= \p{X_{ij}}{X_{k\ell}} &= \delta_{ik}\delta_{j\ell} \\ }$$ where the Kronecker delta symbol is defined as $$\eqalign{ \delta_{ik} &= \begin{cases} {\tt1}\quad {\rm if}\;i=k \\ 0\quad {\rm otherwise} \end{cases} }$$ In words:   If $X_{ij}$ and $X_{k\ell}$ refer to different elements then the derivative is $0$, otherwise it's $\tt1$.

This is analogous to the vector/vector derivative which produces the identity matrix $$\eqalign{ I &= \p{x}{x} \quad\implies\quad {I}_{ij} &= \p{x_i}{x_j} &= \delta_{ij} \\ }$$