I am studying matrix calculus for linear regression and machine learning and I would like to know exactly if the following calculations are correct:
Let $y=\sin(x+yz)$ and $r=\begin{bmatrix}x\\y\\z\end{bmatrix}$
Then the following is the gradient of $y$ i.e., the derivative of $y$ with respect the vector $r$ in denominator layout: $$\frac{\partial y}{\partial r}=\frac{\partial\sin(x+yz)}{\partial r}=\begin{bmatrix}\frac{\partial \sin(x+yz)}{\partial x}\\\frac{\partial\sin(x+yz)}{\partial y}\\\frac{\partial \sin(x+yz}{\partial z}\end{bmatrix}=\begin{bmatrix} \cos(x+yz)\\z\cos(x+yz)\\y\cos(x+yz)\end{bmatrix}$$
In numerator layout it would be:
$$[\cos(x+yz), z\cos(x+yz), y\cos(x+yz)]$$
Now, let $$\mathbf{y}=\begin{bmatrix} e^{xyz}\\x^2z\\yx\end{bmatrix}$$
So in numerator layout: $$\frac{\partial\mathbf{y}}{\partial r}=\begin{bmatrix} \frac{\partial e^{xyz}}{\partial x} & \frac{\partial e^{xyz}}{\partial y} & \frac{\partial e^{xyz}}{\partial z}\\ \frac{\partial x^2z}{\partial x} & \frac{\partial x^2z}{\partial y} & \frac{\partial x^2z}{\partial z}\\ \frac{\partial yx}{\partial x} & \frac{\partial yx}{\partial y} & \frac{\partial yx}{\partial z}\end{bmatrix}\\ =\begin{bmatrix} yze^{xyz} & xze^{xyz} & xye^{xyz}\\2xz& 0 & x^2 \\y & x & 0\end{bmatrix} $$
In denominator layout it would be the transpose of the above matrix?
Now, let
$$Y=\begin{bmatrix} x^2yz & xy^2z \\xyz^2 & \ln(xyz) \end{bmatrix}=\begin{bmatrix} Y_{11} & Y_{12} \\Y_{21} & Y_{22} \end{bmatrix}$$
Then $$\frac{\partial Y}{\partial r}=\begin{bmatrix} \frac{\partial Y_{11}}{\partial x} & \frac{\partial Y_{12}}{\partial x} & \frac{\partial Y_{11}}{\partial y} & \frac{\partial Y_{12}}{\partial y} & \frac{\partial Y_{11}}{\partial z} & \frac{\partial Y_{12}}{\partial z} \\ \frac{\partial Y_{21}}{\partial x} & \frac{\partial Y_{22}}{\partial x} & \frac{\partial Y_{21}}{\partial y} & \frac{\partial Y_{22}}{\partial y} & \frac{\partial Y_{21}}{\partial z} & \frac{\partial Y_{22}}{\partial z}\end{bmatrix}$$
Would this be correct? I'm worrying I mixed up the shape and/or the positions of the members of the matrices $\frac{\partial\mathbf{y}}{\partial r}$ and $\frac{\partial Y}{\partial r}$.
Best Answer
There are different layout notations and each one is correct if used consistently. So, the first one is correct under denominator layout because the output dimension follows the denominator variable's dimension. The second one is correct in numerator layout. But, the third one is wrong because the derivative of a vector wrt a matrix and vice versa is a 3D tensor, not a matrix.