Short story: I read The Elements of Statistical Learning and got frustrated when I was trying to verify some of the results, e.g., given
$$\text{RSS}(\beta) = \left(\mathbf{y}-\mathbf{X}\beta\right)^{T}\left(\mathbf{y}-\mathbf{X}\beta\right)\text{,}$$
then
$$\begin{align}&\dfrac{\partial\text{RSS}}{\partial \beta} = -2\mathbf{X}^{T}\left(\mathbf{y}-\mathbf{X}\beta\right) \\
&\dfrac{\partial^2\text{RSS}}{\partial \beta\text{ }\partial \beta^{T}} = 2\mathbf{X}^{T}\mathbf{X}\text{.}
\end{align}$$
I am looking for a matrix calculus book which is written like your traditional calculus book (i.e., proofs of theorems, examples, exercises on computation, etc.). I have already seen this question and feel that the text by Magnus and Neudecker focuses too much on the theory, and the text I have by Gentle focuses too little on the theory and too much on the computation side.
Is there a happy medium out there which is accessible to someone with a background in undergraduate analysis?
Best Answer
For most matrix questions I always first refer to "The Matrix Cookbook" (see here).
It is regularly updated due to feedback from various sources. There are proofs contained within, however it is mostly intended as a handbook.