I have been trying to find an analytical expression for the following:
$\frac{\partial {X^{+}}}{\partial {X}}$
In my case, $X$ has a constant rank.
I've found the formula for differentiating a pseudoinverse in Goulob's paper (equation 4.12):
$$
\frac{\mathrm d}{\mathrm d x} A^+(x) =
-A^+ \left( \frac{\mathrm d}{\mathrm d x} A \right) A^+
+A^+ A{^+}^T \left( \frac{\mathrm d}{\mathrm d x} A^T \right) (1-A A^+)
+ (1-A^+ A) \left( \frac{\mathrm d}{\mathrm d x} A^T \right) A{^+}^T A^+
$$
but I can't see how to input the original matrix. I've tried
$$
\frac{\mathrm d}{\mathrm d A} A^+(A) =
-A^+ \left( \frac{\mathrm d}{\mathrm d A} A \right) A^+
+A^+ A{^+}^T \left( \frac{\mathrm d}{\mathrm d A} A^T \right) (1-A A^+)
+ (1-A^+ A) \left( \frac{\mathrm d}{\mathrm d A} A^T \right) A{^+}^T A^+
$$
but it doesn't seem to give the right answers. What should $\frac{\mathrm d A^T}{\mathrm d A}$ and $\frac{\mathrm d A}{\mathrm d A}$ evaluate to?
Best Answer
A matrix-matrix gradient will be a 4th order tensor. So it's easier to start with the differential, since that is an ordinary matrix.
For ease of typing, I'll use $P$ for the pseudoinverse of $A$. It's differential is $$\eqalign{ dP &= PP^T\,dA^T\,(I-AP) + (I-PA)\,dA^T\,P^TP -P\,dA\,P \cr\cr }$$ Next we need to factor out $\,dA,\,$ for which we'll need two special 4th order tensors $$\eqalign{ {\mathcal B}_{ijkl} = \delta_{il}\,\delta_{jk} \cr {\mathcal E}_{ijkl} = \delta_{ik}\,\delta_{jl} \cr }$$ These tensors can be used to transpose matrices and to rearrange the order of matrix products $$\eqalign{ X^T &= {\mathcal B}:X \cr A\,X\,B &= A{\mathcal E}B^T:X \cr }$$ where colon represents the inner (Frobenius) product.
Continuing with the pseudoinverse $$\eqalign{ dP &= \Big(PP^T{\mathcal E}(I-AP)^T:{\mathcal B} + (I-PA){\mathcal E}P^TP:{\mathcal B} -P{\mathcal E}P^T\Big):dA \cr \cr \frac{\partial P}{\partial A} &= \Big(PP^T{\mathcal E}(I-AP)^T + (I-PA){\mathcal E}P^TP\Big):{\mathcal B} -P{\mathcal E}P^T \cr\cr }$$ Interestingly, the answer to your last question is $$\eqalign{ {\mathcal B} &= \frac{\partial A^T}{\partial A},\,\,&\,\, {\mathcal E} &= \frac{\partial A}{\partial A} \cr }$$