In sLORETA method there is a cost function as $F = || \phi – KJ||^2$ + $\alpha||J||^2$ which $\phi \in R^{N_E\times 1}$,$J \in R^{N_V\times 1}$ and $K \in R^{N_E\times N_V}$, the solution is $\hat{J} = K^T[KK^T + \alpha I]^+ \phi$ which $+$ denotes the Moore-Penrose pseudo-inverse and $I$ is the identity matrix. For the derivation, i start with differentiation but i cannot insert the Moore-Penrose inverse in the solution. In the relevant literature there's not the proof.
thanks in advance.
[Math] Derivation of Minimum Norm Least Squares (MNLS) with Moore-Penrose pseudoinverse
least squareslinear algebraoptimizationpseudoinverse
Related Solutions
When does the least squares solution exist?
$$\mathbf{A}x = b$$
$$\mathbf{A} \in \mathbb{C}^{m\times n}_{\rho}, \qquad b \in \mathbb{C}^{m}, \qquad x \in \mathbb{C}^{n}$$
In short, when the data vector $b$ is not in the null space: $$ b\notin\color{red}{\mathcal{N}\left( \mathbf{A}^{*} \right)} $$ The least squares solution is the projection of the data vector $b$ onto the range space $\color{blue}{\mathcal{R}\left( \mathbf{A}\right)}$. A projection will exist provided that some component of the data vector is in the range space.
This post works through a derivation of the least squares solution based upon the singular value decomposition of a nonzero matrix. How does the SVD solve the least squares problem?
Another approach is to start with the guaranteed existence of the singular value decomposition $$ \mathbf{A} = \mathbf{U} \, \Sigma \, \mathbf{V}^{*}, $$ and build the Moore-Penrose pseudoinverse $$\mathbf{A}^{+} = \mathbf{V} \Sigma^{+} \mathbf{U}^{*}.$$ The pseudoinverse is a matrix projection operator onto the range space $\color{blue}{\mathcal{R}\left( \mathbf{A}\right)}$.
Eqn. (2.46) proposes to look at the minimizer $x_\alpha$ of the functional $$J_\alpha(x) := |A x - y|^2 + \alpha |x|^2.$$ For any finite $\alpha > 0$, the functional is strictly convex and has a unique minimizer $x_\alpha$; it is the smallest among those $x$ that produce the same residual magnitude $|A x - y|$. Minimization wrt $x$ gives $x_\alpha = (A^\top A + \alpha I)^{-1} A^\top y$. To see this, write the norm $|\cdot|^2$ in terms of the scalar product $\langle \cdot, \cdot \rangle$.
Ad 1. Suppose $A x = y$ has a solution $x^*$. The set of solutions is the convex set $(x^* + \ker A)$. So, there is only one solution that has minimal norm: the orthogonal projection of $0$ onto that set. As $\alpha \searrow 0$, the residual term becomes more important, and $A x = y$ is eventually enforced. Therefore, $x_0 := \lim_{\alpha \searrow 0} x_\alpha$ is the minimal-norm solution of $A x = y$.
Ad 2. If $A x = y$ has no solution, the residual $|A x - y|$ still has a minimum, which is selected for in the limit $\alpha \searrow 0$.
Best Answer
Let
$$f (\mathrm x) := \| \mathrm A \mathrm x - \mathrm b \|_2^2 + \alpha \| \mathrm x \|_2^2 = \mathrm x^T \mathrm A^T \mathrm A \mathrm x - 2 \mathrm b^T \mathrm A \mathrm x + \mathrm b^T \mathrm b + \alpha \, \mathrm x^T \mathrm x$$
Differentiating,
$$\nabla f (\mathrm x) = 2 \mathrm A^T \mathrm A \mathrm x - 2 \mathrm A^T \mathrm b + 2 \alpha \mathrm x$$
Finding where the gradient vanishes, we obtain the following linear system
$$(\mathrm A^T \mathrm A + \alpha \, \mathrm I) \, \mathrm x = \mathrm A^T \mathrm b$$
Let the SVD of $\mathrm A$ be
$$\mathrm A = \mathrm U \Sigma \mathrm V^T = \begin{bmatrix} \mathrm U_1 & \mathrm U_2\end{bmatrix} \begin{bmatrix} \hat\Sigma & \mathrm O\\ \mathrm O & \mathrm O\end{bmatrix} \begin{bmatrix} \mathrm V_1^T\\ \mathrm V_2^T\end{bmatrix}$$
where the zero matrices may be empty. The eigendecomposition of $\mathrm A^T \mathrm A$ is, thus,
$$\mathrm A^T \mathrm A = \mathrm V \Sigma^T \mathrm U^T \mathrm U \Sigma \mathrm V^T = \mathrm V \Sigma^T \Sigma \mathrm V^T = \begin{bmatrix} \mathrm V_1 & \mathrm V_2\end{bmatrix} \begin{bmatrix} \hat\Sigma^2 & \mathrm O\\ \mathrm O & \mathrm O\end{bmatrix} \begin{bmatrix} \mathrm V_1^T\\ \mathrm V_2^T\end{bmatrix}$$
Hence,
$$\mathrm A^T \mathrm A + \alpha \, \mathrm I = \mathrm V \begin{bmatrix} \hat\Sigma^2 + \alpha \, \mathrm I & \mathrm O\\ \mathrm O & \mathrm \alpha \, \mathrm I\end{bmatrix} \mathrm V^T$$
If $\alpha > 0$, then the matrix $\mathrm A^T \mathrm A + \alpha \, \mathrm I$ is invertible. The solution to the linear system is, thus,
$$\hat{\mathrm x} := (\mathrm A^T \mathrm A + \alpha \, \mathrm I)^{-1} \mathrm A^T \mathrm b$$
There is no need to use the pseudoinverse.