This is a pretty straightforward application of the chain rule. Given
$$\begin{aligned}
0 &= f(\lambda, w(\lambda), s(\lambda)) = (1-\lambda s) w - 1
\\[2ex]
\implies 0 &= \frac{{\rm d\,} f(\lambda, w(\lambda), s(\lambda))}{{\rm d\,} \lambda}
= \frac{\partial f}{\partial \lambda}
+ \frac{\partial f}{\partial w}\frac{\partial w}{\partial \lambda}
+ \frac{\partial f}{\partial s}\frac{\partial s}{\partial {\mathbf h}}\frac{\partial {\mathbf h}}{\partial w}\frac{\partial w}{\partial \lambda}
\\[2ex]
&\iff \Big(\frac{\partial f}{\partial w}
+ \frac{\partial f}{\partial s}\frac{\partial s}{\partial {\mathbf h}}\frac{\partial {\mathbf h}}{\partial w}\Big)\frac{\partial w}{\partial \lambda} = -\frac{\partial f}{\partial \lambda}
\end{aligned}$$
So, your derivative can be found by solving the linear system with $-\frac{\partial f}{\partial \lambda} = sw$, $\frac{\partial f}{\partial w} = 1-\lambda s$, $\frac{\partial f}{\partial s}=\lambda w$ and $\frac{\partial s}{\partial {\mathbf h}} = 2{\mathbf B}{\mathbf C}({\mathbf B}'{\mathbf h}-{\mathbf a})$. With regard to $\frac{\partial {\mathbf h}}{\partial w}$, we could do it directly, using $\partial X^{-1}= -X^{-1} (\partial X)X^{-1}$, but we also just apply the implicit function theorem a second time:
$$\begin{aligned}
0 &= g(w, {\mathbf h}(w)) = \left(\mathbf B (\mathbb I + w \mathbf C) \mathbf B'\right) \mathbf h - \mathbf B(\mathbb I + w \mathbf C)\mathbf{a}
\\[2ex]
\implies 0 &= \frac{{\rm d\,} g(w, {\mathbf h}(w))}{{\rm d\,} w}
= \frac{\partial g}{\partial w}
+ \frac{\partial g}{\partial {\mathbf h}}\frac{\partial {\mathbf h}}{\partial w}
\\[2ex]
&\iff \frac{\partial g}{\partial {\mathbf h}}\frac{\partial {\mathbf h}}{\partial w} = -\frac{\partial g}{\partial w}
\end{aligned}$$
where $\frac{\partial g}{\partial w} = {\mathbf B}{\mathbf C}{\mathbf B}'h - {\mathbf B}{\mathbf C}a$ and $\frac{\partial g}{\partial {\mathbf h}} = {\mathbf B}{\mathbf \Gamma} {\mathbf B}'$. So in total your derivative is given by solution of the nested linear system
$$\begin{aligned}
(1)&&&{\mathbf B}{\mathbf \Gamma} {\mathbf B}' \tfrac{\partial \mathbf h}{\partial w} = -{\mathbf B}{\mathbf C}{\mathbf B}'\mathbf h + {\mathbf B}{\mathbf C}\mathbf a
\\
(2)&&&\big(1-\lambda s + 2\lambda w (\mathbf a'- {\mathbf h}'{\mathbf B}){\mathbf C}{\mathbf B}'\tfrac{\partial \mathbf h}{\partial w}\big)\tfrac{\partial w}{\partial \lambda} = sw
\end{aligned}$$
If you want to, you can plug everything back in and try to simplify it. As a final remark, you may wonder why $\frac{\partial s}{\partial {\mathbf h}}$ appears transposed in the formula, this is due to how the tensor-contraction between $\frac{\partial s}{\partial {\mathbf h}}$ and $\frac{\partial {\mathbf h}}{\partial w}$ operates. You can either write it out or realize that the result needs to be a scalar again.
The statement is true if $χ_2^2+\frac{(χ_1χ_3)^2}{χ_1^2+χ_3^2} ≥ χ_4^2/2$, otherwise it might be false.
First, let $c = b-a$ so that the equation to solve becomes
$$\begin{align*}
xx' + yy' + zz' &= χ_1^2cc' + χ_2^2Λcc'Λ + χ_3^2(I_n-Λ)cc'(I_n-Λ)+χ_4^2(a(c+a)'Λ+Λ(c+a)a')\\
&= (χ_1^2+χ_3^2)cc' + (χ_2^2+χ_3^2)Λcc'Λ - χ_3^2(Λcc' + cc'Λ) + 2χ_4^2aa' + χ_4^2 (ac'Λ+Λca') \text{.}
\end{align*}$$
We will guess that $x,y,z$ are of the form $α_ic + β_iΛc + γ_ia$ for $i=1,2,3$, since this is the simplest way to build vectors out of the data we are given. Then
$$\begin{align*}
xx' + yy' + zz' &= \sum_{i=1}^3 α_i^2 cc' + β_i^2 Λcc'Λ + γ_i^2aa' + α_iβ_i(cc'Λ+Λcc') + α_iγ_i(ca'+ac') + β_iγ_i(Λca'+ac'Λ)\\
&= α^2 cc' + β^2 Λcc'Λ + γ^2aa' + αβ(cc'Λ + Λcc') + αγ(ca'+ac')+βγ(Λca'+ac'Λ) \text{.}
\end{align*}$$
(I write $αβ$ for the scalar product of $α$ and $β$.) Matching this expression against the target expression gives the following system.
$$\begin{align*}α^2 &= χ_1^2+χ_3^2\\
β^2 &= χ_2^2+χ_3^2\\
γ^2 &= 2χ_4^2\\
αβ &= -χ_3^2\\
αγ &= 0\\
βγ &= χ_4^2
\end{align*}$$
Let's suppose we are not in an edge case with $χ_i = 0$. We can solve this system geometrically. Choose two orthogonal vectors $α$ and $γ$ with the desired lengths. Then $αβ = -χ_3^2$ and $βγ = χ_4^2$ tell us the projection of $β$ on the plane generated by $α$ and $γ$. The minimal square-norm of $β$ is (I skip the computation) $χ_4^2/2 + \frac{χ_3^4}{χ_1^2+χ_3^2}$. Since $β$ should have a square-norm of $χ_2^2+χ_3^2$, in order to have a solution, we must have $χ_2^2+χ_3^2 ≥ χ_4^2/2 + \frac{χ_3^4}{χ_1^2+χ_3^2}$, or in other words $χ_2^2+\frac{(χ_1χ_3)^2}{χ_1^2+χ_3^2} ≥ χ_4^2/2$. If this is the case, there is a solution.
To find a counter-example, we need to take $χ_2^2+\frac{(χ_1χ_3)^2}{χ_1^2+χ_3^2} < χ_4^2/2$. For instance, with $n=3$ and $u=1$, take $c = (1,1,1)$, $a=(1,2,0)$, $χ_1 = 1$, $χ_2 = 0$, $χ_3 = 1$, $χ_4 = 3$. Then if my computations are correct, we find
$$\begin{pmatrix}
37&64&1\\
64&109&1\\
1&1&2
\end{pmatrix}$$
which is not positive semidefinite.
The condition $χ_2^2+\frac{(χ_1χ_3)^2}{χ_1^2+χ_3^2} ≥ χ_4^2/2$ might even be necessary. To show that, one could try to use the fact that even if this condition fails, there is a solution $β$ but with complex coefficients. This can be used to build a solution $x,y,z$, but there are complex coefficients: can this be used in some way to deduce that $xx'+yy'+zz'$ is not positive definite?
PS: Thank you for the context.
Best Answer
We start with the relation $\mathbf h = \left(\mathbf B {\mathbf \Omega}\mathbf B'\right)^{-1}\mathbf B{\mathbf\Omega}\mathbf{KK'a}$ which gives $ \mathbf B {\mathbf \Omega}\mathbf B'\mathbf h = \mathbf B{\mathbf\Omega}\mathbf{KK'a}.$ Substituting $\mathbf \Omega=\mathbf{I}+\mathbf{KWK'}$, we get \begin{align} \mathbf B \left(\mathbf{I}+\mathbf{KWK}'\right)\mathbf B'\mathbf h &= \mathbf B\left(\mathbf{I}+\mathbf{KWK}'\right)\mathbf{KK'a}\\ \implies \mathbf B\mathbf{KK'a} &=\mathbf {B B}' \mathbf h +\mathbf{B}\mathbf{KWK}'\left(\mathbf B'\mathbf h -\mathbf{a}\right),\tag{1} \end{align} since $\mathbf{K'KK'}=\mathbf{K'}.$ Using the relation $\mathbf W = \left( \mathbf I- \mathbf K'\left(\mathbf B^{\prime}\mathbf h -\mathbf a \right)\left(\mathbf h'\mathbf B-\mathbf a' \right)\mathbf K\right)^{-1}$ and the Sherman-Morrison formula, we obtain \begin{align} \mathbf{KWK}'\left(\mathbf B'\mathbf h -\mathbf{a}\right)&=\mathbf{K}\left( \mathbf I- \mathbf K'\left(\mathbf B^{\prime}\mathbf h -\mathbf a \right)\left(\mathbf h'\mathbf B-\mathbf a' \right)\mathbf K\right)^{-1}\mathbf{K}'\left(\mathbf B'\mathbf h -\mathbf{a}\right)\\ &=\mathbf{K}\left( \mathbf I + \frac{\mathbf K'\left(\mathbf B^{\prime}\mathbf h -\mathbf a \right)\left(\mathbf h'\mathbf B-\mathbf a' \right)\mathbf K}{1-r}\right)\mathbf{K}'\left(\mathbf B'\mathbf h -\mathbf{a}\right)\\ &= \left( 1 + \frac{r}{1-r}\right)\mathbf{K}\mathbf{K}'\left(\mathbf B'\mathbf h -\mathbf{a}\right)= \frac{1}{1-r}\mathbf{K}\mathbf{K}'\left(\mathbf B'\mathbf h -\mathbf{a}\right)\tag{2}, \end{align} where $r\triangleq \left(\mathbf h'\mathbf B-\mathbf a' \right)\mathbf K\mathbf K'\left(\mathbf B^{\prime}\mathbf h -\mathbf a \right)\neq 1$. Combining $(1)$ and $(2)$, we arrive at \begin{align} \mathbf B\mathbf{KK'a} &=\mathbf {B B}' \mathbf h +\frac{1}{1-r}\mathbf{B}\mathbf{K}\mathbf{K}'\left(\mathbf B'\mathbf h -\mathbf{a}\right)\\ \implies \frac{2-r}{1-r}\mathbf B\mathbf{KK'a} &= \mathbf {B}\left(\mathbf I+\frac{1}{1-r}\mathbf{K}\mathbf{K}'\right)\mathbf{ B}' \mathbf h\\ \implies \mathbf B\mathbf{KK'a} &= \mathbf {B}\left(\frac{1-r}{2-r}\mathbf I+\frac{1}{2-r}\mathbf{K}\mathbf{K}'\right)\mathbf{ B}' \mathbf h.\tag{3} \end{align} Since $\mathbf{K}=\begin{bmatrix}\mathbf{I} \\\mathbf 0\end{bmatrix}$, we get that $$ \frac{1-r}{2-r}\mathbf I+\frac{1}{2-r}\mathbf{K}\mathbf{K}' = \frac{1-r}{2-r}\begin{bmatrix}\mathbf I &\mathbf 0\\ \mathbf 0&\mathbf I\end{bmatrix}+\frac{1}{2-r}\begin{bmatrix}\mathbf{I} \\\mathbf 0\end{bmatrix}\begin{bmatrix}\mathbf{I} &\mathbf 0\end{bmatrix} = \begin{bmatrix}\mathbf I &\mathbf 0\\ \mathbf 0& \frac{1-r}{2-r}\mathbf I\end{bmatrix}\triangleq \mathbf\Gamma.$$ Thus, $(3)$ yields $ \mathbf B\mathbf{KK'a} = \mathbf {B}\mathbf\Gamma\mathbf{ B}' \mathbf h\implies \mathbf h=\left(\mathbf {B}\mathbf\Gamma\mathbf{ B}'\right)^{-1}\mathbf B\mathbf{KK'a}.$ Here, $\mathbf {B}\mathbf\Gamma\mathbf{ B}'$ is invertible as $r\neq 1$ due to the Sherman-Morrison formula, and from the fact that $\mathbf B {\mathbf \Omega}\mathbf B'$ is invertible (i.e., $n=\mathrm{rank}\{\mathbf B\}=\mathrm{rank}\{\mathbf B\mathbf B'\} $).
Thus, the proof is complete.