Your boundary condition is $Y(b) = 0$. Looking for solutions of the differential equation of the form $u(y) = e^{ry}$, we find $r^2 = \lambda$, so $r = \pm \sqrt{\lambda}$. Now
the general solution of the differential equation can be written as $u(y) = c_1 e^{\sqrt{\lambda} y} + c_2 e^{-\sqrt{\lambda} y}$. If the boundary condition was $Y(0) = 0$, we would immediately see that $c_1 + c_2 = 0$, so $u(y) = c_1 \left(e^{\sqrt{\lambda} y} - e^{-\sqrt{\lambda} y}\right) = 2 c_1 \sinh(\sqrt{\lambda} y)$. But note that $v(y) = u(y-b)$ is a solution of the differential equation if $u$ is, and $v(b) = u(0)$. So we get
$v(y) = 2 c_1 \sinh(\sqrt{\lambda} (y - b))$. And now absorb the $2$ into the arbitrary
constant $c_1$ to get the book's $Y(y)$.
It is not appropriate to work in the space $H^1_0(U)$ since nonzero boundary conditions are being considered. You will have to assume some regularity of the boundary of $U$.
One version of Green's theorem (see e.g. the appendix in Evans) is that $$- \int_U (\Delta u) v \, dx = \int_U Du \cdot Dv \, dx -\int_{\partial U} \frac{\partial u}{\partial \nu} v \, dS.$$
A weak solution to the problem at hand can be proposed by setting $-\Delta u = f$ in $U$ and $\dfrac{\partial u}{\partial \nu} = -u$ on $\partial U$ so that $$\int_U fv = \int_U Du \cdot Dv + \int_{\partial U} uv \, dS \quad \forall v \in H^1(U)$$
or a bit more precisely
$$\int_U fv = \int_U Du \cdot Dv + \int_{\partial U} \newcommand{\tr}{\mathrm{Tr}\ \! }( \tr u )(\tr v) \, dS \quad \forall v \in H^1(U)$$
where $\tr : H^1(U) \to L^2(\partial U)$ is the trace operator.
An appropriate bilinear form is thus given by $$B[u,v] = \int_U Du \cdot Dv + \int_{\partial U} \newcommand{\tr}{\mathrm{Tr}\ \! }( \tr u )(\tr v) \, dS, \quad u,v \in H^1(U).$$
$B$ is clearly bounded. As far as coercivity goes, it may be helpful to use the Rellich-Kondrachov theorem. I can follow up with a hint if you like.
It remains to show that there is a constant $\alpha > 0$ with the property that $\|u\|_{H^1}^2 \le \alpha B[u,u]$ for all $u \in H^1(U)$. This can be proven by contradition. Otherwise, for every $n \ge \mathbb N$ there would exist $u_n \in H^1(U)$ with the property that $\|u_n\|^2_{H^1} > n B[u_n,u_n]$. For each $n$ define $v_n = \dfrac{u_n}{\|u_n\|_{H^1}}$. Then $v_n \in H^1(U)$, $\|v_n\|_{H^1} = 1$, and $B[v_n,v_n] < \dfrac 1n$ and all $n$.
Here we can invoke Rellich-Kondrachov. Since the family $\{v_n\}$ is bounded in the $H^1$ norm, there is a subsequence $\{v_{n_k}\}$ that converges to a limit $v \in L^2(U)$. However, since $\|Dv_{n_k}\|_{L^2}^2 < \dfrac{1}{n_k}$ it is also true that $Dv_{n_k} \to 0$ in $L^2$. Thus for any $\phi \in C_0^\infty(U)$ you have $$\int_U v D \phi \, dx = \lim_{k \to \infty} \int_U v_{n_k} D \phi \, dx = - \lim_{k \to \infty} \int_U D v_{n_k} \phi \, dx = 0.$$ This means $v \in H^1(U)$ and $D v = 0$, from which you can conclude $v_{n_k} \to v$ in $H^1(U)$. Since $\|v_{n_k}\|_{H^1} = 1$ for all $k$ it follows that $\|v\|_{H^1} = 1$ as well.
Next, since $\|\tr v_{n_k}\|_{L^2(\partial U)}^2 < \dfrac{1}{n_k}$ and the trace operator is bounded there is a constant $C$ for which
$$ \|\tr v\|_{L^2(\partial \Omega)} \le \|\tr v - \tr v_{n_k}\|_{L^2(\partial \Omega)} + \|\tr v_{n_k}\|_{L^2(\partial \Omega)} < \frac{1}{n_k} + C \|v - v_{n_k}\|_{H^1(U)}.$$
Let $k \to \infty$ to find that $\tr v = 0$ in $L^2(\partial U)$.
Can you prove that if $v \in H^1(U)$, $Dv = 0$, and $\tr v = 0$, then $v = 0$? Once you have established that fact you arrive at a contradiction, since $v$ also satisfies $\|v\|_{H^1} = 1$. It follows that $B$ is in fact coercive.
Best Answer
Using separation of variables $u = X(x)Y(y)$, we obtain the ODEs
\begin{align} X'' &= -\lambda X \\ Y'' &= \lambda Y \end{align}
where we used $-\lambda$ for our separation constant, with associated boundary conditions
\begin{align} u(L,y) &= 0 \implies X(L) = 0 \\ u_{x}(0,y) &= 0 \implies X'(0) = 0 \\ u_{y}(x,0) - hu(x,0) &= 0 \implies Y'(0) - hY(0) = 0 \end{align}
Solving the ODE in $X$, we find non-trivial solutions only if $\lambda > 0$ which yields
$$X = A \cos \sqrt{\lambda} x + B \sin \sqrt{\lambda} x$$
Now
\begin{align} X'(0) &= \sqrt{\lambda} B \\ &= 0 \\ \implies B &= 0 \quad \text{(why?)} \\ \therefore X(L) &= A \cos \sqrt{\lambda} L \\ &= 0 \\ \implies \sqrt{\lambda} L &= \frac{(2n + 1) \pi}{2}, \quad n \ge 0 \quad \text{($A \ne 0$ for non-trivial solutions)} \\ \implies \lambda &= \frac{(2n + 1)^{2} \pi^{2}}{4L^{2}}, \quad n \ge 0 \quad(*) \end{align}
and hence
$$X_{n} = A \cos \left( \frac{(2n + 1) \pi x}{2L} \right)$$
Using the eigenvalue $(*)$ and solving the ODE in $Y$ yields
$$Y = \frac{1}{h} \cdot \frac{(2n + 1) \pi }{2L} \cosh \left(\frac{(2n + 1) \pi y}{2L} \right) + \sinh \left(\frac{(2n + 1) \pi y}{2L} \right)$$
and so the general solution is given by
$$u(x,y) = \sum_{n \ge 0} A_{n} \cos \left( \frac{(2n + 1) \pi x}{2L} \right) \left[ \frac{1}{h} \cdot \frac{(2n + 1) \pi }{2L} \cosh \left(\frac{(2n + 1) \pi y}{2L} \right) + \sinh \left(\frac{(2n + 1) \pi y}{2L} \right)\right]$$
Applying the inhomogeneous condition, we find
\begin{align} u(x,L) &= u_{0} \\ &= \sum_{n \ge 0} A_{n} \cos \left( \frac{(2n + 1) \pi x}{2L} \right) \left[ \frac{1}{h} \cdot \frac{(2n + 1) \pi }{2L} \cosh \left(\frac{(2n + 1) \pi}{2} \right) + \sinh \left(\frac{(2n + 1) \pi}{2} \right)\right] \\ &= \sum_{n \ge 0} C_{n} \cos \left( \frac{(2n + 1) \pi x}{2L} \right) \end{align}
where
$$C_{n} = A_{n} \left[ \frac{1}{h} \cdot \frac{(2n + 1) \pi }{2L} \cosh \left(\frac{(2n + 1) \pi}{2} \right) + \sinh \left(\frac{(2n + 1) \pi}{2} \right)\right]$$
You can now solve for the coefficients $C_{n}$ using orthogonality relations. Note that when doing the integrals, a change of variable might help.