You actually had just about everything right, except that you skipped an important step: your normal vector to the surface $ \ \vec{ds} \ $ is correct, but you need to integrate its length over the surface of the cone nappe in order to obtain the surface area.
I'll generalize the problem a little, since the choice of proportions for the cone hides one of the factors in the surface area result. For a cone nappe with a height $ \ h \ $ and a "base radius" $ \ r \ $ , we can use similar triangles to find the parametrization (using your notation)
$$ x \ = \ \left( \frac{r}{h} \right) u \ \cos \ p \ \ , \ \ y \ = \ \left( \frac{r}{h} \right) u \ \sin \ p \ \ , \ \ z \ = \ u \ \ , $$
with the domain $ \ 0 \ \le \ u \ \le \ h \ , \ 0 \ \le \ p \ < \ 2 \pi \ $ . An "upward" normal vector is then given by
$$ \vec{R_u} \ \times \ \vec{R_p} \ \ " = " \ \ \left|\begin{array}{ccc}\mathbf{i}&\mathbf{j}&\mathbf{k}\\ \left( \frac{r}{h} \right) \cos \ p&\left( \frac{r}{h} \right) \sin \ p\quad&1\\ -\left( \frac{r}{h} \right) u \ \sin \ p&\left( \frac{r}{h} \right) u \ \cos \ p\quad&0\end{array}\right| $$
$$ = \ \langle \ -\left( \frac{r}{h} \right) u \ \cos \ p \ \ , \ \ -\left( \frac{r}{h} \right) u \ \sin \ p \ \ , \ \ \left( \frac{r}{h} \right)^2 u \ \rangle \ \ . $$
So, up to this point, your procedure is fine. What is needed now is the "norm" of this vector:
$$ \| \ \vec{R_u} \ \times \ \vec{R_p} \ \| \ \ = \ \ \left[ \ \left( \frac{r}{h} \right)^2 u^2 \ \cos^2 \ p \ + \ \left( \frac{r}{h} \right)^2 u^2 \ \sin^2 \ p \ + \ \left( \frac{r}{h} \right)^4 u^2 \ \right]^{1/2} \ \ . $$
$$ = \ \ \left[ \ \left( \frac{r}{h} \right)^2 u^2 \ + \ \left( \frac{r}{h} \right)^4 u^2 \ \right]^{1/2} \ = \ \left(\frac{r}{h} \right) \ \sqrt{ 1 \ + \ \left( \frac{r}{h} \right)^2 } \ \ u \ \ . $$
It is the "magnitude" of the infinitesimal patches associated with the normal vectors that we wish to integrate over the domain of the parameters. Thus,
$$ S \ \ = \ \ \int_0^{2 \pi} \int_0^h \ \left(\frac{r}{h} \right) \ \sqrt{ 1 \ + \ \left( \frac{r}{h} \right)^2 } \ \ u \ \ du \ dp $$
$$ = \ \ \left(\frac{r}{h} \right) \ \sqrt{ 1 \ + \ \left( \frac{r}{h} \right)^2 } \ \int_0^{2 \pi} dp \ \int_0^h \ u \ \ du $$
$$ = \ \left(\frac{r}{h} \right) \ \sqrt{ 1 \ + \ \left( \frac{r}{h} \right)^2 } \ \cdot \ 2 \pi \ \cdot \ \left(\frac{1}{2}u^2 \right) \vert_0^h \ \ = \ \left(\frac{r}{h} \right) \ \sqrt{ 1 \ + \ \left( \frac{r}{h} \right)^2 } \ \cdot \ \pi \ h^2 $$
$$ = \ \pi \ h \ \cdot \ \left(\frac{r}{h} \right) \ \cdot \ h \ \sqrt{ 1 \ + \ \left( \frac{r}{h} \right)^2 } \ = \ \pi \ r \ \sqrt{ r^2 \ + \ h^2 } \ \ , $$
or $ \ \pi \ $ times the "base radius" times the "slant height" of the cone nappe, as the surface area is frequently expressed. In your use of the "standard cone", for which $ \ r \ = \ h \ $ , this formula gives us $ \ S \ = \ \pi \ \sqrt{2} \ h^2 \ $ , as you will find for your calculations, with the restoration of the omitted step.
Best Answer
Case: $\Omega=\large{\frac\pi2}$
The figure (drawn with Geogebra) shows the cross-section in the plane $x=0$. We want to find the area of what I'll refer to as a "fin", the part of the annulus (region between green and blue circles) interior to the cone (purple) and to the right of the $z$-axis (solid gray).
Define a few symbols:
To parameterize the fin, we first confine ourselves to the disk with boundary $O$, which boils down to replacing $r_O$ with the variable $r$ and allowing it to vary between the appropriate surfaces. Using spherical coordinates, we let $(x,y,z)=(0,r\sin\phi,r\cos\phi)$. Then the equations for the fin's boundaries are
$$\begin{cases} (y-Y)^2 = a^2 z^2 \\ y^2 + z^2 = r_L^2 \\ y^2 + z^2 = r_O^2 \end{cases} \implies \begin{cases} \rho_\lambda(\phi) = \dfrac Y{a\cos\phi+\sin\phi} = \dfrac Y{\sqrt{a^2+1} \sin(\phi + \arctan a)} \\ \rho_L(\phi) = r_L \\ \rho_O(\phi) = r_O \end{cases}$$
Next, we solve for the angles $\Phi_L$ and $\Phi_O$:
$$\rho_\lambda = \rho_{L/O} \implies \phi = \Phi_{L/O} = \arcsin \frac{Y + a \sqrt{\left(1+a^2\right) r_{L/O}^2 - Y^2}}{\left(1+a^2\right) r_{L/O}}$$
by carefully selecting the correct particular solution for the respective $\Phi$.
For $\Phi_L \le \phi \le \Phi_O$, we'll have $r$ vary between $L$ and $\lambda$, then for $\Phi_O\le \phi\le\dfrac\pi2$, it will vary from $L$ to $O$. Thus we parameterize the fin by
$$\vec F(r,\phi) = \left(0, r \sin\phi, r \cos\phi \right)$$
over the union of $(\phi, r) \in \left[\Phi_L,\Phi_O\right] \times \left[\rho_L, \rho_\lambda\right]$ and $(\phi,r) \in \left[\Phi_O, \dfrac\pi2\right] \times \left[\rho_L, \rho_O\right]$.
Below is a plot of the fin (blue and orange) overlaying the cone-sphere-circle system with $(Y,r_L,r_O,a)=(30,5,7,\sqrt{35})$, for which we find
$$\left(\Phi_L,\Phi_O\right) = \left(\arcsin\frac16, \arcsin \frac{5+2\sqrt{210}}{42}\right)$$
By symmetry, the fin's total area can be computed via the integral,
$$2 \left\{\int_{\Phi_L}^{\Phi_O} \int_{\rho_L}^{\rho_\lambda} + \int_{\Phi_O}^\tfrac\pi2 \int_{\rho_L}^{\rho_O} \right\} r \, dr \, d\phi \\ = \int_{\Phi_L}^{\Phi_O} \rho_\lambda^2 \, d\phi + \int_{\Phi_O}^\tfrac\pi2 \rho_O^2 \, d\phi - \int_{\Phi_L}^\tfrac\pi2 \rho_L^2 \, d\phi \\ = \boxed{r_L^2 \Phi_L - r_O^2 \Phi_O + \frac\pi2 \left(r_O^2-r_L^2\right) + \int_{\Phi_L}^{\Phi_O} \rho_\lambda^2 \, d\phi}$$
since $\dfrac{\partial \vec F}{\partial r} \times \dfrac{\partial\vec F}{\partial \phi}=(-r,0,0)$. The $\phi$-integral is quite hairy, but ultimately it's a question of computing an elementary antiderivative of the form
$$\int \frac{a + b \cos(2\phi) + c \sin\phi \sqrt{d + e \cos(2\phi)}}{\left(f + g \cos(2\phi)\right)^2} \, d\phi$$
Case: $\omega < \Omega < {\large\frac\pi2}$
Let $(x,y,z)=(r\color{red}{\cos\Omega}\sin\phi, r\color{red}{\sin\Omega}\sin\phi,r\cos\phi)$.
In the new coordinates, $\lambda$ has a more complicated equation,
$$(Y - \rho_\lambda \sin\phi \sin\Omega)^2 = a^2 \rho_\lambda^2 \left(\cos^2\phi + \cos^2\Omega \sin^2\phi\right) \\ \implies \rho_\lambda = \frac{\frac12 aY \sqrt{3 + \cos(2\phi) + \cos(2\Omega) - \cos(2\phi) \cos(2\Omega)} - Y \sin\phi \sin\Omega}{a^2 \cos^2\phi + a^2 \sin^2\phi \cos^2\Omega - \sin^2\phi \sin^2\Omega}$$
where we've chosen the solution to the quadratic corresponding to the quadrant $0<\Omega<\frac\pi2$. By the same process as before, we determine the polar angles to be
$$\Phi_{L/O} = \arcsin \frac{Y + a \sqrt{\left(1+a^2\right) r_{L/O}^2 - Y^2}}{\left(1+a^2\right) r_{L/O} \color{red}{\sin\Omega}}$$
However, as $\Omega$ approaches a certain angle $\omega$, $O$ will no longer intersect with $\lambda$, while $\Phi_L \to \dfrac\pi2$. This lower bound occurs at
$$\omega = \arcsin {\frac{Y+a\sqrt{\left(1+a^2\right) r_O^2 - Y^2}}{\left(1+a^2\right) r_O}}$$
which is obtained by solving $\Phi_O=\dfrac\pi2$ for $\Omega\in\left(0,\dfrac\pi2\right)$.
The following figures depict two sample fins at $\Omega=\dfrac{5\pi}{12}$ and $\Omega=\omega$. On the left are plots of the functions $\rho(\phi)$ at these $\Omega$. The latter shows $\rho_\lambda$ intersecting with $\rho_L$ exactly once, then asymptotically tending to $\rho_O$ as $\Omega\to\omega^+$.
Case: $\omega' < \Omega \le \omega$
Now, with $O$ no longer delimiting the fin (shown below at $\Omega=\dfrac\pi4$; note the blue arc hovering some distance away from $\lambda$), we solve in similar fashion for $\Omega$ when $\Phi_L=\dfrac\pi2$ to obtain a new lower limit of
$$\omega' = \arcsin {\frac{Y+a\sqrt{\left(1+a^2\right) r_L^2 - Y^2}}{\left(1+a^2\right) r_L}}$$
The area integral reduces somewhat to
$$2 \int_{\Phi_L}^\tfrac\pi2 \int_{\rho_L}^{\rho_\lambda} r \, dr \, d\phi = \int_{\Phi_L}^\tfrac\pi2 \left(\rho_\lambda^2 - \rho_L^2\right) \, d\phi \\ = \boxed{r_L^2 \left(\Phi_L-\frac\pi2\right) + \int_{\Phi_L}^\tfrac\pi2 \rho_\lambda^2 \, d\phi}$$