Case: $\Omega=\large{\frac\pi2}$
The figure (drawn with Geogebra) shows the cross-section in the plane $x=0$. We want to find the area of what I'll refer to as a "fin", the part of the annulus (region between green and blue circles) interior to the cone (purple) and to the right of the $z$-axis (solid gray).
Define a few symbols:
- $Y$ is the $y$-coordinate of the cone $\bbox[#cb42f5,2pt]{\color{white}{\lambda}}$ 's vertex, $(0,Y,0)$
- $r_L$ is the radius of the sphere $\bbox[#00ff37,2pt]{L}$, $x^2+y^2+z^2=r_L^2$
- $r_O$ is the radius of the cylinder containing the circle $\bbox[#42e9f5,2pt]{O}$, $y^2+z^2=r_O^2$
- $a$ is a parameter that ensures $\lambda$ is tangent to $L$; $\lambda$ has equation $(y-Y)^2=a^2(x^2+z^2)$
- $\Phi_L$ and $\Phi_O$ are the polar angles corresponding to the dotted rays where $\lambda$ meets $L$ and $O$, resp. (NB: In comments, I mistook $\Phi_L=0$.)
To parameterize the fin, we first confine ourselves to the disk with boundary $O$, which boils down to replacing $r_O$ with the variable $r$ and allowing it to vary between the appropriate surfaces. Using spherical coordinates, we let $(x,y,z)=(0,r\sin\phi,r\cos\phi)$. Then the equations for the fin's boundaries are
$$\begin{cases} (y-Y)^2 = a^2 z^2 \\ y^2 + z^2 = r_L^2 \\ y^2 + z^2 = r_O^2 \end{cases} \implies \begin{cases} \rho_\lambda(\phi) = \dfrac Y{a\cos\phi+\sin\phi} = \dfrac Y{\sqrt{a^2+1} \sin(\phi + \arctan a)} \\ \rho_L(\phi) = r_L \\ \rho_O(\phi) = r_O \end{cases}$$
Next, we solve for the angles $\Phi_L$ and $\Phi_O$:
$$\rho_\lambda = \rho_{L/O} \implies
\phi = \Phi_{L/O} = \arcsin \frac{Y + a \sqrt{\left(1+a^2\right) r_{L/O}^2 - Y^2}}{\left(1+a^2\right) r_{L/O}}$$
by carefully selecting the correct particular solution for the respective $\Phi$.
For $\Phi_L \le \phi \le \Phi_O$, we'll have $r$ vary between $L$ and $\lambda$, then for $\Phi_O\le \phi\le\dfrac\pi2$, it will vary from $L$ to $O$. Thus we parameterize the fin by
$$\vec F(r,\phi) = \left(0, r \sin\phi, r \cos\phi \right)$$
over the union of $(\phi, r) \in \left[\Phi_L,\Phi_O\right] \times \left[\rho_L, \rho_\lambda\right]$ and $(\phi,r) \in \left[\Phi_O, \dfrac\pi2\right] \times \left[\rho_L, \rho_O\right]$.
Below is a plot of the fin (blue and orange) overlaying the cone-sphere-circle system with $(Y,r_L,r_O,a)=(30,5,7,\sqrt{35})$, for which we find
$$\left(\Phi_L,\Phi_O\right) = \left(\arcsin\frac16, \arcsin \frac{5+2\sqrt{210}}{42}\right)$$
By symmetry, the fin's total area can be computed via the integral,
$$2 \left\{\int_{\Phi_L}^{\Phi_O} \int_{\rho_L}^{\rho_\lambda} + \int_{\Phi_O}^\tfrac\pi2 \int_{\rho_L}^{\rho_O} \right\} r \, dr \, d\phi \\
= \int_{\Phi_L}^{\Phi_O} \rho_\lambda^2 \, d\phi + \int_{\Phi_O}^\tfrac\pi2 \rho_O^2 \, d\phi - \int_{\Phi_L}^\tfrac\pi2 \rho_L^2 \, d\phi \\
= \boxed{r_L^2 \Phi_L - r_O^2 \Phi_O + \frac\pi2 \left(r_O^2-r_L^2\right) + \int_{\Phi_L}^{\Phi_O} \rho_\lambda^2 \, d\phi}$$
since $\dfrac{\partial \vec F}{\partial r} \times \dfrac{\partial\vec F}{\partial \phi}=(-r,0,0)$. The $\phi$-integral is quite hairy, but ultimately it's a question of computing an elementary antiderivative of the form
$$\int \frac{a + b \cos(2\phi) + c \sin\phi \sqrt{d + e \cos(2\phi)}}{\left(f + g \cos(2\phi)\right)^2} \, d\phi$$
Case: $\omega < \Omega < {\large\frac\pi2}$
Let $(x,y,z)=(r\color{red}{\cos\Omega}\sin\phi, r\color{red}{\sin\Omega}\sin\phi,r\cos\phi)$.
In the new coordinates, $\lambda$ has a more complicated equation,
$$(Y - \rho_\lambda \sin\phi \sin\Omega)^2 = a^2 \rho_\lambda^2 \left(\cos^2\phi + \cos^2\Omega \sin^2\phi\right) \\
\implies \rho_\lambda = \frac{\frac12 aY \sqrt{3 + \cos(2\phi) + \cos(2\Omega) - \cos(2\phi) \cos(2\Omega)} - Y \sin\phi \sin\Omega}{a^2 \cos^2\phi + a^2 \sin^2\phi \cos^2\Omega - \sin^2\phi \sin^2\Omega}$$
where we've chosen the solution to the quadratic corresponding to the quadrant $0<\Omega<\frac\pi2$. By the same process as before, we determine the polar angles to be
$$\Phi_{L/O} = \arcsin \frac{Y + a \sqrt{\left(1+a^2\right) r_{L/O}^2 - Y^2}}{\left(1+a^2\right) r_{L/O} \color{red}{\sin\Omega}}$$
However, as $\Omega$ approaches a certain angle $\omega$, $O$ will no longer intersect with $\lambda$, while $\Phi_L \to \dfrac\pi2$. This lower bound occurs at
$$\omega = \arcsin {\frac{Y+a\sqrt{\left(1+a^2\right) r_O^2 - Y^2}}{\left(1+a^2\right) r_O}}$$
which is obtained by solving $\Phi_O=\dfrac\pi2$ for $\Omega\in\left(0,\dfrac\pi2\right)$.
The following figures depict two sample fins at $\Omega=\dfrac{5\pi}{12}$ and $\Omega=\omega$. On the left are plots of the functions $\rho(\phi)$ at these $\Omega$. The latter shows $\rho_\lambda$ intersecting with $\rho_L$ exactly once, then asymptotically tending to $\rho_O$ as $\Omega\to\omega^+$.
Case: $\omega' < \Omega \le \omega$
Now, with $O$ no longer delimiting the fin (shown below at $\Omega=\dfrac\pi4$; note the blue arc hovering some distance away from $\lambda$), we solve in similar fashion for $\Omega$ when $\Phi_L=\dfrac\pi2$ to obtain a new lower limit of
$$\omega' = \arcsin {\frac{Y+a\sqrt{\left(1+a^2\right) r_L^2 - Y^2}}{\left(1+a^2\right) r_L}}$$
The area integral reduces somewhat to
$$2 \int_{\Phi_L}^\tfrac\pi2 \int_{\rho_L}^{\rho_\lambda} r \, dr \, d\phi = \int_{\Phi_L}^\tfrac\pi2 \left(\rho_\lambda^2 - \rho_L^2\right) \, d\phi \\
= \boxed{r_L^2 \left(\Phi_L-\frac\pi2\right) + \int_{\Phi_L}^\tfrac\pi2 \rho_\lambda^2 \, d\phi}$$
Best Answer
In your first formula for $A$ you set $s=\sqrt{r^2-x^2}$, but it should be instead $s=2\sqrt{r^2-x^2}$.