To make things simpler I assume $\alpha_0=0$ (the desired value of $\alpha_0$ can always be added/subtracted at the end); furthermore I take the equator at $\delta=0$, so the range of the latitude $\delta$ is the interval $[-\pi/2,\pi/2]$. Your change of coordinates amounts to changing the standard basis $(e_1,e_, e_3)$ of ${\mathbb R}^3$ to the new basis
$$\bar e_1=(\cos\delta_0,0,\sin\delta_0), \quad \bar e_2=(0,1,0),\quad \bar e_3=(-\sin\delta_0,0,\cos\delta_0)\ .$$
It follows that the new coordinates $\bar x_k$ are given in terms of the old ones $x_i$ by the formulas
$$\bar x_1=\cos\delta_0 x_1 +\sin\delta_0 x_3, \quad \bar x_2=x_2,\quad \bar x_3=-\sin\delta_0 x_1+\cos\delta_0 x_3\ .$$
Now we have to express this in terms of the "geographical" quantities $\alpha$, $\delta$, resp. $\bar\alpha$, $\bar\delta$. On the one hand we have
$$x_1=\cos\delta\cos\alpha,\quad x_2=\cos\delta\sin\alpha, \quad x_3=\sin\delta\ ,$$
and on the other hand
$$\bar\alpha=\arg\Bigl({\bar x_1\over\rho},{\bar x_2\over\rho}\Bigr), \quad \bar\delta=\arcsin(\bar x_3)\ ,$$
where $\rho:=\sqrt{\bar x_1^2+\bar x_2^2}$. Putting it all together some simplifications will result.
First you solve the problem by any means at all, and then you see how to
optimize the solution for computation. I can do the first part, and I’ll
leave the second part to @occulus.
This is a simple problem in spherical trigonometry, once you look at it
close enough, and I managed to find a kind of a solution by that method. But since
@occulus seemed to want a criterion to decide whether a point on the sphere
was or was not within the equiangular quadrilateral (not a rectangle!) that’s
gotten by projecting the rectangular window onto the sphere from a point (the eye)
at the center of the sphere, it seems to me that a nontrigonometric approach
is called for.
The first first thing to observe is that we can always turn a (long,lat) pair of numbers into the
Cartesian coordinates of a point in space, say at unit distance from the origin.
So, if the $(0,0)$-point of the sphere is on the $x$-axis and the poles are on
the $z$-axis, the Cartesian coordinates of $(\theta,\rho)$ are
$(\sin\rho\cos\theta,\sin\rho\sin\theta,\cos\rho)$, where $\rho$ is the “colatitude”,
namely $\pi/2-$ lat., and $\theta$ would be the longitude measured counterclockwise
through the polar axis viewed from the north.
I’m assuming, though @occulus did not make it explicit, that the rectangular window
is positioned so that its center is the closest point to the eye. Call the width
of the rectangle $2w$, and the height $2h$, and suppose that it’s a distance
of $d$ from the eye. Then the apparent half-width $\omega$ has $\tan\omega=w/d$,
and the apparent half-height $\eta$ has $\tan\eta=h/d$.
Now let’s do as @occulus suggests, and put the center of the rectangle on the meridian
of longitude $0$, I’ll call this great circle $g$ (for Greenwich), and suppose,
as I think the poster does, that the rectangle is “level with respect to the
equator”, so that its vertical axis of symmetry coincides with $g$. Let’s suppose
also that the horizontal axis of symmetry of the rectangle crosses the meridian $g$
at the point of colatitude $\rho$. In the discussion that follows, I’m thinking
of everything interesting happening in the northern hemisphere, and my own mental
pictures are based on this, so that $0<\rho<\pi/2$. In the drawing below, I’m thinking
of looking at the sphere from a point outside it, on the $x$-axis, so above the
equator and the meridian $g$.
We have six great circles: the unlabeled horizontal axis of symmetry, the vertical
axis of symmetry $g$, the “upper horizontal edge” $h_1$, the lower horizontal edge
$h_2$, and the vertical edges $v_1$ and $v_2$. One sees that $h_1$ crosses $g$
orthogonally at colatitude $\rho-\eta$, and $h_2$ crosses similarly at colatitude
$\rho+\eta$. The efficient way to describe a great circle is by its pole (there
are actually two!). I’ll use $G=(0,1,0)$ as the pole of $g$, so that a point $P$
is on $g$ if and only if $P\cdot(0,1,0)=0$. And $P$ is to the left of $g$ if and only if
the dot product is negative. Similarly, you see that the pole $H_1$ of $h_1$ has
longitude $\pi$ and colatitude $\pi/2-\rho+\eta$, while the pole $H_2$ of $h_2$ is
at long. $=\pi$, colat. $=\pi/2-\rho-\eta$. The corresponding Cartesian coordinates
are $H_1=(\cos(\rho-\eta),0,-\sin(\rho-\eta))$, and of $H_2$ the same, with $\eta$
replaced by $-\eta$. So you see that for a point $P$ to be in our rectangle,
it must at least have $P\cdot H_1\le0$, $P\cdot H_2\ge0$.
But what are the poles $V_1$ and $V_2$ of $v_1$ and $v_2$, respectively? You see that
you get $v_2$ by rotating $g$ counterclockwise by an amount of $\omega$, only not with
respect to the north pole, but with respect to the pole of the horizontal axis of
symmetry, whose coordinates are just $(-\cos\rho,0,\sin\rho)$. If I did the computations
correctly, we get $V_1=(\sin\omega\sin\rho,\cos\omega,-\sin\omega\cos\rho)$, and
$V_2=(-\sin\omega\sin\rho,\cos\omega,\sin\omega\cos\rho)$. Then the remaining inequalities
are $P\cdot V_1\ge0$, $P\cdot V_2\le0$. The simultaneous satisfaction of the four
inequalities should be equivalent to $P$ being visible through the window.
Best Answer
As you point out, the area element on the sphere depends on latitude. If $u,v$ are uniformly distributed in $[0,1]$, then $$\begin{align*} \phi &= 2\pi u \\\\ \theta &= \cos^{-1}(2v-1) \end{align*}$$ are uniformly distributed over the sphere. Here $\theta$ represents latitude and $\phi$ longitude, both in radians. For further details, more methods and some Mathematica code, see Sphere Point Picking at MathWorld.