I'd ignore the radius $R$ (i.e. take the earth's radius as unit of length) and then convert to Cartesian coordinates:
\begin{align*}
x&=\cos\phi\cos\lambda\\
y&=\cos\phi\sin\lambda\\
z&=\sin\phi
\end{align*}
These vectors point from the center of the sphere to the vertices of your polygon. Two such vertices are joined by a greatcircle arc. That greatcircle is the intersection of your sphere with a plane through the origin. That plane through the origin is defined by its normal vector. The cross product of two vectors is perpendicular to both of them, so you can use the cross product to compute these normal vectors. Writing $v_i=(x_i,y_iz_i)$ for the vertices, you get the normals of the edges as
$$n_i = v_i\times v_{i+1}$$
The angle at a vertex is equal to the angle between the corresponding normal vectors. You can compute that using the dot product, dividing by the lengths to ensure normalization:
$$\alpha_i = \pm\arccos-
\frac{n_{i-1}\cdot n_{i}}{\lVert n_{i-1}\rVert\,\lVert n_i\rVert}$$
The minus sign in there is because for a normalized dot product of $1$, both normals point in the same direction, so you have zero change in direction but the inner angle there is $180$.
So which sign should you choose for the $\pm$ in there? For that, look at the determinant of the $3\times3$ matrix formed by $v_{i-1}$, $v_i$ and $v_{i+1}$. The sign of that matrix will tell you whether the triangle these three vectors form on the surface is oriented clockwise or counter-clockwise. Take the sign from this and the value from the $\arccos$ above and you should be almost done.
As usual with signs, you have a decision to make: which sign is which? Well, one choice of sign (e.g. exactly as described above) will lead you one polygon, the other (with the sign of the determinant flipped, or eqivalently the order of vectors inside the determinant reversed) leads to the complementary polygon. One will have all its vertices in clockwise order, the other in counter-clockwise order. Trying this out on a tiny example (e.g. three points forming three right angles) will tell you which one is which.
Best Answer
Assume we are working on a sphere of radius $1$, or consider the lengths in radians and the areas in steradians.
Extend the sides of length $l$ until they meet. This results in a triangle with sides $$ w,\quad\frac\pi2-\frac l2,\quad\frac\pi2-\frac l2 $$
The Spherical Law of Cosines says that $$ \begin{align} \cos(A) &=\frac{\cos\left(\frac\pi2-\frac l2\right)-\cos\left(\frac\pi2-\frac l2\right)\cos(w)}{\sin\left(\frac\pi2-\frac l2\right)\sin(w)}\\ &=\frac{\sin\left(\frac l2\right)}{\cos\left(\frac l2\right)}\frac{1-\cos(w)}{\sin(w)}\\[6pt] &=\tan\left(\frac l2\right)\tan\left(\vphantom{\frac l2}\frac w2\right) \end{align} $$ One quarter of the spherical excess of the rectangle is $D-\frac\pi2$ and $$ \sin\left(D-\frac\pi2\right)=\tan\left(\frac l2\right)\tan\left(\vphantom{\frac l2}\frac w2\right) $$ Therefore, the area of the rectangle is $$ \bbox[5px,border:2px solid #C0A000]{4\sin^{-1}\left(\tan\left(\frac l2\right)\tan\left(\vphantom{\frac l2}\frac w2\right)\right)} $$ Note that for small $l$ and $w$, this is approximately $lw$; and if $l+w=\pi$ (that is, the rectangle is a great circle), we get an area of $2\pi$ (one half the sphere).
Scaling for a sphere of radius $R$ gives $$ \bbox[5px,border:2px solid #C0A000]{4R^2\sin^{-1}\left(\tan\left(\frac l{2R}\right)\tan\left(\vphantom{\frac l2}\frac w{2R}\right)\right)} $$
Note the similarity to the formula for the area of a spherical right triangle with legs $a$ and $b$: $$ 2\tan^{-1}\left(\tan\left(\vphantom{\frac b2}\frac a2\right)\tan\left(\frac b2\right)\right) $$ or for a sphere of radius $R$, $$ 2R^2\tan^{-1}\left(\tan\left(\vphantom{\frac b2}\frac a{2R}\right)\tan\left(\frac b{2R}\right)\right) $$