I too would interpret the symbols $\alpha$, $\beta$, and $\gamma$ as angles
between the sides, and $a$ as the arc length of a side.
I know of two sets of formulas that might be called the spherical law of cosines.
You can find both of them on the page for
spherical trigonometry at Wolfram MathWorld.
One set of formulas they call the cosine rule for sides:
$$\cos a = \cos b \cos c + \sin b \sin c \cos\alpha$$
(with corresponding formulas with $\cos b$ and $\cos c$, respectively, on the left).
The other set of formulas they call the cosine rule for angles:
$$\cos\alpha = -\cos\beta \cos\gamma + \sin\beta \sin\gamma \cos a$$
(with corresponding formulas with $\cos\beta$ and $\cos\gamma$, respectively, on the left).
You can easily put any of these formulas in a format like the one you have used;
from the cosine rule for angles you would end up with
$$\cos a = \frac{\cos\alpha + \cos\beta \cos\gamma}{\sin\beta \sin\gamma}.$$
Note the "$+$" operation in this formula where you have "$-$".
It turns out this has no effect on your result, since the right-hand operand
turns out to be zero.
Where you make a misstep is here:
$$\frac{\sqrt{6}+\sqrt{2}}{2}=4\cos(\alpha)$$
$$2(\sqrt{6}+\sqrt{2})=\cos(\alpha)$$
You need to divide both sides of the first equation by $4$, so the second equation
should have been
$$\frac{\sqrt{6}+\sqrt{2}}{8} = \cos\alpha.$$
One way you might notice the error is that $2(\sqrt{6}+\sqrt{2}) > 1$,
meaning it cannot be the cosine of any real-valued angle.
The result is not a "nice" angle like $60$ degrees or $75$ degrees, so
either you give a calculator's approximation or you leave it as an expression
involving $\cos^{-1}$.
You could find the other two sides of the triangle using law of cosines,
or you could use the Law of Sines for a spherical triangle, which is
relatively easy to remember:
$$\frac{\sin\alpha}{\sin a} = \frac{\sin\beta}{\sin b} = \frac{\sin\gamma}{\sin c}.$$
That's actually three equations, two of which allow you to solve for your
unknown sides $b$ and $c$.
I'd ignore the radius $R$ (i.e. take the earth's radius as unit of length) and then convert to Cartesian coordinates:
\begin{align*}
x&=\cos\phi\cos\lambda\\
y&=\cos\phi\sin\lambda\\
z&=\sin\phi
\end{align*}
These vectors point from the center of the sphere to the vertices of your polygon. Two such vertices are joined by a greatcircle arc. That greatcircle is the intersection of your sphere with a plane through the origin. That plane through the origin is defined by its normal vector. The cross product of two vectors is perpendicular to both of them, so you can use the cross product to compute these normal vectors. Writing $v_i=(x_i,y_iz_i)$ for the vertices, you get the normals of the edges as
$$n_i = v_i\times v_{i+1}$$
The angle at a vertex is equal to the angle between the corresponding normal vectors. You can compute that using the dot product, dividing by the lengths to ensure normalization:
$$\alpha_i = \pm\arccos-
\frac{n_{i-1}\cdot n_{i}}{\lVert n_{i-1}\rVert\,\lVert n_i\rVert}$$
The minus sign in there is because for a normalized dot product of $1$, both normals point in the same direction, so you have zero change in direction but the inner angle there is $180$.
So which sign should you choose for the $\pm$ in there? For that, look at the determinant of the $3\times3$ matrix formed by $v_{i-1}$, $v_i$ and $v_{i+1}$. The sign of that matrix will tell you whether the triangle these three vectors form on the surface is oriented clockwise or counter-clockwise. Take the sign from this and the value from the $\arccos$ above and you should be almost done.
As usual with signs, you have a decision to make: which sign is which? Well, one choice of sign (e.g. exactly as described above) will lead you one polygon, the other (with the sign of the determinant flipped, or eqivalently the order of vectors inside the determinant reversed) leads to the complementary polygon. One will have all its vertices in clockwise order, the other in counter-clockwise order. Trying this out on a tiny example (e.g. three points forming three right angles) will tell you which one is which.
Best Answer
I don’t believe the formula is correct as quoted.
We’re dealing with an equilateral (and equiangular!) spherical triangle, and to avoid (my own) confusion, I’ll call the side-length $s$ and the angle $\theta$. Both are angles, of course. The most obvious thing is to bisect the triangle into two right triangles, each with hypotenuse $s$ and one side $s/2$, this latter side being opposite the angle $\theta/2$, while the other angle of the triangle is still $\theta$.
Now, one of the standard formulas for right spherical triangles is $$ \cos c=\cot A\cot B\,, $$ and for us, $c=s$, $A=\theta$, $B=\theta/2$. So we differentiate our relation $\cos s=\cot\theta\cot(\theta/2)$, and get \begin{align} -\sin s\frac{ds}{d\theta}&=-\cot\theta\csc^2(\theta/2)\cdot\frac12 -\csc^2\theta\cot(\theta/2)\\ \frac{ds}{d\theta}&=\frac{\frac12\cot\theta\csc^2(\theta/2)+\csc^2\theta\cot(\theta/2)}{\sin s}\,. \end{align} Now let’s test this out on the nicest possible equilateral triangle, the one whose sides and angles are both $90^\circ$. Here, the first term in the numerator drops out, ’cause $\cot90^\circ=0$, but everything else evaluates to $1$, and so even $ds/d\theta=1$ at this test-point. Yet, the formula evaluates to $\sqrt2/2$, unequal to $1$.