I'm trying to get an equation for a solid angle of a segment of octahedron in the same vein as described in this article cubemap-texel-solid-angle. I ended up having to integrate
$$\int \int \frac{1}{(x^2+y^2+(1-x-y)^2)^\frac{3}{2}} \,\mathrm{d}x \,\mathrm{d}y$$ where $0 \leq x \leq 1$ and $0 \leq y \leq 1-x$. That is, integral over a segment of a triangle mapped onto the sphere(one octant). Does anyone know how to integrate that?
Subdivided polyhedrons by courtesy of Gavin Kistner
Update
Thanks to the general formula from this answer:
$$\omega=\cos^{-1}\left(\frac{\cos\alpha-\cos\beta\cos\gamma}{\sin\beta\sin\gamma}\right)-\sin^{-1}\left(\frac{\cos\beta-\cos\alpha\cos\gamma}
{\sin\alpha\sin\gamma}\right)-\sin^{-1}\left(\frac{\cos\gamma-\cos\alpha\cos\beta}{\sin\alpha\sin\beta}\right)$$
We can calculate a solid angle for all the triangles. Here is the gist and the shadertoy. The naive implementation is not numerically stable at small angles.
Update See this answer
Best Answer
The general problem I think you have described is to find the solid angle subtended at one vertex of a tetrahedron.
If we label the vertex in question $O$ and put it at the center of a unit sphere, then project the opposite face onto the sphere, we get a spherical triangle. The lengths of the "sides" of that triangle are the angles $\alpha,$ $\beta,$ and $\gamma$ between the edges of the tetrahedron that meet at $O.$ The angles at the vertices of the spherical triangle are the dihedral angles $A,$ $B,$ and $C$ between the faces of the tetrahedron that meet at $O.$ The usual convention is we use the name $A$ for the angle between the sides of length $\beta$ and $\gamma,$ the name $B$ for the angle between the sides of length $\alpha$ and $\gamma,$ and the name $C$ for the angle between the sides of length $\alpha$ and $\beta.$
The solid angle at $O$ is then the area of the spherical triangle, which in turn is equal to the spherical excess of that triangle, defined as $$ E = A + B + C - \pi. $$
But the information that you seem to be assuming is that you know the three angles $\alpha,$ $\beta,$ and $\gamma.$ So the question becomes how to find $E$ in terms of those angles.
The spherical law of cosines says that $$ \cos\alpha = \cos\beta \cos\gamma + \sin\beta \sin\gamma \cos A. $$ Solving for $A$ we get $$ A = \arccos \left(\frac{\cos\alpha - \cos\beta \cos\gamma} {\sin\beta \sin\gamma}\right) .$$
There are similar formulas involving the angles $B$ and $C,$ with the results $$ B = \arccos \left(\frac{\cos\beta - \cos\alpha \cos\gamma} {\sin\alpha \sin\gamma}\right) $$ and $$ C = \arccos \left(\frac{\cos\gamma - \cos\alpha \cos\beta} {\sin\alpha \sin\beta}\right) .$$
As a result, one formula for the spherical excess is \begin{align} E &= \arccos \left(\frac{\cos\alpha - \cos\beta \cos\gamma} {\sin\beta \sin\gamma}\right) \\ &\qquad + \arccos \left(\frac{\cos\beta - \cos\alpha \cos\gamma} {\sin\alpha \sin\gamma}\right) \\ &\qquad + \arccos \left(\frac{\cos\gamma - \cos\alpha \cos\beta} {\sin\alpha \sin\beta}\right) - \pi. \end{align}
The formula shown in the question is a variation of this formula that can be obtained using the identity $\arccos(x) = \frac\pi2 - \arcsin(x).$
I would be suspicious of the numerical stability of this formula for very small spherical angles (that is, when you have divided your sphere into a very large number of triangular facets), because neither $\arccos(x)$ nor $\arcsin(x)$ is very accurate when $x$ is close to $1.$ You might be better off with another formula such as $$ E = 2 \arctan\left(\frac{\tan\frac\alpha 2 \tan\frac\beta 2 \sin C} {1 + \tan\frac\alpha 2 \tan\frac\beta 2 \cos C}\right), $$ (from here) using formulas such as $$ \cos C = \frac{\cos\gamma - \cos\alpha \cos\beta}{\sin\alpha \sin\beta} $$ and $$ \sin C = \sqrt{1 - \cos^2 C}. $$
This should be fine if the three angles of the triangle are approximately equal (as seems to be the case in your "octahedron"-based construction). If one of the angles is almost $180$ degrees and the other two are almost zero you might want to compute $\sin C$ differently.