Let's think about this from the geometric point of view first. Isn't it strange that two curved surfaces intersect by a couple of circles? After all, circles are flat, they are essentially planar, and yet we have two three-dimensional surfaces intersect by two circles. So, to get to the bottom of this, I would try and determine the equations of these planes.
This can be done by purely algebraic manipulations. Really, multiply the first equation by $2$ and add it to the second one:
$$
2(yz+zx+xy) + x^2+y^2+z^2 = a^2.
$$
This is the same as:
$$
(x + y + z)^2 = a^2,
$$
which means that if $(x,y,z)$ belongs to our intersection, then either $x+y+z=a$ or $x+y+z=-a$. There we have it: our intersection lies entirely in the union of these two parallel planes: $x+y+z=a$ and $x+y+z=-a$.
So basically instead of intersecting a sphere by a cone we could have intersected it with a pair of planes. Clearly, when we intersect a sphere and a pair of planes, we get a pair of circles.
Thanks to @brainjam, who has helped me plug through all the steps to get to my answer. Here it is step by step.
Given the distances $AB,BC,CA$ I need the co-ordinates of the points $A,B,C$ in 3D space to make this work. Because the triangle is arbitrary in its alignment on the sphere, I can rotate the sphere at will. So I placed point $B$ at $(1,0,0)$, and $A$ on the $xz$ plane.
The angle $\angle BOA$ = $\cos^{-1}\left(\dfrac{2-AB^2}{2}\right)$.
Then point $A = (\cos(\angle BOA),0,\sin(\angle BOA))$.
Point $C=(x,y,z)$ can be solved using the distances from points $A,B$ and $OC=1$.
$OC = 1 = \sqrt{x^2+y^2+z^2}$
which gives $1 = x^2+y^2+z^2$.
For x:
$$
\begin{align}
BC &= \sqrt{(x-1)^2+y^2+z^2} \\
BC^2 &= (x-1)^2+y^2+z^2 \\
BC^2 &= x^2 - 2x + 1 + y^2 + z^2 \\
BC^2 &= -2x + 2 \\
-2x &= BC^2-2 \\
x &=1-BC^2/2 \\
\end{align}
$$
For y:
$$
\begin{align}
AC &= \sqrt{(x-A_x)^2+y^2+(z-A_z)^2}\\
AC^2 &= (x-A_x)^2+y^2+(z-A_z)^2\\
y^2 &= AC^2-(x-A_x)^2-(z-A_z)^2\\
y &=\sqrt{AC^2-(x-A_x)^2-(z-A_z)^2}\\
\end{align}
$$
For z, substituting $y^2$ into $BC^2 = (x-1)^2+y^2+z^2$ gives
$$
\begin{align}
BC^2 &= (x-1)^2+z^2 + AC^2-(x-A_x)^2-(z-A_z)^2 \\
(z-A_z)^2-z^2 &= AC^2-BC^2+(x-1)^2-(x-A_x)^2 \\
A_z^2-2zA_z &= AC^2-BC^2+(x-1)^2-(x-A_x)^2 \\
-2zA_z &= AC^2-BC^2+(x-1)^2-(x-A_x)^2-A_z^2 \\
z &= \frac{AC^2-BC^2+(x-1)^2-(x-A_x)^2-A_z^2}{-2A_z} \\
\end{align}
$$
Those three provide $C=(x,y,z)$ in terms of the initial lengths of the sides.
Now that I have the point coordinates of all three points, I can calculate the midpoints $M_{AB},M_{BC},M_{CA}$on each side.
Calculating the angle between each pair of midpoints using the dot product:
$$\angle{M_{AB}OM_{BC}} = \cos^{-1}(M_{AB}\cdot M_{BC}/|M_{AB}||M_{BC}|)$$
and then the chord length between the points projected out along lines $OM_{AB}$ and $OM_{BC}$ to the surface of the sphere, is given by
$$\frac{\sin(M_{AB}OM_{BC})}{\sin(0.5(180-M_{AB}OM_{BC}))}.$$
Best Answer
Any sphere containing the first circle is of the form $$x^2+(y-b)^2+z^2=b^2+r^2$$ for some real number $b$. Similarly, any sphere containing the second circle is of the form $$(x-a)^2+y^2+(z-c)^2 = a^2+b^2-r^2$$ for some real number $c$. Adding the two equations gives us $$2(x^2-ax+y^2-by+z^2-cz) = 0$$ whenever $(x,y,z)$ belongs to both spheres. But this equation tells us that the vectors $(x,y-b,z)$ and $(x-a,y,z-c)$ have dot product zero. This in turns tells us that the normal vectors to the respective spheres are orthogonal at such a point $(x,y,z)$, which is what you needed to show.