I would start with a change of variables (orthogonal transformation) to diagonalize the quadratic form $xy + yz + xz$.
Try $u = (x+y+z)/\sqrt{3}$, $v = (x - y)/\sqrt{2}$, $w = (x + y - 2 z)/\sqrt{6}$.
let the radii of the spheres be $r_1$ & $r_2 $ & the distance between their centers be $d$ such that
$$(r_1-r_2)<d<(r_1+r_2) \quad \text{where, } r_1\geq r_2$$
Now, assuming that one sphere with radius $r_1$ is centered at the origin (0, 0, 0) & another sphere with radius $r_2$ is centered at a point $(d, 0, 0)$ on the $x$ axis. then we can find out the distance of the common chord from the center of each sphere.
then the cone angle subtended by the plane (circle) of the intersection at the center of the sphere with radius $r_1$ is
$$2\cos^{-1}\left(\frac{d^2+r_1^2-r_2^2}{2dr_1}\right)$$ and the cone angle subtended by the plane (circle) of the intersection at the center of the sphere with radius $r_2$ is
$$2\cos^{-1}\left(\frac{d^2+r_2^2-r_1^2}{2dr_2}\right)$$
In this case, the volume of intersection (engulfed) by the spheres is
$$\frac{\pi}{3}\left(d^3+2r_1^3+2r_2^3-3Kr_1^2-3(d-K)(r_2^2+Kd)\right)$$
where, K is a constant & given as
$$K=\frac{d^2+r_1^2-r_2^2}{2d}$$
by substituting the value of constant K & simplifying, the volume of the lens engulfed (common) by the intersecting spheres is
$$\frac{\pi}{12d}\left(d^4+8d(r_1^3+r_2^3)-3(r_1^2-r_2^2)^2-6d^2(r_1^2+r_2^2)\right)$$
Best Answer
Let's think about this from the geometric point of view first. Isn't it strange that two curved surfaces intersect by a couple of circles? After all, circles are flat, they are essentially planar, and yet we have two three-dimensional surfaces intersect by two circles. So, to get to the bottom of this, I would try and determine the equations of these planes.
This can be done by purely algebraic manipulations. Really, multiply the first equation by $2$ and add it to the second one: $$ 2(yz+zx+xy) + x^2+y^2+z^2 = a^2. $$ This is the same as: $$ (x + y + z)^2 = a^2, $$ which means that if $(x,y,z)$ belongs to our intersection, then either $x+y+z=a$ or $x+y+z=-a$. There we have it: our intersection lies entirely in the union of these two parallel planes: $x+y+z=a$ and $x+y+z=-a$.
So basically instead of intersecting a sphere by a cone we could have intersected it with a pair of planes. Clearly, when we intersect a sphere and a pair of planes, we get a pair of circles.