Hint: Think about the relationship between the radius of the circle, the diameter of the circle, and the side length of the square.
$\hskip1.5in$
Do you know the area of a square given its side length?
Polar integration seems to be the best method for creating a general formula for this.
Re-expressing both circles as polar yields this:
$r1 = 1$ and $r2 = cos\theta+\sqrt(a^2-sin^2\theta)$ based off of the general formula for a polar circle, centered at $(1, 0)$ with radius $a$. Really all we need here are the points of intersection which can easily be solved by setting each equation equal and following algebraic and trigonometric rules
$$cos\theta+\sqrt(a^2-sin^2\theta) = 1$$
$$a^2-sin^2\theta=1+cos^2\theta-2cos\theta$$
Replacing $sin^2$ and $cos^2$ with 1 and solving for theta yields:
$$\theta=\pm arcos(a^2-2/-2)$$
From here it should be noted that we must add $\pi$ to the resulting angle because our result accounts for the part of the circle outside of the central unit circle. So our angles from which we base our integral are
$$\theta = \pi \pm arcos(a^2-2/-2)$$
At this point I believe you can evaluate this regularly
$$.5\int_{\theta_1}^{\theta_2} 1^2-[cos\theta+\sqrt({a^2-sin^2\theta})]^2 \,d\theta$$
There may be some errors I missed in my answer however this general method should work
Best Answer
Let $r$ be the radius of the circle and $a$ be half the length of a side of the square.
If $r \leq a$ then the overlapping area is the area of the whole circle $\pi r^2$.
If $r \geq a \sqrt{2}$ then the overlapping area is the area of the whole square $a^2$.
Now to the more interesting case where $a < r < a\sqrt{2}$.
Exploiting the symmetry of the problem we divide the plane to 8 parts as shown in the picture, so that the overlapping area is 8 times the colored area.
Let $\theta = \arccos(a/r)$ the angle shown in the picture. The blue area is a circular sector of angle $\frac{\pi}{4}-\theta$ so its area equals $(\frac{\pi}{4}-\theta)\frac{r^2}{2}$. The red triangle has area $\frac{1}{2}ar\sin\theta$.
Putting all these together, the total overlapping area is $(\pi-4\theta)r^2+4ar\sin\theta$.