Firstly, to remove any doubts, you are correct so far in that:
- The radius of the circle is $10$ (though you kind of went about this in a roundabout method)
- The diagonal is $10\sqrt{2}$
- The square has area $100$
- The area of each quarter-circle is $25\pi$ for a total of $50\pi$
However, you forget that they overlap. This means that while the quarter circles sum to $50\pi$ in area, some of it just doesn't matter, since the other circle is already covering up that bit of space. (If you've ever heard of the inclusion-exclusion principle for stuff like probabilities and counting problems, it has a very similar feel to it.) Consider the below picture, which only has one of the circles in it. Imagine rotating a copy of this square $180^\circ$ and superimposing it on itself: you obviously end up with the original picture you have, but with overlap:
In fact, it is the area of the overlap you are tasked to find. That overlap is the blue area in the picture below:
With the picture color-coded like this, it should not be difficult to convince you that
$$\text{(The blue area)} = \text{(The square's area)} - \text{(The two bits of green area)}$$
In a similar vein, the first picture should show you that the area of one portion of the green area can be obtained by taking the square, and subtracting the area of the quarter circle. Double that for the green area, and subtract that from the area of the square. Then you have your result, the area of the overlapping region.
As Matthew pointed out, there is a misapplication of the inverse power rule when integrating
$$I=\int\mathrm dx\,\sqrt{R^2-x^2}$$
However, if you truly want to avoid trigonometric substitution when evaluating this integral and want to remain strictly in Cartesian coordinates, here is an alternative that may or may not be what you're looking for.
First, rewriting the square root as shown below, we can rewrite the indefinite integral $I$ as
$$\begin{align*}I & =\int\mathrm dx\,\frac {R^2-x^2}{\sqrt{R^2-x^2}}\\ & =R^2\int\frac {\mathrm dx}{\sqrt{R^2-x^2}}\color{brown}{-\int\mathrm dx\,\frac {x^2}{\sqrt{R^2-x^2}}}\end{align*}$$
Next, using integration by parts on $I$ with $u=\sqrt{R^2-x^2}$ and $v=x$, then
$$I=x\sqrt{R^2-x^2}\color{brown}{+\int\mathrm dx\,\frac {x^2}{\sqrt{R^2-x^2}}}$$
Adding the two different expressions for $I$, and observing that the brown expressions cancel each other out, then we're just left with a simplified integral
$$\begin{align*}I & =\frac 12x\sqrt{R^2-x^2}+\frac 12 R^2\int\frac {\mathrm dx}{\sqrt{R^2-x^2}}\\ & =\frac 12x\sqrt{R^2-x^2}+\frac 12 R^2\arcsin\left(\frac xR\right)+C\end{align*}$$
Imposing the limits from $x=0$ to $x=R$, then the expression becomes
$$\int\limits_0^R\mathrm dx\,\sqrt{R^2-x^2}=\frac 12 R^2\arcsin1\color{blue}{=\frac {\pi}4 R^2}$$
Which is indeed the area of a quarter circle!
Best Answer
Imagine a point $P$ moving along the unit circle from $0$ to $\pi/2$. It has a speed in the $x$-direction, and a speed in the $y$-direction (I'll call these $x$-speed and $y$-speed). Note that the $x$-speed of $P$ is constantly changing, and this is reflected in the graph that it draws (the unit circle): we are drawing the $y$-position, but with a changing $x$-speed.
Whereas, the sine graph is drawing the $y$-position with respect to the angle: this is a constant speed. Near an angle of $0$, the point $P$ has a very small $x$-speed, but the sine graph already has a significant $x$-speed. This results in the sine graph covering more area near the start of the movement of $P$ than $P$ does. Thus, the area under the sine wave is bigger than the area under the unit circle.
You could imagine taking the top half of the unit circle and stretching it at the ends where it hits the $x$-axis, to compensate for the slow $x$-speed, and turning it into the sine graph.