Integration is not needed. The area of the circle is $\pi$, so the integral of $c$ over the circleis $\pi c$. It follows that $c=\dfrac{1}{\pi}$.
If you really want to integrate, you could calculate the integral of $c\,dx\,dy$ over the circle in two ways.
(i) Integrate first with respect to $y$, with $y$ going from $-\sqrt{1-x^2}$ to $\sqrt{1-x^2}$. Then integrate with respect to $x$, from $-1$ to $1$.
This is because the top half of the circle has equation $y=\sqrt{1-x^2}$, and the bottom half has equation $y=\sqrt{1-x^2}$. Alternately, integrate from $0$ to $\sqrt{1-x^2}$ and double the result. Somewhat more simply, let $y$ go from $0$ to $\sqrt{1-x^2}$, and then $x$ from $0$ to $1$, and multiply the result by $4$.
Alternately, integrate first with respect to $x$, going from $-\sqrt{1-y^2}$ to $\sqrt{1-y^2}$. This fundamentally the same as what we did above, perhaps a little less familiar-feeling.
(ii) Or else switch to polar coordinates, since circles love polars. The $c\,dx\,dy$ turns into $cr\,dr\,d\theta$, and we integrate from $r=0$ to $r=1$, then from $\theta=0$ to $\theta=2\pi$.
Hints:
$P\left\{ X>Y\right\} =\int f_{Y}\left(y\right)P\left\{ X>Y\mid Y=y\right\} dy=\int f_{Y}\left(y\right)P\left\{ X>y\right\} dy$
Alternative:
$\int_{0}^{\infty}\int_{y}^{\infty}f_{\left(X,Y\right)}\left(x,y\right)dxdy=\int_{0}^{\infty}\int_{y}^{\infty}f_{X}\left(x\right)f_{Y}\left(y\right)dxdy=\int_{0}^{\infty}f_{Y}\left(y\right)\int_{y}^{\infty}f_{X}\left(x\right)dxdy$
Note that integrand $\int_{y}^{\infty}f_{X}\left(x\right)dx$ can
be recognized as: $P\left\{ X>y\right\} $
Since $X$ and $Y$ are independent the joint PDF of $(X,Y)$ is the product of the PDFs of $X$ and $Y$.
Best Answer
Consider the interval $0<x<y<2$. Now $x$ can take any value between $0$ and $y$. Thus we obtain:
$$\int_0^y \frac{3(x^2+y^2)}{16} dx$$
Now $y$ can take any value between $x$ and $2$. Moreover, since $x$ can take any value greater than $0$ we have that $y$ can take any value between $0$ and $2$. Thus we obtain:
$$\int_0^2\int_0^y \frac{3(x^2+y^2)}{16} dxdy$$
Evaluting this integral gives $1$ as desired.