I hope you were using separate unit normals for each segment!
For segment $(-1,0)$ to $(0,1)$ anticlockwise with outer unit normal $\dfrac1{\sqrt{2}}\langle-1,1\rangle$, $x=-t,y=1-t,t:0\rightarrow1, ds=\sqrt{\left(\dfrac{dx}{dt}\right)^2+\left(\dfrac{dy}{dt}\right)^2}~dt=\sqrt{2}~dt$,
$\displaystyle\int_C\mathbf{F}\cdot\mathbf{n}=\int_0^1\dfrac1{\sqrt2}(-x+y^2)~ds=\dfrac{\sqrt2}{\sqrt2}\int_0^1t+(1-t)^2~dt=\dfrac56$.
If you keep the same direction and change to the inner unit normal $\dfrac1{\sqrt{2}}\langle1,-1\rangle$, you get $\displaystyle\int_0^1\dfrac1{\sqrt2}(x-y^2)~ds=\int_0^1-t-(1-t)^2~dt=\dfrac{-5}{6}$, i.e. the sign changes.
If you switch direction to clockwise but keep the outer unit normal $\dfrac1{\sqrt{2}}\langle-1,1\rangle$, you get
$x=t-1,y=t,t:0\rightarrow1,ds=\sqrt{2}~dt$, $\displaystyle\int_0^1\dfrac1{\sqrt2}(-x+y^2)~ds=\int_0^11-t+t^2~dt=\dfrac5{6}$
For this kind of "flux across curve" line integral, $\displaystyle\int_C\mathbf{F}\cdot\mathbf{n}~ds$, it does matter which unit normal you use - switching gives you a sign change. It doesn't matter about the direction of the curve. Why not? We're summing $\mathbf{F}\cdot\mathbf{n}$ along the curve, and this quantity does not depend on the direction of traversal (but does depend on the choice of normal). Further, $ds$, the 'piece of curve', is positive regardless of the direction of traversal. For the same reasons, the general line integral $\displaystyle\int_C\phi(x,y,z)~ds$ doesn't depend on the direction of traversal.
You can contrast this with the "work along curve" line integral,
$\displaystyle\int_C\mathbf{F}\cdot d\mathbf{r}$ where $d\mathbf{r}$ does depend on the direction of traversal. It's $d\mathbf{r}$, the instantaneous vector in the direction of traversal, that changes sign with a change in direction of traversal.
In more physical terms:
- the flux integral is calculating the amount of flow perpendicular to the curve. Adding this up doesn't depend on which way you traverse the curve, but does depend on your idea of inside/outside (the choice of normal).
- the work integral is calculating the amount of work needed to get from A to B. It matters on a given part of the curve whether you're moving with the current/wind or against it, and this depends on the direction of travel.
When you are given problems with two variables but one equation, an easy way to "solve" for one is by eliminating one variable and seeing what you get for the other. In this case, if $Q$ was zero, then you'd have
$$ 1 = -\iint_R \frac{\partial P}{\partial y} dA = \int_C P\,dx $$
and
$$ 1 = -\frac{\partial P}{\partial y}. $$
From this, what is an obvious choice of $P$ based on what you're supposed to end up with? Repeat this for $Q$. Note that I've more or less reverse engineered the problem, but if I just told you what to pick for $P$ and $Q$, you'd be left wondering why other than knowing that it works.
Best Answer
The result is correct indeed we have that the region is symmetric with respect to $y=x$ and the integral for $2x$ is equal to the integral for $2y$ that is
$$\int\int_R 2x\,dA=\int\int_R2y\,dA \implies \int\int_R(2x-2y)dA=0$$