I hope you were using separate unit normals for each segment!
For segment $(-1,0)$ to $(0,1)$ anticlockwise with outer unit normal $\dfrac1{\sqrt{2}}\langle-1,1\rangle$, $x=-t,y=1-t,t:0\rightarrow1, ds=\sqrt{\left(\dfrac{dx}{dt}\right)^2+\left(\dfrac{dy}{dt}\right)^2}~dt=\sqrt{2}~dt$,
$\displaystyle\int_C\mathbf{F}\cdot\mathbf{n}=\int_0^1\dfrac1{\sqrt2}(-x+y^2)~ds=\dfrac{\sqrt2}{\sqrt2}\int_0^1t+(1-t)^2~dt=\dfrac56$.
If you keep the same direction and change to the inner unit normal $\dfrac1{\sqrt{2}}\langle1,-1\rangle$, you get $\displaystyle\int_0^1\dfrac1{\sqrt2}(x-y^2)~ds=\int_0^1-t-(1-t)^2~dt=\dfrac{-5}{6}$, i.e. the sign changes.
If you switch direction to clockwise but keep the outer unit normal $\dfrac1{\sqrt{2}}\langle-1,1\rangle$, you get
$x=t-1,y=t,t:0\rightarrow1,ds=\sqrt{2}~dt$, $\displaystyle\int_0^1\dfrac1{\sqrt2}(-x+y^2)~ds=\int_0^11-t+t^2~dt=\dfrac5{6}$
For this kind of "flux across curve" line integral, $\displaystyle\int_C\mathbf{F}\cdot\mathbf{n}~ds$, it does matter which unit normal you use - switching gives you a sign change. It doesn't matter about the direction of the curve. Why not? We're summing $\mathbf{F}\cdot\mathbf{n}$ along the curve, and this quantity does not depend on the direction of traversal (but does depend on the choice of normal). Further, $ds$, the 'piece of curve', is positive regardless of the direction of traversal. For the same reasons, the general line integral $\displaystyle\int_C\phi(x,y,z)~ds$ doesn't depend on the direction of traversal.
You can contrast this with the "work along curve" line integral,
$\displaystyle\int_C\mathbf{F}\cdot d\mathbf{r}$ where $d\mathbf{r}$ does depend on the direction of traversal. It's $d\mathbf{r}$, the instantaneous vector in the direction of traversal, that changes sign with a change in direction of traversal.
In more physical terms:
- the flux integral is calculating the amount of flow perpendicular to the curve. Adding this up doesn't depend on which way you traverse the curve, but does depend on your idea of inside/outside (the choice of normal).
- the work integral is calculating the amount of work needed to get from A to B. It matters on a given part of the curve whether you're moving with the current/wind or against it, and this depends on the direction of travel.
Best Answer
Your attempt isn't so bad, but when you sweep the hypothenuse by $dx$, the elementary area that you cover isn't $l\,dx$ but $\dfrac l{\sqrt2}dx$ because you need to multiply the base by the height, not by the length.
Said differently, if you consider that you are sweeping perpendicularly to the hypothenuse, then the area is $l\,\dfrac{dx}{\sqrt2}$, because the motion in the perpendicular direction is smaller than in the horizontal direction.
Hence,
$$\int_0^1 \sqrt{2}(1-x)\frac1{\sqrt2}\,dx = \frac{1}{2}.$$