$S$ here is any surface that has $C$ as its boundary, therefore it could be the triangular region made by the three points.
In order to make the dot product think about the geometrical interpretation of the surface integral.
It is just like a normal integral, but it is not on the usual axis/planes, but is on a given surface.
From that you can see that $d\vec S=\nabla dS=\vec n dS$ where $\vec n$ is the normal to the surface.
Therefore, you last paragraph is wrong, what you need is to choose a proper surface and dot its normal.
In a general surface integral of a general vector field this process is as follows:$$\Phi=\int_S \vec V \cdot d\vec S=\int_S (\vec V \cdot \vec n)dS=\int_S V_x dydz+V_ydzdx+V_z dxdy$$
I believe you can go on from here.
Line from $\;(0,0)\to (0,1)\;$ :
$$\ell_1:\;t(0,1)+(1-t)(0,0)=(0,t)\;,\;\;t\in [0,1]\implies$$
$$\int_{\ell_1}Pdx+Qdy=\int_0^1\left((-1)t^2(0\cdot )+(t+1)0^2\right)dt=0$$$${}$$
Line from $\;(0,1)\to \left(\frac12,0\right)\;$ :
$$\ell_2:\;t\left(\frac12,0\right)+(1-t)(0,1)=\left(\frac t2,1-t\right)\;,\;\;t\in [0,1]\implies$$
$$\int_{\ell_2}Pdx+Qdy=\int_0^1\left(\left(\frac t2-1\right)(1-t)^2\cdot\frac12dt+(2-t)\frac{t^2}4(-dt)\right)=$$
$$=\frac14\int_0^1(t-2)(t-1)^2dt+\frac14\int_0^1(t^3-2t^2)dt=\frac14\int_0^1\left(2t^3-6t^2+5t-2\right)dt=$$$${}$$
$$=\frac14\left(\frac12-2+\frac52-2\right)=-\frac14$$
Line from $\;\left(\frac12,0\right)\to (0,0)\;$ :
$$\ell_3:\;t(0,0)+(1-t)\left(\frac12,0\right)=\left(\frac12(1-t),0\right)\;,\;\;t\in [0,1]\implies$$
$$\int_{\ell_3}Pdx+Qdy=\int_0^1\left(-\frac12-t\right)\left(-\frac12\right)+\left(1)0^2\right)dt=\int_0^1\left(\frac14+\frac t2\right)dt=$$
$$=\frac14+\frac14=\frac12$$
so the line integral equals $\;-\cfrac14+\cfrac12=\cfrac14\;$
Using Green's theorem:
$$\frac{\partial Q}{\partial x}=2x(y+1)\;,\;\;\frac{\partial P}{\partial y}=2(x-1)y\implies\frac{\partial Q}{\partial x}-\frac{\partial P}{\partial y}=2(x+y)$$
and the integral becomes on the given region:
$$\int_0^{1/2}\int_0^{-2x+1}2(x+y)dydx=\int_0^{1/2}\left(2x(-2x+1)+(2x-1)^2\right)dx=$$
$$=\int_0^{1/2}(-2x+1)dx=-\frac14+\frac12=\frac14$$
Best Answer
As Pringoooals suggested, use Green's Theorem. The key is to integrate $x$ first, then $y$.
$\displaystyle\int_C(y+e^\sqrt{x}) dx + (xe^{y^2}) dy = \iint_{\Omega}\left[\dfrac{\partial}{\partial x}(xe^{y^2}) - \dfrac{\partial}{\partial x}(y+e^{\sqrt{x}})\right]\,dx\,dy = \iint_{\Omega}(e^{y^2}-1)\,dx\,dy = \int_{0}^{2}\int_{0}^{2y}(e^{y^2}-1)\,dx\,dy = \int_{0}^{2}2y(e^{y^2}-1)\,dy = \left[e^{y^2}-y^2\right]_{0}^{2} = e^4-5$.