$S$ here is any surface that has $C$ as its boundary, therefore it could be the triangular region made by the three points.
In order to make the dot product think about the geometrical interpretation of the surface integral.
It is just like a normal integral, but it is not on the usual axis/planes, but is on a given surface.
From that you can see that $d\vec S=\nabla dS=\vec n dS$ where $\vec n$ is the normal to the surface.
Therefore, you last paragraph is wrong, what you need is to choose a proper surface and dot its normal.
In a general surface integral of a general vector field this process is as follows:$$\Phi=\int_S \vec V \cdot d\vec S=\int_S (\vec V \cdot \vec n)dS=\int_S V_x dydz+V_ydzdx+V_z dxdy$$
I believe you can go on from here.
1) First, while you've chosen the orientation of $A$ correctly, you should realize that the boundary $\partial D$ of the region $D$ contains $C$ and $-A$. This is important.
Now, apply Green's theorem:
$$\oint_{\partial D} F \cdot dr = \int_C F \cdot dr - \int_A F \cdot dr = \int_D (\nabla \times F) \cdot dA$$
You should argue that the area integral is zero. Instead, in your second point ($\ddagger$), you come to this conclusion but without the supporting logic to back it up (which must come from Green's theorem).
(For fun: realize that $\nabla \times F \neq 0$ at the origin, and this is why integrals enclosing the origin pick up $2\pi$.)
2) Why would you pick an ellipse instead of a circle? Green's theorem tells us that $C$ could be an ellipse instead of an arbitrary curve and thus the integral would have to be $2\pi$, but why put yourself through that work?
Responses to supplemental questions:
1.1) Yes, that is the correct logic.
1.2) No, that is the correct expression for the boundary.
1.3) I'm just observing that the line integral enclosing the origin can only be nonzero if the curl is somewhere nonzero in the region enclosed. You can also understand this in relation to complex analysis, with the point at the origin being a pole, which has a nonzero residue.
1.4) Yes, I wrote this incorrectly. I've added the necessary cdot to the original formula.
2.1) It doesn't seem to me there's anything wrong with that integral; it's just not easy to solve (wolfram gives the right answer for specific values of $a$, $b$, but not symbolically). You could try, for example, the polar equation of an ellipse, but I can't guarantee that would be any easier.
Responses to supplemental supplemental questions:
1.3.1) This problem is very specifically constructed so that $F = \hat \theta /r$. Note the hat: this denotes a unit vector. $\hat \theta$ is always tangent to curves of constant $r$ and is in the direction of increasing $\theta$. When you choose a unit circular path, $r = 1$, and $d\mathbf r/d\theta = r \hat \theta$ also, rendering the integral
$$\int_A F \cdot d\boldsymbol \ell = \int_0^{2\pi} \frac{\hat\theta}{r} \cdot \frac{\hat \theta}{r} \, d\theta = \int_0^{2\pi} \hat \theta \cdot \hat \theta \, d\theta$$
Since $\hat \theta \cdot \hat \theta = 1$, the form I argued follows. That is precisely why choosing a unit circular path is so convenient here.
1.4.1) You have the freedom to choose the surface whose boundary is a given curve. That choice of surface then determines the differential. I don't boldface vectors, so you'll have to figure out from context whether a quantity is scalar or vector. Nevertheless, I wrote $dA = \hat z \, dx \, dy$, and $\hat z$ is a unit vector. The choice of surface that goes with this is that of using the flat $xy$-plane, which is the most convenient (I dare say necessary, as you have no information about what $F$ does outside of the $xy$-plane).
Edit: the definition of $\hat \theta$ is
$$\hat \theta = \frac{1}{r} \frac{\partial \mathbf r}{\partial \theta}$$
write $\mathbf r = (r \cos \theta, r \sin \theta)$, and you get
$$\hat \theta = \frac{1}{r} (-r \sin \theta, r \cos \theta) = \left(- \frac{y}{\sqrt{x^2 + y^2}}, \frac{x}{\sqrt{x^2+ y^2}} \right)$$
You should be able to see that $\hat \theta/r = F$ then.
Best Answer
Green's Theorem states that
$$\int_CP\left(x,y\right)\,dx+Q\left(x,y\right)\,dy=\int_\Omega Q_x\left(x,y\right)-P_y\left(x,y\right)\,dA,$$
where $\Omega$ is the region enclosed by the closed curve $C$, which is oriented counterclockwise.
Therefore, your problem can be reformulated as
$$\int_0^1\int_{x^2}^{\sqrt x}\,dy\,dx.$$
I will leave the details to you.
Edit: Figuring out the limits of your integrals is a cal 1 problem. Just draw what the thing looks like:
I treat it as a type 1 region, and it is clear that $0\leq x\leq1$ and that $x^2\leq y\leq\sqrt x$.