No, the vector field is not defined at the origin, so it's divergence is not defined at the origin (some might say its infinite); i.e you only have $\text{div}(F)=0$ on $\Bbb{R}^3\setminus \{0\} $. So, the divergence theorem is not applicable directly to the full ellipsoid which contains the origin. Instead, consider a small closed ball centered around the origin such that it lies entirely inside the ellipsoid.
Now, apply the divergence theorem to the region between the ellipsoid and ball to deduce that the surface integral over the ellipse is equal to surface integral over sphere (be very very careful with which way the unit outward normal vector points). This surface integral should be much easier to calculate directly. Basically the idea is to use the divergence theorem to switch surfaces.
If you do everything right, you should get an answer of $4 \pi. $
You have the right idea by computing the flows, but you did a mistake while solving the differential equation. The solutions of $x' = x$ are $t\mapsto ae^t$, and the solutions of $y'=-y$ are $t\mapsto be^{-t}$.
For the flow of $Y$, the solutions of $x'= y$ and $y' = x$ are $x(t) = x_0\cosh t + y_0 \sinh t$ and $y(t) = y_0\cosh t + x_0 \sinh t$.
Thus the flows of $X$ and $Y$ are
\begin{align}
\varphi_X(t,(x_0,y_0)) &= (x_0e^t,y_0e^{-t}) \\ \varphi_Y(s,(x_0,y_0)) &= (x_0\cosh s + y_0 \sinh s,y_0\cosh s + x_0 \sinh s)
\end{align}
Try now to find $t$ and $s$, and $x_0$ and $y_0$ such that $\varphi_X\left(t,\varphi_Y(s,(x_0,y_0)) \right)\neq \varphi_Y\left(s,\varphi_X(t,(x_0,y_0))\right)$.
I would add two things to help you understand your mistakes:
- when you solved the differential equations, you had two parameters for each solution, that is uncoherent with the fact that you tried to solve a first order linear equation
- if you know the formulation of Lie brackets, the flows of two vector fields $X$ and $Y$ commute if and only if $[X,Y]=0$. Consequently, if you compute $[X,Y]$, and it is non zero, you know that the flows do not commute. But you know more than that: if you identify the non-zero component of $[X,Y]$, you can identify how they do not commute, and that can help you find $s,t,x_0$ and $y_0$
Edit
Here is a precise computation.
\begin{align}
\varphi_X\left(t,\varphi_Y(s,(x_0,y_0)) \right) &= \varphi_X\left(t,(x_0\cosh s + y_0 \sinh s,y_0\cosh s + x_0 \sinh s)\right)\\
&=\left( x_0(\cosh s) e^{t} + y_0(\sinh s) e^t,y_0(\cosh s) e^{-t} + x_0(\sinh s)e^{-t}\right) \\
\varphi_Y(s,\varphi_X(t,x_0,y_0))) &= \varphi_Y\left(s,(x_0e^{t},y_0e^{-t} )\right) \\
&= \left(x_0e^t(\cosh s) +y_0e^{-t}(\sinh s), y_0e^{-t}(\cosh s) + x_0e^t (\sinh s) \right)
\end{align}
Best Answer
The components of the vector field are $X_1(x, y) = -y$ and $X_2(x, y) = x$. These components define a system of ODEs for a curve: $$ \dot{\gamma}(t) = X(\gamma(t)), $$ or more explicitely $$ \begin{align} \dot{\gamma_1}(t) &= X_1(\gamma(t)) = -\gamma_2(t), \\ \dot{\gamma_2}(t) &= X_2(\gamma(t)) = \gamma_1(t). \end{align}$$ The solution $\gamma(t)$ is the flow generated by the vector field. The function $\sigma$ solves the given system of ODES, therefore it is a flow. For further information take a look at this script from the ETH Zurich