You have the right idea by computing the flows, but you did a mistake while solving the differential equation. The solutions of $x' = x$ are $t\mapsto ae^t$, and the solutions of $y'=-y$ are $t\mapsto be^{-t}$.
For the flow of $Y$, the solutions of $x'= y$ and $y' = x$ are $x(t) = x_0\cosh t + y_0 \sinh t$ and $y(t) = y_0\cosh t + x_0 \sinh t$.
Thus the flows of $X$ and $Y$ are
\begin{align}
\varphi_X(t,(x_0,y_0)) &= (x_0e^t,y_0e^{-t}) \\ \varphi_Y(s,(x_0,y_0)) &= (x_0\cosh s + y_0 \sinh s,y_0\cosh s + x_0 \sinh s)
\end{align}
Try now to find $t$ and $s$, and $x_0$ and $y_0$ such that $\varphi_X\left(t,\varphi_Y(s,(x_0,y_0)) \right)\neq \varphi_Y\left(s,\varphi_X(t,(x_0,y_0))\right)$.
I would add two things to help you understand your mistakes:
- when you solved the differential equations, you had two parameters for each solution, that is uncoherent with the fact that you tried to solve a first order linear equation
- if you know the formulation of Lie brackets, the flows of two vector fields $X$ and $Y$ commute if and only if $[X,Y]=0$. Consequently, if you compute $[X,Y]$, and it is non zero, you know that the flows do not commute. But you know more than that: if you identify the non-zero component of $[X,Y]$, you can identify how they do not commute, and that can help you find $s,t,x_0$ and $y_0$
Edit
Here is a precise computation.
\begin{align}
\varphi_X\left(t,\varphi_Y(s,(x_0,y_0)) \right) &= \varphi_X\left(t,(x_0\cosh s + y_0 \sinh s,y_0\cosh s + x_0 \sinh s)\right)\\
&=\left( x_0(\cosh s) e^{t} + y_0(\sinh s) e^t,y_0(\cosh s) e^{-t} + x_0(\sinh s)e^{-t}\right) \\
\varphi_Y(s,\varphi_X(t,x_0,y_0))) &= \varphi_Y\left(s,(x_0e^{t},y_0e^{-t} )\right) \\
&= \left(x_0e^t(\cosh s) +y_0e^{-t}(\sinh s), y_0e^{-t}(\cosh s) + x_0e^t (\sinh s) \right)
\end{align}
$\def\sgn{\operatorname{sgn}}
\def\sD{\mathcal{D}}$I would be grateful if someone could correct me in case there is any typo in the following expressions.
After solving the ODE, the flows for $V$ and for $W$ can be seen to be, respectively,
$$
\begin{aligned}
\theta_t(x,y,z)&=
\begin{cases}
(x+t,0,z),&y=0,t>-x,\\
(x+t,y,z-\arctan\frac{x+t}{y}+\arctan\frac{x}{y}),&y\neq 0.
\end{cases}\\
\psi_s(x,y,z)&=
\begin{cases}
(0,y+s,z),&x=0,s>-y,\\
(x,y+s,z-\arctan\frac{y+s}{x}-\arctan\frac{y}{x}),&x\neq 0.
\end{cases}
\end{aligned}
$$
Denote $\sD_\theta$ and $\sD_\psi$ to the flow domains of $\theta$ and $\psi$, respectively.
Composing in the two possible orders, on the one hand we have that for all pair of times $(t,s)$ and points $p=(x,y,z)$ such that $(s,p)\in\sD_\psi$ and $(t,\psi_s(p))\in\sD_\theta$,
$$
\theta_t(\psi_s(x,y,z))
=
\begin{cases}
(t,y+s,z-\arctan\frac{t}{y+s}) & x=0,s>-y,\\
(x+t,0,z-\arctan\frac{y}{x}) & x\neq 0, y=-s, t>-x\\
(x+t,y+s,\\z+\arctan\frac{y+s}{x}+\arctan\frac{x}{y+s}-\arctan\frac{x+t}{y+s}-\arctan\frac{y}{x})
& x\neq 0, y\neq -s.
\end{cases}
$$
On the other hand, for all times $(t,s)$ and $p=(x,y,z)$ with $(t,p)\in\sD_\theta$ and $(s,\theta_t(p))\in\sD_\psi$,
$$
\psi_s(\theta_t(x,y,z))
=
\begin{cases}
(x+t,s,z+\arctan\frac{s}{x+t}) & y=0,t>-x,\\
(0,y+s,z+\arctan\frac{x}{y}) & y\neq 0, x=-t, s>-y\\
(x+t,y+s,\\z-\arctan\frac{x+t}{y}-\arctan\frac{y}{x+t}+\arctan\frac{y+s}{x+t}+\arctan\frac{x}{y})
& y\neq 0, x\neq -t.
\end{cases}
$$
Recall that for $u\in\mathbb{R}\setminus\{0\}$, we have
$$
\label{1}\tag{1}
\frac{\pi}{2}\sgn u=\arctan u+\arctan\frac{1}{u}.
$$
(This identity follows after differentiating by $u$.)
The only possible choice of $(x,y,z,t,s)$ such that both $\theta_t(\psi_s(x,y,z))$ and $\psi_s(\theta_t(x,y,z))$ are defined and such that at the same time $\theta_t(\psi_s(x,y,z))\neq\psi_s(\theta_t(x,y,z))$, is for $-t\neq x\neq 0\neq y\neq -s$. Indeed, if $(x,y,z,t,s)$ are as such, then both orders of flow compositions are defined and, using \eqref{1}, we have $\theta_t(\psi_s(x,y,z))=\psi_s(\theta_t(x,y,z))$ if and only if
$$
\sgn x\sgn(y+s)+\sgn y\sgn(x+t)=\sgn x\sgn y+\sgn(x+t)\sgn(y+s).
$$
But this identity can be seen to not hold for $(x,y,t,s)=(1,-1,-2,2)$.
What is happening? Consider the characterization from Lee's book of commutativity of vector fields vs commutativity of the flows (it's given here). Then, for $(x,y)=(1,-1)$ even though both $\theta_t(\psi_s(x,y,z))$ and $\psi_s(\theta_t(x,y,z))$ are defined for $(t,s)=(-2,2)$, it happens that there are no open intervals $J,K$ containing $0$ with $-2\in J$, $2\in K$ such that $\theta_t(\psi_s(x,y,z))$ is defined for all $(t,s)\in J\times K$ or $\psi_s(\theta_t(x,y,z))$ is defined for all $(t,s)\in J\times K$. (You can check this yourself by inspecting the formulas for the composites of the flows, trying to move continuously $(t,s)$ from $(-2,2)$ to $(0,0)$.)
Best Answer
There are a couple of problems with your solution. Let me try to explain what they are.
Nothing in the problem statement guarantees that $t\mapsto F(t,s)$ and $t\mapsto F(s,t)$ are maximal integral curves. But they are integral curves. Note that the first paragraph of the proof of Theorem 9.12 in my book shows that any two integral curves for the same vector field with the same initial condition must agree on their common domain (I don't know why I didn't state that as a separate theorem!), and therefore your conclusions in the last sentence above are true.
Theorem 9.44 doesn't apply in this situation. The context of that theorem (stated at the beginning of the "Commuting Vector Fields" section on page 231) is that we are talking about smooth vector fields defined on a smooth manifold. Because $F(\mathbb R^2)$ is typically not going to be a smooth submanifold of $M$, we can't conclude from Theorem 9.44 that $X$ and $Y$ commute on $F(\mathbb R^2)$ if and only if their flows commute on $F(\mathbb R^2)$. (That approach could be made to work in the special case that $F$ is a smooth embedding, once you show that the vector fields $X$ and $Y$ are tangent to the submanifold $N = F(\mathbb R^2)$ and therefore restrict to vector fields on $N$.)
A better approach to this problem is based on the notion of $\boldsymbol F$-related vector fields. Take a look at Propositions 8.30 (naturality of Lie brackets) and 9.6 (naturality of integral curves), and think about applying them to the vector fields $\partial/\partial x$ and $\partial/\partial y$ on $\mathbb R^2$.