I would like the simplest example of the failure of an ODE to be locally diffeomorphic to its linearization, despite being locally homeomorphic to it. More precisely, consider x' = f(x) with f(0) = 0 in R^n. Let A = f'(0) so that the local linearization is x' = Ax. Suppose the eigenvalues of A all have nonzero real part (i.e., 0 is a hyperbolic critical point).
The Hartman-Grobman theorem tells us that there is a homeomorphism of a neighborhood of 0 which conjugates the system x'=f(x) to its linearization x'=Ax. If one reads the elementary `differential equations from the dynamical systems point of view' literature, however, you will gain the false impression that there is a diffeomorphism h : U –> U of a nhood of 0 which does this, and that further, one can even do this with h'(0) = I, the identity matrix. The point of this is to ensure that the trajectories of the nonlinear system are tangent to the trajectories of the linearization: if $h'(0)\neq I$ then this may be false.
Smale's stable manifold theorem gives partial information in this direction, saying that the stable manifolds of the system and its linearization are tangent, and similarly for the unstable manifolds. In 2D, at a saddle, this is sufficient to imply that the separatrices of the original system are tangent to those of the linearization. For a node in 2D, or in higher dimensions, I am under the impression this need not hold. I even think I had worked out an example many years ago, which I no longer recall.
Any enlightenment on this issue would be much appreciated. I am not at all expert in these matters, so welcome any corrections, if I have distorted the facts. I am hoping for a 2 dimensional example.
Added later:
Yuri: resonances and normal forms are definitely relevant. When I get time I will look into the references you suggest.
Here's an example of what I am trying to avoid. Consider a flow on the unit disk with trajectories the radial lines y = mx. Conjugate by $(r,\theta) \mapsto (r,f(r,\theta))$ where $f(0,\theta)$ is constant on, say, $[-\pi/2,\pi/2]$, e.g. $f(r,\theta) = r\theta$ on $[-\pi/2,\pi/2]$ and $(2\theta-\pi) + r(\pi-\theta)$ in the left half plane. Now all the trajectories leaving the unit circle in the right half plane approach 0 along the positive x-axis. So, conjugating with such a homeomorphism has replaced a single trajectory with horizontal tangent by an entire interval of such.
I strongly suspect that this sort of pathology doesn't happen with polynomial flows. Perhaps the normal forms will show this. What I would hope is that for each slope, there is a 1-1 correspondence between the trajectories in the original flow and its linearization approaching or leaving the singular point at that slope. In particular, that you can't have a single trajectory in the linearization but a whole interval of them in the nonlinear flow. A counterexample to this hope would be disappointing, but would settle the matter.
Best Answer
In three dimensions, Hartman gave the example $dx/dt=ax$, $dy/dt=(a-b)y+cxz$, $dz/dt=-bz$ where $a>b>0$ and $c \neq 0$. On the other hand, any $C^2$ planar flow is $C^1$ linearizable (another result by Hartman), so you will not find any polynomial examples in the plane. See Linearization via the Lie Derivative by Chicone & Swanson for references and more details.
(Strengthening the smoothness assumption will not give any more than $C^1$ linearizability even in the planar case, as shown by the $C^{\infty}$ example $dx/dt=-x$, $dy/dt=-2y+x^2$, which is easily solved explicitly: $x(t)=x_0 e^{-t}$, $y(t)=(y_0+x_0^2 t) e^{-2t}$. The solution curves are of the form $y=Ax^2-B x^2 \ln|x|$, and are therefore only $C^1$ at the origin, whereas the linearized system of course has smooth solution curves.)