i will try this one.
A Hamiltonian system is (fully) integrable, which means there are $n$ ($n=$ number of dimensions) independent integrals of motion (note that completely integrable hamiltonian systems are very rare, almost all hamiltonian systems are not completely integrable).
What this states in essence (and intuitively) is that the hamiltonian system of dimension $n$ can be decomposed into a cartesian product of a set of $n$ independent sub-systems (e.g in action-angle representation) which are minimally coupled to each other.
This de-composition into a cartesian product of $n$ independent systems (each of which has bounded energy as the whole system has bounded energy), means topologically is the $n$-dimensional torus $S^1 \times S^1 \times ... \times S^1$ ($n$ factors) which is compact (bounded system is topologically compact).
note $S^1$, literaly means topological circle or topological $1$-dimensional sphere. What it means, is that it represents (since this is topology and not geometry) a compact, bounded 1-dimensional space (1-parameter space). So a hamiltonian system with $n$ independent parameters (integrable) is (should be, locally) topologicaly the cartesian product of $n$ (abstract) $S^1$ spaces ($1$ for each parameter/dimension)
Each $S^1$ space represents a simple harmonic oscilator (a simple periodic system, or in other words a system moving on a circle, see the connection with $S^1$ spaces).
When a (completely) integrable hamiltonian system is de-composed into $n$ indepenent sub-systems, in essence this means that (locally, at every neighborhood of a point of the system phase-space) it can be linearised and represented as a stack of (independent) harmonic oscilators (stacks of $S^1$ spaces). This is the basic theorem of Liouville-Arnold on hamiltonian dynamics
For a simple example of a 3-dimensional (actually 2-dimensional, since the configuration space is the surface of a sphere) hamiltonian system which is completely integrable, see the spherical pendulum and analysis thereof
The spherical pendulum is 2-dimensional system (thus the phase-space is 4-dimesnional) and has a second integral of motion the moment about the vertical axis.
(a link on a more advanced analysis on the dynamics of pendula).
In other words the whole is just the sum of its parts.
What would be the hamiltonian space of a (for example $2$-dimensional) system which the dimensions are not independent (not-integrable).
This means the dimensions are correlated and cannot be de-composed into independent sub-systems (i.e a $2$-dimensional torus $S^1 \times S^1$), so topologically it is a $2$-dimensional sphere ($S^2$).
In a $2$-dimensional sphere the $2$ dimensions are correlated and cannot be made flat (i.e cannot be linearised and mapped into a flat space of same dimension, unlike a $2$-dimensional torus, in other words has what is refered as intrinsic curvature).
Elaborating a little on this.
Of course, if one sees the 2-dimensional torus as a 3D object (in effect this means embedded in a flat 3D euclidean space), it has curvature. This is refered as "external" curvature stemming from the embedding into a 3D space. But if one sees the 2-dim torus as a 2-dimensional surface on its own, it has no (zero) curvature. This is refered to as (intrinsic) curvature (in the riemannian sense).
If one takes the 2-dim torus and cuts it and unfold it, one gets the 2-dimensional cylinder
. If further one cuts the 2-dim cylinder and unfold it, one gets a 2-dimensional flat surface. This means the (intrinsic) curvature of the 2-dim torus is zero and can be mapped into a flat space of the same dimension.
For the 2-dim sphere, this is not possible. There is no way it can be cut and mapped into a flat surface of the same dimension. It has (intrinsic) curvature non-zero and this is also a measure away from flatness (and also a measure of dimension correlation). One example is maps of earth (2-dimensional spherical surface) on a flat paper, one can see that the map contains distortions, since there is no mapping of a sphere into a flat surface.
On the other hand if one takes a flat 2-dim surface and makes one boundary periodic, one gets a 2-dim cylinder, if further makes the other boundary also periodic, one gets the 2-dim torus.
In general the conditions under which any given hamiltonian system is (completely) integrable is a very difficult problem.
Still another way to see this is an analogy with probability spaces. Consider 2 event spaces of 2 physical systems consisting of 2 parameters (lets say 2 coins) $\Omega_{12}$ and $\Omega_{AB}$.
When the system is integrable (i.e the parameters are independent, meaning $P(1|2) = P(1)$) then the event space $\Omega_{12}$ is the cartesian product of each sub-space $\Omega_1 \times \Omega_2$. And each outcome of the total system is just the product of the probabilities of each sub-system.
Now consider a second system where the coins are correlated, meaning $P(A|B) \ne P(A)$.
This space $\Omega_{AB}$ cannot be de-composed into 2 independent sub-spaces $\Omega_A$, $\Omega_B$ as their cartesian product since the sub-spaces are not independent. This corresponds to a non-integrable Hamiltonian system (and a topological $2$-d sphere).
The analog of statistical independence in probability event spaces for hamiltonian systems is exactly the existence and functional (more correctly poisson) independence of the appropriate number of integrals of motion (complete integrability).
In other words for a non-integrable system the whole is more than the sum of its parts.
Hope this is useful to you
PS. You might also want to check: Holonomic System, Non-holonomic System, Integrable System
The OP has different questions on the title and body:
Is the motion of a particle in the surface of a torus chaotic?
Generically, yes, mechanical systems are chaotic — though not always, and the purely ballistic dynamics here in particular is not chaotic [1].
there are ballistic trajectories in the surface of the torus that are not periodic [?]
Yes, there are nonperiodic trajectories, but that doesn't mean they are chaotic.
This is easiest to visualize if you map your torus to a periodic square and realize that, for the ballistic movement, the trajectories are then straight lines: the $x$ and $y$ components of the speed must be constant, defining a linear flow on the torus: when $v_x$ and $v_y$ are commensurate, then the trajectory will close on itself after a finite time and repeat itself, i.e., it's periodic; if they are incommensurate, i.e., if $v_x/v_y$ is irrational, then the trajectory covers densely the torus, never closing on itself, i.e., it's quasiperiodic — which is nonperiodic, but also regular (i.e., not chaotic).
Note that this problem has an interesting connection with billiards.
And it's also related to the tori one often finds in the phase spaces of Hamiltonian systems, which play a central role in fundamental dynamical systems' theorems such as Kolmogorov–Arnold–Moser's.
For the mechanical system of a particle constrained to move on a physical toroidal surface (instead of on formal one), I suppose the constraint force (Coriolis like) would make the system similar to a pendulum, which displays oscillations about its equilibrium, but no chaos when unperturbed. That much is suggested by exercise 4 in http://www.phys.virginia.edu/Education/Qualifier/ClassicalMechanics.pdf and by a calculation by Kirk T MacDonald on a system that additionally considers gravity.
Thus this (more) physical system could have quasiperiodic orbits (with incommensurate oscillation and rotation frequencies), but probably not chaos — finding enough constants of motion would be a way of proving this.
[1] Geodesics on the torus seem not to be chaotic — see, e.g., Mildenhall's paper and Jantzen's.
Best Answer
Consider a non-relativistic massless particle with charge $q$ on a 2D torus
$$\tag{1} x ~\sim~ x + L_x , \qquad y ~\sim~ y + L_y, $$
in a constant non-zero magnetic field $B$ along the $z$-axis.
Locally, we can choose a magnetic vector potential
$$\tag{2} A_x ~=~ \partial_x\Lambda, \qquad A_y ~=~ Bx +\partial_y\Lambda, $$
where $\Lambda(x,y)$ is an arbitrary gauge function. Locally, the Lagrangian (which encodes the Lorentz force) is given as
$$\tag{3} L~=~ q ( A_x\dot{x} + A_y\dot{y})~=~qB~x\dot{y}+ \text{(total time derivative)}. $$
[The ordinary kinetic term $T=\frac{m}{2}(\dot{x}^2+\dot{y}^2)$ is absent since the mass $m=0$. This implies that the characteristic cyclotron frequency of the system is infinite.] The Lagrangian momenta are
$$\tag{4} p_x ~=~ \frac{\partial L}{\partial\dot{x} }~=~A_x, \qquad p_y ~=~ \frac{\partial L}{\partial\dot{x} }~=~A_y. $$
Eq. (4) becomes second class constraints, so that the variables $p_x$ and $p_y$ can be eliminated. The Dirac bracket is non-degenerate in the $xy$-sector:
$$\tag{5} \{y,x\}_{DB}~=~\frac{1}{qB}. $$
[Alternatively, this can be seen using the Faddeev-Jackiw method.] In other words, the two periodic coordinates $x$ and $y$ become each others canonical variable with corresponding symplectic two-form
$$\tag{6} \omega_{DB}~=~qB ~\mathrm{d}x \wedge \mathrm{d}y. $$
The corresponding Hamiltonian $H=0$ vanishes. The classical eqs. of motion
$$\tag{7} \dot{x}~=0~=\dot{y} $$
imply a frozen particle.