There are a lot of defintions of chaos, some are weaker than other ones. See https://hal.archives-ouvertes.fr/hal-00276838/document for a good survey.
The one you should be thinking of is Devaneys' definition of chaos.
He defined it in his book An Introduction to Chaotic Dynamical Systems, 1986.
Unfortunatly this book is not free so I cannot really give the motivation behind it. But we can find a book review by Floris Takens which quotes the book:
DEFINITION 8.5. Let $V$ be a set. $F: V \to V$ is said to be chaotic on $V$ if
- $f$ has sensitive dependence on initial conditions.
- $f$ is topologically transitive.
- periodic points are dense in $V$.
To summarize, a chaotic map possesses three ingredients; unpredictability, indecomposability, and an element of regularity. A chaotic system is unpredictable because of the sensitive dependence on initial conditions. It cannot be broken down or decomposed into two subsystems (two invariant open subsets) which do not interact under f because of topological transitivity. And, in the midst of this random behavior, we nevertheless have an element of regularity, namely the periodic points which are dense.
So for him it is the melange of regularity and unpredictability.
I have two opinions about why periodicity is important. The first one is that it is in some meaning the opposite of minimal system.
A system is said to be minimal if it has no closed invariant subset which implies that:
- Every orbit is dense
- It has no periodic point in meaningful situation (if it has one the dynamic is trivially finite)
So in a minimal system you can follow the orbit of any point to "travel" around all the space.
On the opposite on Devaneys' chaotic system if you pick any point, there is "high probability" (in a topological sense, that is dense) that you pick a periodic point that won't teach you a lot about the rest of the dynamic.
The second idea is that it was inspired by the work of Li and York on 1975 in the famous paper Period Three Implies Chaos which states that if a map $f$ from $\mathbb{R} \to \mathbb{R}$ has a period three point, then it has periodic point for every period. If you restricte this dynamic to a well choosen interval, it has dense periodic point.
So maybe Devaneys was thinking of this kind of phenomena when he wrote his book.
One last thing, it has been proven on the paper, On Devaney's Definition of Chaos by Banks, Cairns and Stacey, 1992, that (3)+(2) implies (1).
Best Answer
Denseness of the periodic points implies an underlying structure. If, in addition, there's a dense orbit (topological mixing) we can say that every open set of the space contains points with very different behavior.
So not only is regular (periodic) behavior possible, it is very close to very 'random looking' behavior... so that not only do we have sensitivity to initial conditions in the sense of diverging iterates but also sensitivity to initial conditions that two 'close' seeds can have very different behaviour.
A major "discovery" of chaos is that seemingly random phenomena can have a relatively simple origin and have hidden patterns. In this case, the "hidden patterns" are the periodic orbits. They tend to be mostly repulsive so they're hard to find and, in that sense, hidden.