Manifold Hypothesis – Physical Interpretation

applied-mathematicsds.dynamical-systemsmachine learningmanifolds

Motivation:

Most dimensionality reduction algorithms assume that the input data are sampled from a manifold $\mathcal{M}$ whose intrinsic dimension $d$ is much smaller than the ambient dimension $D$. In machine learning and applied mathematics circles this is typically known as the manifold hypothesis.

Empirically, this is observed to be true for many kinds of data including text data and natural images. In fact, Carlsson et al. found that the high-dimensional space of natural images has a two-dimensional embedding that is homeomorphic to the Klein bottle [1].

Question:

Might there be a sensible physical interpretation of this phenomenon? My current intuition, from the perspective of dynamical systems, is that if we can collect large amounts of data for a particular process then this process must be stable. Might there be a reason why stable physical processes would tend to have low-dimensional phase spaces?

Might there be a better physical perspective for interpreting this phenomenon?

References:

  1. Carlsson, G., Ishkhanov, T., de Silva, V. et al. On the Local Behavior of Spaces of Natural Images. Int J Comput Vis 76, 1–12. 2008.

  2. Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. TESTING THE MANIFOLD HYPOTHESIS. JOURNAL OF THE AMERICAN MATHEMATICAL SOCIETY
    Volume 29, Number 4, October 2016, Pages 983–1049. 2016.

  3. Bastian Rieck, Markus Banagl, Filip Sadlo, Heike Leitte. Persistent Intersection Homology for the Analysis of Discrete Data. Arxiv. 2019.

  4. D. Chigirev and W. Bialek, Optimal manifold
    representation of data : an information theoretic approach, in Advances in Neural Information Processing Systems 16 161–168, MIT Press, Cambridge MA. 2004.

  5. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323–2326, 2000.

  6. Henry W. Lin, Max Tegmark, and David Rolnick. Why does deep and cheap learning work so well? Arxiv. 2017.

Best Answer

Q: Might there be a reason why stable physical processes would tend to have low-dimensional phase spaces.

Yes. One reason is physical processes have dissipation. E.g., turbulence is "known" to be chaotic dynamics on a low dimensional manifold (i.e., strange attractor) in the infinite dimensional phase space (of $L^2$ velocity fields). Even its dimension can be estimated. See for example:

Doering, Charles R., and John D. Gibbon. "Note on the Constantin-Foias-Temam attractor dimension estimate for two-dimensional turbulence." Physica D: Nonlinear Phenomena 48.2-3 (1991): 471-480.

.

Related Question