In Lee's Introduction to Smooth Manifold (2nd Ed.) the Proposition 2.5 gives two equivalent characterizations of smoothness of a map $F \colon M \to N$ between smooth manifolds $M$ and $N$, the second one is:
F is continuous and there exist smooth atlases $\{(U_\alpha, \varphi_\alpha)\}$ and $\{(V_\beta, \psi_\beta)\}$ for $M$ and $N$ respectively, such that for each $\alpha$ and $\beta$, $\psi_\beta \circ F \circ \varphi_\alpha^{-1}$ is a smooth map from $\varphi_\alpha(U_\alpha \cap F^{-1}(V_\beta))$ to $\psi_\beta(V_\beta)$.
I'm wondering whether there exist the notion of smooth atlas for a smooth manifold. Is it a subset of the smooth structure? I think so. Or is it a smooth atlas for the underlying topological variety? The definition of smooth atlas for a topological variety in the book is the following:
An atlas $\mathcal{A}$ is called a smooth atlas if any two charts in $\mathcal{A}$ are smoothly compatible with each other.
Consider the map $F \colon \mathbb{R} \to \mathbb{R}$ defined by $F(x)=x^2$, $F$ is smooth with respect to the standard smooth structure of $\mathbb{R}$. Now take the smooth atlases $\mathcal{A}_1 = \{(\mathbb{R}, x \mapsto x^3)\}$ and $\mathcal{A}_2 = \{(\mathbb{R},\textrm{id})\}$ the coordinate representation of $F$ is $x \mapsto x^\frac{2}{3}$ which is not smooth.
Best Answer
The book contains a definition of a smooth atlas on a topological manifold, but obviously Lee forgot to say what a smooth atlas on a smooth manifold is. Your guess is correct, it is any subatlas of the smooth structure.
Also your example in correct. Each homeomorphism $h : \mathbb R \to \mathbb R$ determines a smooth structure on the real line, and there are many such smooth structures. The smooth structure determined by $h$ agrees with the standard smooth structure if and only if $h$ is a diffeomorphism in the sense of multivariable calculus. Note, however, that all such smooth manifolds over the real line are diffeomorphic.