Let $S^1$ be the unit circle and $f:S^1 \to S^1$ is a homeomorphism.
We say $f$ is a orientation preserving homeomorphism if any lifting of $f$ to the covering space $\mathbb{R}$ is
strictly increasing and It is called orientation reversing if any lifting of $f$ to the covering space $\mathbb{R}$ is strictly decreasing.
Now I have some question about them :
1- I want to prove that any homeomorphism $f:S^1 \to S^1$ is either orientation preserving or else orientation reversing.
2- The composition of an orientation preserving homeomorphism and an orientation reversing is orientation reversing.
3- If $a,b \in S^1$ then $f$ is orientation preserving if $f(a,b)=(f(a),f(b))$.
And also why the degree of a homeomorphism is either $1$ or $-1$ and does it related to the question 2?
because we know $deg(fog)=deg(f)deg(g)$
Best Answer
Let us first show
Proof. Let $H = \{(x,y) \in J \times J \mid x < y \}$. It is easy to verify that this is a convex subset of $\mathbb R^2$, thus it is path connected and a fortiori connected.
Let $A = \{(x,y) \in H \mid \phi(x) < \phi(y) \}$ and $B= \{(x,y) \in H \mid \phi(x) > \phi(y) \}$. Clearly $A \cap B = \emptyset$ and $A \cup B = H$ (note $\phi(x) = \phi(y)$ implies $x = y$ because $\phi$ is injective). Since $\phi$ is continuous, both $A$ and $B$ are open in $J \times J$, thus also open in $H$. Since $H$ is connected, one of $A$ or $B$ must be $= H$ and the other $= \emptyset$. This means that $\phi$ is either strictly increasing or strictly decreasing.
Next let us show
Proof. Let $p : \mathbb R \to S^1, p(t) = e^{2\pi it}$, be the standard covering map. As a lift of $f : S^1 \to S^1$ we denote any map $F : \mathbb R \to \mathbb R$ such that $p \circ F = f \circ p$.
You certainly know that if $F,F'$ are lifts of $f$, then $$(*) \phantom x F'(t) = F(t) + k \text{ for all } t \text{ with a } \textbf{fixed } k \in \mathbb Z. $$ In fact, we have $e^{2\pi i(F(t) -F'(t))} = e^{2\pi iF(t)}/e^{2\pi i F'(t)} = (p \circ F)(t)/ (p \circ F')(t) = (f \circ p)(t)/ (f \circ p)(t) = 1$, thus $(F - F')(t) = F(t) -F'(t) \in \mathbb Z$ and by continuity of $F - F'$ we see that $(F - F')(t) = k$ for some fixed $k \in \mathbb Z$. This means that $F' = \tau_k \circ F$ with the translation homeomorphism $\tau_k : \mathbb R \to \mathbb R, \tau_k(t) = t + k$.
If $F, G$ are lifts of $f, g$, then $p \circ G \circ F = g \circ p \circ F = g \circ f \circ p$, thus $G\circ F$ is a lift of $g \circ f$.
Let $h$ be a homeomorphism with inverse homeomorphism $h^{-1}$ and let $H, \bar H$ be lifts of $h, h^{-1}$. Then $\bar H \circ H$ is a lift of $h^{-1} \circ h = id$. Since also $id : \mathbb R \to \mathbb R$ is a lift of $id : S^1 \to S^1$, we get $(\tau_k \circ \bar H ) \circ H = \tau_k \circ (\bar H \circ H) = id$ for some $k \in \mathbb Z$. Similarly we get $H \circ \bar H = \tau_r \circ id = \tau_r$ for some $r \in \mathbb Z$. The latter implies $H \circ (\bar H \circ \tau_r^{-1}) = id$. Thus $H$ has a left inverse $H' = \tau_k \circ H$ and a right inverse $H'' = \bar H \circ \tau_r^{-1}$. But now $H'' = id \circ H'' = H' \circ H \circ H'' = H' \circ id = H'$, thus $H$ is a homeomorphism with inverse $H^{-1} = H' = H''$.
Your question 1 is answered by the above two theorems.
By the degree formula $\deg(f \circ g) = \deg(g)\deg(f)$ we see that any homeomorphism $h$ has degree $\pm 1$ (since $\deg(id) = 1)$). In fact, $\pm 1$ are the only elements of $\mathbb Z$ which have a multiplicative inverse. Let $H$ be a lift of $h$. It is a homeomorphism, thus $H$ is either strictly increasing or strictly decreasing. In the first case it must have positive degree, in the second case negative degree. Thus
Thus the degree formula also answers your question 2.
Concerning 3. : It is not really precise how you define "open intervals" $(a,b) \subset S^1$. It seems that if $a, b \in S^1$ are two distinct points, then you move counterclockwise from $a$ to $b$ and all points strictly between $a$ and $b$ constitute $(a,b)$. I think my answer to Open sets on the unit circle $S^1$ explains it more precisely. The open intervals $(a,b) \subset S^1$ are precisely the images $p((s,t))$ of open interval $(s,t) \subset \mathbb R$ such that $0 < t - s < 1$, where we have $a = p(s)$ and $b = p(t)$. Let us show that an orientation preserving homeomorphism $h$ maps $(a,b)$ onto $(h(a),h(b))$.
Clearly $h(a) \ne h(b)$. Let $H$ be a lift of $h$. Then $H(s) < H(t)$ and $H$ maps $(s,t)$ homeomorphically onto $(H(s),H(t))$. We have $p(H(s)) = h(a), p(H(t)) = h(b)$ and $0 < H(t) - H(s) < 1$. Concerning the last inequality: If $H(t) - H(s) = 1$, then $h(b) = h(a)$ which is impossible. If $H(t) - H(s) > 1$, then $p \mid_{(H(s),H(t))}$ is not injective, thus $p \circ H \mid_{(s,t)}$ is not injective which is a contradiction since $p \circ H \mid_{(s,t)} = h \circ p \mid_{(s,t)} = h \mid_{(a,b)}$.