Prove or disprove that if $f$ is continuous in $R$ and it's inverse exists(bijectivity), then $f$ is decreasing or increasing.
Context:
I was solving the following question :
Find all continuous functions satisfying these two conditions. (The domain is $R$)
I. For all $x\in R$, $f(f(x))=x$
II. For all $x>0$, $\int_{-x}^{0} f(t)dt – \int_{0}^{x^2}f(t)dt=x^3$
The first condition implies that $f^{-1}(x)$ exists and it's equal with $f(x)$. But it doesn't imply the differentiability of $f$. But $f$ seems strictly decreasing in $R$.
Best Answer
Assume $f:\mathbb{R} \longrightarrow \mathbb{R}$ is continuous and not monotone.
Thus there exist $a,b,c$ with $a < b < c$ and
$((f(a) \leq f(b), f(c) \leq f(b)$ or $f(b) \leq f(a), f(b) \leq f(c))$.
WLOG, assume $f(a) \leq f(b), f(c) \leq f(b).$
Case $f(a) \leq f(c)$: By IVT, exists $x \in [a,b]$ with $f(x) = f(c)$.
Case $f(c) \leq f(b)$: Similar.
Thus $f$ is not injective.
Consequently, $f^{-1}$ is not bijective.