Only statement (c) is true.
Hint: if $f$ is continuous and strictly increasing, it has a continuous (left-)inverse $f^{-1}$. Consider $f^{-1} \circ f \circ g$.
Counterexamples, in no particular order:
- $f(x) = x$, $g(x)$ is discontinuous
- $f$ is increasing but discontinuous, $g$ is a constant function
- $f$ is increasing but discontinuous, $g(x) = x$
An example of a strictly increasing discontinuous function:
$$
f(x) =
\begin{cases}
x/3 & x \in [0,1/2)\\
(x+1)/3 & x \in [1/2,1]
\end{cases}
$$
Step 1. Suppose $f$ is not continuous at some point $x \in [a,b]$. Then this means there is some $\epsilon > 0$ so that for all $\delta > 0$ we can find $y$ with $|x - y|<\delta$ and $|f(x) - f(y)| \geq \epsilon$.
Step 2. Ok, well, that means for $\delta = \frac{1}{n}$, we can find $y_{n}$ with $|x - y_{n}| < \frac{1}{n}$ and $|f(x) - f(y_{n})| \geq \epsilon$. In particular, we can assume for all $n$, $x < y_{n+1} < y_{n}$ by choosing $y_{n}$ from the interval $(x, \delta_{n}')$ in each step, where $\delta_{n}' = \min\{ y_{n -1}, \frac{1}{n}\}$.
Step 3. Ok, so we have $f(y_{n+1}) < f(y_{n})$ since $f$ is strictly increasing. But for all $n$, we have $|f(x) - f(y_{n})| \geq \epsilon$, i.e., $f(y_{n}) - f(x) \geq \epsilon$ (since $f$ is increasing and $x < y_{n}$). Thus, $f(y_{n}) \geq f(x) + \epsilon$ for all $n$, with $y_{n}$ decreasing down to $x$.
Step 4. Now, since $f$ must be onto, there must be some elements mapped into $(f(x), f(x) + \epsilon)$. But we are about to show that no elements can be mapped into this interval. Here is how:
Case 1: $y > x$. Since $y_{n} \to x$, that means if $y > x$, then $y > y_{n}$ for some $n$, so $y \not \in (f(x), f(x) + \epsilon)$.
Case 2: $y < x$. If $y < x$, then since $f$ is strictly increasing, $f(y) < f(x)$, so we can't have $f(y) \in (f(x), f(x) + \epsilon)$.
Thus, in every possible case, no elements are mapped into $(f(x), f(x) + \epsilon)$, which contradicts that $f$ is onto.
Best Answer
$(3)$:Since $f$ is increasing it is injective and hence has a left inverse .
Moreover $f$ is given to be continuous so the left inverse is also so and hence $f^{-1}\circ (f\circ g)=g$ is continuous which proves $3$.
$(1)$: $f(x)=$\begin{cases} x & x\in [0,\frac{1}{2}]\\\frac{x}{2}&x\in (\frac{1}{2},1]\end{cases}
and $g(x)=x$,then $f\circ g$ is discontinuous at $\frac{1}{2}$.
$(2)$: $f(x)=x^2$;
$g(x)=$\begin{cases} 1 & x\in \Bbb Q\cap[0,1]\\0 &x\in \Bbb Q^c\cap [0,1]\end{cases};then $f\circ g$ is discontinuous