Open intervals $(a,b), a < b$ form a base for the topology of $\mathbb{R}$.
What is $f^{-1}[(a,b)] = \left\{x \in \mathbb{R}: x^2 \in (a,b) \right\}$?
If $b \le 0$, then no square is in $(a,b)$ so then $f^{-1}[(a,b)] = \emptyset$, which is open. So assume $b > 0$. Then $x^2 < b$ iff $x \in (-\sqrt{b},\sqrt{b})$.
If $a < 0$, the $a$ does not impose an extra condition, as all $x^2 \ge 0 > a$ in that case, so
$f^{-1}[(a,b)] = (-\sqrt{b},\sqrt{b})$ if $b > 0, a < 0$, which is an open interval in $\mathbb{R}$ so open.
Otherwise we also need $x^2 > a \ge 0$, so $x < -\sqrt{a}$ or $x > \sqrt{a}$ respectively.
So then $f^{-1}[(a,b)] = (-\sqrt{b}, -\sqrt{a}) \cup (\sqrt{a},\sqrt{b})$ if $b > a \ge 0$, which is open as the union of two open intervals.
This covers all cases, so $f^{-1}[(a,b)]$ is open for all intervals.
Now if $O$ is open, we can write $O = \bigcup_{i \in I} (a_i,b_i)$, for some family of open intervals $(a_i,b_i), i \in I$, as the intervals form a base. But then
$$f^{-1}[O] = f^{-1}[\bigcup_{i \in I} (a_i,b_i)] = \bigcup_{i \in I} f^{-1}[(a_i,b_i)]$$
by standard properties of $f^{-1}$ and the last set is open as unions of open sets are open, and we have shown that the inverse images of the base sets are open.
The set $\left(0, 1\right] = \left(0, \frac32\right) \cap \Big((0,1] \cup (2,3)\Big)$ is an open set in the codomain $(0,1] \cup (2,3)$. We have
$$f^{-1}\Big(\left(0, 1\right]\Big) = \left(0, 1\right]$$
which is not an open set in $(0,2)$. Therefore $f$ is not continuous.
Best Answer
A nice property of metric spaces is the fact that we may characterize the notion of continuity of a mapping $f$ in four different, yet equivalent ways.
Theorem 1: Let $(X,d_X)$ and $(Y,d_Y)$ be metric spaces. Then, the following are equivalent statements:
$f$ is continuous at $a \in X$.
i) $\epsilon-\delta$ requirement.
ii) Suppose $U \subset Y$ is open in $Y$. Then, $f^{-1}(U)$ is open in $ X$.
iii) Suppose $E \subset Y$ is closed in $Y$. Then, $f^{-1}(E)$ is closed in $X$.
iv) If $\{x_n\}_{n \in \mathbb{N}} \to x$, then $\{f(x_n)\}_{n \in \mathbb{N}} \to f(x)$.
Continuity as a Motivation for Topological Spaces
Suppose we wish to move away from the notion of a metric space to a more general space, called a topological space. I want my new space to be one in which there is a well-defined notion of continuity. How then, should I define this new space? Well, why not make use of (3) in Theorem 1? This gives that the only notion required for continuity is that inverse images of open sets are open. Thus, it makes sense for our new space to have some sense of an open set. We then ask: How do open sets behave in $ \mathbb{R}$? Let $\mathscr{O}$ be the collection of all open subsets of $\mathbb{R}$. Then, the following can be easily checked:
$\emptyset$, $\mathbb{R} \in \mathscr{O}$.
For a finite index set $I$, and open sets $\{V_i\}_{i \in I}$, $$ \bigcap \limits_{i \in I} V_i \in \mathscr{O}.$$
For an arbitrary index set $J$, and open sets $\{U_j\}_{j \in J}$, $$\bigcup \limits_{j \in J} V_j \in \mathscr{O}$$
Ahh, you say. Why not then define a space that behaves in this manner? A collection of sets that we will call open sets that behave in the same way $\mathscr{O}$ does. Well, this is precisely the motivation behind the definition of a topology.
Let $X$ be a set. We define a topology on $X$ to be a collection of sets $\mathscr{T}$ whose members we will call open sets, satisfying:
$\emptyset$, $ X \in \mathscr{T}$.
For a finite index set $I$, and open sets $\{V_i\}_{i \in I}$, $$ \bigcap \limits_{i \in I} V_i \in \mathscr{T}.$$
For an arbitrary index set $J$, and open sets $ \{U_j\}_{j \in J}$, $$ \bigcup \limits_{j \in J} U_j \in \mathscr{T}$$ We call $$ (X,\mathscr{T}) $$ a topological space.
This gives a natural definition of continuity in these spaces:
Definition: Let $(X,\mathscr{T})$ and $(Y, \mathscr{V})$ be topological spaces. Let $ f: (X, \mathscr{T}) \to (Y, \mathscr{V})$. We say that $f$ is continuous at $ a \in X$ if, for every open neighborhood $ V \in \mathscr{V}$ of $ f(a)$, there exists an open neighborhood $ U \in \mathscr{T}$ of $a$ so that $ f(x) \in V$ whenever $ x \in U$. If $ f$ is continuous at every $ a \in X$, we say that $ f$ is continuous on $ X$.