A nice property of metric spaces is the fact that we may characterize the notion of continuity of a mapping $f$ in four different, yet equivalent ways.
Theorem 1: Let $(X,d_X)$ and $(Y,d_Y)$ be metric spaces. Then, the following are equivalent statements:
$f$ is continuous at $a \in X$.
i) $\epsilon-\delta$ requirement.
ii) Suppose $U \subset Y$ is open in $Y$. Then, $f^{-1}(U)$ is open in $ X$.
iii) Suppose $E \subset Y$ is closed in $Y$. Then, $f^{-1}(E)$ is closed in $X$.
iv) If $\{x_n\}_{n \in \mathbb{N}} \to x$, then $\{f(x_n)\}_{n \in \mathbb{N}} \to f(x)$.
Continuity as a Motivation for Topological Spaces
Suppose we wish to move away from the notion of a metric space to a more general space, called a topological space. I want my new space to be one in which there is a well-defined notion of continuity. How then, should I define this new space? Well, why not make use of (3) in Theorem 1? This gives that the only notion required for continuity is that inverse images of open sets are open. Thus, it makes sense for our new space to have some sense of an open set. We then ask: How do open sets behave in $ \mathbb{R}$? Let $\mathscr{O}$ be the collection of all open subsets of $\mathbb{R}$. Then, the following can be easily checked:
$\emptyset$, $\mathbb{R} \in \mathscr{O}$.
For a finite index set $I$, and open sets $\{V_i\}_{i \in I}$, $$ \bigcap \limits_{i \in I} V_i \in \mathscr{O}.$$
For an arbitrary index set $J$, and open sets $\{U_j\}_{j \in J}$, $$\bigcup \limits_{j \in J} V_j \in \mathscr{O}$$
Ahh, you say. Why not then define a space that behaves in this manner? A collection of sets that we will call open sets that behave in the same way $\mathscr{O}$ does. Well, this is precisely the motivation behind the definition of a topology.
Let $X$ be a set. We define a topology on $X$ to be a collection of sets $\mathscr{T}$ whose members we will call open sets, satisfying:
$\emptyset$, $ X \in \mathscr{T}$.
For a finite index set $I$, and open sets $\{V_i\}_{i \in I}$, $$ \bigcap \limits_{i \in I} V_i \in \mathscr{T}.$$
For an arbitrary index set $J$, and open sets $ \{U_j\}_{j \in J}$, $$ \bigcup \limits_{j \in J} U_j \in \mathscr{T}$$
We call $$ (X,\mathscr{T}) $$ a topological space.
This gives a natural definition of continuity in these spaces:
Definition: Let $(X,\mathscr{T})$ and $(Y, \mathscr{V})$ be topological spaces. Let $ f: (X, \mathscr{T}) \to (Y, \mathscr{V})$. We say that $f$ is continuous at $ a \in X$ if, for every open neighborhood $ V \in \mathscr{V}$ of $ f(a)$, there exists an open neighborhood $ U \in \mathscr{T}$ of $a$ so that $ f(x) \in V$ whenever $ x \in U$. If $ f$ is continuous at every $ a \in X$, we say that $ f$ is continuous on $ X$.
I guess this exercise is the exercise 5 in section 3 in chapter 1 on p.30 in "Analysis on Manifolds" by James R. Munkres.
On p.26 in this book, there is the definition of "neighborhood":
If $U$ is any open set containing $x_0$, we commonly refer to $U$ simply as a neighborhood of $x_0$.
My solution is here:
Assume that $f$ is continuous.
For each $x\in X$, we choose $U:=X$.
Then, $U$ is a neighborhood of $x$ and $f=f|U$ is continuous.
Conversely, assume that for each $x\in X$, there is a neighborhood $U$ of $x$ such that $f|U$ is continuous.
Let $x$ be any point in $X$.
Let $V$ be any open set in $Y$ which contains $f(x)$.
By assumption, there is a neighborhood $W$ of $x$ such that $f|W$ is continuous.
Since $(f|W)^{-1}(V)$ is an open set in $W$, there is an open set $U$ in $X$ such that $(f|W)^{-1}(V)=W\cap U$.
Since both of $W$ and $U$ are open sets in $X$, their intersection $(f|W)^{-1}(V)$ is also an open set in $X$ and $(f|W)^{-1}(V)$ contains $x$.
Obviously, $f((f|W)^{-1}(V))\subset V$ holds.
Therefore, $f$ is continuous.
Best Answer
Open intervals $(a,b), a < b$ form a base for the topology of $\mathbb{R}$.
What is $f^{-1}[(a,b)] = \left\{x \in \mathbb{R}: x^2 \in (a,b) \right\}$?
If $b \le 0$, then no square is in $(a,b)$ so then $f^{-1}[(a,b)] = \emptyset$, which is open. So assume $b > 0$. Then $x^2 < b$ iff $x \in (-\sqrt{b},\sqrt{b})$. If $a < 0$, the $a$ does not impose an extra condition, as all $x^2 \ge 0 > a$ in that case, so
$f^{-1}[(a,b)] = (-\sqrt{b},\sqrt{b})$ if $b > 0, a < 0$, which is an open interval in $\mathbb{R}$ so open.
Otherwise we also need $x^2 > a \ge 0$, so $x < -\sqrt{a}$ or $x > \sqrt{a}$ respectively.
So then $f^{-1}[(a,b)] = (-\sqrt{b}, -\sqrt{a}) \cup (\sqrt{a},\sqrt{b})$ if $b > a \ge 0$, which is open as the union of two open intervals.
This covers all cases, so $f^{-1}[(a,b)]$ is open for all intervals.
Now if $O$ is open, we can write $O = \bigcup_{i \in I} (a_i,b_i)$, for some family of open intervals $(a_i,b_i), i \in I$, as the intervals form a base. But then
$$f^{-1}[O] = f^{-1}[\bigcup_{i \in I} (a_i,b_i)] = \bigcup_{i \in I} f^{-1}[(a_i,b_i)]$$
by standard properties of $f^{-1}$ and the last set is open as unions of open sets are open, and we have shown that the inverse images of the base sets are open.