I understand in your comment above to Jonas' answer that you would like these things to be broken down into simpler terms.
Think about limit points visually. Suppose you have a point $p$ that is a limit point of a set $E$. What does this mean? In plain terms (sans quantifiers) this means no matter what ball you draw about $p$, that ball will always contain a point of $E$ different from $p.$
For example, look at Jonas' first example above. What you should do wherever you are now is draw the number line, the point $0$, and then points of the set that Jonas described above. Namely draw $1, 1/2, 1/3,$ etc (of course it would not be possible to draw all of them!!).
Now an open ball in the metric space $\mathbb{R}$ with the usual Euclidean metric is just an open interval of the form $(-a,a)$ where $a\in \mathbb{R}$. Now we claim that $0$ is a limit point. How?
Given me an open interval about $0$. For now let it be $(-0.5343, 0.5343)$, a random interval I plucked out of the air. The question now is does this interval contain a point $p$ of the set $\{\frac{1}{n}\}_{n=1}^{\infty}$ different from $0$? Well sure, because by the archimedean property of the reals given any $\epsilon > 0$, we can find $n \in N$ such that
$$0 < \frac{1}{n} < \epsilon.$$
In fact you should be able to see from this immediately that whether or not I picked the open interval $(-0.5343,0.5343)$, $(-\sqrt{2},\sqrt{2})$ or any open interval.
Now let us look at the set $\mathbb{Z}$ as a subset of the reals. What you do now is get a paper, draw the number line and draw some dots on there to represent the integers. Can you see why you are able to draw a ball around an integer that does not contain any other integer?
Having understood this, looks at the following definition below:
$\textbf{Definition:}$ Let $E \subset X$ a metric space. We say that $p$ is a limit point of $E$ if for all $\epsilon > 0$, $B_{\epsilon} (p)$ contains a point of $E$ different from $p$.
$\textbf{The negation:} $ A point $p$ is not a limit point of $E$ if there exists some $\epsilon > 0$ such that $B_{\epsilon} (p)$ contains no point of $E$ different from $p$.
From the negation above, can you see now why every point of $\mathbb{Z}$ satisfies the negation? You already know that you are able to draw a ball around an integer that does not contain any other integer.
"Every finite set is closed" does not imply "every open neighbourhood is infinite".
Let $(X,d)$ be any metric space, and let $F \subseteq X$ be a non-empty finite subset; say $F = \{ x_1, \cdots, x_n \}$ for some $n \in \mathbb{N}$. To prove that $F$ is closed it suffices to prove that $X - F$ is open. So let $p \in X-F$. Then the set $\{ d(p,x_1), \cdots, d(p,x_n) \}$ has a minimum value, say $\delta$, and we know that $\delta > 0$ since $p \not \in F$, and so whenever $d(p,q) < \delta$ we have $q \in X-F$. But this tells you that the (open) ball of radius $\delta$ about $p$ lies in $X-F$, and hence $X-F$ is open.
In your examples, yes, the sets are open and finite, but they are also closed!
The key fact to take home is that sets are allowed to be both open and closed.
However, there is a partial converse: in a dense metric space, every open neighbourhood is infinite.
A metric space $(X,d)$ is dense if for any $x \in X$ and $r > 0$ there exists $y \in X$ with $x \ne y$ and $d(x,y) < r$ $-$ that is, given any point in the space, there are other points which lie arbitrarily close to that point. $\mathbb{R}$ is a dense space, so is $\mathbb{Q}$, so is $(0,1)$, and indeed so is any other non-empty open subset of $\mathbb{R}$.
The fact that open neighbourhoods in dense metric spaces are necessarily infinite is clear from the definition. [It's also clear that a metric space in which every open neighbourhood is infinite is dense, and so the two conditions are equivalent.]
Best Answer
Given Rudin's definition of "neighborhood," if $x\in N_r(p)$ but $x\not=p$, then $N_r(p)$ is not, itself, a neighborhood of $x$. Strictly speaking, a set that is a neighborhood of a point is a ball with that point as its center, so $N_r(p)$ can only be a neighborhood of $p$. What Rudin therefore needs to prove -- and where metric considerations enter in -- is that if $x\in N_r(p)$, then there is some ball centered at $x$, $N_{r'}(x)$, that is completely contained in $N_r(p)$.