Let $(\Omega,\Sigma)$ be a measurable space. An atom of $\Sigma$ is a set $B\in\Sigma$ such that for all $A\subseteq B$ either $A=\emptyset$ or $B=A$. A measurable space is atomic if every element lies in some atom. The $\sigma$-algebra $\Sigma$ is countably generated if there is a countable family of measurable sets such that $\Sigma$ is the smallest $\sigma$-algebra containing all of them. For example $(\mathbb{R},\mathcal{B})$ is countably generated since $\mathcal{B}$ is generated by the open intervals with rational endpoints. The atoms of $\mathcal{B}$ are the singletons.
Proposition: If $\Sigma$ is countably generated, then $(X,\Sigma)$ is atomic.
Proof: If there is a countable family generating $\Sigma$, there is also a countable family closed under complementation that generates $\Sigma$. If $\mathcal{C}$ is such a family, we get all atoms of $\Sigma$ as the intersection of all elements of $\mathcal{C}$ that contain a given point.
Now if $(\Omega,\Sigma,\mu)$ is a probability space, we call $B\in\Sigma$ a $\mu$-atom if $\mu(B)>0$ and for all $A\in\Sigma$ such that $A\subseteq B$, either $\mu(A)=0$ or $\mu(A)=\mu(B)$. The probability space is atomless if it contains no $\mu$-atom.
Lemma: If $(\Omega,\Sigma,\mu)$ is a probability space such that $\Sigma$ is countably generated and $\mu$ takes on only the values $0$ and $1$, then there exists an atom $A\in\Sigma$ such that $\mu(A)=1$.
Proof: Let $\mathcal{C}$ be a countable family closed under complementation that generates $\Sigma$. For each element of $\mathcal{C}$, either itself or its complement has probability one $1$. The intersection of all elements in $\Sigma$ with probability $1$ is an atom with probability $1$.
Proposition: If $(\Omega,\Sigma,\mu)$ is a probability space with $\Sigma$ countably generated, then it is atomless if and only if every atom in $\Sigma$ has probability $0$.
Proof: Clearly, in an atomless probability space, every atom must have probability $0$. Supppose now that $A$ is a $\mu$-atom. Let $A\cap\Sigma=\{A\cap S:S\in\Sigma\}$ be the trace $\sigma$-algebra. It is countably generated too. Then $(A,A\cap\Sigma,1/\mu(A)\cdot\mu)$ is a probability space such that the probability takes on only the values $0$ and $1$. So by the lemma, there is an atom $B$ such that $1/\mu(A)⋅\mu(B)=1$. But $B$ is also an atom of $\Sigma$ and $\mu(B)>0$.
So it follows that a probability measure on $(\mathbb{R},\mathcal{B})$ is atomless if and only if it puts probability $0$ on all singletons, which justifies the definition in the book of Kai Lai Chung.
Finally, an example of a probability space in which each atom has probability $0$ but such that the space is not atomless. Let $\Omega$ be any uncountable set, let $\Sigma$ consists of those subsets of $\Omega$ that are either countable or have an uncountable complement. Let $\mu(A)=0$ if $A$ is countable and $\mu(A)=1$ if its complement is countable. Every set with countable complement is an $\mu$-atom, but the atoms of $\Sigma$ are the singletons which all have probability $0$. Note that $\Sigma$ is not countably generated.
Suppose $A$ is an atom of some Borel measure $\mu$. For simplicity, let us assume $A\subseteq[0,1)$ and $\mu(A)=1$ (it is easy to generalize the argument). For each integer $n$ and each integer $k$ such that $0\leq k<2^n$, let $I_{n,k}=[k/2^n,(k+1)/2^n)$. Since $A$ is an atom, $\mu(A\cap I_{n,k})$ must be either $0$ or $1$. Since these sets (for fixed $n$) partition $A$, we conclude that exactly one of them has measure $1$; that is, there is a unique $k_n$ such that $\mu(A\cap I_{n,k_n})=1$. It is now easy to see that $I_{n,k_n}\subset I_{m,k_m}$ for $n>m$. It follows that $\bigcap_n I_{n,k_n}$ consists of a single point $x$ and that $\mu(A\cap\{x\})=\inf_n \mu(A\cap I_{n,k_n})=1$. That is, $x\in A$, $\{x\}$ is an atom, and $A$ differs from $\{x\}$ by a set of measure zero.
Best Answer
The example you give actually is a $\sigma$-finite Borel measure. Equip $[0,1]$ with the cofinite topology (in which a set is open iff it is either empty or its complement is finite). Then your $\Sigma$ is the Borel $\sigma$-algebra of the cofinite topology (it is a nice exercise to verify this).
However, there is the following result:
Proposition. Let $(X,d)$ be a separable metric space, $\Sigma$ its Borel $\sigma$-algebra, and $\mu$ a $\sigma$-finite measure on $\Sigma$. Then each atom of $\mu$ is the union of a point mass and a null set.
Proof. Let $C$ be a countable dense subset of $X$. For each integer $k\in\mathbb{N}$, we have $\bigcup_{x \in C} B(x, 1/k) = X$. Thus $\bigcup_{x \in C} (A \cap B(x,1/k)) = A$. So by countable additivity, there exists $x_k \in C$ such that $\mu(A \cap B(x_k, 1/k)) > 0$. Since $A$ is an atom, $\mu(A \setminus B(x_k, 1/k)) = 0$. Let $S = \bigcap_k B(x_k, 1/k)$.
Since for each $k$, $S$ is contained in a ball of radius $1/k$, $S$ contains at most one point.
On the other hand, by De Morgan's law and countable additivity, $$\mu(A \setminus S) = \mu\left(\bigcup_k A \setminus B(x_k, 1/k)\right) = 0.$$ Since $\mu(A\cap S)=\mu(A) > 0$, $A\cap S$ is not empty, so $A\cap S$ is a singleton.
Hence $A\cap S$ is a point mass and $A \setminus S$ a null set. $\Box$
So in this case, effectively the only atoms are point masses.
Note that we did not need to assume $X$ was complete.
For non-separable metric spaces, things are harder. For uncountable discrete spaces (which are certainly metric), the question of whether there can be nontrivial atoms is related to whether the cardinality of $X$ is a measurable cardinal, and such questions tend to be independent of the axioms of ZFC. I asked a new question about it: Consistency strength of 0-1 valued Borel measures.