Whatever it is we define "$(a,b)$" to be as a set, what we really want is the following "defining property":
$(a,b) = (c,d)$ if and only if $a=c$ and $b=d$.
There are many ways to achieve this, but this is what we really want to achieve; once we achieve this via some definition, we want to avoid using the actual "guts" of the definition and stick exclusively to that defining property. (Similar to the points made in this answer and comment about how to represent the real numbers as sets).
One way to achieve this "defining property" is via the Kuratowski definition, by defining
$$(a,b) = \Bigl\{ \{a\},\{a,b\}\Bigr\}.$$
We can prove that this is a set, and that this set has the property we want.
There are other ways of achieving the same result; for example, Wiener proposed
$$(a,b) = \Bigl\{ \bigl\{ \{a\},\emptyset\bigr\}, \bigl\{\{b\}\bigr\}\Bigr\},$$
which also has the "defining property".
The problem with your proposal is that it does not have the defining property we want for ordered pairs: for example, $\emptyset\neq\{\emptyset\}$, so we want $(\emptyset,\{\emptyset\}) \neq (\{\emptyset\},\emptyset)$. But in your proposal, we have:
$$\begin{align*}
(\emptyset,\{\emptyset\}) &= \Bigl\{ \{\emptyset\}, \bigl\{\{\emptyset\}\bigr\}, \bigl\{ \emptyset, \{\emptyset\}\bigr\}\Bigr\},\\
(\{\emptyset\},\emptyset) &= \Bigl\{ \bigl\{ \{\emptyset\}\bigr\}, \{\emptyset\}, \bigl\{ \{\emptyset\},\emptyset\bigr\}\Bigr\};
\end{align*}$$
so that $(\emptyset,\{\emptyset\}) = (\{\emptyset\},\emptyset)$.
So the proposal, while a perfectly fine definition of a set, does not achieve the ultimate purpose of defining the ordered pair, and so it should not be the definition of "ordered pair".
Proof that the Kuratowski definition has the "defining property".
If $a=c$ and $b=d$, then
$$(a,b) = \Bigl\{ \{a\}, \{a,b\}\Bigr\} = \Bigl\{ \{c\},\{c,d\}\Bigr\} = (c,d).$$
Assume conversely that $(a,b)=(c,d)$. Then
$\bigcap(a,b) = \bigcap(c,d)$. Since
$$\bigcap(a,b) = \bigcap\Bigl\{ \{a\},\{a,b\}\Bigr\} = \{a\}\cap\{a,b\} = \{a\}$$
and
$$\bigcap(c,d) = \bigcap\Bigr\{ \{c\}, \{c,d\}\Bigr\} = \{c\}\cap\{c,d\} = \{c\},$$
we conclude that $a=c$.
If $b=a$, then $(a,b) = \{\{a\}\} = (c,d) = \{\{c\},\{c,d\}\}$. Therefore, $\{c,d\} \in\{\;\{a\}\;\}$, so $d\in\{a\}$, hence $d=a=b$ and we conclude $d=b$, as desired. Symmetrically, if $c=d$, then $\{a,b\}\in(a,b)=(c,d) = \{\;\{c\}\;\}$, so $\{a,b\}=\{c\}$, hence $b=c=d$ and we again conclude $d=b$ as desired.
If $b\neq a$ and $c\neq d$, then $\bigcup(a,b)-\bigcap(a,b) = \bigcup(c,d)-\bigcap(c,d)$. Since
$$\bigcup(a,b)-\bigcap(a,b) = \bigcup\Bigl\{\{a\},\{a,b\}\Bigr\} - \{a\} = \Bigl( \{a\}\cup \{a,b\}\Bigr)-\{a\} = \{a,b\}-\{a\} = \{b\}$$
and
$$\bigcup(c,d)-\bigcap(c,d) = \bigcup\Bigl\{\{c\},\{c,d\}\Bigr\} - \{c\} = \Bigl(\{c\}\cup \{c,d\}\Bigr)-\{c\} = \{c,d\}-\{c\} = \{d\}$$
(where we've used that $a\neq b$ to conclude that $\{a,b\}-\{a\}=\{b\}$ and we've used $c\neq d$ to conclude $\{c,d\}-\{c\}=\{d\}$), then we have $\{b\}=\{d\}$, hence $b=d$, again as desired.
Thus, if $(a,b)=(c,d)$, then $a=c$ and $b=d$.
Addressing the comments added to the question.
This definition is part of a way to try to define a lot of the things that we use in mathematics on the basis of an axiomatic theory; in this case, we start with Axiomatic Set Theory, where the only notions we have (if we are working in Zermelo-Fraenkel Set Theory) are "set" and "is and element of", together with the axioms that tells us properties of sets and things we can do with sets.
We want to have something that works like what we know as "the ordered pair"; but all we have to work with are sets. So we need to find a way of constructing a set that has the properties we want for the ordered pair.
For a metaphor: the ordered pair is like a car; we know how to drive. But in order to actually have a car, there needs to be an engine and gasoline, and the engine has to work. We are trying to construct that engine so that we can later drive it.
So this is not notation, this is a definition of what the ordered pair is in set theory. We are defining an object, which we call "$(a,b)$", to be the given set. It's not merely how we are writing the ordered pair, is what the ordered pair is if you are interested in actually seeing the engine of the car working. We know what we want "ordered pair" to behave like, but we have to actually construct an object that behaves that way. This is a way of defining an object that does behave that way.
There aren't "two notations" here. We define "the ordered pair with first component $a$ and second component $b$" to be the set
$$\bigl\{ \{a\}, \{a,b\}\bigr\},$$
(which one can prove is indeed a set using the Axioms of Set Theory, if $a$ and $b$ are already in the theory).
Then we prove that "the ordered pair with first component $a$ and second component $b$" is equal to "the ordered pair with first component $c$ and second component $d$" if and only if $a=c$ and $b=d$.
Then we abbreviate "the ordered pair with first component $a$ and second component $b$" by writing "$(a,b)$" (or sometimes "$\langle a,b\rangle$").
"$(a,b)"$ is notation. The other side is the definition of this set.
The definition is the way it is because it works; that's really all we care about. In fact, we forget about the definition pretty much as soon as we can, and simply use the $(a,b)$ and the "defining property." We can do that, because we know that "under the hood" there actually is an engine that does what we need it to do, even if we don't see it working while we are driving the car.
So, there is only one bit of notation, and it's "$(a,b)$". The other side is the definition of what that notation actually is.
The "set notation" doesn't "say" the ordered pair is what you think it is. What we are doing is defining what an ordered pair is, in a theory where the only thing we have are sets and the axioms of set theory. Because sets don't respect order, we cannot rely on simply how we write something; in order to be able to define an ordered pair we need to give a purely set-theoretic definition that actually achieves the purpose we want. Kuratowski's definition of an ordered pair $(a,b)$ to be the set given by $\bigl\{\{a\},\{a,b\}\bigr\}$ achieves this objective, in that the defined object has precisely the property we want an "ordered-pair-whatever-it-may-actually-be" to have. Since this set has that property, we define that set to be what the ordered pair "really is". But we don't actually care about what an ordered pair "really is", we just care about its desired "defining property".
In order for your car to work, there has to be an engine somewhere; but once there is an engine and your car works, you don't need to see the engine working in order to drive the car. The same with the ordered pair: for us to have an "ordered pair" in set theory, we need to be able to construct it somehow using sets. Once we have managed to do that, we don't need to see the actual set, we can just use the fact that there is a set that achieves our desired goal.
Yes, two sets are equal if and only if they have the same elements. So $\bigl\{\{a\},\{a,b\}\bigr\} = \bigl\{ \{a,b\}, \{a\}\bigr\}$. This does not matter. What matters is that $\bigl\{ \{a\}, \{a,b\}\bigr\} = \bigl\{ \{c\},\{c,d\}\bigr\}$ if and only if $a=c$ and $b=d$, because that's what we are going for. The definition of ordered pair by Kuratowski is specifically designed so that the end result "encodes" an order and distinguishes between the "first component" and the "second component" of $(a,b)$. The definition actually achieves this, as I proved above.
There is no problem with "duplicate elements". It's just that the set $\bigl\{\{a\},\{a\}\bigr\}$ is equal to the set $\bigl\{\{a\}\bigr\}$ by the Axiom of Extension, which says that two sets $A$ and $B$ are equal if and only if for every $x$, $x\in A\leftrightarrow x\in B$. The ordered pair $(a,a)$, as a set, is a set which can be written as
$$\bigl\{ \{a\},\{a,a\}\bigr\} \text{ or as }\bigl\{\{a\},\{a\}\bigr\}\text{ or as }\bigl\{ \{a\}\bigr\}.$$
There is no problem with this, because that set has the property that $(c,d)$ is equal to $\bigl\{ \{a\}\bigr\}$ if and only if $c=d=a$, which is exactly what we want.
Again: the whole point of this definition is only that it satisfies the property
$\bigl\{ \{a\},\{a,b\}\bigr\} = \bigl\{ \{c\},\{c,d\}\bigr\}$ if and only if $a=c$ and $b=d$.
Once we have this property, we abbreviate the set $\bigl\{\{a\},\{a,b\}\bigr\}$ as $(a,b)$, and simply use the property listed above.
As others have mentioned, it's typical to define
$$(a_1,\ldots, a_n)=((a_1,a_2,\ldots, a_{n-1}),a_n)$$
though I like to call them "ordered $n$-tuplets" to distinguish from the alternative definition I give below (whose naming convention is rather standard when used for infinite sets, in which the above definition cannot be extended).
I think it's worth noting that there are a lot of different ways to extend the notion of an ordered-pair. For example, once ordered-pairs and ordered-triplets have been defined, you can define $A\times B$ for two sets $A$ and $B$ and a function as an ordered-triple $(A,B,f)$, where $f\subset A\times B$ is well-defined and left-total. For there, you can then "redefine" $n$-tuples to be surjective maps $f: \{0,\ldots, n-1\}\rightarrow A$. (Strictly speaking, we could get away with only having ordered-pairs if we want to just define an $n$-tuple to be a subset of $\{0,\ldots, n-1\}\times A$ for some set $A$ that is well-defined and left-total. But defining functions as triples is useful because it allows one to talk about surjectivity.)
Another approach is defining $(a,b):=\{a,\{a,b\}\}$, though that this satisfies the property that $(a,b)=(c,d)$ implies $a=c,b=d$ requires the Axiom of Foundation.
For the sake of differentiating between the ordered $n$-tuplets as defined above and these $n$-tuplets, I'll denote an $n$-tuple as $\langle a_0,\ldots, a_{n-1}\rangle$ where $a_0,\ldots, a_{n-1}$ are the images of the function $f$ on $0,\ldots, n-1$, respectively. The nice property that $n$-tuples have that $n$-tuplets don't have is that given an $n$-tuple $\mathbf{a}=\langle a_0,\ldots, a_{n-1}\rangle$ and an $m$-tuple $\mathbf{b}=\langle b_0,\ldots, b_{m-1}\rangle$, we have $\mathbf{a}=\mathbf{b}$ if and only if $n=m$ and $a_i=b_i$ for each $i\in\{0,\ldots, n-1\}=\{0,\ldots, m-1\}$, the reason being that two functions being equal implies their domains are equal.
As mentioned above, when we want to extend the notion of an $n$-tuple(t) to deal with indexing things by arbitrary sets, we have to use the notion of an $n$-tuple, defining an $I$-tuple (or a family of sets indexed by $I$) to be a surjective map $f:I\rightarrow A$. If want to regard these $I$-tuples as simply being elements of some infinite Cartesian product $\prod_{i\in I}{A_i}$ (where $(A_i)_{i\in I}$ is a family of sets indexed by $I$), we could choose the elements to either be indexed families $f:I\rightarrow A$ where $A\subset \bigcup_{i\in I}{A_i}$ and $f(i)\in A_i$ for each $i\in I$ (so $f$ is necessarily surjective by definition), or we could choose to take the elements to be simply functions $f:I\rightarrow \bigcup_{i\in I}{A_i}$ where $f(i)\in A_i$ for each $i\in I$. Restricting to the finite case where $I=\{0,\ldots, n-1\}$, we then have a few different ways we could have defined, say, $A\times B$ (though we did use the Kuratowski definition to define a function in the first place, nothing stops us from then considering as "ordered pairs" $2$-tuples rather than $2$-tuplets from now on).
Given these alternatives exist for definitions of $n$-tuple(t)s or infinite cartesian products, a question might be which is better. Ultimately, it's kind of an arbitrary choice, in the sense that there is a natural way to switch between the two that "preserves" how the canonical projections act on the Cartesian products.
Best Answer
Two sets are equal if and only if they share the same elements. Thus there is no distinction between the sets $\{\{a\},\{b\}\}$ and $\{\{b\},\{a\}\}.$ That's why we need a different trick to create a mathematical object involving $a$ and $b$ in some particular order so that $(a,b)\neq(b,a)$ unless $a=b.$ The model $\{\{a\},\{a,b\}\}$ that you cite is the most common one but not the only possibility.
In fact the full requirement for our model is: $(a,b)=(c,d)$ if and only if $a=c$ AND $b=d.$ It is not difficult to verify that the model in your book satisfies that requirement, using the criterion that sets are equal if and only if they contain the same elements: let us show that equality of the ordered pairs implies equality of the corresponding elements.
If the sets $\{\{a\},\{a,b\}\}$ and $\{c\},\{c,d\}\}$ are identical then we must have
$$(\{a\}=\{c\}\hbox{ OR }\{a\}=\{c,d\})\hbox{ AND }(\{a,b\}=\{c\}\hbox{ OR }\{a,b\}=\{c,d\})$$
and
$$(\{c\}=\{a\}\hbox{ OR }\{c\}=\{a,b\})\hbox{ AND }(\{c,d\}=\{a\}\hbox{ OR }\{c,d\}=\{a,b\}).$$
The first two equalities imply that $c$ is an element of the singleton $\{a\},$ so their OR implies $a=c.$
The last two equalities on the first line imply that $b=c$ or $b=d.$
The last two equalities on the second line imply that $d=a$ or $b=d.$
So we either have $b=d$ or $d=a=c=b.$ In either case, $a=c$ and $b=d.$