Quadratic integer ring $\mathcal{O}$ is defined by
\begin{equation}
\mathcal{O}=\begin{cases}
\mathbb{Z}[\sqrt{D}] & \text{if}\ D\equiv2,3\ \pmod 4\\
\mathbb{Z}\left[\frac{1+\sqrt{D}}{2}\right]\ & \text{if}\ D\equiv1\pmod 4
\end{cases}
\end{equation}
where $D$ is square-free.
I understand $\mathbb{Z}\left[\frac{1+\sqrt{D}}{2}\right]$ is not closed under multiplication if $D\equiv2,3\pmod 4$. But still, isn't it more natural to define $\mathcal{O}=\mathbb{Z}[\sqrt{D}]$ for all square-free $D$? (In that case, it really seems like 'quadratic integer'.) I wonder what is the motivation of this definition.
Best Answer
Note: this answer has been edited to put the punchline first so that the rest is more motivated)
The Punchline I think the definition they gave you was a bad one, since usually one starts the study of integer rings (quadratic and otherwise) with conditions like "integrally closed Nötherian domain in which every prime ideal is maximal," or "every ideal factors uniquely into prime ideals," or something else more conceptual or based on properties. After that you usually do some (often-times very hard) work to determine the explicit description. In your case you are just given the explicit description (for the quadratic case) right off the bat, and the lack of symmetry--as you have noted--can be quite striking, until you later realize there's a good reason for it. In your case the problem is that those sets you suggest make more sense fail to satisfy that "integrally closed" part of the definition, eg $\Bbb Z[\sqrt{5}]$ is not integrally closed. The most likely reason is that you're in a basic class where the concepts involved in integer rings have not been taught yet, but since they are such a basic case, they're giving you some practice without the motivation (a confusing, but useful practice if their goal is to prep you for eventually studying the subject more thoroughly).
Since integral closure is about things in your field satisfying monic polynomials, you do some computations to get the description you were given as a definition.
The computations behind the definition you were given
We note that the minimal polynomial for $a+b\sqrt{D}$, assuming $b\ne 0$ is just
$$p_{\alpha}(x)=x^2-2ax+(a^2-Db^2)$$
because it is monic, and has $a+b\sqrt{D}$ as a root by inspection (the other root is $a-b\sqrt{D}$). In order for this to be integral, we need $2a\in\Bbb Z$ and $a^2-Db^2\in\Bbb Z$, and nothing more.
if $2a=2j+1$ is odd, then we compute
$$a^2-Db^2 = {4j^2+4j+1-4Db^2\over 4}$$
If $b\in\Bbb Z$ this is not an integer, since $4j^2+4j-4Db^2+1\equiv 1\mod 4$, so the numerator is not divisible by the denominator, so it must be that $2b=2k+1$, hence we have
$$a^2-Db^2={4(j^2+j-Dk^2-Dk)+1-D\over 4}$$
and for this to be an integer, it is necessary and sufficient that $D\equiv 1\mod 4$. We see then that $\mathcal{O}_K$ is the ring of integers exactly as in your definition, since in all other cases $a,b\in\Bbb Z$--remember, the only way $2a\in\Bbb Z$, but $a\not\in\Bbb Z$ was in the case where we concluded $2b\in\Bbb Z, b\not\in\Bbb Z$ and $D\equiv 1\mod 4$.
For the ambitious student
Why we care about integral closure
The most essential reason is because you get the (generalized) fundamental theorem of arithmetic. Without integral closure, you cannot guarantee the unique factorization because the prime ideals aren't always maximal--i.e. without integral closure your ring is not a Krull domain. We cannot take just any ring that looks natural, we have to do the one that the mathematics definitions lead us to. (Again, the general definitions, not the very specific one you got in this case.)