If $f\colon\Omega\to\mathbb{D}$ is biholomorphic and has an analytic continuation across $\partial\Omega$, it must map $\partial\Omega$ to $\partial\mathbb{D}$. Since $f'$ can have only isolated zeros on $\partial\Omega$, $f$ is locally biholomorphic at almost all points of $\partial\Omega$, and thus $\partial\Omega\cap V = f^{-1}(\partial\mathbb{D}\cap U)$ must be an analytic arc. So that $\partial\Omega$ consists of analytic arcs is essential for the existence of a continuation across the boundary at least if $f$ is injective and the image of $f$ has a boundary consisting of analytic arcs. Sometimes it is of course possible to continue $f$ across the boundary if $\partial\Omega$ is less regular, but a general theorem asserting the existence of a continuation across the boundary needs strong premises.
It seems to me that non-extendability happens at unbounded points
Not only there. If $(\alpha_n)$ is a sequence in $\mathbb{D}\setminus\{0\}$ with
$$\sum_{n=1}^\infty 1 - \lvert\alpha_n\rvert < \infty,$$
the Blaschke product
$$B_{k,\alpha}(z) = z^k\cdot\prod_{n=1}^\infty \frac{\lvert \alpha_n\rvert}{\alpha_n}\frac{z-\alpha_n}{1-\overline{\alpha_n}z}$$
is, for every $k\in\mathbb{N}$ a bounded holomorphic function on $\mathbb{D}$, and it cannot be analytically continued across every part of the boundary containing a limit point of $(\alpha_n)$. Choosing $(\alpha_n)$ so that all points on $\partial\mathbb{D}$ are limit points, you get a bounded holomorphic function whose natural boundary is $\partial\mathbb{D}$.
The definition of a modular form seems extremely unmotivated, and as @AndreaMori has pointed out, whilst the complex analytic approach gives us the quickest route to a definition, it also clouds some of what is really going on.
A good place to start is with the theory of elliptic curves, which have long been objects of geometric and arithmetic interest. One definition of an elliptic curve (over $\mathbb C$) is a quotient of $\mathbb C$ by a lattice $\Lambda = \mathbb Z\tau_1\oplus\mathbb Z\tau_2$, where $\tau_1,\tau_2\in\mathbb C$ are linearly independent over $\mathbb R$ ($\mathbb C$ and $\Lambda$ are viewed as additive groups): i.e.
$$E\cong \mathbb C/\Lambda.$$
In this viewpoint, one can study elliptic curves by studying lattices $\Lambda\subset\mathbb C$. Modular forms will correspond to certain functions of lattices, and by extension, to certain functions of elliptic curves.
Why the upper half plane?
For simplicity, since $\mathbb Z\tau_1 = \mathbb Z(-\tau_1)$, there's no harm in assuming that $\frac{\tau_1}{\tau_2}\in \mathbb H$.
What about $\mathrm{SL}_2(\mathbb Z)$?
When do $(\tau_1,\tau_2)$ and $(\tau_1',\tau_2')$ define the same lattice? Exactly when
$$(\tau_1',\tau_2')=(a\tau_1+b\tau_2,c\tau_1+d\tau_2)$$where $\begin{pmatrix}a&b\\c&d\end{pmatrix}\in\mathrm{SL}_2(\mathbb Z)$. Hence, if we want to consider functions on lattices, they had better be invariant under $\mathrm{SL}_2(\mathbb Z)$.
Functions on lattices:
Suppose we have a function $$F:\{\text{Lattices}\}\to\mathbb C.$$ First observe that multiplying a lattice by a non-zero scalar (i.e. $\lambda\Lambda$ for $\lambda\in\mathbb C^\times$) amounts to rotating and rescaling the lattice. So our function shouldn't do anything crazy to rescaled lattices.
In fact, since we really care about elliptic curves, and $\mathbb C/\Lambda\cong\mathbb C/\lambda\Lambda$ under the isomorphism $z\mapsto \lambda z$, $F$ should be completely invariant under such rescalings - i.e. we should insist that
$$F(\lambda \Lambda) = F(\Lambda).$$
However, if we define $F$ like this, we are forced to insist that $F$ has no poles. This is needlessly restrictive. So what we do instead is require that
$$F(\lambda\Lambda) = \lambda^{-k}F(\Lambda)$$
for some integer $k$; the quotient $F/G$ of two weight $k$ functions gives a fully invariant function, this time with poles allowed.
Where do modular forms come in?
If $\Lambda = \mathbb Z\tau\oplus\mathbb Z$ with $\tau\in\mathbb H$, define a function $f:\mathbb H\to\mathbb C$ by $f(\tau)=F(\Lambda)$. For a general lattice, we have
$$\begin{align}F(\mathbb Z\tau_1\oplus\mathbb Z\tau_2)&=F\left(\tau_2(\mathbb Z({\tau_1}/{\tau_2})\oplus\mathbb Z)\right)\\
&=\tau_2^{-k}f({\tau_1}/{\tau_2})
\end{align}$$
and in particular,
$$\begin{align}f(\tau) &= F(\mathbb Z\tau\oplus\mathbb Z) \\&=F(\mathbb Z(a\tau+b)\oplus\mathbb Z(c\tau+d)) &\text{by }\mathrm{SL}_2(\mathbb Z)\text{ invariance}\\&= (c\tau+d)^{-k} f\left(\frac{a\tau+b}{c\tau+d}\right).\end{align}$$
This answers your first two questions.
At this point, there's no reason to assume that condition (3) holds, and one can study such functions without assuming condition (3). However, imposing cusp conditions is a useful thing to do, as it ensures that the space of weight $k$ modular forms is finite dimensional.
To answer your fourth question, yes, and this is exactly the viewpoint taken in most research done on modular forms and their generalisations, where one considers automorphic representations.
Best Answer
this is clearly false; possibly you didn't quote the book accurately (for example maybe the word "rectangle" is being used in a not quite standard sense, or maybe there are more conditions on $f$...).
Let $R=(0,1)\times(0,1)$ and $S=(0,1)\times(0,2)$. The Riemann mapping theorem shows that there exists a holomorphic $f$ with $f(R)=S$, but clearly if $p$ is a (complex-)linear polynomial then $p(R)\ne S$.
Oh: To make things easier to state let's assume in addition that $f$ extends continuously to the closure $\overline R$. Unless there's something you're not telling us, the reflection argument simply doesn't work unless we know somehow that if $E$ is any edge of $R$ then $f(E)$ is contained in a single edge of $S$.