The theory of modular forms arose out of the study of elliptic integrals (as did the theory of elliptic curves, and much of modern algebraic geometry, and indeed much of modern mathematics). People understood that (complete) elliptic integrals (which we would think of as the number obtained by integrating a de Rham cohomology class, e.g. the one associated to the holomorphic differential on an elliptic curve, over a homology class on the curve) depended on an invariant (what we would think of as the $j$-invariant of the elliptic curve, although historically people used other invariants, often depending on some auxiliary
level structure, such as $\lambda$, or $k$ (the square-root of $\lambda$)). This invariant was called the modulus (which is the origin of the adjective modular in this context).
People knew that if you replaced an elliptic curve by an $N$-isogenous one,
then the elliptic integral would be multiplied by $N$ (in terms of $\mathbb C/\Lambda$, the elliptic integral is just one of the basis elements for $\Lambda$,
and multiplying this by $N$, while keeping the other one fixed, gives a new elliptic curve related to the original one by an $N$-isogeny). They asked themselves how they could describe the modulus for this $N$-isogenous elliptic curve (or integral) in terms of the original one. This led them to find explicit equations for the modular curves $X_0(N)$ (for small values of $N$).
With these kinds of investigations (and remember, these were brilliant people --- Jacobi, Kronecker, Klein, just to mention some spanning a good part of the 19th century), it was natural that they were led to modular forms as well as modular functions (as one example, the Taylor coefficients of elliptic functions give modular forms; as another, the coordinates --- say with respect to Weierstrass elliptic functions --- of $N$-torsion points give level $N$ modular forms).
So all these investigations grew out of the study of elliptic integrals, but became intimately connected with the invention of algebraic topology, the development of complex analysis (by Riemann, and then Schwarz, and then the uniformization theorem), the development of hyperbolic geometry; basically all
the fundamental mathematics of the 19th century that then drove much of the developments of 20th century mathematics.
The connections with arithmetic were also observed early on. Jacobi already introduced theta series and saw the relationship with counting representations by quadratic forms (e.g. he proved that the number of ways of writing $n \geq 0$ as a sum of four squares is equal to $\sum_{d | n, 4 \not\mid d} d$, using weight $2$ modular forms on $\Gamma_0(4)$).
But Kronecker (and maybe Abel, Eisenstein and even Gauss before him) also knew that modular forms, when evaluated at CM elliptic curves (i.e. at quadratic imaginary values of $\tau$) gave algebraic number values in some contexts. Gauss was led to this by the analogy with cyclotomy: $N$-torsion on an elliptic curve was analogous to $N$th roots of $1$ on the unit circle, and the analogy is tighter when the elliptic curve has CM, because then the $N$-torsion points become a cyclic module over the ring of CMs, just as the $N$th roots of $1$ are a cyclic module over $\mathbb Z$ (i.e. a cyclic group).
Kronecker (and again, maybe people before him) realized that CM elliptic curves corresponded to lattices $\Lambda \subset \mathbb C$ that belong to ideal classes in quadratic imaginary fields, and so saw a relationship between CM elliptic curves and class field theory for quadratic imaginary fields (Kronecker's Jugendtraum). This also related to the previous work on evaluating modular forms at CM points.
All this is just to say that even in the 19th century the subject was very deep, and already very connected to number theory, as well as everything else.
Ramanujan knew the theory very well, and discovered new phenomena (e.g. his conjectures on the behavious of $\tau(n)$, defined by $\Delta = q\prod_{n=1}^{\infty} (1- q^n)^{24} = \sum_{n=1}^{\infty} \tau(n) q^n$). Mordell proved Ramanujan's conjecture on the multiplicative nature of $\tau$, and Hecke introduced his operators to systematize Mordell's method of proof.
At this point, the subject moved in a more representation-theoretic and analytic direction, with the generalization to automorphic forms. With the discovery in the 50s, 60s, and 70s of the modularity conjecure for elliptic curves over $\mathbb Q$, and related ideas, the arithmetic theory of modular forms became a central topic again. See this answer on MO for more on that.
Mazur's theorem on torsion points on elliptic curves over $\mathbb Q$ is one of the deepest results that comes from thinking of $X_0(N)$ and $X_1(N)$ directly in modular terms. But already the proofs are more automorphic in nature, and are focussed on the relationships between modular forms, particularly Hecke eigenforms, and Galois representations. That's where the modern focus primarily lies. You can see some of the other answers linked from my webpage (here) for more on that.
Let me close this long discussion by just saying that the passage to Galois representations as a focus is a natural development from Kronecker's Jugendtraum, but reflects a shifting of attention from abelian class field theory for quadratic imaginary fields to non-abelian (more precisely, $\mathrm{GL}_2$) class field theory for $\mathbb Q$. (Note that the former embeds in the latter, since the indcution of a Galois character of a quadratic extension gives a two-dimensional rep. of $G_{\mathbb Q}$.)
Finally, let me mention that the main theme of Mazur's article is congruences between cuspforms and Eisenstein series (this is what the Eisenstein ideal measures), and so it's hard to have one without the other. (In some sense, Eisenstein series are like the trivial Dirichlet character mod $N$, while cuspforms are like the non-trivial characters. Which is more important depends on what you are doing; in many problems you need to consider both.)
hunter's answer is a reasonable one, so I will just elaborate on some details. I'll name the books respectively Neukirch $1$, Neukirch $2$, Diamond Shurman, Silverman, and Advanced Silverman.
First, I am assuming you mean Neukirch's "Algebraic Number Theory"? I don't think he has written a book with the title "Algebraic Geometry".
Now, there are two clear sequences in that list of books, namely
$$\text{Neukirch 1} \longrightarrow \text{Neukirch 2}$$
$$\text{Silverman} \longrightarrow \text{Advanced Silverman}$$
By this I mean that the book on the right presumes knowledge from, and is generally more advanced than, the book on the left. This doesn't mean you have to finish the book on the left from cover to cover to begin the one on the right, but I would advise getting through some introductory chapters of the left before opening the one on the right.
Diamond Shurman is a great book, and gives an introduction to most of the advanced concepts they use. The story they tell is, in very coarse terms, how rational elliptic curves correspond to modular forms. One of the very important ingredients of this is $L$-functions.
Note that there are also intersections among the books. to mention a few; modular forms and functions enter both Silverman and Advanced Silverman, while having a major role in Diamond Shurman. Varieties, including elliptic curves, are also part of Diamond Shurman while being the central topic of both Silvermans. Neukirch 1 devotes much of the latter part on class field theory, which is involved in (at least) the chapter of complex multiplication of Advanced Silverman.
However, in my experience none of these intersections are too serious - whenever you will encounter a topic for the second time, it would be with a different angle, and you will be able to skip past parts you know.
Just reiterating what hunter said, in math it is often wise having an eye on where you are going and trying to figure out what you need, in contrast to just reading textbooks from cover to cover. Opening an advanced book also often gives perspective and direction when studying the more basic material. That being said, here is my reading guide:
- Start Neukirch 1.
- Start Diamond Shurman and Silverman (cpt. 3+4, skim back in 1+2 when needed).
- Finish Neukirch 1 cpt. 1+2, Diamond Shurman cpt. 1++, Silverman cpt 3+4.
- Read Silverman cpt. 1+2, tied together with Neukirch 1 cpt. 3.
- One does not have to see the proofs of class field theory, but it is important to know the results. Look at the relevant chapters of Neukirch. Also continue reading Silverman, the chapters you find relevant.
- Start advanced Silverman, and open Neukirch 2 (important: you should swap between the algebraic and arithmetic parts as needed, do not read it chronologically). Finish Diamond Shurman cpt. 1-5, and consider diving into the later chapters, in combination with the final Neukirch 1 chapter on zeta- and $L$-functions.
From there on you will have a solid grasp on the basic theory, with views into the advanced theory. You should by then also have looked at where you want to go, and be able to take informed decisions as to what to read next from then on.
Best Answer
If you are not familiar with complex analysis, then I recommend Stein and Shakarchi's Complex Analysis, since the last few chapters is essentially about modular forms, but only for the powers of theta functions, to answer the questions about sum of squares of integers. If I remember correctly, the book also contains a proof of the prime number theorem by means of using zeta functions, which could be another interesting part for you. After that, Zagier's Elliptic modular forms chapter in his book 123 of modular forms could also be a great reference for the amazing applications of modular forms.