Note: The following touches only a few aspects far from being representative for the wide connection of complex analysis with number theory.
General Note: When asking for connections of complex analysis with number theory you should delve into analytic number theory. This branch of number theory is roughly divided into additive number theory and multiplicative number theory.
From T. Apostols introductory section of his classic Modular Functions and Dirichlet Series in Number Theory:
Additive number theory is concerned with expressing an integer $n$ as sum of integers from some given set $S$. ...
Let $f(n)$ denote the number of ways $n$ can be written as a sum of elements of $S$. We ask for various properties of $f(n)$, such as its asymptotic behavior for large $n$. In a later chapter we will determine the asymptotic value of the partition function $p(n)$ which counts the number of ways $n$ can be written as a sum of positive integers $\leq n$.
The partition function $p(n)$ and other functions of additive number theory are intimitely related to a class of functions in complex analysis called elliptic modular functions. They play a role in additive number theory analogous to that played by Dirichlet series in multiplicative number theory. ...
We see from his intro that complex analysis plays a key role and also that the asymptotic study of function is essential for gaining insight.
In order to get information about the behavior of numerical sequences $a_n$ for large $n$, we study the corresponding generating functions $F(z)$
\begin{align*}
F(z)=\sum_{n=0}^{\infty}a_nz^n
\end{align*}
as function of a complex variable $z$. Crucial for the asymptotic behavior is the behavior of the function near its singularities.
Asymptotic Behavior: P. Flajolet and R. Sedgewick explain in Analytic Combinatorics this as follows
Comparatively little benefit results from assigning only real values to the variable $z$ that figures in a univariate generating function. In contrast, assigning complex values turns out to have serendipitous consequences. ...
When we do so, a generating function becomes a geometric transformation of the complex plane. This transformation is very regular near the origin—one says that it is analytic (or holomorphic). In other words, near $0$, it only effects a smooth distortion of the complex plane. Farther away from the origin, some cracks start appearing in the picture. These cracks—the dignified name is singularities—correspond to the disappearance of smoothness. It turns out that a function’s singularities provide a wealth of information regarding the function’s coefficients, and especially their asymptotic rate of growth. Adopting geometric point of view for generating functions has a large pay-off.
By focusing on singularities, analytic combinatorics treads in the steps of many respectable older areas of mathematics. For instance, Euler recognized that for the Riemann zeta function $\zeta(s)$ to become infinite (hence have a singularity) at $1$ implies
the existence of infinitely many prime numbers; Riemann, Hadamard, and de la Vall'ee-Poussin later uncovered deep connections between quantitative properties of prime numbers and singularities of $1/\zeta(s)$.
$$ $$
Application of complex analysis: Going back to Apostol's book we can find a famous application of the elliptic modular functions, namely Rademacher's series for the partition function:
From chapter 5:
... The unrestricted partition function $p(n)$ counts the number of ways a positive integer $n$ can be expressed as a sum of positive integers $\leq n$. The number of sums is unrestricted, repetition is allowed, and the order of the summands is not taken into account.
The partition founction is generated by Euler's infinite product
\begin{align*}
F(x)=\prod_{m=1}^{\infty}\frac{1}{1-x^m}=\sum_{n=0}^{\infty}p(n)x^n,
\end{align*}
where $p(0)=1$. Both the product and series converge absolutely and represent the analytic function $F$ in the unit disk $|x|<1$.
...
The partition function $p(n)$ satisfies the asymptotic relation
\begin{align*}
p(n)\sim\frac{e^{K\sqrt{n}}}{4n\sqrt{3}}\qquad \text{as } n\rightarrow \infty,
\end{align*}
where $K=\pi(2/3)^{1/2}$. This was first discovered by Hardy and Ramanujan in 1918 ...
By integrating along a path in the complex plane and considering thereby so-called Ford circles for Farey series (you may want to look at these interesting objects) Rademacher found the following celebrated representation of $p(n)$ for $n\geq 1$ as convergent series:
Apostol (Theorem 5.10): If $n\geq 1$ the partion function $p(n)$ is represented by the convergent series
\begin{align*}
p(n)=\frac{1}{\pi\sqrt{2}}\sum_{k=1}^{\infty}A_k(n)\sqrt{k}\frac{d}{dn}
\left(\frac{\sinh\left\{\frac{\pi}{k}\sqrt{\frac{2}{3}\left(n-\frac{1}{24}\right)}\right\}}{\sqrt{n-\frac{1}{24}}}\right)
\end{align*}
where
\begin{align*}
A_k(n)=\sum_{{0\leq h<k}\atop{(h,k)=1}}e^{\pi i s(h,k)-2\pi i nh/k}
\end{align*}
and with $s(h,k)$ being the Dedekind sum
\begin{align*}
s(h,k)=\sum_{r=1}^{k-1}\frac{r}{k}\left(\frac{hr}{k}-\left[\frac{hr}{k}\right]-\frac{1}{2}\right)
\end{align*}
Here is a last indication how important the study of the singularities in the complex plane of a function is.
Riemann's Zeta Function:
H.M.Edwards discusses in the first chapter of his classic Riemann's Zeta Function Riemann's epoch-making 8-page paper On the number of primes less than a given Magnitude.
From section 1.10 The Product representation of $\zeta(s)$:
A recurrent theme in Riemann's work is the global characterization of analytic function by their singularities. Since the function $\log\zeta(s)$ has logarithmic singularities at the roots $\rho$ of $\zeta(s)$ and no other singularities, it has the same singularities as the formal sum
\begin{align*}
\sum_{p}\log\left(1-\frac{s}{p}\right)\tag{1}.
\end{align*}
Thus if this sum converges and if the function it defines is in some sense as well behaved near $\infty$ as $\log \zeta(s)$ is, then it should follow that the sum (1) differs from $\log \zeta(s)$ by at most an additive constant; setting $s=0$ gives the value $\log \zeta(0)$ for this constant, and hence exponentiation gives
\begin{align*}
\zeta(s)=\zeta(0)\prod_s\left(1-\frac{s}{p}\right)\tag{2}
\end{align*}
as desired. This is essentially the proof of the product formula (2) which Riemann sketches.
Elementary number theory, i.e., modular congruence, linear Diophantine equations, quadratic residues and quadratic reciprocity can be easily studied without experience in higher algebra or calculus, and there a probably several books on the topic accessible to you (in almost every mother tongue). Really, it is a question of taste. One of my favorites is Elementary Number Theory, by Gareth and Josephine Jones. Also, it will cover basic and central concepts of number theory you'll need in any advanced study.
From there, number theory breaks in two major branches (with a great deal of overlap between them, and, of course, not comprehensive branches): analytic and algebraic number theory. For analytic, some calculus may come in hand. As for algebraic, basic higher algebra will be expected.
The standard reference of analytic number theory is Apostol's Introduction to Analytic Number Theory. How much calculus you'll need depends on how deep will you go (it can be single variable, multi variable, complex variable, you'll may also need some general topology, who knows?).
I've never read something specific about algebraic number theory, but the Internet seems to recommend Rosen's & Ireland's A Classical Introduction to Modern Number Theory. From the summary, it appears to cover the basics of algebraic number theory (also, with a lot of overlap with elementary stuff). I've had a good experience with Serre's A Course in Arithmetic, which covers both analytic and algebraic aspects, though it's a very hard book to digest (also, the french original is superior).
From there, you'll probably already have very specific interests from which you will be able to get better references. And for the calculus, algebra and topology, you can find introductory references here on the site. Expect to need at least some knowledge of groups, rings and fields, derivatives, infinite series, topology and complex variables to delve in the most advanced (yet central) areas.
Best Answer
If you have already learnt group theory, I may suggest you to go through the book 'theory of algebraic numbers' by Pollard & Diamond. It's a really good treatise to start off. You don't even need to know the definition of ring to read this book. Everything is given there in a very well setup. After having finished that book, you may pay a look at 'A Classical Introduction to Modern Number Theory' by Ireland & Rosen.