The moduli space of 1-complex dimensional tori (of area 1) is (geometrically) a non-compact orbifold. The non-compactness comes from the fact that on a torus, one may decrease the length of a closed curve down to zero. The universal cover of moduli space is the hyperbolic plane $\mathbb{H}^2$, and the (orbifold) fundamental group of moduli space is $\Gamma=PSL_2(\mathbb{Z})=Mod(T^2)$, which acts on $\partial\mathbb{H}^2=\mathbb{P}^1(\mathbb{R})$ in such a way that it preserves the cusp set $\mathbb{P}^1(\mathbb{Q})$, which parameterize the curves on a torus, corresponding to $\pm(p,q)\in \mathbb{Z}^2, \gcd(p,q)=1$, representing primitive elements of $\pi_1(T^2)=\pi_1(\mathbb{R}^2/\mathbb{Z}^2)=\mathbb{Z}^2$.
Congruence subgroups of $\Gamma$ generally correspond to subgroups which preserve a "level structure" on the torus $T^2$. That is, there is some cover $\tilde{T}^2\to T^2$, and one considers the subgroup of $Mod(T^2)$ which lifts to this cover. More generally, one can intersect subgroups of this sort to obtain congruence subgroups. Under the action of this subgroup of $Mod(T^2)$, the simple closed curves may split up into finitely many orbits, which will correspond to the cusps of the subgroup. For example, if one takes the cover corresponding to the subgroup $\Gamma_0(N)$, corresponding to the subgroup of $PSL_2(\mathbb{Z})$ which preserves the lattice $\mathbb{Z}+N\mathbb{Z} \leq \mathbb{Z}+\mathbb{Z}=\pi_1(T^2)$, then the curves corresponding to $(1,0)=\infty$ and $(0,1)=0\in \mathbb{P}^1(\mathbb{Q})$ will no longer be in the same orbit, so there are at least two cusps.
Now, by the uniformization theorem, for a finite-index subgroup $\tilde{\Gamma}\leq \Gamma$, there is a compact Riemann surface $\overline{X}$ compactifying $X=\mathbb{H}^2/\tilde{\Gamma}$. The holomorphic cusp forms (of even weights) may be interpreted as the forms which descend to the compactified surface $\overline{X}$ as holomorphic differential forms with coefficients in a line bundle. For example, weight 2 cusp forms will correspond to holomorphic 1-forms on $\overline{X}$. The space of these (by the Hodge theorem, or Riemann-Roch) is isomorphic to $H^1(\overline{X},\mathbb{C})\cong \mathbb{C}^g$, where $g$ is the genus of $\overline{X}$. If one did not impose the cusp condition, then $H^1(X)$ would have larger dimension, and there would be further 1-forms not vanishing at the cusps.
In the case of non-holomorphic modular functions, such as Maass wave forms, which correspond to eigenfunctions of the Laplacian on $X$, there are classes of eigenfunctions of the Laplacian (which are not in $L^2(X)$) which come from the cusps, and are represented by Eisenstein series. These forms have a continuous spectrum. Certain arithmetic information though tends to be represented by the cusp forms (which are in $L^2(X)$), which have a discrete spectrum. For example, Selberg's eigenvalue conjecture says that the cusp Maass forms always have eigenvalue $\geq \frac14$ for a congruence subgroup $\tilde{\Gamma}$; the corresponding fact is already known for the Eisenstein eigenfunctions.
first of all, you can try to see who is in the orbit of $\infty$.
If $z$ is in the orbit of $\infty$ then there exist a matrix on $\Gamma_0(p)$ say $\gamma=\left(\begin{matrix} a & b \\ pc & d\end{matrix}\right)$ such that $\gamma \infty =z$. This implies $\frac{a}{pc}=z$, then the numbers in the orbit of infinity are the rational number of the form $\frac{a}{pc}$ where $\gcd(a,pc)=1$ and $a\neq 0$ (because $ad-pbc=1$) .
On the other hand, it is easy to check that the other rational numbers are in the same orbit, and that $0$ is in that orbit.
Best Answer
$SL_2(\Bbb{Z}) i\infty = \Bbb{Q}\cup i\infty$
$SL_2(\Bbb{Z}) i\infty $ is only one point of the modular curve, the one that you need to add to get a compact Riemann surface. The chart is $q\to SL_2(\Bbb{Z}) \frac{\log(q)}{2i\pi}$ for $|q|<1/3$
It works the same way for $\Bbb{Q}\cup i\infty=\bigcup_{j=1}^J G \gamma_j i\infty$ a finite disjoint union, for some $\gamma_j \in SL_2(\Bbb{Z})$, and we need to add $J$ points to $G\backslash \Bbb{H}$ to get a compact Riemann surface. The charts are $q\to G \gamma_j k_j\frac{\log(q)}{2i\pi }$ for some $k_j$.
This is what we mean with the cusps of the modular curves.