The general idea is that, given a subgroup $\Gamma$ of finite index inside $SL_2(\mathbb Z)$ (though also we'd often require that this subgroup be definable by "congruence conditions" on the entries), the quotient $X=\Gamma\backslash{\mathfrak H}$ of the upper half-plane $\mathfrak H$ by $\Gamma$ needs finitely-many points added to it to "compactify" it. These are the "cusps". With fixed $\Gamma$ and fixed "weight" $k$, the cuspforms are the holomorphic weight-$k$ modular forms "vanishing at all cusps". (Since holomorphic modular forms are not actually invariant by $\Gamma$, this notion of vanishing includes some technicalities...) For even weight $2k>2$, the dimension of the space of weight-$2k$ holomorphic modular forms modulo cuspforms is equal to the number of cusps. (For odd weight $2k+1$, depending on $\Gamma$, some cusps can be "irregular", or some other modifier, in the sense of admitting no non-vanishing holomorphic modular form...) Thus, at least for even weight, relative to fixed $\Gamma$, there is an Eisenstein series attached to each cusp, which takes non-zero value there, and value $0$ at all other cusps.
How to exhibit/construct these? The action of $\Gamma$ extends to the compactification, and the isotropy (=stabilizer) subgroup $\Gamma_\sigma$ of a given cusp $\sigma$ makes sense. The corresponding Eisenstein series is a sum over $\Gamma_\sigma\backslash \Gamma$... For $\Gamma=SL_2(\mathbb Z)$ there is a single (equivalence class of) cusp, $i\infty$, and the expression written in the question is one formulaic version of the corresponding formation of Eisenstein series as sum over a coset space of this type.
The dimensions of spaces of holomorphic cuspforms are computable via Riemann-Roch, in effect. This is subtler than computing the dimensions of spaces of holomorphic Eisenstein series.
The moduli space of 1-complex dimensional tori (of area 1) is (geometrically) a non-compact orbifold. The non-compactness comes from the fact that on a torus, one may decrease the length of a closed curve down to zero. The universal cover of moduli space is the hyperbolic plane $\mathbb{H}^2$, and the (orbifold) fundamental group of moduli space is $\Gamma=PSL_2(\mathbb{Z})=Mod(T^2)$, which acts on $\partial\mathbb{H}^2=\mathbb{P}^1(\mathbb{R})$ in such a way that it preserves the cusp set $\mathbb{P}^1(\mathbb{Q})$, which parameterize the curves on a torus, corresponding to $\pm(p,q)\in \mathbb{Z}^2, \gcd(p,q)=1$, representing primitive elements of $\pi_1(T^2)=\pi_1(\mathbb{R}^2/\mathbb{Z}^2)=\mathbb{Z}^2$.
Congruence subgroups of $\Gamma$ generally correspond to subgroups which preserve a "level structure" on the torus $T^2$. That is, there is some cover $\tilde{T}^2\to T^2$, and one considers the subgroup of $Mod(T^2)$ which lifts to this cover. More generally, one can intersect subgroups of this sort to obtain congruence subgroups. Under the action of this subgroup of $Mod(T^2)$, the simple closed curves may split up into finitely many orbits, which will correspond to the cusps of the subgroup. For example, if one takes the cover corresponding to the subgroup $\Gamma_0(N)$, corresponding to the subgroup of $PSL_2(\mathbb{Z})$ which preserves the lattice $\mathbb{Z}+N\mathbb{Z} \leq \mathbb{Z}+\mathbb{Z}=\pi_1(T^2)$, then the curves corresponding to $(1,0)=\infty$ and $(0,1)=0\in \mathbb{P}^1(\mathbb{Q})$ will no longer be in the same orbit, so there are at least two cusps.
Now, by the uniformization theorem, for a finite-index subgroup $\tilde{\Gamma}\leq \Gamma$, there is a compact Riemann surface $\overline{X}$ compactifying $X=\mathbb{H}^2/\tilde{\Gamma}$. The holomorphic cusp forms (of even weights) may be interpreted as the forms which descend to the compactified surface $\overline{X}$ as holomorphic differential forms with coefficients in a line bundle. For example, weight 2 cusp forms will correspond to holomorphic 1-forms on $\overline{X}$. The space of these (by the Hodge theorem, or Riemann-Roch) is isomorphic to $H^1(\overline{X},\mathbb{C})\cong \mathbb{C}^g$, where $g$ is the genus of $\overline{X}$. If one did not impose the cusp condition, then $H^1(X)$ would have larger dimension, and there would be further 1-forms not vanishing at the cusps.
In the case of non-holomorphic modular functions, such as Maass wave forms, which correspond to eigenfunctions of the Laplacian on $X$, there are classes of eigenfunctions of the Laplacian (which are not in $L^2(X)$) which come from the cusps, and are represented by Eisenstein series. These forms have a continuous spectrum. Certain arithmetic information though tends to be represented by the cusp forms (which are in $L^2(X)$), which have a discrete spectrum. For example, Selberg's eigenvalue conjecture says that the cusp Maass forms always have eigenvalue $\geq \frac14$ for a congruence subgroup $\tilde{\Gamma}$; the corresponding fact is already known for the Eisenstein eigenfunctions.
Best Answer
Let $\gamma = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$. Using the transformation law and expanding $f$ as a Fourier series, we have $$\frac{f\left(\frac{a\tau + b}{c \tau + d}\right)}{(c\tau + d)^k} = f(\tau) = \sum_{n=0}^{\infty}a_ne^{2\pi i\tau n}.$$ Then $$\lim_{\textrm{Im}(\tau) \to \infty}\frac{f\left(\frac{a\tau + b}{c \tau + d}\right)}{(c\tau + d)^k} = a_0 \neq 0.$$ On the other hand, since $\frac{a\tau + b}{c\tau + d} \to s$ as $\textrm{Im}(\tau) \to \infty$, if $\lim_{\tau \to s}f(\tau) = 0$ then the above limit would be $0$.