What is the order of growth of coefficients of half-integral weight modular forms on congruence subgroup with character ? There is the usual Hecke trick to compute bounds on the coefficients of integral weights on $SL_2(\mathbb{Z})$ but I don't know how to adapt the argument. I've looked online at many articles but didn't find anything. My guess is that it should be $O(n^{k/2})$ where $k$ is the half-integral weight.
Growth of coefficients of half-integral weight modular forms
complex-analysismodular-formsnumber theory
Related Solutions
The definition of a modular form seems extremely unmotivated, and as @AndreaMori has pointed out, whilst the complex analytic approach gives us the quickest route to a definition, it also clouds some of what is really going on.
A good place to start is with the theory of elliptic curves, which have long been objects of geometric and arithmetic interest. One definition of an elliptic curve (over $\mathbb C$) is a quotient of $\mathbb C$ by a lattice $\Lambda = \mathbb Z\tau_1\oplus\mathbb Z\tau_2$, where $\tau_1,\tau_2\in\mathbb C$ are linearly independent over $\mathbb R$ ($\mathbb C$ and $\Lambda$ are viewed as additive groups): i.e. $$E\cong \mathbb C/\Lambda.$$
In this viewpoint, one can study elliptic curves by studying lattices $\Lambda\subset\mathbb C$. Modular forms will correspond to certain functions of lattices, and by extension, to certain functions of elliptic curves.
Why the upper half plane?
For simplicity, since $\mathbb Z\tau_1 = \mathbb Z(-\tau_1)$, there's no harm in assuming that $\frac{\tau_1}{\tau_2}\in \mathbb H$.
What about $\mathrm{SL}_2(\mathbb Z)$?
When do $(\tau_1,\tau_2)$ and $(\tau_1',\tau_2')$ define the same lattice? Exactly when $$(\tau_1',\tau_2')=(a\tau_1+b\tau_2,c\tau_1+d\tau_2)$$where $\begin{pmatrix}a&b\\c&d\end{pmatrix}\in\mathrm{SL}_2(\mathbb Z)$. Hence, if we want to consider functions on lattices, they had better be invariant under $\mathrm{SL}_2(\mathbb Z)$.
Functions on lattices:
Suppose we have a function $$F:\{\text{Lattices}\}\to\mathbb C.$$ First observe that multiplying a lattice by a non-zero scalar (i.e. $\lambda\Lambda$ for $\lambda\in\mathbb C^\times$) amounts to rotating and rescaling the lattice. So our function shouldn't do anything crazy to rescaled lattices.
In fact, since we really care about elliptic curves, and $\mathbb C/\Lambda\cong\mathbb C/\lambda\Lambda$ under the isomorphism $z\mapsto \lambda z$, $F$ should be completely invariant under such rescalings - i.e. we should insist that
$$F(\lambda \Lambda) = F(\Lambda).$$ However, if we define $F$ like this, we are forced to insist that $F$ has no poles. This is needlessly restrictive. So what we do instead is require that $$F(\lambda\Lambda) = \lambda^{-k}F(\Lambda)$$ for some integer $k$; the quotient $F/G$ of two weight $k$ functions gives a fully invariant function, this time with poles allowed.
Where do modular forms come in?
If $\Lambda = \mathbb Z\tau\oplus\mathbb Z$ with $\tau\in\mathbb H$, define a function $f:\mathbb H\to\mathbb C$ by $f(\tau)=F(\Lambda)$. For a general lattice, we have
$$\begin{align}F(\mathbb Z\tau_1\oplus\mathbb Z\tau_2)&=F\left(\tau_2(\mathbb Z({\tau_1}/{\tau_2})\oplus\mathbb Z)\right)\\ &=\tau_2^{-k}f({\tau_1}/{\tau_2}) \end{align}$$ and in particular, $$\begin{align}f(\tau) &= F(\mathbb Z\tau\oplus\mathbb Z) \\&=F(\mathbb Z(a\tau+b)\oplus\mathbb Z(c\tau+d)) &\text{by }\mathrm{SL}_2(\mathbb Z)\text{ invariance}\\&= (c\tau+d)^{-k} f\left(\frac{a\tau+b}{c\tau+d}\right).\end{align}$$
This answers your first two questions.
At this point, there's no reason to assume that condition (3) holds, and one can study such functions without assuming condition (3). However, imposing cusp conditions is a useful thing to do, as it ensures that the space of weight $k$ modular forms is finite dimensional.
To answer your fourth question, yes, and this is exactly the viewpoint taken in most research done on modular forms and their generalisations, where one considers automorphic representations.
Best Answer
I assume you're referring to cuspforms. My answer assumes cuspforms throughout. The Hecke bound still applies. But one can say better.
It seems that we should have essentially the same bound as in the full integral weight case. That is, we should have that $$ a(n) \ll n^{\frac{k-1}{2} + \epsilon}. $$ In the full integral weight case, one can say a bit more about the $\epsilon$ factor in terms of the divisor function, but that might not be true for half-integral weight.
One reason to suspect that this is the truth is because it holds on average. The standard Rankin--Selberg technique of studying the Dirichlet series $$ D(s) = \sum_{n \geq 1} \frac{\lvert a(n) \rvert^2}{n^{s + k - 1}} $$ applies. One can show that $D(s)$ has meromorphic continuation to $\mathbb{C}$ with polynomial growth in vertical strips, and that the rightmost pole is $s = 1$. This implies that $$ \sum_{n \leq X} \lvert a(n)/n^{\frac{k-1}{2}} \rvert^2 = c X + o(X), $$ and thus that $\lvert a(n) \rvert \approx n^{\frac{k-1}{2}}$ on average.
But unlike the full integral weight case, the results of Deligne don't apply and we do not know how to prove this.