Under a modular transformation the Dedekind $\eta$ function transforms as $$\eta(-1/\tau) = \sqrt{-i \tau}\eta(\tau) \, .\tag*{$(*)$}$$Siegel gives a proof in this paper here that uses complex analytic techniques. However, I am wondering how we can see it in the following fashion. The Dedekind $\eta$ function can be represented in the form$$\eta(\tau) = q^{1/24} \prod_{n = 1}^\infty (1 – q^n) = \sum_{n = -\infty}^\infty (-1)^n q^{{3\over2}(n – 1/6)^2}.$$How can we use the Poisson resummation formula and this representation of the $\eta$ function to verify the modular transformation $(*)$?
[Math] Modular transformations of $\eta(\tau)$
analytic-number-theorycomplex-analysisdedekind-eta-functionmodular-formsnumber theory
Related Solutions
Let $a,b\in\Bbb N$ such that $a+b=24$.
Theorem: The function $$f(\tau)=\eta(a\tau)\eta(b\tau)$$ is a cusp form of weight $1$ for the congruence subgroup $$\Gamma_1(ab)=\left\{\gamma\in\mathrm{SL}_2(\Bbb Z):\gamma\equiv\begin{pmatrix}1 & *\\ 0 & 1\end{pmatrix}\pmod{ab}\right\}.$$
Proof: Since $a+b=24$, we can write $$f(\tau)=q(q^a;q^a)_\infty(q^b;q^b)_\infty,$$ which is invariant under the map $\tau\mapsto \tau+1$, so $$f(\tau+1)=f(\tau).$$ Then, take any $$\gamma=\begin{pmatrix}p_1 & p_2\\ p_3 & p_4\end{pmatrix}\in\Gamma_1(ab),$$ and define $$\gamma (z):= \frac{p_1z+p_2}{p_3z+p_4},$$ as well as $$\gamma_M (z):=\frac{p_1z+Mp_2}{\frac{p_3}{M}z+p_4}$$ for any nonzero integer $M$. We have $$\begin{align} f(\gamma (z))=f\left(\frac{p_1z+p_2}{p_3z+p_4}\right)&=\eta\left(a\frac{p_1z+p_2}{p_3z+p_4}\right)\eta\left(b\frac{p_1z+p_2}{p_3z+p_4}\right)\\ &=\eta\left(\frac{ap_1z+ap_2}{p_3z+p_4}\right)\eta\left(\frac{bp_1z+bp_2}{p_3z+p_4}\right)\\ &=\eta\left(\frac{p_1(az)+ap_2}{\frac{p_3}{a}(az)+p_4}\right)\eta\left(\frac{p_1(bz)+bp_2}{\frac{p_3}{b}(bz)+p_4}\right)\\ &=\eta\left(\gamma_a(az)\right)\eta\left(\gamma_b(bz)\right). \end{align}$$ Then $$\det\gamma_M=\det\begin{pmatrix}p_1 & Mp_2\\ \tfrac{p_3}{M} & p_4\end{pmatrix}=p_1p_4-\left(Mp_2\right)\left(\tfrac{p_3}{M}\right)=p_1p_4-p_2p_3=\det\gamma=1,$$ and since $a$ and $b$ both divide $ab$ and by definition $p_3\equiv 0\pmod{ab}$, we have that $\gamma_a,\gamma_b\in\mathrm{SL}_2(\Bbb Z)$, and thus $$\eta(\gamma_M(Mz))=\varepsilon(\gamma_M)(\tfrac{p_3}{M}Mz+p_4)^{1/2}\eta(Mz)=(p_3z+p_4)^{1/2}\varepsilon_M\eta(Mz),$$ for $M=a,b$. Thus $$f\left(\frac{p_1z+p_2}{p_3z+p_4}\right)=\varepsilon_a\varepsilon_b (p_3z+p_4)f(z).\tag1$$ Here, $$\varepsilon\left[\begin{pmatrix}A & B\\ C & D\end{pmatrix}\right]=\begin{cases} e^{i\pi B/12} & C=0,\, D=1\\ \exp\left[i\pi\left(\tfrac{A+D}{12C}-\tfrac14-s(D,C)\right)\right] & C>0, \end{cases}$$ with $$s(D,C)=\sum_{n=1}^{C-1}\frac{n}{C}\left(\frac{Dn}{C}-\left\lfloor\frac{Dn}{C}\right\rfloor-\frac12\right),$$ and of course $\varepsilon_M=\varepsilon(\gamma_M)$.
From $(1)$ we then know that $f$ is a modular form of weight $1$ for $\Gamma_1(ab)$. As @reuns said, $f$ is a cusp form, because $f^{24}(\tau)=\Delta^{24}(a\tau)\Delta^{24}(b\tau)$, where $\Delta$ is the Modular Discriminant, which is a cusp form.
QED
If we had $\phi \in M_k(\Gamma, \chi)$ for some half-integer $k \in \frac{1}{2} \Bbb Z$ and congruence subgroup $\Gamma$ and a character $\chi$ of finite order on it, then $q(z)^{-1 / 24}$ would be modular as well (in $M_{k - 1/2}(\Gamma, \chi / c)$), which cannot happen. Indeed, since $\chi / c$ has finite order ($\chi$ has order 24), then by raising to a sufficiently large power (some multiple of 24), we get that $$f : z \mapsto e^{2 \pi i d z} \in M_w(\Gamma)$$ for some integers $w, d \in \Bbb Z$.
Take some $N > 1$ so that $\gamma := \begin{pmatrix} 0 & -1 \\ N & 0 \end{pmatrix} \in \Gamma$. Then $$f(\gamma z) = f\left( \frac{-1}{N z} \right) = z^w f(z) \iff \exp(-2 \pi i d / (N z)) = z^w \exp( 2 \pi i d z ) $$ for all $z \in \Bbb H$. But then raising to the power $z$ would give that $z \mapsto \exp( z w \log(z) + 2 \pi i d z)$ is constant, which it isn't.
Best Answer
Let us start from one side of what we want to prove,$$\eta(-1/\tau) = r^{1/24} \prod_{n = 1}^\infty (1 - r^n), \quad r = \exp\left(-{{2\pi i}\over\tau}\right).$$We were given Euler's (quite remarkable) pentagonal number theorem, so we might as well start by using it to transform the product into a sum$$\eta(-1/\tau) = r^{1/24} \sum_{n \in \mathbb{Z}} (-1)^n r^{(3n^2 - n)/2}, \quad r = \exp\left(-{{2\pi i}\over\tau}\right).$$The other hammer we were given is the Poisson summation, and this expression is beginning to look like the right nail: we just need to a find a function $f$ such that each term in the sum above is $f(n)$, and then Fourier transform it. Rewriting the above expression slightly as$$f(n) = \exp\left[\pi i\left(-{1\over{12\tau}} + n - {{(3n^2 - n)}\over\tau}\right)\right],$$it becomes straightforward to extrapolate it to the reals by simply replacing $n$ with a real number $x$. We can now find the Fourier transform (note that $f$ does fall off "sufficiently fast" because $\text{Im}\,\tau > 0$),\begin{align*} \tilde{f}(k) & = \int_{-\infty}^\infty dx\,\exp\left[\pi i\left(-{1\over{12\tau}} + x - 2kx - {{(3x^2 - x)}\over\tau}\right)\right] \\ & = \sqrt{\tau\over{3i}} \exp\left[\pi i\left({{\tau(-2k + 1 + {1\over\tau})^2}\over{12}} - {1\over{12\tau}}\right)\right] \\ & = \sqrt{{-i\tau}\over3}\exp\left[\pi i\left({{\tau(2k - 1)^2}\over{12}} - {{(2k - 1)}\over6}\right)\right],\end{align*}where in the first step we used the Gaussian integral$$\int_{-\infty}^\infty dx\,\exp(-ax^2 + bx + c) = \sqrt{\pi\over a} \exp\left({{b^2}\over{4a}} + c\right),$$valid for $\text{Re}\,a > 0$, which in our case translates to$$\text{Re}\left({{3\pi i}\over\tau}\right) = \text{Im}\left(-{{3\pi}\over\tau}\right) > 0.$$We can finally use the Poisson summation formula to obtain$$\eta(-1/\tau) = \sqrt{{-i\tau}\over3} \sum_{k \in \mathbb{Z}} \exp\left[\pi i\left({{\tau(2k - 1)^2}\over{12}} - {{(2k - 1)}\over6}\right)\right].\tag*{$(*)$}$$How close is this to the desired result? Let us use the pentagonal number theorem on $\eta(\tau)$ as well to transform it into a sum$$\eta(\tau) = q^{1/24} \sum_{n \in \mathbb{Z}} (-1)^n q^{(3n^2 - n)/2} = \sum_{n \in \mathbb{Z}} \exp\left[\pi i\left({{\tau(6n - 1)^2}\over{12}} + n\right)\right].$$This is close! However, there is a factor of $\sqrt{3}$, and the term proportional to $\tau$ in the exponent has $6n - 1$ instead of $2k - 1$. To solve these assorted factors of $3$, we can split the sum in $(*)$ into three sums, with $k = 3l$, $3l + 1$, $3l + 2$. Then$$\eta(-1/\tau) = \sqrt{{{-i\tau}\over3}}\sum_{l \in \mathbb{Z}} \left[ \exp\left[\pi i\left({{\tau(6l - 1)^2}\over{12}} - {{(6l - 1)}\over6}\right)\right] + \exp\left[\pi i\left({{\tau(6l + 1)^2}\over{12}} - {{(6l + 1)}\over6}\right)\right] + \exp\left[\pi i\left({{\tau(6l + 3)^2}\over{12}} - {{(6l + 3)}\over6}\right)\right]\right].$$The first term looks like the desired expression times $e^{\pi i/6}$. In the second term, we can take $l \to -l$ to transform it into the desired expression times $e^{-\pi i/6}$. Since$$e^{\pi i/6} + e^{-\pi i/6} = 2\cos \pi/6 = \sqrt{3},$$we have found the full answer and the last term must vanish!
We can rewrite it as a sum over odd integers,$$\sum_{m \in 2\mathbb{Z} + 1} \exp\left[\pi i\left({{3\tau m^2}\over4} - {m\over2}\right)\right],$$and now separate the positive and negative $m$ parts to find$$\sum_{m \in 2\mathbb{Z} + 1,\,m > 0} \exp\left[\pi i{{3\tau m^2}\over4}\right]\left(e^{\pi im/2} + e^{-\pi im/2}\right),$$which vanishes since $e^{\pi im} = -1$ for $m$ an odd integer.
Therefore, we finally have$$\eta(-1/\tau) = \sqrt{-i\tau} \sum_{l \in \mathbb{Z}} \exp\left[\pi i\left({{\tau(6l - 1)^2}\over{12}} - l\right)\right] = \sqrt{-i\tau}\eta(\tau),$$as we wanted to show.