I am really embarrassed to ask this question since I am supposed to be an expert on modular forms, but here goes anyway since I have been stuck on this
for several days.
Let $q$ be a prime divisor of $N$ such that $q^2\nmid N$, and let $\chi$ be some Dirichlet character modulo $N$. If $F$ is a normalized eigenform (newform of course) in
$S_k(\Gamma_0(N),\chi)$ with Fourier coefficients $a(n)$, it is known from
Atkin–Lehner–Li that if $W_q$ denotes an Atkin–Lehner involution then
$F$ is an eigenform for the action of $W_q$ with eigenvalue $-a(q)/q^{k/2-1}$, and also that $a(q)^2=\chi(q)q^{k/2-1}$.
I am implementing this in Pari/GP and found a contradiction. Most probably
my implementation is wrong, but please follow the following simple example
(I know that David Loeffler had a ticket about a similar problem in Sage, but that was solved).
Consider the space $S_3(\Gamma_0(12),\chi_{-4})$, where as usual $\chi_{-4}$
is the odd character mod 4. It has a single eigenfunction defined over
$\mathbb Q(\sqrt{-3})$, together with its conjugate.
I can choose $W_3=[3,-1;12,-3]$, and $\tau=i=\sqrt{-1}$, so that
$W_3(\tau)=(3i-1)/(12i-3)$. I then compute $1000$ Fourier coefficients
of $F$, and the value $F(i)$ and $F|_3W_3(i)$. Instead of finding
$-a(3)/3^{1/2}$ as ratio, I find $+a(3)/3^{1/2}$ (equal to $i$ and $-i$, but in the wrong order or equivalently with the wrong sign).
I did many other experiments, both in odd and even weights, and with nonreal characters, and my
implementation tells me that the eigenvalue should be
$-a(q)\overline{\chi(q)}/q^{k/2-1}$.
Now I went through Winnie Li's proof which I essentially reproduced in my book with F. Stromberg, and it seems correct. So I am stuck. Can someone at least check the above example, or tell me what I am doing wrong ? Note that
I am not mixing the two possible embeddings of my form: both give me the wrong sign.
ADDED March 27: I am very surprised not to have any (positive or negative)
answer. Since the error is systematic, let me give another example in
even weight with numerical values. Consider $S_2(\Gamma_0(35),\chi_5)$.
It is of dimension $2$ generated by an eigenform and its conjugate. One
specific embedding of the eigenform is (sorry for the GP notation)
$$F(\tau)=q + 2*I*q^2 – I*q^3 – 2*q^4 + (-2 – I)*q^5 + 2*q^6 – I*q^7 + 2*q^9 + O(q^{10}).$$
I take a random point, say $\tau=I/2+1/Pi$. Simply by summing sufficiently
many terms I find $F(\tau)=-0.015178…+0.03676…*I$.
I choose as Atkin–Lehner matrix for the prime $q=7$ the matrix
$W_7=[7,-3;35,-14]$ which has the required properties. I compute
$F|_2W_7(\tau)$ (here I use 2000 terms since the imaginary part is small)
and I find that it is equal to $0.03676+0.015178…*I$, so the eigenvalue
is $-I$. However, Atkin–Lehner-Li tells me it should be equal to $-a(7)$,
which as one sees from above is equal to $+I$, contradiction.
Can someone please check this in Sage, Magma, or for that matter Pari/GP ?
Best Answer
Let's examine the case $N=12$, weight $3$, with central character $\chi_{-4}$. According to the LMFDB, there are two newforms $f$ in this space. I will take it as numerically-verified that for both of these forms, $f|W_3$ is proportional to $f$.
Theorem 2.1 of Atkin-Li (Twists of newforms and pseudo-eigenvalues of W-operators. Invent. Math. 48 (1978), no. 3, 221–243) then states $f|W_3 = \lambda f,$ where $$\lambda = -\frac{|a_f(3)|}{a_f(3)}.$$ Here the minus sign comes from their convention that the Gauss sum of a trivial character is $-1$, under the conditions at hand. Here I multiplied by $|a_f(3)|$ because the pseudo-eigenvalue has absolute value $1$. Indeed, the earlier Proposition 1.1 from that paper shows $\lambda^2 = -1$, so the only question is the sign of $\lambda$. Now the above formula for $\lambda$ becomes $\lambda = - \frac{\overline{a_f(3)}}{|a_f(3)|} = \frac{a_f(3)}{|a_f(3)|}$, since $\lambda = \pm i$. Isn't this the numerical value that is claimed in the original question?
added later: I consulted Theorem 3 of Li's 1975 Math. Annalen paper (Newforms and functional equations). I will continue the discussion of this example. Li defines a version of the Atkin-Lehner operator, denoted $V_3^M$ (I will drop the superscript in what follows). However, I think that her choice of $V_3$ is inconsistent with $W_3$. According to Li's definition, we may choose $V_3 = \begin{pmatrix} -3 & -1 \\ 12 & 3 \end{pmatrix}$. She requires the lower-right entry to be $3$. With $W_3$ given as in the original question, we have $$V_3 W_3^{-1} = \begin{pmatrix} 7 & -2 \\ -24 & 7 \end{pmatrix}.$$ Since $\chi_{-4}(7) = -1$, we have $$f|V_3 = - f |W_3.$$
It should be mentioned that the definition of the Atkin-Lehner operator in the Atkin-Li paper has slightly different congruence conditions. In particular, I checked that the choice of $W_3$ in the original question is consistent with the definition of the Atkin-Lehner definition.
Unless I made some mistakes, I believe the above discussion clears up the problem, at least in this level $12$ example.