This question is a continuation of the following two questions:
Discriminants of indefinite integral binary quadratic forms admitting 3 or 6 torsion.
On certain solutions of a quadratic form equation
Let $f$ be a binary quadratic form with integer coefficients and non-zero discriminant. For $T = \begin{pmatrix}t_1 & t_2 \\ t_3 & t_4 \end{pmatrix} \in \operatorname{GL}_2(\mathbb{R})$, define $f_T(x,y) = f(t_1 x + t_2 y, t_3 x + t_4 y)$. Put
$$\displaystyle \operatorname{Aut}(f) = \{T \in \operatorname{GL}_2(\mathbb{Z}) : f_T = f\}.$$
When $f$ is positive definite, then $\# \operatorname{Aut}(f)$ is easy to determine. In particular, if $f(x,y)$ is reduced, so that it is written as
$$\displaystyle f(x,y) = ax^2 + bxy + cy^2$$
with $|b| \leq a \leq c$ with $ac > 0$, then $\# \operatorname{Aut}(f) = 12$ when $a = b = c = 1$, $\# \operatorname{Aut}(f) = 8$ when $a = c = 1, b = 0$, $\# \operatorname{Aut}(f) = 4$ when $a = \pm b$, and $\#\operatorname{Aut}(f) = 2$ otherwise. When $f$ is reducible, the situation is also easy to account for. The most difficult case is when $f$ is indefinite and irreducible.
One part of the problem is easy. If we consider only positive determinant elements, then they are given by
$$\displaystyle \begin{pmatrix} \dfrac{s+bt}{2} & & -ct \\ \\ at & & \dfrac{s – bt}{2} \end{pmatrix},$$
where $(s,t)$ is a solution to the Pell equation $x^2 – (b^2 – 4ac)y^2 = 4$ (note that this equation is always soluble since $(2,0)$ is a solution).
The problem is the elements of negative determinant. I can show (see the second question linked above) that such elements take the form
$$\displaystyle \begin{pmatrix} m & (bm + cn)/a \\ n & – m\end{pmatrix},$$
where $(m,n)$ is a solution to $am^2 + bmn + cn^2 = a$, which is again always soluble. But there is no guarantee that $(bm+cn)/a$ is an integer. This is guaranteed if $n$ is co-prime to $a$, since then looking at the equation above we see that $a | n(bm+cn)$, whence $a | bm + cn$. There are however examples where this doesn't happen; I believe $5x^2 + 3xy – 11y^2$ is such an example. All of the solutions to $5x^2 + 3xy – 11y^2 = 5$ have $5 | y$, and since $5 \nmid 3$ and $\gcd(m,n) = 1$ by necessity, it is impossible for $3m-11n$ to be divisible by 5 and thus no way for the above matrix to be integral.
However, such examples seem extremely rare. I don't have a lot of computational evidence except to say that there is no such example with $D = b^2 – 4ac < 100$.
The question is, when is the set
$$\displaystyle G^-(f) = \{T \in \operatorname{Aut}(f) : \det T = -1\}$$
non-empty? There are several sufficient conditions for this to be non-empty: if $f$ is equivalent to a diagonal form, if $f$ is equivalent to a form such that $a | b$, , if $f$ is equivalent to a symmetric form, or $f$ is equivalent to a form such that there exists a solution $(m,n)$ to $am^2 + bmn + cn^2 = a$ with $\gcd(n,a) = 1$. Moreover, if the class number is one, then all forms with discriminant $d$ are equivalent and therefore must be equivalent to either $x^2 – (d/4)y^2$ if $d$ is even or $x^2 + xy – (d-1)y^2/4$ if $d$ is odd. Both cases have non-empty $G^-$.
A related question is, suppose that $f_1(x,y) = a_1 x^2 – b_1 y^2$ and $f_2(x,y) = a_2 x^2 – b_2 y^2$ are two non-proportional binary quadratic forms such that $a_1 b_1 = a_2 b_2 > 0$. Then when can $f_1, f_2$ be $\operatorname{GL}_2(\mathbb{Z})$-equivalent?
Best Answer
Here is a necessary and sufficient condition for $G^{-}(f)$ to be non-empty, taken from [1, Exercise 6.21]:
Hint: If $\,^tASA = S$ with $\det(A)= -1$, then show that $\text{Tr}(A) = 0$ and $A^2 = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$. Then reduce the case when $A$ is upper triangular.
If the above conditions hold, the $\text{SL}_2(\mathbb{Z})$-class to which such an $S$ belongs is called an ambig class (sic). This is also what John Robertson calls an ambiguous class in [2]. From this source, we get
[1] T. Ibukiyama, M. Kaneko, "Quadratic Forms and Ideal Theory of Quadratic Fields" in Bernoulli Numbers and Zeta Functions, pp 75-93, 2014.
[2] J. Robertson, "Computing in Quadratic Orders", 2009.