As you mentioned on Chat, Kap and I wrote a note on forms of different discriminants (but positive definite forms, meaning negative discriminants). This was corrected and extended by John Voight, now at Dartmouth; also published.
The best known examples are the pair $x^2 + xy + y^2$ and $x^2 + 3 y^2.$ The proof that these represent the same numbers is some 2 by 2 matrices, some things mod 2. Same for the indefinite pair $x^2 + xy - y^2$ and $x^2 - 5 y^2.$
Probably worth pointing out that the forms $x^2 + xy + 2ky^2$ and $x^2 + (8k-1)y^2$ represent all the same odd numbers, including any odd primes. The latter form does not represent $2$ or $-2,$ if you can say the same about the former form they agree on primes. We called these "Trivial Pairs." Um; as with Gauss, we discard these if the discriminant is square, meaning we demand $8k - 1 \neq -w^2,$ or $k \neq \frac{1 - w^2}{8}.$
The question changes if you allow square discriminants.
There may be infinitely many other indefinite pairs, we did not check.
If the discriminant is not a square, two forms of the same discriminant that share even a single prime are $GL_2 \mathbb Z$ equivalent. In traditional terms, they are either equivalent or opposite.
Forms with square discriminant, such as $xy$ or $x^2 - y^2,$ are unusual in representing entire arithmetic progressions. Primes do not control things.
For self study, I recommend Buell, Binary Quadratic Forms. I find it easier reading than Buchmann and Vollmer. I also recommend L. E. Dickson Introduction to the Theory of Numbers. For just the first section, I also like Cox, Primes of the Form $x^2 + n y^2.$ Cox does a good job on positive forms, genera, composition. No indefinite forms, though, no Pell. As you can see from my answers, I like the first chapter in Conway, The Sensual Quadratic Form. The wonderful thing there is the "Topograph" construction. I have written a bunch of software to tell me how to avoid arithmetic mistakes in drawing those. These give the best way for talking about a fixed indefinite form $A x^2 + B x y + C y^2$ with $B^2 - 4 AC > 0 $ but not a square. The "cycle" method of Lagrange does not do well when $|n|$ is too large, in finding all solutions to $A x^2 + B x y + C y^2 = n.$ Lagrange's method gives all answers when $|n| < \frac{1}{2} \sqrt{B^2 - 4 AC};$ this result is Theorem 85 in Dickson. Oh, both Lagrange and Conway are talking about primitive representations, $\gcd(x,y) = 1.$
A binary quadratic form, with integer coefficients, is some
$$ f(x,y) = A x^2 + B xy + C y^2. $$ The discriminant is
$$ \Delta = B^2 - 4 A C. $$
We will abbreviate this by $$ \langle A,B,C \rangle. $$ It is primitive if $\gcd(A,B,C)=1. $
Standard fact, hard to discover but easy to check: $$ (A x^2 + B x y + C D y^2 ) (C z^2 + B z w + A D w^2 ) = A C X^2 + B X Y + D Y^2,$$
where $ X = x z - D yw, \; Y = A xw + C yz + B yw. $ This gives us Dirichlet's definition of "composition" of quadratic forms of the same discriminant,
$$ \langle A,B,CD \rangle \circ \langle C,B,AD \rangle = \langle AC,B,D \rangle. $$ In particular, if this $D=1,$ the result represents $1$ and is ($SL_2 \mathbf Z$) equivalent to the "principal" form for this discriminant. Oh, duplication or squaring in the group; if $\gcd(A,B)=1,$
$$ \langle A,B,AD \rangle^2 = \langle A^2,B,D \rangle. $$
This comes up with positive forms: $ \langle A,B,C \rangle \circ \langle A,-B,C \rangle = \langle 1,B,AC \rangle $ is principal, the group identity.
Probably should display some $SL_2 \mathbf Z$ equivalence rules, these are how we calculate when things are not quite right for Dirichlet's rule:
$$ \langle A,B,C \rangle \cong \langle C,-B,A \rangle, $$
$$ \langle A,B,C \rangle \cong \langle A, B + 2 A, A + B +C \rangle, $$
$$ \langle A,B,C \rangle \cong \langle A, B - 2 A, A - B +C \rangle. $$
Imaginary first. Suppose we want to know about $\mathbf Q(\sqrt {-47}).$ Reduced positive forms $ \langle A,B,C \rangle $ obey $|B| \leq A \leq C$ and $B \neq -A,$ also whenever $A=C$ we have $B \geq 0.$ Our group of binary forms is
-47
class number 5
all
( 1, 1, 12)
( 2, -1, 6)
( 2, 1, 6)
( 3, -1, 4)
( 3, 1, 4)
This is an abelian group in any case, so it is cyclic of order 5. These are also the five elements in the ring of integers of $\mathbf Q(\sqrt {-47}).$
Here is the mapping from forms to ideals: given $ \langle A,B,C \rangle, $ drop the letter $C.$ That's it.
$$ \langle A,B,C \rangle \mapsto \left[ A, \frac{B + \sqrt \Delta}{2} \right]. $$
Oh, why is this an ideal, rather than just some $\mathbf Z$-lattice? Because, given $\alpha,\beta$ rational integers, $$ \left[ \alpha, \frac{\beta + \sqrt \Delta}{2} \right] $$ is an ideal if and only if
$$ 4 \alpha | ( \Delta - \beta^2 ). $$
Group: we already see how to do
$$ \langle 2,1,6 \rangle^2 \cong \langle 4,1,3 \rangle \cong \langle 3,-1,4 \rangle; $$
$$ \langle 2,1,6 \rangle \circ \langle 3,-1,4 \rangle \cong \langle 2,5,9 \rangle \circ \langle 3,5,6 \rangle \cong \langle 6,5,3 \rangle \cong \langle 3,-5,6 \rangle \cong \langle 3,1,4 \rangle; $$
$$ \langle 2,1,6 \rangle \circ \langle 3,1,4 \rangle \cong \langle 6,1,2 \rangle \cong \langle 2,-1,6 \rangle. $$
$$ \langle 2,1,6 \rangle \circ \langle 2,-1,6 \rangle \cong \langle 1,1,12 \rangle $$
in any case.
Best Answer
Q1)
a) $X^2-Y^2$ is already in diagonal from. It's matrix is $\bigl( \begin{smallmatrix} 1& 0 \\ 0 & -1 \end{smallmatrix} \bigr)$.
b) $aX^2-aY^2$ is already in diagonal from. It's matrix is $\bigl( \begin{smallmatrix} a& 0 \\ 0 & -a \end{smallmatrix} \bigr)$. We have $\bigl( \begin{smallmatrix} a& 0 \\ 0 & -a \end{smallmatrix} \bigr) \sim \bigl( \begin{smallmatrix} 1& 0 \\ 0 & -1 \end{smallmatrix} \bigr)$
c) $X^2-4XY+3Y^2$ can also be written as $(X-2Y)^2-Y^2$
Result a) and b): if $K=\Bbb{R}$ or $\Bbb{C}$ they are always equivalent. If $\sqrt(a)$ is rational and $K=\Bbb{Q}$ then also, and when $K$ is a finite field then if the characteristic is $2$ then always otherwise only when $a=\pm 1$.
Result (b) and (c) always equivalent (same discriminant, rank and signature).
Q2)
i) if in the second we replace $X=U+V$ and $Y=U-V$ then we get $XY=(U+V)(U-V)=U^2-V^2$, so equivalence.
ii) $3X^2-2Y^2 \sim 3(3X^2-2Y^2)=(3X)^2-6Y^2=U^2-6V^2$