This is a sequel of this question where I asked for which positive integer $n$ the
set of primes of the former $x^2+ny^2$ was defined by congruences (a set of primes $P$ is defined by congruences if there is a positive integer $d$ and a subset $A$ of $\mathbb{Z}/d\mathbb{Z}$ such that a prime $p$ is in $P$ if and only if $p$ mod $d$ is in $A$, up to a finite number of exceptions). I was taught there that the answer was "exactly when $n$ is idoneal", that there is finitely many idoneal numbers, and that all are known but perhaps one.
When is the set of primes of the form $x^2+ny^2+mz^2$ ($x,y,z \in \mathbb{Z})$ defined by congruences?
My motivation is not just an idle ternary generalization of the binary case. I really met this question while working on a problem concerning modular forms, and also the slightly more general question, given a fixed positive integer $a$: when is the set of primes $p$ such that $ap$ has the form $x^2+ny^2+mz^2$ ($x,y,z \in \mathbb{Z})$ defined by congruences?
I am well aware that since the set of integers represented by a ternary quadratic form is not stable by multiplication, it is much less
natural to ask the question for prime numbers instead of all positive integers than in the case of a binary quadratic form.
Yet this is really the question for primes that appears in my study (for about a dozen specific ternary forms, actually).
I have found a very interesting paper by Dickson (Ternary quadratic forms and congruences. Ann. of Math. (2) 28 (1926/27), no. 1-4, 333–341.) which solves the question for the integers represented by $x^2+ny^2+mz^2$: there is only a finite explicit numbers of $(n,m)$ such that this set of integers is defined by congruences (in the obvious sense). But the proof does not seem (to me) to be easily generalizable to primes. Other mathscinet research did not give me any more informations.
When I try to think to the question, I meet an even more basic (if perhaps slightli more sophisticated) question that I can't answer:
When is the set of primes of the form $x^2+ny^2+mz^2$ ($x,y,z \in \mathbb{Z})$ Frobenian? (Is it "always"?)
A set of primes $P$ is called Frobenian (a terminology probably introduced by Serre) if there is a finite Galois extension $K/\mathbb{Q}$, and a subset $A$ of Gal$(K/\mathbb{Q})$ stable by conjugacy such that a prime $p$ is in $P$ if and only if Frob${}_p \in A$, except for a finite number of exceptions. A set determined by congruences is a Frobenian set for which
we can take $K$ cyclotomic over $\mathbb{Q}$, which is the same by Kronecker-Weber as abelian over $\mathbb{Q}$. For a quadratic binary
quadratic form (for example $x^2+ny^2$), the set of represented primes is always Frobenian ($K$ can be taken as the
ring class field of $\mathbb{Z}[\sqrt{-n}]$, and $A=\{1\}$, as explained in Cox's book). But I fail to see the reason
(which may nevertheless be trivial) for which the same result would be true for a general ternary quadratic form.
I should add that for my specific ternary forms, I can show that the set is Frobenian, but I am not sure how to extend the argument to all ternary quadratic forms.
Finally, let me say that I would be interested in any book, survey or references on this kind of question (which surely must have been studied), and that I am also interested in analog questions for quaternary quadratic forms (which might be easier,
because of multiplicative properties related to quaternions).
Best Answer
Mostly, you should look at a number of items at
(address updated in 2018):
http://zakuski.math.utsa.edu/~kap/
including Dickson_Diagonal_1939.pdf and Kap_Jagy_Schiemann_1997.pdf to begin with.
Now that I think of it, you also need to read Kap_All_Odd_1995.pdf at the same place, also a new preprint by Jeremy Rouse on his 451 Theorem(s), as it is readily decided whether a form represents the single number 2. In a different direction, you need Wai Kiu Chan and Byeong-Kweon Oh, "Positive Ternary Quadratic Forms with Finitely Many Exceptions," Proceedings of the A.M.S., Volume 132, Number 6, Pages 1567-1573 (2004).
The overriding fact is that ternary forms, like binary forms, are collected together in genera. Unlike binary forms, these vary in size (class number) for a fixed discriminant. The good thing is the result of Jones, every number given by congruence conditions (a finite number of "progressions") is, in fact, represented by at least one form in the genus.
So, an example, $x^2 + 4 y^2 + 9 z^2$ is not regular, so it is not in Dickson's list. The genus of this form represents all numbers not of shape $9 n \pm 3, \; 8 n + 3, \; 4^k (8n+7).$ The other class in the genus is $x^2 + y^2 + 36 z^2.$ Between the two forms, all eligible numbers are represented. It is not difficult to prove that $x^2 + 4 y^2 + 9 z^2$ misses only the single number 2 out of the eligible numbers. So, if you were so minded, you could say that $x^2 + 4 y^2 + 9 z^2$ represents all primes that pass the above restrictions as well as not being $0 \pmod 2.$ As the restriction $9n \pm 3$ does not affect larger primes, just larger composite numbers, one could also say that $x^2 + 4 y^2 + 9 z^2$ represents all primes $p \equiv 1 \pmod 4.$
So there is a built in problem with your formulation. If one of your positive ternaries $x^2 + m y^2 + n z^2$ has finitely many exceptions, in particular finitely many prime exceptions $p_1,p_2,\ldots,p_k,$ the form can be said to represent every prime $p$ fitting the original restrictions and the new restrictions $p \neq 0 \pmod {p_k.}$ As a result, your list of ternary forms is infinite and unprovable. As Franz says, you need a tighter formulation.
If you like, email me your list of forms, we can discuss it.
A better example of the possible horror: Ono and Soundararajan (1997) showed that Ramanujan's form $x^2 + y^2 + 10 z^2$ has only squarefree numbers as sporadics (numbers represented by some form in the genus but not by this form, "exceptions" for Chan and Oh). They also showed that GRH implies that the known list is complete. So, GRH implies that $x^2 + y^2 + 10 z^2$ represents all primes not divisible by any of 3, 7, 31, 43, 67, 79, 223, 307, 2719. The other sporadics are composite. At the same time, it is easy to show that the form represents all numbers $n \equiv 5 \pmod 6,$ first pointed out in a letter from J.S.Hsia to Kaplansky, later a cheap proof by me, and one by Oh.
EDIT: it occurs to me that an alternate property could be: a positive ternary form will be defined to be fungible if its sporadics are all composite. Or perhaps funicular. I looked it up, the best would be frangible.
EDIT TOOOOO: I thought I might find examples of forms $x^2 + m y^2 + n z^2$ that seem to be fungible, or perhaps funicular, or frangible, despite lacking proof. The first example is $$ x^2 + y^2 + 48 z^2 \neq 21 \cdot 9^k $$ compared with the other form in that genus, $2 x^2 + 2 y^2 + 13 z^2 + 2 y z + 2 z x,$ checked on numbers up to 1,250,000. Very similar, $$ x^2 + 4y^2 + 20 z^2 \neq 77 $$ compared with the other form in that genus, $4 x^2 + 4y^2 + 5 z^2,$ also checked on numbers up to 1,250,000. In this second case it is easy to show that each form of the genus represents 4 times any number represented by the other form, and no numbers $2 \pmod 4$ are represented anyway, so only odd numbers come up. Anyway, 21 and 77 are composite. I have not proved these completely, just checked on computer.
EDIT TOOTOOTOO: I got an opinion from Jeremy Rouse. He points out that any positive ternary has two possible causes for having infinitely many numbers missed (compared to its genus), those being high divisibility by anisotropic primes or spinor exceptional classes. These two phenomena affect only finitely many squareclasses. In both cases, we do not increase the set of primes missed, with the result that a positive ternary fails to represent only a finite number of eligible (by congruence conditions) primes. This also explains, to some degree, the reference to Duke and Schulze-Pillot (1990). The final corollary says that any sufficiently large number that is primitively represented by some form in the same spinor genus is represented by the form of interest. There are only a few spinor exceptional squareclasses, so, even in an irregular spinor genus, we can only miss finitely many squarefree numbers, as those other than the spinor exceptional integers are represented by something in the same spinor genus, and primes are squarefree and therefore represented primitively if at all. I think I've caught up now. Note the D_S-P results give no effective bound, so we cannot identify the primes missed without some fortunate accident such as regularity, spinor regularity, regularity with regard to all odd numbers, and so forth.