Edit: as GH noticed, the way I tried to state Montgomery's conjecture is wrong. There were some mistakes in the references I used, which compounded with some mistakes of mine, gave a very poor post. Let me try again, hoping that the questions make more sense now.
Second Edit: I have added a bounty to the question. I would be very happy to give it for an answer to just the first greyed question. I am not comfortable with explicit formulas, but I wonder if the conjecture stated here might be translated into conjecture about the distribution of $0$'s on the critical line (assuming GRH) for Dirichlet L-function, uniformly in the character. This in turn should be known (conjecturally I mean) to specialists…
The Montgomery's conjecture I am talking about is the following:
Let $q \geq 1$ an integer, $a \geq 1$ an integer such that $(a,q)=1$, and
let $$\psi(q,a,x) = \sum_{p^\alpha \leq x, p^\alpha \equiv a[q]} \log p.$$
Then for all $\epsilon>0$, one has
$$(1) \ \ \ \ \ \ \ \ \ \ \ \ \psi(q,a,x)=\frac{x}{\phi(q)} + O(x^{1/2+\epsilon}q^{-1/2})$$
uniformly for $x>q$ (that is, the implied constant depends only of $\epsilon$).
Montgomery himself states a slightly stronger conjecture in [M], with the error term in
$O((x/q)^{1/2+\epsilon})$ and forgets to assume $x>q$. Without the latter condition, the conjecture is
trivially false, as noted by GH (see the comments below). Even with the restriction $x>q$, Montgomery's initial conjecture is false, as
proven in [FG], who proposes the slightly weaker statement above as a replacement: see [FG, conjecture 1(b)].
The conjecture is also given in {IK,17.5] but without the restriction $x>q$.
Since Dirichlet, results on primes in arithmetic progressions are proved using Dirichlet characters.
Hence let $\chi$ be a non-principal Dirichlet character of conductor $q$. Define (as usual)
$$\psi(\chi,x) = \sum_{p^\alpha < x} \chi(p^\alpha) \log p.$$
One has $\psi (\chi,x)=\sum_{a \pmod{q}} \chi(a) \psi(q,a,x)$ and $\psi(q,a,x) = \frac{1}{\phi(q)} \sum_\chi \chi(a)^{-1} \psi(\chi,x)$.
Is it reasonable to conjecture the following estimate, uniformly in the non-principal character $\chi$ and the integer $q$, for $x>q$,
$$(2)\ \ \ \ \ \ \ \ \ \ \ \ \ \psi(\chi,x) = O(x^{1/2+\epsilon} q^{-1/2})$$? Has this been conjectured somewhere?
Note that the Montgomery's estimate (1) for $\psi(q,a,x)$ follows immediately from (2)
by summing over $a$ invertible mod $q$ and dividing by $\phi(q)$.
However, it is not clear that (1) implies (2)
as the error term seems to get multiplied by $\phi(q)$. Yet, I have made some numerical computation, and the estimate for Dirichlet character seems to hold.
Now assuming that the first question has a positive answer, I want to generalize it boldly to Artin's representations. Let $\rho : Gal(K/\mathbb Q) \rightarrow Gl_d(\mathbb C)$ be an irreducible, non-trivial representation of dimension $d$, Artin's conductor $q$ and let $\chi$ be its character. In this context, I define (also as usual):
$$\psi(\chi,x)=\sum_{p^\alpha < x} \chi(Frob_p^\alpha) \log p.$$
What would be a reasonable conjectural estimate for $\psi(\chi,x)$, in terms of $q$, $d$, and $x$, giving back the above conjecture when $d=1$? Has anyone formulated such a conjecture?
Perhaps a starting point is the estimate one gets by usual methods assuming both the GRH for the Artin $L$-function $L(s,\rho)=L(s,\chi)$ and the holomorphy of its $L$-function everywhere, that is, Artin's conjecture for $\rho$ (cf. [IK, Theorem (5.15)]):
$$\psi(\chi,x)=O(x^{1/2} \log(x) (d \log x + \log q)).$$
Can we replace the estimate by $O(x^{1/2+\epsilon} d / q^{1/2})$ for example ? I am at a loss even to guess a reasonable formula.
My motivation is trying to understand the best error terms in effective Chebotarev theorem.
Any conjecture as asked will lead to a new estimate for Chebotarev, and I have a
nice collection of examples to test those versions of Chebotarev against.
References:
[FG] Friedlander, Granville, Limitations of the equi-distribution of primes I, Annals of Maths vol. 129, no2, 1989
[IK] Iwaniec, Kowalski, Analytic Number Theory
[M] Montgomery, Problems about Prime Numbers, in PSPM XXVIII, AMS.
Best Answer
The basic heuristic behind the scenes in all of these conjectures is that there is optimal "square-root" cancellation in the various error terms. The estimate (2) is surely not true; the explicit formula links this to a sum over zeros of the related Dirichlet L-function, and it has zeros on the half-line. Instead, one may hope that $$\psi(\chi, x) = \sum_{n < x} \chi(n) \Lambda(n) = O(x^{1/2 + \varepsilon}).$$ This is square-root in the number of terms, with an extra $x^{\varepsilon}$ thrown in for safety. It's much more subtle to figure out powers of logs.
To get to Montgomery's conjecture, you use the formula $$\psi(q,a,x) = \frac{1}{\phi(q)} \sum_{\chi} \overline{\chi}(a) \psi(\chi, x),$$ and imagine that there is square-root cancellation in the sum over $\chi$. You have roughly $q$ terms, are dividing by roughly $q$, and so in all this amounts to having an error of size $x^{1/2+\varepsilon} q^{-1/2 + \varepsilon}$.