Let $X$ be a scheme and let $G$ be an abstract group acting on $X$ by scheme automorphisms. I'm happy to assume finiteness conditions on $X$ (such as locally Noetherian) and on $G$ (such as $G$ is finite) as necessary.
I understand that it is possible to enlarge the category of schemes in such a way that there is a good "quotient space" $X/G$ associated to this data.
A. What is the correct framework in which to study $X/G$? Is this an example of a quotient stack or perhaps an algebraic space?
There should be a good category of sheaves on $X/G$. Let's agree that the "right" category of sheaves on the scheme $X$ is the category of quasi-coherent sheaves of $\mathcal{O}_X$-modules.
B. What is the "right" category of sheaves on $X/G$?
This should be an abelian category with enough injectives. Assuming the answer to this question is "yes", let's call this category $Qcoh(X/G)$.
C. Is it possible to realise $Qcoh(X/G)$ as a subcategory of the category of abelian sheaves $Sh(\mathcal{C})$ on some Grothendieck site $\mathcal{C}$?
One candidate for such a $\mathcal{C}$ is the set of $G$-stable open subsets of $X$, but presumably this is too naive to work in general.
D. Is there always a morphism of abelian categories $Qcoh(X) \to Qcoh(X/G)$?
On the other hand, it is possible to form the category of $G$-equivariant quasi-coherent sheaves $G-Qcoh(X)$ on $X$.
E. What, if any, is the connection between $G-Qcoh(X)$ and $Qcoh(X/G)$?
I suspect that the answer to this last question is "$Qcoh(X/G)$ is equal to $G-Qcoh(X)$ by definition", in which case the question becomes "Can you shed some light on this definition"?
Finally, references to the literature where similar questions are studied would be greatly appreciated.
Best Answer
I think this question is a good one, but don't expect an encyclopedic answer — MO is not an encyclopedia. Here are some answers, with the disclaimer that I'm a category theorist but not an algebraic geometer.
To question A, by and large the 21st perspective will probably say that it is definitely a stack, in some notion of the word. Certainly there are applications where you do want to consider the "space" quotient $X/G$, in which points in the same orbit are honestly identified. This is like taking a form of "$\pi_0$" of the stack. (Not etale $\pi_0$, certainly, but a form of $\pi_0$ that's valued in spaces rather than sets.)
To questions B, D, and E, the answer is that, as you guessed, the best definition of $\operatorname{QCoh}(X/G)$ is the category of $G$-modules in $\operatorname{QCoh}(X)$, at least when $G$ is a finite group. The geometric intuition is that a quasicoherent sheaf on $X$ is something like a vector bundle over $X$. In the quotient $X/G$, we add an isomorphism between any two points for each way that they are related by an element of $G$. So a vector bundle over $X/G$ should have a fiber over each point of $X$, and an isomorphism between these fibers for each pair of $G$-related points.
There is a quotient morphism $X \to X/G$. The $\operatorname{QCoh}$ functor is best understood as contravariant, just like $\mathcal{O}$ is contravariant. Namely, a geometric morphisms $f: X \to Y$ correspond (modulo details) to symmetric monoidal "linear" functors $f^\ast : \operatorname{QCoh}(Y) \to \operatorname{QCoh}(X)$, which pull back a "vector bundle" along the map. This is certainly true for $\operatorname{QCoh}(X/G) = \operatorname{QCoh}(X)^G$, with the quotient morphism corresponding to the functor "forget the $G$-action". That said, each such functor $f^\ast$ also has a right adjoint $f_\ast : \operatorname{QCoh}(X) \to \operatorname{QCoh}(Y)$, which is not usually symmetric monoidal — it is that takes a "vector bundle" over $X$ and makes it into the "vector bundle" over $Y$ whose stock over $y\in Y$ is the space of sections over $f^{-1}(y)$ of the corresponding bundle on $X$. In the case of the quotient map $X \to X/G$, its right adjoint is the "free" functor, assigning to a quasicoherent module $M$ the corresponding free $G$-module $G \otimes M$. You ask for $f_\ast$ to be "a morphism of abelian categories", which is vague to me. The best definition I know of "morphism of abelian categories" is a right-exact functor (if I have left and right correct), in which case in general pushforward maps are not morphisms — they are instead left-exact. I think that if $G$ is finite, then in fact the pushforward along the quotient map is exact; maybe I need to include that the characteristic of the ground field does not divide the order of $G$.
As for C, as just a subcategory, I'm sure the answer is yes. If you ask for more conditions, the answer is probably still yes, at least in the finite-group case: you should be able to take the category of cocommutative coalgebras in $\operatorname{QCoh}(X/G)$ and find this as sheaves-of-sets on something. But I'd have to think more about details.