For reference, this is Chapter III Proposition 8.5 in Hartshorne. The claim is this
Let $X$ be a noetherian scheme and let $f: X \rightarrow Y$ be a morphism of $X$ to an affine scheme $Y = \text{spec}A$. Then for any quasi-coherent sheaf $\mathcal{F}$ on $X$, we have
$$
R^{i}f_{*}(\mathcal{F}) \simeq H^{i}(X, \mathcal{F})^{\sim}.
$$
He proves this claim in the following steps.
Step 1: He begins by noting that $f_{*}\mathcal{F}$ is indeed quasicoherent on $Y$ since $X$ is noetherian. Taking global sections we find that both functors agree for $i=0$ when $\mathcal{F}$ is a quasi-coherent sheaf.
Step 2: He then notes that since $\sim$ is an exact functor from the category of $A$-module to the category of $\mathcal{O}_{Y}$-modules, both sides are cohomological $\delta$-functors from the category of quasicoherent sheaves on $X$ to the category of $\mathcal{O}_{Y}$-modules.
Step 3: He then notes that since $\mathcal{F}$ can be embedded into a flasque quasicoherent sheaf that both sides are effaceable for $i>0$.
Step 4: Bringing all these observations together he applies a result of Grothendieck that says that effaceable $\delta$-functors are universal and hence the two are unique.
My concern is that he seems to perform some unjustified sleight of hand in step 2. Suddenly we are only talking about $\delta$-functors from the category of quasicoherent sheaves on $X$. But this is not really the functor we are concerned with. These higher direct image functors, and indeed cohomology functors are defined out of the category of $\mathcal{O}_{X}$-modules. Indeed the category of just quasicoherent sheaves doesn't even have enough injectives so constructing cohomological functors out of it is meaningless. I don't see how this is enough to conclude the thing we actually set out to prove.
Is anyone able to put my mind at ease and show me how the original claim follows from these observations and effaceability?
Best Answer
Your assertion that
is correct but not the whole story: indeed, we can compose these functors with the natural inclusion from quasicoherent $\mathcal{O}_X$-modules to all $\mathcal{O}_X$-modules and still have a cohomological $\delta$-functor. Remember what the definition of a cohomological $\delta$-functor is: it's a family of functors $T^n$ indexed by the non-negative integers plus connecting $\delta$-homomorphisms $T^n(C)\to T^{n+1}(A)$ for every short exact sequence $0\to A\to B\to C\to 0$ such that for every morphism of short exact sequences, a certain diagram commutes. So composing with an exact functor on either side (like this inclusion of categories, or the associated sheaf functor taking an $A$-module to a sheaf on $\operatorname{Spec} A$) retains these properties.
Your next assertion that
is false: quasicoherent sheaves have enough injectives. This is a result originally due to Gabber (published after Hartshorne's book, to be sure), and a full proof may be accessed at StacksProject here.
Even if quasicoherent $\mathcal{O}_X$-modules didn't have enough injectives, Hartshorne has proven that one can compute cohomology and higher direct image using just the category of quasi-coherent $\mathcal{O}_X$-modules and get the same answer as one would in the full category of all $\mathcal{O}_X$-modules (EDIT: this requires $X$ noetherian, which was originally omitted in this answer, though satisfied in the scenario the OP is asking about). The first step here is the fact that we can compute the derived functors using an acyclic resolution - this is Hartshorne III.1.2A. Next, by III.3.6, on a noetherian scheme, any quasicoherent sheaf can be embedded in to a flasque quasicoherent sheaf, and by III.8.3 + III.2.5, flasque sheaves are acyclic for both higher direct image and cohomology, respectively. This implies that we can compute both homology and higher direct image via taking flasque resolutions in the category of quasicoherent $\mathcal{O}_X$ modules, and the answers we get doing this match with the answers in the larger category using any injective resolution.
EDIT 4/7/2020: The final paragraph was missing the assumption that $X$ is noetherian, which is used in Hartshorne III.3.6. Gabber's proof immediately before that makes no such assumption, though it is possible that in the non-noetherian case that something may go wrong and the derived functors of $\Gamma:Qcoh(X)\to Ab$ and $\Gamma:\mathcal{O}_X-mod\to Ab$ may disagree, see Roland's answer here (though it contains no explicit counterexamples). One may well point out that the OP has specified that they're working in the noetherian situation, but I thought this answer needed the clarification that things may go wrong in general.