First thing – I don't know any algebraic geometry. I'm trying to understand a little bit about quasi-coherent sheaves but not for the sake of AG, so please rely on as little knowledge as possible.
What follows is an excerpt from Dieudonné's History of Algebraic Geometry, VIII.2.21:
"Serre's principal goal is to extend, as much as possible, to his varieties the results on sheaf cohomology described above for the classical case ($k=\mathbb C$). He restricts to coherent $\mathcal O_X$-modules (in order to be able to use the exact cohomology sequence with the definition of the cohomology groups ("Čech cohomology") of which he avails himself)…"
The exact cohomology sequence mentioned is the one induced by a short exact sequence of abelian sheaves
$$0\longrightarrow \mathcal N\longrightarrow \mathcal G\longrightarrow \mathcal G/\mathcal N\longrightarrow 0$$
From this I understand the original purpose of coherent sheaves is simply the fact they form an abelian category. However, I don't understand why this justifies restricting to coherent sheaves nor why they were specifically chosen from all the abelian subcategories of $\mathcal O_X$-modules:
Couldn't Serre do his homological algebra equally well in $\mathcal
O_X$-$\mathsf{Mod}$? What's the benefit of restricting to
$\mathsf{Coh}(X)$?
Why exactly did coherent sheaves come into play?
Best Answer
All Serre needed to use was quasicoherent sheaves. For these it's a fundamental theorem that Cech cohomology is the "correct" cohomology. That means that it agrees with the cohomology defined abstractly, via derived functors. Since derived functors give rise to long exact sequences, the fundamental theorem implies that short exact sequences of (quasi)coherent sheaves yield long exact sequences in Cech cohomology. This just isn't true for coarser abelian categories like arbitrary sheaves or presheaves.
For an idea of why the theorem is true for quasicoherent sheaves, it's generally true that Cech cohomology becomes the right cohomology when the space has a "good cover," which in this case means a cover whose elements have no higher cohomology for whatever sheaf we're investigating. This is always true for quasicoherent sheaves by a reduction to affine varieties and then commutative algebra, which isn't possible for general sheaves since there may be no cover on which the sheaf is that associated to a module over a ring.