To construct a (coarse or fine) moduli space that is separated, one usually throw away some class of the object in question. For moduli of sheaves people talk about (semi-)stability. A coherent sheaf $E$ on a scheme $X$ is (semi-)stable if it is pure and for all subsheaf $F$ we have $p(F) <(\leq)\text{ } p(E)$, where $p$ is the reduced Hilbert polynomial with respect to some fixed polarization.
My question is, what's the $bad$ property of unstable sheaves so that one has to throw it away to get a $good$ moduli space? Or is this definition merely there to make GIT work?
Best Answer
Stable sheaves are simple, i.e., $\textrm{End}E\simeq \mathbb{C}$. One thing that you want to avoid is the jumping of the automorphism group in a family.
A classical example is to consider a hyperelliptic curve $X$, and $[L]\in\textrm{Pic}^{g-1}X$. If $\pi:X\to \mathbb{P}^1$ is the $g^1_2$, then Grothendieck-Riemann-Roch plus Riemann-Hurwitz tell you that $\pi_\ast L\simeq \mathcal{O}(a-1)\oplus \mathcal{O}(-a-1)$, where $a=h^0(L)$. So you can take a take a family of line bundles over the unit disk $\{L_t\}_{t\in\Delta}$, with $h^0(L_0)=1$, $h^0(L_t)=0$ for $t\in\mathbb{C}^\ast$. Then the generic element will be semistable, $\mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}$, with automorphism group $GL_2$, and over zero you have $\mathcal{O}\oplus \mathcal{O}(-2)$, unstable, with 5-dimensional automorphism group.
And of course you need boundedness, see Donu's comment.
ADDENDUM
Here is an example of how allowing unstable bundles messes up uniqueness of limits (and hence separatedness). Let $X$ be a curve of genus $g\geq 2$, and let $E$ be a semi-stable rank two bundle with $\det E\simeq \mathcal{O}_X$. Let $[L]\in \textrm{Pic}^d X$, $d\geq 2g$. Then $E\otimes L$ is semi-stable of determinant $L^2$. It is globally generated and surjects onto $L^2$, and so $E$ fits in an extension $$ 0\longrightarrow L^{-1}\longrightarrow E\longrightarrow L\longrightarrow 0. $$ Now, take a DVR $R$, $\textrm{Spec }R=\{p,0\}$, where $p$ is the generic point and $0$ the closed point, and consider a family of bundles $\mathcal{F}$ over $\textrm{Spec }R$, for which $\mathcal{F}_0\simeq E$. One can show that if $\mathcal{F}'$ is the elementary transformation of $\mathcal{F}$ along $L$, then $\mathcal{F}_p'\simeq \mathcal{F}_p $, but $\mathcal{F}'_0$ fits in an extension $$ 0\longrightarrow L\longrightarrow \mathcal{F}'_0\longrightarrow L^{-1}\longrightarrow 0. $$ However, by the choice of $L$, $H^1(X,L^2)=0$, so $\mathcal{F}'_0\simeq L\oplus L^{-1}$, an unstable bundle.