To start, here are some references. You can look at Conrad's notes as in Adeel's answer; the first page of those notes already contains what you are asking for in some sense.
Other references include the Stacks project, Vistoli's appendix to his landmark paper or the original paper of Deligne and Mumford. Of course, you can also look at the book of Laumon and Moret-Bailly.
Let me now try to explain a bit how to define the coarse moduli space.
Let $X$ be a finite type Deligne-Mumford algebraic stack of finite type over a noetherian scheme $S$ with finite diagonal.
In Remarque 3.19 of LMB (Laumon, Moret-Bailly) the coarse moduli sheaf of $X$ is defined to be the sheafification of the presheaf $$U\to \{ \mathrm{isomorphism \ classes \ of \ objects \ of \ X_U} \}. $$ This sheaf is not an algebraic space in general.
The paper of Keel and Mori proves that $X$ has a coarse moduli space. The coarse moduli space is defined as a morphism $X\to X^c$ of algebraic stacks with $X^c$ an algebraic space satisfying a certain universal property.
As Niels points out in the comments below and as mentioned above, the coarse space of a stack is not necessarily the coarse sheaf.
The representability of the coarse space $X_{coarse}$ of $X$ by a scheme is a difficult problem, as there are many algebraic spaces which are not representable by a scheme. I'm aware of essentially two non-trivial methods to see that $X_{coarse}$ is a scheme. These are GIT (Mumford) and the methods of Viehweg (see his book). It is with these methods that you can show that the coarse moduli space of the moduli stack of
smooth proper curves of genus $g$, or
principally polarized abelian varieties of fixed degree, or
canonically polarized varieties with fixed Hilbert polynomial, or
hypersurfaces of fixed degree $d\geq 3$ in $\mathbb P^n$ with $n\geq 3$, or
polarized K3 surfaces of fixed degree
is a scheme. (In some special cases you can also argue differently.)
This a general fact and nothing to do with subsheaf of $\Omega^1_X$. If $E$ is a rank two vector bundle with a section, we have an inclusion $O_X\to E$. If you pull back the torsion subsheaf of $E/O_X$, we get an exact sequence, $0\to L\to E\to G\to 0\to 0$, with $L=O_X(S)$, $S$ an effective divisor and $G$ torsion free. One easily checks that the section $O_X\to E(-S)$ vanish only at isolated points.
Best Answer
The Satake compactification of the moduli space of curves $\mathcal M_g$ is the following: consider the image of $\mathcal M_g$ in $\mathcal A_g$ via the Torelli map $j$. Consider the Satake compactification of $\mathcal A_g$ as you describe. Then take the closure of $j(\mathcal M_g)$ in the Satake compactification. This is the Satake compactification of $\mathcal M_g$. This is described briefly in Harris-Morrison's book on moduli of curves. The study of the map from the GIT compactification to the Satake compactification i carried on -to my knowledge- in KNUDSEN, F. The projectivity of the moduli space of stable curves, I. Math. Stand., 39, 19-66, 1976. 11,Math. Stand., 52 161-199,