I'm looking for a way to describe the dual of a coherent torsion sheaf $\mathcal{F}$ on lets say a variety $X$ over $\mathbf{C}$. Because $Hom(\mathcal{F}, \mathcal{O}_X)=0$, I was wondering if there is another way to define a dual sheaf. If $\mathcal{F}$ is a line bundle, supported e.g. on some reduced divisor $D$ on $X$, then we can still dualise by inverting transition functions. Is there maybe are general way to do this that does not involve taking hom – sections? Maybe via some resolution of $\mathcal{F}$? Thanks!
Alternative definition of dual sheaf
algebraic-geometrysheaf-theory
Related Solutions
The question asked for concrete examples and calculations, something I will leave for others to supply. Instead I aspire to share some insight into Serre duality, and by extension the definition of a dualizing sheaf. I hope that provides useful as well.
The lesson summed up in one sentence is: it helps to look at things in the derived setting. The rest of this post is going to try to explain and justify this assertion. I am trying to make this answer as self-contained as possible, so that no prior knowledge of derived categories is required to make it readable. This has contributed to its significant length, however.
Derived categories of sheaves
If you are not familiar with the formalism of derived categories, just imagine that you are working with chain complexes and the morphisms are just chain maps, tweaked so that all weak equivalences (maps which induce isomorphisms on cohomology) are isomorphisms. Informally, if cohomology can't tell two morphisms apart, the derived category identifies them as well.
For a nice-enough scheme $X$ over a nice-enough field $k$ (I am trying to avoid getting too technical here), let $D(X)$ denote the derived category of coherent sheaves on $X.$ As mentioned above, the objects of $D(X)$ are just complexes of coherent sheaves. There is an obvious embedding $\operatorname{Coh}(X)\to D(X)$ that sends every coherent sheaf to the complex with $\mathscr F$ at the $0$-th spot and $0$ everywhere else. This embedding is fully faithful, which is to say that for each pair of coherent sheaves on $X$ we have $$ \operatorname{Hom}_{D(X)}(\mathscr F,\mathscr G) = \operatorname{Hom}_{\mathscr O}(\mathscr F,\mathscr G) $$ (since the entire story takes place over a fixed scheme $X,$ I will write $\mathscr O$ in place of $\mathscr O_X$ and generally omit the subscript $X$ when no confusion is likely to arise from this omission). We will henceforth identify sheaves with their images in $D(X),$ which is to say with complexes concentrated in $0$-th degree.
Various functors between categories of sheaves induce functors on the respective derived categories. A right-exact (left-exact) functor $F\colon \operatorname{Coh}(X)\to\operatorname{Coh}(Y)$ extends to its left (right) derived functor $RF\colon D(X)\to D(Y)$ ($LF\colon D(X)\to D(Y)$). Taking the cohomologies of these produce the right derived functors $R^iF$ and $L^iF$ you might be more faimiliar with. The two derived functors we will be most interested in are the derived tensor product denoted $\mathscr F\overset L\otimes_{\mathscr O}\mathscr G,$ the left derived functor of $\mathscr F\mapsto \mathscr F\otimes_\mathscr O \mathscr G,$ and the derived Hom denoted $R\mathscr{Hom}_{\mathscr O}(\mathscr F,-).$
One last thing to note is that the Hom-sets of the category $D(X)$ carry not only a $k$-vector space structure, as do Homs of $\operatorname{Coh}(X),$ but actually the structure of cochain complexes of vector spaces over $k.$
Serre duality
Now let's look at Serre duality with derived eyes. It obtains the particulary natural form: $$ \operatorname{Hom}_{D(X)}(\mathscr O, \mathscr F) \cong \operatorname{Hom}_{D(X)}(\mathscr F,\omega)^*.\qquad (1) $$ If you find this formula a bit scary (in the "What does it all mean?!" way), don't worry. I will try to write it out a bit more explicitly in a moment. For now, just appreciate what a nice formally-pleasing formula it is: it allows us to relate morphisms into $\mathscr F$ to (duals of) morphisms from $\mathscr F$ into a canonically determined (only dependent on $X,$ not on $\mathscr F$) object $\omega\in \operatorname{ob}D(X).$ This $\omega$ is what is called the dualizing complex.
In the rest of this post we will rewind the definitions to see what this means. Note in particular that both sides are chain complexes (of vector spaces), so we can take their cohomology groups and compare them. Since we are considering everything in derived categories, two complexes being isomorphic is the same as all their cohomology groups agreeing. (Technically that's not quite true. There furtheremore has to exist a morphism of chain complexes from one to the other which induces those isomorphisms on cohomology. But let's ignore this issue for the moment.)
LHS of (1)
There is an obvious identification $\Gamma(X,\mathscr F) =\operatorname{Hom}_{\mathscr O}(\mathscr O,\mathscr F)$ for any sheaf $\mathscr F$ of $\mathscr O$-modules, obtained just by considering what it means to specify a global section. On the other hand, if $V_\bullet$ is a chain complex of the form $0\to V_0 \to V_1\to\ldots,$ then clearly $H^0(V_\bullet)=V_0.$ Therefore taking the $0$-th cohomology of the left side of (1) we get $$ H^0\left(\operatorname{Hom}_{D(X)}(\mathscr O,\mathscr F)\right) \cong \Gamma(X,\mathscr F) $$ and by the same logic taking higher cohomology produces $$ H^i\left(\operatorname{Hom}_{D(X)}(\mathscr O,\mathscr F)\right) \cong H^i(X,\mathscr F). $$
RHS of (1)
Tensoring sheaves of $\mathscr O$-modules with the sheaf $\mathscr O$ does nothing, and so that continues to be true in the derived category. Therefore the (dual of the) right hand side of (1) can be equivalently written as $$ \operatorname{Hom}_{D(X)}(\mathscr F,\omega)\cong \operatorname{Hom}_{D(X)}(\mathscr F\overset{L}{\otimes}\mathscr O,\omega).\qquad (2) $$ Now recall that functorial things that are true in $\operatorname{Coh}(X)$ are also true in $D(X)$ once you replace all the functors with their respective derived functors. So the adjunction $-\otimes_{\mathscr O}\mathscr F \dashv \mathscr{Hom}_{\mathscr O}(\mathscr F,-)$ and the well-know isomorphism $\mathscr{Hom}_{\mathscr O}(\mathscr F,\mathscr O)\otimes_{\mathscr O}\mathscr G\cong \mathscr{Hom}_{\mathscr O}(\mathscr F,\mathscr G)$ (which holds at least when $\mathscr G$ is locally free; luckily we will only use it for $\omega$ and if $X$ is smooth then there will be no problems) extend to the adjuntion and isomorphism $$ -\overset{L}\otimes_{\mathscr O}\mathscr F \dashv R\mathscr{Hom}_{\mathscr O}(\mathscr F,-),\qquad\quad R\mathscr{Hom}_{\mathscr O}(\mathscr F,\mathscr O)\overset L\otimes_{\mathscr O}\mathscr G\cong R\mathscr{Hom}_{\mathscr O}(\mathscr F,\mathscr G). $$ We can use these to continue the chain of isomorphisms (2) as $$ \cong\operatorname{Hom}_{D(X)}(\mathscr O, R\mathscr{Hom}_{\mathscr O}(\mathscr F,\mathscr O)\overset L\otimes_{\mathscr O}\omega) \cong\operatorname{Hom}_{D(X)}(\mathscr O, \mathscr F^{\vee}\overset L\otimes_{\mathscr O}\omega), $$ where $\mathscr F^{\vee}$ is the standard notation for the dual of $\mathscr F$ though note that we are talking in the derived context here, so we are taking the right-derived functor of $\mathscr{Hom}_{\mathscr O}$.
Anyway, just as we did with the LHS, we can now take the $i$-th cohomology of the (dual of) RHS to obtain, by the same reasoning as for LHS, $$ H^i\big(\operatorname{Hom}_{D(X)}(\mathscr F, \mathscr O\overset L\otimes_\mathscr O \omega)\big)= H^i\big(X, \mathscr F^\vee\overset L\otimes_{\mathscr O}\omega\big).\qquad (3) $$ Those with a keen eye will notice that $\mathscr F$ can't be an arbitrary coherent sheaf, but must instead be locally free for this step to have been legal. So let's assume that too.
We should at this point also take into account that in the statement (1) of Serre duality above, the right side appears as a dual. The dual of a chain complex of vector spaces $V_{\bullet}$ of course replaces each $V_i$ with its dual $V^*_i,$ but that isn't all that is deos. Recall that the fucntor $V\mapsto V^*$ is contravariant, therefore it reverses the orders of the arrows. So the dual of the cochain complex $$ 0\to V_0\to V_1\to V_2\to V_3\to \ldots, $$ which, for this purpose may be written more instructively as $$ \ldots\to 0\to 0\to V_0\to V_1\to V_2\to\ldots, $$ is the cochain complex $$ \ldots\to V_2^*\to V_1^*\to V_0^*\to 0\to 0\to\ldots. $$ That is to say, if $V_i$ appears on the $i-$th spot in the complex $V_\bullet,$ then $V_i^*$ appears on the $(-i)$-th spot in the dual complex $X_\bullet^*.$ From this it is easy to see that $H^i(V_\bullet^*)=H^{-i}(V_\bullet)^*,$ so it follows from (3) that the $i$-th cohomology of the right side of (1) is really $$ H^i\big(\operatorname{Hom}_{D(X)}(\mathscr F, \mathscr O\overset L\otimes_\mathscr O \omega)^*\big)= H^{-i}\big(X, \mathscr F^\vee\overset L\otimes_{\mathscr O}\omega\big). $$
Serre duality, pt. 2
Comparing both sides, we now get Serre duality as a natural bijection $$ H^i(X,\mathscr F) \cong H^{-i}\big(X,\mathscr F^{\vee}\overset L\otimes_{\mathscr O} \omega\big)^*\qquad (4) $$ for all integers $i$ (which, when $\mathscr F$ is just a sheaf instead of a complex of sheaves, is interesting only for $i\ge 0$).
So far everything made sense without any restrictions on $X.$ However if we do restrict our attention when $X$ is smooth and of dimension $n,$ we can identify the dualizing complex as $\omega = \Omega^n[n],$ where $\Omega^n = \bigwedge^n\Omega_{X/k}$ is the sheaf of $n$-forms. ''What is that $[n]$ though?'' you might ask.
Suspension operator
The symbol $[i]\colon D(X)\to D(X)$ dentoes the "shift by $i$" operator, generally known as $i$-fold suspension. It takes a complex and shifts it for $i$ places to the left, explicitly $ (\mathscr F_{\bullet}[i])_j =\mathscr F_{i+j} $ for any complex of coherent sheaves $\mathscr F_\bullet$ on $X.$ What is great about this operator is that for any coherent sheaf $\mathscr F$ on $X$ we have $$ H^0\left(\operatorname{Hom}_{D(X)}(\mathscr O, \mathscr F[i])\right) \cong H^i(X,\mathscr F), $$ extending the identification $\Gamma(X,\mathscr F) =H^0\left(\operatorname{Hom}_{D(X)}(\mathscr O,\mathscr F)\right)$ mentioned previously.
An important case to consider is that of point, i.e. when $X=\operatorname{Spec}(k).$ Then $D(X)$ consists objectwise just of complexes of finite dimensional vector spaces over $k$ and for any such complex $V_\bullet$ we get $$ H^j(V_\bullet[i])= H^{i+j}(V_\bullet). $$ Furthermore it is apparent that suspension commutes with Hom, which is to say that $$ \operatorname{Hom}_{D(X)}(\mathscr F, \mathscr G[i]) = \operatorname{Hom}_{D(X)}(\mathscr F,\mathscr G)[i]. $$
Serre duality, pt. 3
With what we just learned about suspension and identification of the dualizing complex on a smooth $n$-dimensional $X$ as $\Omega^n[n],$ formula (4) can be rewritten finally into the classical form of Serre duality $$ H^i(X,\mathscr F)\cong H^{-i}\big(X,\mathscr F^\vee\overset L\otimes_{\mathscr O}\Omega^n[n]\big)^* = H^{n-i}(X,\mathscr F^\vee\otimes_\mathscr O \Omega^n)^*.\qquad (5) $$
(Really, formula (1) says a bit more than formula (5), for having all cohomology groups isomorphic is a neccessary condition for two objects of the derived category to coincide, but it is not sufficient. That said, most proofs of even the classical form of Serre duality proceed by defining a trace map which can be seen as inducing the isomorphism of formula (1) and which yields formula (5) by passing to cohomology.)
Catharsis
You might at this point be wondering what purpose the journey into derived categories served to only restate the original theorem. I would argue the point is, at least for me, that if Serre duality in the form (5) looks a little artificial or mysterious, it is because it is being viewed it in a catgory in which it does not naturally live. If only one is prepared to look at it in the context of the derived category, it assumes the very simple and natural form (1).
Now the dualizing complex is defined just as something which makes this splendid theorem work in more general contexts. Of course it is in general not a sheaf but a proper complex, but that is not that big of a problem. After all, the language of derived categories was defined in a big part in order to provide a natural context for Serre duality and its generalizations.
Trace morphism
The original question also asked why one would define the trace morphism as it is defined. Let us look at formulation (1) of Serre duality again. It is a duality theorem, asserting an isomorphism between (chain complex of) vector spaces and duals, and as any such isomorphism, it comes from a perfect pairing $$ \operatorname{Hom}_{D(X)}(\mathscr O,\mathscr F)\otimes_k\operatorname{Hom}_{D(X)}(\mathscr F,\omega)\to k.\qquad (6) $$ This is the same thing as the trace morphism Hartshorne defines, once you take $n$-th cohomology of it. But again, the only reason cohomology appears there is because one is really talking about things in the derived category.
This pairing is obtained (as can be deduced already form the proof in Hartshorne's book) essentially by composing the two morphisms. The parallel with trace in the context of vector spaces is at this point probably apparent. But in case it is not, allow me to make it explicit.
Take $X$ to be a point, which of course is understood to be $\operatorname{Spec}(k).$ Then sheaves of $\mathscr O$-modules are just vector spaces over $k$ and they are coherent when they are finite dimensional. The structure sheaf $\mathscr O$ corresponds to $k,$ as does the dualizing sheaf $\omega.$ For a vector space $V$ that corresponds to a coherent sheaf $\mathscr F,$ we then have $$ \operatorname{Hom}_{D(X)}(\mathscr O,\mathscr F) = \operatorname{Hom}_k(k, V)\cong V,\qquad \operatorname{Hom}_{D(X)}(\mathscr F,\omega)=\operatorname{Hom}_k(V,k) = V^*. $$ and so Serre duality amounts to specifying a natural isomorphism $V\cong V^{**}$ (the usual one, of course). This too arises form a perfect pairing $V\otimes_k V^*\to k$ given by sending $v\otimes \varphi\mapsto \varphi(v).$ Under the standard isomorphism $V\otimes_k V^*\cong \operatorname{End}_k(V),$ this clearly corresponds to the trace of an endomorphism.
No, this does not follow. For instance you can take any nontrivial stable vector bundle $E$ on $\mathbb{P}^2$ with $c_1 = 0$. Then for generic line $L$ the restriction $E\vert_L$ is trivial by Grauert-Mulich theorem. On the other hand, if $E\vert_U$ is trivial for some open neighborhood of $L$, then the complement of $U$ is zero-dimensional, hence the isomorphism $E\vert_U \to \mathcal{O}_U^r$ extends (via the pushforward along the inclusion $U \to \mathbb{P}^2$) to an isomorphism $E \to \mathcal{O}_{\mathbb{P}^2}^r$.
Assume $F$ is torsion-free and let $E = F^{\vee\vee}$ be its reflexive hull. Then $E$ is locally free, the natural morphism $F \to E$ is injective, and the cokernel is supported on a zero-dimensional scheme. Thus, we have an exact sequence $$ 0 \to F \to E \to Q \to 0 $$ with $E$ locally free and zero-dimensional $\operatorname{supp}(Q)$. If the line $L$ intersects the support of $Q$, then $Tor_1(\mathcal{O}_L,Q) \ne 0$, hence $F\vert_L$ is not locally free. Thus, the intersection is empty, hence $F\vert_L \cong E\vert_L$. Since $E$ is locally free, we conclude that $c_1(E) = 0$. On the other hand, since the support of $Q$ is zero-dimensional, it does not affect $c_1$, hence $c_1(F) = c_1(E) = 0$.
Bonus question: Any torsion-free sheaf on a smooth surface can be obtained from an exact sequence as above.
Best Answer
The construction you are probably after is the derived functor $R\operatorname{\cal{Hom}}(\mathcal F, {\mathcal O}_X)$. This can be computed by replacing $\mathcal F$ by a locally free resolution and then dualizing this resolution. It might not be what you are after because it gives you a complex. For example, if $X$ is smooth and you want to dualize the trivial line bundle $\mathcal O_D$ on a smooth divisor $D$, its dual will be $\mathcal O_D(D)[-1]$*, i.e. the dual line bundle, but seen as a complex in degree $1$.
Here's one reason why you might want to use $R\operatorname{\cal{Hom}}$ instead of naively dualizing: there is a restriction map $\mathcal O_X\to \mathcal O_D$, but if you dualize these sheaves seen as a line bundle on $X$ and on $D$, you would get the same sheaves. There is no "dual map" going the opposite way between then duals, because $\operatorname{Hom}(\mathcal O_D,\mathcal O_X)=0$. There is, however, a dual "map" in $R\operatorname{Hom}(\mathcal O_D[-1],\mathcal O_X)$.
*Sasha is completely right 😬. The resolution of $\mathcal O_D$ is $$ \mathcal O_X(-D)\to \mathcal O_X, $$ which dualized becomes: $$ \mathcal O_X \to \mathcal O_X(D) $$ Which is the same resolution from before, tensored with $\mathcal O_X(D)$, i.e. a resolution of $\mathcal O_D(D)$. However, dualizing has sent the complex in degrees $[-1,0]$ to $[0,1]$, so it's resolving $\mathcal O_D(D)[-1]$.