Yes, that is a fine proof.
As a general principle, if you can write down an isomorphism of vector spaces without making any choices, then you have written down an isomorphism of vector bundles.
Let's make that principle precise in this case. On a small open set $U$, trivialize the vector bundles $V_1$ and $V_2$ with frames (:=sections which form a basis at every point-- note that, seemingly contra the principle, we've made a choice) $e_1, \ldots, e_n$ and $f_1, \ldots, f_m$ (I changed your notation $r_1, r_2$ to $m, n$ to avoid double subscripts).
Then the bundle $V_1 \otimes V_2$ is trivialized with frame $e_i \otimes f_j$ and its determinant trivialized with the singleton frame (alphabetical order)
$$
(e_1 \otimes f_1) \wedge (e_1 \otimes f_2) \wedge \ldots \wedge (e_1 \otimes f_m) \wedge(e_2\otimes f_1) \wedge \ldots \wedge (e_n \otimes f_m)
$$
which we may map to
$$
(e_1\wedge \ldots \wedge e_n)^m \otimes (f_1 \wedge \ldots f_m)^n.
$$
Now it seems so far that our map depends on choices, but it does not. We just need to check that multiplying an $e_i$ by an invertible function, adding $\phi e_i$ to $e_j$ (for $\phi$ an arbitrary function), or flip-flopping $e_i$ and $e_j$ does nothing (and same for $f_i$ and $f_j$). Note that in each case, the given bases for $\text{det}(V_1 \otimes V_2)$ and $\text{det} (V_1)^m \otimes \text{det} (V_2)^n$ multiply by the same scalar function, so the map doesn't change.
We were (allegedly) doing all the above reasoning on a small open set $U$ -- otherwise, there may not exist a frame (the existence of a frame on an open set being equivalent to a vector bundle being trivial). Now suppose we define a global map by doing the same reasoning on ALL open sets. We have to check that if $U$ and $W$ are different, we have defined the same map on $U \cap W$.
But it follows from the independence of choices. The restriction of a choice of frame over $U$ to a frame over $U \cap W$ gives the map of bundles over $U \cap W$, and so does the restriction of a choice of frame over $W$. But we know that the map doesn't depend on a choice. So we've therefore given a global map of vector bundles.
In general it can be general to try to work even more abstractly, i.e. not in terms of picking bases, so that it becomes completely automatic, by the above principle, that a map of vector spaces extends to a map of vector bundles.
I think you can prove this statement from (a weaker form of) the usual Serre duality statement using some compatibilities between derived functors, namely:
- For all $\mathcal{F}^{\bullet}\in D^{b}(X)$ we have a natural isomorphism
$$ R\Gamma(X,-)\circ R\mathcal{Hom}(\mathcal{F}^{\bullet},-)\cong R\operatorname{Hom}(\mathcal{F}^{\bullet},-). $$
- For all $n\in \mathbb{N}$ we have a natural isomorphisms
$$ \operatorname{Hom}(\mathcal{O}[-n],-)\cong H^{n}(R\Gamma(X,-)). $$
- For all $\mathcal{F}^{\bullet},\mathcal{G}^{\bullet},\mathcal{E}^{\bullet}\in D^{b}(X)$ we have an isomorphism
$$ \operatorname{Hom}(\mathcal{F}^{\bullet},\mathcal{E}^{\bullet}\otimes^{L}\mathcal{G}^{\bullet})\cong \operatorname{Hom}(R\mathcal{Hom}(\mathcal{E}^{\bullet},\mathcal{F}^{\bullet}),\mathcal{G}^{\bullet}). $$
This follows from formula (3.15) in Huybrechts' book after applying derived global sections and taking cohomology on degree zero.
The formula (3.15) is functorial, because it is based on a sequence of canonical isomorphisms coming from exercise II.5.1 in Hartshorne's book.
Hence the previous isomorphism is functorial as well.
As a starting point consider the following statement, which is part of the usual Serre duality statement:
For all line bundles $\mathcal{L}$ on $X$ we have a natural perfect pairing
$$ \operatorname{Hom}(\mathcal{L},\omega_{X})\times H^{n}(X,\mathcal{L})\to H^{n}(X,\omega_{X})\cong k $$
We generalize this statement to bounded complexes of coherent sheaves:
For all $\mathcal{F}^{\bullet}\in D^{b}(X)$ we have a natural perfect pairing
$$ \operatorname{Hom}(\mathcal{F}^{\bullet},\omega_{X})\times \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{F}^{\bullet})\to {\operatorname{Hom}(\mathcal{O}_{X}[-n],\omega_{X})}\cong k $$
Proof:
We have to check that the induced morphism
$$ \operatorname{Hom}(\mathcal{F}^{\bullet},\omega_{X})\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{F}^{\bullet})^{\vee} $$
is an isomorphism.
We divide the proof into smaller steps:
- The result is true for finite direct sums of line bundles, because all functors are additive and we know the result for line bundles already.
- The result is true for all coherent sheaves (complexes concentrated on degree zero).
To see this we use that every coherent sheaf $\mathcal{F}$ on a smooth projective variety admits a surjection from a finite direct sum of line bundles $\mathcal{E}=\oplus_{i=1}^{l}\mathcal{L}_{i}$.
So we have a distinguished triangle
$$ \mathcal{K} \to \mathcal{E} \to \mathcal{F} \to \mathcal{K}[1] $$
Apply the cohomological functors $\operatorname{Hom}(-,\omega_{X})$ and $\operatorname{Hom}(\mathcal{O}_{X}[-n],-)^{\vee}$ to obtain a ladder diagram with first row the exact sequence
$$ 0\to \operatorname{Hom}(\mathcal{F},\omega_{X})\to \operatorname{Hom}(\mathcal{E},\omega_{X})\to \operatorname{Hom}(\mathcal{K},\omega_{X}) $$
and second row the exact sequence
$$ 0\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{F})^{\vee}\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{E})^{\vee}\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{K})^{\vee}. $$
The zero on the first row corresponds to a negative ext group between coherent sheaves and the one on the second row follows from Grothendieck vanishing.
By exactness of the rows and the result for finite direct sums of line bundles, the composition $\operatorname{Hom}(\mathcal{F},\omega_{X})\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{E})^{\vee}$ is injective.
By commutativity of the diagram the map $\operatorname{Hom}(\mathcal{F},\omega_{X})\to\operatorname{Hom}(\mathcal{O}_{X},\mathcal{F})^{\vee}$ is injective as well.
Since $\mathcal{F}$ was an arbitrary coherent sheaf on $X$, this is also true for the corresponding map for $\mathcal{K}$.
Hence we can apply the 5-lemma (the version which is sometimes refered to as 4-lemma) to get the isomorphism that we wanted.
- The result is true for all bounded complexes of coherent sheaves.
This can be shown by induction on the length of the complex.
The result is already known for complexes of length zero, so let $\mathcal{F}^{\bullet}$ be a complex of length $n+1$ for some integer $n\geqslant 0$.
For simplicity assume that this complex is concentrated between degrees $-n$ and $0$.
Let $\mathcal{F}^{\bullet}_{\leqslant -1}$ be the stupid truncation (not preserving the cohomology of the complex), so that we get a distinguished triangle
$$ \mathcal{F}^{\bullet}_{\leqslant -1}\to \mathcal{F}^{\bullet}\to \mathcal{F}^{0} \to \mathcal{F}^{\bullet}_{\leqslant -1}[1] $$
By induction hypothesis and arguing with the 5-lemma as before we obtain the desired isomorphism.
[End of proof]
Now we can use the compatibilities mentioned at the beginning to deduce the result in the form stated in Huybrechts' book.
The conclusion is the following:
The category $D^{b}(X)$ has a Serre functor given by
$$ \mathcal{F}^{\bullet} \mapsto \mathcal{F}^{\bullet}\otimes \omega_{X}[n] $$
Proof:
For all $\mathcal{F}^{\bullet}$ and $\mathcal{G}^{\bullet}$ in $D^{b}(X)$ we have the following sequence of functorial isomorphisms:
$$ \operatorname{Hom}(\mathcal{F}^{\bullet},\mathcal{G}^{\bullet}) $$
$$ \cong \operatorname{Hom}(\mathcal{F}^{\bullet}\otimes \omega_{X},\mathcal{G}^{\bullet}\otimes \omega_{X}) $$
$$ \cong \operatorname{Hom}(R\mathcal{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}),\omega_{X}) $$
$$ \cong \operatorname{Hom}(\mathcal{O}_{X}[-n],R\mathcal{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}))^{\vee} $$
$$ \cong \operatorname{Hom}(\mathcal{O}_{X},R\mathcal{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}[n]))^{\vee} $$
$$ \cong H^{0}(R\Gamma(\mathcal{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}[n])))^{\vee}$$
$$ \cong H^{0}(R\operatorname{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}[n]))^{\vee}$$
$$ \cong \operatorname{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}[n])^{\vee}.$$
[End of proof]
Best Answer
You need $E$ to be flat. For an example that doesn't work if $E$ is not flat, take $X=\mathbb P^1$ and $F^\bullet=E=\mathcal O_p[0]$, the structure sheaf of a point concentrated in degree $0$. Since they both supported at a point, we can do this computation on an affine neighborhood of it. For example, let $p=0\in \mathbb A^1$, so that $\mathcal O_p=\frac{k[t]}{tk[t]}$.
We can resolve $E$ by flat sheaves as the complex $$ E\overset{\operatorname{qiso}}\cong \left(k[t]\overset{t}\longrightarrow k[t]\right). $$ Then $F^\bullet\otimes^L E$ is given by tensoring this resolution by $F=\frac{k[t]}{tk[t]}$, so we get the complex $$ F^\bullet\otimes^L E\overset{\operatorname{qiso}}\cong\left(\frac{k[t]}{tk[t]}\overset{t}\longrightarrow \frac{k[t]}{tk[t]}\right)=\left(\frac{k[t]}{tk[t]}\overset{0}\longrightarrow \frac{k[t]}{tk[t]}\right). $$ Now, we can see that $$ H^{-1}(F^\bullet\otimes^L E) \cong \frac{k[t]}{tk[t]} \not\cong 0 =H^{-1}(F^\bullet)\otimes E. $$