Q1 : If $X \to Y \to Z$ are maps of schemes, is there a relation such as
$$\omega_{X/Z} \overset{?}{=} \omega_{Y/Z}|_X \overset{L}{\otimes} \omega_{X/Y}$$
between their dualizing complexes? Or maybe some kind of distinguished triangle?
The reason I think so is that I'm told $\omega_{X/Z}$ is the determinant $\det \mathbb L_{X/Z}$ in the sense of determinants of complexes (if $X \to Z$ is l.c.i.). The cotangent complex always sits in a distinguished triangle
$$\mathbb L_{Y/Z}|_X \to \mathbb L_{X/Z} \to \mathbb L_{X/Y},$$
which leads me to my next question:
Q2 : Does the determinant of complexes send distinguished triangles to tensor products?
Deligne emphasizes this for split exact sequences of vector spaces in La Determinante de Cohomologie: $\det (V \oplus W) = \det V \otimes \det W$ even in the sense of complexes. It's hard for me to imagine that wouldn't extend to triangles in the derived category. The stacks project 0FJW promises to add more details later, so maybe this is obvious.
This came up in a class I'm in. Let $F : X \to X^{(p)}$ be the frobenius. I think surjectivity of the map $F_* \omega_X \to \omega_{X^{(p)}}$ is one of the conditions to be "F-rational." I'd like to understand the cokernel or cone of this morphism in general to see the obstruction to F-rationality and I naively expected something like $\omega_{X/X^{(p)}} = Hom(F_* \mathcal O_X, \mathcal O_{X^{(p)}})$.
Best Answer
A very good reference for these topics is
Lipman, Joseph: Notes on derived functors and Grothendieck duality. Foundations of Grothendieck duality for diagrams of schemes, 1–259, Lecture Notes in Math., 1960, Springer, Berlin, 2009.
For your first question, the issue is the pseudo functoriality of $(-)^!$ together with the characterization of this functor in terms of its value in the structure sheaf.
In more detail, with $f : X \to Y$, $g : Y \to Z$, $h : X \to Z$ and $h = g \circ f$. Assume that all maps are finite type separated map of noetherian schemes. In this case, we have that $h^! \cong f^! \circ g^!$ (loc. cit. Th (4.8.1)).
Second If moreover $f$ is perfect, i.e. $\mathcal{O}_X$ is relatively perfect over $Y$ then $$ f^! \mathcal{F} \cong f^* \mathcal{F} \otimes^L f^! \mathcal{O}_Y $$ (loc. cit. Th (4.9.4)). By introducing the notation $\omega_f = f^! \mathcal{O}_Y$, (and similarly for $g$ and $h$) you get your desired result under the hypothesis mentioned. But beware: in full generality $\omega_f$ is a complex not concentrated in a single degree unless the morphisms are Cohen-Macaulay.
Indeed, as a consequence of the previous discussion, we have the following chain of isomorphisms $$ \omega_h \cong h^! \mathcal{O}_Z \cong f^! g^! \mathcal{O}_Z \cong f^! \omega_g \cong $$ $$ \cong f^* \omega_g \otimes^L f^! \mathcal{O}_Y \cong f^* \omega_g \otimes^L \omega_f $$
As for the formula $\omega_f \cong \det \mathbb L_{f}$, it looks plausible to me under complete intersection hypothesis. I don't know of a published proof. And I don't think it holds under more general hypothesis because without the complete intersection condition, $\mathbb L_{f}$ is not perfect.
Here I interpret $\det$ as something like $L\Lambda^n$, the derived exterior power, where $n$ denotes the relative dimension.
Finally, if $f$ is finite if follows from sheafified duality (loc. cit. Cor. (4.3.6)) that $$ f^! \mathcal{F} \cong \mathbf{R}\mathcal{H}om(f_*\mathcal{O}_X, \mathcal{F})^{\tilde{}} $$ If you substitute by Frobenius you get you last formula, if I understand well.