In Residues and Duality, R. Hartshorne and A. Grothendieck say that there are a plethora of compatibilities that need to be shown in order to have a six functor formalism. For example, if $f:X\to Y$, $g:Y\to Z$, and $h:Z\to W$ are morphisms of ringed spaces, we have isomorphisms
and we would like to be sure that this diagram commutes. While it's simple to see that this diagram commutes, B. Conrad says in Grothendieck Duality and Base Change that some of these compatibilities are quite difficult to verify.
Finally, R. Hartshorne and A. Grothendieck says that "perhaps the language of fibred categories or the techniques of Giraud's thesis will supply what is needed".
I wonder what is the state of the art in this subject and, particularly, if fibred categories are any helpful.
P.S.: the book Triangulated Categories of Mixed Motives by D. C. Cisinski and F. Déglise uses this language to study six functor formalisms but, even though I tried, I admit that it is not clear to me if it solves this problem or not.
Best Answer
If you have an abelian category $A$ with enough injective, for any functor defined on bounded below cochain complexes $\Phi\colon C^+(A)\to D$ sending cohain homotopy equivalences between degree-wise injective cochain complexes to isomorphism, there is a total right derived functor $R\Phi\colon D^+(A)\to D$. This means that there is a natural transformation $\eta_K\colon\Phi(K)\to R\Phi(K)$ for any bounded below cochain complex $K$ and that the pair $(R\Phi,\eta)$ is universal (initial) among all possible choices (i.e. it is the left Kan extension of the functor $\Phi$ along the localization functor from cohain complexes to the derived category). Furthermore, $R\Phi$ is not only a derived functor, but a universal one: this means that, for any functor $\Psi:D\to E$, the pair $(\Psi\circ R\Phi,\Psi(\eta))$ is the right derived functor of the composition $\Psi\circ\Phi$. All this works the same for unbounded complexes is $A$ is an abelian category and if we replace degree-wise injective cochain complexes by a suitable notion (e.g. fibrant of objects in a suitable Quillen model structure).
In particular, in practice, an identification of the form $R(g_*\circ f_*)\cong Rg_*\circ Rf_*$ never comes out of the blue, but rather always as derived functors: the isomorphism above is the unique map which is compatible with the canonical map $(g_*\circ f_*)(K)\to R(g_*\circ f_*)(K)$ and the image by $Rg_*$ of the canonical map $f_*(K)\to Rf_*(K)$ (functorially in $K$). Furthermore, even better, for any functor $\Psi$ whose domain is the codomain of $Rg_*$, the isomorphism $$\Psi(R(g_*\circ f_*)(K))\cong \Psi(Rg_*\circ Rf_*(K))$$ is the unique map which is compatible with the image by $\Psi$ of the canonical map $(g_*\circ f_*)(K)\to R(g_*\circ f_*)(K)$ and of the image by $Rg_*$ of the canonical map $f_*(K)\to Rf_*(K)$ (functorially in $K$).
For the compatibility problem you mention, we have to compare two maps from $R(h_* g_* f_*)$ to $Rh_* Rg_* Rf_*$. Using the universal property of $R(h_* g_* f_*)$, we just have to check that, for any cohain complex $K$, the two induced maps $$(h_* g_* f_*)(K)\to (Rh_* Rg_* Rf_*)(K)$$ agree, which comes right away from the compatibilities mentioned before.
As final remark: all this reasoning can be made in much more general context (e.g. with derived functors defined on homotopy categories of model categories). This extra level of generality means that all the reasoning takes place in a context where there are no possiblities to make sign mistakes because signs do not exist. All the ingredients of the proof are available since the late 1960's.