Let $\mathbf{cRing}$ be a category of commutative rings and let $\mathbf{Set}$ be a category of sets relative to which $\mathbf{cRing}$ is small (Grothendieck universes). The opposite $\mathbf{Aff}$ of the category of commutative rings becomes a site when we equip it with the Grothendieck topology generated by the pretopology consisting of families $\{\,R \to R[s_i^{-1}]\,\}_i$ such that the $s_i$ generate the unit ideal $(1)$ of $R$. The topology is subcanonical and the Yoneda embedding makes $\mathbf{Aff}$ a full subcategory of the topos $\operatorname{Sh}(\mathbf{Aff})$. I denote the Yoneda embedding by $R\mapsto \operatorname{Spec}R$. I call the objects of the presheaf category $\operatorname{Pr}(\mathbf{Aff})$ Z-functors, and I call the objects of the sheaf topos Zariski-local Z-functors. The Grothendieck topology on $\mathbf{Aff}$ induces a Lawvere-Tierney topology $j: \Omega \to \Omega$ on the subobject classifier of the presheaf topos and the Zariski-local Z-functors are precisely the sheaves for this topology.
The open subfunctors of an affine Z-functor $\operatorname{Spec}R$ are by definition (lecture notes by Marc Nieper-Wißkirchen) those of the form $DI\hookrightarrow \operatorname{Spec}R$, where $I$ is an ideal of $R$ and $(DI)A = \{\phi^*: R \to A\,|\,\text{$\phi^*I$ generates (1)\}}$. The closed subfunctors of $\operatorname{Spec}R$ are up to isomorphism of the form $\operatorname{Spec}R/I \to \operatorname{Spec} R$. Once it is clear what the open/closed subobjects of the representable are, one can define topologies on all Z-functors and one can define schemes.
Question: Is there a way to recover the open and closed subobjects from the topology $j$ on the presheaf category?
Edit: I am sorry, I left out important context. I am reading A functional approach to General Topology, and on page 114 section 2.5. the authors hint that it is possible to get a notion of closed maps from a topology on an elementary topos. According to them the closed maps $f: X \to Y$ are those for which both image $\Sigma_f$ and preimage $f^*$ in the subobject fibration commute with the closure operator induced by $j$. I would like if someone with experience to tell me if this will (probably) give me the closed maps I want or something else entirely.
Best Answer
I will explain why I believe it is not possible to define the Zariski-open subobjects in terms of the Grothendieck topology alone.
First of all, there is a very important conceptual difference between topologies on sets and Grothendieck topologies: topologies on a set tell you about which sets are open, but Grothendieck topologies tell you about which sieves cover. For topologies on a set, the axiom that the union of open sets is open means the notion of coverage is inherited from the powerset – there is no freedom to change what it means to cover. By contrast, the raison d'être of Grothendieck topologies is to change the notion of coverage – even if we restrict our attention to subcanonical topologies, this should be clear from the fact that not every epimorphism in the site becomes an epimorphism of sheaves. (This is why I prefer the "coverage" terminology.)
The paragraph above should be reason enough to be sceptical about the possibility of recovering any notion of open subobject from a Grothendieck topology in general. For the category of schemes in particular there are additional difficulties. Recall that a local ring is a ring $A$ that has a unique maximal ideal. The topological space $\operatorname{Spec} A$ has this property: there is a point whose only open neighbourhood is the entire space. Thus, the only Zariski-covering sieve on $\operatorname{Spec} A$ is the maximal sieve! Nonetheless, provided $A$ is not a field, $\operatorname{Spec} A$ does have non-trivial open subspaces. So any attempt to identify open subschemes in terms of e.g. minimal generating subsets of covering sieves is doomed to failure.
So much for open subschemes. What about closed subschemes? In the category of affine schemes, closed immersions are precisely the regular monomorphisms. This fails already in the category of schemes, because there are non-separated schemes. But if you take the category of affine schemes as given there is a general procedure that will construct the Zariski coverage, so it is hard to say that having a Grothendieck topology adds any information here.
I think you are already convinced that the answer to your original question is no, but in the comments you mention the possibility of defining a modality whose fixed points are the open subobjects. It is not clear to me exactly what you mean but there are obstacles here too. A unary operation $\Box$ on subobjects that can be represented by an endomorphism of the subobject classifier must be pullback-stable. In particular, if we assume that $\Box$ preserves the top subobject $\top$, it follows that the operation must be inflationary: indeed, given any monomorphism $f : X \to Y$, we have the following pullback square, $$\require{AMScd} \begin{CD} X @>{\textrm{id}_X}>> X \\ @V{\top_X}VV @VV{f}V \\ X @>>{f}> Y \end{CD}$$ so we must have $f \le \Box f$ in $\textrm{Sub} (Y)$. Thus $\Box$ cannot be a non-trivial interior operator.