That categorical definition is for pre-sheaves, the topological definition is for sheaves.
In topological pre-sheaves, a map is surjective if it is epimorphic for each open set $U$ in $X$.
In topological sheaves, however, we instead have to "sheaf-ify" the definition, and we say that the map is "surjective" if the sheaf-ification of the cokernel map is zero.
Basically, in both cases, you have two categories, $\mathcal{Sh}$ and $\mathcal{PSh}$, and in $\mathcal{PSh}$, the "surjective" maps are the ones that are epimorphisms on each $U$, but in the $\mathcal{Sh}$ catageory, you have a more complicated definition of "surjective" (or "epimorphism.")
Consider, instead, two categories, $\mathcal{Ab}$ the category of abelian groups, and $\mathcal{AbTF}$, the full subcategory of "torsion-free" abelian groups - that is, the abelian groups, $A$, where for any $n\in\mathbb Z$ and $a\in A$, $na=0$ iff $n=0$ or $a=0$.
There is the natural inclusion functor $\mathcal{AbTF}\to\mathcal{Ab}$ and a natural adjoint sending $A\to A/N(A)$ where $N(A)$ is the subgroup of nilpotent elements of $A$.
But in $\mathcal{AbTF}$, the "epimorphisms" are not the ones with cokernel (in $\mathcal{Ab}$) $0$, they are the ones with cokerkels which are nilpotent. So, for example, in $\mathcal{Ab}$, the morphism $\mathbb Z\to\mathbb Z$ sending $x\to 2x$ is not an epimorphism, that same map, when considered as a map in $\mathcal{AbTF}$, is an epimorphism.
So consider the "sheafification" functor $\mathcal{PSh}\to \mathcal{Sh}$ to be much like the functor $\mathcal{Ab}\to\mathcal{AbTF}$.
(I believe, but don't quote me, that $f:A\to B$ in $\mathcal{AbTF}$ is an epimorphism if and only if $f\otimes \mathbb Q:A\otimes \mathbb Q\to B\otimes\mathbb Q$ is an epimorphism in $\mathcal{Ab}$.)
I will explain why I believe it is not possible to define the Zariski-open subobjects in terms of the Grothendieck topology alone.
First of all, there is a very important conceptual difference between topologies on sets and Grothendieck topologies: topologies on a set tell you about which sets are open, but Grothendieck topologies tell you about which sieves cover.
For topologies on a set, the axiom that the union of open sets is open means the notion of coverage is inherited from the powerset – there is no freedom to change what it means to cover.
By contrast, the raison d'être of Grothendieck topologies is to change the notion of coverage – even if we restrict our attention to subcanonical topologies, this should be clear from the fact that not every epimorphism in the site becomes an epimorphism of sheaves.
(This is why I prefer the "coverage" terminology.)
The paragraph above should be reason enough to be sceptical about the possibility of recovering any notion of open subobject from a Grothendieck topology in general.
For the category of schemes in particular there are additional difficulties.
Recall that a local ring is a ring $A$ that has a unique maximal ideal.
The topological space $\operatorname{Spec} A$ has this property: there is a point whose only open neighbourhood is the entire space.
Thus, the only Zariski-covering sieve on $\operatorname{Spec} A$ is the maximal sieve!
Nonetheless, provided $A$ is not a field, $\operatorname{Spec} A$ does have non-trivial open subspaces.
So any attempt to identify open subschemes in terms of e.g. minimal generating subsets of covering sieves is doomed to failure.
So much for open subschemes.
What about closed subschemes?
In the category of affine schemes, closed immersions are precisely the regular monomorphisms.
This fails already in the category of schemes, because there are non-separated schemes.
But if you take the category of affine schemes as given there is a general procedure that will construct the Zariski coverage, so it is hard to say that having a Grothendieck topology adds any information here.
I think you are already convinced that the answer to your original question is no, but in the comments you mention the possibility of defining a modality whose fixed points are the open subobjects.
It is not clear to me exactly what you mean but there are obstacles here too.
A unary operation $\Box$ on subobjects that can be represented by an endomorphism of the subobject classifier must be pullback-stable.
In particular, if we assume that $\Box$ preserves the top subobject $\top$, it follows that the operation must be inflationary: indeed, given any monomorphism $f : X \to Y$, we have the following pullback square,
$$\require{AMScd}
\begin{CD}
X @>{\textrm{id}_X}>> X \\
@V{\top_X}VV @VV{f}V \\
X @>>{f}> Y
\end{CD}$$
so we must have $f \le \Box f$ in $\textrm{Sub} (Y)$.
Thus $\Box$ cannot be a non-trivial interior operator.
Best Answer
This is no trouble: there is a Grothendieck topology on $\mathrm{Top}$ in which the coverings are jointly surjective families of open inclusions, and your notion of sheaf is precisely the same as the notion of sheaf for this covering. There is an issue in talking about the category of presheaves on $\mathrm{Top}$, since the latter is a large category; you can handle this either by restricting the size of your spaces or by allowing presheaves with values in large sets, or, if you really want to consider presheaves on arbitrary spaces valued in small sets, by giving up a few of the nice properties of legitimate Grothendieck toposes.