Since you've asked for an answer using only the concepts in the question, I'll do my best to give such a proof. However, before I do, I want to say that Hanno's suggestion in the comments is definitely the best way to prove this.
Also your proofs have some issues. I won't point out all the differences, because this answer will be long enough as is, but please take note of them.
If $F:C\to D$ is an equivalence of categories, $\alpha : 1\to \Omega\in C$ is a subobject classifier of $C$, then $F\alpha : F1\to F\Omega$ is a subobject classifier of $D$.
Proof:
As you've noted, we should prove that $F1$ is a terminal object of $D$, and that $F\alpha$ remains a monomorphism.
$F1$ is terminal:
Let $X\in D$, since $F$ is essentially surjective, there exists $A\in C$ such that $FA\cong X$. Then $$D(X,F1)\cong D(FA,F1)\cong C(A,1)\cong \{*\}.$$ Thus $F1$ is terminal in $D$, since all hom sets $D(X,F1)$ are one element sets.
$F$ preserves (and reflects) monomorphisms: Let's rephrase the property of being a monomorphism.
$i:X\to Y$ is a monomorphism in a category $C$ if for all objects $Z\in C$, $g,h:Z\to X$, $ig=ih \implies g=h$. In other words, the map $i_* : C(Z,X)\to C(Z,Y)$ is injective for all objects $Z\in C$.
Now suppose $i:X\to Y$ is a monomorphism in $C$, so $i_*:C(Z,X)\to C(Z,Y)$ is injective.
Then we have the diagram
$$\require{AMScd}
\begin{CD}
C(Z,X) @>i\circ ->> C(Z,Y)\\
@V\simeq VFV @V\simeq VFV \\
D(FZ,FX) @>Fi\circ - >> D(FZ,FY), \\
\end{CD}
$$
so $Fi_*$ is injective a map from $D(FZ,FX)$ to $D(FZ,FY)$ for all $Z\in C$. This is not quite what we want though, but it's good enough. For any object $A\in D$, we can find $Z\in C$ such that $\phi : A\xrightarrow{\cong} FZ$. Then the composite
$$D(A,FX)\newcommand\toby\xrightarrow \toby{-\circ \phi^{-1}} D(FZ,FX) \overset{Fi\circ}{\hookrightarrow} D(FZ, FY) \toby{-\circ\phi} D(A,FY)$$
is injective, and reduces to just the map $f\mapsto Fi\circ f \circ \phi^{-1}\circ \phi = Fi\circ f$. Thus $Fi$ is a monomorphism in $D$.
In particular, $F\alpha : F1\to F\Omega$ is a monomorphism.
Note also that if $Fi$ is a monomorphism in $D$, then by the commutative square above, we see that $i$ must have been a monomorphism in $C$.
$F\alpha : F1\to F\Omega$ is a subobject classifier:
Let $i : A\to X$ be a monomorphism in $D$. Choose $B,Y\in C$ such that
$A\cong FB$ and $X\cong FY$. Let $J : FB\to FY$ be the conjugate monomorphism of $i$.
Fixing these isomorphisms, then there is a unique morphism $\varphi : X\to F\Omega$
making
$$
\begin{CD}
A @>>> F1 \\
@VVi V @VV F\alpha V\\
X @>\varphi>> F\Omega \\
\end{CD}
$$
a pullback square if and only if there is a unique morphism $\Phi : FY\to F\Omega$
making
$$
\begin{CD}
FB @>>> F1 \\
@VVJ V @VV F\alpha V\\
FY @>\Phi>> F\Omega \\
\end{CD}
$$
a pullback square.
Thus it suffices to prove that there is such a unique morphism $\Phi$.
Now since $F:C(B,Y)\to D(FB,FY)$ is a bijection, there is some morphism $j:B\to Y$ such that $J=Fj$. By the note in the previous section, $j$ is a monomorphism.
Thus since $\alpha$ is a subobject classifier in $C$, we have a unique morphism $\psi:Y\to \Omega$ such that the following diagram is a pullback square:
$$
\begin{CD}
B @>>> 1 \\
@VVj V @VV \alpha V\\
Y @>\psi>> \Omega. \\
\end{CD}
$$
It just remains to prove that $\Phi = F\psi$ has the desired property.
Certainly the square commutes, and if $\Phi$ were a morphism making the square commute, then since $F$ induces a bijection on hom sets, we could find a map $\psi$ such that $F\psi=\Phi$ making the corresponding square in $C$ commute. Thus it suffices to prove that
$$
\begin{CD}
B @>>> 1 \\
@VVj V @VV \alpha V\\
Y @>\psi>> \Omega \\
\end{CD}
$$
is a pullback square if and only if
$$
\begin{CD}
FB @>>> F1 \\
@VVFj V @VV F\alpha V\\
FY @>F\psi >> F\Omega \\
\end{CD}
$$
is a pullback square.
This is the same idea as before when we were proving $F1$ was terminal, or that $Fi$ is a monomorphism if and only if $i$ is a monomorphism. In fact, the general fact is that if $F$ is an equivalence of categories, then it both preserves and reflects all limits and colimits. I'll prove that $F$ preserves and reflects all limits, since the general language of cones to a diagram is more convenient than talking about pullbacks specifically.
$F$ preserves and reflects all limits:
Lemma: Let $I:A\to C$ be a diagram in $C$. Let $F:C\to D$ be an equivalence of categories. Let $(X,\newcommand\set[1]{\left\{{#1}\right\}}\set{f_a})$
be a cone to $I$. $(X,\set{f_a})$ is a limit cone if and only if $(FX,\set{Ff_a})$ is a limit cone to $FI$.
Proof
First suppose $(FX,\set{Ff_a})$ is a limit cone. Then given a cone
$(Y,\set{g_a})$ to $I$, the cone $(FY,\set{Fg_a})$ induces a unique map
$\alpha_0 : FY\to FX$ such that $Ff_a \circ \alpha_0 = Fg_a$.
Let $\alpha : Y\to X$ be the corresponding map, so that $F\alpha = F\alpha_0$.
Then we have that $f_a \circ \alpha = g_a$, and $\alpha$ is unique, since
if $\tilde{\alpha}$ also had this property, then $F\tilde{\alpha} : FY\to FX$ would
satisfy the same property as $\alpha_0$. So by uniqueness, $F\tilde{\alpha} = \alpha_0= F\alpha$. Hence $\tilde{\alpha}=\alpha$. Thus $(X,\set{f_a})$ is a limit cone.
Therefore $F$ reflects limits.
Now suppose $(X,\set{f_a})$ were a limit cone. If $(Z,\set{G_a})$ is a cone to $FI$ in $D$,
then choose $Y\in C$ with $Z\cong FY$. Then we have a corresponding cone
$(FY,\set{G_a'})$. As usual, we can now reflect the morphisms into $C$ to
get $g_a : Y\to Ia$ such that $(Y,\set{g_a})$ is a cone to $I$ in $C$.
Then the unique morphism $Y\to X$ induced by this cone becomes
a unique morphism $FY\to FX$, and thus a unique morphism $Z\to FX$. Thus $(FX,\set{Ff_a})$ is a limit cone in $D$.
This completes the proof of both the lemma, and the original claim. $\blacksquare$
Best Answer
First of all, when the hint speaks of a "quasi-inverse" it is referring to the following equivalent of the given definition: a functor $F : \mathbf{C} \to \mathbf{D}$ is an equivalence of categories if and only if there exists a functor $G : \mathbf{D} \to \mathbf{C}$ such that $F \circ G \simeq \operatorname{id}_{\mathbf{D}}$ and $G \circ F \simeq \operatorname{id}_{\mathbf{C}}$; and in this case, $G$ is called a quasi-inverse of $F$.
So, one way to follow the hint would be to explain how $\mathbf{s}$ becomes a functor (i.e. how it operates on morphisms, and show it preserves identities and compositions), and then establish isomorphisms $\mathbf{r} \circ \mathbf{s} \simeq \operatorname{id}$ and $\mathbf{s} \circ \mathbf{r} \simeq \operatorname{id}$.
On the other hand, it is possible to proceed using the definition you stated. First, as a preliminary, I don't know if MacLane and Moerdik specified what exactly $\operatorname{Sh}(\mathcal{B})$ means; but the reasonable definition would be that it is the presheaves on the poset category of $\mathcal{B}$ such that whenever $\{ V_i \mid i \in I \} \subseteq \mathcal{B}$ is a cover of $U \in \mathcal{B}$, we have an equalizer diagram $$F(U) \rightarrow \prod_{i\in I} F(V_i) \rightrightarrows \prod_{i, j \in I, W\in \mathcal{B}, W \subseteq V_i \cap V_j} F(W).$$
(The first step would be to see why $\mathbf{r}$ of a sheaf on $X$ would satisfy this condition; I will leave that as an exercise.)
So, first let us see that $\mathbf{r}$ is injective on morphisms; so, suppose that we have two morphisms $f, g : F \to G$ such that $f(V) = g(V)$ whenever $V \in \mathcal{B}$. Then for any open $U$ and $x \in F(U)$, there is a cover of $U$ by elements $\{ V_i \mid i \in I \} \subseteq \mathcal{B}$. Now, by the hypothesis, $$f(x) {\mid_{V_i}} = f(V_i)(x {\mid_{V_i}}) = g(V_i)(x {\mid_{V_i}}) = g(x) {\mid_{V_i}}$$ for each $i$; and by the injectivity part of the equalizer condition defining that $G$ is a sheaf, we conclude that $f(x) = g(x)$. Since this is true for any open $U$ and any $x \in F(U)$, then $f = g$.
Similarly, to see that $\mathbf{r}$ is surjective on morphisms, suppose we have $f : \mathbf{r}(F) \to \mathbf{r}(G)$. Then for any open $U \subseteq X$ and $x \in F(U)$, again choose a cover of $U$ by $\{ V_i \mid i \in I \} \subseteq \mathcal{B}$. (In fact, to forestall questions of the following construction being well-defined, let us use the canonical maximal cover of all elements of $\mathcal{B}$ contained in $U$.) Then for each $i \in I$, define $y_i := f(V_i)(x {\mid_{V_i}})$. Then for each $i,j$, we can find the canonical maximal cover of $V_i \cap V_j$ by $\{ W_k \mid k \in K_{i,j} \} \subseteq \mathcal{B}$. Now for each $k$, we have $$y_i {\mid_{W_k}} = f(V_i)(x {\mid_{V_i}}) {\mid_{W_k}} = f(W_k)((x {\mid_{V_i}}) {\mid_{W_k}}) = F(W_k)(x {\mid_{W_k}}) = y_j {\mid_{W_k}}.$$ Therefore, by the injectivity part of the sheaf condition on $G$, we have $y_i {\mid_{U_i \cap U_j}} = y_j {\mid_{U_i \cap U_j}}$. Then, by the exactness part of the sheaf condition on $G$, there exists a unique $y \in G(U)$ such that $y {\mid_{U_i}} = y_i$. We now define $f'(U)(x) := y$.
It remains to show that $f'$ defines a morphism of sheaves, and that $\mathbf{r}(f') = f$. (Hint for the morphism of sheaves part: given $U' \subseteq U$ and $x \in F(U)$, show that $(f'(U) {\mid_{U'}}) {\mid_{V_i}}$ is equal to $y_i$ when you put $x {\mid_{U'}}$ in place of $x$, and then apply the injectivity part of the sheaf condition on $G$.)
Now, to show that $\mathbf{r}$ is essentially surjective, suppose we have $F \in \operatorname{Sh}(\mathcal{B})$. Then for each open $U$, define $G(U)$ to be the equalizer in the diagram $$G(U) \rightarrow \prod_{V \in \mathcal{B}, V \subseteq U} F(V) \rightrightarrows \prod_{V, V', W \in \mathcal{B}, V \subseteq U, V' \subseteq U, W \subseteq V \cap V'} F(W).$$ The restriction maps of $G$ will then be constructed based on the universal property of equalizers. We now need to see that $G$ is a sheaf on $X$, and that $\mathbf{r}(G) \simeq F$. The latter follows fairly directly from the sheaf condition on $F$.
For the sheaf condition, suppose we have a cover $\{ U_i \mid i \in I \}$ of $U$ and sections $x_i \in G(U_i)$ such that $x_i {\mid_{U_i \cap U_j}} = x_j {\mid_{U_i \cap U_j}}$ for each $i,j$. Then each $x_i$ can be decomposed into the compatible data of an element of $F(V)$ for every $V \in \mathcal{B}$, $V \subseteq U_i$ which we will call $x_i {\mid_V}$. But then, the union of the canonical covers of each $U_i$ will form a cover of $U$; and for each $W$ in this cover, we can choose $i$ such that $W \subseteq U_i$, and define $y_W := x_i {\mid_W}$. If we have two different indices $i,j$ such that $W \subseteq U_i$ and $W \subseteq U_j$, then from the condition $x_i {\mid_{U_i \cap U_j}} = x_j {\mid_{U_i \cap U_j}}$ we get $x_i {\mid_W} = x_j {\mid_W}$, which makes this definition of $y_V$ well-defined. Once we verify the compatibility condition on $(y_W)$, we get a section $z_V \in F(V)$ from the definition of $F$ being a sheaf. It now remains to show that this family of $z_V$ satisfies the compatibility condition from the definition of $G$, and that the section $x \in G(U)$ we get in this way satisfies $x {\mid_{U_i}} = x_i$ for each $i$. It also remains to establish the uniqueness of $x$.
In the above, you can see that our construction in the "essential surjectivity" proof amounted to specifying the object part of a quasi-inverse $\mathbf{s}$, and our construction in the "surjectivity on morphisms" proof amounted to specifying the morphism part of $\mathbf{s}$. (Note that the definition of $\mathbf{s}$ as you wrote it does not necessarily make sense if $\mathcal{B}$ is not closed under intersections.)