That categorical definition is for pre-sheaves, the topological definition is for sheaves.
In topological pre-sheaves, a map is surjective if it is epimorphic for each open set $U$ in $X$.
In topological sheaves, however, we instead have to "sheaf-ify" the definition, and we say that the map is "surjective" if the sheaf-ification of the cokernel map is zero.
Basically, in both cases, you have two categories, $\mathcal{Sh}$ and $\mathcal{PSh}$, and in $\mathcal{PSh}$, the "surjective" maps are the ones that are epimorphisms on each $U$, but in the $\mathcal{Sh}$ catageory, you have a more complicated definition of "surjective" (or "epimorphism.")
Consider, instead, two categories, $\mathcal{Ab}$ the category of abelian groups, and $\mathcal{AbTF}$, the full subcategory of "torsion-free" abelian groups - that is, the abelian groups, $A$, where for any $n\in\mathbb Z$ and $a\in A$, $na=0$ iff $n=0$ or $a=0$.
There is the natural inclusion functor $\mathcal{AbTF}\to\mathcal{Ab}$ and a natural adjoint sending $A\to A/N(A)$ where $N(A)$ is the subgroup of nilpotent elements of $A$.
But in $\mathcal{AbTF}$, the "epimorphisms" are not the ones with cokernel (in $\mathcal{Ab}$) $0$, they are the ones with cokerkels which are nilpotent. So, for example, in $\mathcal{Ab}$, the morphism $\mathbb Z\to\mathbb Z$ sending $x\to 2x$ is not an epimorphism, that same map, when considered as a map in $\mathcal{AbTF}$, is an epimorphism.
So consider the "sheafification" functor $\mathcal{PSh}\to \mathcal{Sh}$ to be much like the functor $\mathcal{Ab}\to\mathcal{AbTF}$.
(I believe, but don't quote me, that $f:A\to B$ in $\mathcal{AbTF}$ is an epimorphism if and only if $f\otimes \mathbb Q:A\otimes \mathbb Q\to B\otimes\mathbb Q$ is an epimorphism in $\mathcal{Ab}$.)
I dont know if this will be helpful to you but it is the intuition that I use to think about things. A sheaf is (intuitively) the set of function of some type, continuous, differentiable, analytic, algebraic etc.. So $\mathcal{F}(U)$ is all such function defined on the open set $U$. The maps $r_{UV}:\mathcal{F}(U)\rightarrow \mathcal{F}(V)$ are restriction, sending a function on $U$ to its restriction to the smaller set $V$.
Viewed in this light all of the machinery of sheaf theory makes perfect sense. And this was the original motivation.
Best Answer
It means that for each open set U inside X, $\mathcal{F}(U)$ is a set, and for two open sets $U \subset V$ we have a morphism of sets $\rho_{V,U} :\mathcal{F}(V) \to \mathcal{F}(U)$ such that $\mathcal{F}$ satisfies the following conditions: