Q1: A very simple example is given in Grothendieck's Tohoku paper "Sur quelques points d'algebre homologiquie", sec. 3.8. Edit: The space is the plane, and the sheaf is constructed by using a union of two irreducible curves intersecting at two points.
Q2: Cech cohomology and derived functor cohomology coincide on a Hausdorff paracompact space (the proof is given in Godement's "Topologie algébrique et théorie des faisceaux"). I don't know of an example on a non paracompact space where they differ.
The correct construction for a topological category is as follows:
If C is a topological category, we can replace it trivially with a simplicial category by taking the simplicial singular complex associated to each hom-space. By abuse of notation, we will call this functor $Sing$.
Now it suffices to give the answer for simplicial categories.
However, to find the classifying space of a simplicial category, we take its associated quasicategory by looking at the homotopy coherent nerve.
The homotopy coherent nerve is usually constructed formally as the adjoint of another functor called $\hat{FU}$, which is the extension of the bar construction $\bar{FU}$ for the associated comonad $FU:Cat\to Cat$ of the free-forgetful adjunction $U:Quiv\rightleftarrows Cat:F$.
Specifically, given any comonad based at $X$, we can form the bar construction, which gives us a functor from $X\to X^{\Delta^{op}}$. This is done by taking objects to be $F_k=F^{k+1}$ the degeneracies to be instances of the comultiplication map $s_i:F_k\to F_{k+1}=F^i\mu F^{k-i}:F^{k+1}\to F^{k+2}$ and faces given by the appropriate application of the counit (the idea is similar to the above, and I leave it as an exercise). In particular, we may take the whole simplicial object in $End(X)$, which gives us our functor $\bar{F}:X\to X^{\Delta^{op}}$
Back to our specific case, we resolve the comonad $FU:Cat\to Cat$ to a functor $\bar{FU}:Cat\to Cat_\Delta$ (since the resolution is trivial on objects , we can say this with a straight face). Restricting $\bar{FU}$ to $\Delta$, which can always be embedded as a full subcategory of $Cat$. By general abstract nonsense, any functor $X\to C$ where C is cocomplete lifts to a unique colimit preserving functor $Psh(X)\to C$ (since taking presheaves gives a "free" cocompletion). Standard notation suggests that we call this functor $\hat{FU}:sSet\to Cat_\Delta$, but following Lurie, we will call it $\mathfrak{C}$. In particular, this functor has a right adjoint called the homtopy coherent nerve, which we can compute as follows:
$$\mathcal{N}(C)_n:=Hom_{Cat_\Delta}(\mathfrak{C}(\Delta^n),C)$$.
for any simplicial category $C$.
Returning to your original case, $BC=\mathcal{N}(Sing(C))$ for a topological category $C$, and for a topological group, we need only notice that a topological group is identical to a one-object Top-enriched category, all of whose morphisms are invertible (something like this).
As for why this is the right definition, I fear I must refer you to Lurie's HTT. It relies on a proof of a certain Quillen equivalence, and alas, the margins are too small...
Edit: Alright, so the reason why they agree is covered in §4.2.4 of HTT, I'm pretty sure.
Best Answer
Clearly looking at sheaves on the geometric realisation gives something too far removed from the simplicial picture. This is essentially because there are too many sheaves on a simplex have (most of which are unrelated to simplicial ideas). What one could do is to consider such sheaves which are constructible with respect to the skeleton filtration, i.e., are constant on each open simplex. This can be described inductively using Artin gluing. I think it amounts to the following for a simplicial set $F$.
For each simplex $c\in F_n$ we have a set $T_c$, the constant value of the sheaf $T$ on the interior of the simplex corresponding to $c$.
For each surjective map $f\colon [n] \to [m]$ in $\Delta$ the corresponding (degeneracy) map on geometric simplices maps the interior of $\Delta_n$ into (onto in fact) the interior of $\Delta_m$ and hence we have a bijection $T_{f(c)} \to T_c$. These bijections are transitive with respect to compositions of $f$'s.
For each injective map $f\colon [m] \to [n]$ in $\Delta$ the corresponding map on geometric simplices maps $\Delta_m$ onto a closed subset of $\Delta_n$. If $j\colon \Delta^o_n \hookrightarrow \Delta_n$ is the inclusion of the interior we get an adjunction map $T \to j_\ast j^\ast T$ and $j_\ast j^\ast T=T_c$ where $T_c$ also denotes the constanct sheaf with value $T_c$. If $f'\colon \Delta^o_m \hookrightarrow \Delta_n$ is the inclusion of the interior composed with $f$ we can restrict the adjunction map to get a map $T_{f(c)}=f'^\ast T \to f'T_c$ and taking global sections we get an actual map $T_{f(c)} \to T_c$. These maps are transitive with respect compositions of $f$'s.
We have a compatibility between maps coming from surjections and injections. Unless something very funny is going on this compatibility should be that we wind up with a function on the comma category $\Delta/F$ which takes surjections $[n] \to [m]$ to isomorphisms.
There is the stronger condition on the sheaf $F$, namely that it is constant on each star of each simplex. This means on the one hand that it is locally constant on the geometric realisation, on the other hand that $T_{f(c)} \to T_c$ is always an isomorpism.
[Added] Some comments intended to give some kind of relation with the answer provided by fpqc. My suggested answer is not homotopy invariant in the sense that a weak (or even homotopy) equivalence of simplicial sets does not induce an equivalence on the category of sheaves. This is so however if one, as per above, adds the condition that all the maps $T_{f(c)} \to T_c$ are isomorphisms. However, that condition is not so good as many maps that are not weak equivalences induces category equivalences (it is enough that the map induce isomorphisms on $\pi_0$ and $\pi_1$). This is a well-known phenomenon and has to do with the fact the $T_c$ are just sets. One could go further and assume that the $T_c$ are topological spaces and the maps $T_{f(c)} \to T_c$ continuous. Of course adding the condition that these maps be homeomorphisms shouldn't be right thing to do, instead one should demand that they be homotopy (or weak) equivalences. Again, this shouldn't be quite it because of the transitivity conditions. We should not have that the composite $T_{g(f(c))} \to T_{f(c)} \to T_c$ should be equal to $T_{g(f(c))} \to T_c$ but rather homotopic to it. Once we have opened that can of worms we should impose higher homotopies between repeated composites. This can no doubt be (has been) done but there seems to be an easier way out. In the first step away from set-valued $T_c$ we have the possibility of they being instead categories. In that case the higher homotopy conditions is that we should have a pseudofunctor $\Delta/F \to \mathcal{C}\mathrm{at}$. Even they are somewhat unpleasant and it is much better to pass to the associated fibred category $\mathcal{T} \to \Delta/F$. In the general case, and admitting that $\Delta/F$ is essentially the same things as $F$ itself, we should therefore look at (Serre) fibrations $X \to |F|$ or if we want to stay completely simplicial, Kan fibrations $X \to F$. This gives another notion of (very flabby) sheaf which now should be homotopy invariant (though that should probably be in the sense of homotopy equivalence of $\Delta$-enriched categories).