The correct construction for a topological category is as follows:
If C is a topological category, we can replace it trivially with a simplicial category by taking the simplicial singular complex associated to each hom-space. By abuse of notation, we will call this functor $Sing$.
Now it suffices to give the answer for simplicial categories.
However, to find the classifying space of a simplicial category, we take its associated quasicategory by looking at the homotopy coherent nerve.
The homotopy coherent nerve is usually constructed formally as the adjoint of another functor called $\hat{FU}$, which is the extension of the bar construction $\bar{FU}$ for the associated comonad $FU:Cat\to Cat$ of the free-forgetful adjunction $U:Quiv\rightleftarrows Cat:F$.
Specifically, given any comonad based at $X$, we can form the bar construction, which gives us a functor from $X\to X^{\Delta^{op}}$. This is done by taking objects to be $F_k=F^{k+1}$ the degeneracies to be instances of the comultiplication map $s_i:F_k\to F_{k+1}=F^i\mu F^{k-i}:F^{k+1}\to F^{k+2}$ and faces given by the appropriate application of the counit (the idea is similar to the above, and I leave it as an exercise). In particular, we may take the whole simplicial object in $End(X)$, which gives us our functor $\bar{F}:X\to X^{\Delta^{op}}$
Back to our specific case, we resolve the comonad $FU:Cat\to Cat$ to a functor $\bar{FU}:Cat\to Cat_\Delta$ (since the resolution is trivial on objects , we can say this with a straight face). Restricting $\bar{FU}$ to $\Delta$, which can always be embedded as a full subcategory of $Cat$. By general abstract nonsense, any functor $X\to C$ where C is cocomplete lifts to a unique colimit preserving functor $Psh(X)\to C$ (since taking presheaves gives a "free" cocompletion). Standard notation suggests that we call this functor $\hat{FU}:sSet\to Cat_\Delta$, but following Lurie, we will call it $\mathfrak{C}$. In particular, this functor has a right adjoint called the homtopy coherent nerve, which we can compute as follows:
$$\mathcal{N}(C)_n:=Hom_{Cat_\Delta}(\mathfrak{C}(\Delta^n),C)$$.
for any simplicial category $C$.
Returning to your original case, $BC=\mathcal{N}(Sing(C))$ for a topological category $C$, and for a topological group, we need only notice that a topological group is identical to a one-object Top-enriched category, all of whose morphisms are invertible (something like this).
As for why this is the right definition, I fear I must refer you to Lurie's HTT. It relies on a proof of a certain Quillen equivalence, and alas, the margins are too small...
Edit: Alright, so the reason why they agree is covered in ยง4.2.4 of HTT, I'm pretty sure.
You're spot on with Hatcher's construction. First he constructs the space $EG$ as a $\Delta$-complex, but this can easily be upgraded to a simplicial set $\mathcal{E}G$ such that $EG$ is the geometric realisation of $\mathcal{E}G$:
In fact let $\mathcal{E}G_n:=G^{n+1}$ and define the face maps as $(g_0, ..., g_n) \mapsto (g_0, ..., \widehat{g_i}, ..., g_n)$ and the degeneracy maps as $(g_0, ..., g_{n-1}) \mapsto (g_0, ..., g_i, g_i, ...., g_n)$. Then $G$ acts by simplicial maps via $g(g_0, ..., g_n) \mapsto (gg_0, ..., gg_n)$ and therefore it also acts simplicially on the realisation $EG:=|\mathcal{E}G|$. In fact $G$ acts freely and $EG$ is contractible so that $EG/G$ is a model of $BG$.
He also explains with the bar notation how this space is exactly equal to the geometric realisation of the nerve of $G$ (even if Hatcher does not use these words). Explicitly: $[g_1|g_2|...|g_n]$ is the image in $EG/G$ of all the simplices $g_0\cdot(1,g_1,g_1g_2,...,g_1..g_n)$ in $EG$. The face maps in bar notation are $[g_1|...|g_n]\mapsto [g_2|...|g_n]$ for $i=0$, $[g_1|...|g_n]\mapsto[g_1|...|g_ig_{i+1}|...g_n]$ for $0<i<n$ and $[g_1|...|g_n]\mapsto[g_1|...|g_{n-1}]$ for $i=n$ which are exactly the face maps for the nerve $N(G)$. The degeneracy maps in bar notation are exactly $[g_1|...|g_{n-1}] \mapsto [g_1|...|g_i|1|g_{i+1}|...|g_{n-1}]$ which is also exactly the same as for $N(G)$. Therefore $EG/G = |N(G)|$ (not just a homotopy equivalency, a very canonical homeomorphism)
Best Answer
I believe that's called the Milgram bar construction: