A category object internal to simplicial sets is the same as a Segal space in which the Segal conditions hold on the nose instead of merely up to weak equivalence. In other words, a category is something whose nerve has unique horn fillers instead of merely contractible spaces of fillers.
The above category objects generate a full sub-(relative category) of Rezk's relative category of complete Segal spaces. As I explain below, Barwick and Kan's work proves that the inclusion of this sub-(relative category) induces an equivalence of homotopy theories.
Barwick and Kan construct a nerve functor $N$ from small relative categories to simplicial spaces. The key point is that anything in the image of this nerve is a category object in the above sense.
Their nerve functor $N$ has a left adjoint $K$, but they also consider a second functor $M$ from simplicial spaces to relative categories. The functors $M$ and $N$ are inverse equivalences of homotopy theories in the sense that there is a zigzag of natural weak equivalences
$$NMX \rightarrow NKX \leftarrow X$$ for any simplicial space $X$,
and a natural weak equivalence
$$MNY \rightarrow Y$$
for any relative category $Y$.
If one restricts the domains of $K$ and $M$ to consist only of category objects, the above natural weak equivalences remain intact. Thus the Barwick+Kan homotopy theory of relative categories is equivalent to the theory of category objects in simplicial spaces.
The answer to Question (ii) is positive. That is to say, there is a weak equivalences between the following functors from Segal topological categories to quasicategories: the composition of the singular complex functor with the homotopy coherent nerve functor, and the realization of the singular complex.
Indeed, the first step in both functors is the same: we take the singular complex of a Segal topological category, which yields a Segal category.
Therefore, the problem reduces to establishing a weak equivalence between the homotopy coherent nerve functor N and the realization functor R,
both considered as functors from Segal categories to quasicategories.
Both functors are homotopy cocontinuous: the homotopy coherent nerve is a Quillen equivalence, and the realization functor by construction.
Since the quasicategory of quasicategories is a reflective localization of ∞-presheaves on Δ, it suffices to construct a weak equivalence between the restructions of N and R along the Yoneda embedding of Δ into Segal categories.
Indeed, both restrictions are weakly equivalent to the Yoneda embedding of Δ into quasicategories, by construction.
Question (i) is not formulated rigorously, but there are rigorously defined functors from quasicategories to Segal categories. For example, one can use the left adjoint of the homotopy coherent nerve functor, which tautologically provides a positive answer.
Other constructions can be obtained by passing from quasicategories to Segal spaces, and then to Segal categories using the constructions of Joyal–Tierney and Bergner. Bergner's book has details of these construction. To see that the resulting functor is indeed the inverse of the functors considered above, consider the two compositions, and show they are weakly equivalent to the corresponding identity functor by restricting along the Yoneda embedding and observing that the resulting restrictions are weakly equivalent by construction.
Best Answer
The most obvious approach is to consider simplicial $\def\Ai{{\sf A}_∞}\Ai$-categories, where $\Ai$ denotes a nonsymmetric operad in simplicial sets that is weakly equivalent to the terminal operad, i.e., the associative operad.
In such a structure, instead of the usual compositions we have maps $$\Ai(n)⨯C(X_{n-1},X_n)⨯⋯⨯C(X_0,X_1)→C(X_0,X_n)$$ that satisfy the usual identities for algebras over operads.
As it turns out, the resulting notion is in some sense equivalent to the usual simplicial categories: every simplicial $\Ai$-category can be rectified to a simplicial category by Proposition 9.2.4 in arXiv:1410.5675v3.
Other generalizations that extend simplicial categories using less obvious notions of coherence for composition include Segal categories and complete Segal spaces. All of these are also equivalent to simplicial categories. The book by Bergner (The Homotopy Theory of (∞,1)-Categories) contains an exposition of these equivalences.