I don't think I have a compelling answer to this question, but maybe some bits and pieces that will be helpful. One point is that all of the examples that you bring up are related to the first: simplicial sets can be used as a model for the homotopy theory of spaces. Pretty much any homotopy theory can be "described" in terms of the homotopy theory of spaces, just like any category can be "described" in terms of the category of sets (via the Yoneda embedding, for example). So if you've decided that "space" means simplicial set, then it's pretty natural to start thinking about presheaves of simplicial sets when you want to think about the homotopy theory of (pre)sheaves of spaces, as in motivic homotopy theory.
But that just brings us to the question "why use simplicial sets as a model for the homotopy theory of spaces"? It's certainly not the only model, and some alternatives have been listed in the other responses. Another alternative is more classical: the category of topological spaces can be used as a model for the homotopy theory of spaces. So, you might ask, why not develop the theory of the cotangent complex using topological commutative rings instead of simplicial commutative rings? There's no reason one couldn't do this; it's just less convenient than the alternative.
There are several things that make simplicial sets very convenient to work with.
1) The category of simplicial sets is very simple: it is described by presheaves on a category with not too many objects and not too many morphisms, so the data of a simplicial set is reasonably concrete and combinatorial. The category of topological spaces (say) is more complicated in comparison, due in part to pathologies in point-set topology which aren't really relevant to the study of homotopy theory.
2) The category of simplices is (op)-sifted. This is related to the concrete observation that the formation of geometric realizations of simplicial sets (or simplicial spaces) commutes with finite products. More generally it guarantees a nice connection between the homotopy theory of simplicial sets and the homotopy theory of bisimplicial sets, which is frequently very useful.
3) The Dold-Kan correspondence tells you that studying simplicial objects in an abelian category is equivalent to studying chain complexes in that abelian category (satisfying certain boundedness conditions). So if you're already convinced that chain complexes are a good way to do homological algebra, it's a short leap to deciding that simplicial objects
are a good way to do homological algebra in nonabelian settings. This also tells you that when you "abelianize" a simplicial construction, you're going to get a chain complex
(as in the story of the cotangent complex: Kahler differentials applied to a simplicial commutative ring yields a chain complex of abelian groups).
4) Simplicial objects arise very naturally in many situations. For example, if
U is a comonad on a category C (arising, say, from a pair of adjoint functors), then applying iterates of U to an object of C gives a simplicial object of C. This sort of thing comes up often when you want to study resolutions. For example, let C be the category of abelian groups, and let U be the comonad U(G) = free group generated by the elements of G
(associated to the adjunction {Groups} <-> {Sets} given by the forgetful functor,free functor). Then the simplicial object I just mentioned is the canonical resolution of any group by free groups. Since "resolutions" play an important role in homotopy theory, it's convenient to work with a model that plays nicely with the combinatorics of the category of simplices. (For example, if we apply the above procedure to a simplicial group, we would get a resolution which was a bisimplicial free group. We can then obtain a simplicial free group by passing to the diagonal (which is a reasonable thing to do by virtue of (2) )).
5) Simplicial sets are related to category theory: the nerve construction gives a fully faithful embedding from the category of small categories to the category of simplicial sets.
Suppose you're interested in higher category theory, and you adopt the position that
"space" = "higher-groupoid" = "higher category in which all morphisms are invertible". If you decide that you're going to model this notion of "space" via Kan complexes, then working with arbitrary simplicial sets gives you a setting where categories (via their nerves)
and higher groupoids (as Kan complexes) both sit naturally. This observation is the starting point for the theory of quasi-categories.
All these arguments really say is that simplicial objects are nice/convenient things to work with. They don't really prove that there couldn't be something nicer/more convenient. For this I'd just offer a sociological argument. The definition of a simplicial set is pretty simple (see (1)), and if there was a simpler definition that worked as well, I suspect that we would be using it already.
There is a standard answer to your question, unless I'm missing something, that I learned from the great survey by Toen and satisfies everything you want. However to satisfy your requirements - ie to get the correct Hochschild complex - you must (explicitly or implicitly) embed simplicial schemes into a larger world of derived algebraic geometry, ie change from rings to simplicial rings (cosimplicial affine schemes) or some variant (homological cdgas in characteristic zero for example).
The construction is "the obvious one" - given any finite simplicial set think of it as a constant functor on your category of affines and construct the mapping space into your given target $X$.
Then pass to functions on the mapping space $X^M$, ie (derived) global sections of the structure sheaf.
When $M=S^1$, then say $BG^{S^1}=G/G$ the adjoint quotient stack. For any (derived) ring $R$ we have
that functions on $Spec(R)^{S^1}$ is Hochschild chains of $R$ (very close to the definition).
Much stronger things are true about these loop spaces, eg for any perfect stack (a very large class of well behaved stacks with affine diagonal map, eg any quasicompact separated scheme) and any finite simplicial set we have $QC(X^M)\simeq QC(X)\otimes M$ ($QC$=quasicoherent sheaves).
$BSL_2$ is such a perfect stack (in characteristic not 2 I think) and $BSL_2^M$ is a great object to consider, with $QC(BSL_2^M)$ being functions on the derived stack of $SL_2$ local systems on your space $M$. Not sure what else you want to hold but this is a very robust nice functorial monoidal construction which has already seen quite a lot of applications.
Best Answer
No, this does not work in the non-commutative case. In general we have $HH(A)=A\otimes_{A\otimes A^{\mathrm{op}}} A$, and this is only a $k$-module, not an algebra. If $A$ is commutative, the tensor product happens to compute coproducts/pushouts of commutative $k$-algebras, and we have $HH(A)=\operatorname{colim}_{S^1}A=x_!(A)$ since $S^1=*\coprod_{*\sqcup *}*$, but in the non-commutative case the tensor product does not have such an interpretation.
To see that the group $S^1$ acts on $HH(A)$ in general, one can use the formalism of factorization homology, $HH(A)=\int_{S^1} A$, which makes the functoriality on $BS^1$ apparent. A more classical approach is to use the fact that the usual "Hochschild complex" extends to a cyclic $k$-module (a functor on Connes' cyclic category $\Lambda$), whose geometric realization acquires an action of $S^1$ due to the $\infty$-groupoid completion of $\Lambda$ being $BS^1$. A reference for the latter approach is Appendix B of the article by Nikolaus and Scholze: https://arxiv.org/pdf/1707.01799.pdf