I don't think I have a compelling answer to this question, but maybe some bits and pieces that will be helpful. One point is that all of the examples that you bring up are related to the first: simplicial sets can be used as a model for the homotopy theory of spaces. Pretty much any homotopy theory can be "described" in terms of the homotopy theory of spaces, just like any category can be "described" in terms of the category of sets (via the Yoneda embedding, for example). So if you've decided that "space" means simplicial set, then it's pretty natural to start thinking about presheaves of simplicial sets when you want to think about the homotopy theory of (pre)sheaves of spaces, as in motivic homotopy theory.
But that just brings us to the question "why use simplicial sets as a model for the homotopy theory of spaces"? It's certainly not the only model, and some alternatives have been listed in the other responses. Another alternative is more classical: the category of topological spaces can be used as a model for the homotopy theory of spaces. So, you might ask, why not develop the theory of the cotangent complex using topological commutative rings instead of simplicial commutative rings? There's no reason one couldn't do this; it's just less convenient than the alternative.
There are several things that make simplicial sets very convenient to work with.
1) The category of simplicial sets is very simple: it is described by presheaves on a category with not too many objects and not too many morphisms, so the data of a simplicial set is reasonably concrete and combinatorial. The category of topological spaces (say) is more complicated in comparison, due in part to pathologies in point-set topology which aren't really relevant to the study of homotopy theory.
2) The category of simplices is (op)-sifted. This is related to the concrete observation that the formation of geometric realizations of simplicial sets (or simplicial spaces) commutes with finite products. More generally it guarantees a nice connection between the homotopy theory of simplicial sets and the homotopy theory of bisimplicial sets, which is frequently very useful.
3) The Dold-Kan correspondence tells you that studying simplicial objects in an abelian category is equivalent to studying chain complexes in that abelian category (satisfying certain boundedness conditions). So if you're already convinced that chain complexes are a good way to do homological algebra, it's a short leap to deciding that simplicial objects
are a good way to do homological algebra in nonabelian settings. This also tells you that when you "abelianize" a simplicial construction, you're going to get a chain complex
(as in the story of the cotangent complex: Kahler differentials applied to a simplicial commutative ring yields a chain complex of abelian groups).
4) Simplicial objects arise very naturally in many situations. For example, if
U is a comonad on a category C (arising, say, from a pair of adjoint functors), then applying iterates of U to an object of C gives a simplicial object of C. This sort of thing comes up often when you want to study resolutions. For example, let C be the category of abelian groups, and let U be the comonad U(G) = free group generated by the elements of G
(associated to the adjunction {Groups} <-> {Sets} given by the forgetful functor,free functor). Then the simplicial object I just mentioned is the canonical resolution of any group by free groups. Since "resolutions" play an important role in homotopy theory, it's convenient to work with a model that plays nicely with the combinatorics of the category of simplices. (For example, if we apply the above procedure to a simplicial group, we would get a resolution which was a bisimplicial free group. We can then obtain a simplicial free group by passing to the diagonal (which is a reasonable thing to do by virtue of (2) )).
5) Simplicial sets are related to category theory: the nerve construction gives a fully faithful embedding from the category of small categories to the category of simplicial sets.
Suppose you're interested in higher category theory, and you adopt the position that
"space" = "higher-groupoid" = "higher category in which all morphisms are invertible". If you decide that you're going to model this notion of "space" via Kan complexes, then working with arbitrary simplicial sets gives you a setting where categories (via their nerves)
and higher groupoids (as Kan complexes) both sit naturally. This observation is the starting point for the theory of quasi-categories.
All these arguments really say is that simplicial objects are nice/convenient things to work with. They don't really prove that there couldn't be something nicer/more convenient. For this I'd just offer a sociological argument. The definition of a simplicial set is pretty simple (see (1)), and if there was a simpler definition that worked as well, I suspect that we would be using it already.
EDIT: Tom Goodwillie, in the comments, points out (interpreted in a quite charitable way) that there are two mistakes with the following argument. The $\pi_1$-obstruction does exist. However:
It misreads the question and assumes that there is some portion of an actual diagram which is homotopy commutative.
Even given that, it makes a mistake: it asserts that we can ignore the indeterminacy. This is wrong. For any space in the diagram, the space of maps to the terminal object $S^1$ is not simply-connected, but instead is homotopy equivalent to $S^1$. This allows you to simply erase the obstruction in $\pi_1$ by simply making different choices of homotopies.
As such, it's advisable that what's written below be demoted and I'll try to get a correct version later.
One of the classical obstructions to realizability is for cubical diagrams. Here, the category $I$ is the poset of subsets of $\{0,1,2\}$. Given such a diagram which is homotopy commutative, you can get twelve maps (one per edge in the cube) and six homotopies (one per face, which are well-defined up to "multiplication by an element in $\pi_1$") and the collection of all possible ways to compose maps, or compose maps with homotopies, gives rise to a hexagon in the space $Map(X,Z)$. Here $X$ is the image of initial object and $Z$ is the image of the terminal object in the cube. If the diagram is actually commutative then you can choose your homotopies to be trivial, and get a trivial hexagon; if it is equivalent to an honestly commutative diagram then you can choose your six homotopies on faces so that the hexagon in $Map(X,Z)$ can be filled in with a disc, or equivalently "represents the trivial element in $\pi_1$".
All scare quotes in the above are places where I'm neglecting basepoints. You have to be more careful in a real-world situation.
Here's an example where all the spaces in the diagram are contractible, except for the terminal one. This makes it easy to ignore the indeterminacy.
Let $Z$ be $S^1$, viewed as a quotient of $[0,1]$. We have six subsets of $Z$, which are the images of
$$
[0,1/3], [1/3, 2/3], [2/3, 1], \{0\}, \{1/3\}, \{2/3\}
$$
We get a corresponding cubical diagram in the homotopy category as follows. The space $S^1$ and its six subspaces define an honestly commutative diagram which is almost all of a commutative cube, except that it's missing the initial vertex. Let the initial vertex be a point $\ast$, which maps isomorphically to all three objects $\{0\}, \{1/3\}, \{2/3\}$.
(Sorry, I'm not really up to TeXing up the commutative diagram on MO today, it would be much easier to grasp.)
This diagram in the homotopy category is homotopy commutative. In fact, all the spaces are connected and the sources of nontrivial morphisms are contractible, so the diagram has no choice but to commute. The six homotopies all occur in contractible mapping spaces, so there is no $\pi_1$-indeterminacy. The hexagon maps to the mapping space $Map(\ast,S^1) \cong S^1$, and if you use the most obvious choices of homotopies then the hexagon maps to $S^1$ by a homotopy equivalence.
In this instance you can choose a witness to each commutativity diagram. The failure to rectify homotopy commutativity occurs here because your witnesses aren't telling stories that are compatible.
Best Answer
Most of the things that stop working are things related to procedures in commutative algebra that don't preserve exactness. The tensor product of simplicial modules always has to be the derived tensor product in order to be meaningful, etc.
One of the sticky points is that "free" is not the same as "polynomial" in higher degrees except in characteristic zero, and this makes building simplicial rings from a generators-and-relations perspective quite difficult.
For example, if R is an ordinary ring viewed as a constant simplicial commutative ring (with homotopy R, concentrated in degree 0), then you can take the free simplicial R-algebra on a class x in degree n; let me call it A. If n=0, the homotopy groups of A are just the polynomial algebra R[x] concentrated in degree zero. If n=1, the homotopy groups of A are an exterior algebra over R on a generator in degree 1. If n=2, the homotopy groups of A are a divided power algebra on a over R on a generator in degree 2 (a simplicial commutative ring always has a divided power structure on its higher homotopy groups). If n>3 and 2R = 0, then the answer is always a countable tensor product of exterior algebras over R. If n > 3 and pR = 0, you get algebras whose structure is simply complicated (because it involves iterated Tor starting with a divided power algebra).
Similar remarks apply to trying to construct an ideal generated by a collection of elements in homotopy groups which are anything other than a regular sequence in $\pi_0$ - you are unlikely to get the "quotient" you expect.