I really like this question, I've been trying to sort out some of these ideas for a little while. I don't know the answer to your questions about conilpotence and twisting morphisms vs twisted arrows. I do have reason to believe that twisted arrows between A and C are the same as the twisted arrows from A to conil(C) but I don't know how to prove that.
I think Gabriel's answer is worth expanding on. Lurie is describing an adjunction between infinity categories: Alg and Coalg. The bar and cobar construction you mention are between categories---let me denote them by ALG and COALG [and I mean conilpotent coalg]---and so must be equipped with weak equivalences in order to induce functors on the infinity categories.
To model Alg, we equip ALG with quasi-isomorphisms. To model Coalg [or rather, conilCoalg], we must equip COALG with quasi-isomorphisms too. However, the classical bar and cobar construction are not homotopical between these relative (or model) categories.
However, we have a second class of weak equivalences on COALG---you called them fancy---that makes this adjunction into a Quillen pair, and as you point out, this Quillen pair is an equivalence. Gabriel's point, though, is that (COALG, fancy) left localizes to (COALG, quasi-iso). Conjugating this localization by the ``bar-cobar as equivalence between (ALG, quasi-iso) and (COALG, fancy)" will show you that this left adjoint from (COALG, fancy) to (COALG, quasi-iso) models Lurie's infinity left adjoint from Alg to Coalg, and it is defined by something that looks like the classical bar construction.
First of all, there are important differences between the notions of strict $n$-category, weak $n$-category, and $(\infty,n)$-category. The easiest notion is that of a strict $n$-category, and there's no doubt about the definition there: a strict $0$-category is a set, and by induction a strict $n$-category is a category enriched in the category of $(n-1)$-categories.
It's good that you cited Baez and Dolan's paper, which introduced an early model for the notion of a weak $n$-category. Between 1995 and 2001 there was a huge proliferation of other models. Morally, they should be categories weakly enriched in the category of weak $(n-1)$-categories, but there are many ways to define a weak enrichment, because there are many ways of keeping track of higher cells and how they combine. In 2004 there was a conference to try to get everyone together and figure out the commonalities between the models, and which were equivalent to which others. It did not result in one emerging as the "standard" model, and I don't think you should expect that to happen any time soon. However, we now know that models for weak $n$-categories broadly fall into two camps. Wikipedia says it nicely:
There are basically two classes of theories: those in which the higher cells and higher compositions are realized algebraically (most remarkably Michael Batanin's theory of weak higher categories) and those in which more topological models are used (e.g. a higher category as a simplicial set satisfying some universality properties).
Wikipedia also says "Several definitions have been given, and telling when they are equivalent, and in what sense, has become a new object of study in category theory." This matches my understanding of the field as it currently stands. I think of higher category theory as being interested in questions about the many models for weak $n$-categories. That's different from the study of $(\infty,n)$-categories, which is situated more in homotopy theory.
Now, others might come along and say "$(\infty,n)$-categories are the right thing" because MathOverflow has a larger representation of homotopy theorists than higher category theorists. You might get the same feel from reading the nLab, again based on who writes there. But if you go hang out in Sydney, Australia, where higher category theory is alive and well, you will not hear people saying $(\infty,n)$-categories are the "right" model or that the unicity theorem for $(\infty,n)$-categories solves the problem from 2004 of figuring out which models of weak $n$-categories are equivalent.
There is also plenty of ongoing work related to the stabilization hypothesis, tangle hypothesis, and cobordism hypothesis in various models of weak $n$-categories. For example, Batanin recently proved the stabilization hypothesis for Rezk's model based on $\Theta_n$-spaces. Then Batanin and I gave another proof that holds for a whole class of definitions of weak $n$-categories, including Rezk's model. Way back in 1998, Carlos Simpson proved the stabilization hypothesis for Tamsamani's definition of weak n-categories. Gepner and Haugseng proved the stabilization hypothesis for $(\infty,n)$-categories and the type of weak enrichment you'd get using Haugseng's PhD thesis (on enriched $\infty$-categories). Of course, famously, Lurie wrote thousands of pages towards proving the cobordism hypothesis for $(\infty,n)$-categories, and Ayala and Francis gave a shorter proof using factorization homology.
I'm sure there's lots of literature I missed, and I'm sure some will disagree with me in saying "yes, it is still valuable to study models of weak $n$-categories instead of only studying $(\infty,n)$-categories." But you asked for references so here are a bunch to get you started.
Best Answer
My answer would be that his insight was firstly that it pays to take what Grothendieck said in his various long manuscripts, extremely seriously and then to devote a very large amount of thought, time and effort. Many of the methods in HTT have been available from the 1980s and the importance of quasi-categories as a way to boost higher dimensional category theory was obvious to Boardman and Vogt even earlier. Lurie then put in an immense amount of work writing down in detail what the insights from that period looked like from a modern viewpoint.It worked as the progress since that time had provided tools ripe for making rapid progress on several linked problems. His work since HTT continues the momentum that he has built up.
As far as `abstracter than thou' goes, I believe that Grothendieck's ideas in Pursuing Stacks were not particularly abstract and Lurie's continuation of that trend is not either. Once you see that there are some good CONCRETE models for $\infty$-categories the geometry involved gets quite concrete as well. Simplicial sets are not particularly abstract things, although they can be a bit scary when you meet them first. Quasi-categories are then a simple-ish generalisation of categories, but where you can use both categorical insights and homotopy insights. That builds a good intuition about infinity categories... now bring in modern algebraic topology with spectra, etc becoming available.