Reading this site for the last nearly two years I've learned a lot about LaTeX3 (which I had previously understood to be a defunct pipe dream) and LuaTeX (of which I was previously totally unaware). As I have said before, TeX itself is a bad programming language, though a good typesetting engine, but even as a typesetting engine, it is hobbled by its limitations as a language. What I mean by both of these, in short, is that it does not provide an easy way to manipulate its own representations of data. Others have also had this problem, and sometimes it can be solved with LuaTeX, which opens up a lot of the internals to the programming environment.
Now, as I understand it, LaTeX3 also provides a lot of programming tools within TeX, with the aim of making package creation and maintenance more robust. Joseph Wright has explained this in at least one place (and therefore probably many more). These are "algorithmic" tools, while the direction of LuaTeX development appears to be "data structure" tools. Of course, Lua is also a general-purpose programming language and as such provides perfectly good algorithmic tools as well.
My question:
Why use LaTeX3 programming over Lua, when LuaTex is available?
I understand that LuaTeX is very much experimental now, but then, so is LaTeX3. I also understand that as a result, LuaTeX is not necessarily used by anyone "ordinary", but then, it is intended as the ultimate successor to pdfTeX, which is now the default engine in the major TeX distributions. For all the reasons that TeX is a terrible programming language, LaTeX3 will still suffer from irritating expansion issues, lack of arithmetic capabilities, poor (or at least poor man's) variables, and so on. However, Knuth himself has written that he would have used a better language if one had been available. Now that "we" are inserting Lua into the TeX executable it appears that this alternative has been identified (ironically, that link is about PyTeX, based on Python).
Now that \directlua is feasible, I don't see why it is not better in every way simply to make all "function" macros (those that do not directly store text) hook into that one primitive and then do all their programming in Lua. Loops, variables, recursion, and so on can all be done in a comprehensible way. A lot of effort in the development of LaTeX3 appears to be in managing the programming language and its use. So, I ask again:
Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?
This suggests a related question to me:
Is the separation of LuaTeX and LaTeX3 a result (or at least an artifact) of the non-communication among developers that Ahmed Musa described in his comment to this answer? What kind of cooperation is there between these two projects to reduce duplication of effort?
(An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side. Then TeX itself would be a "macro package" on top of LuaTeX and one would not be bound to its strange design choices. An answer to why that should or should not be done might be too long for me to request, however.)
I don't personally think that this is too bloggy for this site, and I hope others agree. At least, given the confluence of expertise here, it seems to be the best place to ask the question.
I have accepted Frank Mittelbach's answer, though all the answers are extremely informative, because I think it hits all the main points most directly. One thing that neither it nor Joseph Wright's answer quite manages to come out and say directly is that ConTeXt does do the Lua-programming thing at least to some extent, and it does not solve all algorithmic problems because LuaTeX does not actually rewrite the underpinnings of TeX so much as stitch in a Lua layer that can interact with them.
I guess the one-line answer to my question is that my expectations of LuaTeX are much more grand than its reality, much like my "fantastical" idea.
Best Answer
(I guess I could be called a member of one of the teams ;-) this is my view)
I thought of staying out of this debate, but perhaps some words of clarification or, let's say, some thoughts are in order after all.
First of all this is the wrong question imho: LaTeX3 has different goals to LuaTeX and those goals may well be still a defunct pipe dream, but if so they are unlikely to be resolved by pure Lua either.
So if one wants to develop an argument along those lines then it should be more like "Why does LaTeX3 use an underlying programming language based on eTeX and not on LuaTeX where a lot of functionality would be available in a "simpler" way?"
But LaTeX3 is really about three to four different levels
See for example my talk at TUG 2011: http://www.latex-project.org/papers/
Here is a sketch of the architecture structure:
The chosen underlying engine (at the moment is any TeX engine with e-TeX extension). The programming level is what we call "expl3" and that is what I guess you are referring to if you say "LaTeX3" (and I sometimes do that too). However, it is only the bottom box in the above diagram. The more interesting parts are those that are above the programming level (and largely a pipe dream but moving nicely along now that the foundation on the programming level is stable). And for this part there is no comparison against Lua.
To build the upper layers it is extremely important to have a stable underlying base. As @egreg mentioned in chat: compare the package situation in 2.09 to the package situation in 2e. The moment there were standard protocols to build and interface packages the productivity increased a lot. However, the underlying programming level in 2e was and is still a mess which made a lot of things very complicated and often impossible to do in a reliable manner. Thus the need for a better underlying programming layer.
However, that programming layer is build on eTeX not because eTeX is the superior engine (it is compared to TeX but not with respect to other extensions, be it Lua or some other engine) but because it is a stable base available everywhere.
So eTeX + expl3 is a programming layer that the LaTeX3 team can rely on of not being further developed and changed (other than by us). It is also a base that is immediately available to everybody in the TeX world with the same level of functionality as all engines in use are implementing eTeX extensions.
Any larger level of modifications/improvements in the underlying engine is a hindrance to build the upper layers. True, some things may not work and some things may be more complicated to solve but the tasks we are looking at (well I am) the majority are very much independent of that layer anyway.
To make a few examples:
good algorithms for automatically doing complex page layout aren't there (as algorithms) so Lua will not help you here unless somebody comes up with such algorithms first.
something like "coffins", is thinking about approaching "design" and the importance here is how to think about it, not how to implement it (that comes second) -- see Is there no easier way to float objects and set margins? or LaTeX3 and pauper's coffins for examples
Having said that, the moment LuaTeX would be stable similar to eTeX (or a subset of LuaTeX at least) there might well good reasons for replacing the underlying engine and the program layer implementation. But it is not the focus (for now).
Could happen. But only if that "LuaTeX" would no longer be a moving target (because LaTeX3 on top would be moving target enough).
Side remark: @PatrickGundlach in his answer speculated that this answer that the LaTeX3 goal is backwards compatibility. Wrong. The same people that are coming down on you very strong about compatibility for LaTeX2e have a different mindset here. We do not believe that the interesting open questions that couldn't get resolved for 2e could be resolved in any form or shape with LaTeX3 in a document-compatible manner.
Input compatible: probably. But output compatibility for old documents, no chance if you want to get anything right.
But in any case, this is not an argument for or against implementing the ideas we are working on one day with a LuaTeX engine.
As I tried to explain, there is not much overlap in the first place. There is much more overlap in conceptual ideas on the level ConTeXt viz. LaTeX.
That brings us to a completely different level of discussion, namely is based on LuaTeX, or anything else for that matter, a completely different approach to a typesetting engine possible? That is a very interesting thought, but as @Patrick explained it isn't done with leaving TeX to do the typesetting and do everything else in a different language. So far such concepts have failed whether it was NTS or anything else because fundamentally (in my believe) we haven't yet grasped how to come up with a successful and different model for the TeX macro approach (as ugly as it might look in places).