Will has written a couple of short answers, I'll add a bit of detail.
There are several things that are wrong with LaTeX2e. Most obviously to end users, the kernel is rather inflexible. As a result, to get what you want you almost always need to load quite a number of packages, or do some low-level hacks of the kernel. That is not really ideal: I think most people would be happier not having to remember 'load package X to do ...' for every document they write. So the first challenge is to write a kernel which does many of the common tasks (such as those covered by KOMA-Script or memoir) without needing add-ons. The current aim is to include 'most of The LaTeX Companion': that is a lot of material.
The second challenge is that programming LaTeX2e is a mix of TeX, documented LaTeX, undocumented LaTeX and picking up stuff from packages. That is not a good thing: what is needed is documented system for programming. That is at least partly what expl3 is about: it provides a lot of lower-level programming material, although there is still lots to do.
The next things are 'big challenges' (we know how to do the two things above, it's just lots of work, whereas here I'm talking about things we have to work out how to do). Actually providing a way to make design and coding separate is not easy. The CSS model from HTML does not answer all of the issues we have in a typesetting system, so a new model is being developed (as xtemplate). There are then questions such as a better output routine, to allow things like grid typesetting, more complex column and float placement and so on. None of this is easy.
You also cannot ignore the fact that LaTeX is written by people in their spare time. There have been periods when the various people involved have had a lot of things on outside of LaTeX. It is pretty clear that some momentum was lost due to writing The LaTeX Companion, peoples jobs, other LaTeX packages, etc.
Put together, these issues mean that the project is very challenging. Looking at it today, both Will and I are active in trying to get the work done. We've both only joined the LaTeX Project relatively recently, and are putting in the serious effort that is needed. One of the aims of releasing material to CTAN as it gets to a stable state is to demonstrate that delivery is possible. For example, both expl3 and xparse are usable and do give real benefits.
One final point. LaTeX2e was released in 1994, and since then has acquired a lot of users. Even so, some people still use LaTeX2.09 concepts (such as \documentstyle). LaTeX3 will face a much bigger 'hill' than LaTeX2e did, and it will only succeed if the benefits are big enough to get a critical mass of users. To do that, we need to deliver on a lot of the issues with LaTeX2e. (See for example the 'there is nothing to improve' answer.)
There is always more that can be done, and I am focused on delivery (I don't want to load lots of support packages any more than anyone else does). One place to help out would be to work on documentation. We have some stuff, but we know that it needs improving. Feel free to drop the Project a line with offers of assistance or send a pull request to LaTeX3 GitHub repository!
(I guess I could be called a member of one of the teams ;-) this is my view)
I thought of staying out of this debate, but perhaps some words of clarification or, let's say, some thoughts are in order after all.
LaTeX3 versus pure Lua
First of all this is the wrong question imho: LaTeX3 has different goals to LuaTeX and those goals may well be still a defunct pipe dream, but if so they are unlikely to be resolved by pure Lua either.
So if one wants to develop an argument along those lines then it should be more like "Why does LaTeX3 use an underlying programming language based on eTeX and not on LuaTeX where a lot of functionality would be available in a "simpler" way?"
But LaTeX3 is really about three to four different levels
- underlying engine
- programming level
- typesetting element layer
- designer interface foundation layer
- document representation layer
See for example my talk at TUG 2011: http://www.latex-project.org/papers/
Here is a sketch of the architecture structure:
The chosen underlying engine (at the moment is any TeX engine with e-TeX extension). The programming level is what we call "expl3" and that is what I guess you are referring to if you say "LaTeX3" (and I sometimes do that too). However, it is only the bottom box in the above diagram. The more interesting parts are those that are above the programming level (and largely a pipe dream but moving nicely along now that the foundation on the programming level is stable). And for this part there is no comparison against Lua.
Why use LaTeX3 programming over Lua, when LuaTeX is available?
To build the upper layers it is extremely important to have a stable underlying base. As @egreg mentioned in chat: compare the package situation in 2.09 to the package situation in 2e. The moment there were standard protocols to build and interface packages the productivity increased a lot. However, the underlying programming level in 2e was and is still a mess which made a lot of things very complicated and often impossible to do in a reliable manner. Thus the need for a better underlying programming layer.
However, that programming layer is build on eTeX not because eTeX is the superior engine (it is compared to TeX but not with respect to other extensions, be it Lua or some other engine) but because it is a stable base available everywhere.
So eTeX + expl3 is a programming layer that the LaTeX3 team can rely on of not being further developed and changed (other than by us). It is also a base that is immediately available to everybody in the TeX world with the same level of functionality as all engines in use are implementing eTeX extensions.
Any larger level of modifications/improvements in the underlying engine is a hindrance to build the upper layers. True, some things may not work and some things may be more complicated to solve but the tasks we are looking at (well I am) the majority are very much independent of that layer anyway.
To make a few examples:
good algorithms for automatically doing complex page layout aren't there (as algorithms) so Lua will not help you here unless somebody comes up with such algorithms first.
something like "coffins", is thinking about approaching "design" and the importance here is how to think about it, not how to implement it (that comes second) -- see Is there no easier way to float objects and set margins? or LaTeX3 and pauper's coffins for examples
Having said that, the moment LuaTeX would be stable similar to eTeX (or a subset of LuaTeX at least) there might well good reasons for replacing the underlying engine and the program layer implementation. But it is not the focus (for now).
Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?
Could happen. But only if that "LuaTeX" would no longer be a moving target (because LaTeX3 on top would be moving target enough).
Side remark: @PatrickGundlach in his answer speculated that this
answer that the LaTeX3 goal is backwards compatibility. Wrong. The
same people that are coming down on you very strong about
compatibility for LaTeX2e have a different mindset here. We do not
believe that the interesting open questions that couldn't get resolved
for 2e could be resolved in any form or shape with LaTeX3 in a
document-compatible manner.
Input compatible: probably. But output compatibility for old
documents, no chance if you want to get anything right.
But in any case, this is not an argument for or against implementing the ideas we are working on one day with a LuaTeX engine.
Is the separation of LuaTeX and LaTeX3 a result (or at least an artifact) of the non-communication among developers that Ahmed Musa described in his comment to this answer? What kind of cooperation is there between these two projects to reduce duplication of effort?
As I tried to explain, there is not much overlap in the first place. There is much more overlap in conceptual ideas on the level ConTeXt viz. LaTeX.
An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side.
That brings us to a completely different level of discussion, namely is based on LuaTeX, or anything else for that matter, a completely different approach to a typesetting engine possible? That is a very interesting thought, but as @Patrick explained it isn't done with leaving TeX to do the typesetting and do everything else in a different language. So far such concepts have failed whether it was NTS or anything else because fundamentally (in my believe) we haven't yet grasped how to come up with a successful and different model for the TeX macro approach (as ugly as it might look in places).
Best Answer
Developments in Lua and LaTeX3 are separate, but there is some convergence in the sense that both address issues for LaTeX programmers. It's also the case that the benefits you will see from these developments depend on the nature of your documents.
LuaTeX adds to TeX functionality which is not available any other way, but also makes general programming a lot easier. The general programming stuff helps package authors. For example, Lua is used by
fontspecwhen loaded in LuaLaTeX to find fonts system. Some of the new abilities show up in particular areas, where TeX 'runs out of steam'. One that stands out is in Arabic typesetting, which is very context-dependent and which TeX does not do so well at. LuaTeX-based approaches can address this type of issue.LaTeX3 developments are also focussed at the programming end the moment making it easier to maintain packages. For example, XeLaTeX users tend to use
fontspec, andfontspecis written in the LaTeX3 programming system. In the longer term, the aim is to address a number of underlying issues with LaTeX in an eventual LaTeX3 format. This should wrap up a lot of package functionality kernel, and also allow us to address some long-standing awkward issues (for example, that adding vertical space can have odd effects due to the way TeX handles the main vertical list). A richer set of kernel tools should also cut down on multiple-package compatibilities. Now, it's possible that this will have very little impact on your documents, other than perhaps giving you shorter preambles.