What is the best way to learn to programming in LaTeX3, and where can I get the best information in your opinion. I know that does not sound like a specific question, but I want to deepen on LaTeX and do not know where to start.I've done my research on the web and I have found dozens sites referred to LaTeX3, but which of these are appropriate for someone who starts now?
[Tex/LaTex] Programming with LaTeX3
latex3
Related Solutions
(I guess I could be called a member of one of the teams ;-) this is my view)
I thought of staying out of this debate, but perhaps some words of clarification or, let's say, some thoughts are in order after all.
LaTeX3 versus pure Lua
First of all this is the wrong question imho: LaTeX3 has different goals to LuaTeX and those goals may well be still a defunct pipe dream, but if so they are unlikely to be resolved by pure Lua either.
So if one wants to develop an argument along those lines then it should be more like "Why does LaTeX3 use an underlying programming language based on eTeX and not on LuaTeX where a lot of functionality would be available in a "simpler" way?"
But LaTeX3 is really about three to four different levels
- underlying engine
- programming level
- typesetting element layer
- designer interface foundation layer
- document representation layer
See for example my talk at TUG 2011: http://www.latex-project.org/papers/
Here is a sketch of the architecture structure:
The chosen underlying engine (at the moment is any TeX engine with e-TeX extension). The programming level is what we call "expl3" and that is what I guess you are referring to if you say "LaTeX3" (and I sometimes do that too). However, it is only the bottom box in the above diagram. The more interesting parts are those that are above the programming level (and largely a pipe dream but moving nicely along now that the foundation on the programming level is stable). And for this part there is no comparison against Lua.
Why use LaTeX3 programming over Lua, when LuaTeX is available?
To build the upper layers it is extremely important to have a stable underlying base. As @egreg mentioned in chat: compare the package situation in 2.09 to the package situation in 2e. The moment there were standard protocols to build and interface packages the productivity increased a lot. However, the underlying programming level in 2e was and is still a mess which made a lot of things very complicated and often impossible to do in a reliable manner. Thus the need for a better underlying programming layer.
However, that programming layer is build on eTeX not because eTeX is the superior engine (it is compared to TeX but not with respect to other extensions, be it Lua or some other engine) but because it is a stable base available everywhere.
So eTeX + expl3 is a programming layer that the LaTeX3 team can rely on of not being further developed and changed (other than by us). It is also a base that is immediately available to everybody in the TeX world with the same level of functionality as all engines in use are implementing eTeX extensions.
Any larger level of modifications/improvements in the underlying engine is a hindrance to build the upper layers. True, some things may not work and some things may be more complicated to solve but the tasks we are looking at (well I am) the majority are very much independent of that layer anyway.
To make a few examples:
good algorithms for automatically doing complex page layout aren't there (as algorithms) so Lua will not help you here unless somebody comes up with such algorithms first.
something like "coffins", is thinking about approaching "design" and the importance here is how to think about it, not how to implement it (that comes second) -- see Is there no easier way to float objects and set margins? or LaTeX3 and pauper's coffins for examples
Having said that, the moment LuaTeX would be stable similar to eTeX (or a subset of LuaTeX at least) there might well good reasons for replacing the underlying engine and the program layer implementation. But it is not the focus (for now).
Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?
Could happen. But only if that "LuaTeX" would no longer be a moving target (because LaTeX3 on top would be moving target enough).
Side remark: @PatrickGundlach in his answer speculated that this answer that the LaTeX3 goal is backwards compatibility. Wrong. The same people that are coming down on you very strong about compatibility for LaTeX2e have a different mindset here. We do not believe that the interesting open questions that couldn't get resolved for 2e could be resolved in any form or shape with LaTeX3 in a document-compatible manner.
Input compatible: probably. But output compatibility for old documents, no chance if you want to get anything right.
But in any case, this is not an argument for or against implementing the ideas we are working on one day with a LuaTeX engine.
Is the separation of LuaTeX and LaTeX3 a result (or at least an artifact) of the non-communication among developers that Ahmed Musa described in his comment to this answer? What kind of cooperation is there between these two projects to reduce duplication of effort?
As I tried to explain, there is not much overlap in the first place. There is much more overlap in conceptual ideas on the level ConTeXt viz. LaTeX.
An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side.
That brings us to a completely different level of discussion, namely is based on LuaTeX, or anything else for that matter, a completely different approach to a typesetting engine possible? That is a very interesting thought, but as @Patrick explained it isn't done with leaving TeX to do the typesetting and do everything else in a different language. So far such concepts have failed whether it was NTS or anything else because fundamentally (in my believe) we haven't yet grasped how to come up with a successful and different model for the TeX macro approach (as ugly as it might look in places).
As I have tried to explain in answer to the question "LaTeX3 versus pure lua" my vision of LaTeX3 is really a system with multiple layers that provide interfaces for different kind of roles. These layers are
- the underlying engine (some TeX variant)
- the programming layer (the core language, i.e.,
expl3) - the typesetting foundation layer (providing higher-level concepts for typesetting)
- the typesetting element layer (templates for all type of document elements)
- the designer interface foundation layer
- the class designer layer (where instances of document elements with specific settings are defined)
- document representation layer (that provides the input syntax, i.e., how the author uses elements)
If you look at it from the perspective of user roles then there are at least three or four roles that you can clearly distinguish:
- The Programmer (template and functionality provider)
- The Document Type Designer (defines which elements are available; abstract syntax and semantics)
- The Designer (typography and layout)
- The Author (content)
As a consequence the LaTeX3 Project needs different kind of help depending on what layer or role we are looking at.
The "Author" is using, say, list structures by specifying something like \begin{itemize} \item in his documents. Or perhaps by writing <ul> ...</ul> or whatever the UI representation offers to him.
The "Document Type Designer" defines what kind of abstract document elements are available, and what attributes or arguments those elements provide at the author level. E.g., he may specify that a certain class of documents provides the display lists "enumerate", "itemize" and "description".
The "Programmer" on the other hand implements templates (that offer customizations) for such document elements, e.g., for display lists. What kind of customization possibilities should be provided by the "Programmer" is the domain of the "Document Designer", he drives what kind of flexibility he needs for the design. In most cases the "Document Designer" should be able to simply draw templates (already written) from a template library and only focus on the design, i.e., instantiating the templates with values so that the desired layout for "itemize" lists, etc. is created.
In real life a single person may end up playing more than one role, but it is important to recognise that all of them come with different requirements with respect to interfaces and functionality.
Programming Layer
The programm layer consists of a core language layer (called expl3 (EXP erimental L aTeX 3) for historical reasons and now we got stuck with it :-)) and two more components: the "Typesetting foundation layer" that we are currently working on and the "Typesetting Element Layer" that is going to provide customizable objects for the design layer. expl3 is in many parts already fairly complete and usable the other two are under construction.
Help is needed for this layer in
- helping by extending and completing the regression test suite for
expl3 - helping with providing good or better documentation
- possibly helping in coding additional core functionality but that requires in contrast to the first two points a good amount of commitment and experience with core language as otherwise the danger is too high the the final results end up being inconsistent
Once we are a bit further with the "Typesetting foundation layer" we would need help in
- providing higher-level functionality, perhaps rewriting existing packages/code for elements making use of extended possibilities
Two steps down the road (once the "Designer Layer" is closer to being finalized) we would need help with
- developing templates for all kind of elements
In summary for this part we need help from people interested in programming in TeX and expl3 and/or interested in providing documentation (but for this a thorough understanding of the programming concepts is necessary too).
Design Layer
The intention of the design layer is to provide interfaces that allow specifying layout and typography styles in a declarative way. On the implementation side there are a number of prototype implementations (most notably xtemplate and the recent reimplementation of the ldb). Those need to get unified into a common model which requires some more experimentation and proably also some further thoughts.
But the real importance of this layer is not the implementation of its interfaces but the conceptual view of it: provisioning a rich declarative method (or methods) to describe design and layout. I.e., enabling a designer to think not in programs but in visual representations and relationships.
So here is the big area where people can help that do not feel they want or can program TeX's bowls. What would be extremely helpful (and in fact not just for LaTeX3) would be
- collecting and classifying a huge set of layouts and designs
- designs for individual document elements (such as headings, TOCs, etc)
- document designs that include relationships between document elements
- thinking about declarative good ways to specify such designs
- what needs to be specified
- to what extend and with what flexibility
I believe that this is a huge task (but rewarding in itself) and already the first part of collecting existing design specifications will be a major undertaking and will need coordination and probably a lot of work. But it will be a huge asset to test any implementation and interfaces for this layer later on.
Document Interface Layer
If we get the separation done correctly, then this layer should effectively offer nothing more than a front end for parsing the document syntax and transforming it into an internal standarized form. This means that on this layer one should not see any (or not much) coding or computation.
It is envisioned that alternative document syntax models can be provided. At the moment we have a draft solution, the xparse package which offers a document syntax in the style of LaTeX2e, that is with *-forms, optional arguments in brackets, etc. but with a few more bells and whistles such as a more generalized concept of default values, support for addition delimiters for arguments and so on. It is fairly conventional though. In addition when it was written the clear separation of layers wasn't well-defined and so the package also contains components for for conditional programming that I don't any longer think should be there.
Bottom line on what is needed for this layer is to
- think about good syntax for providing document content from "the author" perspective
- think about good syntax for providing document content from an "application to typesetting" perspective, i.e., the syntax and structure for automated typesetting where the content is prepared by a system/application rather than by a human
The two might want need structures (in fact most likely need, as automation works much better with structures that do no have a lot of alternative possibilities ad shortcuts etc) and even if just looking at the human author a lot of open questions need answering. And these answers may or may not be to painfully stick with existing LaTeX2e conventions in all cases (or perhaps with any?).
None of this requires coding or expl3 experience. What it requires is familarity with existing input concepts, a feel for where the pain points are currently and the willingness to think and discuss how alternatives and extensions could look like.
In Summary
Basically help is possible on any level and it doesn't need to involve programming. Thoughts are sprincled thoughout the article above, but here are a few more high-lights:
- help with developening/improving the core programming layer by
- joining the effort to improve the test suite
- help improving the existing (or not existing) documentation
- joining the effort to produce core or auxilary code modules
- help on the design layer by
- collecting and classifying design tasks (what is needed ...)
- thinking and suggesting ways to describe layout requirements in a declarative manner
- help on shaping the document interface layer
The concepts, as well as their implementation, are under discussion on the list LATEX-L. You can join this list, which is intended solely for discussing ideas and concepts for future versions of LaTeX, by sending mail to listserv@URZ.UNI-HEIDELBERG.DE containing the line
SUBSCRIBE LATEX-L Your Name
Once you are subscribed you can post by email sending the post to LATEX-L@LISTSERV.UNI-HEIDELBERG.DE.
Gmane hosts a web interface, including archives, of the LATEX-L list: http://news.gmane.org/group/gmane.comp.tex.latex.latex3. If you are subscribed to LATEX-L as outlined above you can use this web interface to participate in discussions on the list, otherwise you can only read the entries made by others.
The list has only a fairly low traffic right now as actual implementation and development tasks are typically discussed directly between the few active implementors. But this might change if further people join.
And something else ...
The people on the LaTeX3 team are also committed to keep LaTeX2e stable and even while there isn't that much to do these days there is the need to resolve bug reports (if they concern the 2e core), provide new distributions once in a while, etc. All this is work that takes effort or remains undone or incomplete. Thus here too, it helps the LaTeX3 efforts if we get help to free up resources.
Most notably: we recently restarted an effort to get Babel for LaTeX2e back on track and there any help is welcome. If you want to join this effort get in touch with @JavierBezos or myself or contact us on LATEX-L.
Best Answer
Currently, we don't have a Programming in LaTeX3 book, and even if we did there would be lots of gaps. So there is no 'complete' solution. However, there are some resources. First, I'd point to the introduction to LaTeX3 ideas which gives a broad-brush overview of the concepts. I have written a short series of blog posts on the topic: I've summarised the 'story so far'. There is also the complete command reference, as well as course of the questions here tagged latex3 or expl3.