Today, TeX is a big program, LuaTeX, PDFTeX, METAPOST etc.
Where is the public accessing source code for hacking LuaTeX or other implementations. For example, I need to embed TeX into C program or creating Ruby object TeX.
[Tex/LaTex] Where is source code of TeX, LuaTeX, pdfTeX
tex-core
Related Solutions
pdfTeX is intended to offer complete compatibility with Knuth's TeX, and thus if the e-TeX extensions are not enabled should act in the same way.
XeTeX is based on the e-TeX code and does not set out to break any compatibility with Knuth's TeX unless it is absolutely necessary (i.e. there is no reimplementation of algorithms unless this relates to adding new features). However, there are places that differences occur. As noted in the question, XeTeX can load system fonts. When this is done, a new approach to placing boxes on the page is used. Possibly only those with deep involvement in that code can comment on whether it was absolutely necessary not to support the {} approach to breaking ligatures, but as noted that doesn't always work anyway. Changes also occur where the classical TeX syntax is extended. For example, allowing more than two ^ is done to allow access to the full Unicode range. However, as shown in http://wiki.contextgarden.net/Encodings_and_Regimes that leads to code which does different things in 8-bit and Unicode TeX engines:
\def\"{0}\expandafter\def\csname^^^^^00022\endcsname{1}
\ifnum\"=0 \message{tex82}\else\message{newstuff}\fi
LuaTeX is a very different case. The designers have decided to revisit a number of Knuth's decisions: the LuaTeX manual covers the detail (there is quite a bit). For example, the fact that {} does not inhibit a ligature is deliberate and relates to how LuaTeX process the input and represents it in internal data structures. LuaTeX treats hyphenation as a property of language not of font. As such, LuaTeX can hyphenate words using different fonts if the language does not change. As a result, hyphenation is governed by per-language primitives. (LuaTeX can also hyphenate the first word of a paragraph, which Knuth's TeX does not do and which is nowadays a 'feature'.) LuaTeX also has the same issues as XeTeX in terms of extending primitives for Unicode working: see the demo above for example.
Worth noting is that as LuaTeX supports callbacks functionality unchanged by the engine may be altered by Lua code. An obvious example is the \font primitive. This is not extended by the engine, but is by a Lua-based font loader: plain and LaTeX users share the same code here while ConTeXt has its own (related) loader.
For both XeTeX and LuaTeX it is worth noting that the extension of math mode to allow a number of additional Unicode math parameters to be used (all prefixed \Umath...) means that math mode spacing may change if these additional data points are available, principally when using a Unicode math font.
The bottom line from all of this is that if you have an 8-bit document written for pdfTeX, e-TeX or indeed TeX90 you should be able to use pdfTeX to process it unchanged. XeTeX will give the same result with almost all files of the same form assuming they don't contain any engine tests or similar, and assuming that the contain no driver-specific code (XeTeX uses the xdvipdfmx driver in all cases, pdfTeX may use dvips, dvipdfmx or direct PDF output). LuaTeX may change the behaviour of such documents, including but not limited to hyphenation, line breaking, ligature formation and so on.
Looking at the question purely in terms of primitives, we have to decide if we are comparing XeTeX and LuaTeX with TeX90, e-TeX or pdfTeX1.40. The question seems to be focussed on 'current' engines, so I will take pdfTeX 1.40 as the 'reference (it incorporates the e-TeX modifications to TeX90 plus a range of additional primitives). As noted in the part above, some behaviours are changes in XeTeX and LuaTeX. I'll note where possible any TeX90/e-TeX/pdfTeX variations which seem important in this context. Quite a bit of this information is available in the LuaTeX manual.
As XeTeX and LuaTeX allow Unicode input, and primitives which are followed by the <number> of a character are affected by the change:
\char\lccode\uccode\catcode\sfcode\efcode(LuaTeX-only: see below)\lpcode\rpcode\chardef
These all accept the full Unicode range (up to 0x10FFFF) with the newer engines: pdfTeX like e-TeX and TeX90 allows only the 8-bit range (maximum 0xFF).
LuaTeX extends the range of registers allowed beyond that of e-TeX. Thus while pdfTeX and XeTeX allow up to 32767 box, count, dimen, muskip, marks and toks registers, LuaTeX allows a 16-bit range (max is 65535). This affects the primitives
\count\dimen\skip\muskip\marks\toks\countdef\dimendef\skipdef\muskipdef\toksdef\box\unhbox\unvbox\copy\unhcopy\unvcopy\wd\ht\dp\setbox\vsplit
The \font primitive is extended by XeTeX to allow loading of system fonts with the syntax
\font⟨name⟩="⟨font identifier⟩⟨font options⟩:⟨font features⟩" ⟨TeX font options⟩
where the ⟨font identifier⟩ may be given in square brackets for a file name or without with a 'friendly' (system) name. This is not the case in LuaTeX: as noted above, LuaTeX is normally used with a Lua-based font loader which modifies the primitive via a callback.
LuaTeX allows file names to be given in braces as primitive sytnax, for example
\input{file name}
This affects the primitives
\font(note: this is purely to do with the file name of the font)\input\openin\openout
pdfTeX adds a number of primitives to e-TeX, some related to PDF creation, some for microtypography and some general utilities. As XeTeX is based directly on e-TeX and not on pdfTeX, it only features some of these where they have been ported across. Some of the primitive are also renamed as they are no PDF-related. Thus XeTeX includes the following concepts introduced by pdfTeX:
\lpcode\rpcode\pdfpageheight\pdfpagewidth\pdfsavepos\pdflastxpos\pdflastypos\ifincsname\ifprimitive(\ifpdfprimitivein pdfTeX)\primitive(\pdfprimitivein pdfTeX)\strcmp(\pdfstrcmpin pdfTeX`)\shellescape(\pdfshellescapein pdfTeX)\normaldeviate(TL'19 onward,\pdfnormaldeviatein pdfTeX)\uniformdeviate(TL'19 onward,\pdfuniformdeviatein pdfTeX)\randomseed(TL'19 onward,\pdfrandomseedin pdfTeX)\setrandomseed(TL'19 onward,\pdfsetrandomseedin pdfTeX)\elapsedtime(TL'19 onward,\pdfelapsedtimein pdfTeX)\resettimer(TL'19 onward,\pdfresettimerin pdfTeX)\filedump(TL'19 onward,\pdffiledumpin pdfTeX)\filemoddate(TL'19 onward,\pdffilemoddatein pdfTeX)\filesize(TL'19 onward,\pdffilesizein pdfTeX)\mdfivesum(TL'19 onward,\pdfmdfivesumin pdfTeX)
but not for example \efcode (as noted above), \pdfliteral or many others.
LuaTeX is based on pdfTeX and retains some of the primitives introduced there, renames some to remove 'pdf' and drops others. As well as primitives marked as experimental or deprecated in pdfTeX 1.40, LuaTeX also removes the primitives:
\pdfelapsedtime\pdfescapehex\pdfescapename\pdfescapestring\pdffiledump\pdffilemoddate\pdffilesize\pdflastmatch\pdfmatch\pdfmdfivesum\pdfresettimer\pdfshellescape\pdfstrcmp\pdfunescapehex
and provides
\primitive\ifprimitive\ifabsnum\ifabsdim
without 'pdf' in the name. It also moves all of the 'back end' concepts (to do with producing PDF output) to three new primitives which implement the functionality of the various PDF-related \pdf... primitives from pdfTeX.
Currently, XeTeX and pdfTeX use the 'TeX--XeT' model for right-to-left typesetting while LuaTeX uses one derived from Omega/Aleph. As such, it does not feature the primitives
\TeXXeTstate\beginR\beginL\endR\endL
(Note that there has been suggestion that XeTeX may at some stage move from TeX--XeT to the Omega model.)
LuaTeX also alters the behaviour of \endlinechar and \newlinechar: the maximum value is 127 while setting any value below zero stores -1.
Both XeTeX and LuaTeX add new primitives to TeX and the behaviour of these of course requires the appropriate engine. Note in particular that new primitives for Unicode math handling (\Umath...) are available in both engines. The also both feature \suppressfontnotfounderror.
The argument for \kern is a ⟨dimen⟩, so TeX first looks for a ⟨decimal number⟩ (which can be an integer in any admissible representation, or a fractional number in decimal notation, with period or comma for separating the fractional part) and then for a ⟨unit of measure⟩, that is, a pair of letters representing one of the accepted units (pt, cm, mm, in, dd, pc, cc, bp, sp, em, ex). Any number of spaces between the numbers and the units are ignored, an optional space can follow the unit (it is looked for with expansion). Any other pair of letters will trigger an error message
! Illegal unit of measure (pt inserted).
Thus \kern37pclike has a correct value for \kern, but \kern 37 pc like would have given exactly the same result.
Be careful with \kern, because it's a command that does different things if called in vertical or horizontal (or math) mode. Using \enspace before having started a paragraph will make you scratch your head.
The argument to \hskip should instead be a ⟨skip⟩; the same as before applies, but after the ⟨dimen⟩ stating the natural width, TeX looks for the keyword plus or minus (with expansion). If it finds it, it looks again for a ⟨dimen⟩ with the same rules as before, but the ⟨decimal number⟩ can also be followed by fil, fill or filll (not the real truth, but a good approximation) again with a trailing optional space that's looked for and ignored. Notwithstanding if plus is found or not, TeX looks for a minus keyword, same rules.
So \hskip 2pt minuscule chances of errors will trigger a puzzling error message.
The same as before is valid whenever TeX is looking for a ⟨dimen⟩ or a ⟨skip⟩. In case of doubts, particularly with a ⟨skip⟩, add \relax, which is what \setlength, \hspace and \vspace do in LaTeX.
Best Answer
TeX Live:
XeTeX:
LuaTeX
anonsvn/anonsvn)MetaPost
anonsvn/anonsvn)pdfTeX