[Tex/LaTex] LuaLaTeX: How to use a \char directive inside a string.gsub function

luatexstrings

Consider the following MWE:

% !TEX TS-program = lualatex
\documentclass{article}

\usepackage{fontspec} 
\setmainfont[Ligatures=NoCommon]{Latin Modern Roman}

\usepackage{luacode,luatexbase} 
\begin{luacode}
function dosub ( s )
   s =  string.gsub ( s , 'ff', '\\char64256{}') 
   return ( s )
end
--luatexbase.add_to_callback ( "process_input_buffer", dosub, "dosub" )
\end{luacode}

\begin{document}
off \directlua{ tex.sprint ( dosub ( \luastring{off} ) ) } off
\end{document}

The heart of the code is the function dosub, which employs the Lua function string.gsub. It is set to replace instances of ff with the glyph that contains the ff ligature. (You'll have to trust me that, for the font at hand, the ff-ligature glyph is located in "slot" 64256.) Note that, for now, the instruction luatexbase.add_to_callback instruction is commented out. (A -- (double dash) string initiates a Lua comment.)

When this MWE is run, one gets:

Observe that the middle word, which is generated via a \directlua call to dosub, correctly contains the ff-ligature, whereas the first and third words do not (once again correctly, since automatic ligature generation is disabled).

The trouble starts when I uncomment the instruction

luatexbase.add_to_callback ( "process_input_buffer", dosub, "dosub" )

Upon recompiling, the following, fairly incomprehensible, error message results:

(/usr/local/texlive/2015/texmf-dist/tex/context/base/supp-pdf.mkii

[Loading MPS to PDF converter (version 2006.09.02).]

\scratchcounter=\count290

\scratchdimen=\dimen261

\scratchbox=\box256

! Missing number, treated as zero.

\let

l.275 \let

\pdflastform=\pdflastxform

?

I suspect this is somewhat related to the presence of a TeX macro — \char — in the replacement string part of the string.gsub function. To wit, if I replace '\\char64256{}' with gg (i.e., a constant string), no error message is generated (and the three instances of "ff" in the body of the document are automatically replaced with "gg").

Do I need to "wrap" or "protect" the TeX macro in some special way in order to enable the successful use of "luatexbase.add_to_callback"? Is there something else I should do? About my computing setup: I'm running MacTeX2015 (with all available updates thru this morning applied) on a MacBookPro running MacOSX 10.10.5 "Yosemite".

Best Answer

There is function unicode.utf8.char for direct unicode character inserting in Lua functions:

% !TEX TS-program = lualatex
\documentclass{article}

\usepackage{fontspec} 
\setmainfont[Ligatures=NoCommon]{Latin Modern Roman}

\usepackage{luacode,luatexbase} 
\begin{luacode}
local uchar = unicode.utf8.char
function dosub ( s )
   s =  string.gsub ( s , 'ff', uchar(64256)) 
   return ( s )
end
\end{luacode}

\AtBeginDocument{%
      \luaexec{luatexbase.add_to_callback ( "process_input_buffer", dosub, "dosub" )}%
    }

\begin{document}
off \directlua{ tex.sprint ( dosub ( \luastring{off} ) ) } off
\end{document}

But the main issue in your code is that the callback is inserted too early and it probably replaces ff chars in some macros loaded in \AtBeginDocument. So other solution is to insert the callback in \AtBeginDocument as well, which reduces the risk of such collision (you should do that even in the first method):

% !TEX TS-program = lualatex
\documentclass{article}

\usepackage{fontspec} 
\setmainfont[Ligatures=NoCommon]{Latin Modern Roman}

\usepackage{luacode,luatexbase} 
\begin{luacode}
function dosub ( s )
   s =  string.gsub ( s , 'ff', '\\char64256{}') 
   return ( s )
end
\end{luacode}
\AtBeginDocument{%
  \luaexec{luatexbase.add_to_callback ( "process_input_buffer", dosub, "dosub" )}%
}


\begin{document}
off \directlua{ tex.sprint ( dosub ( \luastring{off} ) ) } off
\end{document}

Edit:

There is also another catch, what if your document body include some macro with ff in a name? To fix that, we can use such function:

\begin{luacode}
local uchar = unicode.utf8.char

function dosub ( s )
  local x = s:gsub('(\\?)([%a%@]+)', function(back,text)
     if back~="" then 
        return back .. text  
     end
     return  text:gsub ( 'ff', uchar(64256)) 
  end)
  print("x", x)
  return x
end
luatexbase.add_to_callback ( "process_input_buffer", dosub, "dosub" )
\end{luacode}

with s:gsub('(\\?)([%a%@]+)', function(back,text) we catch all words, including macros. If variable back is not empty string, the current word is a macro and we need to return it unprocessed. Otherwise, we can apply ff replacing regexp.

Note that in this case add_to_callback is used without AtBeginDocument, because when \offer macro is defined in the preamble, it's text wouldn't be replaced. Because we now skip macros, it shouldn't matter.

And as closing remarks I would add that node processing callbacks are much better for this kind of hacks, exactly because of these problems with macros.

For instance the following code:

local uchar = unicode.utf8.char
local fchar = string.byte("f")
local glyph_id = node.id("glyph")
local glue_id = node.id("glue")


local function next_status(n, node_table)
  local node_table = node_table or {}
  table.insert(node_table, n)
  if not n then return false end
  if n.id == glyph_id and n.char == fchar then
    return true, node_table
  elseif n.id == glyph_id or n.id == glue_id then 
    return false
  else
    return next_status(n.next, node_table)
  end
end


local function node_dosub(nodes)
  for n in node.traverse(nodes) do
    if n.id == glyph_id and n.char == fchar then
      local next, node_table = next_status(n.next)
      if next == true then
        n.char =  64256
        for _, x in ipairs(node_table) do
          node.remove(nodes, x)
        end
      end
    end
  end
  return nodes
end

luatexbase.add_to_callback ( "pre_linebreak_filter", node_dosub, "node_dosub" )

it is more complicated, because we can't operate on string level, but on individual nodes. lot of node types exists, glyph nodes with node.id 37 are important for us. every glyph node has char field, holding the character code. When glyph with f character is found, we peek next nodes to find whether there is another f glyph next to this one. when it is found, we replace current character with code for ff ligature and delete next f glyph.

Related Solutions

[Tex/LaTex] Ploting a function using mplib in lualatex

This is how I would do it in ConTeXt. I don't know if similar functionality is available in a LaTeX package. Basically, I do the calculations in lua and write the result inside a \startMPcode .. \stopMPcode environment.

\startluacode
  require("math")
  local sin = math.sin
  local cos = math.cos 
  local pi  = math.pi 

  local function scale(x) 
    return x .. " * 1cm"
  end
  local function pair(x,y) 
    return "(" .. scale(x) .. "," .. scale(y) .. ")"
  end

  local function point(f,x)
    return pair(x, f(x))
  end

  userdata = userdata or {}

  function userdata.draw_function(f, a, b, n) 
    local step = (b-a)/n
    local path = {}
    local i = 0 
    for x = a, b, step do 
      path[i] = point(f,x) 
      i = i + 1
    end  
    path = table.concat(path, "--")

    context("draw " .. path .. " withpen pencircle scaled 2bp;")
  end

  function userdata.draw_x_axis(from, to)
      context(" drawarrow " .. pair(from,0) .. " -- " .. pair(to,0) .. "; ")
      context(" label.rt ( btex $x$ etex , " .. pair(to,0) .. ") ;")
  end

  function userdata.draw_y_axis(from, to)
      context(" drawarrow " .. pair(0, from) .. " -- " .. pair(0,to) .. "; ")
      context(" label.top ( btex $y$ etex , " .. pair (0,to) .. ") ;")
  end

  function userdata.plot()

    userdata.draw_x_axis(-2, 6.5) 
    userdata.draw_y_axis(-2.2, 2.2)


    local f = function (x) 
      return sin(2*x) + cos(x)
    end

    userdata.draw_function(f, 0, 2*pi, 40)
  end

\stopluacode

\starttext
  \ctxlua
    {context.startMPcode() ;
      userdata.plot() ;
     context.stopMPcode() ; }
\stoptext

However, this assumes that everything that is written inside teh userdata.plot() function is MP code. If you also want that function to print stuff to the TeX stream, then you can store all the MP code inside a \startMPdrawing and \stopMPdrawing environment and retrieve them using \getMPdrawing.

\startluacode
  require("math")
  local sin = math.sin
  local cos = math.cos 
  local pi  = math.pi 

  local function scale(x) 
    return x .. " * 1cm"
  end
  local function pair(x,y) 
    return "(" .. scale(x) .. "," .. scale(y) .. ")"
  end

  local function point(f,x)
    return pair(x, f(x))
  end

  userdata = userdata or {}

  function userdata.draw_function(f, a, b, n) 
    local step = (b-a)/n
    local path = {}
    local i = 0 
    for x = a, b, step do 
      path[i] = point(f,x) 
      i = i + 1
    end  
    path = table.concat(path, "--")

  context.startMPdrawing()
    context("draw " .. path .. " withpen pencircle scaled 2bp;")
  context.stopMPdrawing()
  end

  function userdata.draw_x_axis(from, to)
    context.startMPdrawing()
      context(" drawarrow " .. pair(from,0) .. " -- " .. pair(to,0) .. "; ")
      context(" label.rt ( btex $x$ etex , " .. pair(to,0) .. ") ;")
    context.stopMPdrawing()
  end

  function userdata.draw_y_axis(from, to)
    context.startMPdrawing()
      context(" drawarrow " .. pair(0, from) .. " -- " .. pair(0,to) .. "; ")
      context(" label.top ( btex $y$ etex , " .. pair (0,to) .. ") ;")
    context.stopMPdrawing()
  end

  function userdata.plot()
    userdata.draw_x_axis(-2, 6.5) 
    userdata.draw_y_axis(-2.2, 2.2)


    local f = function (x) 
      return sin(2*x) + cos(x)
    end

    userdata.draw_function(f, 0, 2*pi, 40)
  end

\stopluacode

\starttext
  \ctxlua
    { context.resetMPdrawing() ;
      userdata.plot() ;
      context.MPdrawingdonetrue() ;
      context.getMPdrawing() ; }
\stoptext

[Tex/LaTex] How to suppress the operation of a luatex-defined macro on a string if the string is part of macro or a label

Here is my solution to this problem, which also uses the ligaturing callback (reusing lots of code from the earlier answer).

Instead of attempting to do the actual hyphenation in the processing function, my code one inserts whatsit nodes at the key spots. Those whatsit nodes then prohibit ligature building at those spots.

\documentclass{article}
\usepackage{fontspec}
\usepackage{luacode}%,luatexbase}
\setmainfont[Renderer=Basic]{Latin Modern Roman}
%\defaultfontfeatures{Ligatures={TeX,NoCommon}}
%\setmainfont{Linux Libertine O}
\usepackage[margin=1cm]{geometry}
\begin{luacode}
local glyph = node.id('glyph')
local glue = node.id("glue")
local whatsit = node.id("whatsit")
local userdefined
for n,v in pairs(node.whatsits()) do
  if v == 'user_defined' then userdefined = n end
end
local identifier = 123456  -- any unique identifier 
local noliga={}
debug=false
function debug_info(s)
  if debug then
    texio.write_nl(s)
  end
end
local blocknode = node.new(whatsit, userdefined)
blocknode.type = 100
blocknode.user_id = identifier

function process_ligatures(nodes,tail)
  local s={}
  local current_node=nodes--node.copy(nodes)
  local build_liga_table =  function(strlen,t)
    local p={}
    for i = 1, strlen do
      p[i]=0
    end
    for k,v in pairs(t) do
      debug_info("Match: "..v[3])
      local c= string.find(noliga[v[3]],"|")
      local correction=1
      while c~=nil do
         debug_info("Position "..(v[1]+c))
         p[v[1]+c-correction] = 1
         c = string.find(noliga[v[3]],"|",c+1)  
         correction=correction+1
      end   
    end
    debug_info("Liga table: "..table.concat(p, ""))
    return p
  end
  local apply_ligatures=function(head,ligatures)
     local i=1
     local hh=head
     local last=node.tail(head)
     for curr in node.traverse_id(glyph,head) do
       if ligatures[i]==1 then
         debug_info("Current glyph: "..unicode.utf8.char(curr.char))
         node.insert_before(hh,curr, node.copy(blocknode))
         hh=curr
       end 
       last=curr
       if i==#ligatures then 
         debug_info("Leave node list on position: "..i)
         break 
       end
       i=i+1
     end
     if(last~=nil) then
       debug_info("Last char: "..unicode.utf8.char(last.char))
     end--]]
  end
  for t in node.traverse(nodes) do
    if t.id==glyph then
      s[#s+1]=string.lower(unicode.utf8.char(t.char))
    elseif t.id== glue then
      local f=string.gsub(table.concat(s,""),"[\\?!,\\.]+","") -- add all interpunction
      local throwliga={}    
      for k, v in pairs(noliga) do
        local count=1
        local match= string.find(f,k)
        while match do
          count=match
          debug_info("pattern match: "..f .." - "..k)  
          local n = match + string.len(k)-1
          table.insert(throwliga,{match,n,k})
          match= string.find(f,k,count+1)
        end
      end
      if #throwliga==0 then 
        debug_info("No ligature substitution for: "..f)  
      else
        debug_info("Do ligature substitution for: "..f)  
        local ligabreaks=build_liga_table(f:len(),throwliga)
        apply_ligatures(current_node,ligabreaks)
      end
      s={}
      current_node=t
    end    
  end
  -- node.ligaturing(nodes) -- not needed, luaotfload does ligaturing
end
function suppress_liga(s,t)
  noliga[s]=t
end
function drop_special_nodes (nodes,tail)
  for t in node.traverse(nodes) do
     if t.id == whatsit and t.subtype == userdefined and t.user_id == identifier then
        node.remove(nodes,t)
        node.free(t)
     end
  end
end
luatexbase.add_to_callback("ligaturing", process_ligatures,"Filter ligatures", 1) 
--luatexbase.add_to_callback("ligaturing", drop_special_nodes,"Drop filter ligatures", 2) 
\end{luacode}
\newcommand\suppressligature[2]{
\directlua{
    suppress_liga("\luatexluaescapestring{#1}","\luatexluaescapestring{#2}")
}
}
\newcommand\debugon{%
\directlua{
debug=true
}
}
\begin{document}

\suppressligature{fifi}{f|ifi}
\suppressligature{grafi}{graf|i}
\suppressligature{lfful}{lf|ful} 
\suppressligature{fflink}{ff|link}
\suppressligature{iflich}{if|lich}
\suppressligature{uflauf}{uf|lauf}
\suppressligature{ufform}{uf|form}
\debugon

shelfful 
cufflink
unbegreiflich 
Auflaufform
offen

\end{document}

As you can see, the code does not do any ligaturing at all (!) as that is handled by luaotfload in the pre_linebreak_filter.

However, this also creates a minor glitch: the added whatsits also prevent kerning at those spots, but they cannot be removed here because that would re-enable the ligatures once luaotfload comes into play. I do not know enough of the internals of lualatex to fix this (minor) problem.

Best Answer

Related Solutions

[Tex/LaTex] Ploting a function using mplib in lualatex

[Tex/LaTex] How to suppress the operation of a luatex-defined macro on a string if the string is part of macro or a label

Related Question