[Tex/LaTex] Using LuaLaTeX and SQLite3

luatex

I am trying to access SQLite databases from LuaLaTeX but am not able to load an installed lsqlite3 package. Here's what I tried so far:

Took a brandnew Xubuntu 14.04 installation
Installed a complete TeX Live 2014
Installed Luarocks (which installed Lua5.1 on the way)
Used Luarocks to install lsqlite3

uwe@luabuntu:/media/uwe$ luarocks list

Installed rocks:

lsqlite3
0.9.1-2 (installed) – /usr/local/lib/luarocks/rocks

Appended the package.path in the following TeX file:

\documentclass{article}    
\usepackage{luacode}

\begin{document}

\begin{luacode*}
  package.path="/usr/local/lib/luarocks/rocks;/usr/local/lib/luarocks/rocks/lsqlite3; 
  /usr/local/lib/lua/5.1/lsqlite3.so" .. package.path

  require("lsqlite3")
\end{luacode*}

Hello Lua!

\end{document}

The error I receive is the following:

! LuaTeX error [\directlua]:3: module 'lsqlite3' not found:
  no field package.preload['lsqlite3']
  [luatexbase.loader] Search failed
  [kpse lua searcher] file not found: 'lsqlite3'
  [kpse C searcher] file not found: 'lsqlite3'
  [oberdiek.luatex.kpse_module_loader]-eroux Search failed
stack traceback:
  [C]: in function 'require'
  [\directlua]:3: in main chunk.
\luacode@dbg@exec ...code@maybe@printdbg {#1} #1 }

l.9 \end{luacode*}

What must be set to have LuaLaTeX find the lsqlite3 package?

EDIT

I have tried to adjust everything following the comments below:

Adjusted /usr/local/texlive/2014/texmf.cnf
Put everything into the example

Still get the error about the missing lsqlite3 module with the following example

\documentclass{article}
\directlua{
require "lualoader"
}

\usepackage{fontspec}
\usepackage[english]{babel}
\begin{document}

\directlua{package.path='/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/home/uwe/.luarocks/share/lua/5.1/?.lua;/home/uwe/.luarocks/share/lua/5.1/?/init.lua;/usr/share/lua/5.1//?.lua;/usr/share/lua/5.1//?/init.lua;./?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua'
package.cpath='/usr/local/lib/lua/5.1/?.so;/home/uwe/.luarocks/lib/lua/5.1/?.so;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/lib/x86_64-linux-gnu/lua/5.1/?.so;/usr/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so'
require "lualoader"
}

\directlua{%
local sqlite3 = require("lsqlite3")
local db = sqlite3.open_memory()

db:exec[[
  CREATE TABLE test (id INTEGER PRIMARY KEY, content);

  INSERT INTO test VALUES (NULL, 'Hello World');
  INSERT INTO test VALUES (NULL, 'Hello Lua');
  INSERT INTO test VALUES (NULL, 'Hello Sqlite3')
]]

for row in db:nrows("SELECT * FROM test") do
  tex.print(row.id .. " : ".. row.content )
end

}
\end{document}

Best Answer

It is possible to modify the package loading mechanism in Lua. In the case of LuaTeX, the kpse library is used to load modules instead of the default mechanism that uses package.path and package.cpath. So setting these variables will not have any effect. But nothing prevents us from using both methods using a Lua module, lualoader.lua. Save the snippet below as a file in the directory containing your document:

-- lualoader.lua
-- this is copied from luatexbase.loader
local make_loader = function(path, pos, loadfunc)
  local default_loader = package.searchers[pos]
  local loader = function(name)
    local file, _ = package.searchpath(name,path)
    if not file then
      local msg = "\n\t[lualoader] Search failed"
      local ret = default_loader(name)
      if type(ret) == "string" then
        return msg ..ret
      elseif type(ret) == "nil" then
        return msg
      else
        return ret
      end
    end
    local loader,err = loadfunc(file, name)
    if not loader then
      return "\n\t[lualoader] Loading error:\n\t"..err
    end
    return loader
  end
  package.searchers[pos] = loader
end

local binary_loader = function(file, name)
  local symbol = name:gsub("%.","_")
  return package.loadlib(file, "luaopen_"..symbol)
end

make_loader(package.path,2,loadfile)
make_loader(package.cpath,3, binary_loader)

We use the function make_loader to insert a function searching package.path or package.cpath at a specified index in the package.searchers table. 2 for lua files, 3 for binary modules. We also need to use different functions to load a module depending on its type. We use loadfile for Lua files and slightly more complicated function for binary modules that uses package.loadlib.

As an example, we can try loading lsqlite3 from LuaTeX:

\documentclass{article}
\directlua{
require "lualoader"
}

\usepackage{fontspec}
\usepackage[english]{babel}
\begin{document}
\directlua{%
local sqlite3 = require("lsqlite3")

local db = sqlite3.open_memory()

db:exec[[
  CREATE TABLE test (id INTEGER PRIMARY KEY, content);

  INSERT INTO test VALUES (NULL, 'Hello World');
  INSERT INTO test VALUES (NULL, 'Hello Lua');
  INSERT INTO test VALUES (NULL, 'Hello Sqlite3')
]]

for row in db:nrows("SELECT * FROM test") do
  tex.print(row.id .. " : ".. row.content )
end

}
\end{document}

It is important to run LuaLaTeX with the --shell-escape flag, because binary libraries are blocked by default. With --shell-escape you can execute external programs and call binary libraries.

The example yields:

enter image description here

Edit:

As Uwe pointed out, luarocks as installed by some Linux package managers may not work correctly. I experienced this on Fedora, so I installed luarocks from source. Another solution may be to set package.path and package.cpath to the directories listed by the command

luarocks path

On my own system, this yields:

$ luarocks path
export LUA_PATH='/home/michal/.luarocks/share/lua/5.2/?.lua;/home/michal/.luarocks/share/lua/5.2/?/init.lua;/usr/local/share/lua/5.2/?.lua;/usr/local/share/lua/5.2/?/init.lua;/usr/share/lua/5.2/?.lua;/usr/share/lua/5.2/?/init.lua;/usr/lib/lua/5.2/?.lua;/usr/lib/lua/5.2/?/init.lua;./?.lua'
export LUA_CPATH='/home/michal/.luarocks/lib/lua/5.2/?.so;/usr/local/lib/lua/5.2/?.so;/usr/lib/lua/5.2/?.so;/usr/lib/lua/5.2/loadall.so;./?.so'

So a modified preamble would be:

\directlua{    
package.cpath = '/home/michal/.luarocks/lib/lua/5.2/?.so;/usr/local/lib/lua/5.2/?.so;/usr/lib/lua/5.2/?.so;/usr/lib/lua/5.2/loadall.so;./?.so'
package.path = '/home/michal/.luarocks/share/lua/5.2/?.lua;/home/michal/.luarocks/share/lua/5.2/?/init.lua;/usr/local/share/lua/5.2/?.lua;/usr/local/share/lua/5.2/?/init.lua;/usr/share/lua/5.2/?.lua;/usr/share/lua/5.2/?/init.lua;/usr/lib/lua/5.2/?.lua;/usr/lib/lua/5.2/?/init.lua;./?.lua'
require "lualoader"
}

Added

Thanks to Mico's for experimenting the issue. The only font that seems to give problem is "LastResort" that should be kept in the black list.

Doing

sudo nano /usr/local/texlive/2012/texmf-dist/tex/luatex/luaotfload/otfl-blacklist.cnf

and putting a % in front of each line after

% Mac OS X TTC fonts, this list need to be filtered out

followed by a call of

mkluatexfontdb

will make all .ttc fonts available for LuaLaTeX.

[Tex/LaTex] lualatex and string.format

With package luatextra you have already loaded package luacode. This package provides help with the dealing of special characters, see its documentation. In this case \luaexec together with \% helps. Inside \luaexec \% is a macro that expands to a harmless percent character. Remember, % is usually the comment character in TeX.

\documentclass[ngerman, pagesize]{scrartcl}
\usepackage{luatextra}

\begin{document}

\directlua{
CumCDP_2012 = 1.00E-6
ICumCDP_WIP_2012 = 2.00E-7
ICumCDP_R = 1.00E-7
ICumCDP = ICumCDP_WIP_2012 + ICumCDP_R
}

\begin{tabular}{cr}
Summe: & \directlua{tex.sprint(ICumCDP)}\\
Summe: & \luaexec{tex.sprint(string.format("\%e", 51.2))}\\             
\end{tabular}

\end{document}

Answer to the comment with `\%` in `\directlua`:

I expect that using \% within \directlua will not work any longer with the new LuTeX 0.74 with Lua 5.2. Taco Hoekwater (author of LuaTeX) wrote in the luatex mailing list in January 2013 with subject "About lua 5.2 changes":

But also, the string parser has changed:

Lua 5.1 understands the following backslash escapes:

    \a \b \f \n \r \t \v \\ \' \"
              (these all have their expected C meanings)
    \<LF>     (insert a line feed into the string)
    \<CR>     (internally converted to <LF>, then inserted)
    \<d[dd]>  (1-3 decimal digits, result range is 0-255)

Lua 5.2 adds:

    \z        (skips following whitespace in input)
    \x<xx>    (2 case-insensitive hex digits)

Anything else following a backslash is an error in Lua 5.2,
but was silently ignored in 5.1.

With the standard definition of \% in LaTeX (\chardef), the string would get the percent character with preceding backslash. In Lua 5.1 the backslash would be ignored. However in Lua 5.2 the percent character would be "anything else" causing an error now.

Best Answer

Related Solutions

[Tex/LaTex] using Hoefler Text font under lualatex and MacOSX

Added

[Tex/LaTex] lualatex and string.format

Answer to the comment with \% in \directlua:

Related Question

Answer to the comment with `\%` in `\directlua`: