[Tex/LaTex] How to Bengali v.2 (bng2) be used to typeset Bengali with XeTeX

fontsfontspecindicopentypexetex

This is a follow-up to this question about typesetting Bengali.

As explained in both answers there, XeLaTeX can typeset Bengali given a suitable font using polyglossia and fontspec. For example,

\documentclass[a4paper]{article}
\usepackage{geometry,fontspec,polyglossia}
\setmainlanguage[variant=british]{english}
\setotherlanguage{bengali}
\newfontfamily\bengalifont{Noto Sans Bengali}[Script=Bengali]
\begin{document}
x, y, z whatever\dots

\begin{bengali}
  আমি
\end{bengali}
\end{document}

produces

[Note that XeTeX is required. LuaTeX does not give correct results.]

As I explain in my answer, the font I used in this example actually offers two different versions of Bengali script, which correspond to two different OpenType scripts for Bengali:

beng            Bengali
bng2            Bengali v.2

As shown above, using the first one is straightforward.

What about the second? As Arun Debray explains in discussion following his answer, there are posts elsewhere suggesting this is possible, but the example which would have shown how to realise this possibility is no longer available at the sign-posted location. That is, the trail goes cold at this point.

Hence, Arun Debray and I thought this question worth asking:

How should the second be used?

Disclaimer: I know nothing whatsoever about Bengali. I am told that the sample above is Bengali and that the output is correct. However, if I was not told this, it could as easily be Psyptizamen and I would never know the difference.

Though this question is about Bengali, several other scripts (e.g. Devanagari, Tamil) have two OpenType versions, so whatever difference there is between these is not specific to just Bengali.

Best Answer

This has been answered in the comments on the question by Arun Debray (also at the other question) and by Will Robertson, fontspec developer; just turning it into an "answer" to get this question off the unanswered list.

Briefly, in the command

\newfontfamily\bengalifont{Noto Sans Bengali}[Script=Bengali]

the Script=Bengali is just a convenience, part of fontspec's pre-defined mapping of common names to OpenType script tags. As documented in section "Defining new scripts and languages" of the fontspec manual, you can define your own scripts with \newfontscript.

Thus, if you wish, you can forget about the default fontspec-defined Script=Bengali and define your own explicitly:

\documentclass[a4paper]{article}
\usepackage{geometry,fontspec,polyglossia}
\setmainlanguage[variant=british]{english}
\setotherlanguage{bengali}
\newfontscript{BengaliOpenTypeOld}{beng}
\newfontscript{BengaliOpenTypeNew}{bng2}
\newfontfamily\bengalifont{Noto Sans Bengali}[Script=BengaliOpenTypeNew]
\begin{document}
x, y, z whatever\dots

\begin{bengali}
  আমি
\end{bengali}
\end{document}

and switch between Script=BengaliOpenTypeNew and Script=BengaliOpenTypeOld as you wish.

Aside: The rest of this answer is completely tangential, but somewhat related to the motivation for asking this question (looking at the other question): the reason XeTeX is required and LuaTeX does not give correct results (currently) is that XeTeX uses the system libraries—such as Harfbuzz—for complex text layout aka text shaping (glyph reordering, glyph positioning, etc.), while LuaTeX hopes to minimize external dependencies and implement everything in Lua code, and this (IMO highly ambitious) work has, at the moment, simply not been done for Indic scripts other than reasonable support for Devanagari script and some basic support for Malayalam script. (See font-odv.lua in ConTeXt source code.)

For example, the word "আমি" consists of three Unicode "characters" (codepoints) in this order:

where the glyph for the vowel-sign needs to be placed to the left of the consonant. This is done by Harfbuzz (or on Windows, possibly DirectWrite),

but in LuaTeX the glyphs are picked from the font and simply placed one after another meaninglessly:

Related Solutions

[Tex/LaTex] Towards a \ucshape and \textuc command for uppercase text (XeTeX, LuaTeX)

A full implementation in LuaTeX:

\documentclass{article}
\usepackage{fontspec}
\usepackage{xcolor}
\usepackage{luacode}
\usepackage{luatexbase}
\usepackage{luatexbase-attr}

\newluatexattribute\uppercaseattr


\begin{luacode*}
  local ucattr = luatexbase.attributes.uppercaseattr
  local GLYPH = node.id("glyph")
  local function makeuppercase(head)
      local orighead = head
      local string = unicode.utf8
      while head do
          if head.id == GLYPH then
              local att = node.has_attribute(head,ucattr)
              if att then
                  if head.char == 223 then -- ß
                      -- insert two 'S' glyphs
                      head.char = 83
                      orighead = node.insert_before(orighead,head,node.copy(head))
                  elseif head.char == 64258 then -- fl
                      head.char = 70 -- F
                      orighead = node.insert_before(orighead,head,node.copy(head))
                      head.char = 76 -- L
                  elseif head.char == 64256 then -- ff
                      head.char = 70 -- F
                      orighead = node.insert_before(orighead,head,node.copy(head))
                      head.char = 70 -- F
                  elseif head.char == 64257 then -- fi
                      head.char = 70 -- F
                      orighead = node.insert_before(orighead,head,node.copy(head))
                      head.char = 73 -- I
                  elseif head.char == 64259 then -- ffi
                      head.char = 70 -- F
                      orighead = node.insert_before(orighead,head,node.copy(head))
                      head.char = 70 -- F
                      orighead = node.insert_before(orighead,head,node.copy(head))
                      head.char = 73 -- I
                  elseif head.char == 64260 then -- ffl
                      head.char = 70 -- F
                      orighead = node.insert_before(orighead,head,node.copy(head))
                      head.char = 70 -- F
                      orighead = node.insert_before(orighead,head,node.copy(head))
                      head.char = 76 -- L
                  else
                      head.char = string.byte(string.upper(string.char(head.char)))
                  end
              end
          end
          head = head.next
      end
      return orighead
  end

  function makeuppercase_hbox(head,groupcode)
      local orighead = head
      if groupcode == "adjusted_hbox" or groupcode == "hbox" then
           makeuppercase(head)
      end
      return orighead
  end

  luatexbase.add_to_callback("hpack_filter",makeuppercase_hbox,"makeuppercasehbox")
  luatexbase.add_to_callback("pre_linebreak_filter",makeuppercase,"makeuppercase")
\end{luacode*}

\newcommand*\ucshape{\uppercaseattr=1}
\DeclareTextFontCommand{\textuc}{\ucshape}

\begin{document}
\hsize 7.2cm
\newcommand\sample{Draußen \i \ij\ ffl fluffiest fish \textit{König} \textcolor{blue}{àéîàáâãäåæ} çèéêëìíîï  ff \hbox{ðñòóôõö} ùúûüýþÿœš}


Lowercase {\ucshape \sample} Lowercase

Lowercase \textuc{\sample} Lowercase

% textcolor doesn't work in LaTeX's \MakeUppercase
% Lowercase \MakeUppercase{\sample} Lowercase


\end{document}

which yields

enter image description here

The program above just manipulates the nodelist iff the given attribute is set (to any value). Attributes are grouped, just like any TeX assignment.

It's a bit more complicated than I thought because we need to treat \hbox{}es separately.

[Tex/LaTex] Can we stop recommending fontspec as first choice in case lualatex is used

One reason to rely on fontspec over luainputenc comes from the package documentation of the second package. The abstract says simply:

Input encoding management for LuaTeX, needed only for compatibility with old documents. For new documents, using UTF-8 encoding and Unicode fonts is strongly recommended. You’ve been warned!

Then, the first section of the document, entitled "Overview: When (not) to use this package" reads:

This package is strictly meant for compatibility. It is usefull [sic] in the two (overlapping) following cases:

Your source is not encoded in UTF-8 and you don’t want to reencode it for some reason.

Your document is using legacy 8-bit fonts (with fontenc), as opposed to modern Unicode fonts (most probably with fontspec or luaotfload and fontenc with option EU2).

The same section then continues (in part):

luainputenc has several modes of operation. By default, it basically turns LuaTEX into an 8-bit engine, which means you loose half of the benefits from using LuaTEX.

In any event, I think it is fairly clear that, as the author of the package himself does not recommend using luainputenc for "new" documents, there is little reason for others to recommend it to other users. It does not seem to be recommended for the purpose you are using the package. (Which is perhaps fine: you have taken the time to explore what best suits your needs.)

Of course, one may choose to use this package over fontspec (I wish LuaTeX-based documents compiled more quickly, too!), but it seems a stretch to say that fontspec (or luaotfload) should not be recommended---especially to new users---before luainputenc. I am certainly not inclined to do so.

Best Answer

Related Solutions

[Tex/LaTex] Towards a \ucshape and \textuc command for uppercase text (XeTeX, LuaTeX)

[Tex/LaTex] Can we stop recommending fontspec as first choice in case lualatex is used

Related Question