[Tex/LaTex] It doesn’t hyphenate words ending with “—”

fontspechyphenationspacingspanishxetex

UPDATE

Well. After a while, I decided to see again about this “problem”, and just discovered that in pdfLaTeX, using \usepackage[utf8]{inputenc} and the — (the unicode —, not the ligature ---) it works perfect (at least in what I tried). Minor edit: as @cfr mentions in her answer (I forgot it), it is in fact possible to use --- (the ligature) if you use T1 encoding and \hyphenchar\font=\string"7F. In both methods (wether I use the em-dash — or ---) microtype works perfectly.

Now, the problem remains in XeLaTeX. I would like to clear my idea: I want the em-dash to behave correctly; it doesn't matter if I have to type — (the unicode em-dash —) or --- (the usual LaTeX way); it should (among others) hyphenate words correctly; be always together to the word; in case it's followed by a comma/period, they should be together; and, microtype should work, e.g., the hyphen should still hang in the margin.

Of course, there is a basic solution (which works in any engine): indicate to XeLaTeX the breaking points, for instance ocur\-recone\-stedoc\-umento---. But I'm looking for an automatic solution.

If you have anything to say, please, say it, it's welcomed!

Note: Every — you see in the code is an em-dash.

I'writing a paper, and XeLaTeX doesn't hyphenate words which end with – (traditional LaTeX ---). After reading the comments, I realized this is a common problem also in LaTeX (not only XeLaTeX). Here it is a minimal working example:

This code (full example at the bottom of the question) outputs

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

enter image description here

If we substitute the last — by a comma, for example it hyphenates correctly

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento, con algunas palabras más.

enter image description here

Moreover, if the phrase ends with —. XeLaTeX (or whoever is doing this) takes the full stop to the next line

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento—.

enter image description here

Here is a full minimal working example

\documentclass{scrartcl}
\usepackage[hmargin = 4cm]{geometry}
\usepackage{fontspec}

\begin{document}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

—Hola, esto es un texto absurdo —para ejemplificar lo que ocurreconestedocumento— con algunas palabras más.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

Any ideas?

Best Answer

You can prepare the following example:

\input ucode
\input lmfonts

\hsize=12cm

—Hola, esto es un texto absurdo —para ejemplificar lo que
ocurreconestedocumento— con algunas palabras más.

\end

and you can try to process it by 1) xetex test and 2) xetex -fmt pdfcsplain test. You will see different results: 1) the long word isn't hyphenated, 2) the long word is hyphenated.

You can study the difference in code settings, hyphenation settings etc. IMHO LuaLaTeX copies the settings from xetex's eplain, no from pdfcsplain. So, you have the problem.

Edit: Where is the difference? We can see the following setting in xetex.ini and in xelatex.ini:

 \XeTeXdashbreakstate=1

Bingo! Here is the problem. Set \XeTeXdashbreakstate=0 and your words will be hyphenated.

Related Solutions

[Tex/LaTex] Allow Latex to hyphenate words with dashes

Traditional typographical practice is to avoid hyphenating such words, so such a rule was coded by Knuth into the TeX program.

However, it is still possible to insert a hyphen that allows breaks, by fooling TeX into thinking we have two separate words (by inserting empty glue). This can be quite easily done if we're willing to type a special sequence like \Hyphdash or \-/ instead of a hyphen, which is what is provided by packages like extdash. But if we insist on typing regular hyphens into the TeX file, then “classical” TeX engines (meaning, other than LuaTeX) provide only one way of having regular hyphens be treated as something else, and that is to change the catcode of - to active, so that it becomes a macro that expands to something longer. But this would cause problems whenever - is used anywhere other than regular text, which is quite common (see examples of the problem in Steven's answer). I imagine that if a good solution existed for this in the classical TeX world, some package like extdash would have covered it.

(Actually, there's a perfectly sensible approach which for some reason is not very popular in the TeX world: and that is to preprocess your input file with some “smart” external script, replacing only the hyphens that occur in regular text with \Hyphdash (or its expansion) or whatever. But for some reason, people want to do everything inside TeX.)

With LuaTeX, which allows greater access to TeX internals, a lot of things become possible, and a lot of catcode and macro-related headaches are gone. For example, LuaTeX has a hyphenate callback which can be used to modify the regular hyphenation it does. This callback gets a list of nodes, into which we can insert discretionary nodes (hyphens) wherever we choose. This is much easier than having to do everything with macros (it is hard to set up appropriate input which later expands into exactly what we need, with no other interference). And there's even a lang.hyphenate function, which we can use to simply do what the regular hyphenation does.

The following seems to work: put the following in a file, say hyphenateall.lua:

function hyphenate_always(head, tail)
   local n = head
   while n do
      if node.type(n.id) == 'glyph' and n.char == string.byte('-') then
         -- Insert an infinite penalty before, and a zero-width glue node after, the hyphen.
         -- Like writing "\nobreak-\hspace{0pt}" or equivalently "\penalty10000-\hskip0pt"
         local p = node.new(node.id('penalty'))
         p.penalty = 10000
         head, p = node.insert_before(head, n, p)
         local g = node.new(node.id('glue'))
         head, g = node.insert_after(head, n, g)
         n = g
      end
      n = n.next
   end
   lang.hyphenate(head, tail)
end

luatexbase.add_to_callback('hyphenate', hyphenate_always, 'Hyphenate even words containing hyphens')

and just put \directlua{require('hyphenateall')} in your .tex file.

For example (borrowing the example from Steven's answer):

\documentclass{article}
% Not needed -- used in this example just to show it's not needed :-)
\newcommand\?{\nobreak-\hspace{0pt}}
\begin{document}

\begin{minipage}[t]{0in}
\hbox{Before:} often hungry Works: often\?hungry What about: often-hungry
\end{minipage}\hspace{1in}%
\directlua{require('hyphenateall')}%
\begin{minipage}[t]{0in}
 \hbox{After:} often hungry Works: often\?hungry What about: often-hungry
\end{minipage}

\end{document}

Note that after the \directlua call, we did not need the special syntax to get the extra hyphenation.

Hyphenate \texttt

You can exploit xurl.

\documentclass{article}
\usepackage{hyperref}
\usepackage{xurl}

\ExplSyntaxOn

\NewDocumentCommand{\clearurl}{m}
  {
    \tl_set:Nn \l__clearurl_url_tl {#1}
    \regex_replace_all:nnN {.*:\/\/(?:www.)?(.*[^\/])\/?} {\1} \l__clearurl_url_tl
    \clearurl_print:nV { #1 } \l__clearurl_url_tl
  }

\tl_new:N \l__clearurl_url_tl
\cs_new_protected:Nn \clearurl_print:nn
  {
    \href{#1}{\nolinkurl{#2}}
  }
\cs_generate_variant:Nn \clearurl_print:nn { nV }

\ExplSyntaxOff

\begin{document}

\clearurl{https://www.google.it/books/edition/Mirifici_Logarithmorum_Canonis_Construct/VukHAQAAIAAJ?hl=en&gbpv=1&dq=mirifici+logarithmorum&pg=PT1&printsec=frontcover}

\end{document}

UPDATE

Any ideas?

Best Answer

Related Solutions

[Tex/LaTex] Allow Latex to hyphenate words with dashes

Hyphenate \texttt

Related Question