[Tex/LaTex] Expandable full expansion of tokens that preserves catcodes

expansionpdftex

Is it possible to fully expand tokens in an expandable manner and preserves category codes? I'd like to do this just using pdfTeX.

I'm looking for something that would work like this:

\def\foo{foo}
\fullyexpand{\foo bar baz}

which would expand to the 10 tokens "foobar baz" and retain the category codes.

If we drop the expandable requirement, then this is easy.

\def\fullyexpand#1{\edef\fetemp{#1}\fetemp}

If we drop the retain catcode requirement, it's doable, but less straight-forward. Here's the best I could come up with.

\def\gobbleprefix#1#2\femarker{%
        \ifnum\escapechar<0
                #1%
        \else\ifnum\escapechar>255
                #1%
        \fi\fi
        #2%
}
\def\fullyexpand#1{%
        \expandafter\expandafter\expandafter\gobbleprefix
        \expandafter\string\csname#1\endcsname\femarker
}

I was hoping to combine the ε-TeX extension \scantokens with the above (by an appropriate, trivial modification of \gobbleprefix), but of course, that does not work. \scantokens's expansion is empty and pdfTeX just acts like it has opened a new file.

(As an aside, it amuses me that not expanding tokens is expandable—using \unexpanded—but expanding tokens isn't, at least not obviously so.)

Best Answer

Did you try using \romannumeral? This is used a lot for this type of thing (see for example the \exp_args:Nf concept in expl3):

\def\fullyexpand#1{\romannumeral - `0#1}

This works because TeX will keep expanding #1 looking for a number, which will always turn out to be negative, so the Roman numeral will vanish. Note that this solution will stop on the first non-expandable token, unlike an \edef which will keep going.

It's possible to build a function which can expand using \romannumeral 'around' unexpandable tokens. For example, the following code will work reasonably well:

\long\def\fullyexpand#1{%
  \csname donothing\fullyexpandauxi{#1}{}%
}
\long\def\fullyexpandauxi#1{%
  \expandafter\fullyexpandauxii\romannumeral -`0#1\fullyexpandend
}
\long\def\fullyexpandauxii#1#2\fullyexpandend#3{%
  \ifx\donothing#2\donothing
    \expandafter\fullyexpandend
  \else
    \expandafter\fullyexpandloop
  \fi
  {#1}{#2}{#3}%
}
\long\def\fullyexpandend#1#2#3{\endcsname#3#1}
\long\def\fullyexpandloop#1#2#3{%
  \fullyexpandauxi{#2}{#3#1}%
}
\def\donothing{}

However, this is not the same as \expanded, for a few reasons. First, my implementation will strip out spaces in the argument (as it does a loop, and TeX will skip spaces). Braces will also get stripped out. A bit of testing also reveals that \romannumeral will expand \protected functions here, whereas \expanded does not. I'd also note that the above code needs some guards adding for a blank (empty or all space) argument, as currently things fail in these cases.

With current release LuaTeX one can use \expanded, which does more-or-less the same as an \edef but is expandable (it doesn't required doubled # tokens also). This primitive will be in TeX Live 2019 pdfTeX/e-pTeX/e-upTeX, and hopefully in XeTeX (yet to be confirmed). As a precursor to this, expl3 has a macro-based emulation, slow but working, which does token-by-token examination and allows 'e-type' expansion.

On the aside, it is possible to use \scantokens expandably, but as you may have found this can be tricky and it is usually necessary to have a (non-expandable) change of \everyeof first. LuaTeX addresses this issue with the \scantextokens primitive, which combines this end-of-file stuff directly into the primitive. Of course, if you are using LuaTeX then the original problem is solvable anyway, since \expanded is available.

A fully expandable sanitizer

The following is an implementation of a \Sanitize command that:

Completely removes all control sequences and balanced braces in its argument.
Does not choke on nested braces.
Keeps spaces where they were requested either by " " or "\ " (for after macros).
Is fully expandable (i.e. can be put in \edef or \csname).

Edit: This is a revised version. My initial code had a few minor bugs that were a major pain to fix, and this is substantially rewritten. I think it's clearer, too.

How it works

There are three states: sanitizing spaces, sanitizing groups, and sanitizing tokens. We scan for "words" one at a time, then within each "word" look for groups that might be hiding spaces (TeX's macro scanner will only absorb delimited arguments with balanced braces). Finally, once we are satisfied that we are looking at genuinely contiguous tokens, we scan one at a time and throw out the ones that are control sequences, leaving only explicitly specified spaces (" " or "\ ").

From the inside out, the operation looks like this:

\SanitizeTokens is a big nested conditional that tests its argument against the various special cases. During the sweep for spaces, all space characters were converted to \SanitizedSpace tokens, and they are now converted to \RealSpaces. Both \SanitizedSpace and \SanitizeStop are macros that expand to themselves, and since they are private, this means that testing against them via \ifx is a reliable way to detect the exact control sequences (in the first version, these were \countdef tokens, which have the same property but are not quite as private).
\SanitizeGroups uses the tricky \def\SanitizeGroups#1#{ construction discussed in this question: Macros with # as the last parameter. It is the most legitimate such use I can imagine: its point is to detect groups, which you can't do using plain macro expansion in any other way. It guarantees that #1 has no groups in it, and since this comes after space elimination, it also has no spaces in it, so we can run \SanitizeTokens straight away. We then "enter" the group and go back to eliminating spaces.
\SanitizeSpaces uses pattern matching to grab the first chunk of text up until a space, excluding of course those spaces that are in groups. There is a technical trick here: every use of this macro has {} right after it, before the text. The point of that is so that the argument scanner doesn't remove braces around a group constituting an entire "word" between spaces. If that happens, then we will erroneously treat it as though it's been cleared of spaces when, in fact, it has not. (Any unsanitized spaces would be eaten by \SanitizeTokens because argument scanning ignores spaces.)
There are of course some cute utility macros. My favorite is \IfNoGapToStop, which is called like this: \IfNoGapToStop.X. \SanitizeStop, with X being the quantity potentially containing a gap. If it has none, then the first gap is the visible space after the period; if it has a gap, then the two periods are in different components, and both arguments of \IfNoGapToStop are nonempty.

Aside from the structural changes from the previous version, this one correctly preserves spaces at the boundaries of groups. (That version didn't explicitly scan for groups, but eliminated them as a side effect of absorbing tokens. That works, but it also makes it impossible to be sure when you are looking at a group, which may have spaces, rather than a single token.)

Oh, and of course: the algorithm is no longer stupid. The last version rescanned the entire initial portion of the text repeatedly while looking for words (the point of that was so as not to "lose" those tokens before sanitizing them). Now I crawl through the words one at a time, so there's no problem with abandoning each one when looking for the next. That turns a quadratic algorithm into a linear one.

This is not my preferred way of writing TeX anymore (for that, you should read this answer: How to write readable commands) but pgfkeys is really not the tool for this kind of textual parsing.

\documentclass{article}

\makeatletter
\newcommand\Sanitize[1]{%
 \SanitizeSpaces{}#1 \SanitizeStop
}

% This loops through and replaces all spaces (outside brace groups) with \SanitizedSpace's.
% Then it goes for the control sequences.
% All calls to this should put a {} right before the content, to inhibit the gobbling of braces
% if there is a group right at the beginning.
\def\SanitizeSpaces#1 #2\SanitizeStop{%
 \IfEmpty{#2}% Last word
  {\IfEmpty{#1}% No content at all
   {}% Nothing to do
   {\SanitizeGroups#1{\SanitizeStop}}%
  }%
  % No need for a trailing space anymore: there's already one from the initial call
  {\SanitizeGroups#1\SanitizedSpace{\SanitizeStop}\SanitizeSpaces{}#2\SanitizeStop}%
}

% Sanitize tokens up to the next group, then go back to doing spaces.
\def\SanitizeGroups#1#{%
 \SanitizeTokens#1\SanitizeStop
 \EnterGroup
}

% Sanitize the next group from the top.
\newcommand\EnterGroup[1]{%
 \ifx\SanitizeStop#1%
  \expandafter\@gobble
 \else
  \expandafter\@firstofone
 \fi
 {\SanitizeSpaces{}#1 \SanitizeStop\SanitizeGroups}%
}

\newcommand\SanitizeTokens[1]{%
 \ifx\SanitizeStop#1%
 \else
  \ifx\SanitizedSpace#1%
   \RealSpace
  \else
   \ifx\ #1%
    \RealSpace
   \else
    \if\relax\noexpand#1%
    \else
     #1%
    \fi
   \fi
  \fi
  \expandafter\SanitizeTokens
 \fi
}

% We use TeX's proclivity to eat braces even for delimited arguments to eat the braces if #1 
% happens to be just {}, which we put in.
% Even if we didn't put it in, {} is going to get thrown out when \SanitizeSpaces gets to it.
\newcommand\IfEmpty[1]{%
 \IfOneTokenToStop.#1\SanitizeStop
  {% #1 has at most space tokens
   % and thus is nonempty if and only if there is a gap:
   \IfNoGapToStop.#1. \SanitizeStop
  }
  {% #1 has non-space tokens
   \@secondoftwo
  }%
}

% Checks for a gap in #1, meaning #2 is nonempty
% This should only be used with \IfEmpty
\def\IfNoGapToStop#1 #2\SanitizeStop{%
 % It's enough to check for one token, since #2 is never just spaces
 \IfOneTokenToStop.#2\SanitizeStop
}

\def\IfOneTokenToStop#1#2{% From \IfEmpty, #1 is always a .
 \ifx\SanitizeStop#2%
  % If #2 is multi-token, the rest of it will fall in the one-token case and be passed over.
  % If not, well, that's what we asked for.
  \expandafter\@firstoftwo
 \else
  \expandafter\GobbleToStopAndSecond
 \fi
}

\def\GobbleToStopAndSecond#1\SanitizeStop{%
 \@secondoftwo
}
\makeatother

\def\SanitizeStop{\SanitizeStop}
\def\SanitizedSpace{\SanitizedSpace}
\def\RealSpace{ }

\begin{document}
\setlength\parindent{0pt}\tt

% Torture test
\edef\a{%
 \Sanitize{ Word1 \macro{Word2 Word3}{\macro\ Word4}{ Word5} {Word6 }{}Word7{ }{{Word8}} }
}\meaning\a

\a
\medskip

% Examples
\edef\a{%
 \Sanitize{\emph{This} sentence has \TeX\ macros and {grouping}. }
}\meaning\a

\a
\medskip

\edef\a{%
 \Sanitize{{A}{ gratuitously {nested} sentence {}{{with many} layers}}.}
}\meaning\a

\a
\medskip

\end{document}

[Tex/LaTex] A \typeout (or expandable command) that can expand once – or with variable levels of expansion

This has nothing to do with \typeout except insofar as its argument is fully expanded because of \write, as you noted. You could ask the same of \edef, and the answer would be to use a sufficiently elaborate combination of \expandafter, \noexpand, and \unexpanded to force each token to expand the desired number of times. For example:

*\def\a{\b} \def\b{\c} \def\c{c}

*\edef\x{\noexpand \a}

*\show\x 
> \x=macro:
->\a .
<*> \show\x

? 

*\edef\x{\expandafter\noexpand \a}

*\show\x
> \x=macro:
->\b .
<*> \show\x

? 

*\edef\x{\expandafter\expandafter\expandafter\noexpand \a}

*\show\x
> \x=macro:
->\c .
<*> \show\x

? 

*\edef\x{%
  \expandafter\expandafter\expandafter\expandafter
  \expandafter\expandafter\expandafter\noexpand\x}

*\show\x
> \x=macro:
->c.
<*> \show\x

The number of \expandafters increases exponentially. The question of how to write a single expandable command that expands the desired number of times is addressed here. Note that the answer given there still only expands one token, so you can't apply it to an entire text. It is possible to play tricks with \unexpanded (a \noexpand for many tokens) and with token list registers (which, inside \edef, expand to their contents and then make the result temporarily unexpandable) to achieve similar effects.

You might be wondering, if zero expansions and infinite expansions are allowed on an entire text, why not one or more? The answer is that in fact, "infinite" expansions are done token-by-token and there is no notion of expanding the "whole text". To do zero expansions TeX simply has to dump the token list as given; the way that \meaning apparently does one expansion is that it actually changes the \catcodes of its output so that it is no longer expandable: those aren't real macros you see. You can achieve the same effect yourself if you use \scantokens in combination with a \catcode change...except that \catcode is not expandable, so you can't do that in \edef or \write. Or rather, if you do, you will just see the string "\catcode" in the result.

Anyway, if you want to expand a "whole text" a certain number of times, you could mean one of three things:

Expand the first token, and then expand the first token of the result, and so on, that many times.
Expand the first expandable token, then the first expandable token of the result, and so on, that many times.
Expand the first expandable token; then, in the expanded text, expand everything (n - 1) times in the same way. Once you're done, move to the first "untouched" token and continue in this way to the end of the text. (Thus, expanding to a certain "depth" across the entire text.)

Rules 1 and 2 have different results:

\def\a{a}
a\a

where expanding "once" gives either "a\a" or "aa" depending on the rule. Rule 1 is relatively easy to implement using \multiexpandafter (from the other question) and \unexpanded. Rule 2 is probably impossible without embedding \expandafter and \noexpand into all your macros. Rule 3 is probably Turing-complete.

Best Answer

Related Solutions

[Tex/LaTex] Can one define an expandable command that removes control sequences from its argument

A fully expandable sanitizer

How it works

[Tex/LaTex] A \typeout (or expandable command) that can expand once – or with variable levels of expansion

Related Question