[Tex/LaTex] How to examine a token

expansionmacrostex-core

… let me count the ways.
I can test it with \if and \ifx
and \show its \meaning

And there I stop, both in my knowledge and my poetic ability (though some might argue that the latter never started).

Background: I'm still trying to get my head around the statement of what \noexpand does:

\noexpand temporarily sets the next token to \relax.

The purpose of setting it to \relax seems to be that since \relax is not expandable, the next token does not get expanded the next time that TeX tries to expand it. But it does get expanded the time after that, so that seems to be the meaning of "temporarily".

But is the fact that it is temporarily set to \relax detectable? To test this, I want to examine what TeX thinks \noexpand\foo is, but that "temporarily" keeps getting in the way: it's all a bit quantum since the act of observing a \noexpand\foo seems to change it in some way.

Anyway, so I want to build a test suite to see if the value of \noexpand\foo is detectable in a significant way and for that, I need to know all the ways that I can get at its value, such as conditionals and \show. And that's my question.

Question: What ways are there of examining a macro or active token in TeX which reveal something about what it points/expands to?

Best Answer

Hoping that I understand the question, the options are:

\show, which displays the meaning of a token in the terminal. For registers, this shows the register number rather than the content of the register. Primitives give you an identity, for example \show = \show, whereas for macros you get the expansion.
\showthe, which displays the content of a register at the terminal.
\meaning, which places very similar information in the input stream, with category codes for all tokens 'other' except for spaces. This is useful for both displaying the meaning of macros and for various tests and tokenization tricks. For example, LaTeX uses \meaning to sanitize category codes (although nowadays \scantokens might be a better choice.)
\ifx, which does a comparison based on the meaning of two tokens. The two do not have to be \let to one another, but do have to be identically-defined for the test to be true. So something like
```
\def\testa{stuff}
\def\testb{stuff}
\ifx\testa\testb
```
is true but
```
\def\testa{stuff}
\long\def\testb{stuff}
\ifx\testa\testb
```
is false.
\if, which tests if two unexpandable tokens are identical, ignoring category codes. The two tokens business means that
```
\def\test{aa}
\if\test\SomeOtherThing
```
will compare the two a characters here, and that \SomeOtherThing is part of the true branch.

Related Solutions

[Tex/LaTex] How to do \scantokens inside \edef without triggering “Runaway definition”

In short, yes. Try

\edef\foo{\scantokens{Test\noexpand}}

\everyeof={\noexpand}
\edef\foo{\scantokens{text}}

in which the \noexpand ‘hides’ the EOF marker by turning it into \relax.

You might like to take a look at the definition of expl3's \tl_rescan:nn and \tl_set_rescan:Nnn, both wrappers for \scantokens based on the ideas in Heiko Oberdiek's catchfile package.

Then you can write

\tl_set_rescan:Nnn \foo { «catcode setup» } { «rescanned tokens» }

\tl_rescan:nn { «catcode setup» } { «tokens to re-scan (in a group)» }

The latter cannot be used inside an \edef but can otherwise come in handy when you're processing a macro argument that has already been tokenised. In practise I find the set_rescan version a little more convenient.

[Tex/LaTex] Does \noexpand have to be a primitive

(Further updated answer, with a look deep down the Lion's Mouth in the last section.)

Joseph gave an answer (“No!”) to your actual question; I'll try to get your (and my) head around the “temporarily”. Let's start from TeX-by-Topic, bottom of page 129:

The \noexpand command is expandable, and its expansion is the following token. The meaning of that token is made temporarily equal to \relax, so that it cannot be expanded further.

At first I dismissed this as a colourful description, but one can easily see that TeX-by-Topic is right: The following short plain TeX file

\def\foo{bar}
\expandafter\show\noexpand\foo
\bye

prints

> \foo=\relax.
<recently read> \notexpanded: \foo

to the terminal, so \foo is indeed “made temporarily equal to \relax”. What I find very interesting is the \notexpanded: \foo; this \notexpanded isn't mentioned in the TeXbook or in TeX-by-Topic, and I only found one occurrence in Knuth's torture test for TeX. (Another test is to put \tracingcommands=2 \noexpand\foo into your TeX file; then you'll find a \relax in your log file.)

In the TeXbook there's a description that I find easier chewable than the one from TeX-by-Topic (since it doesn't contain “temporarily”): On page 214 it says

\noexpand⟨token⟩. The expansion is the token itself; but that token is interpreted as if its meaning were ‘\relax’ if it is a control sequence that would ordinarily be expanded by TeX’s expansion rules.

So what does “temporarily” mean?

In practise, “temporarily equal to \relax” means the following: When TeX expands \noexpand\foo, the result is \foo with the temporary meaning \relax, and as soon as TeX continues expanding, \foo gets back its original meaning. A typical use case of \noexpand, from Tex-by-Topic (page 130):

\edef\one{\def\noexpand\two{\the\prevdepth}}

Inside the \edef, TeX doesn't expand \two but continues with the next token { (which is also not expandable). The \noexpand is needed since otherwise TeX would try to expand \two. This would cause an error if \two is undefined, and it would cause much more trouble if \two has been defined before. (Before \def one doesn't need a \noexpand since \def is not expandable.) If you now \show the \one, then you see that the real \two, not a \relaxed one is inside \one:

> \one=macro:
->\def \two {-1000.0pt}.

What's happening behind the scenes?

Let me first say this in the picture of TeX's mouth and stomach. In general, TeX continues chewing on a token list in its mouth as long as the first token is expandable. When the first token is unexpandable, it's swallowed for further processing in the stomach. Now \noexpand indeed wraps the token following it in some protective coating. However, this coating is chewable; figure that it's made of sugar. Thus, when TeX chews on such a coated token, the coating is removed (and the token is expandable again), but TeX gets a sensory flash and thinks “Wow, this tastes like \relax, I want to swallow it.” And sure enough, if the now uncoated token is the first in TeX's mouth (think of “first” as “closest to the throat”), then it is swallowed.

Two examples (the 2nd one being rather academic):

You have, e.g.,
```
\edef\bar{\noexpand\foo\anothercs}
```
Then inside the \edef, \noexpand\foo expands to a \relaxed \foo. This is converted back to \foo, immediately swallowed, and TeX continues with expanding the next token \anothercs.
You expand \noexpand\foo twice, e.g. with
```
\def\foo{bar}
\expandafter\expandafter\expandafter\show\noexpand\foo
\bye
```
Then on the terminal you get
```
> \foo=macro:
->bar.
```
Thus, expanding \noexpand\foo once gives a \relaxed \foo (as seen above), and the expansion of the \relaxed \foo is again the original \foo. And this is not swallowed since it's not the first token!

Disclaimer: The above I found out by observation; it might be that some details are not entirely correct.

Some final remarks

Your attempt at defining \noexpand as macro fails for several reasons. One technical point is that \let#1\relax is not a good idea if #1 isn't a control sequence. Another point is that it's not really helpful to do \let#1\relax #1; this has the same effect as \relax as far as I can tell. (And as pointed out in the other answers, \let is not expandable.)