TeX Core – Expandable Way to Differentiate a Character Token from an Equivalent Control Sequence in Plain TeX

conditionalsplain-textex-core

As the title says: is there any completely reliable, expandable method to differentiate between a control sequence that has been made equivalent to a character token and the character token itself? (Assuming the meaning of particular primitives was not changed: otherwise clearly all bets are off.) For example, consider the following (staying with plain TeX to keep superfluous elements to the minimum):

\font\ecrm=ecrm1000 \ecrm
\long\def\test#1#2{\hbox{%
    \vbox{\hsize=15em\escapechar=`\\ \tt\detokenize{#1}}%
    \vbox{#2}\par%
}}
\long\def\testexp#1{\test{#1}{#1}}
\long\def\testif#1{\test{#1}{#1\else not \fi equal}}
\long\def\exec#1{{\escapechar=`\\ \tt\detokenize{#1}}#1\par}

\begingroup
\exec{\let\^=^}
\exec{\escapechar=-1}
\bigskip
\testexp{\meaning^}
\testexp{\meaning\^}
\testexp{\string^}
\testexp{\string\^}
\testexp{\detokenize{^}}
\testexp{\detokenize{\^}}
\bigskip
\testif{\if^\^}
\testif{\if^\noexpand\^}
\testif{\ifcat^\^}
\testif{\ifcat^\noexpand\^}
\testif{\ifx^\^}
\bigskip
\exec{\def\tmpa{^}\def\tmpb{\^}}
\testif{\ifx\tmpa\tmpb}
\endgroup

\bye

enter image description here

(For convenience I do use the ε-TeX primitive \detokenize here, but that is not the primary point, hence the tex-core tag.)

I have found no way to make any of the conditionals recognize the difference between the two tokens: the actual character token and the control sequence. (I knew that these were not supposed to differentiate between them, but went and tested them just in case.) If I additionally assume that \escapechar is not printable, as is the case here then not even \meaning, \string, or \detokenize will treat them any differently.

That said, the TeX engine clearly does know the difference, as when these are made to be the replacement text of macros, \ifx recognizes them as not the same. Likewise when they are set as delimiters of macro arguments. However, if we want an expandable comparison of the two tokens as, say, the arguments of a macro, then this is not something we can do. Neither can we change the value of \escapechar: if it is not printable, we are stuck with it.

Is there a way to make the comparison in an expandable manner, one I have not thought of? If not in (e)TeX, then maybe in XeTeX or LuaTeX?

P.S. Another, somewhat related, weird corner case of tokens that I find are hard to tell apart: the control sequence with the empty name and — assuming, say, the normal escape character — the one with the name csname\endcsname (which is a situation where I really fail to comprehend TeX going out of its way change the rules for that one special case, messing up the injectivity of \string on macros). I know that LuaTeX’s \csstring does kind of solve that though.

Best Answer

David's solution expects that you will compare something with the fixed token ^. You cannot redefine the \testifhatx to something other at expansion level only. My solution works with two arbitrary tokens, not only with ^ and another one.

\long\def\testeq#1#2{%
    \ifx#1#2%
       \testsingleletter#1\iftrue
           \testsingleletter#2\iftrue TRUE%
           \else FALSE%
           \fi
       \else \testsingleletter#1\iftrue FALSE%
           \else TRUE%
           \fi
       \fi
    \else FALSE
    \fi
}
\long\def\testsingleletter#1{\expandafter\testsingleletterA\string#1\end}
\long\def\testsingleletterA#1#2\end{\ifx&#2&\else\expandafter\unless\fi}

\let \^=^

\testeq ^ ^   % prints TRUE

\testeq \a \^ % prints FALSE

\testeq ^ \^  % prints FALSE

\bye

Edit I created a second version, whereby you can process something like \fi in the parameter, as per your comment.

\long\def\splitif#1{#1\expandafter\ignoresecond\else\expandafter\usesecond\fi}
\long\def\usesecond#1#2{#2}  \long\def\ignoresecond#1#2{#1}

\long\def\isequal#1#2{\splitif {\ifx#1#2}%
   {\splitif {\testsinglechar#1\iftrue}%
      {\splitif{\testsinglechar#2\iftrue}{1}{0}}%
      {\splitif{\testsinglechar#2\iftrue}{0}{1}}}%
   {0}%
}
\long\def\testsinglechar#1{\expandafter\testsinglecharA\string#1\end}
\long\def\testsinglecharA#1#2\end{\ifx&#2&\else\expandafter\unless\fi}

\long\def\test #1#2{\ifnum\isequal#1#2>0 TRUE\else FALSE\fi}

\let \^=^

\test ^ ^   % prints TRUE

\test \a \^ % prints FALSE

\test ^ \^  % prints FALSE

\bye
Related Question