[Tex/LaTex] What exactly does \@doendpe do

conditionalsenvironmentslatex-basemacrosparagraphs

Quoting from the LaTeX2e sources:

[\@endparenv, \@doendpe]

To suppress the paragraph indentation in text immediately following a
paragraph-making environment, \everypar is changed to remove the
space, and \par is redefined to restore \everypar. Instead of
redefining \par and \everpar, \@endparenv was changed to set
the @endpe switch, letting \end redefine \par and \everypar.

This allows paragraph-making environments to work right when called by
other environments. (Changed 27 Oct 86)

These are the relevant code lines (with line breaks added to improve readability):

\def\@endparenv{%
  \addpenalty\@endparpenalty
  \addvspace\@topsepadd
  \@endpetrue
}
\def\@doendpe{%
  \@endpetrue% <---?
  \def\par{%
    \@restorepar
    \everypar{}%
    \par
    \@endpefalse% <---?
  }%
  \everypar{%
    {\setbox\z@\lastbox}%
    \everypar{}%
    \@endpefalse% <---?
  }%
}

While I roughly comprehend \@doendpe and its \everypar trickery, I'm lost with regard to two details (marked with <---? in the above code):

  • As the definition of \begin includes \@endpefalse, why is it necessary to set \@endpefalse at the end of \@doendpe's \par/\everypar redefinitions?

  • Why is setting \@endpetrue at the start of \@doendpe necessary? Isn't \@doendpe only called if the @endpe switch is set to true?

I suspect that a possible answer involves the (to me) cryptical remark about "paragraph-making environments [working] right when called by other environments", but would be grateful for a full explanation.

Bonus question: If I were to define a custom list environment that at its end would set \@endpefalse and instead (with the appropriate \aftergroup) call a carbon copy of \@doendpe (say \@doendcustomenv), would the code lines marked wih <---? above be beneficial or superfluous in this copy?

Best Answer

For our purposes, \begin{#1} is roughly

\begingroup \@endpefalse \csname #1\endcsname

and \end{#1} is roughly

\csname end#1\endcsname\expandafter \endgroup \if@endpe \@doendpe \fi

Since \expandafter expands \if@endpe before the group ends, the value of the switch that is used is the one within the environment. If the \csname #1\endcsname part of a \begin{#1} features \@endpetrue, then \@doendpe will be expanded. However, it will be expanded outside the environment, where the switch has been returned to its former value.

Consider the following document. The environment parenvA suppresses the indentation afterwards. In the second case, \@endparenv is called within two groups, and causes \@doendpe to be called still within the parenvB environment. If the definition of \@doendpe did not have \@endpetrue, then it would simply define \par and \everypar within the parenvB environment. These definitions would disappear at the end of the group, and the next paragraph would be indented. So instead, \@doendpe makes sure to carry the truth of the switch with it beyond environment boundaries, until the next \par or \everypar.

\documentclass{article}
\makeatletter
\newenvironment{parenvA}{}{\par\@endparenv}
\newenvironment{parenvB}{}{}
\makeatother
\usepackage{lipsum}
\begin{document}
% case 1
\begin{parenvA}
  \lipsum[1]
\end{parenvA}
\lipsum[2]
% case 2
\begin{parenvB}
  \begin{parenvA}
    \lipsum[3]
  \end{parenvA}
\end{parenvB}
\lipsum[4]
% case 3
\begin{parenvB}
  \begin{parenvA}
    \lipsum[5]
  \end{parenvA}
  \lipsum[6]
\end{parenvB}
\lipsum[7]
\end{document}

Once this is established, both \par and \everypar must have \@endpefalse, otherwise the last paragraph in the example above would lose its indentation: indeed, the \@doendpe triggered just outside the parenvA environment contains \@endpetrue, but we don't want the indentation to be suppressed after that \end{parenvB}, because there has already been a paragraph.

Now why do we need to work with both \everypar and \par? Because there are actually two cases. If the user leaves no blank line after \end{parenvA}, then the first letter, inherently horizontal for TeX, causes \everypar to be inserted, and to eat the indentation (with \lastbox). However, if the user leaves a blank line, TeX's eyes replace that by a \par token, which resets the normal definitions of \par and \everypar, and do not eat any indentation (try adding some blank lines in the file above and see what happens).

For your bonus question, well, if you remove those lines, the indentation will only be removed if the next \par (blank line) or \everypar (triggered by horizontal material) is within the same group/environment. Not necessarily what you want.

Related Question