[Tex/LaTex] How to do literate programming in TeX

literate-programming

How can I do literate programming in TeX where TeX, or some variant, is both the used programming language and what is used for typesetting the document? Is there more than one way to do it or different approaches for different TeX variants? I would appreciate some examples.

Best Answer

Given that the question is on how to do literate programming for TeX code then one answer as already mentioned in the comment by David is using doc or xdoc package for documentation and docstrip.tex for generating for generating runable code.

The doc/docstrip combination is not the best possible way to do things (there are better approaches that could be imagined), but it offers one immense advantage, which probably explains why a large proportion of packages these days are documented in this manner:

It doesn't require any additional program at the user end and therefore works whereever a TeX system of any flavor is implemented.
It also provides the possibility to generate variants of your code from the same sources, e.g., a production version and a version with additional tracing code.

On the downside

It requires the code to be structured sequencially (at least unless you want to work artificially with a number of output files).
It has some somewhat strange conventions (as I see it these days, even if I'm the one put them in). One is that the documentation is hidden behind % signs. The initial reasoning was that you could use such a file directly as code without any processing (the docstrip program was added later.
Another one is that the code has to be surrounded by
```
%    \begin{macrocode}
 ... code ...
%    \end{macrocode}
```
where the last line has to be %followed by exactly 4 spaces followed by \end{macrocode}.

However, it served fairly well as it seems. Documentation is found as part of the LaTeX distribution.

For documentation look at doc.pdf and docstrip.pdf in the LaTeX distribution and for examples of use take a look at any .dtx file in a TeX installation.

Related Solutions

[Tex/LaTex] How to create step by step annotated programming tutorial in LaTeX

An interactive version with beamer may be constructed along the following lines:

\documentclass{beamer}

\begin{document}

\begin{frame}[fragile]%
\begin{semiverbatim}
\frametitle{Input \&\ Output}
\#include <cstdio>
\only<1>{(1)?}\only<2->{int main() \{}
   \only<3>{(3)?}\only<4->{long a,b,c;}
   \only<4>{(4)?}\only<5->{std::scanf("\%ld\%ld", \&a, \&b);}
   \only<5>{(5)?}\only<6->{c = a * b;}
   \only<6>{(6)?}\only<7->{std::printf("\%ld\textbackslash n", c);}
   \only<2>{(2)?}\only<3->{return 0;}
 \}
\end{semiverbatim}
\end{frame}

\end{document}

enter image description here

[Tex/LaTex] Different approach to literate programming for LaTeX

As discussed in earlier in the comments, it is quite possible to solve the problem of defining classes with this method. Granted, as you said, it takes two runs, but it is possible to make a document that will deal with the writing (and some minimal printing) on the first run, and that will process class-defined commands on the second run only (if correctly marked up, obviously).

In order to perform this check, I rely on the existence (or lack thereof) of an .aux file, because it is both easy – most IDEs can delete it in one click, it allows for overwrites, etc. – and user friendly in case of problem – as I expect most users to try to delete this file first if something goes wrong.

The only requirement is that we store the code in a class file instead of a package. The following MWE implements this solution, and also a macro-based solution to the problem of multiple files (you should write one file after the other, not mix them up haphazardly). Due to the fact that I am using comma-separated lists for the macros, it might make things easier to index, e.g. adding an optional argument for the type of thing created (macro, environment, counter, etc.).

The .cls file:

\NeedsTeXFormat{LaTeX2e}
\ProvidesClass{literate}

\newif\if@firstrun
    \@firstruntrue
\newif\if@iscsname
    \@iscsnamefalse
\newcounter{c@copyright}

\IfFileExists{\jobname.aux}
    {\IfFileExists{\jobname.cls}
        {\LoadClass{\jobname}}{}
      \IfFileExists{\jobname.sty}
        {\RequirePackage{\jobname}}{}
      \@firstrunfalse}
    {\LoadClass{book}}

\newcommand{\filetype}[1]{%
    \if@firstrun
        \begingroup\lst@EndWriteFile
    \fi
    \def\usefiletype{#1}
    \listadd{\file@list}{\listingshead\lstinputlisting[style=plain]{\jobname.#1}\addvspace{\baselineskip}}
    \setcounter{c@copyright}{0}
}

\newcommand{\listingshead}{%
    \section{Listings for \jobname.\usefiletype}%
    The file contains the following code:
}

\RequirePackage{etoolbox,xstring,xcolor,listings,verbatim,docmute,makeidx}
\lst@RequireAspects{writefile}
\makeindex

\newcommand{\numbercolor}{\color[HTML]{655643}}% Ash
\newcommand{\macrocolor}{\color[HTML]{78948D}}% Teal
\newcommand{\commentcolor}{\color[HTML]{BF4D28}}% Saffron
\newcommand{\mathcolor}{\textcolor[HTML]{CF872E}}% Ochre
\newcommand{\emphcolor}{\color{cyan}}
\newcommand{\identifiercolor}{\color{black}}
\newcommand{\bracketcolor}{\color{black}}

\lstset{
    language=[AlLaTeX]TeX,
    breaklines=true,
    columns=spaceflexible,
    emptylines=0,
    breakindent=1em,
    tabsize=4,
    basicstyle=\ttfamily,
    texcsstyle=*\macrocolor,
    commentstyle=\commentcolor\itshape,
    identifierstyle=\identifiercolor,
    morestring=[b]$,
    stringstyle=\mathcolor,
}

\lstdefinestyle{blockcode}{
    frame=leftline,
    framerule=.2ex,
    rulecolor=\numbercolor,
    numbers=left,
    numberstyle=\footnotesize\numbercolor,
    firstnumber=auto,
}

\lstdefinestyle{samplecode}{
    numbers=left,
    numberfirstline=true,
    stepnumber=10000,
    firstnumber=1,
    numberstyle=\numbercolor\makebox[1em][c]{\Large$\star$}\@gobble,
}

\lstdefinestyle{macrocode}{
    numbers=left,
    numberfirstline=true,
    stepnumber=10000,
    firstnumber=1,
    numberstyle=\numbercolor\makebox[1em][c]{\P}\@gobble,
    identifierstyle=\identifiercolor\itshape,
    literate=   {\{}{{\bracketcolor\{}{\identifiercolor$\langle$}}{2}
            {\}}{{\identifiercolor$\,\rangle$}{\bracketcolor\}}}{2}
            {[}{{\bracketcolor[}{\identifiercolor$\langle$}}{2}
            {]}{{\identifiercolor$\,\rangle$}{\bracketcolor]}}{2}
            {<}{{\identifiercolor$\langle$}}{1}
            {>}{{\identifiercolor$\,\rangle$}}{1},
}

\lstdefinestyle{inlinecode}{
    literate=   {\{}{{\bracketcolor\{}}{1}
            {\}}{{\bracketcolor\}}}{1}
            {[}{{\bracketcolor[}}{1}
            {]}{{\bracketcolor]}}{1},
}

\lstdefinestyle{plain}{
    style=blockcode,
    texcsstyle=*\color{black},
    emphstyle=\color{black},
    commentstyle={\color{black}\itshape},
    stringstyle=\color{black},  
}

\newcommand{\addcs}[1]{\lstset{moretexcs={#1},}}

% Create commands for globally adding indexation patterns. By default, it is assumed to be a control sequence
\newcommand{\defaultindexdef}{}
\newcommand{\addindexdef}{}

\newcommand{\setdefaultindex}[3]{%
    \def\defaultindexdef
        {\@iscsnametrue
          \expandafter\ifstrempty{#1}
            {\def\macro@type{}}
            {\def\macro@type{#1}}
          \expandafter\ifstrempty{#2}{}{\def\macro@format{#2}}
          \expandafter\ifstrempty{#3}{}{\def\macro@index{#3}}}
}
\newcommand{\addtoindex}{\@ifstar{\s@addtoindex}{\@addtoindex}}
\newcommand{\@addtoindex}[4]{%
    \apptocmd{\addindexdef}
        {\expandafter\ifstrequal\expandafter{\macrotype}{#1}
            {\@iscsnamefalse
              \expandafter\ifstrempty{#2}
                {\def\macro@type{}}
                {\def\macro@type{#2}}
              \expandafter\ifstrempty{#3}{}{\def\macro@format{#3}}
              \expandafter\ifstrempty{#4}{}{\def\macro@index{#4}}}
            {}}
        {}{}
}
\newcommand{\s@addtoindex}[4]{%
    \apptocmd{\addindexdef}
        {\expandafter\ifstrequal\expandafter{\macrotype}{#1}
            {\@iscsnametrue
              \expandafter\ifstrempty{#2}{}{\def\macro@type{#2}}
              \expandafter\ifstrempty{#3}{}{\def\macro@format{#3}}
              \expandafter\ifstrempty{#4}{}{\def\macro@index{#4}}}
            {}}
        {}{}
}

% Set default index
\setdefaultindex
    {\macrotype}
    {\macroname}
    {\entryname @\string\texttt{{\string\textbackslash}\entryname}\space(\entrytype)}

% Create a command for setting the macro type based on the optional argument to code
\newcommand{\set@macro@type}[2]{
    % Create helper macros for text
    \def\macrotype{#1}
    \def\macroname{\if@iscsname\textbackslash\fi#2}
    % Create helper macros for indexes
    \StrSubstitute{#2}{@}{"@}[\entryname]
    \def\entrytype{\macro@type}
    % Default definitions
    \defaultindexdef
    % Specific type definitions
    \ifstrequal{#1}{m}
        {\@iscsnametrue
          \def\macro@type{}
          \def\macro@index{%
            \entryname @\string\texttt{{\string\textbackslash}\entryname}}}{}
    \ifstrequal{#1}{l}
        {\@iscsnametrue
          \def\macro@type{length}}{}
    \ifstrequal{#1}{d}
        {\@iscsnametrue
          \def\macro@type{dimension}}{}
    \ifstrequal{#1}{e}
        {\@iscsnametrue
          \def\macro@type{environment}
          \def\macro@format{\textbackslash#2\par\textbackslash end#2}}{}
    \ifstrequal{#1}{f}
        {\@iscsnametrue
          \def\macro@type{float}
          \def\macro@format{\textbackslash#2\par\textbackslash end#2}}{}
    \ifstrequal{#1}{p}
        {\@iscsnamefalse
          \def\macro@type{package}
          \def\macro@index{%
            \entryname @\string\texttt{\entryname}\entrytype}}{}
    \ifstrequal{#1}{o}
        {\@iscsnamefalse
          \def\macro@type{option}
          \def\macro@index{%
            \entryname @\string\texttt{\entryname}\entrytype}}{}
    \ifstrequal{#1}{c}
        {\@iscsnamefalse
          \def\macro@type{counter}
          \def\macro@index{%
            \entryname @\string\texttt{\string\textit{\entryname}}\space(\entrytype)}}{}
    % Append user created macros
    \addindexdef
}


\newcommand{\docindex}[2][]{%
    \ifstrempty{#1}
        {\@ifundefined{index@#2}
            {\ClassError{literate}{Wrong item to index}{Control sequences must be indexed directly (no optional argument)\MessageBreak For parameters, write their type as an optional argument}}
            {\index{\csname index@#2\endcsname}}}
        {\@ifundefined{index@#1@#2}
            {\ClassError{literate}{Wrong item to index}{Control sequences must be indexed directly (no optional argument)\MessageBreak For parameters, write their type as an optional argument}}
            {\index{\csname index@#1@#2\endcsname}}}}

\newcommand{\processtexcs}[2][]{%
    \set@macro@type{#1}{#2}
    \if@iscsname
        \def\@index@name{index@#2}
    \else
        \def\@index@name{index@#1@#2}
    \fi
    \def\@index@content{\macro@index}
    \expandafter\xdef\csname\@index@name\endcsname
        {\macro@index}
    \texttt{\macro@format}\index{\@nameuse{\@index@name}}\par%
}

\lstnewenvironment{code}[2][m]
    {\lstset{name=code@\usefiletype,style=blockcode,moretexcs={#2}}%
      \AfterEndEnvironment{code}{\lstset{moretexcs={#2},}}%
          \@ifundefined{usefiletype}{\ClassError{literate}{Missing filetype definition}{You must specify which file to write to with the command \string\filetype}}{}%
      \addvspace{\baselineskip}%
      \marginpar{%
        \hskip0pt\vskip.3\baselineskip%
        \forcsvlist{\processtexcs[#1]}{#2}}%
      \csname\@lst @SetFirstNumber\endcsname%
      \if@firstrun%
        \lst@BeginAlsoWriteFile{\jobname.\usefiletype}%
        \ifnum\value{c@copyright}=0\lstinputlisting{\jobname.copyright}\fi%
      \fi}
    {\if@firstrun\endgroup\fi%
      \csname\@lst @SaveFirstNumber\endcsname%
      \stepcounter{c@copyright}%
      \addvspace{\baselineskip}}

\lstMakeShortInline[style=inlinecode]{|}

\lstnewenvironment{macro}
    {\lstset{name=macro@\usefiletype,style=macrocode,}%
      \addvspace{\baselineskip}%
      \csname\@lst @SetFirstNumber\endcsname}
    {\csname\@lst @SaveFirstNumber\endcsname
      \addvspace{\baselineskip}}

\if@firstrun
    \lstnewenvironment{copywrite}
        {\if@firstrun\lst@BeginWriteFile{\jobname.copyright}\fi}
        {\if@firstrun\lst@EndWriteFile\fi}
    \newenvironment{example}
        {\expandafter\comment}
        {\expandafter\endcomment}
\else
    \newenvironment{copywrite}
        {\expandafter\comment}
        {\expandafter\endcomment}
    \lstnewenvironment{example}
        {\lstset{name=example@\usefiletype,style=samplecode,}%
          \addvspace{\baselineskip}%
          \lst@BeginAlsoWriteFile{\jobname.tmp}}
        {\lst@EndWriteFile%
          \noindent\llap{\numbercolor\makebox[1em][c]{$\rightarrow$}\hspace{\lst@numbersep}}%
          \input{\jobname.tmp}%
          \addvspace{\baselineskip}}
\fi

\newcommand{\printlistings}{%
    \if@firstrun\else
        \forlistloop{}{\file@list}
    \fi
}

\AtEndDocument{\if@firstrun\begingroup\lst@EndWriteFile\fi}

And the .tex example of a self-documented class + package:

\documentclass{literate}

\addcs{ProvidesClass, definecolor, textcolor}

\addtoindex{col}{colour}{\macroname}{\entrytype!\entryname @\string\texttt{\entryname}}

\begin{document}

\begin{copywrite}
%
% This is a sample copyright notice that will be printed verbatim at the beginning of all files.
%
\end{copywrite}

\section{The first section}

\filetype{cls}

The class (|.cls|) begins thus:

\begin{code}{}
\NeedsTeXFormat{LaTeX2e}
\ProvidesClass{MWE}
    [2011/03/20 v1 Some sample class]
\LoadClass{tufte-book}
\end{code}

Then we create a new macro:

\begin{code}{mymacro,my@macro}
% Some comment
\newcommand{\mymacro}[2][]{#1{#2}}
\newcommand{\my@macro}[1]{$#1$}
\end{code}

It can be used this way:

\begin{macro}
\mymacro[format]{text}
\end{macro}

And outputs this:

\begin{example}
\mymacro[\emph]{some text}
\end{example}

Then we create a counter:

\begin{code}[c]{something}
% Some comment
\newcounter{something}
\end{code}

\section{The second section}

\filetype{sty}

We also create an |.sty| file:

\begin{code}{}
\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{MWE}
    [2011/03/20 v1 Some sample package]
\end{code}

\pagebreak A new environment:

\begin{code}[e]{myenv}
\newenvironment{myenv}{}{}
\end{code}

A new color:

\begin{code}[col]{ochre}
\usepackage{xcolor}
\definecolor{ochre}{HTML}{CF872E}
\end{code}

\begin{example}
\textcolor{ochre}{Some ochre text!}
\end{example}

But perhaps should we return to talking about |\mymacro|\docindex{mymacro}, and later we'll switch back to colours\docindex[col]{ochre}.

\printlistings

\printindex

\end{document}

Requires 2 runs of LaTeX to print the documentation + one run of MakeIndex and one final run of LaTeX.

Edit (April 3): Added an AfterEndEnvironment hook so the new macro names are set globally after the first code environment (fixes a bug in syntax highlighting).

Edit (April 10): Added several new things:

A macro for adding some control sequences to listings' colouring, without indexing them: \addcs{}, which takes a CSV list as its argument.
An environment that writes a copyright file and prints it at the beginning of every new file: \begin{copywrite}.
A command that inputs the listings of all created files, with a "heading" that can be customised via \renewcommand{\listingshead} – by default, I have set it to print a section title and a short sentence. (Could be made more customisable I suppose…)
Semi-automatic indexing (see below).
Some error messages, because wrong indexing commands are easy to produce and the LaTeX error message was really, really unhelpful.

Several macros that automatically index the marked content of every code environment based on its type. It also works with @ characters in macro names. By default, it is assumed that the string is a macro but you can change it using the optional argument (e.g. \begin{code}[e]{myenv} for an environment).

I have defined macros (m, default), environments (e), floats (f), lengths (l), dimensions (d), counters (c), packages (p) and options (o). New types can be defined implicitly:

\begin{code}[great-macro]{mygreatmacro}

in which case they will be treated as macros. They can also be defined explicitly in the preamble with:

\addtoindex{<short name>}{<name to index>}{<format>}{<indexing code>}
% The starred version is for control sequences, the normal version for everything else
% Helper macros are `\macroname`, `\entryname` and `\entrytype`

In the main file, you then need to write \docindex{mymacro} for control sequences, or \docindex[c]{mycounter} for everything else. This is made necessary by the fact that, while you can have only one control sequence with a given name, you can have several strings of other types with the same name (say the \chapter command vs the chapter counter and perhaps a chapter colour).

I will try to set up a repository with the complete file (which includes options at class selection), and try to work on some documentation as things are getting a bit complex now.

And I would indeed be very happy to get any feedback regarding bugs and possible features or improvements – in the hope that it can become a package once tested and cleaned up.

Best Answer

Related Solutions

[Tex/LaTex] How to create step by step annotated programming tutorial in LaTeX

[Tex/LaTex] Different approach to literate programming for LaTeX

Related Question