[Tex/LaTex] Different approach to literate programming for LaTeX

documentationliterate-programming

I have spent a couple of hours rethinking the literate style tools we have for LaTeX, which are essentially doc, docstrip and ltxdoc. Although the traditional workflow with a good editor can be less of a pain, imprisoning the code in guards and percentage marks can be disruptive to the flow of writing code, especially to newcomers. With this in mind and also to try and improve the typography and the functionality of the code, I came up with an alternative.

The central idea is to be able to use any class as the base class (in the MWE I have used the tufte-book) and to be able to have pretty printing using a package such as listings. The approach is to use only one .tex file which when compiled produces the documentation as well as the class, package or files.

Essentially it saves code verbatim when is defined. The MWE example only outputs to one .sty file, but it is possible to save to as many files as one needs (simulating the docstrip guards). A small installation script can be incorporated and self generated with LuaLaTeX if one wants to also simulate docstrips ability to install files in different directories.

What precautions should one take in developing such a code and are there any essential features of the doc/docstrip approach that cannot be incorporated in my approach?

\documentclass{tufte-book}
\usepackage{microtype,soul}
\makeatletter
\usepackage[charter]{mathdesign}
 \def\rmdefault{bch} % not scaled
 \def\ttdefault{blg}
\usepackage{xcolor,filecontents,ragged2e}
\usepackage[listings, theorems]{tcolorbox}
\definecolor{theblue} {rgb}{0.02,0.04,0.48}
\definecolor{thered}  {rgb}{0.65,0.04,0.07}
\definecolor{thegreen}{rgb}{0.06,0.44,0.08}
\definecolor{thegrey} {gray}{0.5}
\definecolor{theshade}{gray}{0.94}
\definecolor{theframe}{gray}{0.75}

\lstloadlanguages{[LaTeX]TeX, [primitive]TeX}

% Emphasis
\newcommand\emphasis[2][thered]{\lstset{emph={newcommand,def,gdef,#2},
   emphstyle={\ttfamily\textcolor{#1}}}}%

\lstset{language={[LaTeX]TeX},
      escapeinside={{(*@}{@*)}}, 
       numbers=left, gobble=2,
       stepnumber=1,numbersep=5pt, 
       numberstyle={\footnotesize\color{gray}},firstnumber=last,
       breaklines=true,
       framesep=5pt,
       basicstyle=\small\ttfamily,
       showstringspaces=false,
     % keywordstyle=\ttfamily\textcolor{thegreen},
      stringstyle=\color{orange},
      commentstyle=\color{black},
      rulecolor=\color{theshade},
      breakatwhitespace=true,
     showspaces=false,  % shows spacing symbol
      xleftmargin=0pt,
      xrightmargin=5pt,
      aboveskip=3pt plus1pt minus1pt, % compact the code looks ugly in type
      belowskip=7pt plus1pt minus1pt,  % user responsible to insert any skips
      backgroundcolor=\color{theshade}
}
\lst@RequireAspects{writefile}

\lstnewenvironment{Macro}[1][Test]
    {\emphasis{#1}\marginpar{\vskip0.5\baselineskip\footnotesize\color{thered}\texttt{\string#1}}
   \lst@BeginAlsoWriteFile{textsamples.sty}}
   {\endgroup}

\newcommand\lorem{Fusce adipiscing justo nec ante. Nullam in enim.
 Pellentesque felis orci, sagittis ac, malesuada et, facilisis in,
 ligula. Nunc non magna sit amet mi aliquam dictum. In mi. Curabitur
 sollicitudin justo sed quam et quadd. \par}


\DeclareRobustCommand{\todo}[1]{\sidenote{\hl{#1}}}
\author{Y Lazarides}\publisher{Camel Press}
\title{\parindent0pt A New Approach in\\ Documenting Macros}
\begin{document}

\backmatter
\maketitle
\parindent0pt


\mainmatter

\tableofcontents

\chapter{The Approach}
\section{User Documentation}
The user documentation can be styled a bit better. The user should be able to 
write the package or class in a more natural style.\todo{Write a few more notes here.}
\medskip

\begin{tcblisting}{colframe=thegreen,boxrule=1pt,colback=thered!5,listing options={style=tcblatex}}
The macro  \verb+\LaTeXe+, typesets the \LaTeXe logo.
\end{tcblisting}

\section{Implementation}

\lorem

\Macro[\ProvidesPackage]
  \NeedsTeXFormat{LaTeX2e}
  \ProvidesPackage{textsamples}[2012/02/13 v1.0 sample texts]
\endMacro

\lorem


\begin{Macro}[\lorem]
  \newcommand\lorem{\leavevmode 
    Fusce adipiscing justo nec ante. Nullam in enim.
    Pellentesque felis orci, sagittis ac, malesuada et, facilisis in,
    ligula. Nunc non magna sit amet mi aliquam dictum. 
    In mi. Curabitur sollicitudin justo sed quam et quadd.
    \par
  }
\end{Macro}

\lorem

\emphasis{\tex}
\begin{Macro}[\tex]
  \gdef{\tex}{%
      \TeX\xspace (*@\sidenote{You can even place sidenotes.% 
      These notes have no impact on the %
      code being written to the file.} @*)
  }
\end{Macro}
 % closes the open file
\begingroup\lst@EndWriteFile

\section{Full listing of {textsamples.sty}}
\lstinputlisting{textsamples.sty}
\end{document}

Note, I have used listings for the verbatim writes and tcolorbox for the example self-running box. It also omits doc's many useful macros which would still need to be adapted.

Best Answer

As discussed in earlier in the comments, it is quite possible to solve the problem of defining classes with this method. Granted, as you said, it takes two runs, but it is possible to make a document that will deal with the writing (and some minimal printing) on the first run, and that will process class-defined commands on the second run only (if correctly marked up, obviously).

In order to perform this check, I rely on the existence (or lack thereof) of an .aux file, because it is both easy – most IDEs can delete it in one click, it allows for overwrites, etc. – and user friendly in case of problem – as I expect most users to try to delete this file first if something goes wrong.

The only requirement is that we store the code in a class file instead of a package. The following MWE implements this solution, and also a macro-based solution to the problem of multiple files (you should write one file after the other, not mix them up haphazardly). Due to the fact that I am using comma-separated lists for the macros, it might make things easier to index, e.g. adding an optional argument for the type of thing created (macro, environment, counter, etc.).

The .cls file:

\NeedsTeXFormat{LaTeX2e}
\ProvidesClass{literate}

\newif\if@firstrun
    \@firstruntrue
\newif\if@iscsname
    \@iscsnamefalse
\newcounter{c@copyright}

\IfFileExists{\jobname.aux}
    {\IfFileExists{\jobname.cls}
        {\LoadClass{\jobname}}{}
      \IfFileExists{\jobname.sty}
        {\RequirePackage{\jobname}}{}
      \@firstrunfalse}
    {\LoadClass{book}}

\newcommand{\filetype}[1]{%
    \if@firstrun
        \begingroup\lst@EndWriteFile
    \fi
    \def\usefiletype{#1}
    \listadd{\file@list}{\listingshead\lstinputlisting[style=plain]{\jobname.#1}\addvspace{\baselineskip}}
    \setcounter{c@copyright}{0}
}

\newcommand{\listingshead}{%
    \section{Listings for \jobname.\usefiletype}%
    The file contains the following code:
}

\RequirePackage{etoolbox,xstring,xcolor,listings,verbatim,docmute,makeidx}
\lst@RequireAspects{writefile}
\makeindex

\newcommand{\numbercolor}{\color[HTML]{655643}}% Ash
\newcommand{\macrocolor}{\color[HTML]{78948D}}% Teal
\newcommand{\commentcolor}{\color[HTML]{BF4D28}}% Saffron
\newcommand{\mathcolor}{\textcolor[HTML]{CF872E}}% Ochre
\newcommand{\emphcolor}{\color{cyan}}
\newcommand{\identifiercolor}{\color{black}}
\newcommand{\bracketcolor}{\color{black}}

\lstset{
    language=[AlLaTeX]TeX,
    breaklines=true,
    columns=spaceflexible,
    emptylines=0,
    breakindent=1em,
    tabsize=4,
    basicstyle=\ttfamily,
    texcsstyle=*\macrocolor,
    commentstyle=\commentcolor\itshape,
    identifierstyle=\identifiercolor,
    morestring=[b]$,
    stringstyle=\mathcolor,
}

\lstdefinestyle{blockcode}{
    frame=leftline,
    framerule=.2ex,
    rulecolor=\numbercolor,
    numbers=left,
    numberstyle=\footnotesize\numbercolor,
    firstnumber=auto,
}

\lstdefinestyle{samplecode}{
    numbers=left,
    numberfirstline=true,
    stepnumber=10000,
    firstnumber=1,
    numberstyle=\numbercolor\makebox[1em][c]{\Large$\star$}\@gobble,
}

\lstdefinestyle{macrocode}{
    numbers=left,
    numberfirstline=true,
    stepnumber=10000,
    firstnumber=1,
    numberstyle=\numbercolor\makebox[1em][c]{\P}\@gobble,
    identifierstyle=\identifiercolor\itshape,
    literate=   {\{}{{\bracketcolor\{}{\identifiercolor$\langle$}}{2}
            {\}}{{\identifiercolor$\,\rangle$}{\bracketcolor\}}}{2}
            {[}{{\bracketcolor[}{\identifiercolor$\langle$}}{2}
            {]}{{\identifiercolor$\,\rangle$}{\bracketcolor]}}{2}
            {<}{{\identifiercolor$\langle$}}{1}
            {>}{{\identifiercolor$\,\rangle$}}{1},
}

\lstdefinestyle{inlinecode}{
    literate=   {\{}{{\bracketcolor\{}}{1}
            {\}}{{\bracketcolor\}}}{1}
            {[}{{\bracketcolor[}}{1}
            {]}{{\bracketcolor]}}{1},
}

\lstdefinestyle{plain}{
    style=blockcode,
    texcsstyle=*\color{black},
    emphstyle=\color{black},
    commentstyle={\color{black}\itshape},
    stringstyle=\color{black},  
}

\newcommand{\addcs}[1]{\lstset{moretexcs={#1},}}

% Create commands for globally adding indexation patterns. By default, it is assumed to be a control sequence
\newcommand{\defaultindexdef}{}
\newcommand{\addindexdef}{}

\newcommand{\setdefaultindex}[3]{%
    \def\defaultindexdef
        {\@iscsnametrue
          \expandafter\ifstrempty{#1}
            {\def\macro@type{}}
            {\def\macro@type{#1}}
          \expandafter\ifstrempty{#2}{}{\def\macro@format{#2}}
          \expandafter\ifstrempty{#3}{}{\def\macro@index{#3}}}
}
\newcommand{\addtoindex}{\@ifstar{\s@addtoindex}{\@addtoindex}}
\newcommand{\@addtoindex}[4]{%
    \apptocmd{\addindexdef}
        {\expandafter\ifstrequal\expandafter{\macrotype}{#1}
            {\@iscsnamefalse
              \expandafter\ifstrempty{#2}
                {\def\macro@type{}}
                {\def\macro@type{#2}}
              \expandafter\ifstrempty{#3}{}{\def\macro@format{#3}}
              \expandafter\ifstrempty{#4}{}{\def\macro@index{#4}}}
            {}}
        {}{}
}
\newcommand{\s@addtoindex}[4]{%
    \apptocmd{\addindexdef}
        {\expandafter\ifstrequal\expandafter{\macrotype}{#1}
            {\@iscsnametrue
              \expandafter\ifstrempty{#2}{}{\def\macro@type{#2}}
              \expandafter\ifstrempty{#3}{}{\def\macro@format{#3}}
              \expandafter\ifstrempty{#4}{}{\def\macro@index{#4}}}
            {}}
        {}{}
}

% Set default index
\setdefaultindex
    {\macrotype}
    {\macroname}
    {\entryname @\string\texttt{{\string\textbackslash}\entryname}\space(\entrytype)}

% Create a command for setting the macro type based on the optional argument to code
\newcommand{\set@macro@type}[2]{
    % Create helper macros for text
    \def\macrotype{#1}
    \def\macroname{\if@iscsname\textbackslash\fi#2}
    % Create helper macros for indexes
    \StrSubstitute{#2}{@}{"@}[\entryname]
    \def\entrytype{\macro@type}
    % Default definitions
    \defaultindexdef
    % Specific type definitions
    \ifstrequal{#1}{m}
        {\@iscsnametrue
          \def\macro@type{}
          \def\macro@index{%
            \entryname @\string\texttt{{\string\textbackslash}\entryname}}}{}
    \ifstrequal{#1}{l}
        {\@iscsnametrue
          \def\macro@type{length}}{}
    \ifstrequal{#1}{d}
        {\@iscsnametrue
          \def\macro@type{dimension}}{}
    \ifstrequal{#1}{e}
        {\@iscsnametrue
          \def\macro@type{environment}
          \def\macro@format{\textbackslash#2\par\textbackslash end#2}}{}
    \ifstrequal{#1}{f}
        {\@iscsnametrue
          \def\macro@type{float}
          \def\macro@format{\textbackslash#2\par\textbackslash end#2}}{}
    \ifstrequal{#1}{p}
        {\@iscsnamefalse
          \def\macro@type{package}
          \def\macro@index{%
            \entryname @\string\texttt{\entryname}\entrytype}}{}
    \ifstrequal{#1}{o}
        {\@iscsnamefalse
          \def\macro@type{option}
          \def\macro@index{%
            \entryname @\string\texttt{\entryname}\entrytype}}{}
    \ifstrequal{#1}{c}
        {\@iscsnamefalse
          \def\macro@type{counter}
          \def\macro@index{%
            \entryname @\string\texttt{\string\textit{\entryname}}\space(\entrytype)}}{}
    % Append user created macros
    \addindexdef
}


\newcommand{\docindex}[2][]{%
    \ifstrempty{#1}
        {\@ifundefined{index@#2}
            {\ClassError{literate}{Wrong item to index}{Control sequences must be indexed directly (no optional argument)\MessageBreak For parameters, write their type as an optional argument}}
            {\index{\csname index@#2\endcsname}}}
        {\@ifundefined{index@#1@#2}
            {\ClassError{literate}{Wrong item to index}{Control sequences must be indexed directly (no optional argument)\MessageBreak For parameters, write their type as an optional argument}}
            {\index{\csname index@#1@#2\endcsname}}}}

\newcommand{\processtexcs}[2][]{%
    \set@macro@type{#1}{#2}
    \if@iscsname
        \def\@index@name{index@#2}
    \else
        \def\@index@name{index@#1@#2}
    \fi
    \def\@index@content{\macro@index}
    \expandafter\xdef\csname\@index@name\endcsname
        {\macro@index}
    \texttt{\macro@format}\index{\@nameuse{\@index@name}}\par%
}

\lstnewenvironment{code}[2][m]
    {\lstset{name=code@\usefiletype,style=blockcode,moretexcs={#2}}%
      \AfterEndEnvironment{code}{\lstset{moretexcs={#2},}}%
          \@ifundefined{usefiletype}{\ClassError{literate}{Missing filetype definition}{You must specify which file to write to with the command \string\filetype}}{}%
      \addvspace{\baselineskip}%
      \marginpar{%
        \hskip0pt\vskip.3\baselineskip%
        \forcsvlist{\processtexcs[#1]}{#2}}%
      \csname\@lst @SetFirstNumber\endcsname%
      \if@firstrun%
        \lst@BeginAlsoWriteFile{\jobname.\usefiletype}%
        \ifnum\value{c@copyright}=0\lstinputlisting{\jobname.copyright}\fi%
      \fi}
    {\if@firstrun\endgroup\fi%
      \csname\@lst @SaveFirstNumber\endcsname%
      \stepcounter{c@copyright}%
      \addvspace{\baselineskip}}

\lstMakeShortInline[style=inlinecode]{|}

\lstnewenvironment{macro}
    {\lstset{name=macro@\usefiletype,style=macrocode,}%
      \addvspace{\baselineskip}%
      \csname\@lst @SetFirstNumber\endcsname}
    {\csname\@lst @SaveFirstNumber\endcsname
      \addvspace{\baselineskip}}

\if@firstrun
    \lstnewenvironment{copywrite}
        {\if@firstrun\lst@BeginWriteFile{\jobname.copyright}\fi}
        {\if@firstrun\lst@EndWriteFile\fi}
    \newenvironment{example}
        {\expandafter\comment}
        {\expandafter\endcomment}
\else
    \newenvironment{copywrite}
        {\expandafter\comment}
        {\expandafter\endcomment}
    \lstnewenvironment{example}
        {\lstset{name=example@\usefiletype,style=samplecode,}%
          \addvspace{\baselineskip}%
          \lst@BeginAlsoWriteFile{\jobname.tmp}}
        {\lst@EndWriteFile%
          \noindent\llap{\numbercolor\makebox[1em][c]{$\rightarrow$}\hspace{\lst@numbersep}}%
          \input{\jobname.tmp}%
          \addvspace{\baselineskip}}
\fi

\newcommand{\printlistings}{%
    \if@firstrun\else
        \forlistloop{}{\file@list}
    \fi
}

\AtEndDocument{\if@firstrun\begingroup\lst@EndWriteFile\fi}

And the .tex example of a self-documented class + package:

\documentclass{literate}

\addcs{ProvidesClass, definecolor, textcolor}

\addtoindex{col}{colour}{\macroname}{\entrytype!\entryname @\string\texttt{\entryname}}

\begin{document}

\begin{copywrite}
%
% This is a sample copyright notice that will be printed verbatim at the beginning of all files.
%
\end{copywrite}

\section{The first section}

\filetype{cls}

The class (|.cls|) begins thus:

\begin{code}{}
\NeedsTeXFormat{LaTeX2e}
\ProvidesClass{MWE}
    [2011/03/20 v1 Some sample class]
\LoadClass{tufte-book}
\end{code}

Then we create a new macro:

\begin{code}{mymacro,my@macro}
% Some comment
\newcommand{\mymacro}[2][]{#1{#2}}
\newcommand{\my@macro}[1]{$#1$}
\end{code}

It can be used this way:

\begin{macro}
\mymacro[format]{text}
\end{macro}

And outputs this:

\begin{example}
\mymacro[\emph]{some text}
\end{example}

Then we create a counter:

\begin{code}[c]{something}
% Some comment
\newcounter{something}
\end{code}

\section{The second section}

\filetype{sty}

We also create an |.sty| file:

\begin{code}{}
\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{MWE}
    [2011/03/20 v1 Some sample package]
\end{code}

\pagebreak A new environment:

\begin{code}[e]{myenv}
\newenvironment{myenv}{}{}
\end{code}

A new color:

\begin{code}[col]{ochre}
\usepackage{xcolor}
\definecolor{ochre}{HTML}{CF872E}
\end{code}

\begin{example}
\textcolor{ochre}{Some ochre text!}
\end{example}

But perhaps should we return to talking about |\mymacro|\docindex{mymacro}, and later we'll switch back to colours\docindex[col]{ochre}.

\printlistings

\printindex

\end{document}

Requires 2 runs of LaTeX to print the documentation + one run of MakeIndex and one final run of LaTeX.

Edit (April 3): Added an AfterEndEnvironment hook so the new macro names are set globally after the first code environment (fixes a bug in syntax highlighting).

Edit (April 10): Added several new things:

  • A macro for adding some control sequences to listings' colouring, without indexing them: \addcs{}, which takes a CSV list as its argument.

  • An environment that writes a copyright file and prints it at the beginning of every new file: \begin{copywrite}.

  • A command that inputs the listings of all created files, with a "heading" that can be customised via \renewcommand{\listingshead} – by default, I have set it to print a section title and a short sentence. (Could be made more customisable I suppose…)

  • Semi-automatic indexing (see below).

  • Some error messages, because wrong indexing commands are easy to produce and the LaTeX error message was really, really unhelpful.

Several macros that automatically index the marked content of every code environment based on its type. It also works with @ characters in macro names. By default, it is assumed that the string is a macro but you can change it using the optional argument (e.g. \begin{code}[e]{myenv} for an environment).

I have defined macros (m, default), environments (e), floats (f), lengths (l), dimensions (d), counters (c), packages (p) and options (o). New types can be defined implicitly:

\begin{code}[great-macro]{mygreatmacro}

in which case they will be treated as macros. They can also be defined explicitly in the preamble with:

\addtoindex{<short name>}{<name to index>}{<format>}{<indexing code>}
% The starred version is for control sequences, the normal version for everything else
% Helper macros are `\macroname`, `\entryname` and `\entrytype`

In the main file, you then need to write \docindex{mymacro} for control sequences, or \docindex[c]{mycounter} for everything else. This is made necessary by the fact that, while you can have only one control sequence with a given name, you can have several strings of other types with the same name (say the \chapter command vs the chapter counter and perhaps a chapter colour).


I will try to set up a repository with the complete file (which includes options at class selection), and try to work on some documentation as things are getting a bit complex now.

And I would indeed be very happy to get any feedback regarding bugs and possible features or improvements – in the hope that it can become a package once tested and cleaned up.

Related Question