[Tex/LaTex] How to generate “You can’t use \unskip in vertical mode”

errorsprogrammingtex-core

We all know (or believe so) that \unskip is not possible in main vertical mode and this statement can be found in various answers on the site. And yes LaTeX does a lot of gymnastics in \addvspace or \addpenalty to deal with this problem. So I was a bit surprised when in The second column always appears lower than the first one in the multicol environment a simple workaround using \unskip worked.

This was on the main vertical list!

So looking into the TeX program itself, the condition is that the error will be triggered if the last item on the list is a glue item and this glue item has already been contributed to the current page. But when does this happen?

XXX
\par
\vskip 1cm 
\vskip 5cm
\unskip\unskip
XXX
\bye

This works perfectly and according to the TeXbook there are only 5 cases when stuff gets moved from recent contributions to the current page (page 122):

Here is a list of the times when that can happen: (a) At the beginning
or end of a paragraph, provided that this paragraph is being
contributed to the main vertical list. (b) At the beginning or end of
a displayed equation within such a paragraph. (c) After completing an
\halign in vertical mode. (d) After contributing a box or penalty or
insertion to the main vertical list. (e) After an \output routine
has ended.

Unfortunately it is not quite accurate (probably there are some other places discussing this further that I have overlooked). For example, if you add an empty line after the \vskip in the example above, then it triggers the error even though that \par clearly doesn't trigger or ends any paragraph.

I can imagine to create it from case (e), e.g., by making the output routine add a glue to the main vertical list, though I haven't tried that.

Any other cases in which this could be triggered in reality?

Best Answer

Looking at the source tex.web, I gather the following, none of which I am sure of. From the (separate) lines

primitive("unskip",remove_item,glue_node);

any_mode(remove_item): delete_last;

I see that \unskip invokes the procedure delete_last, which has the following comment in tex.web:

Like \lastbox, this command is not allowed in vertical mode (except internal vertical mode), since the current list in vertical mode is sent to the page builder. But if we happen to be able to implement it in vertical mode, we do.

In other words, TeX tries its best to remove something, but this is sometimes impossible. In fact, the code for delete_last "apologizes for [the] inability to do the operation now" if the current mode is vmode and (I believe) there are no recent contributions, unless \unskip follows a non-glue node (in which case \unskip does nothing). Your assessment that we need to find out when things are moved out of the recent contributions is thus correct.

The main vertical list is split into two parts: from page_head to page_tail lies the current page, and from contrib_head to contrib_tail lie recent contributions. Searching for contrib_, I find the variables used in several places:

in the procedure show_activities, link(contrib_head) is queried, to know if there are any recent contributions;
the procedure build_page appears to move material from the recent contributions to the current page, so it is crucial for us;
link(contrib_head):=link(p) is used in two places to add a node to the recent contributions;
q:=new_skip_param(top_skip_code) then link(contrib_head):=q adds the topskip to the page when the first box is added;
when breaking the page at a given node, whatever follows in the current page is placed before the recent contributions, by manipulating contrib_head and contrib_tail;
the default output routine appears to place the current page back into the recent contributions (except of course what has already been split and placed into \box255), then shipout \box255;
the same happens after a user-defined output routine (save for the \box255 business).

From this, we see that build_page is probably what you are after. It is called in the following cases (labelled with a letter corresponding to your list):

(a) when a paragraph is started in vmode (by a horizontal command such as \indent, a letter, \discretionary etc), after inserting \everypar into the input stream;
(b) after insering the \everydisplay token list in the input stream and having set various parameters upon entering display mode, if the surrounding mode is vmode;
(b) after going back to horizontal mode (and saving the language etc) after a display is finished, if the surrounding mode is vmode;
(a) after ending a paragraph with \par in horizontal mode if the resulting mode is vmode;
(c) when \halign ends in vmode;
(d) after inserting a penalty in vmode;
(d) within the procedure box_end, if the finished box (or box register, or...) is non-null and is to be added to the main vertical list (rather than stored, shipped out, or used as a leader);
(d) when an \insert or \vadjust ends with an end-group character, if the resulting mode is vmode;
(e) after TeX has finished a user's output routine, checked that \box255 is empty, and placed all the current page material into recent contributions (see above);
when encountering \par in vmode;
after inserting \hbox to \hsize{}\vfill\penalty-'10000000000 if there is still some material in the main vertical list when meeting the primitive \end.

A few differences crop up compared to your list. After a default output routine, build_page does not seem to be used. The case of \end is missing from your list. More importantly, the case of \par in unrestricted vertical mode is not explicitly given in your list (although (a) would fit it, I guess). This case explains why

XXX\par
\vskip 1cm
%
\unskip

fails upon removing the %. Contrarily to expectations, \par in vertical mode is not ignored: it moves material from the recent contributions to the current page, but also resets some parameters (\looseness, \hangindent, \hangafter and \parshape). This comes to me as a surprise.

I found one last thing. The procedure build_page is not used in append_glue because the procedure append_glue "is used in at least one place where that would be a mistake". I do not know where the mistake would be. But this explains why \vskip 1cm does not make TeX move any material to the current page.

Related Solutions

[Tex/LaTex] way to get at the value of \prevdepth from the main vertical list inside the output routine

@wipet in his answer has shown how to resolve the problem as long as we can assume that that the \prevdepth of all material in the document is sufficiently small (that it lies below \maxdepth in fact). In that case we can use the depth of box 255 as a measure for the \prevdepthcalculations that will have taken place if there is any remainder in recent contributions, and use this to adjust the new depth to match that. An if there isn't any remainder then this doesn't really matter either. This trick is in fact already mentioned by Don Knuth in the TeXbook where he discusses an output routine that add index headings in random places in the text (and normal text has this nice property of having a depth smaller than \maxdepth ... normally but unfortunately not always.

However, this stops working if this can't guaranteed and the general case is what I was and am after. He is also correct in stating that finding out what the value of \prevdepth on the main vertical list is isn't really going to help us as such, so my question above was partly incorrect: we also need to know if there is in fact something on the recent contributions and if so deal with it.

So what I tried to get my head around over the last days if that is really impossible to find out within base TeX (or if it needs an extension in luatex or is already available there and resolved in ConTeXt ... would be interesting to learn about it beside that claim @Martin you made). And as it stands now it is possible in TeX after all. The solution is a bit complex and perhaps it can be simplified further but it is not so complex that it can't be used (and hopefully not too complex that I overlooked some cases).

The main idea is not to try to determine the situation directly but instead use 2 output routines in succession that just ensure that the recent contributions are empty. Along the way we gather enough information to afterwards dissect the material gathered and do whatever we like.

My initial idea of using \aftergroup to regain control and have a look doesn't work because the token inserted that way will not be executed when the output routine ends. Instead TeX immediately calls again on a procedure "buildpage" that takes anything on recent contributions and moves it to the main vertical list and only once that has happened my inserted token is looked at (in other words too late).

So the more complicated approach is that the first OR puts box255 back but only it has changed \vsize to the largest possible dimen. Additionally it uses \aftergroup so that we gain control later again. As we have changed \vsize we will get everything including the material recent contributions onto the main vertical list and only then our control token kicks in. Finally we change the output routine to a second one and then return.

The token inserted by \aftergroup will then issue a forcing penalty (actually it does a little more, see below) so that everything is grabbed and the second OR is called.

Inside that OR we are now in a better situation:

we know that recent contributions is empty (except for a (\penalty 10000)
we can store away what was gathered in box255
or we could manipulate it using \vsplit etc to get, for example the amount split of that we would have gotten in the first OR
and we can use \aftergroup to gain control after the OR has ended.
the latter allows us to change \prevdepth on the main vertical list to represent whatever is needed

And that should (I believe) do the trick completely.

Here is the (more or less documented code including a bit of test data to play out various scenarios). It is a bit longish but largely because I tried to properly document the most important aspects and some of the more subtle points. Enjoy:

\tracingoutput1
\showboxbreadth\maxdimen\showboxdepth\maxdimen
\tracingpages1
\tracingonline1

\vsize20\baselineskip

\lineskip=13pt  % for identifications

% this is our trial material used below. We will arrange things so
% that the first para will be longer than a page so that we will end
% up with some material on recent contributions. The OR is actually
% then triggered when the first ``p'' is seen. Alternatively one can
% uncomment the \vskip or the \penalty in which case the OR will be
% triggered by them or you could uncomment ``Line2 and3'' then the
% break happens somewhere in the middle of the first paragraph (in
% vmode inn that case)

\def\testmaterial{%
  Line 1 \hfil\break
%  Line 2 \hfil\break
%  Line 3 \hfil\break
  \vadjust{\penalty -333 }
  some text some text some text some text some text some text
  some text some text some text
   and some more text ggg \vrule depth 88pt

%\ vskip 17pt
% \penalty 15

  pppppppppppppppppppppppppp

  \showlists
}


% now this here is just to see the whole stuff being processed by the
% normal OR and see the \showlists result for it. One can then compare
% that to the showlists result we get later to check for differences

\testmaterial

\vfill

\eject


%==============================================

\newdimen\savedvsize
\newbox\savedORbox


% now for a set of special output routines:
% the first one does the following
%
% - save away current \vsize and set it to \maxdimen
% - then unbox 255 and readd the output penalty (unless it is 10000)
% - set up a new output routine for the next time
% - finally install control with \aftergroup\addendpenalty
%

% The point here is that the \aftergroup token is not actually
% directly executed the moment the OR ends. If TeX ends an OR it looks
% at the recent contribution and if they are not empty it will call
% ``buildpage'' and move them to the main vertical list. And only if
% this has happened the \aftergroup token gets executed.  Now given
% that we set \vsize to the largest possible dimen this means that all
% the remainder that was not used first time around will now end up on
% the main vertial list and only then \addendpenalty kicks in.


\output{%
  \global\savedvsize\vsize
  \global\vsize\maxdimen
%--- tracing --------------
  \showthe\outputpenalty
  \showlists
%--------------------------
  \unvbox255
%
% above I claimed we put \outputpenalty back (which we should) but to
% make things more visible I put back a special penalty that can be
% easily recognised in \showlists
%
%  \penalty \ifnum\outputenalty=10000 0 \else \outputpenalty \fi
  \penalty-777
  \global\output{\ORtwo}%
  \aftergroup\addendpenalty
}

% The macro \addendpenalty is used with \aftergroup from the output
% routine to gain control again. It adds a penalty to trigger the next
% output routine. However, we are quite likely in horizontal mode when
% the OR returns (just have seen the start of a paragraph) so we check
% for this. If true we remove the indentation box and end the
% paragraph. As a result all that get contributed is \parskip but no
% box (so \prevdepth is not touched). We signal with the penalty value
% whether or not we have seen hmode as we will have to remove that
% extra parskip in the next OR.

\def\addendpenalty{%
 \ifhmode
   \setbox0\lastbox\par\penalty-10010
 \else
   \penalty-10011 
 \fi}


% the second Or now should receive everything that was on the main
% vertical list with the recent contributions being empty (well empty
% except for a \penalty10000 that TeX puts in the place where it
% triggered the OR).
%
% What we have to do now is to remove the surplus \parskip at the
% bottom of 255 if we have been in hmode before. This is something we
% can determine by looking at the \outputpenalty that should be -10010
% in that case (otherwise -10011)
%
% then we save all of 255 in a spare box and return from the OR. To
% gain control afterwards we issue \aftergroup\XXX

\def\ORtwo{%
%--- tracing --------------
  \showthe\outputpenalty
  \showbox255\showlists
%--------------------------
  \ifnum\outputpenalty=-10010
    \setbox255=\vbox
      {\unvbox255
       \unskip  % this gets rid of the \parskip from hmode
      }
  \fi
  \global\setbox\savedORbox\box255
  \aftergroup\XXX
}

% Note that now the macro \XXX is immediately called when the OR ends
% as the recent contributions are empty now. Thus this macro now is
% getting us in a good shape:
%
% - it can access \prevdepth and \prevgraf (which is in fact having
% the same issue) and it can change them as necessary.
%
% - it has the complete main vertical list at its disposal (saved
% in \savedORbox)
%
%  - there is nothing left in recent contributions so anything
%  following is new material, so we can now arrange everything to our
%  liking and reprocess


\def\XXX{%
%--- tracing --------------
  \showthe\prevdepth  % this is finally the outer one and we could
                      % change it if needed
  \showlists          % nothing on it not even the penalty remains
                      % only prevdepth and prevgraf set (incorrectly)
%--------------------------
%  \global\vsize\savedvsize
  \global\vsize20\baselineskip
  \global\output{\plainoutput}%
  \unvbox\savedORbox
}

% what we do above is set the vsize back to 20 baselines set up the
% plainoutput routine again and reprocess and we get 100% the same as
% in the initial test (well, in one place there is penalty 777 but
% that was just mark that spot, normally we would have \outputpenalty
% there which was 0. In the original there was nothing in this space
% only glue but that is equivalent


% and here is now the real test: we set a very short vsize so the
% first OR is triggered with \testmaterial and some part of it ends up
% in recent contributions.

\vsize=3\baselineskip

\testmaterial


\bye

[Tex/LaTex] \prevdepth and \noalign

this plain tex file

\def\foo#1{\edef\tmp{\immediate\write20{#1: \the\prevdepth}}\tmp}


\foo{1}
g


\foo{2}

\vbox{
\foo{3}

Q

\foo{4}
}

\foo{5}

a,

\foo{6}

\halign{#\hfil\cr
\noalign{\foo{7}}
 z\cr}

\foo{8}


\bye

produces

1: -1000.0pt
2: 1.94444pt
3: -1000.0pt
4: 1.94444pt
5: 1.94444pt
6: 1.94444pt
7: 1.94444pt
8: 0.0pt

showing that \prevdepth has the magic -1000pt value at the start of the main vertical list and at the start of a \vbox but not in the \noalign.

Best Answer

Related Solutions

[Tex/LaTex] way to get at the value of \prevdepth from the main vertical list inside the output routine

[Tex/LaTex] \prevdepth and \noalign

Related Question