[Tex/LaTex] What value TeX uses as its minimal unit

Question

TeX uses scaled point as its minimal unit. At least it's said so in the TeXbook:

TeX represents all dimensions internally as an integer multiple of the tiny units called sp.

65536 sp = 2^{16} sp = 1 pt.

And later on,

TeX actually does its calculations with integer multiples of 2^{-16}…

BUT, compile this and you will see, that 0.00000762939453125 starts to be distinguishable from zero, which means that it's TeX's minimal unit. (0.00000762939453125 is exactly 2^{-17})

\line{\hskip 0pt plus 16383.99999fil and now?\hskip 0pt plus 0.000007629394531249fill}
\line{\hskip 0pt plus 16383.99999fil and now?\hskip 0pt plus 0.000007629394531250fill}
\bye

So, what is the real minimal unit: 2^{-16} or 2^{-17} ?

Answer 1

When you say

\dimen0=0.00000762939453125pt

a \showthe\dimen0 instruction will answer

> 0.00002pt.

because this is what 1sp looks like when shown in points.

What happens? TeX does binary arithmetic, deep down inside the program. The number you input is recognized as greater than zero (one should look into the program code) and so the dimension is set to the smallest positive one.

Similarly, after

\skip0=0pt plus 0.000007629394531250fill \showthe\skip0

you get

> 0.0pt plus 0.00002fill.

so never is a positive dimension less than 1sp used. The value you input can appear less than 1sp, but the value that TeX uses is 1sp.

The conversion from decimal number to scaled integer is described in module 102 of tex.web; the integral part is just multiplied by 65536, so it's only interesting for numbers in the form 0.d₀d₁…d_k–1

The final scaled integer is stored in a which is initialized as 0. Then we start from the far right and at each step a is assigned the result of

(a + d_i * 2¹⁷) div 10

Finally, a is assigned the value (a + 1) div 2 (with truncation)

If we carry out this algorithm, we get the following numbers as values of a (in parentheses the digit examined)

(5)  65536
(2)  32768
(1)  16384
(3)  40960
(5)  69632
(4)  59392
(9) 123904
(3)  51712
(9) 123136
(2)  38528
(6)  82496
(7) 100000
(0)  10000
(0)   1000
(0)    100
(0)     10
(0)      1
final step (1+1) div 2 = 1

The div operation works by truncation.

I leave as an exercise to show that 0.0000152587890625 = 2^–16 produces 1 as well and that 0.000007629394531249 produces 0 instead.

As you see, the algorithm uses 2¹⁷ to ensure 16 bit precision of the fractional part.