I have only basic knowledge of the TeX internals, so I hope what I am saying here is not complete nonsense. If I understand correctly, in TeX, horizontal positioning / line breaking is done using a rather advanced algorithm based on a paragraph-level optimization (Knuth-Plass); it allows for concepts such as glue and line break penalties, and gives considerably better results than the more naive approach of, say, MS Word.
On the other hand, I seem to understand that nothing similar is used for vertical positioning of text. Lines are shipped out as soon as they are done and pages are filled using the obvious greedy algorithm. Widows and orphans are handled using special tricks.
[EDIT: this is not entirely correct, see Hendrik Vogt's answer below — there is an optimization algorithm involved, although it's only local optimization over a single page and not KP]
So, a question arises naturally to me:
why isn't something similar to Knuth-Plass used also for page breaks and the vertical layout of text? Are there major complications?
Best Answer
this was addressed by Knuth in a q&a session in st. petersburg, florida, published in tugboat: - TUG'95: Questions and Answers with Prof. Donald E. Knuth, pp.18 (bottom of column 2) - 20; the session was republished in Digital Typography, with the relevant question starting on p.594.
the page-breaking problem was also the subject of Michael Plass' dissertation, Optimal Pagination Techniques for Automatic Typesetting Systems, posted on the tug web site.