[Tex/LaTex] pdflatex suddenly takes much longer to compile document – how to find the reason

biberbiblatexpdftexperformanceunicode

I've done some major changes in my document (dissertation, nearly 300 pages) and now compiling time has increased nearly by factor 3!

One pdflatex run took normally less than 3 minutes before and now takes about 6 minutes, for complete compiling with latexmk it took 26 minutes!!!

What I've changed:

  • switched all source files (.tex and .bib) to UTF8
  • had to switch from bibtex/biblatex to biber/biblatex

My questions:

  • Can the change to UTF8 slow down compiling?
  • I read about some problems with kpathsea checking deeps directory paths during each compiling. How can I find out if I have such a problem?
    I formerly had 2 TeXLive distros on my machine, but currently only have TeXLive 2014 installed.

Hard and Software

  • MacBook 13" (2007)
  • MacOS X 10.6.8
  • TeXLive 2014
  • Aquamacs 2.5 as editor
  • compiling is currently done from command line (pdflatex or latexmk)

Further investigations of the suspects:

1. TEXINPUTS

the command kpsexpand \$TEXINPUTS gives the following result:
(added linebreaks after each path)

.:{/Users/mbg4/Library/texlive/2014/texmf-config,
 /Users/mbg4/Library/texlive/2014/texmf-var,
/Users/mbg4/Library/texmf,
!!/usr/local/texlive/2014/texmf-config,
!!/usr/local/texlive/2014/texmf-var,
!!/usr/local/texlive/texmf-local, 
!!/usr/local/texlive/2014/texmf-dist}/tex/{kpsewhich,generic,}//

The directory /Users/mbg4/Library/texmf does not exist. However, there is a directory /Users/mbg4/Libary/texlive with the following content

texlive mbg4$ ls -ld $(find .)
drwxrwxr-x  6 mbg4  mbg4      204 16 Jan 12:16 .
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./.DS_Store
drwxrwxr-x  4 mbg4  mbg4      136 18 Mär  2011 ./2010
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/.DS_Store
drwxrwxr-x  6 mbg4  mbg4      204 29 Jul  2011 ./2010/texmf-var
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/texmf-var/.DS_Store
drwxrwxr-t  3 mbg4  mbg4      102 29 Jul  2011 ./2010/texmf-var/fonts
drwxrwxr-t  3 mbg4  mbg4      102 29 Jul  2011 ./2010/texmf-var/fonts/tfm
drwxrwxr-t  3 mbg4  mbg4      102 29 Jul  2011 ./2010/texmf-var/fonts/tfm/public
drwxrwxr-t  2 mbg4  mbg4       68 29 Jul  2011 ./2010/texmf-var/fonts/tfm/public/pandora
drwxrwxr-x  4 mbg4  mbg4      136 18 Mär  2011 ./2010/texmf-var/luatex-cache
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/texmf-var/luatex-cache/.DS_Store
drwxrwxr-x  4 mbg4  mbg4      136 18 Mär  2011 ./2010/texmf-var/luatex-cache/context
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/texmf-var/luatex-cache/context/.DS_Store
drwxrwxr-x  4 mbg4  mbg4      136 18 Mär  2011 ./2010/texmf-var/luatex-cache/context/e9159e878e989105b2ef2d27fad97515
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2010/texmf-var/luatex-cache/context/e9159e878e989105b2ef2d27fad97515/.DS_Store
drwxrwxr-x  2 mbg4  mbg4       68 11 Feb  2011 ./2010/texmf-var/luatex-cache/context/e9159e878e989105b2ef2d27fad97515/trees
drwxrwxr-x  3 mbg4  mbg4      102 18 Dez  2010 ./2010/texmf-var/texdoc
-rw-r--r--  1 mbg4  mbg4   818250 10 Jul  2011 ./2010/texmf-var/texdoc/cache-tlpdb.lua
drwxrwxr-x  4 mbg4  mbg4      136  1 Mär 12:50 ./2011
-rw-r--r--@ 1 mbg4  mbg4     6148  1 Mär 12:50 ./2011/.DS_Store
drwxrwxr-x  3 mbg4  mbg4      102 15 Dez  2011 ./2011/texmf-var
drwxrwxr-x  3 mbg4  mbg4      102 15 Dez  2011 ./2011/texmf-var/texdoc
-rw-r--r--  1 mbg4  mbg4   879199 15 Dez  2011 ./2011/texmf-var/texdoc/cache-tlpdb.lua
drwxrwxr-x  3 mbg4  mbg4      102 16 Jan 12:16 ./2014
drwxrwxr-x  4 mbg4  mbg4      136 16 Jan 21:32 ./2014/texmf-var
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts/pk
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts/pk/ljfour
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts/pk/ljfour/public
drwxrwxr-t  3 mbg4  mbg4      102 16 Jan 21:32 ./2014/texmf-var/fonts/pk/ljfour/public/wasy
-rw-r--r--  1 mbg4  mbg4    15908 16 Jan 21:32 ./2014/texmf-var/fonts/pk/ljfour/public/wasy/wasy10.657pk
drwxrwxr-x  3 mbg4  mbg4      102 16 Jan 12:16 ./2014/texmf-var/texdoc
-rw-r--r--  1 mbg4  mbg4  1142858 18 Feb 00:22 ./2014/texmf-var/texdoc/cache-tlpdb.lua

2. Tikz/PGFplots

It's commented out, so I think it should have no impact at all, but I still try to remove it.

  • But: I use IPE 7 for creating my graphics and had tried to embed a tikz tree in an IPE document. I removed the tikz tree, but I forgot to remove the usepackage commands in the IPE latex preamble of this document. And the resulting PDF (created with IPE) is used via includegraphics in my pdflatex document. So I'll see if it helps removing the tikz code from there.

3. Datatool

  • I don't use datatool, at least I'm not aware of it and a grep -ir datatool …

4. Unicode

not yet investigated

5. biber

  • biber seems to have a large impact. I use this biblatex setup currently:

    \usepackage[natbib=true, style=numeric-comp, backend=biber, defernumbers,useprefix,firstinits=true, maxnames=99, maxcitenames=3]{biblatex}

and as soon as I replace backend=biber to "backend=bibtex8`, the compiling time was reduced dramatically (from nearly 5 minutes for 98 pages to 37 seconds – only 1 pdflatex run).

Switching back to biber, I get a compiling time of 2min40sec – I'm quite confused, maybe some cache issues?

to be continued here: Why does biber increase compilation time of pdflatex runs dramatically (factor 2.8!!)?

Remarks:

  • I know that this is very little information at the moment and I'll do some more tests to find the problem, but some general suggestions and comments how to approach that problem would help me very much!
  • I'm using version control and I'll try to go back to the original state before the compiling time went up and try to compare the outputs.

Best Answer

It seems that using backend=biber is causing the amazing slow down,

I've asked a new focused question here to discuss this issue: Why does biber increase compilation time of pdflatex runs dramatically (factor 2.8!!)?

Performance comparison biber / bibtex8:

Test setup

  • my LaTeX document, limited to one long chapter by \includeonly,
  • 98 pages with many many references, many of them in large tables.
  • large .bib file (5.6 MB) with >3000 references, 350 of them cited in the document
  • I'm compiling from OS X terminal to measure the duration of the compilation:
    latexmk -C; time pdflatex main.tex

Test results

with biber:

  • biber cache removed once with rm -rfbiber --cache``

  • latexmk -C; time pdflatex Diss.tex
    Output written on Diss.pdf (83 pages, 2130562 bytes).
    real 2m17.499s
    user 1m47.626s
    sys 0m1.030s

  • biber Diss
    real 0m27.257s
    user 0m20.568s
    sys 0m1.247s

  • pdflatex Diss.tex Output written on Diss.pdf (95 pages, 2453607 bytes).
    real 4m7.708s
    user 3m27.864s
    sys 0m1.029s

  • again pdflatex Diss.tex
    Output written on Diss.pdf (95 pages, 2452370 bytes). real 4m33.769s user 3m36.491s sys 0m1.252s

with bibtex8

  • remove all aux and bbl files etc.
  • latexmk -C; time pdflatex Diss.tex
    Output written on Diss.pdf (82 pages, 2130383 bytes).
    real 0m48.470s
    user 0m19.161s
    sys 0m0.522s
  • time bibtex Diss
    (There were 200 error messages) sure :-(
    real 0m1.258s
    user 0m0.850s
    sys 0m0.061s

  • time pdflatex Diss.tex
    Output written on Diss.pdf (95 pages, 2452225 bytes).
    real 2m26.433s
    user 1m44.921s
    sys 0m0.932s

  • second pdflatex Diss.tex
    Output written on Diss.pdf (95 pages, 2450936 bytes).
    real 2m27.148s
    user 1m51.958s
    sys 0m0.953s

Related Question