[Tex/LaTex] Externalizing TikZ-Images takes very long

performancetikz-externaltikz-pgf

Introduction

Hello, I am facing very long compile times of more than 14 minutes for a ~130 page document, including around 40 PGF-Plots/TikZ-Images. Due to LaTeX's memory limits and (in fact) for speed up reasons, I decided to use the external-package.

Then I noticed, that each image takes up to 25 seconds to render (looking into my pdflatex.log from Miktex 2.9.
Edit: When compiling the same TikZ-images using TikzEdt, it takes only a split second from beginning to the end of compilation and the PDF being there.

Today, I created a MWE for demonstrating this effect, which seems to increase with the number of other packages loaded:

MWE (find it also here: Github)

\documentclass[]{article}

\usepackage{pgfplots}
\usepackage{graphicx} 

% Comment out the next three lines for faster compilation time
\usepgfplotslibrary{external}
\tikzexternalize
\tikzsetexternalprefix{figtemp/}

\begin{document}

\section{First}
    \begin{tikzpicture}
        \tikz \draw[thick,rounded corners=8pt]
        (0,0) -- (0,2) -- (1,3.25) -- (2,2) -- (2,0) -- (0,2) -- (2,2) -- (0,0) -- (2,0);
    \end{tikzpicture}

\section{Second}
    \begin{tikzpicture}
        \draw (-1.5,0) -- (1.5,0);
        \draw (0,-1.5) -- (0,1.5);
    \end{tikzpicture}

\section{Third}
    \begin{tikzpicture}
        \draw (0,0) arc (0:315:1.75cm and 1cm);
    \end{tikzpicture}

\end{document}

This example gives me the two respective pdflatex.log-Files, depending on whether externalization is enabled or disabled:

Externalization enabled

Total time: ~14.0 s

2019-03-05 13:21:31,240+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 13:21:31,246+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:31,246+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:31,246+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:33,607+0100 INFO  pdflatex - executing write18 shell command: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname "figtemp/TeX-TikZ-slow-MWE-figure0" "\def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}"
2019-03-05 13:21:33,885+0100 INFO  pdflatex - starting with command line: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname figtemp/TeX-TikZ-slow-MWE-figure0 \def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}
2019-03-05 13:21:33,891+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:33,891+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:33,891+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:36,177+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:21:36,326+0100 INFO  pdflatex - executing write18 shell command: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname "figtemp/TeX-TikZ-slow-MWE-figure1" "\def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}"
2019-03-05 13:21:36,614+0100 INFO  pdflatex - starting with command line: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname figtemp/TeX-TikZ-slow-MWE-figure1 \def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}
2019-03-05 13:21:36,619+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:36,620+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:36,620+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:39,167+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:21:39,311+0100 INFO  pdflatex - executing write18 shell command: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname "figtemp/TeX-TikZ-slow-MWE-figure2" "\def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}"
2019-03-05 13:21:39,583+0100 INFO  pdflatex - starting with command line: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname figtemp/TeX-TikZ-slow-MWE-figure2 \def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}
2019-03-05 13:21:39,592+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:39,592+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:39,592+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:41,817+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:21:42,029+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:21:42,485+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 13:21:42,489+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:42,489+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:42,489+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:45,223+0100 INFO  pdflatex - finishing with exit code 0

second run, using the externalized images

Total time: ~2.4 s

2019-03-05 14:15:03,628+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 14:15:03,629+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 14:15:03,629+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 14:15:03,629+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 14:15:06,080+0100 INFO  pdflatex - finishing with exit code 0

Externalization disabled

Total time: ~5.1 s

2019-03-05 13:23:19,734+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 13:23:19,742+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:23:19,742+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:23:19,742+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:23:22,045+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:23:22,476+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 13:23:22,485+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:23:22,485+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:23:22,485+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:23:24,842+0100 INFO  pdflatex - finishing with exit code 0

Conclusion

Externalizing my three (very simple) TikZ-Images blows up compilation time almost by a factor of 3 in the first run.

When using a tool like TikzEdt, all my TikZ-images compile in a split second.

What can I do, to speed this up?

Edit:

I found out that TikzEdt uses a precompiled .fmt file, what makes it so fast. As I've got no experience in precompiling, I could use some help for this and how to pass it to the tikzexternalize calls of pdflatex. In this question someone is tryin the same, but I can't seem to organize my preamble to work as desired and meanwhile speed up the externalized image compilation.

Best Answer

\tikzexternalize will always take longer than compiling without externalization on the first run. It needs to create the figures just like it has to do w/o tikzexternalize, and on top of the usual work, it has to export them to files. This includes overhead that is added by calling new instances of pdflatex and interacting with the OS separately for each picture.

The performance gain is best illustrated with a slightly more intensive TikZ example, as suggested by @Phelype Oleinik:

\documentclass{article}
\usepackage{pgfplots}
\usepackage{graphicx} 
\usepgfplotslibrary{external}
\tikzexternalize
\tikzsetexternalprefix{figtemp/}
\begin{document}
\section{First}
        \begin{tikzpicture}
                \foreach \i in {1,...,10000} {
                    \draw[thin,rounded corners=8pt] ({\i*0.05},0) -- ({\i*0.05},2) -- ({1+\i*0.05},3.25) -- ({2+\i*0.05},2) -- ({2+\i*0.05},0) -- ({\i*0.05},2) -- ({2+\i*0.05},2) -- ({\i*0.05},0) -- ({2+\i*0.05},0);
                }
        \end{tikzpicture}
\end{document}

The following relationships for run times should be expected:

[First run with \tikzexternalize] > [Run without \tikzexternalize] >>> [Later runs with \tikzexternalize].

Edit: To get a feeling for the additional work the first tikzexternalize run has to do, compared to the plain run w/o tikzexternalized, you can study the log files in the figtemp folder.

As a very crude way of getting a benchmark for how long it should take, you can see how fast your computer is in creating an empty tikz image in a single pdflatex run:

\documentclass{article}
%your preable as in the main document
\begin{document}
\section{First}
        \begin{tikzpicture}
            \coordinate (aa) at (0,0)
        \end{tikzpicture}
\end{document}

As this is (very) roughly equivalent to the extra work tikzexternalize has to do for each picture during the first run. So the first run should overall take:

t₀ + n t₁ seconds, where t₀ is the time it takes to compile the document without tikzexternalize and t₁ is the time the TikZ picture above takes to compile.

Related Solutions

[Tex/LaTex] Problem combining tikzexternalize, tikzscale and external graphics

Ok, I've found a workaround for this. As this has been bothering me for a long time and also includes some other issues to consider, I'll answer this step by step.

Task: Create a tikzpicture consisting of an external graphics (.png,.jpg,.pdf) that will not result in an error when used with tikzscale and \tikzexternalize.

Questions: Why a tikzpicture and not just \includegraphics?
-Because I want to add some elements to the picture, which is easy to do in tikz.
Why using tikzscale?
-Because I like to have my figures to have the same width as the text while keeping the fontsize of axes, nodes etc. independent of the figure's width. tikzscale is the tool for this.
Why \tikzexternalize?
-Not using \tikzexternalize means that every tikzpicture is recompiled every time, the document is compiled. Depending on the tikzpicture's complexity, this will take a lot of time.

The first approach:
The first approach can be seen in the original question. I placed a node in a tikzpicture and filled it with \includegraphics. This results in an error. The document is produced correctly, but the tikzpictureis not externalized i.e. saved in a separate .pdf.

The second approach:
In some earlier work I learned that pgfplots' \addplot graphics actually works with tikzscale and \tikzexternalize. The idea is now to put the exterael graphic in the axis environment and to hide the axes, ticks and labels. Let's assume we want a tikzpicture containing the following graphics (example-image-16x10.jpg included in the mwe package):

The code then reads as follows (the tikzpicture is written in a separate file test.tikz to load it with tikzscale's \includegraphics). note the ticks=none and axis lines=none options to hide the axes, ticks and labels:

\documentclass[10pt]{scrartcl}
\usepackage[english]{babel}
\usepackage{tikz}
\usetikzlibrary{calc}
\usetikzlibrary{external}
\tikzexternalize[shell escape=-enable-write18]
\usepackage{tikzscale}
\usepackage{pgfplots}
\usepackage{mwe}
\usepackage{filecontents}
\pgfplotsset{compat=1.10}
\begin{document}
\begin{filecontents}{test.tikz}
\begin{tikzpicture}
\begin{axis}[
    xmin=0,xmax=1,ymin=0,ymax=1,
    ticks=none,
    axis lines=none,
    ]
    \addplot graphics [xmin=0,xmax=1,ymin=0,ymax=1] {example-image-16x10};
\end{axis}
\end{tikzpicture}
\end{filecontents}
First we will include the picture as a normal graphics:
\begin{figure}[hb]
    \centering
    \includegraphics[width=0.5\textwidth]{example-image-16x10}
    \caption{This is the figure without using \textit{tikzpicture}.}
\end{figure}
\newline Next, we include the figure as a \textit{tikzpicture} using \textit{tikzscale}.
\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{test.tikz}
    \caption{This is my figure in \textit{tikzpicture}'s \textit{axis} environment. Unfortunaltely it is stretched into a square shape.}
    \end{figure}
\end{document}

When one compiles this code (twice), one will note, that the graphics in the tikzpicture is streched vertically, as pgfplots' axis environment tends to crate square shaped plots. To keep the original image's aspect ratio, one has to define a new \addplot command that will measure the original image's aspect ratio and scale the axes accordingly as demonstrated in an earlier question \addplot graphics: maintaining image's aspect ratio despite different scaling of axes.
The final code is then:

\documentclass[10pt]{scrartcl}
\usepackage[english]{babel}
\usepackage{tikz}
\usetikzlibrary{calc}
\usetikzlibrary{external}
\tikzexternalize[shell escape=-enable-write18]
\usepackage{tikzscale}
\usepackage{pgfplots}
\usepackage{mwe}
\usepackage{filecontents}
\pgfplotsset{compat=1.10}

\makeatletter
\newcommand\addplotgraphicsnatural[2][]{%
    \begingroup
    \pgfqkeys{/pgfplots/plot graphics}{#1}%
    \setbox0=\hbox{\includegraphics{#2}}%
    \pgfmathparse{\wd0/(\pgfkeysvalueof{/pgfplots/plot graphics/xmax} - \pgfkeysvalueof{/pgfplots/plot graphics/xmin})}%
    \let\xunit=\pgfmathresult
    \pgfmathparse{\ht0/(\pgfkeysvalueof{/pgfplots/plot graphics/ymax} - \pgfkeysvalueof{/pgfplots/plot graphics/ymin})}%
    \let\yunit=\pgfmathresult
    \xdef\marshal{%
        \noexpand\pgfplotsset{unit vector ratio={\xunit\space \yunit}}%
    }%
    \endgroup
    \marshal
    \addplot graphics[#1] {#2};
}   
\makeatother


\begin{document}
\begin{filecontents}{test.tikz}
\begin{tikzpicture}
\begin{axis}[
    xmin=0,xmax=1,ymin=0,ymax=1,
    ticks=none,
    axis lines=none,
    ]
    \addplotgraphicsnatural [xmin=0,xmax=1,ymin=0,ymax=1] {example-image-16x10};
\end{axis}
\end{tikzpicture}
\end{filecontents}
First we will include the picture as a normal graphics:
\begin{figure}[hb]
    \centering
    \includegraphics[width=0.5\textwidth]{example-image-16x10}
    \caption{This is the figure without using \textit{tikzpicture}.}
\end{figure}
\newline Next, we include the figure as a \textit{tikzpicture} using \textit{tikzscale}.
\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{test.tikz}
    \caption{This is my figure in \textit{tikzpicture}'s \textit{axis} environment. Unfortunaltely it is stretched into a square shape.}
    \end{figure}
\end{document}

This will produce the desired document as well as the externalized figure. I don't know whether there is a more elegant way to do this, but it works.

[Tex/LaTex] Cannot run tikz externalize with lualatex but it used to work

You need the shellesc package

\documentclass{article}
\usepackage{shellesc}
% main document, called main.tex
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize % activate!
\begin{document}
\loggingall
\begin{tikzpicture}
  \node {root}
    child {node {left}}
    child {node {right}
      child {node {child}}
      child {node {child}}
    };
\end{tikzpicture}

A simple image is \tikz \fill (0,0) circle(5pt);.
\end{document}