[Tex/LaTex] Lualatex, Pdflatex fail with \externalize

luatexpdftextikz-externaltikz-pgf

Here is my problem:
I have a 100+ page document with ~100 tikz plots.
Some plots contain 9000+ points, and compiling them with pdflatex -shell-escape (or whatever main_memory_size) fails.
I can compile it with lualatex though, but it takes forever.
I decided to use:

    \usetikzlibrary{external}
    \tikzset{external/system call={lualatex \tikzexternalcheckshellescape -shell-                escape -halt-on-error -interaction=nonstopmode -jobname "\image" "\texsource"}}
    \tikzexternalize[prefix=TikzFigures/]
    \tikzset{external/force remake}

And it fails with

    "16684 words still in use, bla-bla ...may be you did not enable shell-escape         ...bla-bla write18...please refer to picture-17.log",

but there is no error message in that log.
If I switch back to pdflatex, with proper adjustments, error log says again that I'm out of memory.
I thought lualatex manages memory dynamically, and externalize should work when regular compilation fails, but I got the opposite case.
I have 4-core i7 with 8Gb RAM, hardly half is used.. What to do?
I use TexNic center and MikTex, so in the editor options of course I hap put the -shell-escape as well as -enable-write18 in all possible combinations for both pdflatex output profile and lualatex profile.
I have a main .tex file with several included chapters, with separate tikzpistures in .tex files as well.
Moreover, if I delete all the images from the TikzFigures/ folder, now both lualatex and pdflatex fail already at the first image. If I comment out lines responsible for externalization, it compiles. It takes 5 minutes and 500Mb of memory, but it compiles with one warning about scrbook\float@addtolist that it is about to get outdated.

    \documentclass[fontsize=12,BCOR=0mm]{scrbook}
    \usepackage{etex}
    \usepackage[USenglish]{babel}
    \usepackage{url}
    \usepackage{natbib}
    \usepackage{breakcites}
    \usepackage{enumerate}
    \usepackage{caption}
    \usepackage{nomencl}
    \usepackage{setspace}
    \usepackage{listings}
    \usepackage{booktabs}
    \usepackage{tikz}
    \usepackage{tikzscale}
    \usepackage{pgf}
    \usepackage{pgfplots}
    \usepackage{subfig}
    \pgfplotsset{compat=1.8}
    \pgfplotsset{colormap={my_hot}{[1cm]color(0cm)=(black);
    color(3cm)=(red);color(4cm)=(orange);color(5cm)=(yellow);color(8cm)=(white)}}

    \pgfplotsset{every axis/.append style={line width=1pt, 
    tick style={tick align=outside, line width=0.6pt}}}
    \pgfplotsset{ticks=both}%/.append style={thin}}
    \pgfplotsset{ylabel shift=-4pt}
    \pgfplotsset{yticklabel style={xshift=0.1cm}}
    \pgfplotsset{label style={font=\small}}
    \pgfplotsset{every axis legend/.style={inner ysep=1.5pt}}
    \usetikzlibrary{automata, arrows}
    \usepackage{color}
    \definecolor{light-gray}{gray}{0.95}
    \lstset{backgroundcolor=\color{light-gray}}%, xleftmargin=\noindent} 


    \usepackage{multibib}


    \usetikzlibrary{external}
    \tikzset{external/system call={lualatex \tikzexternalcheckshellescape 
    -enable-write18 -halt-on-error -interaction=nonstopmode 
    -jobname "\image" "\texsource"}}
    \tikzexternalize[prefix=TikzFigures/]
    \tikzset{external/force remake}

    \makenomenclature
    \newcommand{\keywords}[1]{\par\addvspace\baselineskip
    \noindent\keywordname\enspace\ignorespaces#1}
    \setcapindent{0pt} 
    \renewcommand{\nomname}{List of Symbols}

    \makeatletter 
    \renewenvironment{thebibliography}[1]
     {\subsection*{\refname}%
      \@mkboth{\MakeUppercase\refname}{\MakeUppercase\refname}
      \list{\@biblabel{\@arabic\c@enumiv}}
    {\settowidth\labelwidth{\@biblabel{#1}}
    \leftmargin\labelwidth
    \advance\leftmargin\labelsep
    \@openbib@code
    \usecounter{enumiv}
    \let\p@enumiv\@empty
    \renewcommand\theenumiv{\@arabic\c@enumiv}}
      \sloppy
      \clubpenalty4000
      \@clubpenalty \clubpenalty
      \widowpenalty4000%
      \sfcode`\.\@m}
     {\def\@noitemerr
       {\@latex@warning{Empty `thebibliography' environment}}
      \endlist} 
    \makeatother

    \makeindex
    \begin{document}
    \pagenumbering{roman}
    \input{title.tex}
    \maketitle


    \include{abstract}

    \include{acknowledgements}

    \tableofcontents
    \listoftables 
    \listoffigures 

    \clearpage
    \pagenumbering{arabic}

    \include{Abstract}
    \include{Introduction}
    \include{Algorithms}
    \include{Modeling}
    \include{Performance}
    \include{Conclusions}


    \pagestyle{plain}
    \cleardoublepage
    \addcontentsline{toc}{chapter}{List of Symbols}

    \setlength{\nomitemsep}{-0.7\parsep}

    \begin{spacing}{0.4}
    \printnomenclature [14mm]
    \end{spacing}

    \bibliography{paper-bibfile-short}


    \end{document}

! Package tikz Error: Sorry, the system call 'lualatex --shell-escape -intera
ion=nonstopmode -jobname "TikzFigures/phd_thesis-figure17" "\def\tikzexternal
aljob{phd_thesis}\input{phd_thesis}"' did NOT result in a usable output file
ikzFigures/phd_thesis-figure17' (expected one of .pdf:.jpg:.jpeg:.png:). Plea
verify that you have enabled system calls. For pdflatex, this is 'pdflatex -
ell-escape'. Sometimes it is also named 'write 18' or something like that. Or
aybe the command simply failed? Error messages can be found in 'TikzFigures/p
_thesis-figure17.log'. If you continue now, I'll try to typeset the picture.

Best Answer

Why not using pdflatex with increasing memory manually.

here is an example (numbers are arbitrary large, you may want try some other values)

  \tikzset{external/system call= {pdflatex -save-size=80000 
                           -pool-size=10000000 
                           -extra-mem-top=50000000 
                           -extra-mem-bot=10000000 
                           -main-memory=90000000 
                           \tikzexternalcheckshellescape 
                           -halt-on-error 
                           -interaction=batchmode
                           -jobname "\image" "\texsource"}}

Related Solutions

[Tex/LaTex] Using gnuplot in tikzpicture environment using pgf-package

It´s been at while but i couldn´t find the time untill now. So here ist what i have done to solve my problem.

Example

\documentclass [12pt, a4paper, headinclude, titlepage]{scrartcl}                
\usepackage[ngerman]{babel}                                                     
\usepackage[fixlanguage]{babelbib}
\selectbiblanguage{german}
\usepackage[utf8]{inputenc}                                                     
\usepackage[T1]{fontenc}    
\usepackage[a4paper]{geometry}                                                                                                                                                  
\usepackage[fleqn]{amsmath}                                                                                                                 
\usepackage{pgfplots}                                                           
\usepackage{pgf}
\usepackage{tikz}                                                               
\usepackage{float}                                                              

\pgfplotsset{width=7cm, compat=newest}

\begin{document}
\begin{figure}[H] 
\centering
\begin{tikzpicture}
 \begin{axis}
     [  ylabel=$\kappa$ / (S/m) , xlabel=c / (mol/m$^3$), title=\textbf{c gegen $\kappa$}, 
        xmin=0, xmax=130,
        ymin=0,  ymax=0.3,
        width=0.6\textwidth, height=0.45\textwidth, x axis line style={-}, y axis line                     style={-},
      ]     
         \addplot [only marks, gray, raw gnuplot, id=raw3] function{plot 'Daten.txt' u 1:2};
             \addplot [orange, dashed, raw gnuplot, id=raw3] function{plot 'ber.txt' u 1:3};
     \end{axis} 
\end{tikzpicture}
\caption{Auftragung der Messwerte c gegen $\kappa$ (grau) und der berechneten Werte (orange)}
\end{figure}
\end{document}

My configurations: - Gnuplot and LaTeX installed - path variable checked (windows) - included "-shell-escape" in the pdfLaTeX option (easy in texstudio) - ensured that the name of the texfile includes no space, umlauts or other special characters; otherwise the .table file is not created. - If the memory of LaTeX is exceeded, swaped the plot in an extra file and included it in the main file

Run pdfLaTeX twice and voila there should be a plot like the one below.

Here is the plot: plot createt with gnuplot in latex

[Tex/LaTex] Problem combining tikzexternalize, tikzscale and external graphics

Ok, I've found a workaround for this. As this has been bothering me for a long time and also includes some other issues to consider, I'll answer this step by step.

Task: Create a tikzpicture consisting of an external graphics (.png,.jpg,.pdf) that will not result in an error when used with tikzscale and \tikzexternalize.

Questions: Why a tikzpicture and not just \includegraphics?
-Because I want to add some elements to the picture, which is easy to do in tikz.
Why using tikzscale?
-Because I like to have my figures to have the same width as the text while keeping the fontsize of axes, nodes etc. independent of the figure's width. tikzscale is the tool for this.
Why \tikzexternalize?
-Not using \tikzexternalize means that every tikzpicture is recompiled every time, the document is compiled. Depending on the tikzpicture's complexity, this will take a lot of time.

The first approach:
The first approach can be seen in the original question. I placed a node in a tikzpicture and filled it with \includegraphics. This results in an error. The document is produced correctly, but the tikzpictureis not externalized i.e. saved in a separate .pdf.

The second approach:
In some earlier work I learned that pgfplots' \addplot graphics actually works with tikzscale and \tikzexternalize. The idea is now to put the exterael graphic in the axis environment and to hide the axes, ticks and labels. Let's assume we want a tikzpicture containing the following graphics (example-image-16x10.jpg included in the mwe package):

The code then reads as follows (the tikzpicture is written in a separate file test.tikz to load it with tikzscale's \includegraphics). note the ticks=none and axis lines=none options to hide the axes, ticks and labels:

\documentclass[10pt]{scrartcl}
\usepackage[english]{babel}
\usepackage{tikz}
\usetikzlibrary{calc}
\usetikzlibrary{external}
\tikzexternalize[shell escape=-enable-write18]
\usepackage{tikzscale}
\usepackage{pgfplots}
\usepackage{mwe}
\usepackage{filecontents}
\pgfplotsset{compat=1.10}
\begin{document}
\begin{filecontents}{test.tikz}
\begin{tikzpicture}
\begin{axis}[
    xmin=0,xmax=1,ymin=0,ymax=1,
    ticks=none,
    axis lines=none,
    ]
    \addplot graphics [xmin=0,xmax=1,ymin=0,ymax=1] {example-image-16x10};
\end{axis}
\end{tikzpicture}
\end{filecontents}
First we will include the picture as a normal graphics:
\begin{figure}[hb]
    \centering
    \includegraphics[width=0.5\textwidth]{example-image-16x10}
    \caption{This is the figure without using \textit{tikzpicture}.}
\end{figure}
\newline Next, we include the figure as a \textit{tikzpicture} using \textit{tikzscale}.
\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{test.tikz}
    \caption{This is my figure in \textit{tikzpicture}'s \textit{axis} environment. Unfortunaltely it is stretched into a square shape.}
    \end{figure}
\end{document}

When one compiles this code (twice), one will note, that the graphics in the tikzpicture is streched vertically, as pgfplots' axis environment tends to crate square shaped plots. To keep the original image's aspect ratio, one has to define a new \addplot command that will measure the original image's aspect ratio and scale the axes accordingly as demonstrated in an earlier question \addplot graphics: maintaining image's aspect ratio despite different scaling of axes.
The final code is then:

\documentclass[10pt]{scrartcl}
\usepackage[english]{babel}
\usepackage{tikz}
\usetikzlibrary{calc}
\usetikzlibrary{external}
\tikzexternalize[shell escape=-enable-write18]
\usepackage{tikzscale}
\usepackage{pgfplots}
\usepackage{mwe}
\usepackage{filecontents}
\pgfplotsset{compat=1.10}

\makeatletter
\newcommand\addplotgraphicsnatural[2][]{%
    \begingroup
    \pgfqkeys{/pgfplots/plot graphics}{#1}%
    \setbox0=\hbox{\includegraphics{#2}}%
    \pgfmathparse{\wd0/(\pgfkeysvalueof{/pgfplots/plot graphics/xmax} - \pgfkeysvalueof{/pgfplots/plot graphics/xmin})}%
    \let\xunit=\pgfmathresult
    \pgfmathparse{\ht0/(\pgfkeysvalueof{/pgfplots/plot graphics/ymax} - \pgfkeysvalueof{/pgfplots/plot graphics/ymin})}%
    \let\yunit=\pgfmathresult
    \xdef\marshal{%
        \noexpand\pgfplotsset{unit vector ratio={\xunit\space \yunit}}%
    }%
    \endgroup
    \marshal
    \addplot graphics[#1] {#2};
}   
\makeatother


\begin{document}
\begin{filecontents}{test.tikz}
\begin{tikzpicture}
\begin{axis}[
    xmin=0,xmax=1,ymin=0,ymax=1,
    ticks=none,
    axis lines=none,
    ]
    \addplotgraphicsnatural [xmin=0,xmax=1,ymin=0,ymax=1] {example-image-16x10};
\end{axis}
\end{tikzpicture}
\end{filecontents}
First we will include the picture as a normal graphics:
\begin{figure}[hb]
    \centering
    \includegraphics[width=0.5\textwidth]{example-image-16x10}
    \caption{This is the figure without using \textit{tikzpicture}.}
\end{figure}
\newline Next, we include the figure as a \textit{tikzpicture} using \textit{tikzscale}.
\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{test.tikz}
    \caption{This is my figure in \textit{tikzpicture}'s \textit{axis} environment. Unfortunaltely it is stretched into a square shape.}
    \end{figure}
\end{document}

This will produce the desired document as well as the externalized figure. I don't know whether there is a more elegant way to do this, but it works.

Best Answer

Related Solutions

[Tex/LaTex] Using gnuplot in tikzpicture environment using pgf-package

[Tex/LaTex] Problem combining tikzexternalize, tikzscale and external graphics

Related Question