Skip to contentSkip to conversation

Enable parallel runs of pdflatex

PATCH Superseded Akira Yokosawa 2 participants 3 messages Jan 27, 2022 Kbuild

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Enable parallel runs of pdflatex

Parallel build of PDF figures and .fcv snippets by "make -jN" worked when a single PDF target was given to "make", such as "make neatfreak; make -j4 eb".
On the other hand, "make -j3 2c 1c eb" ended up in various errors during parallel runs of pdflatex.

It turns out that the cause of the error is the \include{} commands in appendix/appendix.tex and perfbook-lt.tex.
\include{} switches to its own .aux file, in this case, appendix/questions.aux, appendix/toyrcu.aux, and so on. As their names are common to different PDF targets, those .aux files can be overwritten/corrupted by parallel runs of pdflatex.

\include{} implies \cleardoublepage both in front of and next to it. In perfbook, included LaTeX sources have \chapter{} commands and the implied page breaks are redundant.

By replacing \include{} with \input{}, parallel runs of pdflatex can be enabled.

There were a couple of minor issues in Makefile WRT parallel runs of runfirstlatex.sh.

When some of perfbook-xxx.tex files already existed, runfirstlatex.sh for other main perfbook-yyy.tex could be invoked prematurely before that .tex file was ready.

Fix it by adding stricter dependencies of perfbook-xxx.aux on perfbook-xxx.tex.
Also remove some redundant dependencies around here.

Note: LATEXSOURCES contains perfbook-lt.tex

---
Hi Paul,

For quite a while, I was wondering why "make -j3 2c 1c eb" didn't work as expected.

I think I have managed to get it work.

But changes in Makefile need extra testing. Especially, those involving parallelizing.

So I'd like you to give this a fair amount of testing before pushing it.

Thanks, Akira
--

 Makefile              | 6 +++---
 appendix/appendix.tex | 8 ++++----
 perfbook-lt.tex       | 4 ++--
 3 files changed, 9 insertions(+), 9 deletions(-)
Makefile +3 -3
index 6ee849fd..9946be3a 100644
@@ -222,14 +222,14 @@ $(PDFTARGETS): %.pdf: %.tex %.bbl
 $(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux $(BIBSOURCES)
 	bibtex $(basename $@)
 
-$(PDFTARGETS:.pdf=.aux): $(LATEXGENERATED) $(LATEXSOURCES) $(LST_SOURCES)
+$(PDFTARGETS:.pdf=.aux): %.aux: %.tex $(LATEXGENERATED) $(LST_SOURCES)
 ifeq ($(NEWTXTEXT),)
 	$(error Font package 'newtx' not found. See #9 in FAQ-BUILD.txt)
 endif
 	sh utilities/runfirstlatex.sh $(basename $@)
 
-autodate.tex: perfbook-lt.tex $(LATEXSOURCES) $(BIBSOURCES) \
-    $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \
+autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(PDFTARGETS_OF_EPS) \
+    $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \
     $(GITREFSTAGS) utilities/autodate.sh
 	sh utilities/autodate.sh
 
appendix/appendix.tex +4 -4
index f2640157..edcfe7ef 100644
@@ -2,9 +2,9 @@
 % mainfile: ../perfbook.tex
 % SPDX-License-Identifier: CC-BY-SA-3.0
 
-\include{appendix/questions/questions}
-\include{appendix/toyrcu/toyrcu}
-\include{appendix/whymb/whymemorybarriers}
-\include{appendix/styleguide/styleguide}
+\input{appendix/questions/questions}
+\input{appendix/toyrcu/toyrcu}
+\input{appendix/whymb/whymemorybarriers}
+\input{appendix/styleguide/styleguide}
 \renewcommand{\bottomtitlespace}{.08\textheight}
 \QuickQuizAnswers
perfbook-lt.tex +2 -2
index 085ce55d..d70dde66 100644
@@ -667,12 +667,12 @@
 
 % Credits
 \setcounter{secnumdepth}{-1} % surpress section numbering in backmatter
-\include{ack}
+\input{ack}
 
 % Index if enabled
 \IfIndexOn{
 \IfIndexHier{
-\include{indexsee}
+\input{indexsee}
 }{}
 \phantomsection
 \IfEbookSize{

base-commit: f24ccfbd345eab8a70d6c08e71c84f18df46a3a1

Signature · 1 line
2.17.1
#1 lore raw
Quoted 40 lines · Akira Yokosawa

Parallel build of PDF figures and .fcv snippets by "make -jN" worked when a single PDF target was given to "make", such as "make neatfreak; make -j4 eb".
On the other hand, "make -j3 2c 1c eb" ended up in various errors during parallel runs of pdflatex.

It turns out that the cause of the error is the \include{} commands in appendix/appendix.tex and perfbook-lt.tex.
\include{} switches to its own .aux file, in this case, appendix/questions.aux, appendix/toyrcu.aux, and so on. As their names are common to different PDF targets, those .aux files can be overwritten/corrupted by parallel runs of pdflatex.

\include{} implies \cleardoublepage both in front of and next to it. In perfbook, included LaTeX sources have \chapter{} commands and the implied page breaks are redundant.

By replacing \include{} with \input{}, parallel runs of pdflatex can be enabled.

There were a couple of minor issues in Makefile WRT parallel runs of runfirstlatex.sh.

When some of perfbook-xxx.tex files already existed, runfirstlatex.sh for other main perfbook-yyy.tex could be invoked prematurely before that .tex file was ready.

Fix it by adding stricter dependencies of perfbook-xxx.aux on perfbook-xxx.tex.
Also remove some redundant dependencies around here.

Note: LATEXSOURCES contains perfbook-lt.tex

---
Hi Paul,

For quite a while, I was wondering why "make -j3 2c 1c eb" didn't work as expected.

Quoted 7 lines

I think I have managed to get it work.

But changes in Makefile need extra testing. Especially, those involving parallelizing.

So I'd like you to give this a fair amount of testing before pushing it.

OK, I am applying it to an experimental repo. Let's give it a go:

	$ time make -j32 2c 1c eb a4 nq sf sfnq
	[ . . . ]
	'perfbook-sf.pdf' is ready.
	'perfbook-1c.pdf' is ready.
	'perfbook.pdf' is ready.
	'perfbook-sfnq.pdf' is ready.
	'perfbook-a4.pdf' is ready.
	'perfbook-eb.pdf' is ready.
	'perfbook-nq.pdf' is ready.
	real    2m31.412s
	user    16m45.778s
	sys     1m48.385s

Not bad, actually! Might be worth redoing my release scripts to take advantage of this. ;-)

I (very) quickly glanced at the PDFs, and they look OK.

I then did a "make clean" and:

	$ time make
	[ . . . ]
	'perfbook.pdf' is ready.
	utilities/punctcheck.sh
	utilities/cleverefcheck.sh
	real    1m47.823s
	user    1m45.715s
	sys     0m1.226s

Another "make clean" and:

	$ time make -j32
	[ . . . ]
	'perfbook.pdf' is ready.
	utilities/punctcheck.sh
	utilities/cleverefcheck.sh
	real    1m49.484s
	user    1m47.206s
	sys     0m1.437s

So, as one would expect, the sweet spot for -j is when rebuilding all the images from scratch or when building multiple PDFs.

Nice!

Huh. It would be good to have a way of diff-ing PDF files. Then a single-threaded-generated PDF could be automatically compared to its counterpart from a concurrent build. But the search engines were not kind to me.

Thoughts?

Thanx, Paul

Quoted 71 lines

Thanks, Akira
--

 Makefile              | 6 +++---
 appendix/appendix.tex | 8 ++++----
 perfbook-lt.tex       | 4 ++--
 3 files changed, 9 insertions(+), 9 deletions(-)
Makefile +3 -3
index 6ee849fd..9946be3a 100644
@@ -222,14 +222,14 @@ $(PDFTARGETS): %.pdf: %.tex %.bbl
 $(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux $(BIBSOURCES)
 	bibtex $(basename $@)
 
-$(PDFTARGETS:.pdf=.aux): $(LATEXGENERATED) $(LATEXSOURCES) $(LST_SOURCES)
+$(PDFTARGETS:.pdf=.aux): %.aux: %.tex $(LATEXGENERATED) $(LST_SOURCES)
 ifeq ($(NEWTXTEXT),)
 	$(error Font package 'newtx' not found. See #9 in FAQ-BUILD.txt)
 endif
 	sh utilities/runfirstlatex.sh $(basename $@)
 
-autodate.tex: perfbook-lt.tex $(LATEXSOURCES) $(BIBSOURCES) \
-    $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \
+autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(PDFTARGETS_OF_EPS) \
+    $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \
     $(GITREFSTAGS) utilities/autodate.sh
 	sh utilities/autodate.sh
 
appendix/appendix.tex +4 -4
index f2640157..edcfe7ef 100644
@@ -2,9 +2,9 @@
 % mainfile: ../perfbook.tex
 % SPDX-License-Identifier: CC-BY-SA-3.0
 
-\include{appendix/questions/questions}
-\include{appendix/toyrcu/toyrcu}
-\include{appendix/whymb/whymemorybarriers}
-\include{appendix/styleguide/styleguide}
+\input{appendix/questions/questions}
+\input{appendix/toyrcu/toyrcu}
+\input{appendix/whymb/whymemorybarriers}
+\input{appendix/styleguide/styleguide}
 \renewcommand{\bottomtitlespace}{.08\textheight}
 \QuickQuizAnswers
perfbook-lt.tex +2 -2
index 085ce55d..d70dde66 100644
@@ -667,12 +667,12 @@
 
 % Credits
 \setcounter{secnumdepth}{-1} % surpress section numbering in backmatter
-\include{ack}
+\input{ack}
 
 % Index if enabled
 \IfIndexOn{
 \IfIndexHier{
-\include{indexsee}
+\input{indexsee}
 }{}
 \phantomsection
 \IfEbookSize{

base-commit: f24ccfbd345eab8a70d6c08e71c84f18df46a3a1

Signature · 1 line
2.17.1
#2 lore raw
Quoted 61 lines · Paul E. McKenney
Quoted 40 lines · Akira Yokosawa

Parallel build of PDF figures and .fcv snippets by "make -jN" worked when a single PDF target was given to "make", such as "make neatfreak; make -j4 eb".
On the other hand, "make -j3 2c 1c eb" ended up in various errors during parallel runs of pdflatex.

It turns out that the cause of the error is the \include{} commands in appendix/appendix.tex and perfbook-lt.tex.
\include{} switches to its own .aux file, in this case, appendix/questions.aux, appendix/toyrcu.aux, and so on. As their names are common to different PDF targets, those .aux files can be overwritten/corrupted by parallel runs of pdflatex.

\include{} implies \cleardoublepage both in front of and next to it. In perfbook, included LaTeX sources have \chapter{} commands and the implied page breaks are redundant.

By replacing \include{} with \input{}, parallel runs of pdflatex can be enabled.

There were a couple of minor issues in Makefile WRT parallel runs of runfirstlatex.sh.

When some of perfbook-xxx.tex files already existed, runfirstlatex.sh for other main perfbook-yyy.tex could be invoked prematurely before that .tex file was ready.

Fix it by adding stricter dependencies of perfbook-xxx.aux on perfbook-xxx.tex.
Also remove some redundant dependencies around here.

Note: LATEXSOURCES contains perfbook-lt.tex

---
Hi Paul,

For quite a while, I was wondering why "make -j3 2c 1c eb" didn't work as expected.

Quoted 7 lines

I think I have managed to get it work.

But changes in Makefile need extra testing. Especially, those involving parallelizing.

So I'd like you to give this a fair amount of testing before pushing it.

OK, I am applying it to an experimental repo. Let's give it a go:

	$ time make -j32 2c 1c eb a4 nq sf sfnq
	[ . . . ]
	'perfbook-sf.pdf' is ready.
	'perfbook-1c.pdf' is ready.
	'perfbook.pdf' is ready.
	'perfbook-sfnq.pdf' is ready.
	'perfbook-a4.pdf' is ready.
	'perfbook-eb.pdf' is ready.
	'perfbook-nq.pdf' is ready.
Quoted 6 lines
	real    2m31.412s
	user    16m45.778s
	sys     1m48.385s

Not bad, actually! Might be worth redoing my release scripts to take advantage of this. ;-)

Nice!

Quoted 30 lines

I (very) quickly glanced at the PDFs, and they look OK.

I then did a "make clean" and:

	$ time make
	[ . . . ]
	'perfbook.pdf' is ready.
	utilities/punctcheck.sh
	utilities/cleverefcheck.sh
	real    1m47.823s
	user    1m45.715s
	sys     0m1.226s

Another "make clean" and:

	$ time make -j32
	[ . . . ]
	'perfbook.pdf' is ready.
	utilities/punctcheck.sh
	utilities/cleverefcheck.sh
	real    1m49.484s
	user    1m47.206s
	sys     0m1.437s

So, as one would expect, the sweet spot for -j is when rebuilding all the images from scratch or when building multiple PDFs.

Quoted 8 lines

Nice!

Huh. It would be good to have a way of diff-ing PDF files. Then a single-threaded-generated PDF could be automatically compared to its counterpart from a concurrent build. But the search engines were not kind to me.

Thoughts?

How about comparing .log files of two builds?

In my trial, "diff -u perfbook-single.log perfbook-multi.log" yields only a hunk of build time difference as shown below:

    --- perfbook-single.log	2022-01-27 14:30:01.659959433 +0900

    +++ perfbook-multi.log	2022-01-27 14:33:57.556196036 +0900

    @@ -1,4 +1,4 @@

    -This is pdfTeX, Version 3.141592653-2.6-1.40.23 (TeX Live 2021) (preloaded format=pdflatex 2022.1.24)  27 JAN 2022 14:28

    +This is pdfTeX, Version 3.141592653-2.6-1.40.23 (TeX Live 2021) (preloaded format=pdflatex 2022.1.24)  27 JAN 2022 14:32

     entering extended mode

      restricted \write18 enabled.

      %&-line parsing enabled.

Not a direct comparison, but the log has a summary near the end:

    Here is how much of TeX's memory you used:
     74049 strings out of 479599
     1597759 string characters out of 5876039
     1950680 words of memory out of 5000000
     80683 multiletter control sequences out of 15000+600000
     692091 words of font info for 721 fonts, out of 8000000 for 9000
     1816 hyphenation exceptions out of 8191
     105i,19n,133p,1187b,1660s stack positions out of 5000i,500n,10000p,200000b,80000s

     [long list of referenced font files]

    Output written on perfbook.pdf (638 pages, 8031215 bytes).
    PDF statistics:
     25510 PDF objects out of 26593 (max. 8388607)
     24061 compressed objects within 241 object streams
     7119 named destinations out of 7423 (max. 500000)
     166134 words of extra memory for PDF output out of 184870 (max. 10000000)

I think any visible difference in two PDFs is likely to be reflected in this summary.

Indirect hints they may be, at least they provide partial assurance.

Thanks, Akira

Quoted 10 lines

Thanx, Paul

Quoted 7 lines

Thanks, Akira
--

 Makefile              | 6 +++---
 appendix/appendix.tex | 8 ++++----
 perfbook-lt.tex       | 4 ++--
 3 files changed, 9 insertions(+), 9 deletions(-)

[...]