Enable parallel runs of pdflatex
PATCH 1/1 Enable parallel runs of pdflatex
Parallel build of PDF figures and .fcv snippets by "make -jN" worked when a single PDF target was given to "make", such as "make neatfreak; make -j4 eb".
On the other hand, "make -j3 2c 1c eb" ended up in various errors during parallel runs of pdflatex.
It turns out that the cause of the error is the \include{} commands in appendix/appendix.tex and perfbook-lt.tex.
\include{} switches to its own .aux file, in this case, appendix/questions.aux, appendix/toyrcu.aux, and so on. As their names are common to different PDF targets, those .aux files can be overwritten/corrupted by parallel runs of pdflatex.
\include{} implies \cleardoublepage both in front of and next to it. In perfbook, included LaTeX sources have \chapter{} commands and the implied page breaks are redundant.
By replacing \include{} with \input{}, parallel runs of pdflatex can be enabled.
There were a couple of minor issues in Makefile WRT parallel runs of runfirstlatex.sh.
When some of perfbook-xxx.tex files already existed, runfirstlatex.sh for other main perfbook-yyy.tex could be invoked prematurely before that .tex file was ready.
Fix it by adding stricter dependencies of perfbook-xxx.aux on perfbook-xxx.tex.
Also remove some redundant dependencies around here.
Note: LATEXSOURCES contains perfbook-lt.tex
---
Hi Paul,
For quite a while, I was wondering why "make -j3 2c 1c eb" didn't work as expected.
I think I have managed to get it work.
But changes in Makefile need extra testing. Especially, those involving parallelizing.
So I'd like you to give this a fair amount of testing before pushing it.
Thanks, Akira
--
Makefile | 6 +++--- appendix/appendix.tex | 8 ++++---- perfbook-lt.tex | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-)
Makefile +3 -3
@@ -222,14 +222,14 @@ $(PDFTARGETS): %.pdf: %.tex %.bbl $(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux $(BIBSOURCES) bibtex $(basename $@) -$(PDFTARGETS:.pdf=.aux): $(LATEXGENERATED) $(LATEXSOURCES) $(LST_SOURCES) +$(PDFTARGETS:.pdf=.aux): %.aux: %.tex $(LATEXGENERATED) $(LST_SOURCES) ifeq ($(NEWTXTEXT),) $(error Font package 'newtx' not found. See #9 in FAQ-BUILD.txt) endif sh utilities/runfirstlatex.sh $(basename $@) -autodate.tex: perfbook-lt.tex $(LATEXSOURCES) $(BIBSOURCES) \ - $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \ +autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(PDFTARGETS_OF_EPS) \ + $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \ $(GITREFSTAGS) utilities/autodate.sh sh utilities/autodate.sh
appendix/appendix.tex +4 -4
@@ -2,9 +2,9 @@ % mainfile: ../perfbook.tex % SPDX-License-Identifier: CC-BY-SA-3.0 -\include{appendix/questions/questions} -\include{appendix/toyrcu/toyrcu} -\include{appendix/whymb/whymemorybarriers} -\include{appendix/styleguide/styleguide} +\input{appendix/questions/questions} +\input{appendix/toyrcu/toyrcu} +\input{appendix/whymb/whymemorybarriers} +\input{appendix/styleguide/styleguide} \renewcommand{\bottomtitlespace}{.08\textheight} \QuickQuizAnswers
perfbook-lt.tex +2 -2
@@ -667,12 +667,12 @@ % Credits \setcounter{secnumdepth}{-1} % surpress section numbering in backmatter -\include{ack} +\input{ack} % Index if enabled \IfIndexOn{ \IfIndexHier{ -\include{indexsee} +\input{indexsee} }{} \phantomsection \IfEbookSize{
base-commit: f24ccfbd345eab8a70d6c08e71c84f18df46a3a1
Signature · 1 line
2.17.1
Quoted 40 lines · Akira Yokosawa
Parallel build of PDF figures and .fcv snippets by "make -jN" worked when a single PDF target was given to "make", such as "make neatfreak; make -j4 eb".
On the other hand, "make -j3 2c 1c eb" ended up in various errors during parallel runs of pdflatex.It turns out that the cause of the error is the \include{} commands in appendix/appendix.tex and perfbook-lt.tex.
\include{} switches to its own .aux file, in this case, appendix/questions.aux, appendix/toyrcu.aux, and so on. As their names are common to different PDF targets, those .aux files can be overwritten/corrupted by parallel runs of pdflatex.\include{} implies \cleardoublepage both in front of and next to it. In perfbook, included LaTeX sources have \chapter{} commands and the implied page breaks are redundant.
By replacing \include{} with \input{}, parallel runs of pdflatex can be enabled.
There were a couple of minor issues in
MakefileWRT parallel runs ofrunfirstlatex.sh.When some of perfbook-xxx.tex files already existed,
runfirstlatex.shfor other main perfbook-yyy.tex could be invoked prematurely before that .tex file was ready.Fix it by adding stricter dependencies of perfbook-xxx.aux on perfbook-xxx.tex.
Also remove some redundant dependencies around here.Note: LATEXSOURCES contains perfbook-lt.tex
Signed-off-by Akira Yokosawa <[email protected]>---
Hi Paul,For quite a while, I was wondering why "make -j3 2c 1c eb" didn't work as expected.
Quoted 7 lines
I think I have managed to get it work.
But changes in
Makefileneed extra testing. Especially, those involving parallelizing.So I'd like you to give this a fair amount of testing before pushing it.
OK, I am applying it to an experimental repo. Let's give it a go:
$ time make -j32 2c 1c eb a4 nq sf sfnq [ . . . ] 'perfbook-sf.pdf' is ready. 'perfbook-1c.pdf' is ready. 'perfbook.pdf' is ready. 'perfbook-sfnq.pdf' is ready. 'perfbook-a4.pdf' is ready. 'perfbook-eb.pdf' is ready. 'perfbook-nq.pdf' is ready.
real 2m31.412s user 16m45.778s sys 1m48.385s
Not bad, actually! Might be worth redoing my release scripts to take advantage of this. ;-)
I (very) quickly glanced at the PDFs, and they look OK.
I then did a "make clean" and:
$ time make [ . . . ] 'perfbook.pdf' is ready. utilities/punctcheck.sh utilities/cleverefcheck.sh
real 1m47.823s user 1m45.715s sys 0m1.226s
Another "make clean" and:
$ time make -j32 [ . . . ] 'perfbook.pdf' is ready. utilities/punctcheck.sh utilities/cleverefcheck.sh
real 1m49.484s user 1m47.206s sys 0m1.437s
So, as one would expect, the sweet spot for -j is when rebuilding all the images from scratch or when building multiple PDFs.
Nice!
Huh. It would be good to have a way of diff-ing PDF files. Then a single-threaded-generated PDF could be automatically compared to its counterpart from a concurrent build. But the search engines were not kind to me.
Thoughts?
Thanx, Paul
Quoted 71 lines
Thanks, Akira
--Makefile | 6 +++--- appendix/appendix.tex | 8 ++++---- perfbook-lt.tex | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-)Makefile +3
-3@@ -222,14 +222,14 @@ $(PDFTARGETS): %.pdf: %.tex %.bbl $(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux $(BIBSOURCES) bibtex $(basename $@) -$(PDFTARGETS:.pdf=.aux): $(LATEXGENERATED) $(LATEXSOURCES) $(LST_SOURCES) +$(PDFTARGETS:.pdf=.aux): %.aux: %.tex $(LATEXGENERATED) $(LST_SOURCES) ifeq ($(NEWTXTEXT),) $(error Font package 'newtx' not found. See #9 in FAQ-BUILD.txt) endif sh utilities/runfirstlatex.sh $(basename $@) -autodate.tex: perfbook-lt.tex $(LATEXSOURCES) $(BIBSOURCES) \ - $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \ +autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(PDFTARGETS_OF_EPS) \ + $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \ $(GITREFSTAGS) utilities/autodate.sh sh utilities/autodate.shappendix/appendix.tex +4
-4@@ -2,9 +2,9 @@ % mainfile: ../perfbook.tex % SPDX-License-Identifier: CC-BY-SA-3.0 -\include{appendix/questions/questions} -\include{appendix/toyrcu/toyrcu} -\include{appendix/whymb/whymemorybarriers} -\include{appendix/styleguide/styleguide} +\input{appendix/questions/questions} +\input{appendix/toyrcu/toyrcu} +\input{appendix/whymb/whymemorybarriers} +\input{appendix/styleguide/styleguide} \renewcommand{\bottomtitlespace}{.08\textheight} \QuickQuizAnswersperfbook-lt.tex +2
-2@@ -667,12 +667,12 @@ % Credits \setcounter{secnumdepth}{-1} % surpress section numbering in backmatter -\include{ack} +\input{ack} % Index if enabled \IfIndexOn{ \IfIndexHier{ -\include{indexsee} +\input{indexsee} }{} \phantomsection \IfEbookSize{base-commit:
f24ccfbd345eab8a70d6c08e71c84f18df46a3a1Signature · 1 line
2.17.1
Quoted 61 lines · Paul E. McKenney
Quoted 40 lines · Akira Yokosawa
Parallel build of PDF figures and .fcv snippets by "make -jN" worked when a single PDF target was given to "make", such as "make neatfreak; make -j4 eb".
On the other hand, "make -j3 2c 1c eb" ended up in various errors during parallel runs of pdflatex.It turns out that the cause of the error is the \include{} commands in appendix/appendix.tex and perfbook-lt.tex.
\include{} switches to its own .aux file, in this case, appendix/questions.aux, appendix/toyrcu.aux, and so on. As their names are common to different PDF targets, those .aux files can be overwritten/corrupted by parallel runs of pdflatex.\include{} implies \cleardoublepage both in front of and next to it. In perfbook, included LaTeX sources have \chapter{} commands and the implied page breaks are redundant.
By replacing \include{} with \input{}, parallel runs of pdflatex can be enabled.
There were a couple of minor issues in
MakefileWRT parallel runs ofrunfirstlatex.sh.When some of perfbook-xxx.tex files already existed,
runfirstlatex.shfor other main perfbook-yyy.tex could be invoked prematurely before that .tex file was ready.Fix it by adding stricter dependencies of perfbook-xxx.aux on perfbook-xxx.tex.
Also remove some redundant dependencies around here.Note: LATEXSOURCES contains perfbook-lt.tex
Signed-off-by Akira Yokosawa <[email protected]>---
Hi Paul,For quite a while, I was wondering why "make -j3 2c 1c eb" didn't work as expected.
Quoted 7 lines
I think I have managed to get it work.
But changes in
Makefileneed extra testing. Especially, those involving parallelizing.So I'd like you to give this a fair amount of testing before pushing it.
OK, I am applying it to an experimental repo. Let's give it a go:
$ time make -j32 2c 1c eb a4 nq sf sfnq [ . . . ] 'perfbook-sf.pdf' is ready. 'perfbook-1c.pdf' is ready. 'perfbook.pdf' is ready. 'perfbook-sfnq.pdf' is ready. 'perfbook-a4.pdf' is ready. 'perfbook-eb.pdf' is ready. 'perfbook-nq.pdf' is ready.
Quoted 6 lines
real 2m31.412s user 16m45.778s sys 1m48.385sNot bad, actually! Might be worth redoing my release scripts to take advantage of this. ;-)
Nice!
Quoted 30 lines
I (very) quickly glanced at the PDFs, and they look OK.
I then did a "make clean" and:
$ time make [ . . . ] 'perfbook.pdf' is ready. utilities/punctcheck.sh utilities/cleverefcheck.shreal 1m47.823s user 1m45.715s sys 0m1.226sAnother "make clean" and:
$ time make -j32 [ . . . ] 'perfbook.pdf' is ready. utilities/punctcheck.sh utilities/cleverefcheck.shreal 1m49.484s user 1m47.206s sys 0m1.437sSo, as one would expect, the sweet spot for -j is when rebuilding all the images from scratch or when building multiple PDFs.
Quoted 8 lines
Nice!
Huh. It would be good to have a way of diff-ing PDF files. Then a single-threaded-generated PDF could be automatically compared to its counterpart from a concurrent build. But the search engines were not kind to me.
Thoughts?
How about comparing .log files of two builds?
In my trial, "diff -u perfbook-single.log perfbook-multi.log" yields only a hunk of build time difference as shown below:
--- perfbook-single.log 2022-01-27 14:30:01.659959433 +0900
+++ perfbook-multi.log 2022-01-27 14:33:57.556196036 +0900
@@ -1,4 +1,4 @@
-This is pdfTeX, Version 3.141592653-2.6-1.40.23 (TeX Live 2021) (preloaded format=pdflatex 2022.1.24) 27 JAN 2022 14:28
+This is pdfTeX, Version 3.141592653-2.6-1.40.23 (TeX Live 2021) (preloaded format=pdflatex 2022.1.24) 27 JAN 2022 14:32
entering extended mode
restricted \write18 enabled.
%&-line parsing enabled.Not a direct comparison, but the log has a summary near the end:
Here is how much of TeX's memory you used:
74049 strings out of 479599
1597759 string characters out of 5876039
1950680 words of memory out of 5000000
80683 multiletter control sequences out of 15000+600000
692091 words of font info for 721 fonts, out of 8000000 for 9000
1816 hyphenation exceptions out of 8191
105i,19n,133p,1187b,1660s stack positions out of 5000i,500n,10000p,200000b,80000s
[long list of referenced font files]
Output written on perfbook.pdf (638 pages, 8031215 bytes).
PDF statistics:
25510 PDF objects out of 26593 (max. 8388607)
24061 compressed objects within 241 object streams
7119 named destinations out of 7423 (max. 500000)
166134 words of extra memory for PDF output out of 184870 (max. 10000000)I think any visible difference in two PDFs is likely to be reflected in this summary.
Indirect hints they may be, at least they provide partial assurance.
Thanks, Akira
Quoted 10 lines
Thanx, Paul
Quoted 7 lines
Thanks, Akira
--Makefile | 6 +++--- appendix/appendix.tex | 8 ++++---- perfbook-lt.tex | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-)
[...]