Enable parallel runs of pdflatex
v3 cover Enable parallel runs of pdflatex
Hi Paul,
I said earlier in v2's thread:
Quoted 8 lines
This patch changes both LaTeX code and
Makefile. SyncTeX database is affected by LaTeX code change because it is a collection of "LaTeX code (file:line:char) <-> coordinates in PDF".
So we can't see if changes inMakefileaffect PDF or not.I'll split this patch into a set of "preparatory LaTeX change" and "
Makefileupdate" patches.
I did break up the patch (1/4 and 2/4), but it did not work as I had expected.
Because of the Git-commit info in the footer area, SyncTeX database can't be identical when you move to another commit.
So, I restored the Makefile as of 2022.01.25 for reference (3/4) and added a script to collect and compare SyncTeX databases from reference, sequential, and parallel runs of pdflatex (4/4).
Please give it a try.
You can override defaults of TARGETS (2c 1c eb) and JOBS (4) by saying, e.g.:
TARGETS="2c 1c eb nq 1cnq ebnq" JOBS=8 ./utilities/parallel-latex-regress.sh
It will take quite a while for sequential runs of pdflatex with the synctex option enabled. So I'd suggest starting from a short list.
In my tests so far, no mismatch was detected.
Thanks, Akira
--
Akira Yokosawa (4):
Replace \include{} with \input{} for parallel pdflatex runs
Makefile: Fix issues WRT parallel runs of pdflatex Restore Makefile as of 2022.01.25
Add script for parallel-pdflatex-run regression test
Makefile | 12 +- Makefile.2022.01.25 | 608 ++++++++++++++++++++++++++++ appendix/appendix.tex | 8 +- perfbook-lt.tex | 4 +- utilities/parallel-latex-regress.sh | 75 ++++ 5 files changed, 696 insertions(+), 11 deletions(-) create mode 100644 Makefile.2022.01.25 create mode 100755 utilities/parallel-latex-regress.sh
base-commit: f24ccfbd345eab8a70d6c08e71c84f18df46a3a1
Signature · 1 line
2.17.1
PATCH v3 1/4 Replace \include{} with \input{} for parallel pdflatex runs
Parallel build of .pdf and .fcv files by "make -jN" worked when a single PDF target was specified.
On the other hand, "make -j3 2c 1c eb" ended up in various errors during parallel runs of pdflatex.
It turns out that the cause of the error is the \include{} commands in appendix/appendix.tex and perfbook-lt.tex.
\include{} switches to its own .aux file, in this case, appendix/questions.aux, appendix/toyrcu.aux, and so on. As their names are common to different PDF targets, those .aux files can be overwritten/corrupted by parallel runs of pdflatex.
\include{} implies \cleardoublepage both in front of and next to it. In perfbook, included LaTeX sources have \chapter{} commands and the implied page breaks are redundant.
By replacing \include{} commands with \input{} commands, parallel runs of pdflatex can be enabled.
--- appendix/appendix.tex | 8 ++++---- perfbook-lt.tex | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-)
appendix/appendix.tex +4 -4
@@ -2,9 +2,9 @@ % mainfile: ../perfbook.tex % SPDX-License-Identifier: CC-BY-SA-3.0 -\include{appendix/questions/questions} -\include{appendix/toyrcu/toyrcu} -\include{appendix/whymb/whymemorybarriers} -\include{appendix/styleguide/styleguide} +\input{appendix/questions/questions} +\input{appendix/toyrcu/toyrcu} +\input{appendix/whymb/whymemorybarriers} +\input{appendix/styleguide/styleguide} \renewcommand{\bottomtitlespace}{.08\textheight} \QuickQuizAnswers
perfbook-lt.tex +2 -2
@@ -667,12 +667,12 @@ % Credits \setcounter{secnumdepth}{-1} % surpress section numbering in backmatter -\include{ack} +\input{ack} % Index if enabled \IfIndexOn{ \IfIndexHier{ -\include{indexsee} +\input{indexsee} }{} \phantomsection \IfEbookSize{
Signature · 1 line
2.17.1
PATCH v3 2/4 Makefile: Fix issues WRT parallel runs of pdflatex
In reply to Akira YokosawaThere is a couple of minor issues in Makefile WRT parallel runs of runfirstlatex.sh.
When some of perfbook-xxx.tex files already exist, runfirstlatex.sh for other main perfbook-yyy.tex can be invoked prematurely before that .tex file gets ready.
Fix it by adding stricter dependencies of perfbook-xxx.aux on perfbook-xxx.tex.
Also sort out dependencies around here. autodate.tex now depends only on files in the Git repo. Instead, perfbook_flat.tex acts as a barrier between the file conversion phase and the pdflatex phase.
--- Makefile | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
Makefile +7 -5
@@ -222,18 +222,20 @@ $(PDFTARGETS): %.pdf: %.tex %.bbl $(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux $(BIBSOURCES) bibtex $(basename $@) -$(PDFTARGETS:.pdf=.aux): $(LATEXGENERATED) $(LATEXSOURCES) $(LST_SOURCES) +$(PDFTARGETS:.pdf=.aux): %.aux: %.tex $(LATEXGENERATED) ifeq ($(NEWTXTEXT),) $(error Font package 'newtx' not found. See #9 in FAQ-BUILD.txt) endif sh utilities/runfirstlatex.sh $(basename $@) -autodate.tex: perfbook-lt.tex $(LATEXSOURCES) $(BIBSOURCES) \ - $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \ - $(GITREFSTAGS) utilities/autodate.sh +autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(SOURCES_OF_SNIPPET) \ + $(LST_SOURCES) $(FIGSOURCES) $(DOTSOURCES) $(EPSORIGIN) \ + $(SVGSOURCES) $(GITREFSTAGS) \ + utilities/autodate.sh sh utilities/autodate.sh -perfbook_flat.tex: autodate.tex +perfbook_flat.tex: autodate.tex $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) \ + $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) ifndef LATEXPAND $(error --> $@: latexpand not found. Please install it) endif
Signature · 1 line
2.17.1
PATCH v3 3/4 Restore Makefile as of 2022.01.25
In reply to Akira YokosawaFor regression test of parallel pdflatex runs, carry old Makefile as Makefile.2022.01.25 for reference.
--- Makefile.2022.01.25 | 608 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 608 insertions(+) create mode 100644 Makefile.2022.01.25
Makefile.2022.01.25 new file +608 -0
@@ -0,0 +1,608 @@ +SHELL = /bin/bash + +GITREFSTAGS := $(shell ls -d .git/refs/tags 2>/dev/null) + +LATEXSOURCES = \ + perfbook-lt.tex \ + legal.tex \ + summary.tex \ + glossary.tex \ + qqz.sty origpub.sty \ + glsdict.tex indexsee.tex \ + pfbook.cls \ + ushyphex.tex pfhyphex.tex \ + ack.tex \ + */*.tex \ + */*/*.tex + +LST_SOURCES := $(wildcard CodeSamples/formal/promela/*.lst) \ + $(wildcard appendix/styleguide/*.c) + +SUB_QQZ := qqzhowto.tex qqzintro.tex qqzcpu.tex qqztoolsoftrade.tex \ + qqzcount.tex qqzSMPdesign.tex qqzlocking.tex qqzowned.tex \ + qqzdefer.tex qqzdatastruct.tex qqzdebugging.tex qqzformal.tex \ + qqztogether.tex qqzadvsync.tex qqzmemorder.tex qqzeasy.tex \ + qqzfuture.tex qqzquestions.tex qqztoyrcu.tex qqzwhymb.tex + +LATEXGENERATED = autodate.tex qqz.tex contrib.tex origpub.tex sub_qqz +# Note: Empty target "sub_qqz" is used on behalf of $(SUB_QQZ) to prevent +# parallel runs of divideqqz.pl. + +TWOCOLTARGETS := mstx msr msn msnt sf nq sfnq ix df +EBTARGETS := $(foreach v,nq sf sfnq ix df,eb$(v)) +ABBREVTARGETS := lt hb a4 1c tcb msns mss eb $(TWOCOLTARGETS) $(foreach v,$(TWOCOLTARGETS),1c$(v)) $(EBTARGETS) + +PDFTARGETS := perfbook.pdf $(foreach v,$(ABBREVTARGETS),perfbook-$(v).pdf) +GENERATED_MAIN := $(filter-out perfbook-lt.tex,$(foreach v,$(ABBREVTARGETS),perfbook-$(v).tex)) perfbook.tex + +EPSSOURCES_FROM_TEX := \ + SMPdesign/DiningPhilosopher5.eps \ + SMPdesign/DiningPhilosopher5TB.eps \ + SMPdesign/DiningPhilosopher4part-b.eps \ + SMPdesign/DiningPhilosopher5PEM.eps + +PDFTARGETS_OF_TEX := $(EPSSOURCES_FROM_TEX:%.eps=%.pdf) + +DOTSOURCES := $(wildcard */*.dot) + +EPSSOURCES_FROM_DOT := $(DOTSOURCES:%.dot=%.eps) + +FIGSOURCES := $(wildcard */*.fig) $(wildcard */*/*.fig) + +EPSSOURCES_FROM_FIG := $(FIGSOURCES:%.fig=%.eps) + +SVGSOURCES := $(wildcard */*.svg) +FAKE_EPS_FROM_SVG := $(SVGSOURCES:%.svg=%.eps) +PDFTARGETS_OF_SVG := $(SVGSOURCES:%.svg=%.pdf) + +OBSOLETE_FILES = extraction $(FAKE_EPS_FROM_SVG) CodeSamples/snippets.mk + +EPSSOURCES_DUP := \ + $(wildcard */*.eps) \ + $(wildcard */*/*.eps) \ + $(wildcard */*/*/*.eps) \ + $(wildcard */*/*/*/*.eps) \ + $(wildcard */*/*/*/*/*.eps) \ + $(EPSSOURCES_FROM_TEX) \ + $(EPSSOURCES_FROM_DOT) \ + $(EPSSOURCES_FROM_FIG) + +EPSSOURCES_OLD := \ + $(wildcard CodeSamples/*/*/OLD-*/*.eps) \ + $(wildcard CodeSamples/*/*/OLD-*/*/*.eps) \ + $(wildcard CodeSamples/*/*/*/OLD-*/*.eps) \ + $(wildcard CodeSamples/*/*/*/OLD-*/*/*.eps) + +EPSSOURCES := $(sort $(filter-out $(EPSSOURCES_OLD),$(filter-out $(OBSOLETE_FILES),$(EPSSOURCES_DUP)))) + +PDFTARGETS_OF_EPS := $(EPSSOURCES:%.eps=%.pdf) + +EPSORIGIN := $(filter-out $(EPSSOURCES_FROM_TEX) $(EPSSOURCES_FROM_DOT) $(EPSSOURCES_FROM_FIG),$(EPSSOURCES)) + +GNUPLOT_ORIG := $(shell grep -l gnuplot $(EPSORIGIN)) +GNUPLOT_ORIG_NOFIXFONTS := $(shell grep -l '/FontMatrix' $(GNUPLOT_ORIG)) +GNUPLOT_ORIG_NEEDFIXFONTS := $(filter-out $(GNUPLOT_ORIG_NOFIXFONTS),$(GNUPLOT_ORIG)) +EPSORIG_NOFIXFONTS := $(filter-out $(GNUPLOT_ORIG_NEEDFIXFONTS),$(EPSORIGIN)) + +PDFTARGETS_OF_EPSORIG := $(EPSORIGIN:%.eps=%.pdf) +PDFTARGETS_OF_GNUPLOT_NEEDFIXFONTS := $(GNUPLOT_ORIG_NEEDFIXFONTS:%.eps=%.pdf) +PDFTARGETS_OF_EPSORIG_NOFIXFONTS := $(EPSORIG_NOFIXFONTS:%.eps=%.pdf) + +PDFTARGETS_OF_EPSOTHER := $(filter-out $(PDFTARGETS_OF_EPSORIG) $(PDFTARGETS_OF_TEX),$(PDFTARGETS_OF_EPS)) + +BIBSOURCES := bib/*.bib alphapf.bst + +# required commands +DOT := $(shell which dot 2>/dev/null) +FIG2EPS := $(shell which fig2eps 2>/dev/null) +INKSCAPE := $(shell which inkscape 2>/dev/null) +ifdef INKSCAPE + INKSCAPE_ONE := $(shell inkscape --version 2>/dev/null | grep -c "Inkscape 1") +endif +LATEXPAND := $(shell which latexpand 2>/dev/null) +QPDF := $(shell which qpdf 2>/dev/null) + +# required fonts +STEELFONT := $(shell fc-list | grep -c -i steel) +URWPS := $(shell fc-list | grep "Nimbus Mono PS" | wc -l) + +# required font packages +FONTPACKAGES := $(shell kpsewhich newtxtext.sty nimbusmono.sty newtxtt.sty newtxsf.sty inconsolata.sty couriers.sty) +NEWTXTEXT := $(findstring newtxtext,$(FONTPACKAGES)) +NIMBUSMONO := $(findstring nimbusmono,$(FONTPACKAGES)) +NEWTXTT := $(findstring newtxtt,$(FONTPACKAGES)) +COURIERS := $(findstring couriers,$(FONTPACKAGES)) +NEWTXSF := $(findstring newtxsf,$(FONTPACKAGES)) +INCONSOLATA := $(findstring inconsolata,$(FONTPACKAGES)) +FREESANS := $(shell fc-list | grep FreeSans | wc -l) +DEJAVUMONO := $(shell fc-list | grep "DejaVu Sans Mono" | wc -l) + +# for line break in error text +define n + + +endef + +ifeq ($(URWPS),0) +FIXSVGFONTS = utilities/fixsvgfonts.sh +FIXANEPSFONTS = utilities/fixanepsfonts.sh +FIXFONTS = utilities/fixfonts.sh +else +FIXSVGFONTS = utilities/fixsvgfonts-urwps.sh +FIXANEPSFONTS = utilities/fixanepsfonts-urwps.sh +FIXFONTS = utilities/fixfonts-urwps.sh +endif +ifeq ($(FREESANS),0) + RECOMMEND_FREEFONT := 1 +else + RECOMMEND_FREEFONT := 0 +endif +ifeq ($(DEJAVUMONO),0) + RECOMMEND_DEJAVU := 1 +else + RECOMMEND_DEJAVU := 0 +endif + +STEELFONTID := $(shell fc-list | grep -i steel | grep -c Steel) + +LINELABEL_ENV_BEGIN := $(shell grep -l -F '\begin{linelabel}' $(LATEXSOURCES)) +LINELABEL_ENV_END := $(shell grep -l -F '\end{linelabel}' $(LATEXSOURCES)) +LINEREF_ENV_BEGIN := $(shell grep -l -F '\begin{lineref}' $(LATEXSOURCES)) +LINEREF_ENV_END := $(shell grep -l -F '\end{lineref}' $(LATEXSOURCES)) +LINELABEL_ENV := $(sort $(LINELABEL_ENV_BEGIN) $(LINELABEL_ENV_END)) +LINEREF_ENV := $(sort $(LINEREF_ENV_BEGIN) $(LINEREF_ENV_END)) + +CREFPTN := '\\[Cc](ln)?ref{[^}]+}\s*{[^}]+}' +CREFPAIR := $(shell grep -l -zo -E $(CREFPTN) $(LATEXSOURCES)) + +SOURCES_OF_SNIPPET_ALL := $(shell grep -r -l -F '\begin{snippet}' CodeSamples) +SOURCES_OF_LITMUS := $(shell grep -r -l -F '\begin[snippet]' CodeSamples) +SOURCES_OF_LTMS := $(patsubst %.litmus,%.ltms,$(SOURCES_OF_LITMUS)) +SOURCES_OF_SNIPPET := $(filter-out $(SOURCES_OF_LTMS),$(SOURCES_OF_SNIPPET_ALL)) $(SOURCES_OF_LITMUS) +GEN_SNIPPET_D = utilities/gen_snippet_d.pl utilities/gen_snippet_d.sh + +default = $(PERFBOOK_DEFAULT) + +ifeq ($(default),) + targ = perfbook.pdf +else + targ = $(default) +endif + +chkpagegroup = $(PERFBOOK_CHKPAGEGROUP) + +ifeq ($(PERFBOOK_PAPER),A4) + PERFBOOK_BASE = perfbook-a4.tex +else +ifeq ($(PERFBOOK_PAPER),HB) + PERFBOOK_BASE = perfbook-hb.tex +else + PERFBOOK_BASE = perfbook-lt.tex +endif +endif + +BASE_DEPENDS := perfbook.tex $(foreach v,tcb 1c msns mss mstx msr msn msnt sf nq ix df,perfbook-$(v).tex) + +.PHONY: all touchsvg clean distclean neatfreak 2c ls-unused $(ABBREVTARGETS) +.PHONY: mslm perfbook-mslm.pdf mslmmsg +.PHONY: qq perfbook-qq.pdf qqmsg +.PHONY: help help-official help-full help-semiofficial help-paper help-draft +.PHONY: help-experimental help-prefixed +.PHONY: paper-clean periodcheck punctcheck punctcheck-auto +.PHONY: cleanfigs cleanfigs-eps cleanfigs-svg figs + +all: punctcheck-auto + +ifeq ($(MAKECMDGOALS),clean) +else ifeq ($(MAKECMDGOALS),distclean) +else ifeq ($(MAKECMDGOALS),neatfreak) +else +-include CodeSamples/snippets.d +endif + +2c: perfbook.pdf + +mslm: perfbook-mslm.pdf +perfbook-mslm.pdf: mslmmsg + +qq: perfbook-qq.pdf +perfbook-qq.pdf: qqmsg + +mslmmsg: perfbook.pdf + @echo "perfbook-mslm.pdf is promoted to default target," + @echo "built as perfbook.pdf." + +qqmsg: perfbook.pdf + @echo "perfbook-qq.pdf is promoted to default target," + @echo "built as perfbook.pdf." + +$(PDFTARGETS): %.pdf: %.tex %.bbl + sh utilities/runlatex.sh $(basename $@) + +$(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux $(BIBSOURCES) + bibtex $(basename $@) + +$(PDFTARGETS:.pdf=.aux): $(LATEXGENERATED) $(LATEXSOURCES) $(LST_SOURCES) +ifeq ($(NEWTXTEXT),) + $(error Font package 'newtx' not found. See #9 in FAQ-BUILD.txt) +endif + sh utilities/runfirstlatex.sh $(basename $@) + +autodate.tex: perfbook-lt.tex $(LATEXSOURCES) $(BIBSOURCES) \ + $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \ + $(GITREFSTAGS) utilities/autodate.sh + sh utilities/autodate.sh + +perfbook_flat.tex: autodate.tex +ifndef LATEXPAND + $(error --> $@: latexpand not found. Please install it) +endif + @if [ ! -z "$(LINELABEL_ENV)" -a "$(LINELABEL_ENV)" != " " ]; then \ + echo "'linelabel' used as environment in $(LINELABEL_ENV)." ; \ + echo "------" ; \ + grep -n -B 2 -A 2 -F 'linelabel' $(LINELABEL_ENV) ; \ + echo "------" ; \ + echo "Substitute 'fcvlabel' for 'linelabel' in $(LINELABEL_ENV)." ; \ + exit 1 ; \ + fi + @if [ ! -z "$(LINEREF_ENV)" -a "$(LINEREF_ENV)" != " " ]; then \ + echo "'lineref' used as environment in $(LINEREF_ENV)." ; \ + echo "------" ; \ + grep -n -B 2 -A 2 -F 'lineref' $(LINEREF_ENV) ; \ + echo "------" ; \ + echo "Substitute 'fcvref' for 'lineref' in $(LINEREF_ENV)." ; \ + exit 1 ; \ + fi + @if [ ! -z "$(CREFPAIR)" -a "$(CREFPAIR)" != " " ]; then \ + echo "------" ; \ + if grep -q -E $(CREFPTN) $(CREFPAIR) ; then \ + grep -n -B 2 -A 2 -E $(CREFPTN) $(CREFPAIR) ; \ + else \ + grep -zo -B 2 -A 2 -E $(CREFPTN) $(CREFPAIR) ; \ + echo ; \ + fi ; \ + echo "------" ; \ + echo "Need to use \[Cc]refrange or \[Cc]lnrefrangein $(CREFPAIR)." ; \ + exit 1 ; \ + fi + echo > qqz.tex + echo > contrib.tex + echo > origpub.tex + latexpand --empty-comments perfbook-lt.tex 1> $@ 2> /dev/null + +qqz.tex: perfbook_flat.tex + sh utilities/extractqqz.sh < $< | perl utilities/qqzreorder.pl > $@ + +contrib.tex: perfbook_flat.tex qqz.tex + cat $^ | sh utilities/extractcontrib.sh > $@ + +origpub.tex: perfbook_flat.tex + sh utilities/extractorigpub.sh < $< > $@ + +# Empty target to generate $(SUB_QQZ) files +sub_qqz: qqz.tex + utilities/divideqqz.pl + @touch sub_qqz + +perfbook.tex: $(PERFBOOK_BASE) + cp $< $@ + +perfbook-tcb.tex: $(PERFBOOK_BASE) + sed -e 's/{tblcptop}{true}/{tblcptop}{false}/' < $< > $@ + +perfbook-1c.tex: $(PERFBOOK_BASE) + sed -e 's/setboolean{twocolumn}{true}/setboolean{twocolumn}{false}/' < $< > $@ + +perfbook-hb.tex: perfbook-lt.tex + sed -e 's/setboolean{hardcover}{false}/setboolean{hardcover}{true}/' < $< > $@ + +perfbook-eb.tex: perfbook-lt.tex + sed -e 's/setboolean{ebooksize}{false}/setboolean{ebooksize}{true}/' < $< > $@ + sed -i 's/setboolean{twocolumn}{true}/setboolean{twocolumn}{false}/' $@ + +perfbook-msns.tex: $(PERFBOOK_BASE) + sed -e 's/%msfontstub/\\usepackage{courier}/' < $< > $@ + +perfbook-mss.tex: $(PERFBOOK_BASE) +ifeq ($(COURIERS),) + $(error Font package 'courier-scaled' not found. See #9 in FAQ-BUILD.txt) +endif + sed -e 's/%msfontstub/\\usepackage[scaled=.94]{couriers}/' < $< > $@ + +perfbook-mstx.tex: $(PERFBOOK_BASE) +perfbook-1cmstx.tex: perfbook-1c.tex +perfbook-mstx.tex perfbook-1cmstx.tex: + sed -e 's/%msfontstub/\\renewcommand*\\ttdefault{txtt}/' < $< > $@ + +perfbook-msr.tex: $(PERFBOOK_BASE) +perfbook-1cmsr.tex: perfbook-1c.tex +perfbook-msr.tex perfbook-1cmsr.tex: +ifeq ($(NIMBUSMONO),) + $(error Font package 'nimbus15' not found. See #9 in FAQ-BUILD.txt) +endif + sed -e 's/%msfontstub/\\usepackage[scaled=.94]{nimbusmono}/' \ + -e 's/{nimbusavail}{false}/{nimbusavail}{true}/' < $< > $@ + +perfbook-msn.tex: $(PERFBOOK_BASE) +perfbook-1cmsn.tex: perfbook-1c.tex +perfbook-msn.tex perfbook-1cmsn.tex: +ifeq ($(NIMBUSMONO),) + $(error Font package 'nimbus15' not found. See #9 in FAQ-BUILD.txt) +endif + sed -e 's/\\renewcommand\*\\ttdefault{lmtt}//' \ + -e 's/{lmttforcode}{true}/{lmttforcode}{false}/' \ + -e 's/{nimbusavail}{false}/{nimbusavail}{true}/' < $< > $@ + +perfbook-msnt.tex: $(PERFBOOK_BASE) +perfbook-1cmsnt.tex: perfbook-1c.tex +perfbook-msnt.tex perfbook-1cmsnt.tex: +ifeq ($(NEWTXTT),) + $(error Font package 'newtxtt' not found.$nInstall it or try 'make mstx' instead. See #9 in FAQ-BUILD.txt) +endif +ifeq ($(NIMBUSMONO),) + $(error Font package 'nimbus15' not found. See #9 in FAQ-BUILD.txt) +endif + sed -e 's/%msfontstub/\\usepackage[zerostyle=a]{newtxtt}/' \ + -e 's/{qqzbg}{false}/{qqzbg}{true}/' \ + -e 's/{nimbusavail}{false}/{nimbusavail}{true}/' < $< > $@ + +perfbook-sf.tex: $(PERFBOOK_BASE) +perfbook-1csf.tex: perfbook-1c.tex +perfbook-ebsf.tex: perfbook-eb.tex +perfbook-sf.tex perfbook-1csf.tex perfbook-ebsf.tex: +ifeq ($(NEWTXSF),) + $(error Font package 'newtxsf' not found. See #9 in FAQ-BUILD.txt) +endif +ifeq ($(INCONSOLATA),) + $(error Font package 'inconsolata' not found. See #9 in FAQ-BUILD.txt) +endif +ifeq ($(NIMBUSMONO),) + $(error Font package 'nimbus15' not found. See #9 in FAQ-BUILD.txt) +endif + sed -e 's/setboolean{sansserif}{false}/setboolean{sansserif}{true}/' \ + -e 's/{nimbusavail}{false}/{nimbusavail}{true}/' \ + -e 's/%msfontstub/\\usepackage[var0]{inconsolata}[2013\/07\/17]/' < $< > $@ + +perfbook-nq.tex: $(PERFBOOK_BASE) +perfbook-sfnq.tex: perfbook-sf.tex +perfbook-1cnq.tex: perfbook-1c.tex +perfbook-1csfnq.tex: perfbook-1csf.tex +perfbook-ebnq.tex: perfbook-eb.tex +perfbook-ebsfnq.tex: perfbook-ebsf.tex +perfbook-nq.tex perfbook-sfnq.tex perfbook-1cnq.tex perfbook-1csfnq.tex perfbook-ebnq.tex perfbook-ebsfnq.tex: + sed -e 's/setboolean{qqzbg}{true}/setboolean{qqzbg}{false}/' \ + -e 's/setboolean{noqqz}{false}/setboolean{noqqz}{true}/' < $< > $@ + +perfbook-ix.tex: $(PERFBOOK_BASE) +perfbook-1cix.tex: perfbook-1c.tex +perfbook-ebix.tex: perfbook-eb.tex +perfbook-ix.tex perfbook-1cix.tex perfbook-ebix.tex: + sed -e 's/setboolean{qqzbg}{true}/setboolean{qqzbg}{false}/' \ + -e 's/setboolean{indexhl}{false}/setboolean{indexhl}{true}/' < $< > $@ + +perfbook-df.tex: $(PERFBOOK_BASE) +perfbook-1cdf.tex: perfbook-1c.tex +perfbook-ebdf.tex: perfbook-eb.tex +perfbook-df.tex perfbook-1cdf.tex perfbook-ebdf.tex: + sed -e 's/setboolean{qqzbg}{true}/setboolean{qqzbg}{false}/' \ + -e 's/setboolean{indexon}{true}/setboolean{indexon}{false}/' < $< > $@ + +perfbook-a4.tex: perfbook-lt.tex +perfbook-a4.tex: + sed -e 's/{afourpaper}{false}/{afourpaper}{true}/' < $< > $@ + +# Rules related to perfbook_html are removed as of May, 2016 + +$(EPSSOURCES_FROM_TEX): $(FIXANEPSFONTS) $(FIXFONTS) +$(EPSSOURCES_FROM_TEX): %.eps: %.tex + @echo "$< --> $(suffix $@)" + sh utilities/mpostcheck.sh + @latex -output-directory=$(shell dirname $<) $< > /dev/null 2>&1 + @dvips -Pdownload35 -E $(patsubst %.tex,%.dvi,$<) -o $@ > /dev/null 2>&1 + @sh $(FIXANEPSFONTS) $@ + +$(EPSSOURCES_FROM_DOT): $(FIXANEPSFONTS) $(FIXFONTS) +$(EPSSOURCES_FROM_DOT): %.eps: %.dot + @echo "$< --> $(suffix $@)" +ifndef DOT + $(error $< --> $@: dot not found. Please install graphviz) +endif + @dot -Tps -o $@ $< + @sh $(FIXANEPSFONTS) $@ + +$(EPSSOURCES_FROM_FIG): $(FIXANEPSFONTS) $(FIXFONTS) +$(EPSSOURCES_FROM_FIG): %.eps: %.fig + @echo "$< --> $(suffix $@)" +ifndef FIG2EPS + $(error $< --> $@: fig2eps not found. Please install fig2ps) +endif + @fig2eps --nogv $< > /dev/null 2>&1 + @sh $(FIXANEPSFONTS) $@ + +# .eps --> .pdf rules +ifdef USE_A2PING + include a2ping-rule.mk +else + include epstopdf-rule.mk +endif + +$(PDFTARGETS_OF_SVG): $(FIXSVGFONTS) +$(PDFTARGETS_OF_SVG): %.pdf: %.svg + @echo "$< --> $(suffix $@)" +ifeq ($(STEELFONT),0) + $(error "Steel City Comic" font not found. See #1 in FAQ.txt) +endif +ifndef INKSCAPE + $(error $< --> $@ inkscape not found. Please install it) +endif +ifeq ($(STEELFONTID),0) + @sh $(FIXSVGFONTS) < $< | sed -e 's/Steel City Comic/Test/g' > $<i +else + @sh $(FIXSVGFONTS) < $< > $<i +endif +ifeq ($(RECOMMEND_FREEFONT),1) + $(info Nice-to-have font package 'gnu-freefont' not found. See #9 in FAQ-BUILD.txt) +endif +ifeq ($(RECOMMEND_DEJAVU),1) + $(info Nice-to-have font package 'dejavu' not found. See #9 in FAQ-BUILD.txt) +endif +ifeq ($(INKSCAPE_ONE),0) + @inkscape --export-pdf=$@ $<i > /dev/null 2>&1 +else + @inkscape -o $@ $<i > /dev/null 2>&1 +endif + @rm -f $<i +ifeq ($(chkpagegroup),on) +ifndef QPDF + $(error qpdf not found. Please install it) +endif + @echo "checking page group in $@" + @qpdf --qdf $@ $@q + @./utilities/extpagegroup.pl < $@q > $@p + @diff -q -w $@p pagegroup + @rm -f $@q $@p +endif + +CodeSamples/snippets.d: $(SOURCES_OF_SNIPPET) $(GEN_SNIPPET_D) + sh ./utilities/gen_snippet_d.sh + +$(FCVSNIPPETS): + @echo "$< --> $(suffix $@)" + @utilities/fcvextract.pl $< $(subst +,\\+,$(subst @,:,$(basename $(notdir $@)))) > $@ + @utilities/checkfcv.pl $@ + +$(FCVSNIPPETS_VIA_LTMS): + @echo "$< --> $(suffix $@)" + @utilities/fcvextract.pl $< $(subst +,\\+,$(subst @,:,$(basename $(notdir $@)))) > $@ + @utilities/checkfcv.pl $@ + +$(FCVSNIPPETS_LTMS): + @echo "$< --> $(suffix $@)" + @utilities/reorder_ltms.pl $< > $@ + +help-official: + @echo "Official targets (Latin Modern Typewriter for monospace font):" + @echo " Full, Abbr." + @echo " perfbook.pdf, 2c: (default) 2-column layout" + @echo " perfbook-1c.pdf, 1c: 1-column layout" + @echo "Note:" + @echo " Official targets now enable indexing and Quick-Quiz framing." + +help-semiofficial: + @echo + @echo "Semi-official targets:" + @echo " Full, Abbr." + @echo " perfbook-nq.pdf, nq: 2c without inline Quick Quizzes (chapterwise Qs)" + @echo " perfbook-sf.pdf, sf: 2c with sans serif font" + @echo " perfbook-sfnq.pdf, sfnq: sf + nq" + +help-paper: + @echo + @echo "Set env variable PERFBOOK_PAPER to change paper size:" + @echo " PERFBOOK_PAPER=A4: a4paper" + @echo " PERFBOOK_PAPER=HB: hard cover book" + @echo " other (default): letterpaper" + @echo "Note:" + @echo " Modified PERFBOOK_PAPER takes effect after \"make paper-clean\"." + @echo + @echo "Paper-size specific targets (independent of PERFBOOK_PAPER):" + @echo " perfbook-lt.pdf, lt: 2c layout on letterpaper" + @echo " perfbook-hb.pdf, hb: 2c layout for hard cover book" + @echo " perfbook-a4.pdf, a4: 2c layout on a4paper" + @echo " perfbook-eb.pdf, eb: 1c layout for ebook reader (WIP)" + +help: help-official help-paper + @echo + @echo "\"make help-full\" will show the full list of available targets." + +help-draft: + @echo + @echo "Targets for draft check, non-framed Quick Quizzes (quicker build)" + @echo " perfbook-ix.pdf, ix: for draft check, with indexed terms highlighted" + @echo " perfbook-df.pdf, df: for draft check, without indexing" + +help-prefixed: + @echo + @echo "Prefixed targets:" + @echo " \"1c*\" such as \"1cnq\", \"1csf\", and \"1cix\" are for 1c-layout." + @echo " \"ebnq\", \"ebsf\", \"ebsfnq\", \"ebix\", and \"ebdf\" are for ebook-size 1c-layout," + @echo " independent of PERFBOOK_PAPER. (WIP)" + +help-experimental: + @echo + @echo "Experimental targets:" + @echo " perfbook-msnt.pdf, msnt: newtxtt as monospace (non-slashed 0)" + @echo " perfbook-mstx.pdf, mstx: txtt as monospace" + @echo " perfbook-msr.pdf, msr: regular thickness courier clone as monospace" + @echo " perfbook-msn.pdf, msn: narrow courier clone as monospace" + @echo + @echo "Historical targets:" + @echo " perfbook-tcb.pdf, tcb: table caption at bottom (First Edition)" + @echo " perfbook-msns.pdf, msns: non-scaled courier (First Edition)" + @echo " perfbook-mss.pdf, mss: scaled courier (default in early 2017)" + @echo + @echo "Notes:" + @echo " - \"msnt\" requires \"newtxtt\". \"mstx\" is a fallback target for older TeX env." + @echo " - \"msr\" and \"msn\" require \"nimbus15\"." + @echo " - \"msn\" doesn't cover bold face monospace." + @echo " - \"sf\" requires \"newtxsf\"." + @echo " - All the targets except for \"msn\" use \"Latin Modern Typewriter\" font" + @echo " for code snippets." + +help-full: help-official help-paper help-semiofficial help-draft help-prefixed help-experimental + +clean: + find . -name '*.aux' -o -name '*.blg' \ + -o -name '*.dvi' -o -name '*.log' \ + -o -name '*.qqz' -o -name '*.toc' -o -name '*.bbl' \ + -o -name '*.pdfp' -o -name '*.pdfq' | xargs rm -f + rm -f perfbook_flat.tex perfbook*.out $(GENERATED_MAIN) + rm -f $(LATEXGENERATED) + rm -f qqz*.tex + rm -f perfbook*.idx perfbook*.ind perfbook*.ilg perfbook*.ist + rm -f perfbook*.acn perfbook*.acr perfbook*.alg + rm -f perfbook*.glg perfbook*.glo perfbook*.gls perfbook*.glsdefs + rm -f CodeSamples/snippets.d + rm -f *.synctex* + @rm -f $(OBSOLETE_FILES) + +paper-clean: + rm -f $(BASE_DEPENDS) + +distclean: clean + sh utilities/cleanpdf.sh + rm -f $(EPSSOURCES_FROM_DOT) $(EPSSOURCES_FROM_TEX) $(EPSSOURCES_FROM_FIG) + find . -name '*.fcv' -o -name '*.ltms' | xargs rm -f + +touchsvg: + find . -name '*.svg' | xargs touch + +ls-unused: + find . -name .unused | xargs ls + +neatfreak: distclean + find . -name '*.pdf' | xargs rm -f + +cleanfigs-eps: + rm -f $(EPSSOURCES_FROM_TEX) $(EPSSOURCES_FROM_DOT) $(EPSSOURCES_FROM_FIG) + rm -f $(PDFTARGETS_OF_EPS) + +cleanfigs-svg: + rm -f $(PDFTARGETS_OF_SVG) + +cleanfigs: cleanfigs-eps cleanfigs-svg + +figs: $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) + +punctcheck: + utilities/punctcheck.sh + utilities/cleverefcheck.sh + +punctcheck-auto: $(targ) + utilities/punctcheck.sh + utilities/cleverefcheck.sh + +periodcheck: punctcheck + +.SECONDEXPANSION: +$(ABBREVTARGETS): %: [email protected]
Signature · 1 line
2.17.1
PATCH v3 4/4 Add script for parallel-pdflatex-run regression test
In reply to Akira Yokosawautilities/parallel-latex-regress.sh runs three phases of "make -j$JOBS $TARGETS" with SyncTeX option enabled.
Phase 1: Sequential pdflatex runs by Makefile.2022.01.25 Phase 2: Sequential pdflatex runs by Makefile Phase 3: Parallel pdflatex runs by Makefile
It then compares collected SyncTeX databases and prints out the result.
Default values of variables TARGETS, JOBS, TMP, and REFMAKE can be overridden by saying, e.g.:
TARGETS="ix 1cix ebix" ./utilities/parallel-latex-regress.sh
Target names given to TARGETS are not sanity-checked at the moment.
--- utilities/parallel-latex-regress.sh | 75 +++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100755 utilities/parallel-latex-regress.sh
utilities/parallel-latex-regress.sh new file +75 -0
@@ -0,0 +1,75 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Regression test of parallel pdflatex runs. +# Copyright (C) Akira Yokosawa, 2022 +# +# Authors: Akira Yokosawa <[email protected]> + +TARGETS=${TARGETS:-2c 1c eb} +JOBS=${JOBS:-4} +TMP=${TMP:-/tmp} +REFMAKE=${REFMAKE:-make -f Makefile.2022.01.25} +export LATEX_OPT=-synctex=1 + +copy_synctex_db () { + for t in $TARGETS + do + case $t in + 2c ) base=perfbook;; + * ) base=perfbook-$t;; + esac + cp $base.synctex.gz $TMP/$base.$1.synctex.gz + done +} + +# Reference sequential builds + +$REFMAKE neatfreak +$REFMAKE -j$JOBS perfbook_flat.tex +$REFMAKE $TARGETS + +copy_synctex_db ref + +# Sequential builds + +make neatfreak +make -j$JOBS perfbook_flat.tex +make $TARGETS + +copy_synctex_db sequ + +# Parallel builds + +make neatfreak +make -j$JOBS $TARGETS + +copy_synctex_db para + +cmp_err=0 + +for t in $TARGETS +do + case $t in + 2c ) base=perfbook;; + * ) base=perfbook-$t;; + esac + + if ! cmp $TMP/$base.ref.synctex.gz $TMP/$base.sequ.synctex.gz + then + cmp_err=`expr $cmp_err + 1` + fi + if ! cmp $TMP/$base.ref.synctex.gz $TMP/$base.para.synctex.gz + then + cmp_err=`expr $cmp_err + 1` + fi +done + +if test $cmp_err -eq 0 +then + echo "No mismatch detected in SyncTeX databases." + exit 0 +else + echo "Mismatch detected in SyncTeX databases!" + exit 1 +fi
Signature · 1 line
2.17.1
A message that is not in the archive (sent off-list or not yet received): 20220130043603.GR4285@paulmck-ThinkPad-P17-Gen-1
Quoted 36 lines · Paul E. McKenney
Quoted 27 lines · Akira Yokosawa
Hi Paul,
I said earlier in v2's thread:
Quoted 8 lines
This patch changes both LaTeX code and
Makefile. SyncTeX database is affected by LaTeX code change because it is a collection of "LaTeX code (file:line:char) <-> coordinates in PDF".
So we can't see if changes inMakefileaffect PDF or not.I'll split this patch into a set of "preparatory LaTeX change" and "
Makefileupdate" patches.I did break up the patch (1/4 and 2/4), but it did not work as I had expected.
Because of the Git-commit info in the footer area, SyncTeX database can't be identical when you move to another commit.So, I restored the
Makefileas of 2022.01.25 for reference (3/4) and added a script to collect and compare SyncTeX databases from reference, sequential, and parallel runs of pdflatex (4/4).Please give it a try.
You can override defaults of TARGETS (2c 1c eb) and JOBS (4) by saying, e.g.:
Quoted 7 lines
TARGETS="2c 1c eb nq 1cnq ebnq" JOBS=8
./utilities/parallel-latex-regress.shIt will take quite a while for sequential runs of pdflatex with the synctex option enabled. So I'd suggest starting from a short list.
In my tests so far, no mismatch was detected.
Quoted 8 lines
Thank you, especially for the script! I applied these patches, but I get an exit code of 1. Ah, but that is because "2cnq" is not a thing. Trying again with:
TARGETS="2c 1c eb 1cnq nq ebnq" time
./utilities/parallel-latex-regress.shAnd I still got mismatches and an exit code of 1, with output shown below. Any hints?
Paul, you did find a regression in the changes of 2/4. I did not expect the script would actually catch such a bug.
The "sorting out" in 2/4 caused autodate.sh to see temporary untracked files from concurrent conversions of figures.
I'm sending a followup patch (5/4) partially reverting 2/4. I should have run the test under a newly cloned Git repository...
Thanks, Akira
Quoted 5 lines
Thanx, Paul
------------------------------------------------------------------------
[...]
Quoted 16 lines
/tmp/perfbook.ref.synctex.gz/tmp/perfbook.sequ.synctex.gzdiffer: byte 3690, line 6/tmp/perfbook.ref.synctex.gz/tmp/perfbook.para.synctex.gzdiffer: byte 3690, line 6/tmp/perfbook-1c.ref.synctex.gz/tmp/perfbook-1c.sequ.synctex.gzdiffer: byte 3720, line 5/tmp/perfbook-1c.ref.synctex.gz/tmp/perfbook-1c.para.synctex.gzdiffer: byte 3720, line 5/tmp/perfbook-eb.ref.synctex.gz/tmp/perfbook-eb.sequ.synctex.gzdiffer: byte 3705, line 8/tmp/perfbook-eb.ref.synctex.gz/tmp/perfbook-eb.para.synctex.gzdiffer: byte 3705, line 8/tmp/perfbook-1cnq.ref.synctex.gz/tmp/perfbook-1cnq.sequ.synctex.gzdiffer: byte 3722, line 4/tmp/perfbook-1cnq.ref.synctex.gz/tmp/perfbook-1cnq.para.synctex.gzdiffer: byte 3722, line 4/tmp/perfbook-nq.ref.synctex.gz/tmp/perfbook-nq.sequ.synctex.gzdiffer: byte 24, line 1/tmp/perfbook-nq.ref.synctex.gz/tmp/perfbook-nq.para.synctex.gzdiffer: byte 24, line 1/tmp/perfbook-ebnq.ref.synctex.gz/tmp/perfbook-ebnq.sequ.synctex.gzdiffer: byte 3701, line 7/tmp/perfbook-ebnq.ref.synctex.gz/tmp/perfbook-ebnq.para.synctex.gzdiffer: byte 3701, line 7 Mismatch detected in SyncTeX databases!
Command exited with non-zero status 1
2230.23user 110.37system 27:21.46elapsed 142%CPU (0avgtext+0avgdata 127116maxresident)k 0inputs+4072584outputs (263249major+15113217minor)pagefaults 0swaps
[RFT PATCH -perfbook v3 5/4] Makefile: Fix regression of parallel run of autodate.sh
In reply to Akira YokosawaCommit xxxxxxxxxxxx ("Makefile: Fix issues WRT parallel runs of pdflatex") changed dependency around autodate.tex so that it depends only on files in the Git repository.
However, this change causes autodate.sh to see untracked temporary intermediate files from concurrent conversion of figures. It ends up adding the "(m)" maker in autodate.tex even when there is no untracked file in the beginning.
Fix the regression by partially reverting xxxxxxxxxxxx and restoring dependencies of autodate.tex on converted PDF and .fcv files.
Note: This regression was caught by parallel-latex-regress.sh when Paul ran it under clean Git repository.
---
Paul,
I can merge this with 2/4, but I'd rather keep this partial revert as an example of regression found by the script.
If you push this series to master, please amend the commit ids in the change log.
Thanks, Akira
--
Makefile | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
Makefile +4 -6
@@ -228,14 +228,12 @@ ifeq ($(NEWTXTEXT),) endif sh utilities/runfirstlatex.sh $(basename $@) -autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(SOURCES_OF_SNIPPET) \ - $(LST_SOURCES) $(FIGSOURCES) $(DOTSOURCES) $(EPSORIGIN) \ - $(SVGSOURCES) $(GITREFSTAGS) \ - utilities/autodate.sh +autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(LST_SOURCES) \ + $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \ + $(GITREFSTAGS) utilities/autodate.sh sh utilities/autodate.sh -perfbook_flat.tex: autodate.tex $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) \ - $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) +perfbook_flat.tex: autodate.tex ifndef LATEXPAND $(error --> $@: latexpand not found. Please install it) endif
Signature · 1 line
2.17.1
Re: [RFT PATCH -perfbook v3 5/4] Makefile: Fix regression of parallel run of autodate.sh
Quoted 12 lines · Akira Yokosawa
Commit xxxxxxxxxxxx ("
Makefile: Fix issues WRT parallel runs of pdflatex") changed dependency around autodate.tex so that it depends only on files in the Git repository.However, this change causes
autodate.shto see untracked temporary intermediate files from concurrent conversion of figures. It ends up adding the "(m)" maker in autodate.tex even when there is no untracked file in the beginning.Fix the regression by partially reverting xxxxxxxxxxxx and restoring dependencies of autodate.tex on converted PDF and .fcv files.
Quoted 6 lines
Note: This regression was caught by
parallel-latex-regress.shwhen Paul ran it under clean Git repository.Fixes xxxxxxxxxxxx ("Makefile: Fix issues WRT parallel runs of pdflatex")Reported-by Paul E. McKenney <[email protected]>Signed-off-by Akira Yokosawa <[email protected]>
Much better! Queued with SHA-1 applied and pushed, thank you!!!
At least this way we tested the test. ;-)
Thanx, Paul
Quoted 41 lines
---
Paul,I can merge this with 2/4, but I'd rather keep this partial revert as an example of regression found by the script.
If you push this series to master, please amend the commit ids in the change log.
Thanks, Akira
--Makefile | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)Makefile +4
-6@@ -228,14 +228,12 @@ ifeq ($(NEWTXTEXT),) endif sh utilities/runfirstlatex.sh $(basename $@) -autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(SOURCES_OF_SNIPPET) \ - $(LST_SOURCES) $(FIGSOURCES) $(DOTSOURCES) $(EPSORIGIN) \ - $(SVGSOURCES) $(GITREFSTAGS) \ - utilities/autodate.sh +autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(LST_SOURCES) \ + $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \ + $(GITREFSTAGS) utilities/autodate.sh sh utilities/autodate.sh -perfbook_flat.tex: autodate.tex $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) \ - $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) +perfbook_flat.tex: autodate.tex ifndef LATEXPAND $(error --> $@: latexpand not found. Please install it) endifSignature · 1 line
2.17.1