Makefile: optimize noop runs, add shared.mak
Subsystems (2)
v2 cover Makefile: optimize noop runs, add shared.mak
As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/
This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).
This v2 re-roll addresses trivial commit message/comment grammar/typo issues pointed out by Eric Sunshine, thanks Eric!
Ævar Arnfjörð Bjarmason (8):
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" template
Documentation/Makefile | 63 +++------------------- Makefile | 118 +++++++++++++---------------------------- config.mak.uname | 1 - shared.mak | 109 +++++++++++++++++++++++++++++++++++++ t/Makefile | 3 ++ t/interop/Makefile | 3 ++ templates/Makefile | 8 ++- 7 files changed, 160 insertions(+), 145 deletions(-) create mode 100644 shared.mak
Range-diff against v1:
1: f74b47662b7 = 1: b652fc78fda Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it 2: b0c63abe091 = 2: b0c9be581a6 Makefile: disable GNU make built-in wildcard rules 3: c6c6f7cf8d8 = 3: ed6fd1c0bd1 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) 4: ed64cd1bd4a = 4: 4c6d8089fff Makefile: move ".SUFFIXES" rule to shared.mak 5: 1749085b929 = 5: f1f02c71dbc Makefile: move $(comma), $(empty) and $(space) to shared.mak 6: c25284b24cf = 6: fb877060d6b Makefile: add "$(QUIET)" boilerplate to shared.mak 7: 3daef7672be = 7: 90d804ea9a0 Makefile: use $(wspfx) for $(QUIET...) in shared.mak 8: aca560ca410 ! 8: 59c1b7032db Makefiles: add and use wildcard "mkdir -p" template
@@ Commit message
But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run
- of a performance test similar to thatnoted downthread of [1] in [2]
+ of a performance test similar to that noted downthread of [1] in [2]
shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):
@@ shared.mak: ifndef V
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
-+## "mkdir -p" when running in parallel, but won't need to exhaustively ++## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule.
Signature · 1 line
2.34.1.1215.g6e154b84c77
PATCH v2 1/8 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
This does have the potential downside that if e.g. templates/Makefile would like to include this "shared.mak" in the future the semantics of such a Makefile will change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.
This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
--- Documentation/Makefile | 6 +++--- Makefile | 13 +++---------- shared.mak | 9 +++++++++ t/Makefile | 3 +++ t/interop/Makefile | 3 +++ templates/Makefile | 3 +++ 6 files changed, 24 insertions(+), 13 deletions(-) create mode 100644 shared.mak
Documentation/Makefile +3 -3
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Guard against environment variables MAN1_TXT = MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n:: $(MAKE) -C po $@ endif -# Delete the target file on error -.DELETE_ON_ERROR: - .PHONY: FORCE
Makefile +3 -10
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # The default target of this Makefile is... all::
@@ -2169,16 +2172,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell strip: $(PROGRAMS) git$X $(STRIP) $(STRIP_OPTS) $^ -### Flags affecting all rules - -# A GNU make extension since gmake 3.72 (released in late 1994) to -# remove the target of rules if commands in those rules fail. The -# default is to only do that if make itself receives a signal. Affects -# all targets, see: -# -# info make --index-search=.DELETE_ON_ERROR -.DELETE_ON_ERROR: - ### Target-specific flags and dependencies # The generic compilation pattern rule and automatically
shared.mak new file +9 -0
@@ -0,0 +1,9 @@ +### Flags affecting all rules + +# A GNU make extension since gmake 3.72 (released in late 1994) to +# remove the target of rules if commands in those rules fail. The +# default is to only do that if make itself receives a signal. Affects +# all targets, see: +# +# info make --index-search=.DELETE_ON_ERROR +.DELETE_ON_ERROR:
t/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Run tests # # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS
templates/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # make and install sample templates ifndef V
Signature · 1 line
2.34.1.1215.g6e154b84c77
PATCH v2 2/8 Makefile: disable GNU make built-in wildcard rules
In reply to Ævar Arnfjörð BjarmasonOverride built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.
The speed-up may wary. I've seen 1-10% depending on the speed of the local disk, caches, -jN etc. Running:
strace -f -c -S calls make -j1 NO_TCLTK=Y
Shows that we reduce the number of syscalls we make, mostly in "stat" calls.
We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early. Doing so might make us a bit faster still. But doing so is a much bigger hammer, since it will disable all built-in rules, some (all?) of which can be seen with:
make -f/dev/null -p | grep -v -e ^# -e ^$
We may have something that relies on them, so let's go for the more isolated optimization here that gives us most or all of the wins.
1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html
--- shared.mak | 11 +++++++++++ 1 file changed, 11 insertions(+)
shared.mak +11 -0
@@ -1,3 +1,14 @@ +### Remove GNU make implicit rules + +## This speeds things up since we don't need to look for and stat() a +## "foo.c,v" every time a rule referring to "foo.c" is in play. See +## "make -p -f/dev/null | grep ^%::'". +%:: %,v +%:: RCS/%,v +%:: RCS/% +%:: s.% +%:: SCCS/s.% + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.34.1.1215.g6e154b84c77
PATCH v2 3/8 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
In reply to Ævar Arnfjörð BjarmasonCombine the definitions of $(FIND_SOURCE_FILES) and $(LIB_H) to speed up the Makefile, as these are the two main expensive $(shell) commands that we execute unconditionally.
When see what was in $(FOUND_SOURCE_FILES) that wasn't in $(LIB_H) via the ad-hoc test of:
$(error $(filter-out $(LIB_H),$(filter %.h,$(ALL_SOURCE_FILES))))
$(error $(filter-out $(ALL_SOURCE_FILES),$(filter %.h,$(LIB_H))))We'll get, respectively:
Makefile:850: *** t/helper/test-tool.h. Stop.
Makefile:850: *** . Stop.I.e. we only had a discrepancy when it came to t/helper/test-tool.h. In terms of correctness this was broken before, but now works:
$ make t/helper/test-tool.hco
HDR t/helper/test-tool.hThis speeds things up a lot:
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10
Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1
Time (mean ± σ): 159.9 ms ± 6.8 ms [User: 137.2 ms, System: 28.0 ms]
Range (min … max): 154.6 ms … 175.9 ms 10 runs Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0
Time (mean ± σ): 100.0 ms ± 1.3 ms [User: 84.2 ms, System: 20.2 ms]
Range (min … max): 98.8 ms … 102.8 ms 10 runs
Summary
'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran
1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1'--- Makefile | 54 ++++++++++++++++++++++++++---------------------------- 1 file changed, 26 insertions(+), 28 deletions(-)
Makefile +26 -28
@@ -826,12 +826,33 @@ GENERATED_H += hook-list.h .PHONY: generated-hdrs generated-hdrs: $(GENERATED_H) -LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \ +## Exhaustive lists of our source files, either dynamically generated, +## or hardcoded. +SOURCES_CMD = ( \ + git ls-files \ + '*.[hcS]' \ + '*.sh' \ + ':!*[tp][0-9][0-9][0-9][0-9]*' \ + ':!contrib' \ + 2>/dev/null || \ $(FIND) . \ - -name .git -prune -o \ - -name t -prune -o \ - -name Documentation -prune -o \ - -name '*.h' -print))) + \( -name .git -type d -prune \) \ + -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ + -o \( -name contrib -type d -prune \) \ + -o \( -name build -type d -prune \) \ + -o \( -name 'trash*' -type d -prune \) \ + -o \( -name '*.[hcS]' -type f -print \) \ + -o \( -name '*.sh' -type f -print \) \ + | sed -e 's|^\./||' \ + ) +FOUND_SOURCE_FILES := $(shell $(SOURCES_CMD)) + +FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) +FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES)) + +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) + +LIB_H = $(FOUND_H_SOURCES) LIB_OBJS += abspath.o LIB_OBJS += add-interactive.o
@@ -2758,26 +2779,6 @@ perl/build/man/man3/Git.3pm: perl/Git.pm $(QUIET_GEN)mkdir -p $(dir $@) && \ pod2man $< $@ -FIND_SOURCE_FILES = ( \ - git ls-files \ - '*.[hcS]' \ - '*.sh' \ - ':!*[tp][0-9][0-9][0-9][0-9]*' \ - ':!contrib' \ - 2>/dev/null || \ - $(FIND) . \ - \( -name .git -type d -prune \) \ - -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ - -o \( -name contrib -type d -prune \) \ - -o \( -name build -type d -prune \) \ - -o \( -name 'trash*' -type d -prune \) \ - -o \( -name '*.[hcS]' -type f -print \) \ - -o \( -name '*.sh' -type f -print \) \ - | sed -e 's|^\./||' \ - ) - -FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES)) - $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \ echo $(FOUND_SOURCE_FILES) | xargs etags -a -o $@+ && \
@@ -2987,9 +2988,6 @@ check: $(GENERATED_H) exit 1; \ fi -FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) -COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) - %.cocci.patch: %.cocci $(COCCI_SOURCES) $(QUIET_SPATCH) \ if test $(SPATCH_BATCH_SIZE) = 0; then \
Signature · 1 line
2.34.1.1215.g6e154b84c77
PATCH v2 4/8 Makefile: move ".SUFFIXES" rule to shared.mak
In reply to Ævar Arnfjörð BjarmasonThis was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.
This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms]
Range (min … max): 112.8 ms … 148.4 ms 26 runs Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs
Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'
--- Makefile | 2 -- shared.mak | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
@@ -2549,8 +2549,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S)) ASM_OBJ := $(ASM_SRC:S=o) C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS)) -.SUFFIXES: - $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir) $(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +5 -0
@@ -9,6 +9,11 @@ %:: s.% %:: SCCS/s.% +## Likewise delete default $(SUFFIXES). See: +## +## info make --index-search=.DELETE_ON_ERROR +.SUFFIXES: + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.34.1.1215.g6e154b84c77
Re: [PATCH v2 4/8] Makefile: move ".SUFFIXES" rule to shared.mak
Quoted 5 lines · Ævar Arnfjörð Bjarmason
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms]
Range (min … max): 112.8 ms … 148.4 ms 26 runsQuoted 7 lines
Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runsSummary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'
Nice speed-up. Though I am not sure I totally understand where it comes from ;). Reading 30248886ce8 and the documentation on .SUFFIXES from [1], I am still unclear. I guess removing the obsolete built-in suffix rules gives make less work to do in general?
So long as we're not depending on any of these, this seems like a nice little boost to me.
shared.mak line 15
Quoted 12 lines
shared.mak +4
-0@@ -9,6 +9,11 @@ %:: s.% %:: SCCS/s.% +## Likewise delete default $(SUFFIXES). See: +## +## info make --index-search=.DELETE_ON_ERROR +.SUFFIXES:
Hmm. s/DELETE_ON_ERROR/SUFFIXES? Or perhaps I'm holding this whole thing incorrectly:
~/s/git [nand] (ab/make-noop) $ info make --index-search=.DELETE_ON_ERROR no index entries found for '.DELETE_ON_ERROR'
~/s/git [nand] (ab/make-noop) $ info make --index-search=.SUFFIXES no index entries found for '.SUFFIXES'
Thanks,
Taylor
[1]: https://www.gnu.org/software/make/manual/html_node/Suffix-Rules.html
Re: [PATCH v2 4/8] Makefile: move ".SUFFIXES" rule to shared.mak
Quoted 10 lines · Taylor Blau
Quoted 9 lines · Ævar Arnfjörð Bjarmason
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man' Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1 Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms] Range (min … max): 112.8 ms … 148.4 ms 26 runsBenchmark 2: make -C Documentation -j1 man' in 'HEAD~0 Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms] Range (min … max): 89.8 ms … 111.8 ms 32 runs
Quoted 8 lines
Quoted 3 lines
Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'Nice speed-up. Though I am not sure I totally understand where it comes from ;). Reading
30248886ce8and the documentation on .SUFFIXES from [1], I am still unclear. I guess removing the obsolete built-in suffix rules gives make less work to do in general?
I'll update the commit message, but basically the same applies as for 2/8 here (<[email protected]>), or if you run "make" with "--debug=a". I.e. if before/after this change you do:
git clean -dxf; make -C Documentation/ --debug=a git-status.1 >/tmp/b 2>&1
git clean -dxf; make -C Documentation/ --debug=a git-status.1 >/tmp/a 2>&1You'll get:
$ wc -l /tmp/[ba]
6515 /tmp/a
144051 /tmp/b
150566 totalWhich e.g. for "git-status.txt" starts with (I cut a lot out, there's way more than this just for that file):
Considering target file 'git-status.txt'.
Looking for an implicit rule for 'git-status.txt'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.o'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.c'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.cc'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.C'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.cpp'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.p'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.f'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.F'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.m'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.r'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.s'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.S'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.mod'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.sh'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.o'.
[...]
I.e. given a foo.txt "make" by default will exhaustively consider foo foo.txt.c, foo.txt.cpp etc. etc.
This is all ultimately a part of make's implicit rule mechanism. I.e. even if you have no Makefile at all you can run "make main" if you have a main.c in your directory, and it'll discover it and compile it with implicit rules, unless you explicitl disable them, e.g. with "-r":
$ rm Makefile hello; make hello
rm: cannot remove 'Makefile': No such file or directory
rm: cannot remove 'hello': No such file or directory
cc hello.o -o hello
$ rm Makefile hello; make -r hello
rm: cannot remove 'Makefile': No such file or directory
make: *** No rule to make target 'hello'. Stop.Quoted 2 lines
So long as we're not depending on any of these, this seems like a nice little boost to me.
Yes, definitely!
Quoted 16 lines
shared.mak line 15
Quoted 12 lines
shared.mak +4
-0@@ -9,6 +9,11 @@ %:: s.% %:: SCCS/s.% +## Likewise delete default $(SUFFIXES). See: +## +## info make --index-search=.DELETE_ON_ERROR +.SUFFIXES:Hmm. s/DELETE_ON_ERROR/SUFFIXES? Or perhaps I'm holding this whole thing incorrectly:
Quoted 4 lines
~/s/git [nand] (ab/make-noop) $ info make --index-search=.DELETE_ON_ERROR no index entries found for '.DELETE_ON_ERROR'
~/s/git [nand] (ab/make-noop) $ info make --index-search=.SUFFIXES no index entries found for '.SUFFIXES'
Yes that's a copy/paste error, it should be .SUFFIXES.
But both commands should work, or emit an error like:
$ info doesnotexist --index-search=.DELETE_ON_ERROR
info: No menu item 'doesnotexist' in node '(dir)Top'A broken OS package? Self-built "make"?
In any case I could also link to
https://www.gnu.org/software/make/manual/html_node/Suffix-Rules.html;
which is the same information online (but may not match your local "make" version)>
PATCH v2 5/8 Makefile: move $(comma), $(empty) and $(space) to shared.mak
In reply to Ævar Arnfjörð BjarmasonMove these variables over to the shared.max, we'll make use of them in a subsequent commit. There was no reason for these to be "simply expanded variables", so let's use the normal lazy "=" assignment here.
See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.
--- Makefile | 4 ---- shared.mak | 8 ++++++++ 2 files changed, 8 insertions(+), 4 deletions(-)
Makefile +0 -4
@@ -1279,10 +1279,6 @@ endif ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS) ALL_LDFLAGS = $(LDFLAGS) -comma := , -empty := -space := $(empty) $(empty) - ifdef SANITIZE SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag)) BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)
shared.mak +8 -0
@@ -23,3 +23,11 @@ # # info make --index-search=.DELETE_ON_ERROR .DELETE_ON_ERROR: + +### Global variables + +## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among other Makefile syntax. +comma = , +empty = +space = $(empty) $(empty)
Signature · 1 line
2.34.1.1215.g6e154b84c77
PATCH v2 6/8 Makefile: add "$(QUIET)" boilerplate to shared.mak
In reply to Ævar Arnfjörð BjarmasonThe $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.
Since we're not using the environment to pass these around we don't need to export the "QUIET_GEN" and "QUIET_BUILT_IN" variables anymore. The "QUIET_GEN" variable is used in "git-gui/Makefile" and "gitweb/Makefile", but they've got their own definition for those. The "QUIET_BUILT_IN" variable is only used in the top-level "Makefile". We still need to export the "V" variable.
--- Documentation/Makefile | 32 ------------------------- Makefile | 33 -------------------------- config.mak.uname | 1 - shared.mak | 53 ++++++++++++++++++++++++++++++++++++++++++ templates/Makefile | 5 ---- 5 files changed, 53 insertions(+), 71 deletions(-)
Documentation/Makefile +0 -32
@@ -218,38 +218,6 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR)) ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)' endif -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring $(MAKEFLAGS),w),w) -PRINT_DIR = --no-print-directory -else # "make -w" -NO_SUBDIR = : -endif - -ifneq ($(findstring $(MAKEFLAGS),s),s) -ifndef V - QUIET = @ - QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; - QUIET_XMLTO = @echo ' ' XMLTO $@; - QUIET_DB2TEXI = @echo ' ' DB2TEXI $@; - QUIET_MAKEINFO = @echo ' ' MAKEINFO $@; - QUIET_DBLATEX = @echo ' ' DBLATEX $@; - QUIET_XSLTPROC = @echo ' ' XSLTPROC $@; - QUIET_GEN = @echo ' ' GEN $@; - QUIET_STDERR = 2> /dev/null - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir - - QUIET_LINT_GITLINK = @echo ' ' LINT GITLINK $<; - QUIET_LINT_MANSEC = @echo ' ' LINT MAN SEC $<; - QUIET_LINT_MANEND = @echo ' ' LINT MAN END $<; - - export V -endif -endif - all: html man html: $(DOC_HTML)
Makefile +0 -33
@@ -1971,39 +1971,6 @@ ifndef PAGER_ENV PAGER_ENV = LESS=FRX LV=-c endif -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring w,$(MAKEFLAGS)),w) -PRINT_DIR = --no-print-directory -else # "make -w" -NO_SUBDIR = : -endif - -ifneq ($(findstring s,$(MAKEFLAGS)),s) -ifndef V - QUIET_CC = @echo ' ' CC $@; - QUIET_AR = @echo ' ' AR $@; - QUIET_LINK = @echo ' ' LINK $@; - QUIET_BUILT_IN = @echo ' ' BUILTIN $@; - QUIET_GEN = @echo ' ' GEN $@; - QUIET_LNCP = @echo ' ' LN/CP $@; - QUIET_XGETTEXT = @echo ' ' XGETTEXT $@; - QUIET_MSGFMT = @echo ' ' MSGFMT $@; - QUIET_GCOV = @echo ' ' GCOV $@; - QUIET_SP = @echo ' ' SP $<; - QUIET_HDR = @echo ' ' HDR $(<:hcc=h); - QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir - export V - export QUIET_GEN - export QUIET_BUILT_IN -endif -endif - ifdef NO_INSTALL_HARDLINKS export NO_INSTALL_HARDLINKS endif
config.mak.uname +0 -1
@@ -720,7 +720,6 @@ vcxproj: git diff-index --cached --quiet HEAD -- # Make .vcxproj files and add them - unset QUIET_GEN QUIET_BUILT_IN; \ perl contrib/buildsystems/generate -g Vcxproj git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
shared.mak +53 -0
56 lines not on this page · raw message
templates/Makefile +0 -5
@@ -2,11 +2,6 @@ include ../shared.mak # make and install sample templates - -ifndef V - QUIET = @ -endif - INSTALL ?= install TAR ?= tar RM ?= rm -f
Signature · 1 line
2.34.1.1215.g6e154b84c77
PATCH v2 7/8 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
In reply to Ævar Arnfjörð BjarmasonChange the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding, to guarantee that it's consistent with the "TRACK_template" template.
$ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
-> GIT-CFLAGS PARAMETERS (changed)
-> CC grep.o--- shared.mak | 54 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 30 insertions(+), 24 deletions(-)
shared.mak +30 -24
@@ -32,6 +32,12 @@ comma = , empty = space = $(empty) $(empty) +## wspfx: the whitespace prefix padding for $(QUIET...) and similarly +## aligned output. +wspfx = $(space)$(space)$(space) +wspfx_SQ = '$(subst ','\'',$(wspfx))' +# ' closing quote to appease Emacs make-mode.elxo + ### Quieting ## common QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
@@ -47,39 +53,39 @@ ifneq ($(findstring s,$(MAKEFLAGS)),s) ifndef V ## common QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ + QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo $(wspfx_SQ) SUBDIR $$subdir; \ $(MAKE) $(PRINT_DIR) -C $$subdir QUIET = @ - QUIET_GEN = @echo ' ' GEN $@; + QUIET_GEN = @echo $(wspfx_SQ) GEN $@; ## Used in "Makefile" - QUIET_CC = @echo ' ' CC $@; - QUIET_AR = @echo ' ' AR $@; - QUIET_LINK = @echo ' ' LINK $@; - QUIET_BUILT_IN = @echo ' ' BUILTIN $@; - QUIET_LNCP = @echo ' ' LN/CP $@; - QUIET_XGETTEXT = @echo ' ' XGETTEXT $@; - QUIET_MSGFMT = @echo ' ' MSGFMT $@; - QUIET_GCOV = @echo ' ' GCOV $@; - QUIET_SP = @echo ' ' SP $<; - QUIET_HDR = @echo ' ' HDR $(<:hcc=h); - QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_CC = @echo $(wspfx_SQ) CC $@; + QUIET_AR = @echo $(wspfx_SQ) AR $@; + QUIET_LINK = @echo $(wspfx_SQ) LINK $@; + QUIET_BUILT_IN = @echo $(wspfx_SQ) BUILTIN $@; + QUIET_LNCP = @echo $(wspfx_SQ) LN/CP $@; + QUIET_XGETTEXT = @echo $(wspfx_SQ) XGETTEXT $@; + QUIET_MSGFMT = @echo $(wspfx_SQ) MSGFMT $@; + QUIET_GCOV = @echo $(wspfx_SQ) GCOV $@; + QUIET_SP = @echo $(wspfx_SQ) SP $<; + QUIET_HDR = @echo $(wspfx_SQ) HDR $(<:hcc=h); + QUIET_RC = @echo $(wspfx_SQ) RC $@; + QUIET_SPATCH = @echo $(wspfx_SQ) SPATCH $<; ## Used in "Documentation/Makefile" - QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; - QUIET_XMLTO = @echo ' ' XMLTO $@; - QUIET_DB2TEXI = @echo ' ' DB2TEXI $@; - QUIET_MAKEINFO = @echo ' ' MAKEINFO $@; - QUIET_DBLATEX = @echo ' ' DBLATEX $@; - QUIET_XSLTPROC = @echo ' ' XSLTPROC $@; - QUIET_GEN = @echo ' ' GEN $@; + QUIET_ASCIIDOC = @echo $(wspfx_SQ) ASCIIDOC $@; + QUIET_XMLTO = @echo $(wspfx_SQ) XMLTO $@; + QUIET_DB2TEXI = @echo $(wspfx_SQ) DB2TEXI $@; + QUIET_MAKEINFO = @echo $(wspfx_SQ) MAKEINFO $@; + QUIET_DBLATEX = @echo $(wspfx_SQ) DBLATEX $@; + QUIET_XSLTPROC = @echo $(wspfx_SQ) XSLTPROC $@; + QUIET_GEN = @echo $(wspfx_SQ) GEN $@; QUIET_STDERR = 2> /dev/null - QUIET_LINT_GITLINK = @echo ' ' LINT GITLINK $<; - QUIET_LINT_MANSEC = @echo ' ' LINT MAN SEC $<; - QUIET_LINT_MANEND = @echo ' ' LINT MAN END $<; + QUIET_LINT_GITLINK = @echo $(wspfx_SQ) LINT GITLINK $<; + QUIET_LINT_MANSEC = @echo $(wspfx_SQ) LINT MAN SEC $<; + QUIET_LINT_MANEND = @echo $(wspfx_SQ) LINT MAN END $<; export V endif
Signature · 1 line
2.34.1.1215.g6e154b84c77
PATCH v2 8/8 Makefiles: add and use wildcard "mkdir -p" template
In reply to Ævar Arnfjörð BjarmasonAdd a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).
As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.
I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.
But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
Time (mean ± σ): 2.914 s ± 0.062 s [User: 2.449 s, System: 0.489 s]
Range (min … max): 2.834 s … 3.020 s 10 runs Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s]
Range (min … max): 2.229 s … 2.397 s 10 runs
Summary
'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'
So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.
This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://gitlab.com/avar/git-hyperfine/
--- Documentation/Makefile | 25 +++---------------------- Makefile | 12 +++++++----- shared.mak | 17 +++++++++++++++++ 3 files changed, 27 insertions(+), 27 deletions(-)
Documentation/Makefile +3 -22
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo print-man1: @for i in $(MAN1_TXT); do echo $$i; done -## Lint: Common -.build: - $(QUIET)mkdir $@ -.build/lint-docs: | .build - $(QUIET)mkdir $@ - ## Lint: gitlink -.build/lint-docs/gitlink: | .build/lint-docs - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT)) -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config $(LINT_DOCS_GITLINK): lint-gitlink.perl $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \ $< \ $(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt lint-docs-gitlink: $(LINT_DOCS_GITLINK) ## Lint: man-end-blurb -.build/lint-docs/man-end-blurb: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@ .PHONY: lint-docs-man-end-blurb -lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB) ## Lint: man-section-order -.build/lint-docs/man-section-order: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@ .PHONY: lint-docs-man-section-order lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
Makefile +7 -5
@@ -2714,7 +2714,8 @@ all:: $(MOFILES) endif po/build/locale/%/LC_MESSAGES/git.mo: po/%.po - $(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $< + $(call mkdir_p_parent_template) + $(QUIET_MSGFMT)$(MSGFMT) -o $@ $< LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm) LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2730,15 +2731,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS)) endif perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES - $(QUIET_GEN)mkdir -p $(dir $@) && \ + $(call mkdir_p_parent_template) + $(QUIET_GEN) \ sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \ -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \ -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \ < $< > $@ perl/build/man/man3/Git.3pm: perl/Git.pm - $(QUIET_GEN)mkdir -p $(dir $@) && \ - pod2man $< $@ + $(call mkdir_p_parent_template) + $(QUIET_GEN)pod2man $< $@ $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \
@@ -2872,7 +2874,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $( all:: $(TEST_PROGRAMS) $(test_bindir_programs) bin-wrappers/%: wrap-for-bin.sh - @mkdir -p bin-wrappers + $(call mkdir_p_parent_template) $(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \ -e 's|@@BUILD_DIR@@|$(shell pwd)|' \ -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
shared.mak +17 -0
@@ -59,6 +59,8 @@ ifndef V QUIET = @ QUIET_GEN = @echo $(wspfx_SQ) GEN $@; + QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D); + ## Used in "Makefile" QUIET_CC = @echo $(wspfx_SQ) CC $@; QUIET_AR = @echo $(wspfx_SQ) AR $@;
@@ -90,3 +92,18 @@ ifndef V export V endif endif + +### Templates + +## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@ +## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not +## needed. +## +## Is racy, but in a good way; we might redundantly (and safely) +## "mkdir -p" when running in parallel, but won't need to exhaustively create +## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule. +define mkdir_p_parent_template +$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D))) +endef
Signature · 1 line
2.34.1.1215.g6e154b84c77
On Fri, Dec 24 2021, Ævar Arnfjörð Bjarmason wrote:
A gentle reminder about considering picking up this topic that got lost around the holidays.
This one is independent of the other Makefile topic I've got outstanding at
https://lore.kernel.org/git/[email protected]/
This thread doesn't have any reviews, but I see that's because I screwed up and omitted the In-Reply-To at the time. There was feeback on the v1 of this:
https://lore.kernel.org/git/[email protected]/
That v1 was in turn split off from a larger series, and as the range-diff for that v1 shows the split-off version was was mostly unchanged from that version, which had a few eyeballs on it.
Quoted 58 lines
As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).
This v2 re-roll addresses trivial commit message/comment grammar/typo issues pointed out by Eric Sunshine, thanks Eric!
Ævar Arnfjörð Bjarmason (8):
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" templateDocumentation/Makefile | 63 +++------------------- Makefile | 118 +++++++++++++---------------------------- config.mak.uname | 1 - shared.mak | 109 +++++++++++++++++++++++++++++++++++++ t/Makefile | 3 ++ t/interop/Makefile | 3 ++ templates/Makefile | 8 ++- 7 files changed, 160 insertions(+), 145 deletions(-) create mode 100644 shared.makRange-diff against v1:
1: f74b47662b7 = 1: b652fc78fda Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it 2: b0c63abe091 = 2: b0c9be581a6 Makefile: disable GNU make built-in wildcard rules 3: c6c6f7cf8d8 = 3: ed6fd1c0bd1 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) 4: ed64cd1bd4a = 4: 4c6d8089fff Makefile: move ".SUFFIXES" rule to shared.mak 5: 1749085b929 = 5: f1f02c71dbc Makefile: move $(comma), $(empty) and $(space) to shared.mak 6: c25284b24cf = 6: fb877060d6b Makefile: add "$(QUIET)" boilerplate to shared.mak 7: 3daef7672be = 7: 90d804ea9a0 Makefile: use $(wspfx) for $(QUIET...) in shared.mak 8: aca560ca410 ! 8: 59c1b7032db Makefiles: add and use wildcard "mkdir -p" template@@ Commit message
But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run
- of a performance test similar to thatnoted downthread of [1] in [2]
+ of a performance test similar to that noted downthread of [1] in [2]
shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):@@ shared.mak: ifndef V
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
-+## "mkdir -p" when running in parallel, but won't need to exhaustively ++## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule.
v3 cover Makefile: optimize noop runs, add shared.mak
In reply to Ævar Arnfjörð BjarmasonAs the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/
This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).
This v3 re-roll:
* Addresses a minor comment/documentation issue pointed out by Taylor Blau.
* Adjusts the ".SUFFIXES" commit message to note the source of the speed-up from that change, in response to a question from Taylor.
* Changes the contrib/scalar/Makefile to make use of the new shared.mak, which allows for deleting some copy/pasted code in it in favor of the same shared logic.
The original version of these patches was written before that file landed in-tree.
For v2, see: https://lore.kernel.org/git/[email protected]/
Ævar Arnfjörð Bjarmason (9):
scalar Makefile: set the default target after the includes
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" template
Documentation/Makefile | 63 ++------------------ Makefile | 118 ++++++++++++-------------------------- config.mak.uname | 1 - contrib/scalar/Makefile | 19 ++---- contrib/scalar/t/Makefile | 3 + shared.mak | 109 +++++++++++++++++++++++++++++++++++ t/Makefile | 3 + t/interop/Makefile | 3 + t/perf/Makefile | 3 + templates/Makefile | 8 +-- 10 files changed, 170 insertions(+), 160 deletions(-) create mode 100644 shared.mak
Range-diff against v2:
-: ----------- > 1: 2404c4d8b96 scalar Makefile: set the default target after the includes 1: 97dccacce20 ! 2: 96a490bec54 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
@@ Makefile: shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
# The generic compilation pattern rule and automatically
+ ## contrib/scalar/Makefile ##
+@@
++# Import tree-wide shared Makefile behavior and libraries ++include ../../shared.mak
++
+ QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
+ QUIET_SUBDIR1 =
+
+
+ ## contrib/scalar/t/Makefile ##
+@@
++# Import tree-wide shared Makefile behavior and libraries ++include ../../../shared.mak
++
+ # Run scalar tests
+ #
+ # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin +
## shared.mak (new) ##
@@
+### Flags affecting all rules
@@ t/interop/Makefile
export GIT_TEST_OPTIONS
+ ## t/perf/Makefile ##
+@@
++# Import tree-wide shared Makefile behavior and libraries ++include ../../shared.mak
++
+ -include ../../config.mak
+ export GIT_TEST_OPTIONS
+
+
## templates/Makefile ##
@@
+# Import tree-wide shared Makefile behavior and libraries
2: b2bf32ab071 = 3: 9392e3c3e97 Makefile: disable GNU make built-in wildcard rules 3: 275aba624fa = 4: 07cf9daa9d6 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) 4: 7103c40de64 ! 5: 16f2e3ff35b Makefile: move ".SUFFIXES" rule to shared.mak
@@ Commit message
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'
+ The reason for that can be seen when comparing that run with
+ "--debug=a". Without this change making a target like "git-status.1"
+ will cause "make" to consider not only "git-status.txt", but
+ "git-status.txt.o", as well as numerous other implicit suffixes such
+ as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after
+ example.
+
+ So this is causing us to omit a bunch of work we didn't need to
+ do. For making "git-status.1" the "--debug=a" output is reduced from
+ ~140k lines to ~6k.
+
+ 1. https://lore.kernel.org/git/[email protected]/
+
## Makefile ##
@@ shared.mak
+## Likewise delete default $(SUFFIXES). See:
+##
-+## info make --index-search=.DELETE_ON_ERROR
++## info make --index-search=.SUFFIXES
+.SUFFIXES:
+
### Flags affecting all rules
5: 4a5c647b5e7 = 6: 1b6ecb27f02 Makefile: move $(comma), $(empty) and $(space) to shared.mak
6: 940299d2a03 ! 7: 471067deefc Makefile: add "$(QUIET)" boilerplate to shared.mak
@@ config.mak.uname: vcxproj:
git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
+ ## contrib/scalar/Makefile ##
+@@
+ # Import tree-wide shared Makefile behavior and libraries
+ include ../../shared.mak
+
+-QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
+-QUIET_SUBDIR1 =
+-
+-ifneq ($(findstring s,$(MAKEFLAGS)),s)
+-ifndef V
+- QUIET_GEN = @echo ' ' GEN $@;
+- QUIET_SUBDIR0 = +@subdir=
+- QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \
+- $(MAKE) $(PRINT_DIR) -C $$subdir
+-else
+- export V
+-endif
+-endif
+-
+ include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak
+
## shared.mak ##
@@
comma = ,
7: 1e13fee526d = 8: 510306d2219 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
8: 250b32540d9 = 9: 85bb74aa32f Makefiles: add and use wildcard "mkdir -p" templateSignature · 1 line
2.35.1.1175.gf9e1b23ea35
PATCH v3 1/9 scalar Makefile: set the default target after the includes
Make have the "contrib/scalar/Makefile" be stylistically consistent with the top-level "Makefile" in including other makefiles before setting the default target.
This adjusts code added in 0a43fb22026 (scalar: create a rudimentary executable, 2021-12-03), it's a style-only change, in a subsequent commit the "QUIET" boilerplate at the beginning of this file will be retrieved via an include, and having an "all:" between the two set of "include"'s after that change would look odd.
--- contrib/scalar/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
contrib/scalar/Makefile +2 -2
@@ -12,12 +12,12 @@ else endif endif -all: - include ../../config.mak.uname -include ../../config.mak.autogen -include ../../config.mak +all: + TARGETS = scalar$(X) scalar.o GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
Re: [PATCH v3 1/9] scalar Makefile: set the default target after the includes
Quoted 3 lines · Ævar Arnfjörð Bjarmason
Make have the "contrib/scalar/Makefile" be stylistically consistent with the top-level "
Makefile" in including other makefiles before setting the default target.
I think that goal makes sense, but the patch looks like doing a different thing. The top-level "Makefile" does it correctly, by starting the file with
# The default target of this Makefile is... all::
that declares that saying "make" is equivalent to "make all" upfront.
The patch does not do that; it moves "all" even lower.
What am I missing?
Quoted 29 lines
This adjusts code added in
0a43fb22026(scalar: create a rudimentary executable, 2021-12-03), it's a style-only change, in a subsequent commit the "QUIET" boilerplate at the beginning of this file will be retrieved via an include, and having an "all:" between the two set of "include"'s after that change would look odd.Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- contrib/scalar/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)contrib/scalar/Makefile +2
-2@@ -12,12 +12,12 @@ else endif endif -all: - include ../../config.mak.uname -include ../../config.mak.autogen -include ../../config.mak +all: + TARGETS = scalar$(X) scalar.o GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
PATCH v3 2/9 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
In reply to Ævar Arnfjörð BjarmasonWe have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
This does have the potential downside that if e.g. templates/Makefile would like to include this "shared.mak" in the future the semantics of such a Makefile will change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.
This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
--- Documentation/Makefile | 6 +++--- Makefile | 13 +++---------- contrib/scalar/Makefile | 3 +++ contrib/scalar/t/Makefile | 3 +++ shared.mak | 9 +++++++++ t/Makefile | 3 +++ t/interop/Makefile | 3 +++ t/perf/Makefile | 3 +++ templates/Makefile | 3 +++ 9 files changed, 33 insertions(+), 13 deletions(-) create mode 100644 shared.mak
Documentation/Makefile +3 -3
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Guard against environment variables MAN1_TXT = MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n:: $(MAKE) -C po $@ endif -# Delete the target file on error -.DELETE_ON_ERROR: - .PHONY: FORCE
Makefile +3 -10
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # The default target of this Makefile is... all::
@@ -2194,16 +2197,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell strip: $(PROGRAMS) git$X $(STRIP) $(STRIP_OPTS) $^ -### Flags affecting all rules - -# A GNU make extension since gmake 3.72 (released in late 1994) to -# remove the target of rules if commands in those rules fail. The -# default is to only do that if make itself receives a signal. Affects -# all targets, see: -# -# info make --index-search=.DELETE_ON_ERROR -.DELETE_ON_ERROR: - ### Target-specific flags and dependencies # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir QUIET_SUBDIR1 =
contrib/scalar/t/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../../shared.mak + # Run scalar tests # # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
@@ -0,0 +1,9 @@ +### Flags affecting all rules + +# A GNU make extension since gmake 3.72 (released in late 1994) to +# remove the target of rules if commands in those rules fail. The +# default is to only do that if make itself receives a signal. Affects +# all targets, see: +# +# info make --index-search=.DELETE_ON_ERROR +.DELETE_ON_ERROR:
t/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Run tests # # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS
t/perf/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS
templates/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # make and install sample templates ifndef V
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile line 5
Quoted 10 lines · Ævar Arnfjörð Bjarmason
Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # The default target of this Makefile is... all::
Continuing with the theme of [1/9], this change to Makefile gets my firm NAK. The first two lines MUST stay to be the first lines.
Otherwise, a mistaken patch that adds rules to shared.make will make the first of these rules, not "all", as the default target.
Just move it below to the third line or so and you'd be OK.
Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Quoted 16 lines · Junio C Hamano
Makefile line 5
Quoted 10 lines · Ævar Arnfjörð Bjarmason
Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # The default target of this Makefile is... all::Continuing with the theme of [1/9], this change to
Makefilegets my firm NAK. The first two lines MUST stay to be the first lines.
Quoted 4 lines
Otherwise, a mistaken patch that adds rules to shared.make will make the first of these rules, not "all", as the default target.
Just move it below to the third line or so and you'd be OK.
Understood, I can/will do that.
But wouldn't working this into this series be an even better option?
The ".DEFAULT_GOAL" variable is new in, 3.81 (01 Apr 2006), which I think is old enough to hard depend on. We use $(or ...) added in the same release in config.mak.dev.
We definitely have a hard dependency on "Version 3.80 (03 Oct 2002)" $(eval ...) etc.
I checked and RHEL5 (which is otherwise the oldest OS we've supported) has it, but RHEL4 doesn't (it has 3.80). So I think it's safe to have a hard dependency on it and other things in 3.81 at this point.
Makefile +0 -3
@@ -1,9 +1,6 @@ # Import tree-wide shared Makefile behavior and libraries include shared.mak -# The default target of this Makefile is... -all:: - # Define V=1 to have a more verbose compile. # # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
shared.mak +5 -0
@@ -1,3 +1,8 @@ +### By setting the default goal to "all" we override any implicit +### setting once "make" sees our first target, which it'll ignore if +### .DEFAULT_GOAL was explicitly set. +.DEFAULT_GOAL = all + ### Remove GNU make implicit rules ## This speeds things up since we don't need to look for and stat() a
Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Quoted 1 line · Ævar Arnfjörð Bjarmason
But wouldn't working this into this series be an even better option?
No.
What our current Makefile does (i.e. to have the default goal at the beginning, even though the recipe for it may live elsewhere) is a very well understood and established pattern, and there is no reason to start doing the same thing in a different way, even if it is safe to assume that the "different way" may be available everywhere. We are not gaining anything from doing it differently.
Especially it is not something I want to see as "while at it" change, which adds more to the review cycle without improving anything.
Quoted 3 lines
The ".DEFAULT_GOAL" variable is new in, 3.81 (01 Apr 2006), which I think is old enough to hard depend on. We use $(or ...) added in the same release in config.mak.dev.
Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
In reply to Ævar Arnfjörð BjarmasonHi Ævar
Quoted 10 lines · Ævar Arnfjörð Bjarmason
We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own
7b76d6bf221(Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) anddb10fc6c09f(doc: simplifyMakefileusing .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
Quoted 5 lines
This does have the potential downside that if e.g. templates/Makefile would like to include this "shared.mak" in the future the semantics of such a
Makefilewill change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.
I was confused about the mention of templates/Makefile in this paragraph, it seems to be saying that the behavior would change in the future if we included shared.mak in templates/Makefile but this patch does exactly that.
Also does this patch mean we're now using .DELETE_ON_ERROR in places where we were not previously using it?
Best Wishes
Phillip
Quoted 146 lines
This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- Documentation/Makefile | 6 +++--- Makefile | 13 +++---------- contrib/scalar/Makefile | 3 +++ contrib/scalar/t/Makefile | 3 +++ shared.mak | 9 +++++++++ t/Makefile | 3 +++ t/interop/Makefile | 3 +++ t/perf/Makefile | 3 +++ templates/Makefile | 3 +++ 9 files changed, 33 insertions(+), 13 deletions(-)create mode 100644 shared.mak
Documentation/Makefile +3
-3@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Guard against environment variables MAN1_TXT = MAN5_TXT =@@ -524,7 +527,4 @@ doc-l10n install-l10n:: $(MAKE) -C po $@ endif -# Delete the target file on error -.DELETE_ON_ERROR: - .PHONY: FORCEMakefile +3
-10@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # The default target of this Makefile is... all::@@ -2194,16 +2197,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell strip: $(PROGRAMS) git$X $(STRIP) $(STRIP_OPTS) $^ -### Flags affecting all rules - -# A GNU make extension since gmake 3.72 (released in late 1994) to -# remove the target of rules if commands in those rules fail. The -# default is to only do that if make itself receives a signal. Affects -# all targets, see: -# -# info make --index-search=.DELETE_ON_ERROR -.DELETE_ON_ERROR: - ### Target-specific flags and dependencies # The generic compilation pattern rule and automaticallycontrib/scalar/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir QUIET_SUBDIR1 =contrib/scalar/t/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../../shared.mak + # Run scalar tests # # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelinshared.mak new file +9
-0@@ -0,0 +1,9 @@ +### Flags affecting all rules + +# A GNU make extension since gmake 3.72 (released in late 1994) to +# remove the target of rules if commands in those rules fail. The +# default is to only do that if make itself receives a signal. Affects +# all targets, see: +# +# info make --index-search=.DELETE_ON_ERROR +.DELETE_ON_ERROR:t/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Run tests # # Copyright (c) 2005 Junio C Hamanot/interop/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONSt/perf/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONStemplates/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # make and install sample templates ifndef V
Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Quoted 18 lines · Phillip Wood
Hi Ævar
Quoted 6 lines · Ævar Arnfjörð Bjarmason
We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own7b76d6bf221(Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) anddb10fc6c09f(doc: simplifyMakefileusingQuoted 8 lines
.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
This does have the potential downside that if
e.g. templates/Makefile
would like to include this "shared.mak" in the future the semantics of such aMakefilewill change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.
Quoted 4 lines
I was confused about the mention of templates/Makefile in this paragraph, it seems to be saying that the behavior would change in the future if we included shared.mak in templates/Makefile but this patch does exactly that.
Yes, oops! It's a zombie comment that I forgot to adjust from an earlier version of this where that wasn't the case. Will adjust & re-roll.
Quoted 2 lines
Also does this patch mean we're now using .DELETE_ON_ERROR in places where we were not previously using it?
Yes, we'll now use it in those other Makefiles as well. The commits referenced in the second paragrap of the commit message argue for this being safe, and I've reviewed the logic myself & don't expect any problems with it.
As the GNU make manual itself notes the cases where that would cause problems are really obscure, but it's not the default out of an abundance of backwards compatibility caution in GNU make.
Quoted 137 lines
Quoted 137 lines
This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- Documentation/Makefile | 6 +++--- Makefile | 13 +++---------- contrib/scalar/Makefile | 3 +++ contrib/scalar/t/Makefile | 3 +++ shared.mak | 9 +++++++++ t/Makefile | 3 +++ t/interop/Makefile | 3 +++ t/perf/Makefile | 3 +++ templates/Makefile | 3 +++ 9 files changed, 33 insertions(+), 13 deletions(-)create mode 100644 shared.mak
Documentation/Makefile +3
-2@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Guard against environment variables MAN1_TXT = MAN5_TXT =@@ -524,7 +527,4 @@ doc-l10n install-l10n:: $(MAKE) -C po $@ endif -# Delete the target file on error -.DELETE_ON_ERROR: - .PHONY: FORCEMakefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # The default target of this Makefile is... all:: @@ -2194,16 +2197,6 @@ shell_compatibility_test:
please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $^
-### Flags affecting all rulesMakefile +0
-9- -# A GNU make extension since gmake 3.72 (released in late 1994) to -# remove the target of rules if commands in those rules fail. The -# default is to only do that if make itself receives a signal. Affects -# all targets, see: -# -# info make --index-search=.DELETE_ON_ERROR -.DELETE_ON_ERROR: - ### Target-specific flags and dependencies # The generic compilation pattern rule and automaticallycontrib/scalar/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir QUIET_SUBDIR1 = diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefileindex
6170672bb37..01e82e56d15100644contrib/scalar/t/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../../shared.mak + # Run scalar tests # # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelinshared.mak new file +9
-0@@ -0,0 +1,9 @@ +### Flags affecting all rules + +# A GNU make extension since gmake 3.72 (released in late 1994) to +# remove the target of rules if commands in those rules fail. The +# default is to only do that if make itself receives a signal. Affects +# all targets, see: +# +# info make --index-search=.DELETE_ON_ERROR +.DELETE_ON_ERROR:t/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Run tests # # Copyright (c) 2005 Junio C Hamanot/interop/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS diff --git a/t/perf/Makefile b/t/perf/Makefileindex
2465770a782..e4808aebed0100644t/perf/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS diff --git a/templates/Makefile b/templates/Makefileindex
d22a71a3999..636cee52f51100644templates/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # make and install sample templates ifndef V
Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Quoted 29 lines · Ævar Arnfjörð Bjarmason
Quoted 18 lines · Phillip Wood
Hi Ævar
Quoted 6 lines · Ævar Arnfjörð Bjarmason
We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own7b76d6bf221(Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) anddb10fc6c09f(doc: simplifyMakefileusingQuoted 8 lines
.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
This does have the potential downside that if
e.g. templates/Makefile
would like to include this "shared.mak" in the future the semantics of such aMakefilewill change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.Quoted 4 lines
I was confused about the mention of templates/Makefile in this paragraph, it seems to be saying that the behavior would change in the future if we included shared.mak in templates/Makefile but this patch does exactly that.
Yes, oops! It's a zombie comment that I forgot to adjust from an earlier version of this where that wasn't the case. Will adjust & re-roll.
Quoted 7 lines
Quoted 2 lines
Also does this patch mean we're now using .DELETE_ON_ERROR in places where we were not previously using it?
Yes, we'll now use it in those other Makefiles as well. The commits referenced in the second paragrap of the commit message argue for this being safe, and I've reviewed the logic myself & don't expect any problems with it.
Thanks for elaborating, maybe it is worth spelling explicitly in the commit message that this is turning on .DELETE_ON_ERROR in places we didn't previously use it. I had a look at the commit message you referenced and it seems to make a good case for using .DELETE_ON_ERROR. Having a shared makefile for common code makes sense and the speed ups from some of the other commits are nice.
Best Wishes
Phillip
Quoted 142 lines
As the GNU make manual itself notes the cases where that would cause problems are really obscure, but it's not the default out of an abundance of backwards compatibility caution in GNU make.
Quoted 137 lines
Quoted 137 lines
This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- Documentation/Makefile | 6 +++--- Makefile | 13 +++---------- contrib/scalar/Makefile | 3 +++ contrib/scalar/t/Makefile | 3 +++ shared.mak | 9 +++++++++ t/Makefile | 3 +++ t/interop/Makefile | 3 +++ t/perf/Makefile | 3 +++ templates/Makefile | 3 +++ 9 files changed, 33 insertions(+), 13 deletions(-)create mode 100644 shared.mak
Documentation/Makefile +3
-2@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Guard against environment variables MAN1_TXT = MAN5_TXT =@@ -524,7 +527,4 @@ doc-l10n install-l10n:: $(MAKE) -C po $@ endif -# Delete the target file on error -.DELETE_ON_ERROR: - .PHONY: FORCEMakefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # The default target of this Makefile is... all:: @@ -2194,16 +2197,6 @@ shell_compatibility_test:
please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $^
-### Flags affecting all rulesMakefile +0
-9- -# A GNU make extension since gmake 3.72 (released in late 1994) to -# remove the target of rules if commands in those rules fail. The -# default is to only do that if make itself receives a signal. Affects -# all targets, see: -# -# info make --index-search=.DELETE_ON_ERROR -.DELETE_ON_ERROR: - ### Target-specific flags and dependencies # The generic compilation pattern rule and automaticallycontrib/scalar/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir QUIET_SUBDIR1 = diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefileindex
6170672bb37..01e82e56d15100644contrib/scalar/t/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../../shared.mak + # Run scalar tests # # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelinshared.mak new file +9
-0@@ -0,0 +1,9 @@ +### Flags affecting all rules + +# A GNU make extension since gmake 3.72 (released in late 1994) to +# remove the target of rules if commands in those rules fail. The +# default is to only do that if make itself receives a signal. Affects +# all targets, see: +# +# info make --index-search=.DELETE_ON_ERROR +.DELETE_ON_ERROR:t/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Run tests # # Copyright (c) 2005 Junio C Hamanot/interop/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS diff --git a/t/perf/Makefile b/t/perf/Makefileindex
2465770a782..e4808aebed0100644t/perf/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS diff --git a/templates/Makefile b/templates/Makefileindex
d22a71a3999..636cee52f51100644templates/Makefile +3
-0@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # make and install sample templates ifndef V
Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Quoted 36 lines · Phillip Wood
Quoted 28 lines · Ævar Arnfjörð Bjarmason
Quoted 18 lines · Phillip Wood
Hi Ævar
Quoted 6 lines · Ævar Arnfjörð Bjarmason
We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own7b76d6bf221(Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) anddb10fc6c09f(doc: simplifyMakefileusingQuoted 8 lines
.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
This does have the potential downside that if
e.g. templates/Makefile
would like to include this "shared.mak" in the future the semantics of such aMakefilewill change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.Quoted 4 lines
I was confused about the mention of templates/Makefile in this paragraph, it seems to be saying that the behavior would change in the future if we included shared.mak in templates/Makefile but this patch does exactly that.
Yes, oops! It's a zombie comment that I forgot to adjust from an earlier
version of this where that wasn't the case. Will adjust & re-roll.Quoted 6 lines
Quoted 2 lines
Also does this patch mean we're now using .DELETE_ON_ERROR in places where we were not previously using it?
Yes, we'll now use it in those other Makefiles as well. The commits referenced in the second paragrap of the commit message argue for this being safe, and I've reviewed the logic myself & don't expect any problems with it.
Quoted 6 lines
Thanks for elaborating, maybe it is worth spelling explicitly in the commit message that this is turning on .DELETE_ON_ERROR in places we didn't previously use it. I had a look at the commit message you referenced and it seems to make a good case for using .DELETE_ON_ERROR. Having a shared makefile for common code makes sense and the speed ups from some of the other commits are nice.
Thanks, yes, willdo :) I have a re-roll of this queued up for submission, after sitting on it for a bit longer to shake out any potential last-minute issues...
PATCH v3 3/9 Makefile: disable GNU make built-in wildcard rules
In reply to Ævar Arnfjörð BjarmasonOverride built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.
The speed-up may wary. I've seen 1-10% depending on the speed of the local disk, caches, -jN etc. Running:
strace -f -c -S calls make -j1 NO_TCLTK=Y
Shows that we reduce the number of syscalls we make, mostly in "stat" calls.
We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early. Doing so might make us a bit faster still. But doing so is a much bigger hammer, since it will disable all built-in rules, some (all?) of which can be seen with:
make -f/dev/null -p | grep -v -e ^# -e ^$
We may have something that relies on them, so let's go for the more isolated optimization here that gives us most or all of the wins.
1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html
--- shared.mak | 11 +++++++++++ 1 file changed, 11 insertions(+)
shared.mak +11 -0
@@ -1,3 +1,14 @@ +### Remove GNU make implicit rules + +## This speeds things up since we don't need to look for and stat() a +## "foo.c,v" every time a rule referring to "foo.c" is in play. See +## "make -p -f/dev/null | grep ^%::'". +%:: %,v +%:: RCS/%,v +%:: RCS/% +%:: s.% +%:: SCCS/s.% + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
Re: [PATCH v3 3/9] Makefile: disable GNU make built-in wildcard rules
Quoted 5 lines · Ævar Arnfjörð Bjarmason
Override built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to
stat()so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.
;-)
Quoted 1 line
The speed-up may wary. I've seen 1-10% depending on the speed of the
vary, you mean?
Quoted 6 lines
local disk, caches, -jN etc. Running:
strace -f -c -S calls make -j1 NO_TCLTK=Y
Shows that we reduce the number of syscalls we make, mostly in "stat" calls.
Naturally understandable.
I do remember getting surprised after seeing output from "make -d" and how many paths were considered for possible places that a source file can come from.
Quoted 14 lines
shared.mak +11
-0+### Remove GNU make implicit rules + +## This speeds things up since we don't need to look for and stat() a +## "foo.c,v" every time a rule referring to "foo.c" is in play. See +## "make -p -f/dev/null | grep ^%::'". +%:: %,v +%:: RCS/%,v +%:: RCS/% +%:: s.% +%:: SCCS/s.% + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
PATCH v3 4/9 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
In reply to Ævar Arnfjörð BjarmasonCombine the definitions of $(FIND_SOURCE_FILES) and $(LIB_H) to speed up the Makefile, as these are the two main expensive $(shell) commands that we execute unconditionally.
When see what was in $(FOUND_SOURCE_FILES) that wasn't in $(LIB_H) via the ad-hoc test of:
$(error $(filter-out $(LIB_H),$(filter %.h,$(ALL_SOURCE_FILES))))
$(error $(filter-out $(ALL_SOURCE_FILES),$(filter %.h,$(LIB_H))))We'll get, respectively:
Makefile:850: *** t/helper/test-tool.h. Stop.
Makefile:850: *** . Stop.I.e. we only had a discrepancy when it came to t/helper/test-tool.h. In terms of correctness this was broken before, but now works:
$ make t/helper/test-tool.hco
HDR t/helper/test-tool.hThis speeds things up a lot:
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10
Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1
Time (mean ± σ): 159.9 ms ± 6.8 ms [User: 137.2 ms, System: 28.0 ms]
Range (min … max): 154.6 ms … 175.9 ms 10 runs Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0
Time (mean ± σ): 100.0 ms ± 1.3 ms [User: 84.2 ms, System: 20.2 ms]
Range (min … max): 98.8 ms … 102.8 ms 10 runs
Summary
'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran
1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1'--- Makefile | 54 ++++++++++++++++++++++++++---------------------------- 1 file changed, 26 insertions(+), 28 deletions(-)
Makefile +26 -28
@@ -833,12 +833,33 @@ GENERATED_H += hook-list.h .PHONY: generated-hdrs generated-hdrs: $(GENERATED_H) -LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \ +## Exhaustive lists of our source files, either dynamically generated, +## or hardcoded. +SOURCES_CMD = ( \ + git ls-files \ + '*.[hcS]' \ + '*.sh' \ + ':!*[tp][0-9][0-9][0-9][0-9]*' \ + ':!contrib' \ + 2>/dev/null || \ $(FIND) . \ - -name .git -prune -o \ - -name t -prune -o \ - -name Documentation -prune -o \ - -name '*.h' -print))) + \( -name .git -type d -prune \) \ + -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ + -o \( -name contrib -type d -prune \) \ + -o \( -name build -type d -prune \) \ + -o \( -name 'trash*' -type d -prune \) \ + -o \( -name '*.[hcS]' -type f -print \) \ + -o \( -name '*.sh' -type f -print \) \ + | sed -e 's|^\./||' \ + ) +FOUND_SOURCE_FILES := $(shell $(SOURCES_CMD)) + +FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) +FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES)) + +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) + +LIB_H = $(FOUND_H_SOURCES) LIB_OBJS += abspath.o LIB_OBJS += add-interactive.o
@@ -2789,26 +2810,6 @@ perl/build/man/man3/Git.3pm: perl/Git.pm $(QUIET_GEN)mkdir -p $(dir $@) && \ pod2man $< $@ -FIND_SOURCE_FILES = ( \ - git ls-files \ - '*.[hcS]' \ - '*.sh' \ - ':!*[tp][0-9][0-9][0-9][0-9]*' \ - ':!contrib' \ - 2>/dev/null || \ - $(FIND) . \ - \( -name .git -type d -prune \) \ - -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ - -o \( -name contrib -type d -prune \) \ - -o \( -name build -type d -prune \) \ - -o \( -name 'trash*' -type d -prune \) \ - -o \( -name '*.[hcS]' -type f -print \) \ - -o \( -name '*.sh' -type f -print \) \ - | sed -e 's|^\./||' \ - ) - -FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES)) - $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \ echo $(FOUND_SOURCE_FILES) | xargs etags -a -o $@+ && \
@@ -3018,9 +3019,6 @@ check: $(GENERATED_H) exit 1; \ fi -FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) -COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) - %.cocci.patch: %.cocci $(COCCI_SOURCES) $(QUIET_SPATCH) \ if test $(SPATCH_BATCH_SIZE) = 0; then \
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
PATCH v3 5/9 Makefile: move ".SUFFIXES" rule to shared.mak
In reply to Ævar Arnfjörð BjarmasonThis was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.
This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms]
Range (min … max): 112.8 ms … 148.4 ms 26 runs Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs
Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'
The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.
So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.
1. https://lore.kernel.org/git/[email protected]/
--- Makefile | 2 -- shared.mak | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S)) ASM_OBJ := $(ASM_SRC:S=o) C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS)) -.SUFFIXES: - $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir) $(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +5 -0
@@ -9,6 +9,11 @@ %:: s.% %:: SCCS/s.% +## Likewise delete default $(SUFFIXES). See: +## +## info make --index-search=.SUFFIXES +.SUFFIXES: + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
Re: [PATCH v3 5/9] Makefile: move ".SUFFIXES" rule to shared.mak
Quoted 7 lines · Ævar Arnfjörð Bjarmason
This was added in
30248886ce8(Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.This doesn't benefit the main
Makefileat all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:
Nice.
PATCH v3 6/9 Makefile: move $(comma), $(empty) and $(space) to shared.mak
In reply to Ævar Arnfjörð BjarmasonMove these variables over to the shared.max, we'll make use of them in a subsequent commit. There was no reason for these to be "simply expanded variables", so let's use the normal lazy "=" assignment here.
See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.
--- Makefile | 4 ---- shared.mak | 8 ++++++++ 2 files changed, 8 insertions(+), 4 deletions(-)
Makefile +0 -4
@@ -1289,10 +1289,6 @@ endif ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS) ALL_LDFLAGS = $(LDFLAGS) -comma := , -empty := -space := $(empty) $(empty) - ifdef SANITIZE SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag)) BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)
shared.mak +8 -0
@@ -23,3 +23,11 @@ # # info make --index-search=.DELETE_ON_ERROR .DELETE_ON_ERROR: + +### Global variables + +## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among other Makefile syntax. +comma = , +empty = +space = $(empty) $(empty)
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
Re: [PATCH v3 6/9] Makefile: move $(comma), $(empty) and $(space) to shared.mak
Quoted 1 line · Ævar Arnfjörð Bjarmason
Move these variables over to the shared.max, we'll make use of them in
max?
Quoted 2 lines
a subsequent commit. There was no reason for these to be "simply expanded variables", so let's use the normal lazy "=" assignment here.
Seems to do unrelated things "while at it" which is a common pattern that slows the reviews down.
Quoted 6 lines
shared.mak +0
-0@@ -23,3 +23,11 @@ #
shared.mak line 33
Quoted 10 lines
shared.mak +8
-0# info make --index-search=.DELETE_ON_ERROR .DELETE_ON_ERROR: + +### Global variables + +## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among other Makefile syntax. +comma = , +empty = +space = $(empty) $(empty)
In any case, wanting to have a $(space) is not specific to the top-level Makefile, and it is a good idea to move them to the shared file.
PATCH v3 7/9 Makefile: add "$(QUIET)" boilerplate to shared.mak
In reply to Ævar Arnfjörð BjarmasonThe $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.
Since we're not using the environment to pass these around we don't need to export the "QUIET_GEN" and "QUIET_BUILT_IN" variables anymore. The "QUIET_GEN" variable is used in "git-gui/Makefile" and "gitweb/Makefile", but they've got their own definition for those. The "QUIET_BUILT_IN" variable is only used in the top-level "Makefile". We still need to export the "V" variable.
--- Documentation/Makefile | 32 ------------------------- Makefile | 33 ------------------------- config.mak.uname | 1 - contrib/scalar/Makefile | 14 ----------- shared.mak | 53 +++++++++++++++++++++++++++++++++++++++++ templates/Makefile | 5 ---- 6 files changed, 53 insertions(+), 85 deletions(-)
Documentation/Makefile +0 -32
@@ -218,38 +218,6 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR)) ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)' endif -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring $(MAKEFLAGS),w),w) -PRINT_DIR = --no-print-directory -else # "make -w" -NO_SUBDIR = : -endif - -ifneq ($(findstring $(MAKEFLAGS),s),s) -ifndef V - QUIET = @ - QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; - QUIET_XMLTO = @echo ' ' XMLTO $@; - QUIET_DB2TEXI = @echo ' ' DB2TEXI $@; - QUIET_MAKEINFO = @echo ' ' MAKEINFO $@; - QUIET_DBLATEX = @echo ' ' DBLATEX $@; - QUIET_XSLTPROC = @echo ' ' XSLTPROC $@; - QUIET_GEN = @echo ' ' GEN $@; - QUIET_STDERR = 2> /dev/null - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir - - QUIET_LINT_GITLINK = @echo ' ' LINT GITLINK $<; - QUIET_LINT_MANSEC = @echo ' ' LINT MAN SEC $<; - QUIET_LINT_MANEND = @echo ' ' LINT MAN END $<; - - export V -endif -endif - all: html man html: $(DOC_HTML)
Makefile +0 -33
@@ -2001,39 +2001,6 @@ ifndef PAGER_ENV PAGER_ENV = LESS=FRX LV=-c endif -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring w,$(MAKEFLAGS)),w) -PRINT_DIR = --no-print-directory -else # "make -w" -NO_SUBDIR = : -endif - -ifneq ($(findstring s,$(MAKEFLAGS)),s) -ifndef V - QUIET_CC = @echo ' ' CC $@; - QUIET_AR = @echo ' ' AR $@; - QUIET_LINK = @echo ' ' LINK $@; - QUIET_BUILT_IN = @echo ' ' BUILTIN $@; - QUIET_GEN = @echo ' ' GEN $@; - QUIET_LNCP = @echo ' ' LN/CP $@; - QUIET_XGETTEXT = @echo ' ' XGETTEXT $@; - QUIET_MSGFMT = @echo ' ' MSGFMT $@; - QUIET_GCOV = @echo ' ' GCOV $@; - QUIET_SP = @echo ' ' SP $<; - QUIET_HDR = @echo ' ' HDR $(<:hcc=h); - QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir - export V - export QUIET_GEN - export QUIET_BUILT_IN -endif -endif - ifdef NO_INSTALL_HARDLINKS export NO_INSTALL_HARDLINKS endif
config.mak.uname +0 -1
@@ -727,7 +727,6 @@ vcxproj: git diff-index --cached --quiet HEAD -- # Make .vcxproj files and add them - unset QUIET_GEN QUIET_BUILT_IN; \ perl contrib/buildsystems/generate -g Vcxproj git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
contrib/scalar/Makefile +0 -14
@@ -1,20 +1,6 @@ # Import tree-wide shared Makefile behavior and libraries include ../../shared.mak -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring s,$(MAKEFLAGS)),s) -ifndef V - QUIET_GEN = @echo ' ' GEN $@; - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir -else - export V -endif -endif - include ../../config.mak.uname -include ../../config.mak.autogen -include ../../config.mak
shared.mak +53 -0
56 lines not on this page · raw message
templates/Makefile +0 -5
@@ -2,11 +2,6 @@ include ../shared.mak # make and install sample templates - -ifndef V - QUIET = @ -endif - INSTALL ?= install TAR ?= tar RM ?= rm -f
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
Re: [PATCH v3 7/9] Makefile: add "$(QUIET)" boilerplate to shared.mak
Quoted 2 lines · Ævar Arnfjörð Bjarmason
The $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.
Thank you thank you thank you.
It has been disturbing me a lot to see these copied-and-pasted duplication across Makefiles.
PATCH v3 8/9 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
In reply to Ævar Arnfjörð BjarmasonChange the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding, to guarantee that it's consistent with the "TRACK_template" template.
$ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
-> GIT-CFLAGS PARAMETERS (changed)
-> CC grep.o--- shared.mak | 54 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 30 insertions(+), 24 deletions(-)
shared.mak +30 -24
@@ -32,6 +32,12 @@ comma = , empty = space = $(empty) $(empty) +## wspfx: the whitespace prefix padding for $(QUIET...) and similarly +## aligned output. +wspfx = $(space)$(space)$(space) +wspfx_SQ = '$(subst ','\'',$(wspfx))' +# ' closing quote to appease Emacs make-mode.elxo + ### Quieting ## common QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
@@ -47,39 +53,39 @@ ifneq ($(findstring s,$(MAKEFLAGS)),s) ifndef V ## common QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ + QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo $(wspfx_SQ) SUBDIR $$subdir; \ $(MAKE) $(PRINT_DIR) -C $$subdir QUIET = @ - QUIET_GEN = @echo ' ' GEN $@; + QUIET_GEN = @echo $(wspfx_SQ) GEN $@; ## Used in "Makefile" - QUIET_CC = @echo ' ' CC $@; - QUIET_AR = @echo ' ' AR $@; - QUIET_LINK = @echo ' ' LINK $@; - QUIET_BUILT_IN = @echo ' ' BUILTIN $@; - QUIET_LNCP = @echo ' ' LN/CP $@; - QUIET_XGETTEXT = @echo ' ' XGETTEXT $@; - QUIET_MSGFMT = @echo ' ' MSGFMT $@; - QUIET_GCOV = @echo ' ' GCOV $@; - QUIET_SP = @echo ' ' SP $<; - QUIET_HDR = @echo ' ' HDR $(<:hcc=h); - QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_CC = @echo $(wspfx_SQ) CC $@; + QUIET_AR = @echo $(wspfx_SQ) AR $@; + QUIET_LINK = @echo $(wspfx_SQ) LINK $@; + QUIET_BUILT_IN = @echo $(wspfx_SQ) BUILTIN $@; + QUIET_LNCP = @echo $(wspfx_SQ) LN/CP $@; + QUIET_XGETTEXT = @echo $(wspfx_SQ) XGETTEXT $@; + QUIET_MSGFMT = @echo $(wspfx_SQ) MSGFMT $@; + QUIET_GCOV = @echo $(wspfx_SQ) GCOV $@; + QUIET_SP = @echo $(wspfx_SQ) SP $<; + QUIET_HDR = @echo $(wspfx_SQ) HDR $(<:hcc=h); + QUIET_RC = @echo $(wspfx_SQ) RC $@; + QUIET_SPATCH = @echo $(wspfx_SQ) SPATCH $<; ## Used in "Documentation/Makefile" - QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; - QUIET_XMLTO = @echo ' ' XMLTO $@; - QUIET_DB2TEXI = @echo ' ' DB2TEXI $@; - QUIET_MAKEINFO = @echo ' ' MAKEINFO $@; - QUIET_DBLATEX = @echo ' ' DBLATEX $@; - QUIET_XSLTPROC = @echo ' ' XSLTPROC $@; - QUIET_GEN = @echo ' ' GEN $@; + QUIET_ASCIIDOC = @echo $(wspfx_SQ) ASCIIDOC $@; + QUIET_XMLTO = @echo $(wspfx_SQ) XMLTO $@; + QUIET_DB2TEXI = @echo $(wspfx_SQ) DB2TEXI $@; + QUIET_MAKEINFO = @echo $(wspfx_SQ) MAKEINFO $@; + QUIET_DBLATEX = @echo $(wspfx_SQ) DBLATEX $@; + QUIET_XSLTPROC = @echo $(wspfx_SQ) XSLTPROC $@; + QUIET_GEN = @echo $(wspfx_SQ) GEN $@; QUIET_STDERR = 2> /dev/null - QUIET_LINT_GITLINK = @echo ' ' LINT GITLINK $<; - QUIET_LINT_MANSEC = @echo ' ' LINT MAN SEC $<; - QUIET_LINT_MANEND = @echo ' ' LINT MAN END $<; + QUIET_LINT_GITLINK = @echo $(wspfx_SQ) LINT GITLINK $<; + QUIET_LINT_MANSEC = @echo $(wspfx_SQ) LINT MAN SEC $<; + QUIET_LINT_MANEND = @echo $(wspfx_SQ) LINT MAN END $<; export V endif
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
Re: [PATCH v3 8/9] Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Quoted 7 lines · Ævar Arnfjörð Bjarmason
Change the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding, to guarantee that it's consistent with the "
TRACK_template" template.$ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->' -> GIT-CFLAGS PARAMETERS (changed) -> CC grep.o
What is a TRACK_template? Is this some "rebase -i" gone bad?
PATCH v3 9/9 Makefiles: add and use wildcard "mkdir -p" template
In reply to Ævar Arnfjörð BjarmasonAdd a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).
As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.
I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.
But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
Time (mean ± σ): 2.914 s ± 0.062 s [User: 2.449 s, System: 0.489 s]
Range (min … max): 2.834 s … 3.020 s 10 runs Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s]
Range (min … max): 2.229 s … 2.397 s 10 runs
Summary
'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'
So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.
This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://gitlab.com/avar/git-hyperfine/
--- Documentation/Makefile | 25 +++---------------------- Makefile | 12 +++++++----- shared.mak | 17 +++++++++++++++++ 3 files changed, 27 insertions(+), 27 deletions(-)
Documentation/Makefile +3 -22
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo print-man1: @for i in $(MAN1_TXT); do echo $$i; done -## Lint: Common -.build: - $(QUIET)mkdir $@ -.build/lint-docs: | .build - $(QUIET)mkdir $@ - ## Lint: gitlink -.build/lint-docs/gitlink: | .build/lint-docs - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT)) -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config $(LINT_DOCS_GITLINK): lint-gitlink.perl $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \ $< \ $(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt lint-docs-gitlink: $(LINT_DOCS_GITLINK) ## Lint: man-end-blurb -.build/lint-docs/man-end-blurb: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@ .PHONY: lint-docs-man-end-blurb -lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB) ## Lint: man-section-order -.build/lint-docs/man-section-order: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@ .PHONY: lint-docs-man-section-order lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
Makefile +7 -5
@@ -2745,7 +2745,8 @@ all:: $(MOFILES) endif po/build/locale/%/LC_MESSAGES/git.mo: po/%.po - $(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $< + $(call mkdir_p_parent_template) + $(QUIET_MSGFMT)$(MSGFMT) -o $@ $< LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm) LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2761,15 +2762,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS)) endif perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES - $(QUIET_GEN)mkdir -p $(dir $@) && \ + $(call mkdir_p_parent_template) + $(QUIET_GEN) \ sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \ -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \ -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \ < $< > $@ perl/build/man/man3/Git.3pm: perl/Git.pm - $(QUIET_GEN)mkdir -p $(dir $@) && \ - pod2man $< $@ + $(call mkdir_p_parent_template) + $(QUIET_GEN)pod2man $< $@ $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \
@@ -2903,7 +2905,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $( all:: $(TEST_PROGRAMS) $(test_bindir_programs) bin-wrappers/%: wrap-for-bin.sh - @mkdir -p bin-wrappers + $(call mkdir_p_parent_template) $(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \ -e 's|@@BUILD_DIR@@|$(shell pwd)|' \ -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
shared.mak +17 -0
@@ -59,6 +59,8 @@ ifndef V QUIET = @ QUIET_GEN = @echo $(wspfx_SQ) GEN $@; + QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D); + ## Used in "Makefile" QUIET_CC = @echo $(wspfx_SQ) CC $@; QUIET_AR = @echo $(wspfx_SQ) AR $@;
@@ -90,3 +92,18 @@ ifndef V export V endif endif + +### Templates + +## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@ +## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not +## needed. +## +## Is racy, but in a good way; we might redundantly (and safely) +## "mkdir -p" when running in parallel, but won't need to exhaustively create +## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule. +define mkdir_p_parent_template +$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D))) +endef
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
v4 cover Makefile: optimize noop runs, add shared.mak
In reply to Ævar Arnfjörð BjarmasonAs the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/
This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).
This v4 re-roll (for v3, see
https://lore.kernel.org/git/[email protected]/):
* The "all" boilerplate goes first now, before "include"
* Elaborated on DELETE_ON_ERROR ina commit message.
* No longer change a ":=" to "=" while moving code.
* Rephrased other commit messages (one of which referred to a function from another future series)
* Typo fix in commit message.
Ævar Arnfjörð Bjarmason (9):
scalar Makefile: use "The default target of..." pattern
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" template
Documentation/Makefile | 63 ++------------------ Makefile | 118 ++++++++++++-------------------------- config.mak.uname | 1 - contrib/scalar/Makefile | 20 ++----- contrib/scalar/t/Makefile | 3 + shared.mak | 109 +++++++++++++++++++++++++++++++++++ t/Makefile | 3 + t/interop/Makefile | 3 + t/perf/Makefile | 3 + templates/Makefile | 8 +-- 10 files changed, 171 insertions(+), 160 deletions(-) create mode 100644 shared.mak
Range-diff against v3:
1: 2404c4d8b96 < -: ----------- scalar Makefile: set the default target after the includes -: ----------- > 1: 26c6bb897cf scalar Makefile: use "The default target of..." pattern 2: 96a490bec54 ! 2: 74692458b70 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
@@ Commit message
.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
- This does have the potential downside that if e.g. templates/Makefile
- would like to include this "shared.mak" in the future the semantics of
- such a Makefile will change, but as noted in the above commits (and
- GNU make's own documentation) any such change would be for the better,
- so it's safe to do this.
+ I.e. this changes the behavior of existing rules in the altered
+ Makefiles (except "Makefile" & "Documentation/Makefile"). I'm
+ confident that this is safe having read the relevant rules in those
+ Makfiles, and as the GNU make manual notes that it isn't the default
+ behavior is out of an abundance of backwards compatibility
+ caution. From edition 0.75 of its manual, covering GNU make 4.3:
- This also doesn't introduce a bug by e.g. having this
+ [Enabling '.DELETE_ON_ERROR' is] almost always what you want
+ 'make' to do, but it is not historical practice; so for
+ compatibility, you must explicitly request it.
+
+ This doesn't introduce a bug by e.g. having this
".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
+ It does increase the danger that any Makefile without an explicit "The
+ default target of this Makefile is..." snippet to define the default
+ target as "all" could have its default rule changed if our new
+ shared.mak ever defines a "real" rule. In subsequent commits we'll be
+ careful not to do that, and such breakage would be obvious e.g. in the
+ case of "make -C t".
+
+ We might want to make that less fragile still (e.g. by using
+ ".DEFAULT_GOAL" as noted in the preceding commit), but for now let's
+ simply include "shared.mak" without adding that boilerplate to all the
+ Makefiles that don't have it already. Most of those are already
+ exposed to that potential caveat e.g. due to including "config.mak*".
+
## Documentation/Makefile ##
@@ Documentation/Makefile: doc-l10n install-l10n::
## Makefile ##
@@
-+# Import tree-wide shared Makefile behavior and libraries -+include shared.mak
-+
# The default target of this Makefile is... all::
++# Import tree-wide shared Makefile behavior and libraries
++include shared.mak
++
+ # Define V=1 to have a more verbose compile.
+ #
+ # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
@@ Makefile: shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $^
@@ Makefile: shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
## contrib/scalar/Makefile ##
@@
+ # The default target of this Makefile is...
+ all::
+
+# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak
+
- QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
- QUIET_SUBDIR1 =
-
+ include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak
## contrib/scalar/t/Makefile ##
@@
3: 9392e3c3e97 ! 3: 0fbdeeffc7b Makefile: disable GNU make built-in wildcard rules
@@ Commit message
contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.
- The speed-up may wary. I've seen 1-10% depending on the speed of the
+ The speed-up may vary. I've seen 1-10% depending on the speed of the
local disk, caches, -jN etc. Running:
strace -f -c -S calls make -j1 NO_TCLTK=Y
4: 07cf9daa9d6 = 4: ea6b835308a Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) 5: 16f2e3ff35b = 5: c2339694cf7 Makefile: move ".SUFFIXES" rule to shared.mak 6: 1b6ecb27f02 ! 6: 741fdfd48e2 Makefile: move $(comma), $(empty) and $(space) to shared.mak
@@ Metadata
## Commit message ##
Makefile: move $(comma), $(empty) and $(space) to shared.mak
- Move these variables over to the shared.max, we'll make use of them in
- a subsequent commit. There was no reason for these to be "simply
- expanded variables", so let's use the normal lazy "=" assignment here.
+ Move these variables over to the shared.mak, we'll make use of them in
+ a subsequent commit.
+
+ Note that there's reason for these to be "simply expanded variables",
+ i.e. to use ":=" assignments instead of lazily expanded "="
+ assignments. We could use "=", but let's leave this as-is for now for
+ ease of review.
See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.
@@ shared.mak
+
+## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among other Makefile syntax.
-+comma = ,
-+empty =
-+space = $(empty) $(empty)
++comma := ,
++empty :=
++space := $(empty) $(empty)
7: 471067deefc ! 7: a723cbce270 Makefile: add "$(QUIET)" boilerplate to shared.mak
@@ config.mak.uname: vcxproj:
## contrib/scalar/Makefile ##
-@@
- # Import tree-wide shared Makefile behavior and libraries
- include ../../shared.mak
+@@ contrib/scalar/Makefile: include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak
-QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 =
@@ contrib/scalar/Makefile
-endif
-endif
-
- include ../../config.mak.uname
- -include ../../config.mak.autogen
- -include ../../config.mak
+ TARGETS = scalar$(X) scalar.o
+ GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a +
## shared.mak ##
@@
- comma = ,
- empty =
- space = $(empty) $(empty)
+ comma := ,
+ empty :=
+ space := $(empty) $(empty)
+
+### Quieting
+## common
8: 510306d2219 ! 8: 3733b0c8df1 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
@@ Commit message
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Change the mostly move-only change in the preceding commit to use the
- $(wspfx) variable for defining the QUIET padding, to guarantee that
- it's consistent with the "TRACK_template" template.
+ $(wspfx) variable for defining the QUIET padding. This refactoring
+ will make it easier to emit that exact amount of padding in functions
+ that we might add to shared.mak in the future.
- $ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
- -> GIT-CFLAGS PARAMETERS (changed)
- -> CC grep.o
+ Such a function is not part of this patch series, but a
+ "TRACK_template" that I'd like to add as a follow-up to it makes use
+ of this. Let's make this change now while modifying these QUIET rules
+ is fresh in our minds.
## shared.mak ##
-@@ shared.mak: comma = ,
- empty =
- space = $(empty) $(empty)
+@@ shared.mak: comma := ,
+ empty :=
+ space := $(empty) $(empty)
+## wspfx: the whitespace prefix padding for $(QUIET...) and similarly +## aligned output.
9: 85bb74aa32f = 9: 4cc4aeabb20 Makefiles: add and use wildcard "mkdir -p" template
Signature · 1 line
2.35.1.1228.g56895c6ee86
PATCH v4 1/9 scalar Makefile: use "The default target of..." pattern
Make have the "contrib/scalar/Makefile" be stylistically consistent with the top-level "Makefile" in first declaring "all" to be the default rule, follwed by including other Makefile snippets.
This adjusts code added in 0a43fb22026 (scalar: create a rudimentary executable, 2021-12-03), it's a style-only change, in a subsequent commit the "QUIET" boilerplate at the beginning of this file will be retrieved via an include, and having an "all:" between the two set of "include"'s after that change would look odd.
As noted in [1] using ".DEFAULT_GOAL = all" is another way to do this in more modern GNU make versions, which we already have a hard dependency on, but let's leave any such change for a future improvement and go with using our established pattern consistently for now.
1. https://lore.kernel.org/git/[email protected]/
--- contrib/scalar/Makefile | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
contrib/scalar/Makefile +7 -6
@@ -1,3 +1,10 @@ +# The default target of this Makefile is... +all:: + +include ../../config.mak.uname +-include ../../config.mak.autogen +-include ../../config.mak + QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir QUIET_SUBDIR1 =
@@ -12,12 +19,6 @@ else endif endif -all: - -include ../../config.mak.uname --include ../../config.mak.autogen --include ../../config.mak - TARGETS = scalar$(X) scalar.o GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
Signature · 1 line
2.35.1.1228.g56895c6ee86
Re: [PATCH v4 1/9] scalar Makefile: use "The default target of..." pattern
Quoted 3 lines · Ævar Arnfjörð Bjarmason
Make have the "contrib/scalar/Makefile" be stylistically consistent with the top-level "
Makefile" in first declaring "all" to be the default rule, follwed by including otherMakefilesnippets.
Sorry, but I cannot quite parse it. Perhaps remove "have"?
Quoted 5 lines
This adjusts code added in
0a43fb22026(scalar: create a rudimentary executable, 2021-12-03), it's a style-only change, in a subsequent commit the "QUIET" boilerplate at the beginning of this file will be retrieved via an include, and having an "all:" between the two set of "include"'s after that change would look odd.
In other words, raising these includes to the top is not style-only but is more significant change. I think these two are logically distinct changes, but I am fine to see both in the same commit, as long as they are explained as such.
Quoted 6 lines
As noted in [1] using ".DEFAULT_GOAL = all" is another way to do this in more modern GNU make versions, which we already have a hard dependency on, but let's leave any such change for a future improvement and go with using our established pattern consistently for now.
I would suggest dropping this paragraph. In the discussion, we didn't even establish that such a change would be an "improvement", and in fact, not everything new is an improvement. If we know .DEFAULT_GOAL is not used, readers of Makefile do not have to run around and look for it to figure out what happens when "make" is run without argument.
PATCH v4 2/9 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
In reply to Ævar Arnfjörð BjarmasonWe have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
I.e. this changes the behavior of existing rules in the altered Makefiles (except "Makefile" & "Documentation/Makefile"). I'm confident that this is safe having read the relevant rules in those Makfiles, and as the GNU make manual notes that it isn't the default behavior is out of an abundance of backwards compatibility caution. From edition 0.75 of its manual, covering GNU make 4.3:
[Enabling '.DELETE_ON_ERROR' is] almost always what you want 'make' to do, but it is not historical practice; so for compatibility, you must explicitly request it.
This doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
It does increase the danger that any Makefile without an explicit "The default target of this Makefile is..." snippet to define the default target as "all" could have its default rule changed if our new shared.mak ever defines a "real" rule. In subsequent commits we'll be careful not to do that, and such breakage would be obvious e.g. in the case of "make -C t".
We might want to make that less fragile still (e.g. by using ".DEFAULT_GOAL" as noted in the preceding commit), but for now let's simply include "shared.mak" without adding that boilerplate to all the Makefiles that don't have it already. Most of those are already exposed to that potential caveat e.g. due to including "config.mak*".
--- Documentation/Makefile | 6 +++--- Makefile | 13 +++---------- contrib/scalar/Makefile | 3 +++ contrib/scalar/t/Makefile | 3 +++ shared.mak | 9 +++++++++ t/Makefile | 3 +++ t/interop/Makefile | 3 +++ t/perf/Makefile | 3 +++ templates/Makefile | 3 +++ 9 files changed, 33 insertions(+), 13 deletions(-) create mode 100644 shared.mak
Documentation/Makefile +3 -3
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Guard against environment variables MAN1_TXT = MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n:: $(MAKE) -C po $@ endif -# Delete the target file on error -.DELETE_ON_ERROR: - .PHONY: FORCE
Makefile +3 -10
@@ -1,6 +1,9 @@ # The default target of this Makefile is... all:: +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # Define V=1 to have a more verbose compile. # # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
@@ -2194,16 +2197,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell strip: $(PROGRAMS) git$X $(STRIP) $(STRIP_OPTS) $^ -### Flags affecting all rules - -# A GNU make extension since gmake 3.72 (released in late 1994) to -# remove the target of rules if commands in those rules fail. The -# default is to only do that if make itself receives a signal. Affects -# all targets, see: -# -# info make --index-search=.DELETE_ON_ERROR -.DELETE_ON_ERROR: - ### Target-specific flags and dependencies # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
@@ -1,6 +1,9 @@ # The default target of this Makefile is... all:: +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + include ../../config.mak.uname -include ../../config.mak.autogen -include ../../config.mak
contrib/scalar/t/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../../shared.mak + # Run scalar tests # # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
@@ -0,0 +1,9 @@ +### Flags affecting all rules + +# A GNU make extension since gmake 3.72 (released in late 1994) to +# remove the target of rules if commands in those rules fail. The +# default is to only do that if make itself receives a signal. Affects +# all targets, see: +# +# info make --index-search=.DELETE_ON_ERROR +.DELETE_ON_ERROR:
t/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Run tests # # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS
t/perf/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS
templates/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # make and install sample templates ifndef V
Signature · 1 line
2.35.1.1228.g56895c6ee86
PATCH v4 3/9 Makefile: disable GNU make built-in wildcard rules
In reply to Ævar Arnfjörð BjarmasonOverride built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.
The speed-up may vary. I've seen 1-10% depending on the speed of the local disk, caches, -jN etc. Running:
strace -f -c -S calls make -j1 NO_TCLTK=Y
Shows that we reduce the number of syscalls we make, mostly in "stat" calls.
We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early. Doing so might make us a bit faster still. But doing so is a much bigger hammer, since it will disable all built-in rules, some (all?) of which can be seen with:
make -f/dev/null -p | grep -v -e ^# -e ^$
We may have something that relies on them, so let's go for the more isolated optimization here that gives us most or all of the wins.
1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html
--- shared.mak | 11 +++++++++++ 1 file changed, 11 insertions(+)
shared.mak +11 -0
@@ -1,3 +1,14 @@ +### Remove GNU make implicit rules + +## This speeds things up since we don't need to look for and stat() a +## "foo.c,v" every time a rule referring to "foo.c" is in play. See +## "make -p -f/dev/null | grep ^%::'". +%:: %,v +%:: RCS/%,v +%:: RCS/% +%:: s.% +%:: SCCS/s.% + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1228.g56895c6ee86
PATCH v4 4/9 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
In reply to Ævar Arnfjörð BjarmasonCombine the definitions of $(FIND_SOURCE_FILES) and $(LIB_H) to speed up the Makefile, as these are the two main expensive $(shell) commands that we execute unconditionally.
When see what was in $(FOUND_SOURCE_FILES) that wasn't in $(LIB_H) via the ad-hoc test of:
$(error $(filter-out $(LIB_H),$(filter %.h,$(ALL_SOURCE_FILES))))
$(error $(filter-out $(ALL_SOURCE_FILES),$(filter %.h,$(LIB_H))))We'll get, respectively:
Makefile:850: *** t/helper/test-tool.h. Stop.
Makefile:850: *** . Stop.I.e. we only had a discrepancy when it came to t/helper/test-tool.h. In terms of correctness this was broken before, but now works:
$ make t/helper/test-tool.hco
HDR t/helper/test-tool.hThis speeds things up a lot:
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10
Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1
Time (mean ± σ): 159.9 ms ± 6.8 ms [User: 137.2 ms, System: 28.0 ms]
Range (min … max): 154.6 ms … 175.9 ms 10 runs Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0
Time (mean ± σ): 100.0 ms ± 1.3 ms [User: 84.2 ms, System: 20.2 ms]
Range (min … max): 98.8 ms … 102.8 ms 10 runs
Summary
'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran
1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1'--- Makefile | 54 ++++++++++++++++++++++++++---------------------------- 1 file changed, 26 insertions(+), 28 deletions(-)
Makefile +26 -28
@@ -833,12 +833,33 @@ GENERATED_H += hook-list.h .PHONY: generated-hdrs generated-hdrs: $(GENERATED_H) -LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \ +## Exhaustive lists of our source files, either dynamically generated, +## or hardcoded. +SOURCES_CMD = ( \ + git ls-files \ + '*.[hcS]' \ + '*.sh' \ + ':!*[tp][0-9][0-9][0-9][0-9]*' \ + ':!contrib' \ + 2>/dev/null || \ $(FIND) . \ - -name .git -prune -o \ - -name t -prune -o \ - -name Documentation -prune -o \ - -name '*.h' -print))) + \( -name .git -type d -prune \) \ + -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ + -o \( -name contrib -type d -prune \) \ + -o \( -name build -type d -prune \) \ + -o \( -name 'trash*' -type d -prune \) \ + -o \( -name '*.[hcS]' -type f -print \) \ + -o \( -name '*.sh' -type f -print \) \ + | sed -e 's|^\./||' \ + ) +FOUND_SOURCE_FILES := $(shell $(SOURCES_CMD)) + +FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) +FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES)) + +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) + +LIB_H = $(FOUND_H_SOURCES) LIB_OBJS += abspath.o LIB_OBJS += add-interactive.o
@@ -2789,26 +2810,6 @@ perl/build/man/man3/Git.3pm: perl/Git.pm $(QUIET_GEN)mkdir -p $(dir $@) && \ pod2man $< $@ -FIND_SOURCE_FILES = ( \ - git ls-files \ - '*.[hcS]' \ - '*.sh' \ - ':!*[tp][0-9][0-9][0-9][0-9]*' \ - ':!contrib' \ - 2>/dev/null || \ - $(FIND) . \ - \( -name .git -type d -prune \) \ - -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ - -o \( -name contrib -type d -prune \) \ - -o \( -name build -type d -prune \) \ - -o \( -name 'trash*' -type d -prune \) \ - -o \( -name '*.[hcS]' -type f -print \) \ - -o \( -name '*.sh' -type f -print \) \ - | sed -e 's|^\./||' \ - ) - -FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES)) - $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \ echo $(FOUND_SOURCE_FILES) | xargs etags -a -o $@+ && \
@@ -3018,9 +3019,6 @@ check: $(GENERATED_H) exit 1; \ fi -FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) -COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) - %.cocci.patch: %.cocci $(COCCI_SOURCES) $(QUIET_SPATCH) \ if test $(SPATCH_BATCH_SIZE) = 0; then \
Signature · 1 line
2.35.1.1228.g56895c6ee86
PATCH v4 5/9 Makefile: move ".SUFFIXES" rule to shared.mak
In reply to Ævar Arnfjörð BjarmasonThis was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.
This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms]
Range (min … max): 112.8 ms … 148.4 ms 26 runs Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs
Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'
The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.
So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.
1. https://lore.kernel.org/git/[email protected]/
--- Makefile | 2 -- shared.mak | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S)) ASM_OBJ := $(ASM_SRC:S=o) C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS)) -.SUFFIXES: - $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir) $(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +5 -0
@@ -9,6 +9,11 @@ %:: s.% %:: SCCS/s.% +## Likewise delete default $(SUFFIXES). See: +## +## info make --index-search=.SUFFIXES +.SUFFIXES: + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1228.g56895c6ee86
PATCH v4 6/9 Makefile: move $(comma), $(empty) and $(space) to shared.mak
In reply to Ævar Arnfjörð BjarmasonMove these variables over to the shared.mak, we'll make use of them in a subsequent commit.
Note that there's reason for these to be "simply expanded variables", i.e. to use ":=" assignments instead of lazily expanded "=" assignments. We could use "=", but let's leave this as-is for now for ease of review.
See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.
--- Makefile | 4 ---- shared.mak | 8 ++++++++ 2 files changed, 8 insertions(+), 4 deletions(-)
Makefile +0 -4
@@ -1289,10 +1289,6 @@ endif ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS) ALL_LDFLAGS = $(LDFLAGS) -comma := , -empty := -space := $(empty) $(empty) - ifdef SANITIZE SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag)) BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)
shared.mak +8 -0
@@ -23,3 +23,11 @@ # # info make --index-search=.DELETE_ON_ERROR .DELETE_ON_ERROR: + +### Global variables + +## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among other Makefile syntax. +comma := , +empty := +space := $(empty) $(empty)
Signature · 1 line
2.35.1.1228.g56895c6ee86
PATCH v4 7/9 Makefile: add "$(QUIET)" boilerplate to shared.mak
In reply to Ævar Arnfjörð BjarmasonThe $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.
Since we're not using the environment to pass these around we don't need to export the "QUIET_GEN" and "QUIET_BUILT_IN" variables anymore. The "QUIET_GEN" variable is used in "git-gui/Makefile" and "gitweb/Makefile", but they've got their own definition for those. The "QUIET_BUILT_IN" variable is only used in the top-level "Makefile". We still need to export the "V" variable.
--- Documentation/Makefile | 32 ------------------------- Makefile | 33 ------------------------- config.mak.uname | 1 - contrib/scalar/Makefile | 14 ----------- shared.mak | 53 +++++++++++++++++++++++++++++++++++++++++ templates/Makefile | 5 ---- 6 files changed, 53 insertions(+), 85 deletions(-)
Documentation/Makefile +0 -32
@@ -218,38 +218,6 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR)) ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)' endif -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring $(MAKEFLAGS),w),w) -PRINT_DIR = --no-print-directory -else # "make -w" -NO_SUBDIR = : -endif - -ifneq ($(findstring $(MAKEFLAGS),s),s) -ifndef V - QUIET = @ - QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; - QUIET_XMLTO = @echo ' ' XMLTO $@; - QUIET_DB2TEXI = @echo ' ' DB2TEXI $@; - QUIET_MAKEINFO = @echo ' ' MAKEINFO $@; - QUIET_DBLATEX = @echo ' ' DBLATEX $@; - QUIET_XSLTPROC = @echo ' ' XSLTPROC $@; - QUIET_GEN = @echo ' ' GEN $@; - QUIET_STDERR = 2> /dev/null - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir - - QUIET_LINT_GITLINK = @echo ' ' LINT GITLINK $<; - QUIET_LINT_MANSEC = @echo ' ' LINT MAN SEC $<; - QUIET_LINT_MANEND = @echo ' ' LINT MAN END $<; - - export V -endif -endif - all: html man html: $(DOC_HTML)
Makefile +0 -33
@@ -2001,39 +2001,6 @@ ifndef PAGER_ENV PAGER_ENV = LESS=FRX LV=-c endif -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring w,$(MAKEFLAGS)),w) -PRINT_DIR = --no-print-directory -else # "make -w" -NO_SUBDIR = : -endif - -ifneq ($(findstring s,$(MAKEFLAGS)),s) -ifndef V - QUIET_CC = @echo ' ' CC $@; - QUIET_AR = @echo ' ' AR $@; - QUIET_LINK = @echo ' ' LINK $@; - QUIET_BUILT_IN = @echo ' ' BUILTIN $@; - QUIET_GEN = @echo ' ' GEN $@; - QUIET_LNCP = @echo ' ' LN/CP $@; - QUIET_XGETTEXT = @echo ' ' XGETTEXT $@; - QUIET_MSGFMT = @echo ' ' MSGFMT $@; - QUIET_GCOV = @echo ' ' GCOV $@; - QUIET_SP = @echo ' ' SP $<; - QUIET_HDR = @echo ' ' HDR $(<:hcc=h); - QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir - export V - export QUIET_GEN - export QUIET_BUILT_IN -endif -endif - ifdef NO_INSTALL_HARDLINKS export NO_INSTALL_HARDLINKS endif
config.mak.uname +0 -1
@@ -727,7 +727,6 @@ vcxproj: git diff-index --cached --quiet HEAD -- # Make .vcxproj files and add them - unset QUIET_GEN QUIET_BUILT_IN; \ perl contrib/buildsystems/generate -g Vcxproj git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
contrib/scalar/Makefile +0 -14
@@ -8,20 +8,6 @@ include ../../config.mak.uname -include ../../config.mak.autogen -include ../../config.mak -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring s,$(MAKEFLAGS)),s) -ifndef V - QUIET_GEN = @echo ' ' GEN $@; - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir -else - export V -endif -endif - TARGETS = scalar$(X) scalar.o GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
shared.mak +53 -0
56 lines not on this page · raw message
templates/Makefile +0 -5
@@ -2,11 +2,6 @@ include ../shared.mak # make and install sample templates - -ifndef V - QUIET = @ -endif - INSTALL ?= install TAR ?= tar RM ?= rm -f
Signature · 1 line
2.35.1.1228.g56895c6ee86
PATCH v4 8/9 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
In reply to Ævar Arnfjörð BjarmasonChange the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding. This refactoring will make it easier to emit that exact amount of padding in functions that we might add to shared.mak in the future.
Such a function is not part of this patch series, but a "TRACK_template" that I'd like to add as a follow-up to it makes use of this. Let's make this change now while modifying these QUIET rules is fresh in our minds.
--- shared.mak | 54 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 30 insertions(+), 24 deletions(-)
shared.mak +30 -24
@@ -32,6 +32,12 @@ comma := , empty := space := $(empty) $(empty) +## wspfx: the whitespace prefix padding for $(QUIET...) and similarly +## aligned output. +wspfx = $(space)$(space)$(space) +wspfx_SQ = '$(subst ','\'',$(wspfx))' +# ' closing quote to appease Emacs make-mode.elxo + ### Quieting ## common QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
@@ -47,39 +53,39 @@ ifneq ($(findstring s,$(MAKEFLAGS)),s) ifndef V ## common QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ + QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo $(wspfx_SQ) SUBDIR $$subdir; \ $(MAKE) $(PRINT_DIR) -C $$subdir QUIET = @ - QUIET_GEN = @echo ' ' GEN $@; + QUIET_GEN = @echo $(wspfx_SQ) GEN $@; ## Used in "Makefile" - QUIET_CC = @echo ' ' CC $@; - QUIET_AR = @echo ' ' AR $@; - QUIET_LINK = @echo ' ' LINK $@; - QUIET_BUILT_IN = @echo ' ' BUILTIN $@; - QUIET_LNCP = @echo ' ' LN/CP $@; - QUIET_XGETTEXT = @echo ' ' XGETTEXT $@; - QUIET_MSGFMT = @echo ' ' MSGFMT $@; - QUIET_GCOV = @echo ' ' GCOV $@; - QUIET_SP = @echo ' ' SP $<; - QUIET_HDR = @echo ' ' HDR $(<:hcc=h); - QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_CC = @echo $(wspfx_SQ) CC $@; + QUIET_AR = @echo $(wspfx_SQ) AR $@; + QUIET_LINK = @echo $(wspfx_SQ) LINK $@; + QUIET_BUILT_IN = @echo $(wspfx_SQ) BUILTIN $@; + QUIET_LNCP = @echo $(wspfx_SQ) LN/CP $@; + QUIET_XGETTEXT = @echo $(wspfx_SQ) XGETTEXT $@; + QUIET_MSGFMT = @echo $(wspfx_SQ) MSGFMT $@; + QUIET_GCOV = @echo $(wspfx_SQ) GCOV $@; + QUIET_SP = @echo $(wspfx_SQ) SP $<; + QUIET_HDR = @echo $(wspfx_SQ) HDR $(<:hcc=h); + QUIET_RC = @echo $(wspfx_SQ) RC $@; + QUIET_SPATCH = @echo $(wspfx_SQ) SPATCH $<; ## Used in "Documentation/Makefile" - QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; - QUIET_XMLTO = @echo ' ' XMLTO $@; - QUIET_DB2TEXI = @echo ' ' DB2TEXI $@; - QUIET_MAKEINFO = @echo ' ' MAKEINFO $@; - QUIET_DBLATEX = @echo ' ' DBLATEX $@; - QUIET_XSLTPROC = @echo ' ' XSLTPROC $@; - QUIET_GEN = @echo ' ' GEN $@; + QUIET_ASCIIDOC = @echo $(wspfx_SQ) ASCIIDOC $@; + QUIET_XMLTO = @echo $(wspfx_SQ) XMLTO $@; + QUIET_DB2TEXI = @echo $(wspfx_SQ) DB2TEXI $@; + QUIET_MAKEINFO = @echo $(wspfx_SQ) MAKEINFO $@; + QUIET_DBLATEX = @echo $(wspfx_SQ) DBLATEX $@; + QUIET_XSLTPROC = @echo $(wspfx_SQ) XSLTPROC $@; + QUIET_GEN = @echo $(wspfx_SQ) GEN $@; QUIET_STDERR = 2> /dev/null - QUIET_LINT_GITLINK = @echo ' ' LINT GITLINK $<; - QUIET_LINT_MANSEC = @echo ' ' LINT MAN SEC $<; - QUIET_LINT_MANEND = @echo ' ' LINT MAN END $<; + QUIET_LINT_GITLINK = @echo $(wspfx_SQ) LINT GITLINK $<; + QUIET_LINT_MANSEC = @echo $(wspfx_SQ) LINT MAN SEC $<; + QUIET_LINT_MANEND = @echo $(wspfx_SQ) LINT MAN END $<; export V endif
Signature · 1 line
2.35.1.1228.g56895c6ee86
Re: [PATCH v4 8/9] Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Quoted 9 lines · Ævar Arnfjörð Bjarmason
Change the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding. This refactoring will make it easier to emit that exact amount of padding in functions that we might add to shared.mak in the future.
Such a function is not part of this patch series, but a "
TRACK_template" that I'd like to add as a follow-up to it makes use of this. Let's make this change now while modifying these QUIET rules is fresh in our minds.
I do not see why this change is still part of this series, then.
shared.mak ifneq ($(findstring s,$(MAKEFLAGS)),s) · line 64
Quoted 7 lines
## Used in "
Makefile"shared.mak +2
-2- QUIET_CC = @echo ' ' CC $@; - QUIET_AR = @echo ' ' AR $@; + QUIET_CC = @echo $(wspfx_SQ) CC $@; + QUIET_AR = @echo $(wspfx_SQ) AR $@;
Look how much uglier these lines got.
Without the benefit TRACK_template brings us, whatever that is, this change looks more of an unnecessary churn. Let's leave it to the later series that introduces the user that justifies this change.
PATCH v4 9/9 Makefiles: add and use wildcard "mkdir -p" template
In reply to Ævar Arnfjörð BjarmasonAdd a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).
As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.
I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.
But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
Time (mean ± σ): 2.914 s ± 0.062 s [User: 2.449 s, System: 0.489 s]
Range (min … max): 2.834 s … 3.020 s 10 runs Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s]
Range (min … max): 2.229 s … 2.397 s 10 runs
Summary
'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'
So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.
This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://gitlab.com/avar/git-hyperfine/
--- Documentation/Makefile | 25 +++---------------------- Makefile | 12 +++++++----- shared.mak | 17 +++++++++++++++++ 3 files changed, 27 insertions(+), 27 deletions(-)
Documentation/Makefile +3 -22
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo print-man1: @for i in $(MAN1_TXT); do echo $$i; done -## Lint: Common -.build: - $(QUIET)mkdir $@ -.build/lint-docs: | .build - $(QUIET)mkdir $@ - ## Lint: gitlink -.build/lint-docs/gitlink: | .build/lint-docs - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT)) -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config $(LINT_DOCS_GITLINK): lint-gitlink.perl $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \ $< \ $(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt lint-docs-gitlink: $(LINT_DOCS_GITLINK) ## Lint: man-end-blurb -.build/lint-docs/man-end-blurb: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@ .PHONY: lint-docs-man-end-blurb -lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB) ## Lint: man-section-order -.build/lint-docs/man-section-order: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@ .PHONY: lint-docs-man-section-order lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
Makefile +7 -5
@@ -2745,7 +2745,8 @@ all:: $(MOFILES) endif po/build/locale/%/LC_MESSAGES/git.mo: po/%.po - $(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $< + $(call mkdir_p_parent_template) + $(QUIET_MSGFMT)$(MSGFMT) -o $@ $< LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm) LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2761,15 +2762,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS)) endif perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES - $(QUIET_GEN)mkdir -p $(dir $@) && \ + $(call mkdir_p_parent_template) + $(QUIET_GEN) \ sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \ -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \ -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \ < $< > $@ perl/build/man/man3/Git.3pm: perl/Git.pm - $(QUIET_GEN)mkdir -p $(dir $@) && \ - pod2man $< $@ + $(call mkdir_p_parent_template) + $(QUIET_GEN)pod2man $< $@ $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \
@@ -2903,7 +2905,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $( all:: $(TEST_PROGRAMS) $(test_bindir_programs) bin-wrappers/%: wrap-for-bin.sh - @mkdir -p bin-wrappers + $(call mkdir_p_parent_template) $(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \ -e 's|@@BUILD_DIR@@|$(shell pwd)|' \ -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
shared.mak +17 -0
@@ -59,6 +59,8 @@ ifndef V QUIET = @ QUIET_GEN = @echo $(wspfx_SQ) GEN $@; + QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D); + ## Used in "Makefile" QUIET_CC = @echo $(wspfx_SQ) CC $@; QUIET_AR = @echo $(wspfx_SQ) AR $@;
@@ -90,3 +92,18 @@ ifndef V export V endif endif + +### Templates + +## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@ +## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not +## needed. +## +## Is racy, but in a good way; we might redundantly (and safely) +## "mkdir -p" when running in parallel, but won't need to exhaustively create +## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule. +define mkdir_p_parent_template +$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D))) +endef
Signature · 1 line
2.35.1.1228.g56895c6ee86
Re: [PATCH v4 9/9] Makefiles: add and use wildcard "mkdir -p" template
Quoted 3 lines · Ævar Arnfjörð Bjarmason
Add a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in
8650c6298c1(doc lint: make "lint-docs" non-.PHONY, 2021-10-15).
Sounds good.
Quoted 5 lines
But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):
Cute.
Quoted 14 lines
shared.mak +2
-0@@ -59,6 +59,8 @@ ifndef V QUIET = @ QUIET_GEN = @echo $(wspfx_SQ) GEN $@; + QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D); + ## Used in "Makefile" QUIET_CC = @echo $(wspfx_SQ) CC $@; QUIET_AR = @echo $(wspfx_SQ) AR $@;
shared.mak ifndef V · line 94
Quoted 3 lines
shared.mak +0
-0@@ -90,3 +92,18 @@ ifndef V export V endif endif
It is unfortunate that this came after [8/9] and needs manual resolution to reorder to leave 8/9 for later series.
Quoted 7 lines · Ævar Arnfjörð Bjarmason
As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).
Other than a few things I noticed and commented on, 1-7 and 9 looked all sensible.
Thanks.
Quoted 62 lines · Ævar Arnfjörð Bjarmason
As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).
This v4 re-roll (for v3, see
https://lore.kernel.org/git/[email protected]/):* The "all" boilerplate goes first now, before "include"
* Elaborated onDELETE_ON_ERRORina commit message.
* No longer change a ":=" to "=" while moving code.
* Rephrased other commit messages (one of which referred to a function from another future series)
* Typo fix in commit message.Ævar Arnfjörð Bjarmason (9):
scalarMakefile: use "The default target of..." pattern
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" templateDocumentation/Makefile | 63 ++------------------ Makefile | 118 ++++++++++++-------------------------- config.mak.uname | 1 - contrib/scalar/Makefile | 20 ++----- contrib/scalar/t/Makefile | 3 + shared.mak | 109 +++++++++++++++++++++++++++++++++++ t/Makefile | 3 + t/interop/Makefile | 3 + t/perf/Makefile | 3 + templates/Makefile | 8 +-- 10 files changed, 171 insertions(+), 160 deletions(-)create mode 100644 shared.mak
Range-diff against v3:
1: 2404c4d8b96 < -: ----------- scalar Makefile: set the default target after the includes -: ----------- > 1: 26c6bb897cf scalar Makefile: use "The default target of..." pattern 2: 96a490bec54 ! 2: 74692458b70 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it@@ Commit message
.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.- This does have the potential downside that if e.g. templates/Makefile
- would like to include this "shared.mak" in the future the semantics of
- such aMakefilewill change, but as noted in the above commits (and
- GNU make's own documentation) any such change would be for the better,
- so it's safe to do this.
+ I.e. this changes the behavior of existing rules in the altered
+ Makefiles (except "Makefile" & "Documentation/Makefile"). I'm
+ confident that this is safe having read the relevant rules in those
+ Makfiles, and as the GNU make manual notes that it isn't the default
+ behavior is out of an abundance of backwards compatibility
+ caution. From edition 0.75 of its manual, covering GNU make 4.3:
Quoted 8 lines
- This also doesn't introduce a bug by e.g. having this
+ [Enabling '.DELETE_ON_ERROR' is] almost always what you want
+ 'make' to do, but it is not historical practice; so for
+ compatibility, you must explicitly request it.
+
+ This doesn't introduce a bug by e.g. having this
".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
I think this is much clearer now,
Best Wishes
Phillip
Quoted 162 lines
+ It does increase the danger that any Makefile without an explicit "The + default target of this Makefile is..." snippet to define the default + target as "all" could have its default rule changed if our new + shared.mak ever defines a "real" rule. In subsequent commits we'll be + careful not to do that, and such breakage would be obvious e.g. in the + case of "make -C t".+
+ We might want to make that less fragile still (e.g. by using
+ ".DEFAULT_GOAL" as noted in the preceding commit), but for now let's
+ simply include "shared.mak" without adding that boilerplate to all the
+ Makefiles that don't have it already. Most of those are already
+ exposed to that potential caveat e.g. due to including "config.mak*".
+Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>##
Documentation/Makefile##
@@Documentation/Makefile: doc-l10n install-l10n::##
Makefile##
@@
-+# Import tree-wide sharedMakefilebehavior and libraries -+include shared.mak
-+
# The default target of thisMakefileis... all::++# Import tree-wide shared
Makefilebehavior and libraries
++include shared.mak
++
+ # Define V=1 to have a more verbose compile.
+ #
+ # DefineSHELL_PATHto a POSIX shell if your/bin/shis broken.
@@Makefile:shell_compatibility_test:please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $^
@@Makefile:shell_compatibility_test:please_set_SHELL_PATH_to_a_more_modern_shell## contrib/scalar/Makefile ##
@@
+ # The default target of thisMakefileis...
+ all::
+
+# Import tree-wide sharedMakefilebehavior and libraries +include ../../shared.mak
+
-QUIET_SUBDIR0= +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1=
-
+ include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak## contrib/scalar/t/Makefile ##
@@
3:9392e3c3e97! 3:0fbdeeffc7bMakefile: disable GNU make built-in wildcard rules
@@ Commit message
contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.- The speed-up may wary. I've seen 1-10% depending on the speed of the
+ The speed-up may vary. I've seen 1-10% depending on the speed of the
local disk, caches, -jN etc. Running:strace -f -c -S calls make -j1 NO_TCLTK=Y
4: 07cf9daa9d6 = 4: ea6b835308a Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) 5: 16f2e3ff35b = 5: c2339694cf7 Makefile: move ".SUFFIXES" rule to shared.mak 6: 1b6ecb27f02 ! 6: 741fdfd48e2 Makefile: move $(comma), $(empty) and $(space) to shared.mak@@ Metadata
## Commit message ##
Makefile: move $(comma), $(empty) and $(space) to shared.mak- Move these variables over to the shared.max, we'll make use of them in
- a subsequent commit. There was no reason for these to be "simply
- expanded variables", so let's use the normal lazy "=" assignment here.
+ Move these variables over to the shared.mak, we'll make use of them in
+ a subsequent commit.
+
+ Note that there's reason for these to be "simply expanded variables",
+ i.e. to use ":=" assignments instead of lazily expanded "="
+ assignments. We could use "=", but let's leave this as-is for now for
+ ease of review.See
425ca6710b2(Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.
@@ shared.mak
+
+## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among otherMakefilesyntax.
-+comma = ,
-+empty =
-+space = $(empty) $(empty)
++comma := ,
++empty :=
++space := $(empty) $(empty)
7:471067deefc! 7:a723cbce270Makefile: add "$(QUIET)" boilerplate to shared.mak
@@ config.mak.uname: vcxproj:## contrib/scalar/Makefile ##
-@@
- # Import tree-wide sharedMakefilebehavior and libraries
- include ../../shared.mak
+@@ contrib/scalar/Makefile: include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak-QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 =
@@ contrib/scalar/Makefile
-endif
-endif
-
- include ../../config.mak.uname
- -include ../../config.mak.autogen
- -include ../../config.mak
+ TARGETS = scalar$(X) scalar.o
+ GITLIBS =../../common-main.o../../libgit.a ../../xdiff/lib.a +## shared.mak ##
@@
- comma = ,
- empty =
- space = $(empty) $(empty)
+ comma := ,
+ empty :=
+ space := $(empty) $(empty)
+
+### Quieting
+## common
8:510306d2219! 8:3733b0c8df1Makefile: use $(wspfx) for $(QUIET...) in shared.mak
@@ Commit message
Makefile: use $(wspfx) for $(QUIET...) in shared.makChange the mostly move-only change in the preceding commit to use the
- $(wspfx) variable for defining the QUIET padding, to guarantee that
- it's consistent with the "TRACK_template" template.
+ $(wspfx) variable for defining the QUIET padding. This refactoring
+ will make it easier to emit that exact amount of padding in functions
+ that we might add to shared.mak in the future.- $ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
- -> GIT-CFLAGS PARAMETERS (changed)
- -> CC grep.o
+ Such a function is not part of this patch series, but a
+ "TRACK_template" that I'd like to add as a follow-up to it makes use
+ of this. Let's make this change now while modifying these QUIET rules
+ is fresh in our minds.Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>## shared.mak ##
-@@ shared.mak: comma = ,
- empty =
- space = $(empty) $(empty)
+@@ shared.mak: comma := ,
+ empty :=
+ space := $(empty) $(empty)+## wspfx: the whitespace prefix padding for $(QUIET...) and similarly +## aligned output.
9:85bb74aa32f= 9:4cc4aeabb20Makefiles: add and use wildcard "mkdir -p" template
v5 cover Makefile: optimize noop runs, add shared.mak
In reply to Ævar Arnfjörð BjarmasonAs the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/
This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).
This v5 re-roll (for v3, see:
https://lore.kernel.org/git/[email protected]/):
* Drops the "wspfx" patch.
* Rephrases the 1/8 commit message, dropping the mention of .DEFAULT_GOAL etc.
I think this should be ready to advance now per Junio's comment on v4
(https://lore.kernel.org/git/[email protected]/):
Other than a few things I noticed and commented on, 1-7 and 9 looked all sensible.
Ævar Arnfjörð Bjarmason (8):
scalar Makefile: use "The default target of..." pattern
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefiles: add and use wildcard "mkdir -p" template
Documentation/Makefile | 63 ++------------------ Makefile | 118 ++++++++++++-------------------------- config.mak.uname | 1 - contrib/scalar/Makefile | 20 ++----- contrib/scalar/t/Makefile | 3 + shared.mak | 103 +++++++++++++++++++++++++++++++++ t/Makefile | 3 + t/interop/Makefile | 3 + t/perf/Makefile | 3 + templates/Makefile | 8 +-- 10 files changed, 165 insertions(+), 160 deletions(-) create mode 100644 shared.mak
Range-diff against v4:
1: 26c6bb897cf ! 1: 7547bf3e481 scalar Makefile: use "The default target of..." pattern
@@ Metadata
## Commit message ##
scalar Makefile: use "The default target of..." pattern
- Make have the "contrib/scalar/Makefile" be stylistically consistent
- with the top-level "Makefile" in first declaring "all" to be the
- default rule, follwed by including other Makefile snippets.
+ Make the "contrib/scalar/Makefile" be stylistically consistent with
+ the top-level "Makefile" in first declaring "all" to be the default
+ rule, followed by including other Makefile snippets.
This adjusts code added in 0a43fb22026 (scalar: create a rudimentary
- executable, 2021-12-03), it's a style-only change, in a subsequent
- commit the "QUIET" boilerplate at the beginning of this file will be
- retrieved via an include, and having an "all:" between the two set of
- "include"'s after that change would look odd.
-
- As noted in [1] using ".DEFAULT_GOAL = all" is another way to do this
- in more modern GNU make versions, which we already have a hard
- dependency on, but let's leave any such change for a future
- improvement and go with using our established pattern consistently for
- now.
-
- 1. https://lore.kernel.org/git/[email protected]/
+ executable, 2021-12-03), it further ensures that when we add another
+ "include" file in a subsequent commit that the included file won't be
+ the one to define our default target.
2: 74692458b70 = 2: 91795eccc32 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it 3: 0fbdeeffc7b = 3: 9f42f40b518 Makefile: disable GNU make built-in wildcard rules 4: ea6b835308a = 4: 034779ff7fb Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) 5: c2339694cf7 = 5: 18e0a6985f1 Makefile: move ".SUFFIXES" rule to shared.mak 6: 741fdfd48e2 = 6: 422dee02ae9 Makefile: move $(comma), $(empty) and $(space) to shared.mak 7: a723cbce270 = 7: 21bf1e6e01c Makefile: add "$(QUIET)" boilerplate to shared.mak 8: 3733b0c8df1 < -: ----------- Makefile: use $(wspfx) for $(QUIET...) in shared.mak 9: 4cc4aeabb20 ! 8: e6a93cae81d Makefiles: add and use wildcard "mkdir -p" template
@@ Makefile: test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_
## shared.mak ##
@@ shared.mak: ifndef V
QUIET = @
- QUIET_GEN = @echo $(wspfx_SQ) GEN $@;
+ QUIET_GEN = @echo ' ' GEN $@;
+ QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D);
+
## Used in "Makefile"
- QUIET_CC = @echo $(wspfx_SQ) CC $@;
- QUIET_AR = @echo $(wspfx_SQ) AR $@;
+ QUIET_CC = @echo ' ' CC $@;
+ QUIET_AR = @echo ' ' AR $@;
@@ shared.mak: ifndef V
export V
endif
Signature · 1 line
2.35.1.1230.ga6e6579e98c
PATCH v5 1/8 scalar Makefile: use "The default target of..." pattern
Make the "contrib/scalar/Makefile" be stylistically consistent with the top-level "Makefile" in first declaring "all" to be the default rule, followed by including other Makefile snippets.
This adjusts code added in 0a43fb22026 (scalar: create a rudimentary executable, 2021-12-03), it further ensures that when we add another "include" file in a subsequent commit that the included file won't be the one to define our default target.
--- contrib/scalar/Makefile | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
contrib/scalar/Makefile +7 -6
@@ -1,3 +1,10 @@ +# The default target of this Makefile is... +all:: + +include ../../config.mak.uname +-include ../../config.mak.autogen +-include ../../config.mak + QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir QUIET_SUBDIR1 =
@@ -12,12 +19,6 @@ else endif endif -all: - -include ../../config.mak.uname --include ../../config.mak.autogen --include ../../config.mak - TARGETS = scalar$(X) scalar.o GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
Signature · 1 line
2.35.1.1230.ga6e6579e98c
PATCH v5 2/8 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
In reply to Ævar Arnfjörð BjarmasonWe have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
I.e. this changes the behavior of existing rules in the altered Makefiles (except "Makefile" & "Documentation/Makefile"). I'm confident that this is safe having read the relevant rules in those Makfiles, and as the GNU make manual notes that it isn't the default behavior is out of an abundance of backwards compatibility caution. From edition 0.75 of its manual, covering GNU make 4.3:
[Enabling '.DELETE_ON_ERROR' is] almost always what you want 'make' to do, but it is not historical practice; so for compatibility, you must explicitly request it.
This doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.
It does increase the danger that any Makefile without an explicit "The default target of this Makefile is..." snippet to define the default target as "all" could have its default rule changed if our new shared.mak ever defines a "real" rule. In subsequent commits we'll be careful not to do that, and such breakage would be obvious e.g. in the case of "make -C t".
We might want to make that less fragile still (e.g. by using ".DEFAULT_GOAL" as noted in the preceding commit), but for now let's simply include "shared.mak" without adding that boilerplate to all the Makefiles that don't have it already. Most of those are already exposed to that potential caveat e.g. due to including "config.mak*".
--- Documentation/Makefile | 6 +++--- Makefile | 13 +++---------- contrib/scalar/Makefile | 3 +++ contrib/scalar/t/Makefile | 3 +++ shared.mak | 9 +++++++++ t/Makefile | 3 +++ t/interop/Makefile | 3 +++ t/perf/Makefile | 3 +++ templates/Makefile | 3 +++ 9 files changed, 33 insertions(+), 13 deletions(-) create mode 100644 shared.mak
Documentation/Makefile +3 -3
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Guard against environment variables MAN1_TXT = MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n:: $(MAKE) -C po $@ endif -# Delete the target file on error -.DELETE_ON_ERROR: - .PHONY: FORCE
Makefile +3 -10
@@ -1,6 +1,9 @@ # The default target of this Makefile is... all:: +# Import tree-wide shared Makefile behavior and libraries +include shared.mak + # Define V=1 to have a more verbose compile. # # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
@@ -2194,16 +2197,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell strip: $(PROGRAMS) git$X $(STRIP) $(STRIP_OPTS) $^ -### Flags affecting all rules - -# A GNU make extension since gmake 3.72 (released in late 1994) to -# remove the target of rules if commands in those rules fail. The -# default is to only do that if make itself receives a signal. Affects -# all targets, see: -# -# info make --index-search=.DELETE_ON_ERROR -.DELETE_ON_ERROR: - ### Target-specific flags and dependencies # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
@@ -1,6 +1,9 @@ # The default target of this Makefile is... all:: +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + include ../../config.mak.uname -include ../../config.mak.autogen -include ../../config.mak
contrib/scalar/t/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../../shared.mak + # Run scalar tests # # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
@@ -0,0 +1,9 @@ +### Flags affecting all rules + +# A GNU make extension since gmake 3.72 (released in late 1994) to +# remove the target of rules if commands in those rules fail. The +# default is to only do that if make itself receives a signal. Affects +# all targets, see: +# +# info make --index-search=.DELETE_ON_ERROR +.DELETE_ON_ERROR:
t/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # Run tests # # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS
t/perf/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak + -include ../../config.mak export GIT_TEST_OPTIONS
templates/Makefile +3 -0
@@ -1,3 +1,6 @@ +# Import tree-wide shared Makefile behavior and libraries +include ../shared.mak + # make and install sample templates ifndef V
Signature · 1 line
2.35.1.1230.ga6e6579e98c
PATCH v5 3/8 Makefile: disable GNU make built-in wildcard rules
In reply to Ævar Arnfjörð BjarmasonOverride built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.
The speed-up may vary. I've seen 1-10% depending on the speed of the local disk, caches, -jN etc. Running:
strace -f -c -S calls make -j1 NO_TCLTK=Y
Shows that we reduce the number of syscalls we make, mostly in "stat" calls.
We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early. Doing so might make us a bit faster still. But doing so is a much bigger hammer, since it will disable all built-in rules, some (all?) of which can be seen with:
make -f/dev/null -p | grep -v -e ^# -e ^$
We may have something that relies on them, so let's go for the more isolated optimization here that gives us most or all of the wins.
1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html
--- shared.mak | 11 +++++++++++ 1 file changed, 11 insertions(+)
shared.mak +11 -0
@@ -1,3 +1,14 @@ +### Remove GNU make implicit rules + +## This speeds things up since we don't need to look for and stat() a +## "foo.c,v" every time a rule referring to "foo.c" is in play. See +## "make -p -f/dev/null | grep ^%::'". +%:: %,v +%:: RCS/%,v +%:: RCS/% +%:: s.% +%:: SCCS/s.% + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1230.ga6e6579e98c
Re: [PATCH v5 3/8] Makefile: disable GNU make built-in wildcard rules
On Thu, Mar 03, 2022 at 05:04:14PM +0100, Ævar Arnfjörð Bjarmason wrote:
[...]
Quoted 10 lines
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- shared.mak | 11 +++++++++++ 1 file changed, 11 insertions(+)shared.mak +0
-0
shared.mak line 5
Quoted 5 lines
shared.mak +5
-0@@ -1,3 +1,14 @@ +### Remove GNU make implicit rules + +## This speeds things up since we don't need to look for and stat() a +## "foo.c,v" every time a rule referring to "foo.c" is in play. See +## "make -p -f/dev/null | grep ^%::'".
^
A bit late to the party, but are you missing an opening single quote there?
Quoted 12 lines
shared.mak +6
-0+%:: %,v +%:: RCS/%,v +%:: RCS/% +%:: s.% +%:: SCCS/s.% + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) toSignature · 1 line
2.35.1.1230.ga6e6579e98c
Re: [PATCH v5 3/8] Makefile: disable GNU make built-in wildcard rules
Quoted 20 lines · Rene Kita
On Thu, Mar 03, 2022 at 05:04:14PM +0100, Ævar Arnfjörð Bjarmason wrote:
[...]
Quoted 10 lines
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- shared.mak | 11 +++++++++++ 1 file changed, 11 insertions(+)shared.mak +0
-0shared.mak line 5
Quoted 5 lines
shared.mak +5
-0@@ -1,3 +1,14 @@ +### Remove GNU make implicit rules + +## This speeds things up since we don't need to look for and stat() a +## "foo.c,v" every time a rule referring to "foo.c" is in play. See +## "make -p -f/dev/null | grep ^%::'".^
Quoted 2 lines
A bit late to the party, but are you missing an opening single quote there?
Yes, that's a typo. Oops!
PATCH v5 4/8 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
In reply to Ævar Arnfjörð BjarmasonCombine the definitions of $(FIND_SOURCE_FILES) and $(LIB_H) to speed up the Makefile, as these are the two main expensive $(shell) commands that we execute unconditionally.
When see what was in $(FOUND_SOURCE_FILES) that wasn't in $(LIB_H) via the ad-hoc test of:
$(error $(filter-out $(LIB_H),$(filter %.h,$(ALL_SOURCE_FILES))))
$(error $(filter-out $(ALL_SOURCE_FILES),$(filter %.h,$(LIB_H))))We'll get, respectively:
Makefile:850: *** t/helper/test-tool.h. Stop.
Makefile:850: *** . Stop.I.e. we only had a discrepancy when it came to t/helper/test-tool.h. In terms of correctness this was broken before, but now works:
$ make t/helper/test-tool.hco
HDR t/helper/test-tool.hThis speeds things up a lot:
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10
Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1
Time (mean ± σ): 159.9 ms ± 6.8 ms [User: 137.2 ms, System: 28.0 ms]
Range (min … max): 154.6 ms … 175.9 ms 10 runs Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0
Time (mean ± σ): 100.0 ms ± 1.3 ms [User: 84.2 ms, System: 20.2 ms]
Range (min … max): 98.8 ms … 102.8 ms 10 runs
Summary
'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran
1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1'--- Makefile | 54 ++++++++++++++++++++++++++---------------------------- 1 file changed, 26 insertions(+), 28 deletions(-)
Makefile +26 -28
@@ -833,12 +833,33 @@ GENERATED_H += hook-list.h .PHONY: generated-hdrs generated-hdrs: $(GENERATED_H) -LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \ +## Exhaustive lists of our source files, either dynamically generated, +## or hardcoded. +SOURCES_CMD = ( \ + git ls-files \ + '*.[hcS]' \ + '*.sh' \ + ':!*[tp][0-9][0-9][0-9][0-9]*' \ + ':!contrib' \ + 2>/dev/null || \ $(FIND) . \ - -name .git -prune -o \ - -name t -prune -o \ - -name Documentation -prune -o \ - -name '*.h' -print))) + \( -name .git -type d -prune \) \ + -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ + -o \( -name contrib -type d -prune \) \ + -o \( -name build -type d -prune \) \ + -o \( -name 'trash*' -type d -prune \) \ + -o \( -name '*.[hcS]' -type f -print \) \ + -o \( -name '*.sh' -type f -print \) \ + | sed -e 's|^\./||' \ + ) +FOUND_SOURCE_FILES := $(shell $(SOURCES_CMD)) + +FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) +FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES)) + +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) + +LIB_H = $(FOUND_H_SOURCES) LIB_OBJS += abspath.o LIB_OBJS += add-interactive.o
@@ -2789,26 +2810,6 @@ perl/build/man/man3/Git.3pm: perl/Git.pm $(QUIET_GEN)mkdir -p $(dir $@) && \ pod2man $< $@ -FIND_SOURCE_FILES = ( \ - git ls-files \ - '*.[hcS]' \ - '*.sh' \ - ':!*[tp][0-9][0-9][0-9][0-9]*' \ - ':!contrib' \ - 2>/dev/null || \ - $(FIND) . \ - \( -name .git -type d -prune \) \ - -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ - -o \( -name contrib -type d -prune \) \ - -o \( -name build -type d -prune \) \ - -o \( -name 'trash*' -type d -prune \) \ - -o \( -name '*.[hcS]' -type f -print \) \ - -o \( -name '*.sh' -type f -print \) \ - | sed -e 's|^\./||' \ - ) - -FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES)) - $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \ echo $(FOUND_SOURCE_FILES) | xargs etags -a -o $@+ && \
@@ -3018,9 +3019,6 @@ check: $(GENERATED_H) exit 1; \ fi -FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES)) -COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) - %.cocci.patch: %.cocci $(COCCI_SOURCES) $(QUIET_SPATCH) \ if test $(SPATCH_BATCH_SIZE) = 0; then \
Signature · 1 line
2.35.1.1230.ga6e6579e98c
PATCH v5 5/8 Makefile: move ".SUFFIXES" rule to shared.mak
In reply to Ævar Arnfjörð BjarmasonThis was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.
This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms]
Range (min … max): 112.8 ms … 148.4 ms 26 runs Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs
Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'
The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.
So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.
1. https://lore.kernel.org/git/[email protected]/
--- Makefile | 2 -- shared.mak | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S)) ASM_OBJ := $(ASM_SRC:S=o) C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS)) -.SUFFIXES: - $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir) $(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +5 -0
@@ -9,6 +9,11 @@ %:: s.% %:: SCCS/s.% +## Likewise delete default $(SUFFIXES). See: +## +## info make --index-search=.SUFFIXES +.SUFFIXES: + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1230.ga6e6579e98c
Re: [PATCH v5 5/8] Makefile: move ".SUFFIXES" rule to shared.mak
Quoted 59 lines · Ævar Arnfjörð Bjarmason
This was added in
30248886ce8(Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.This doesn't benefit the main
Makefileat all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man' Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1 Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms] Range (min … max): 112.8 ms … 148.4 ms 26 runsBenchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runsSummary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.
So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.
1. https://lore.kernel.org/git/[email protected]/
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- Makefile | 2 -- shared.mak | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)Makefile +0
-2@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S)) ASM_OBJ := $(ASM_SRC:S=o) C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS)) -.SUFFIXES: - $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir) $(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)shared.mak +0
-0@@ -9,6 +9,11 @@ %:: s.%
shared.mak line 19
Quoted 10 lines
shared.mak +5
-0%:: SCCS/s.% +## Likewise delete default $(SUFFIXES). See: +## +## info make --index-search=.SUFFIXES +.SUFFIXES: + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) to
I confess I really don't understand why, but as part of testing v2.36.0-rc0 on Cygwin, I've started getting errors building the info pages, and bisect points to this commit as the culprit.
Specifically, I've been running
git clean -dffx && make configure && ./configure && make -j4 info
Without this commit, that gets me a successful build; there's a bunch of noisy warnings that have been hanging around for a long time, and I think are fundamentally due to the slightly mismatched documentation libraries that Cygwin has. With this commit, I get the same noisy warnings, but I also get the error "could not open .texi: No such file or directory".
I have to confess, I don't really understand this aspect of GNU Make, so I'm not sure if this is a problem with Cygwin having a bad toolset or there being something about my environment that means this doesn't work, but regardless, it's currently causing the Cygwin Git builds to fail.
Re: [PATCH v5 5/8] Makefile: move ".SUFFIXES" rule to shared.mak
Quoted 85 lines · Adam Dinwoodie
Quoted 59 lines · Ævar Arnfjörð Bjarmason
This was added in
30248886ce8(Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.This doesn't benefit the main
Makefileat all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man' Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1 Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms] Range (min … max): 112.8 ms … 148.4 ms 26 runsBenchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runsSummary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.
So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.
1. https://lore.kernel.org/git/[email protected]/
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- Makefile | 2 -- shared.mak | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)Makefile +0
-2@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S)) ASM_OBJ := $(ASM_SRC:S=o) C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS)) -.SUFFIXES: - $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir) $(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)shared.mak +0
-0@@ -9,6 +9,11 @@ %:: s.%shared.mak line 19
Quoted 10 lines
shared.mak +5
-0%:: SCCS/s.% +## Likewise delete default $(SUFFIXES). See: +## +## info make --index-search=.SUFFIXES +.SUFFIXES: + ### Flags affecting all rules # A GNU make extension since gmake 3.72 (released in late 1994) toI confess I really don't understand why, but as part of testing v2.36.0-rc0 on Cygwin, I've started getting errors building the info pages, and bisect points to this commit as the culprit.
Specifically, I've been running
git clean -dffx && make configure && ./configure && make -j4 infoWithout this commit, that gets me a successful build; there's a bunch of noisy warnings that have been hanging around for a long time, and I think are fundamentally due to the slightly mismatched documentation libraries that Cygwin has. With this commit, I get the same noisy warnings, but I also get the error "could not open .texi: No such file or directory".
Quoted 4 lines
I have to confess, I don't really understand this aspect of GNU Make, so I'm not sure if this is a problem with Cygwin having a bad toolset or there being something about my environment that means this doesn't work, but regardless, it's currently causing the Cygwin Git builds to fail.
Hi. I can reproduce this locally, will look at it and fix it, sorry.
PATCH 1/1 Documentation/Makefile: fix "make info" regression in dad9cd7d518
In reply to Adam DinwoodieFix a regression in my dad9cd7d518 (Makefile: move ".SUFFIXES" rule to shared.mak, 2022-03-03). As explained in the GNU make documentation for the $* variable, available at:
info make --index-search='$*'
This rule relied on ".texi" being in the default list of suffixes, as seen at:
make -f/dev/null -p | grep -v -e ^# -e ^$|grep -F .SUFFIXES
The documentation explains what was going on here:
In an explicit rule, there is no stem; so '$*' cannot be determined in that way. Instead, if the target name ends with a recognized suffix (*note Old-Fashioned Suffix Rules: Suffix Rules.), '$*' is set to the target name minus the suffix. For example, if the target name is 'foo.c', then '$*' is set to 'foo', since '.c' is a suffix. GNU 'make' does this bizarre thing only for compatibility with other implementations of 'make'. You should generally avoid using '$*' except in implicit rules or static pattern rules.
If the target name in an explicit rule does not end with a recognized suffix, '$*' is set to the empty string for that rule.
I.e. this rule added back in 5cefc33bffd (Documentation: add gitman.info target, 2007-12-10) was resolving gitman.texi from gitman.info. We can instead just use the more obvious $< variable referring to the prerequisite.
This was the only use of $* in our Makefiles in an explicit rule, the three remaining ones are all implicit rules, and therefore didn't depend on the ".SUFFIXES" list.
---
Quoted 2 lines · Adam Dinwoodie
With this commit, I get the same noisy warnings, but I also get the error "could not open .texi: No such file or directory".
Sorry about the regression. This fixes it, and as noted above I'm pretty sure this was the only fallout of the change.
(I didn't have building the full extended documentation as part of my local build, but I'll be adding it now).
Documentation/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/Makefile +1 -1
@@ -390,7 +390,7 @@ gitman.texi: $(MAN_XML) cat-texi.perl texi.xsl $(RM) $@+ gitman.info: gitman.texi - $(QUIET_MAKEINFO)$(MAKEINFO) --no-split --no-validate $*.texi + $(QUIET_MAKEINFO)$(MAKEINFO) --no-split --no-validate $< $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml $(QUIET_DB2TEXI)$(DOCBOOK2X_TEXI) --to-stdout $*.xml >$@
Signature · 1 line
2.35.1.1604.g35dc6517170
Re: [PATCH] Documentation/Makefile: fix "make info" regression in dad9cd7d518
Quoted 40 lines · Ævar Arnfjörð Bjarmason
Fix a regression in my
dad9cd7d518(Makefile: move ".SUFFIXES" rule to shared.mak, 2022-03-03). As explained in the GNU make documentation for the $* variable, available at:info make --index-search='$*'This rule relied on ".texi" being in the default list of suffixes, as seen at:
make -f/dev/null -p | grep -v -e ^# -e ^$|grep -F .SUFFIXESThe documentation explains what was going on here:
In an explicit rule, there is no stem; so '$*' cannot be determined in that way. Instead, if the target name ends with a recognized suffix (*note Old-Fashioned Suffix Rules: Suffix Rules.), '$*' is set to the target name minus the suffix. For example, if the target name is '
foo.c', then '$*' is set to 'foo', since '.c' is a suffix. GNU 'make' does this bizarre thing only for compatibility with other implementations of 'make'. You should generally avoid using '$*' except in implicit rules or static pattern rules.If the target name in an explicit rule does not end with a recognized suffix, '$*' is set to the empty string for that rule.
I.e. this rule added back in
5cefc33bffd(Documentation: add gitman.info target, 2007-12-10) was resolving gitman.texi from gitman.info. We can instead just use the more obvious $< variable referring to the prerequisite.This was the only use of $* in our Makefiles in an explicit rule, the three remaining ones are all implicit rules, and therefore didn't depend on the ".SUFFIXES" list.
Reported-by Adam Dinwoodie <[email protected]>Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>---
On Tue, Apr 05 2022, Adam Dinwoodie wrote:
Quoted 8 lines
Quoted 2 lines
With this commit, I get the same noisy warnings, but I also get the error "could not open .texi: No such file or directory".
Sorry about the regression. This fixes it, and as noted above I'm pretty sure this was the only fallout of the change.
(I didn't have building the full extended documentation as part of my local build, but I'll be adding it now).
Confirmed this patch fixes things for me. Thanks for the quick fix!
Re: [PATCH] Documentation/Makefile: fix "make info" regression in dad9cd7d518
Quoted 3 lines · Adam Dinwoodie
Confirmed this patch fixes things for me. Thanks for the quick fix!
Tested-by Adam Dinwoodie <[email protected]>
Thanks, both.
Re: [PATCH] Documentation/Makefile: fix "make info" regression in dad9cd7d518
Quoted 7 lines · Junio C Hamano
Quoted 3 lines · Adam Dinwoodie
Confirmed this patch fixes things for me. Thanks for the quick fix!
Tested-by Adam Dinwoodie <[email protected]>Thanks, both.
Indeed; thanks for the fix, Ævar. This sub-thread was a fun read for me to get to learn about a new "make" quirk. You learn something new every day ;-).
Thanks,
Taylor
PATCH v5 6/8 Makefile: move $(comma), $(empty) and $(space) to shared.mak
In reply to Ævar Arnfjörð BjarmasonMove these variables over to the shared.mak, we'll make use of them in a subsequent commit.
Note that there's reason for these to be "simply expanded variables", i.e. to use ":=" assignments instead of lazily expanded "=" assignments. We could use "=", but let's leave this as-is for now for ease of review.
See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.
--- Makefile | 4 ---- shared.mak | 8 ++++++++ 2 files changed, 8 insertions(+), 4 deletions(-)
Makefile +0 -4
@@ -1289,10 +1289,6 @@ endif ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS) ALL_LDFLAGS = $(LDFLAGS) -comma := , -empty := -space := $(empty) $(empty) - ifdef SANITIZE SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag)) BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)
shared.mak +8 -0
@@ -23,3 +23,11 @@ # # info make --index-search=.DELETE_ON_ERROR .DELETE_ON_ERROR: + +### Global variables + +## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among other Makefile syntax. +comma := , +empty := +space := $(empty) $(empty)
Signature · 1 line
2.35.1.1230.ga6e6579e98c
PATCH v5 7/8 Makefile: add "$(QUIET)" boilerplate to shared.mak
In reply to Ævar Arnfjörð BjarmasonThe $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.
Since we're not using the environment to pass these around we don't need to export the "QUIET_GEN" and "QUIET_BUILT_IN" variables anymore. The "QUIET_GEN" variable is used in "git-gui/Makefile" and "gitweb/Makefile", but they've got their own definition for those. The "QUIET_BUILT_IN" variable is only used in the top-level "Makefile". We still need to export the "V" variable.
--- Documentation/Makefile | 32 ------------------------- Makefile | 33 ------------------------- config.mak.uname | 1 - contrib/scalar/Makefile | 14 ----------- shared.mak | 53 +++++++++++++++++++++++++++++++++++++++++ templates/Makefile | 5 ---- 6 files changed, 53 insertions(+), 85 deletions(-)
Documentation/Makefile +0 -32
@@ -218,38 +218,6 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR)) ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)' endif -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring $(MAKEFLAGS),w),w) -PRINT_DIR = --no-print-directory -else # "make -w" -NO_SUBDIR = : -endif - -ifneq ($(findstring $(MAKEFLAGS),s),s) -ifndef V - QUIET = @ - QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; - QUIET_XMLTO = @echo ' ' XMLTO $@; - QUIET_DB2TEXI = @echo ' ' DB2TEXI $@; - QUIET_MAKEINFO = @echo ' ' MAKEINFO $@; - QUIET_DBLATEX = @echo ' ' DBLATEX $@; - QUIET_XSLTPROC = @echo ' ' XSLTPROC $@; - QUIET_GEN = @echo ' ' GEN $@; - QUIET_STDERR = 2> /dev/null - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir - - QUIET_LINT_GITLINK = @echo ' ' LINT GITLINK $<; - QUIET_LINT_MANSEC = @echo ' ' LINT MAN SEC $<; - QUIET_LINT_MANEND = @echo ' ' LINT MAN END $<; - - export V -endif -endif - all: html man html: $(DOC_HTML)
Makefile +0 -33
@@ -2001,39 +2001,6 @@ ifndef PAGER_ENV PAGER_ENV = LESS=FRX LV=-c endif -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring w,$(MAKEFLAGS)),w) -PRINT_DIR = --no-print-directory -else # "make -w" -NO_SUBDIR = : -endif - -ifneq ($(findstring s,$(MAKEFLAGS)),s) -ifndef V - QUIET_CC = @echo ' ' CC $@; - QUIET_AR = @echo ' ' AR $@; - QUIET_LINK = @echo ' ' LINK $@; - QUIET_BUILT_IN = @echo ' ' BUILTIN $@; - QUIET_GEN = @echo ' ' GEN $@; - QUIET_LNCP = @echo ' ' LN/CP $@; - QUIET_XGETTEXT = @echo ' ' XGETTEXT $@; - QUIET_MSGFMT = @echo ' ' MSGFMT $@; - QUIET_GCOV = @echo ' ' GCOV $@; - QUIET_SP = @echo ' ' SP $<; - QUIET_HDR = @echo ' ' HDR $(<:hcc=h); - QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir - export V - export QUIET_GEN - export QUIET_BUILT_IN -endif -endif - ifdef NO_INSTALL_HARDLINKS export NO_INSTALL_HARDLINKS endif
config.mak.uname +0 -1
@@ -727,7 +727,6 @@ vcxproj: git diff-index --cached --quiet HEAD -- # Make .vcxproj files and add them - unset QUIET_GEN QUIET_BUILT_IN; \ perl contrib/buildsystems/generate -g Vcxproj git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
contrib/scalar/Makefile +0 -14
@@ -8,20 +8,6 @@ include ../../config.mak.uname -include ../../config.mak.autogen -include ../../config.mak -QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 = - -ifneq ($(findstring s,$(MAKEFLAGS)),s) -ifndef V - QUIET_GEN = @echo ' ' GEN $@; - QUIET_SUBDIR0 = +@subdir= - QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \ - $(MAKE) $(PRINT_DIR) -C $$subdir -else - export V -endif -endif - TARGETS = scalar$(X) scalar.o GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
shared.mak +53 -0
56 lines not on this page · raw message
templates/Makefile +0 -5
@@ -2,11 +2,6 @@ include ../shared.mak # make and install sample templates - -ifndef V - QUIET = @ -endif - INSTALL ?= install TAR ?= tar RM ?= rm -f
Signature · 1 line
2.35.1.1230.ga6e6579e98c
PATCH v5 8/8 Makefiles: add and use wildcard "mkdir -p" template
In reply to Ævar Arnfjörð BjarmasonAdd a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).
As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.
I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.
But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):
$ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
Time (mean ± σ): 2.914 s ± 0.062 s [User: 2.449 s, System: 0.489 s]
Range (min … max): 2.834 s … 3.020 s 10 runs Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s]
Range (min … max): 2.229 s … 2.397 s 10 runs
Summary
'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'
So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.
This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://gitlab.com/avar/git-hyperfine/
--- Documentation/Makefile | 25 +++---------------------- Makefile | 12 +++++++----- shared.mak | 17 +++++++++++++++++ 3 files changed, 27 insertions(+), 27 deletions(-)
Documentation/Makefile +3 -22
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo print-man1: @for i in $(MAN1_TXT); do echo $$i; done -## Lint: Common -.build: - $(QUIET)mkdir $@ -.build/lint-docs: | .build - $(QUIET)mkdir $@ - ## Lint: gitlink -.build/lint-docs/gitlink: | .build/lint-docs - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT)) -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config $(LINT_DOCS_GITLINK): lint-gitlink.perl $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \ $< \ $(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt lint-docs-gitlink: $(LINT_DOCS_GITLINK) ## Lint: man-end-blurb -.build/lint-docs/man-end-blurb: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@ .PHONY: lint-docs-man-end-blurb -lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB) ## Lint: man-section-order -.build/lint-docs/man-section-order: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@ .PHONY: lint-docs-man-section-order lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
Makefile +7 -5
@@ -2745,7 +2745,8 @@ all:: $(MOFILES) endif po/build/locale/%/LC_MESSAGES/git.mo: po/%.po - $(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $< + $(call mkdir_p_parent_template) + $(QUIET_MSGFMT)$(MSGFMT) -o $@ $< LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm) LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2761,15 +2762,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS)) endif perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES - $(QUIET_GEN)mkdir -p $(dir $@) && \ + $(call mkdir_p_parent_template) + $(QUIET_GEN) \ sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \ -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \ -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \ < $< > $@ perl/build/man/man3/Git.3pm: perl/Git.pm - $(QUIET_GEN)mkdir -p $(dir $@) && \ - pod2man $< $@ + $(call mkdir_p_parent_template) + $(QUIET_GEN)pod2man $< $@ $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \
@@ -2903,7 +2905,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $( all:: $(TEST_PROGRAMS) $(test_bindir_programs) bin-wrappers/%: wrap-for-bin.sh - @mkdir -p bin-wrappers + $(call mkdir_p_parent_template) $(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \ -e 's|@@BUILD_DIR@@|$(shell pwd)|' \ -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
shared.mak +17 -0
@@ -53,6 +53,8 @@ ifndef V QUIET = @ QUIET_GEN = @echo ' ' GEN $@; + QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D); + ## Used in "Makefile" QUIET_CC = @echo ' ' CC $@; QUIET_AR = @echo ' ' AR $@;
@@ -84,3 +86,18 @@ ifndef V export V endif endif + +### Templates + +## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@ +## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not +## needed. +## +## Is racy, but in a good way; we might redundantly (and safely) +## "mkdir -p" when running in parallel, but won't need to exhaustively create +## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule. +define mkdir_p_parent_template +$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D))) +endef
Signature · 1 line
2.35.1.1230.ga6e6579e98c