Skip to contentSkip to conversation

Makefile: optimize noop runs, add shared.mak

PATCH Superseded Ævar Arnfjörð Bjarmason 6 participants 67 messages Dec 24, 2021 – Apr 11, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover Makefile: optimize noop runs, add shared.mak

As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/

This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).

This v2 re-roll addresses trivial commit message/comment grammar/typo issues pointed out by Eric Sunshine, thanks Eric!

Ævar Arnfjörð Bjarmason (8):
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" template

 Documentation/Makefile |  63 +++-------------------
 Makefile               | 118 +++++++++++++----------------------------
 config.mak.uname       |   1 -
 shared.mak             | 109 +++++++++++++++++++++++++++++++++++++
 t/Makefile             |   3 ++
 t/interop/Makefile     |   3 ++
 templates/Makefile     |   8 ++-
 7 files changed, 160 insertions(+), 145 deletions(-)
 create mode 100644 shared.mak

Range-diff against v1:

1:  f74b47662b7 = 1:  b652fc78fda Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
2:  b0c63abe091 = 2:  b0c9be581a6 Makefile: disable GNU make built-in wildcard rules
3:  c6c6f7cf8d8 = 3:  ed6fd1c0bd1 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
4:  ed64cd1bd4a = 4:  4c6d8089fff Makefile: move ".SUFFIXES" rule to shared.mak
5:  1749085b929 = 5:  f1f02c71dbc Makefile: move $(comma), $(empty) and $(space) to shared.mak
6:  c25284b24cf = 6:  fb877060d6b Makefile: add "$(QUIET)" boilerplate to shared.mak
7:  3daef7672be = 7:  90d804ea9a0 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
8:  aca560ca410 ! 8:  59c1b7032db Makefiles: add and use wildcard "mkdir -p" template

@@ Commit message

But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run
- of a performance test similar to thatnoted downthread of [1] in [2]
+ of a performance test similar to that noted downthread of [1] in [2]
shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):

@@ shared.mak: ifndef V
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
-+## "mkdir -p" when running in parallel, but won't need to exhaustively ++## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule.

Signature · 1 line
2.34.1.1215.g6e154b84c77
#1 lore raw

PATCH v2 1/8 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.

See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.

This does have the potential downside that if e.g. templates/Makefile would like to include this "shared.mak" in the future the semantics of such a Makefile will change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.

This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

---
 Documentation/Makefile |  6 +++---
 Makefile               | 13 +++----------
 shared.mak             |  9 +++++++++
 t/Makefile             |  3 +++
 t/interop/Makefile     |  3 +++
 templates/Makefile     |  3 +++
 6 files changed, 24 insertions(+), 13 deletions(-)
 create mode 100644 shared.mak
Documentation/Makefile +3 -3
index ed656db2ae9..ba27456c86a 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # Guard against environment variables
 MAN1_TXT =
 MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n::
 	$(MAKE) -C po $@
 endif
 
-# Delete the target file on error
-.DELETE_ON_ERROR:
-
 .PHONY: FORCE
Makefile +3 -10
index 75ed168adbc..7dddea3caf3 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
 # The default target of this Makefile is...
 all::
 
@@ -2169,16 +2172,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
 strip: $(PROGRAMS) git$X
 	$(STRIP) $(STRIP_OPTS) $^
 
-### Flags affecting all rules
-
-# A GNU make extension since gmake 3.72 (released in late 1994) to
-# remove the target of rules if commands in those rules fail. The
-# default is to only do that if make itself receives a signal. Affects
-# all targets, see:
-#
-#    info make --index-search=.DELETE_ON_ERROR
-.DELETE_ON_ERROR:
-
 ### Target-specific flags and dependencies
 
 # The generic compilation pattern rule and automatically
shared.mak new file +9 -0
new file mode 100644
index 00000000000..0170bb397ae
@@ -0,0 +1,9 @@
+### Flags affecting all rules
+
+# A GNU make extension since gmake 3.72 (released in late 1994) to
+# remove the target of rules if commands in those rules fail. The
+# default is to only do that if make itself receives a signal. Affects
+# all targets, see:
+#
+#    info make --index-search=.DELETE_ON_ERROR
+.DELETE_ON_ERROR:
t/Makefile +3 -0
index 46cd5fc5273..056ce55dcc9 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # Run tests
 #
 # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
index 31a4bbc716a..6911c2915a7 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 -include ../../config.mak
 export GIT_TEST_OPTIONS
 
templates/Makefile +3 -0
index d22a71a3999..636cee52f51 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # make and install sample templates
 
 ifndef V
Signature · 1 line
2.34.1.1215.g6e154b84c77
#2 lore raw

PATCH v2 2/8 Makefile: disable GNU make built-in wildcard rules

In reply to Ævar Arnfjörð Bjarmason

Override built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.

The speed-up may wary. I've seen 1-10% depending on the speed of the local disk, caches, -jN etc. Running:

strace -f -c -S calls make -j1 NO_TCLTK=Y

Shows that we reduce the number of syscalls we make, mostly in "stat" calls.

We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early. Doing so might make us a bit faster still. But doing so is a much bigger hammer, since it will disable all built-in rules, some (all?) of which can be seen with:

    make -f/dev/null -p | grep -v -e ^# -e ^$

We may have something that relies on them, so let's go for the more isolated optimization here that gives us most or all of the wins.

1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html

---
 shared.mak | 11 +++++++++++
 1 file changed, 11 insertions(+)
shared.mak +11 -0
index 0170bb397ae..29f0e69ecb9 100644
@@ -1,3 +1,14 @@
+### Remove GNU make implicit rules
+
+## This speeds things up since we don't need to look for and stat() a
+## "foo.c,v" every time a rule referring to "foo.c" is in play. See
+## "make -p -f/dev/null | grep ^%::'".
+%:: %,v
+%:: RCS/%,v
+%:: RCS/%
+%:: s.%
+%:: SCCS/s.%
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.34.1.1215.g6e154b84c77
#3 lore raw

PATCH v2 3/8 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)

In reply to Ævar Arnfjörð Bjarmason

Combine the definitions of $(FIND_SOURCE_FILES) and $(LIB_H) to speed up the Makefile, as these are the two main expensive $(shell) commands that we execute unconditionally.

When see what was in $(FOUND_SOURCE_FILES) that wasn't in $(LIB_H) via the ad-hoc test of:

    $(error $(filter-out $(LIB_H),$(filter %.h,$(ALL_SOURCE_FILES))))
    $(error $(filter-out $(ALL_SOURCE_FILES),$(filter %.h,$(LIB_H))))

We'll get, respectively:

    Makefile:850: *** t/helper/test-tool.h.  Stop.
    Makefile:850: *** .  Stop.

I.e. we only had a discrepancy when it came to t/helper/test-tool.h. In terms of correctness this was broken before, but now works:

    $ make t/helper/test-tool.hco
        HDR t/helper/test-tool.h

This speeds things up a lot:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10
    Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1
      Time (mean ± σ):     159.9 ms ±   6.8 ms    [User: 137.2 ms, System: 28.0 ms]
      Range (min … max):   154.6 ms … 175.9 ms    10 runs
    Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0
      Time (mean ± σ):     100.0 ms ±   1.3 ms    [User: 84.2 ms, System: 20.2 ms]
      Range (min … max):    98.8 ms … 102.8 ms    10 runs

    Summary
      'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran
        1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1'
---
 Makefile | 54 ++++++++++++++++++++++++++----------------------------
 1 file changed, 26 insertions(+), 28 deletions(-)
Makefile +26 -28
index 7dddea3caf3..d3ce0d22199 100644
@@ -826,12 +826,33 @@ GENERATED_H += hook-list.h
 .PHONY: generated-hdrs
 generated-hdrs: $(GENERATED_H)
 
-LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \
+## Exhaustive lists of our source files, either dynamically generated,
+## or hardcoded.
+SOURCES_CMD = ( \
+	git ls-files \
+		'*.[hcS]' \
+		'*.sh' \
+		':!*[tp][0-9][0-9][0-9][0-9]*' \
+		':!contrib' \
+		2>/dev/null || \
 	$(FIND) . \
-	-name .git -prune -o \
-	-name t -prune -o \
-	-name Documentation -prune -o \
-	-name '*.h' -print)))
+		\( -name .git -type d -prune \) \
+		-o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
+		-o \( -name contrib -type d -prune \) \
+		-o \( -name build -type d -prune \) \
+		-o \( -name 'trash*' -type d -prune \) \
+		-o \( -name '*.[hcS]' -type f -print \) \
+		-o \( -name '*.sh' -type f -print \) \
+		| sed -e 's|^\./||' \
+	)
+FOUND_SOURCE_FILES := $(shell $(SOURCES_CMD))
+
+FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))
+FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))
+
+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
+
+LIB_H = $(FOUND_H_SOURCES)
 
 LIB_OBJS += abspath.o
 LIB_OBJS += add-interactive.o
@@ -2758,26 +2779,6 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
 	$(QUIET_GEN)mkdir -p $(dir $@) && \
 	pod2man $< $@
 
-FIND_SOURCE_FILES = ( \
-	git ls-files \
-		'*.[hcS]' \
-		'*.sh' \
-		':!*[tp][0-9][0-9][0-9][0-9]*' \
-		':!contrib' \
-		2>/dev/null || \
-	$(FIND) . \
-		\( -name .git -type d -prune \) \
-		-o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
-		-o \( -name contrib -type d -prune \) \
-		-o \( -name build -type d -prune \) \
-		-o \( -name 'trash*' -type d -prune \) \
-		-o \( -name '*.[hcS]' -type f -print \) \
-		-o \( -name '*.sh' -type f -print \) \
-		| sed -e 's|^\./||' \
-	)
-
-FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES))
-
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
 	echo $(FOUND_SOURCE_FILES) | xargs etags -a -o $@+ && \
@@ -2987,9 +2988,6 @@ check: $(GENERATED_H)
 		exit 1; \
 	fi
 
-FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))
-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
-
 %.cocci.patch: %.cocci $(COCCI_SOURCES)
 	$(QUIET_SPATCH) \
 	if test $(SPATCH_BATCH_SIZE) = 0; then \
Signature · 1 line
2.34.1.1215.g6e154b84c77
#4 lore raw

PATCH v2 4/8 Makefile: move ".SUFFIXES" rule to shared.mak

In reply to Ævar Arnfjörð Bjarmason

This was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.

This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
    Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
      Time (mean ± σ):     121.7 ms ±   8.8 ms    [User: 105.8 ms, System: 18.6 ms]
      Range (min … max):   112.8 ms … 148.4 ms    26 runs

Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs

Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

---
 Makefile   | 2 --
 shared.mak | 5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
index d3ce0d22199..e30093b83d3 100644
@@ -2549,8 +2549,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S))
 ASM_OBJ := $(ASM_SRC:S=o)
 C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS))
 
-.SUFFIXES:
-
 $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
 	$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
 $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +5 -0
index 29f0e69ecb9..155ac84f867 100644
@@ -9,6 +9,11 @@
 %:: s.%
 %:: SCCS/s.%
 
+## Likewise delete default $(SUFFIXES). See:
+##
+##     info make --index-search=.DELETE_ON_ERROR
+.SUFFIXES:
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.34.1.1215.g6e154b84c77
#5 lore raw

Re: [PATCH v2 4/8] Makefile: move ".SUFFIXES" rule to shared.mak

Quoted 5 lines · Ævar Arnfjörð Bjarmason
    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
    Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
      Time (mean ± σ):     121.7 ms ±   8.8 ms    [User: 105.8 ms, System: 18.6 ms]
      Range (min … max):   112.8 ms … 148.4 ms    26 runs
Quoted 7 lines

Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs

Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

Nice speed-up. Though I am not sure I totally understand where it comes from ;). Reading 30248886ce8 and the documentation on .SUFFIXES from [1], I am still unclear. I guess removing the obsolete built-in suffix rules gives make less work to do in general?

So long as we're not depending on any of these, this seems like a nice little boost to me.

shared.mak line 15

Quoted 12 lines
shared.mak +4 -0
index 29f0e69ecb9..155ac84f867 100644
@@ -9,6 +9,11 @@
 %:: s.%
 %:: SCCS/s.%
 
+## Likewise delete default $(SUFFIXES). See:
+##
+##     info make --index-search=.DELETE_ON_ERROR
+.SUFFIXES:

Hmm. s/DELETE_ON_ERROR/SUFFIXES? Or perhaps I'm holding this whole thing incorrectly:

~/s/git [nand] (ab/make-noop) $ info make --index-search=.DELETE_ON_ERROR no index entries found for '.DELETE_ON_ERROR'
~/s/git [nand] (ab/make-noop) $ info make --index-search=.SUFFIXES no index entries found for '.SUFFIXES'

Thanks,
Taylor

[1]: https://www.gnu.org/software/make/manual/html_node/Suffix-Rules.html

#6 lore raw

Re: [PATCH v2 4/8] Makefile: move ".SUFFIXES" rule to shared.mak

Quoted 10 lines · Taylor Blau
Quoted 9 lines · Ævar Arnfjörð Bjarmason
    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
    Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
      Time (mean ± σ):     121.7 ms ±   8.8 ms    [User: 105.8 ms, System: 18.6 ms]
      Range (min … max):   112.8 ms … 148.4 ms    26 runs
    Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
      Time (mean ± σ):      97.5 ms ±   8.0 ms    [User: 80.1 ms, System: 20.1 ms]
      Range (min … max):    89.8 ms … 111.8 ms    32 runs
Quoted 8 lines
Quoted 3 lines

Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

Nice speed-up. Though I am not sure I totally understand where it comes from ;). Reading 30248886ce8 and the documentation on .SUFFIXES from [1], I am still unclear. I guess removing the obsolete built-in suffix rules gives make less work to do in general?

I'll update the commit message, but basically the same applies as for 2/8 here (<[email protected]>), or if you run "make" with "--debug=a". I.e. if before/after this change you do:

    git clean -dxf; make -C Documentation/ --debug=a git-status.1 >/tmp/b 2>&1
    git clean -dxf; make -C Documentation/ --debug=a git-status.1 >/tmp/a 2>&1

You'll get:

    $ wc -l /tmp/[ba]
       6515 /tmp/a
     144051 /tmp/b
     150566 total

Which e.g. for "git-status.txt" starts with (I cut a lot out, there's way more than this just for that file):

Considering target file 'git-status.txt'.
Looking for an implicit rule for 'git-status.txt'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.o'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.c'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.cc'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.C'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.cpp'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.p'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.f'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.F'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.m'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.r'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.s'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.S'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.mod'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.sh'.
- Trying pattern rule with stem 'git-status.txt'.
- Trying implicit prerequisite 'git-status.txt.o'.
[...]

I.e. given a foo.txt "make" by default will exhaustively consider foo foo.txt.c, foo.txt.cpp etc. etc.

This is all ultimately a part of make's implicit rule mechanism. I.e. even if you have no Makefile at all you can run "make main" if you have a main.c in your directory, and it'll discover it and compile it with implicit rules, unless you explicitl disable them, e.g. with "-r":

    $ rm Makefile hello; make hello
    rm: cannot remove 'Makefile': No such file or directory
    rm: cannot remove 'hello': No such file or directory
    cc   hello.o   -o hello
    $ rm Makefile hello; make -r hello
    rm: cannot remove 'Makefile': No such file or directory
    make: *** No rule to make target 'hello'.  Stop.
Quoted 2 lines

So long as we're not depending on any of these, this seems like a nice little boost to me.

Yes, definitely!

Quoted 16 lines

shared.mak line 15

Quoted 12 lines
shared.mak +4 -0
index 29f0e69ecb9..155ac84f867 100644
@@ -9,6 +9,11 @@
 %:: s.%
 %:: SCCS/s.%
 
+## Likewise delete default $(SUFFIXES). See:
+##
+##     info make --index-search=.DELETE_ON_ERROR
+.SUFFIXES:

Hmm. s/DELETE_ON_ERROR/SUFFIXES? Or perhaps I'm holding this whole thing incorrectly:

Quoted 4 lines

~/s/git [nand] (ab/make-noop) $ info make --index-search=.DELETE_ON_ERROR no index entries found for '.DELETE_ON_ERROR'
~/s/git [nand] (ab/make-noop) $ info make --index-search=.SUFFIXES no index entries found for '.SUFFIXES'

Yes that's a copy/paste error, it should be .SUFFIXES.

But both commands should work, or emit an error like:

    $ info doesnotexist --index-search=.DELETE_ON_ERROR
    info: No menu item 'doesnotexist' in node '(dir)Top'

A broken OS package? Self-built "make"?

In any case I could also link to
https://www.gnu.org/software/make/manual/html_node/Suffix-Rules.html;
which is the same information online (but may not match your local "make" version)>

#7 lore raw

PATCH v2 5/8 Makefile: move $(comma), $(empty) and $(space) to shared.mak

In reply to Ævar Arnfjörð Bjarmason

Move these variables over to the shared.max, we'll make use of them in a subsequent commit. There was no reason for these to be "simply expanded variables", so let's use the normal lazy "=" assignment here.

See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.

---
 Makefile   | 4 ----
 shared.mak | 8 ++++++++
 2 files changed, 8 insertions(+), 4 deletions(-)
Makefile +0 -4
index e30093b83d3..85f99e66b07 100644
@@ -1279,10 +1279,6 @@ endif
 ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS)
 ALL_LDFLAGS = $(LDFLAGS)
 
-comma := ,
-empty :=
-space := $(empty) $(empty)
-
 ifdef SANITIZE
 SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag))
 BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)
shared.mak +8 -0
index 155ac84f867..b34fb948c0f 100644
@@ -23,3 +23,11 @@
 #
 #    info make --index-search=.DELETE_ON_ERROR
 .DELETE_ON_ERROR:
+
+### Global variables
+
+## comma, empty, space: handy variables as these tokens are either
+## special or can be hard to spot among other Makefile syntax.
+comma = ,
+empty =
+space = $(empty) $(empty)
Signature · 1 line
2.34.1.1215.g6e154b84c77
#8 lore raw

PATCH v2 6/8 Makefile: add "$(QUIET)" boilerplate to shared.mak

In reply to Ævar Arnfjörð Bjarmason

The $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.

Since we're not using the environment to pass these around we don't need to export the "QUIET_GEN" and "QUIET_BUILT_IN" variables anymore. The "QUIET_GEN" variable is used in "git-gui/Makefile" and "gitweb/Makefile", but they've got their own definition for those. The "QUIET_BUILT_IN" variable is only used in the top-level "Makefile". We still need to export the "V" variable.

---
 Documentation/Makefile | 32 -------------------------
 Makefile               | 33 --------------------------
 config.mak.uname       |  1 -
 shared.mak             | 53 ++++++++++++++++++++++++++++++++++++++++++
 templates/Makefile     |  5 ----
 5 files changed, 53 insertions(+), 71 deletions(-)
Documentation/Makefile +0 -32
index ba27456c86a..0f4ebdeda8a 100644
@@ -218,38 +218,6 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR))
 ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)'
 endif
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring $(MAKEFLAGS),w),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring $(MAKEFLAGS),s),s)
-ifndef V
-	QUIET		= @
-	QUIET_ASCIIDOC	= @echo '   ' ASCIIDOC $@;
-	QUIET_XMLTO	= @echo '   ' XMLTO $@;
-	QUIET_DB2TEXI	= @echo '   ' DB2TEXI $@;
-	QUIET_MAKEINFO	= @echo '   ' MAKEINFO $@;
-	QUIET_DBLATEX	= @echo '   ' DBLATEX $@;
-	QUIET_XSLTPROC	= @echo '   ' XSLTPROC $@;
-	QUIET_GEN	= @echo '   ' GEN $@;
-	QUIET_STDERR	= 2> /dev/null
-	QUIET_SUBDIR0	= +@subdir=
-	QUIET_SUBDIR1	= ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			  $(MAKE) $(PRINT_DIR) -C $$subdir
-
-	QUIET_LINT_GITLINK	= @echo '   ' LINT GITLINK $<;
-	QUIET_LINT_MANSEC	= @echo '   ' LINT MAN SEC $<;
-	QUIET_LINT_MANEND	= @echo '   ' LINT MAN END $<;
-
-	export V
-endif
-endif
-
 all: html man
 
 html: $(DOC_HTML)
Makefile +0 -33
index 85f99e66b07..d8051278519 100644
@@ -1971,39 +1971,6 @@ ifndef PAGER_ENV
 PAGER_ENV = LESS=FRX LV=-c
 endif
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring w,$(MAKEFLAGS)),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring s,$(MAKEFLAGS)),s)
-ifndef V
-	QUIET_CC       = @echo '   ' CC $@;
-	QUIET_AR       = @echo '   ' AR $@;
-	QUIET_LINK     = @echo '   ' LINK $@;
-	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
-	QUIET_GEN      = @echo '   ' GEN $@;
-	QUIET_LNCP     = @echo '   ' LN/CP $@;
-	QUIET_XGETTEXT = @echo '   ' XGETTEXT $@;
-	QUIET_MSGFMT   = @echo '   ' MSGFMT $@;
-	QUIET_GCOV     = @echo '   ' GCOV $@;
-	QUIET_SP       = @echo '   ' SP $<;
-	QUIET_HDR      = @echo '   ' HDR $(<:hcc=h);
-	QUIET_RC       = @echo '   ' RC $@;
-	QUIET_SPATCH   = @echo '   ' SPATCH $<;
-	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			 $(MAKE) $(PRINT_DIR) -C $$subdir
-	export V
-	export QUIET_GEN
-	export QUIET_BUILT_IN
-endif
-endif
-
 ifdef NO_INSTALL_HARDLINKS
 	export NO_INSTALL_HARDLINKS
 endif
config.mak.uname +0 -1
index a3a779327f8..cfc2416c935 100644
@@ -720,7 +720,6 @@ vcxproj:
 	git diff-index --cached --quiet HEAD --
 
 	# Make .vcxproj files and add them
-	unset QUIET_GEN QUIET_BUILT_IN; \
 	perl contrib/buildsystems/generate -g Vcxproj
 	git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
 
shared.mak +53 -0

56 lines not on this page · raw message

templates/Makefile +0 -5
index 636cee52f51..367ad00c24c 100644
@@ -2,11 +2,6 @@
 include ../shared.mak
 
 # make and install sample templates
-
-ifndef V
-	QUIET = @
-endif
-
 INSTALL ?= install
 TAR ?= tar
 RM ?= rm -f
Signature · 1 line
2.34.1.1215.g6e154b84c77
#9 lore raw

PATCH v2 7/8 Makefile: use $(wspfx) for $(QUIET...) in shared.mak

In reply to Ævar Arnfjörð Bjarmason

Change the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding, to guarantee that it's consistent with the "TRACK_template" template.

    $ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
     -> GIT-CFLAGS PARAMETERS (changed)
     -> CC grep.o
---
 shared.mak | 54 ++++++++++++++++++++++++++++++------------------------
 1 file changed, 30 insertions(+), 24 deletions(-)
shared.mak +30 -24
index d6f70f3d6c9..f8ebaf30569 100644
@@ -32,6 +32,12 @@ comma = ,
 empty =
 space = $(empty) $(empty)
 
+## wspfx: the whitespace prefix padding for $(QUIET...) and similarly
+## aligned output.
+wspfx = $(space)$(space)$(space)
+wspfx_SQ = '$(subst ','\'',$(wspfx))'
+# ' closing quote to appease Emacs make-mode.elxo
+
 ### Quieting
 ## common
 QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
@@ -47,39 +53,39 @@ ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
 ## common
 	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo $(wspfx_SQ) SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
 
 	QUIET          = @
-	QUIET_GEN      = @echo '   ' GEN $@;
+	QUIET_GEN      = @echo $(wspfx_SQ) GEN $@;
 
 ## Used in "Makefile"
-	QUIET_CC       = @echo '   ' CC $@;
-	QUIET_AR       = @echo '   ' AR $@;
-	QUIET_LINK     = @echo '   ' LINK $@;
-	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
-	QUIET_LNCP     = @echo '   ' LN/CP $@;
-	QUIET_XGETTEXT = @echo '   ' XGETTEXT $@;
-	QUIET_MSGFMT   = @echo '   ' MSGFMT $@;
-	QUIET_GCOV     = @echo '   ' GCOV $@;
-	QUIET_SP       = @echo '   ' SP $<;
-	QUIET_HDR      = @echo '   ' HDR $(<:hcc=h);
-	QUIET_RC       = @echo '   ' RC $@;
-	QUIET_SPATCH   = @echo '   ' SPATCH $<;
+	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
+	QUIET_AR       = @echo $(wspfx_SQ) AR $@;
+	QUIET_LINK     = @echo $(wspfx_SQ) LINK $@;
+	QUIET_BUILT_IN = @echo $(wspfx_SQ) BUILTIN $@;
+	QUIET_LNCP     = @echo $(wspfx_SQ) LN/CP $@;
+	QUIET_XGETTEXT = @echo $(wspfx_SQ) XGETTEXT $@;
+	QUIET_MSGFMT   = @echo $(wspfx_SQ) MSGFMT $@;
+	QUIET_GCOV     = @echo $(wspfx_SQ) GCOV $@;
+	QUIET_SP       = @echo $(wspfx_SQ) SP $<;
+	QUIET_HDR      = @echo $(wspfx_SQ) HDR $(<:hcc=h);
+	QUIET_RC       = @echo $(wspfx_SQ) RC $@;
+	QUIET_SPATCH   = @echo $(wspfx_SQ) SPATCH $<;
 
 ## Used in "Documentation/Makefile"
-	QUIET_ASCIIDOC	= @echo '   ' ASCIIDOC $@;
-	QUIET_XMLTO	= @echo '   ' XMLTO $@;
-	QUIET_DB2TEXI	= @echo '   ' DB2TEXI $@;
-	QUIET_MAKEINFO	= @echo '   ' MAKEINFO $@;
-	QUIET_DBLATEX	= @echo '   ' DBLATEX $@;
-	QUIET_XSLTPROC	= @echo '   ' XSLTPROC $@;
-	QUIET_GEN	= @echo '   ' GEN $@;
+	QUIET_ASCIIDOC	= @echo $(wspfx_SQ) ASCIIDOC $@;
+	QUIET_XMLTO	= @echo $(wspfx_SQ) XMLTO $@;
+	QUIET_DB2TEXI	= @echo $(wspfx_SQ) DB2TEXI $@;
+	QUIET_MAKEINFO	= @echo $(wspfx_SQ) MAKEINFO $@;
+	QUIET_DBLATEX	= @echo $(wspfx_SQ) DBLATEX $@;
+	QUIET_XSLTPROC	= @echo $(wspfx_SQ) XSLTPROC $@;
+	QUIET_GEN	= @echo $(wspfx_SQ) GEN $@;
 	QUIET_STDERR	= 2> /dev/null
 
-	QUIET_LINT_GITLINK	= @echo '   ' LINT GITLINK $<;
-	QUIET_LINT_MANSEC	= @echo '   ' LINT MAN SEC $<;
-	QUIET_LINT_MANEND	= @echo '   ' LINT MAN END $<;
+	QUIET_LINT_GITLINK	= @echo $(wspfx_SQ) LINT GITLINK $<;
+	QUIET_LINT_MANSEC	= @echo $(wspfx_SQ) LINT MAN SEC $<;
+	QUIET_LINT_MANEND	= @echo $(wspfx_SQ) LINT MAN END $<;
 
 	export V
 endif
Signature · 1 line
2.34.1.1215.g6e154b84c77
#10 lore raw

PATCH v2 8/8 Makefiles: add and use wildcard "mkdir -p" template

In reply to Ævar Arnfjörð Bjarmason

Add a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).

As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.

I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.

But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
    Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
      Time (mean ± σ):      2.914 s ±  0.062 s    [User: 2.449 s, System: 0.489 s]
      Range (min … max):    2.834 s …  3.020 s    10 runs

Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s]
Range (min … max): 2.229 s … 2.397 s 10 runs

Summary
'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'

So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.

This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://gitlab.com/avar/git-hyperfine/

---
 Documentation/Makefile | 25 +++----------------------
 Makefile               | 12 +++++++-----
 shared.mak             | 17 +++++++++++++++++
 3 files changed, 27 insertions(+), 27 deletions(-)
Documentation/Makefile +3 -22
index 0f4ebdeda8a..1eb9192dae8 100644
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo
 print-man1:
 	@for i in $(MAN1_TXT); do echo $$i; done
 
-## Lint: Common
-.build:
-	$(QUIET)mkdir $@
-.build/lint-docs: | .build
-	$(QUIET)mkdir $@
-
 ## Lint: gitlink
-.build/lint-docs/gitlink: | .build/lint-docs
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
 LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT))
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config
 $(LINT_DOCS_GITLINK): lint-gitlink.perl
 $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \
 		$< \
 		$(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
 lint-docs-gitlink: $(LINT_DOCS_GITLINK)
 
 ## Lint: man-end-blurb
-.build/lint-docs/man-end-blurb: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb
 $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl
 $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@
 .PHONY: lint-docs-man-end-blurb
-lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB)
 
 ## Lint: man-section-order
-.build/lint-docs/man-section-order: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order
 $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl
 $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@
 .PHONY: lint-docs-man-section-order
 lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
Makefile +7 -5
index d8051278519..5474c27f2b1 100644
@@ -2714,7 +2714,8 @@ all:: $(MOFILES)
 endif
 
 po/build/locale/%/LC_MESSAGES/git.mo: po/%.po
-	$(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $<
+	$(call mkdir_p_parent_template)
+	$(QUIET_MSGFMT)$(MSGFMT) -o $@ $<
 
 LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm)
 LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2730,15 +2731,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS))
 endif
 
 perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN) \
 	sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \
 	    -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \
 	    -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \
 	< $< > $@
 
 perl/build/man/man3/Git.3pm: perl/Git.pm
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
-	pod2man $< $@
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN)pod2man $< $@
 
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
@@ -2872,7 +2874,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(
 all:: $(TEST_PROGRAMS) $(test_bindir_programs)
 
 bin-wrappers/%: wrap-for-bin.sh
-	@mkdir -p bin-wrappers
+	$(call mkdir_p_parent_template)
 	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
 	     -e 's|@@BUILD_DIR@@|$(shell pwd)|' \
 	     -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
shared.mak +17 -0
index f8ebaf30569..7787f81856c 100644
@@ -59,6 +59,8 @@ ifndef V
 	QUIET          = @
 	QUIET_GEN      = @echo $(wspfx_SQ) GEN $@;
 
+	QUIET_MKDIR_P_PARENT  = @echo $(wspfx_SQ) MKDIR -p $(@D);
+
 ## Used in "Makefile"
 	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
 	QUIET_AR       = @echo $(wspfx_SQ) AR $@;
@@ -90,3 +92,18 @@ ifndef V
 	export V
 endif
 endif
+
+### Templates
+
+## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@
+## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
+## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a
+## "a/prefix/dir/file". This can instead be inserted at the start of
+## the "a/prefix/dir/file" rule.
+define mkdir_p_parent_template
+$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D)))
+endef
Signature · 1 line
2.34.1.1215.g6e154b84c77
#11 lore raw
In reply to Ævar Arnfjörð Bjarmason

On Fri, Dec 24 2021, Ævar Arnfjörð Bjarmason wrote:

A gentle reminder about considering picking up this topic that got lost around the holidays.

This one is independent of the other Makefile topic I've got outstanding at
https://lore.kernel.org/git/[email protected]/

This thread doesn't have any reviews, but I see that's because I screwed up and omitted the In-Reply-To at the time. There was feeback on the v1 of this:
https://lore.kernel.org/git/[email protected]/

That v1 was in turn split off from a larger series, and as the range-diff for that v1 shows the split-off version was was mostly unchanged from that version, which had a few eyeballs on it.

Quoted 58 lines

As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/

This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).

This v2 re-roll addresses trivial commit message/comment grammar/typo issues pointed out by Eric Sunshine, thanks Eric!

Ævar Arnfjörð Bjarmason (8):
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" template

 Documentation/Makefile |  63 +++-------------------
 Makefile               | 118 +++++++++++++----------------------------
 config.mak.uname       |   1 -
 shared.mak             | 109 +++++++++++++++++++++++++++++++++++++
 t/Makefile             |   3 ++
 t/interop/Makefile     |   3 ++
 templates/Makefile     |   8 ++-
 7 files changed, 160 insertions(+), 145 deletions(-)
 create mode 100644 shared.mak

Range-diff against v1:

1:  f74b47662b7 = 1:  b652fc78fda Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
2:  b0c63abe091 = 2:  b0c9be581a6 Makefile: disable GNU make built-in wildcard rules
3:  c6c6f7cf8d8 = 3:  ed6fd1c0bd1 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
4:  ed64cd1bd4a = 4:  4c6d8089fff Makefile: move ".SUFFIXES" rule to shared.mak
5:  1749085b929 = 5:  f1f02c71dbc Makefile: move $(comma), $(empty) and $(space) to shared.mak
6:  c25284b24cf = 6:  fb877060d6b Makefile: add "$(QUIET)" boilerplate to shared.mak
7:  3daef7672be = 7:  90d804ea9a0 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
8:  aca560ca410 ! 8:  59c1b7032db Makefiles: add and use wildcard "mkdir -p" template

@@ Commit message

But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run
- of a performance test similar to thatnoted downthread of [1] in [2]
+ of a performance test similar to that noted downthread of [1] in [2]
shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):

@@ shared.mak: ifndef V
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
-+## "mkdir -p" when running in parallel, but won't need to exhaustively ++## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule.

#12 lore raw

v3 cover Makefile: optimize noop runs, add shared.mak

In reply to Ævar Arnfjörð Bjarmason

As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/

This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).

This v3 re-roll:

* Addresses a minor comment/documentation issue pointed out by Taylor Blau.

* Adjusts the ".SUFFIXES" commit message to note the source of the speed-up from that change, in response to a question from Taylor.

* Changes the contrib/scalar/Makefile to make use of the new shared.mak, which allows for deleting some copy/pasted code in it in favor of the same shared logic.

The original version of these patches was written before that file landed in-tree.

For v2, see: https://lore.kernel.org/git/[email protected]/

Ævar Arnfjörð Bjarmason (9):
scalar Makefile: set the default target after the includes
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" template

 Documentation/Makefile    |  63 ++------------------
 Makefile                  | 118 ++++++++++++--------------------------
 config.mak.uname          |   1 -
 contrib/scalar/Makefile   |  19 ++----
 contrib/scalar/t/Makefile |   3 +
 shared.mak                | 109 +++++++++++++++++++++++++++++++++++
 t/Makefile                |   3 +
 t/interop/Makefile        |   3 +
 t/perf/Makefile           |   3 +
 templates/Makefile        |   8 +--
 10 files changed, 170 insertions(+), 160 deletions(-)
 create mode 100644 shared.mak

Range-diff against v2:

 -:  ----------- >  1:  2404c4d8b96 scalar Makefile: set the default target after the includes
 1:  97dccacce20 !  2:  96a490bec54 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

@@ Makefile: shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell

# The generic compilation pattern rule and automatically

+ ## contrib/scalar/Makefile ##
+@@
++# Import tree-wide shared Makefile behavior and libraries ++include ../../shared.mak
++
+ QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
+ QUIET_SUBDIR1 =
+
+
+ ## contrib/scalar/t/Makefile ##
+@@
++# Import tree-wide shared Makefile behavior and libraries ++include ../../../shared.mak
++
+ # Run scalar tests
+ #
+ # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin +
## shared.mak (new) ##
@@
+### Flags affecting all rules
@@ t/interop/Makefile
export GIT_TEST_OPTIONS

+ ## t/perf/Makefile ##
+@@
++# Import tree-wide shared Makefile behavior and libraries ++include ../../shared.mak
++
+ -include ../../config.mak
+ export GIT_TEST_OPTIONS
+
+
## templates/Makefile ##
@@
+# Import tree-wide shared Makefile behavior and libraries

 2:  b2bf32ab071 =  3:  9392e3c3e97 Makefile: disable GNU make built-in wildcard rules
 3:  275aba624fa =  4:  07cf9daa9d6 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
 4:  7103c40de64 !  5:  16f2e3ff35b Makefile: move ".SUFFIXES" rule to shared.mak

@@ Commit message
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

+ The reason for that can be seen when comparing that run with
+ "--debug=a". Without this change making a target like "git-status.1"
+ will cause "make" to consider not only "git-status.txt", but
+ "git-status.txt.o", as well as numerous other implicit suffixes such
+ as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after
+ example.
+
+ So this is causing us to omit a bunch of work we didn't need to
+ do. For making "git-status.1" the "--debug=a" output is reduced from
+ ~140k lines to ~6k.
+
+ 1. https://lore.kernel.org/git/[email protected]/
+

## Makefile ##
@@ shared.mak

+## Likewise delete default $(SUFFIXES). See:
+##
-+## info make --index-search=.DELETE_ON_ERROR
++## info make --index-search=.SUFFIXES
+.SUFFIXES:
+
### Flags affecting all rules

 5:  4a5c647b5e7 =  6:  1b6ecb27f02 Makefile: move $(comma), $(empty) and $(space) to shared.mak
 6:  940299d2a03 !  7:  471067deefc Makefile: add "$(QUIET)" boilerplate to shared.mak
    @@ config.mak.uname: vcxproj:
      	git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
      
     
    + ## contrib/scalar/Makefile ##
    +@@
    + # Import tree-wide shared Makefile behavior and libraries
    + include ../../shared.mak
    + 
    +-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
    +-QUIET_SUBDIR1  =
    +-
    +-ifneq ($(findstring s,$(MAKEFLAGS)),s)
    +-ifndef V
    +-	QUIET_GEN      = @echo '   ' GEN $@;
    +-	QUIET_SUBDIR0  = +@subdir=
    +-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
    +-			 $(MAKE) $(PRINT_DIR) -C $$subdir
    +-else
    +-	export V
    +-endif
    +-endif
    +-
    + include ../../config.mak.uname
    + -include ../../config.mak.autogen
    + -include ../../config.mak
    +
      ## shared.mak ##
     @@
      comma = ,
 7:  1e13fee526d =  8:  510306d2219 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
 8:  250b32540d9 =  9:  85bb74aa32f Makefiles: add and use wildcard "mkdir -p" template
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#13 lore raw

PATCH v3 1/9 scalar Makefile: set the default target after the includes

Make have the "contrib/scalar/Makefile" be stylistically consistent with the top-level "Makefile" in including other makefiles before setting the default target.

This adjusts code added in 0a43fb22026 (scalar: create a rudimentary executable, 2021-12-03), it's a style-only change, in a subsequent commit the "QUIET" boilerplate at the beginning of this file will be retrieved via an include, and having an "all:" between the two set of "include"'s after that change would look odd.

---
 contrib/scalar/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
contrib/scalar/Makefile +2 -2
index 231b1ee1796..5b12a437426 100644
@@ -12,12 +12,12 @@ else
 endif
 endif
 
-all:
-
 include ../../config.mak.uname
 -include ../../config.mak.autogen
 -include ../../config.mak
 
+all:
+
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#14 lore raw

Re: [PATCH v3 1/9] scalar Makefile: set the default target after the includes

Quoted 3 lines · Ævar Arnfjörð Bjarmason

Make have the "contrib/scalar/Makefile" be stylistically consistent with the top-level "Makefile" in including other makefiles before setting the default target.

I think that goal makes sense, but the patch looks like doing a different thing. The top-level "Makefile" does it correctly, by starting the file with

# The default target of this Makefile is... all::

that declares that saying "make" is equivalent to "make all" upfront.

The patch does not do that; it moves "all" even lower.

What am I missing?

Quoted 29 lines

This adjusts code added in 0a43fb22026 (scalar: create a rudimentary executable, 2021-12-03), it's a style-only change, in a subsequent commit the "QUIET" boilerplate at the beginning of this file will be retrieved via an include, and having an "all:" between the two set of "include"'s after that change would look odd.

---
 contrib/scalar/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
contrib/scalar/Makefile +2 -2
index 231b1ee1796..5b12a437426 100644
@@ -12,12 +12,12 @@ else
 endif
 endif
 
-all:
-
 include ../../config.mak.uname
 -include ../../config.mak.autogen
 -include ../../config.mak
 
+all:
+
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
#15 lore raw

PATCH v3 2/9 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

In reply to Ævar Arnfjörð Bjarmason

We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.

See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.

This does have the potential downside that if e.g. templates/Makefile would like to include this "shared.mak" in the future the semantics of such a Makefile will change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.

This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

---
 Documentation/Makefile    |  6 +++---
 Makefile                  | 13 +++----------
 contrib/scalar/Makefile   |  3 +++
 contrib/scalar/t/Makefile |  3 +++
 shared.mak                |  9 +++++++++
 t/Makefile                |  3 +++
 t/interop/Makefile        |  3 +++
 t/perf/Makefile           |  3 +++
 templates/Makefile        |  3 +++
 9 files changed, 33 insertions(+), 13 deletions(-)
 create mode 100644 shared.mak
Documentation/Makefile +3 -3
index ed656db2ae9..ba27456c86a 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # Guard against environment variables
 MAN1_TXT =
 MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n::
 	$(MAKE) -C po $@
 endif
 
-# Delete the target file on error
-.DELETE_ON_ERROR:
-
 .PHONY: FORCE
Makefile +3 -10
index 6f0b4b775fe..d378ec22545 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
 # The default target of this Makefile is...
 all::
 
@@ -2194,16 +2197,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
 strip: $(PROGRAMS) git$X
 	$(STRIP) $(STRIP_OPTS) $^
 
-### Flags affecting all rules
-
-# A GNU make extension since gmake 3.72 (released in late 1994) to
-# remove the target of rules if commands in those rules fail. The
-# default is to only do that if make itself receives a signal. Affects
-# all targets, see:
-#
-#    info make --index-search=.DELETE_ON_ERROR
-.DELETE_ON_ERROR:
-
 ### Target-specific flags and dependencies
 
 # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
index 5b12a437426..6fb5cc8b701 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
 QUIET_SUBDIR1  =
 
contrib/scalar/t/Makefile +3 -0
index 6170672bb37..01e82e56d15 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../../shared.mak
+
 # Run scalar tests
 #
 # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
new file mode 100644
index 00000000000..0170bb397ae
@@ -0,0 +1,9 @@
+### Flags affecting all rules
+
+# A GNU make extension since gmake 3.72 (released in late 1994) to
+# remove the target of rules if commands in those rules fail. The
+# default is to only do that if make itself receives a signal. Affects
+# all targets, see:
+#
+#    info make --index-search=.DELETE_ON_ERROR
+.DELETE_ON_ERROR:
t/Makefile +3 -0
index 46cd5fc5273..056ce55dcc9 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # Run tests
 #
 # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
index 31a4bbc716a..6911c2915a7 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 -include ../../config.mak
 export GIT_TEST_OPTIONS
 
t/perf/Makefile +3 -0
index 2465770a782..e4808aebed0 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 -include ../../config.mak
 export GIT_TEST_OPTIONS
 
templates/Makefile +3 -0
index d22a71a3999..636cee52f51 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # make and install sample templates
 
 ifndef V
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#16 lore raw

Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

Makefile line 5

Quoted 10 lines · Ævar Arnfjörð Bjarmason
Makefile +3 -0
index 6f0b4b775fe..d378ec22545 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
 # The default target of this Makefile is...
 all::

Continuing with the theme of [1/9], this change to Makefile gets my firm NAK. The first two lines MUST stay to be the first lines.

Otherwise, a mistaken patch that adds rules to shared.make will make the first of these rules, not "all", as the default target.

Just move it below to the third line or so and you'd be OK.

#17 lore raw

Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

Quoted 16 lines · Junio C Hamano

Makefile line 5

Quoted 10 lines · Ævar Arnfjörð Bjarmason
Makefile +3 -0
index 6f0b4b775fe..d378ec22545 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
 # The default target of this Makefile is...
 all::

Continuing with the theme of [1/9], this change to Makefile gets my firm NAK. The first two lines MUST stay to be the first lines.

Quoted 4 lines

Otherwise, a mistaken patch that adds rules to shared.make will make the first of these rules, not "all", as the default target.

Just move it below to the third line or so and you'd be OK.

Understood, I can/will do that.

But wouldn't working this into this series be an even better option?

The ".DEFAULT_GOAL" variable is new in, 3.81 (01 Apr 2006), which I think is old enough to hard depend on. We use $(or ...) added in the same release in config.mak.dev.

We definitely have a hard dependency on "Version 3.80 (03 Oct 2002)" $(eval ...) etc.

I checked and RHEL5 (which is otherwise the oldest OS we've supported) has it, but RHEL4 doesn't (it has 3.80). So I think it's safe to have a hard dependency on it and other things in 3.81 at this point.

Makefile +0 -3
index 549ca6e7a5c..03321af7e3e 100644
@@ -1,9 +1,6 @@
 # Import tree-wide shared Makefile behavior and libraries
 include shared.mak
 
-# The default target of this Makefile is...
-all::
-
 # Define V=1 to have a more verbose compile.
 #
 # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
shared.mak +5 -0
index 7ba04049c8a..1034dc758f1 100644
@@ -1,3 +1,8 @@
+### By setting the default goal to "all" we override any implicit
+### setting once "make" sees our first target, which it'll ignore if
+### .DEFAULT_GOAL was explicitly set.
+.DEFAULT_GOAL = all
+
 ### Remove GNU make implicit rules
 
 ## This speeds things up since we don't need to look for and stat() a
#18 lore raw

Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

Quoted 1 line · Ævar Arnfjörð Bjarmason

But wouldn't working this into this series be an even better option?

No.

What our current Makefile does (i.e. to have the default goal at the beginning, even though the recipe for it may live elsewhere) is a very well understood and established pattern, and there is no reason to start doing the same thing in a different way, even if it is safe to assume that the "different way" may be available everywhere. We are not gaining anything from doing it differently.

Especially it is not something I want to see as "while at it" change, which adds more to the review cycle without improving anything.

Quoted 3 lines

The ".DEFAULT_GOAL" variable is new in, 3.81 (01 Apr 2006), which I think is old enough to hard depend on. We use $(or ...) added in the same release in config.mak.dev.

#19 lore raw

Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

In reply to Ævar Arnfjörð Bjarmason

Hi Ævar

Quoted 10 lines · Ævar Arnfjörð Bjarmason

We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.

See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.

Quoted 5 lines

This does have the potential downside that if e.g. templates/Makefile would like to include this "shared.mak" in the future the semantics of such a Makefile will change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.

I was confused about the mention of templates/Makefile in this paragraph, it seems to be saying that the behavior would change in the future if we included shared.mak in templates/Makefile but this patch does exactly that.

Also does this patch mean we're now using .DELETE_ON_ERROR in places where we were not previously using it?

Best Wishes

Phillip

Quoted 146 lines

This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

---
  Documentation/Makefile    |  6 +++---
  Makefile                  | 13 +++----------
  contrib/scalar/Makefile   |  3 +++
  contrib/scalar/t/Makefile |  3 +++
  shared.mak                |  9 +++++++++
  t/Makefile                |  3 +++
  t/interop/Makefile        |  3 +++
  t/perf/Makefile           |  3 +++
  templates/Makefile        |  3 +++
  9 files changed, 33 insertions(+), 13 deletions(-)

create mode 100644 shared.mak

Documentation/Makefile +3 -3
index ed656db2ae9..ba27456c86a 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
  # Guard against environment variables
  MAN1_TXT =
  MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n::
  	$(MAKE) -C po $@
  endif
 
-# Delete the target file on error
-.DELETE_ON_ERROR:
-
  .PHONY: FORCE
Makefile +3 -10
index 6f0b4b775fe..d378ec22545 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
  # The default target of this Makefile is...
  all::
 
@@ -2194,16 +2197,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
  strip: $(PROGRAMS) git$X
  	$(STRIP) $(STRIP_OPTS) $^
 
-### Flags affecting all rules
-
-# A GNU make extension since gmake 3.72 (released in late 1994) to
-# remove the target of rules if commands in those rules fail. The
-# default is to only do that if make itself receives a signal. Affects
-# all targets, see:
-#
-#    info make --index-search=.DELETE_ON_ERROR
-.DELETE_ON_ERROR:
-
  ### Target-specific flags and dependencies
 
  # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
index 5b12a437426..6fb5cc8b701 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
  QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
  QUIET_SUBDIR1  =
 
contrib/scalar/t/Makefile +3 -0
index 6170672bb37..01e82e56d15 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../../shared.mak
+
  # Run scalar tests
  #
  # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
new file mode 100644
index 00000000000..0170bb397ae
@@ -0,0 +1,9 @@
+### Flags affecting all rules
+
+# A GNU make extension since gmake 3.72 (released in late 1994) to
+# remove the target of rules if commands in those rules fail. The
+# default is to only do that if make itself receives a signal. Affects
+# all targets, see:
+#
+#    info make --index-search=.DELETE_ON_ERROR
+.DELETE_ON_ERROR:
t/Makefile +3 -0
index 46cd5fc5273..056ce55dcc9 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
  # Run tests
  #
  # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
index 31a4bbc716a..6911c2915a7 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
  -include ../../config.mak
  export GIT_TEST_OPTIONS
 
t/perf/Makefile +3 -0
index 2465770a782..e4808aebed0 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
  -include ../../config.mak
  export GIT_TEST_OPTIONS
 
templates/Makefile +3 -0
index d22a71a3999..636cee52f51 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
  # make and install sample templates
 
  ifndef V
#20 lore raw

Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

Quoted 18 lines · Phillip Wood

Hi Ævar

Quoted 6 lines · Ævar Arnfjörð Bjarmason

We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using

Quoted 8 lines

.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
This does have the potential downside that if
e.g. templates/Makefile
would like to include this "shared.mak" in the future the semantics of such a Makefile will change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.

Quoted 4 lines

I was confused about the mention of templates/Makefile in this paragraph, it seems to be saying that the behavior would change in the future if we included shared.mak in templates/Makefile but this patch does exactly that.

Yes, oops! It's a zombie comment that I forgot to adjust from an earlier version of this where that wasn't the case. Will adjust & re-roll.

Quoted 2 lines

Also does this patch mean we're now using .DELETE_ON_ERROR in places where we were not previously using it?

Yes, we'll now use it in those other Makefiles as well. The commits referenced in the second paragrap of the commit message argue for this being safe, and I've reviewed the logic myself & don't expect any problems with it.

As the GNU make manual itself notes the cases where that would cause problems are really obscure, but it's not the default out of an abundance of backwards compatibility caution in GNU make.

Quoted 137 lines
Quoted 137 lines

This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

---
  Documentation/Makefile    |  6 +++---
  Makefile                  | 13 +++----------
  contrib/scalar/Makefile   |  3 +++
  contrib/scalar/t/Makefile |  3 +++
  shared.mak                |  9 +++++++++
  t/Makefile                |  3 +++
  t/interop/Makefile        |  3 +++
  t/perf/Makefile           |  3 +++
  templates/Makefile        |  3 +++
  9 files changed, 33 insertions(+), 13 deletions(-)

create mode 100644 shared.mak

Documentation/Makefile +3 -2
index ed656db2ae9..ba27456c86a 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
  # Guard against environment variables
  MAN1_TXT =
  MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n::
  	$(MAKE) -C po $@
  endif
  -# Delete the target file on error
-.DELETE_ON_ERROR:
-
  .PHONY: FORCE
Makefile +3 -0
index 6f0b4b775fe..d378ec22545 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
  # The default target of this Makefile is...
  all::
  @@ -2194,16 +2197,6 @@ shell_compatibility_test:

please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $^
-### Flags affecting all rules

Makefile +0 -9
-
-# A GNU make extension since gmake 3.72 (released in late 1994) to
-# remove the target of rules if commands in those rules fail. The
-# default is to only do that if make itself receives a signal. Affects
-# all targets, see:
-#
-#    info make --index-search=.DELETE_ON_ERROR
-.DELETE_ON_ERROR:
-
  ### Target-specific flags and dependencies
    # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
index 5b12a437426..6fb5cc8b701 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
  QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
  QUIET_SUBDIR1  =
  diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefile

index 6170672bb37..01e82e56d15 100644

contrib/scalar/t/Makefile +3 -0
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../../shared.mak
+
  # Run scalar tests
  #
  # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
new file mode 100644
index 00000000000..0170bb397ae
@@ -0,0 +1,9 @@
+### Flags affecting all rules
+
+# A GNU make extension since gmake 3.72 (released in late 1994) to
+# remove the target of rules if commands in those rules fail. The
+# default is to only do that if make itself receives a signal. Affects
+# all targets, see:
+#
+#    info make --index-search=.DELETE_ON_ERROR
+.DELETE_ON_ERROR:
t/Makefile +3 -0
index 46cd5fc5273..056ce55dcc9 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
  # Run tests
  #
  # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
index 31a4bbc716a..6911c2915a7 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
  -include ../../config.mak
  export GIT_TEST_OPTIONS
  diff --git a/t/perf/Makefile b/t/perf/Makefile

index 2465770a782..e4808aebed0 100644

t/perf/Makefile +3 -0
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
  -include ../../config.mak
  export GIT_TEST_OPTIONS
  diff --git a/templates/Makefile b/templates/Makefile

index d22a71a3999..636cee52f51 100644

templates/Makefile +3 -0
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
  # make and install sample templates
    ifndef V
#21 lore raw

Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

Quoted 29 lines · Ævar Arnfjörð Bjarmason
Quoted 18 lines · Phillip Wood

Hi Ævar

Quoted 6 lines · Ævar Arnfjörð Bjarmason

We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using

Quoted 8 lines

.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
This does have the potential downside that if
e.g. templates/Makefile
would like to include this "shared.mak" in the future the semantics of such a Makefile will change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.

Quoted 4 lines

I was confused about the mention of templates/Makefile in this paragraph, it seems to be saying that the behavior would change in the future if we included shared.mak in templates/Makefile but this patch does exactly that.

Yes, oops! It's a zombie comment that I forgot to adjust from an earlier version of this where that wasn't the case. Will adjust & re-roll.

Quoted 7 lines
Quoted 2 lines

Also does this patch mean we're now using .DELETE_ON_ERROR in places where we were not previously using it?

Yes, we'll now use it in those other Makefiles as well. The commits referenced in the second paragrap of the commit message argue for this being safe, and I've reviewed the logic myself & don't expect any problems with it.

Thanks for elaborating, maybe it is worth spelling explicitly in the commit message that this is turning on .DELETE_ON_ERROR in places we didn't previously use it. I had a look at the commit message you referenced and it seems to make a good case for using .DELETE_ON_ERROR. Having a shared makefile for common code makes sense and the speed ups from some of the other commits are nice.

Best Wishes

Phillip

Quoted 142 lines

As the GNU make manual itself notes the cases where that would cause problems are really obscure, but it's not the default out of an abundance of backwards compatibility caution in GNU make.

Quoted 137 lines
Quoted 137 lines

This also doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

---
   Documentation/Makefile    |  6 +++---
   Makefile                  | 13 +++----------
   contrib/scalar/Makefile   |  3 +++
   contrib/scalar/t/Makefile |  3 +++
   shared.mak                |  9 +++++++++
   t/Makefile                |  3 +++
   t/interop/Makefile        |  3 +++
   t/perf/Makefile           |  3 +++
   templates/Makefile        |  3 +++
   9 files changed, 33 insertions(+), 13 deletions(-)

create mode 100644 shared.mak

Documentation/Makefile +3 -2
index ed656db2ae9..ba27456c86a 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
   # Guard against environment variables
   MAN1_TXT =
   MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n::
   	$(MAKE) -C po $@
   endif
   -# Delete the target file on error
-.DELETE_ON_ERROR:
-
   .PHONY: FORCE
Makefile +3 -0
index 6f0b4b775fe..d378ec22545 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
   # The default target of this Makefile is...
   all::
   @@ -2194,16 +2197,6 @@ shell_compatibility_test:

please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $^
-### Flags affecting all rules

Makefile +0 -9
-
-# A GNU make extension since gmake 3.72 (released in late 1994) to
-# remove the target of rules if commands in those rules fail. The
-# default is to only do that if make itself receives a signal. Affects
-# all targets, see:
-#
-#    info make --index-search=.DELETE_ON_ERROR
-.DELETE_ON_ERROR:
-
   ### Target-specific flags and dependencies
     # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
index 5b12a437426..6fb5cc8b701 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
   QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
   QUIET_SUBDIR1  =
   diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefile

index 6170672bb37..01e82e56d15 100644

contrib/scalar/t/Makefile +3 -0
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../../shared.mak
+
   # Run scalar tests
   #
   # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
new file mode 100644
index 00000000000..0170bb397ae
@@ -0,0 +1,9 @@
+### Flags affecting all rules
+
+# A GNU make extension since gmake 3.72 (released in late 1994) to
+# remove the target of rules if commands in those rules fail. The
+# default is to only do that if make itself receives a signal. Affects
+# all targets, see:
+#
+#    info make --index-search=.DELETE_ON_ERROR
+.DELETE_ON_ERROR:
t/Makefile +3 -0
index 46cd5fc5273..056ce55dcc9 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
   # Run tests
   #
   # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
index 31a4bbc716a..6911c2915a7 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
   -include ../../config.mak
   export GIT_TEST_OPTIONS
   diff --git a/t/perf/Makefile b/t/perf/Makefile

index 2465770a782..e4808aebed0 100644

t/perf/Makefile +3 -0
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
   -include ../../config.mak
   export GIT_TEST_OPTIONS
   diff --git a/templates/Makefile b/templates/Makefile

index d22a71a3999..636cee52f51 100644

templates/Makefile +3 -0
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
   # make and install sample templates
     ifndef V
#22 lore raw

Re: [PATCH v3 2/9] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

Quoted 36 lines · Phillip Wood
Quoted 28 lines · Ævar Arnfjörð Bjarmason
Quoted 18 lines · Phillip Wood

Hi Ævar

Quoted 6 lines · Ævar Arnfjörð Bjarmason

We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.
See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using

Quoted 8 lines

.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.
This does have the potential downside that if
e.g. templates/Makefile
would like to include this "shared.mak" in the future the semantics of such a Makefile will change, but as noted in the above commits (and GNU make's own documentation) any such change would be for the better, so it's safe to do this.

Quoted 4 lines

I was confused about the mention of templates/Makefile in this paragraph, it seems to be saying that the behavior would change in the future if we included shared.mak in templates/Makefile but this patch does exactly that.

Yes, oops! It's a zombie comment that I forgot to adjust from an earlier
version of this where that wasn't the case. Will adjust & re-roll.

Quoted 6 lines
Quoted 2 lines

Also does this patch mean we're now using .DELETE_ON_ERROR in places where we were not previously using it?

Yes, we'll now use it in those other Makefiles as well. The commits referenced in the second paragrap of the commit message argue for this being safe, and I've reviewed the logic myself & don't expect any problems with it.

Quoted 6 lines

Thanks for elaborating, maybe it is worth spelling explicitly in the commit message that this is turning on .DELETE_ON_ERROR in places we didn't previously use it. I had a look at the commit message you referenced and it seems to make a good case for using .DELETE_ON_ERROR. Having a shared makefile for common code makes sense and the speed ups from some of the other commits are nice.

Thanks, yes, willdo :) I have a re-roll of this queued up for submission, after sitting on it for a bit longer to shake out any potential last-minute issues...

#23 lore raw

PATCH v3 3/9 Makefile: disable GNU make built-in wildcard rules

In reply to Ævar Arnfjörð Bjarmason

Override built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.

The speed-up may wary. I've seen 1-10% depending on the speed of the local disk, caches, -jN etc. Running:

strace -f -c -S calls make -j1 NO_TCLTK=Y

Shows that we reduce the number of syscalls we make, mostly in "stat" calls.

We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early. Doing so might make us a bit faster still. But doing so is a much bigger hammer, since it will disable all built-in rules, some (all?) of which can be seen with:

    make -f/dev/null -p | grep -v -e ^# -e ^$

We may have something that relies on them, so let's go for the more isolated optimization here that gives us most or all of the wins.

1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html

---
 shared.mak | 11 +++++++++++
 1 file changed, 11 insertions(+)
shared.mak +11 -0
index 0170bb397ae..29f0e69ecb9 100644
@@ -1,3 +1,14 @@
+### Remove GNU make implicit rules
+
+## This speeds things up since we don't need to look for and stat() a
+## "foo.c,v" every time a rule referring to "foo.c" is in play. See
+## "make -p -f/dev/null | grep ^%::'".
+%:: %,v
+%:: RCS/%,v
+%:: RCS/%
+%:: s.%
+%:: SCCS/s.%
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#24 lore raw

Re: [PATCH v3 3/9] Makefile: disable GNU make built-in wildcard rules

Quoted 5 lines · Ævar Arnfjörð Bjarmason

Override built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.

;-)

Quoted 1 line

The speed-up may wary. I've seen 1-10% depending on the speed of the

vary, you mean?

Quoted 6 lines

local disk, caches, -jN etc. Running:

strace -f -c -S calls make -j1 NO_TCLTK=Y

Shows that we reduce the number of syscalls we make, mostly in "stat" calls.

Naturally understandable.

I do remember getting surprised after seeing output from "make -d" and how many paths were considered for possible places that a source file can come from.

Quoted 14 lines
shared.mak +11 -0
+### Remove GNU make implicit rules
+
+## This speeds things up since we don't need to look for and stat() a
+## "foo.c,v" every time a rule referring to "foo.c" is in play. See
+## "make -p -f/dev/null | grep ^%::'".
+%:: %,v
+%:: RCS/%,v
+%:: RCS/%
+%:: s.%
+%:: SCCS/s.%
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
#25 lore raw

PATCH v3 4/9 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)

In reply to Ævar Arnfjörð Bjarmason

Combine the definitions of $(FIND_SOURCE_FILES) and $(LIB_H) to speed up the Makefile, as these are the two main expensive $(shell) commands that we execute unconditionally.

When see what was in $(FOUND_SOURCE_FILES) that wasn't in $(LIB_H) via the ad-hoc test of:

    $(error $(filter-out $(LIB_H),$(filter %.h,$(ALL_SOURCE_FILES))))
    $(error $(filter-out $(ALL_SOURCE_FILES),$(filter %.h,$(LIB_H))))

We'll get, respectively:

    Makefile:850: *** t/helper/test-tool.h.  Stop.
    Makefile:850: *** .  Stop.

I.e. we only had a discrepancy when it came to t/helper/test-tool.h. In terms of correctness this was broken before, but now works:

    $ make t/helper/test-tool.hco
        HDR t/helper/test-tool.h

This speeds things up a lot:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10
    Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1
      Time (mean ± σ):     159.9 ms ±   6.8 ms    [User: 137.2 ms, System: 28.0 ms]
      Range (min … max):   154.6 ms … 175.9 ms    10 runs
    Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0
      Time (mean ± σ):     100.0 ms ±   1.3 ms    [User: 84.2 ms, System: 20.2 ms]
      Range (min … max):    98.8 ms … 102.8 ms    10 runs

    Summary
      'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran
        1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1'
---
 Makefile | 54 ++++++++++++++++++++++++++----------------------------
 1 file changed, 26 insertions(+), 28 deletions(-)
Makefile +26 -28
index d378ec22545..185bfa71df3 100644
@@ -833,12 +833,33 @@ GENERATED_H += hook-list.h
 .PHONY: generated-hdrs
 generated-hdrs: $(GENERATED_H)
 
-LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \
+## Exhaustive lists of our source files, either dynamically generated,
+## or hardcoded.
+SOURCES_CMD = ( \
+	git ls-files \
+		'*.[hcS]' \
+		'*.sh' \
+		':!*[tp][0-9][0-9][0-9][0-9]*' \
+		':!contrib' \
+		2>/dev/null || \
 	$(FIND) . \
-	-name .git -prune -o \
-	-name t -prune -o \
-	-name Documentation -prune -o \
-	-name '*.h' -print)))
+		\( -name .git -type d -prune \) \
+		-o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
+		-o \( -name contrib -type d -prune \) \
+		-o \( -name build -type d -prune \) \
+		-o \( -name 'trash*' -type d -prune \) \
+		-o \( -name '*.[hcS]' -type f -print \) \
+		-o \( -name '*.sh' -type f -print \) \
+		| sed -e 's|^\./||' \
+	)
+FOUND_SOURCE_FILES := $(shell $(SOURCES_CMD))
+
+FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))
+FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))
+
+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
+
+LIB_H = $(FOUND_H_SOURCES)
 
 LIB_OBJS += abspath.o
 LIB_OBJS += add-interactive.o
@@ -2789,26 +2810,6 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
 	$(QUIET_GEN)mkdir -p $(dir $@) && \
 	pod2man $< $@
 
-FIND_SOURCE_FILES = ( \
-	git ls-files \
-		'*.[hcS]' \
-		'*.sh' \
-		':!*[tp][0-9][0-9][0-9][0-9]*' \
-		':!contrib' \
-		2>/dev/null || \
-	$(FIND) . \
-		\( -name .git -type d -prune \) \
-		-o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
-		-o \( -name contrib -type d -prune \) \
-		-o \( -name build -type d -prune \) \
-		-o \( -name 'trash*' -type d -prune \) \
-		-o \( -name '*.[hcS]' -type f -print \) \
-		-o \( -name '*.sh' -type f -print \) \
-		| sed -e 's|^\./||' \
-	)
-
-FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES))
-
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
 	echo $(FOUND_SOURCE_FILES) | xargs etags -a -o $@+ && \
@@ -3018,9 +3019,6 @@ check: $(GENERATED_H)
 		exit 1; \
 	fi
 
-FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))
-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
-
 %.cocci.patch: %.cocci $(COCCI_SOURCES)
 	$(QUIET_SPATCH) \
 	if test $(SPATCH_BATCH_SIZE) = 0; then \
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#26 lore raw

PATCH v3 5/9 Makefile: move ".SUFFIXES" rule to shared.mak

In reply to Ævar Arnfjörð Bjarmason

This was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.

This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
    Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
      Time (mean ± σ):     121.7 ms ±   8.8 ms    [User: 105.8 ms, System: 18.6 ms]
      Range (min … max):   112.8 ms … 148.4 ms    26 runs

Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs

Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.

So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.

1. https://lore.kernel.org/git/[email protected]/

---
 Makefile   | 2 --
 shared.mak | 5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
index 185bfa71df3..1462016a3f4 100644
@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S))
 ASM_OBJ := $(ASM_SRC:S=o)
 C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS))
 
-.SUFFIXES:
-
 $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
 	$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
 $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +5 -0
index 29f0e69ecb9..1dda948df09 100644
@@ -9,6 +9,11 @@
 %:: s.%
 %:: SCCS/s.%
 
+## Likewise delete default $(SUFFIXES). See:
+##
+##     info make --index-search=.SUFFIXES
+.SUFFIXES:
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#27 lore raw

Re: [PATCH v3 5/9] Makefile: move ".SUFFIXES" rule to shared.mak

Quoted 7 lines · Ævar Arnfjörð Bjarmason

This was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.

This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:

Nice.

#28 lore raw

PATCH v3 6/9 Makefile: move $(comma), $(empty) and $(space) to shared.mak

In reply to Ævar Arnfjörð Bjarmason

Move these variables over to the shared.max, we'll make use of them in a subsequent commit. There was no reason for these to be "simply expanded variables", so let's use the normal lazy "=" assignment here.

See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.

---
 Makefile   | 4 ----
 shared.mak | 8 ++++++++
 2 files changed, 8 insertions(+), 4 deletions(-)
Makefile +0 -4
index 1462016a3f4..81b2eaa0355 100644
@@ -1289,10 +1289,6 @@ endif
 ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS)
 ALL_LDFLAGS = $(LDFLAGS)
 
-comma := ,
-empty :=
-space := $(empty) $(empty)
-
 ifdef SANITIZE
 SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag))
 BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)
shared.mak +8 -0
index 1dda948df09..f6b589ffd8f 100644
@@ -23,3 +23,11 @@
 #
 #    info make --index-search=.DELETE_ON_ERROR
 .DELETE_ON_ERROR:
+
+### Global variables
+
+## comma, empty, space: handy variables as these tokens are either
+## special or can be hard to spot among other Makefile syntax.
+comma = ,
+empty =
+space = $(empty) $(empty)
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#29 lore raw

Re: [PATCH v3 6/9] Makefile: move $(comma), $(empty) and $(space) to shared.mak

Quoted 1 line · Ævar Arnfjörð Bjarmason

Move these variables over to the shared.max, we'll make use of them in

max?

Quoted 2 lines

a subsequent commit. There was no reason for these to be "simply expanded variables", so let's use the normal lazy "=" assignment here.

Seems to do unrelated things "while at it" which is a common pattern that slows the reviews down.

Quoted 6 lines
shared.mak +0 -0
index 1dda948df09..f6b589ffd8f 100644
@@ -23,3 +23,11 @@
 #

shared.mak line 33

Quoted 10 lines
shared.mak +8 -0
 #    info make --index-search=.DELETE_ON_ERROR
 .DELETE_ON_ERROR:
+
+### Global variables
+
+## comma, empty, space: handy variables as these tokens are either
+## special or can be hard to spot among other Makefile syntax.
+comma = ,
+empty =
+space = $(empty) $(empty)

In any case, wanting to have a $(space) is not specific to the top-level Makefile, and it is a good idea to move them to the shared file.

#30 lore raw

PATCH v3 7/9 Makefile: add "$(QUIET)" boilerplate to shared.mak

In reply to Ævar Arnfjörð Bjarmason

The $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.

Since we're not using the environment to pass these around we don't need to export the "QUIET_GEN" and "QUIET_BUILT_IN" variables anymore. The "QUIET_GEN" variable is used in "git-gui/Makefile" and "gitweb/Makefile", but they've got their own definition for those. The "QUIET_BUILT_IN" variable is only used in the top-level "Makefile". We still need to export the "V" variable.

---
 Documentation/Makefile  | 32 -------------------------
 Makefile                | 33 -------------------------
 config.mak.uname        |  1 -
 contrib/scalar/Makefile | 14 -----------
 shared.mak              | 53 +++++++++++++++++++++++++++++++++++++++++
 templates/Makefile      |  5 ----
 6 files changed, 53 insertions(+), 85 deletions(-)
Documentation/Makefile +0 -32
index ba27456c86a..0f4ebdeda8a 100644
@@ -218,38 +218,6 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR))
 ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)'
 endif
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring $(MAKEFLAGS),w),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring $(MAKEFLAGS),s),s)
-ifndef V
-	QUIET		= @
-	QUIET_ASCIIDOC	= @echo '   ' ASCIIDOC $@;
-	QUIET_XMLTO	= @echo '   ' XMLTO $@;
-	QUIET_DB2TEXI	= @echo '   ' DB2TEXI $@;
-	QUIET_MAKEINFO	= @echo '   ' MAKEINFO $@;
-	QUIET_DBLATEX	= @echo '   ' DBLATEX $@;
-	QUIET_XSLTPROC	= @echo '   ' XSLTPROC $@;
-	QUIET_GEN	= @echo '   ' GEN $@;
-	QUIET_STDERR	= 2> /dev/null
-	QUIET_SUBDIR0	= +@subdir=
-	QUIET_SUBDIR1	= ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			  $(MAKE) $(PRINT_DIR) -C $$subdir
-
-	QUIET_LINT_GITLINK	= @echo '   ' LINT GITLINK $<;
-	QUIET_LINT_MANSEC	= @echo '   ' LINT MAN SEC $<;
-	QUIET_LINT_MANEND	= @echo '   ' LINT MAN END $<;
-
-	export V
-endif
-endif
-
 all: html man
 
 html: $(DOC_HTML)
Makefile +0 -33
index 81b2eaa0355..9f93ee1532c 100644
@@ -2001,39 +2001,6 @@ ifndef PAGER_ENV
 PAGER_ENV = LESS=FRX LV=-c
 endif
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring w,$(MAKEFLAGS)),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring s,$(MAKEFLAGS)),s)
-ifndef V
-	QUIET_CC       = @echo '   ' CC $@;
-	QUIET_AR       = @echo '   ' AR $@;
-	QUIET_LINK     = @echo '   ' LINK $@;
-	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
-	QUIET_GEN      = @echo '   ' GEN $@;
-	QUIET_LNCP     = @echo '   ' LN/CP $@;
-	QUIET_XGETTEXT = @echo '   ' XGETTEXT $@;
-	QUIET_MSGFMT   = @echo '   ' MSGFMT $@;
-	QUIET_GCOV     = @echo '   ' GCOV $@;
-	QUIET_SP       = @echo '   ' SP $<;
-	QUIET_HDR      = @echo '   ' HDR $(<:hcc=h);
-	QUIET_RC       = @echo '   ' RC $@;
-	QUIET_SPATCH   = @echo '   ' SPATCH $<;
-	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			 $(MAKE) $(PRINT_DIR) -C $$subdir
-	export V
-	export QUIET_GEN
-	export QUIET_BUILT_IN
-endif
-endif
-
 ifdef NO_INSTALL_HARDLINKS
 	export NO_INSTALL_HARDLINKS
 endif
config.mak.uname +0 -1
index 4352ea39e9b..7727b707b74 100644
@@ -727,7 +727,6 @@ vcxproj:
 	git diff-index --cached --quiet HEAD --
 
 	# Make .vcxproj files and add them
-	unset QUIET_GEN QUIET_BUILT_IN; \
 	perl contrib/buildsystems/generate -g Vcxproj
 	git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
 
contrib/scalar/Makefile +0 -14
index 6fb5cc8b701..139b2a2c8c8 100644
@@ -1,20 +1,6 @@
 # Import tree-wide shared Makefile behavior and libraries
 include ../../shared.mak
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring s,$(MAKEFLAGS)),s)
-ifndef V
-	QUIET_GEN      = @echo '   ' GEN $@;
-	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			 $(MAKE) $(PRINT_DIR) -C $$subdir
-else
-	export V
-endif
-endif
-
 include ../../config.mak.uname
 -include ../../config.mak.autogen
 -include ../../config.mak
shared.mak +53 -0

56 lines not on this page · raw message

templates/Makefile +0 -5
index 636cee52f51..367ad00c24c 100644
@@ -2,11 +2,6 @@
 include ../shared.mak
 
 # make and install sample templates
-
-ifndef V
-	QUIET = @
-endif
-
 INSTALL ?= install
 TAR ?= tar
 RM ?= rm -f
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#31 lore raw

Re: [PATCH v3 7/9] Makefile: add "$(QUIET)" boilerplate to shared.mak

Quoted 2 lines · Ævar Arnfjörð Bjarmason

The $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.

Thank you thank you thank you.

It has been disturbing me a lot to see these copied-and-pasted duplication across Makefiles.

#32 lore raw

PATCH v3 8/9 Makefile: use $(wspfx) for $(QUIET...) in shared.mak

In reply to Ævar Arnfjörð Bjarmason

Change the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding, to guarantee that it's consistent with the "TRACK_template" template.

    $ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
     -> GIT-CFLAGS PARAMETERS (changed)
     -> CC grep.o
---
 shared.mak | 54 ++++++++++++++++++++++++++++++------------------------
 1 file changed, 30 insertions(+), 24 deletions(-)
shared.mak +30 -24
index 20be9ddf5a6..6e0f103b65f 100644
@@ -32,6 +32,12 @@ comma = ,
 empty =
 space = $(empty) $(empty)
 
+## wspfx: the whitespace prefix padding for $(QUIET...) and similarly
+## aligned output.
+wspfx = $(space)$(space)$(space)
+wspfx_SQ = '$(subst ','\'',$(wspfx))'
+# ' closing quote to appease Emacs make-mode.elxo
+
 ### Quieting
 ## common
 QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
@@ -47,39 +53,39 @@ ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
 ## common
 	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo $(wspfx_SQ) SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
 
 	QUIET          = @
-	QUIET_GEN      = @echo '   ' GEN $@;
+	QUIET_GEN      = @echo $(wspfx_SQ) GEN $@;
 
 ## Used in "Makefile"
-	QUIET_CC       = @echo '   ' CC $@;
-	QUIET_AR       = @echo '   ' AR $@;
-	QUIET_LINK     = @echo '   ' LINK $@;
-	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
-	QUIET_LNCP     = @echo '   ' LN/CP $@;
-	QUIET_XGETTEXT = @echo '   ' XGETTEXT $@;
-	QUIET_MSGFMT   = @echo '   ' MSGFMT $@;
-	QUIET_GCOV     = @echo '   ' GCOV $@;
-	QUIET_SP       = @echo '   ' SP $<;
-	QUIET_HDR      = @echo '   ' HDR $(<:hcc=h);
-	QUIET_RC       = @echo '   ' RC $@;
-	QUIET_SPATCH   = @echo '   ' SPATCH $<;
+	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
+	QUIET_AR       = @echo $(wspfx_SQ) AR $@;
+	QUIET_LINK     = @echo $(wspfx_SQ) LINK $@;
+	QUIET_BUILT_IN = @echo $(wspfx_SQ) BUILTIN $@;
+	QUIET_LNCP     = @echo $(wspfx_SQ) LN/CP $@;
+	QUIET_XGETTEXT = @echo $(wspfx_SQ) XGETTEXT $@;
+	QUIET_MSGFMT   = @echo $(wspfx_SQ) MSGFMT $@;
+	QUIET_GCOV     = @echo $(wspfx_SQ) GCOV $@;
+	QUIET_SP       = @echo $(wspfx_SQ) SP $<;
+	QUIET_HDR      = @echo $(wspfx_SQ) HDR $(<:hcc=h);
+	QUIET_RC       = @echo $(wspfx_SQ) RC $@;
+	QUIET_SPATCH   = @echo $(wspfx_SQ) SPATCH $<;
 
 ## Used in "Documentation/Makefile"
-	QUIET_ASCIIDOC	= @echo '   ' ASCIIDOC $@;
-	QUIET_XMLTO	= @echo '   ' XMLTO $@;
-	QUIET_DB2TEXI	= @echo '   ' DB2TEXI $@;
-	QUIET_MAKEINFO	= @echo '   ' MAKEINFO $@;
-	QUIET_DBLATEX	= @echo '   ' DBLATEX $@;
-	QUIET_XSLTPROC	= @echo '   ' XSLTPROC $@;
-	QUIET_GEN	= @echo '   ' GEN $@;
+	QUIET_ASCIIDOC	= @echo $(wspfx_SQ) ASCIIDOC $@;
+	QUIET_XMLTO	= @echo $(wspfx_SQ) XMLTO $@;
+	QUIET_DB2TEXI	= @echo $(wspfx_SQ) DB2TEXI $@;
+	QUIET_MAKEINFO	= @echo $(wspfx_SQ) MAKEINFO $@;
+	QUIET_DBLATEX	= @echo $(wspfx_SQ) DBLATEX $@;
+	QUIET_XSLTPROC	= @echo $(wspfx_SQ) XSLTPROC $@;
+	QUIET_GEN	= @echo $(wspfx_SQ) GEN $@;
 	QUIET_STDERR	= 2> /dev/null
 
-	QUIET_LINT_GITLINK	= @echo '   ' LINT GITLINK $<;
-	QUIET_LINT_MANSEC	= @echo '   ' LINT MAN SEC $<;
-	QUIET_LINT_MANEND	= @echo '   ' LINT MAN END $<;
+	QUIET_LINT_GITLINK	= @echo $(wspfx_SQ) LINT GITLINK $<;
+	QUIET_LINT_MANSEC	= @echo $(wspfx_SQ) LINT MAN SEC $<;
+	QUIET_LINT_MANEND	= @echo $(wspfx_SQ) LINT MAN END $<;
 
 	export V
 endif
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#33 lore raw

Re: [PATCH v3 8/9] Makefile: use $(wspfx) for $(QUIET...) in shared.mak

Quoted 7 lines · Ævar Arnfjörð Bjarmason

Change the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding, to guarantee that it's consistent with the "TRACK_template" template.

    $ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
     -> GIT-CFLAGS PARAMETERS (changed)
     -> CC grep.o

What is a TRACK_template? Is this some "rebase -i" gone bad?

#34 lore raw

PATCH v3 9/9 Makefiles: add and use wildcard "mkdir -p" template

In reply to Ævar Arnfjörð Bjarmason

Add a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).

As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.

I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.

But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
    Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
      Time (mean ± σ):      2.914 s ±  0.062 s    [User: 2.449 s, System: 0.489 s]
      Range (min … max):    2.834 s …  3.020 s    10 runs

Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s]
Range (min … max): 2.229 s … 2.397 s 10 runs

Summary
'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'

So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.

This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://gitlab.com/avar/git-hyperfine/

---
 Documentation/Makefile | 25 +++----------------------
 Makefile               | 12 +++++++-----
 shared.mak             | 17 +++++++++++++++++
 3 files changed, 27 insertions(+), 27 deletions(-)
Documentation/Makefile +3 -22
index 0f4ebdeda8a..1eb9192dae8 100644
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo
 print-man1:
 	@for i in $(MAN1_TXT); do echo $$i; done
 
-## Lint: Common
-.build:
-	$(QUIET)mkdir $@
-.build/lint-docs: | .build
-	$(QUIET)mkdir $@
-
 ## Lint: gitlink
-.build/lint-docs/gitlink: | .build/lint-docs
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
 LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT))
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config
 $(LINT_DOCS_GITLINK): lint-gitlink.perl
 $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \
 		$< \
 		$(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
 lint-docs-gitlink: $(LINT_DOCS_GITLINK)
 
 ## Lint: man-end-blurb
-.build/lint-docs/man-end-blurb: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb
 $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl
 $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@
 .PHONY: lint-docs-man-end-blurb
-lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB)
 
 ## Lint: man-section-order
-.build/lint-docs/man-section-order: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order
 $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl
 $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@
 .PHONY: lint-docs-man-section-order
 lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
Makefile +7 -5
index 9f93ee1532c..549ca6e7a5c 100644
@@ -2745,7 +2745,8 @@ all:: $(MOFILES)
 endif
 
 po/build/locale/%/LC_MESSAGES/git.mo: po/%.po
-	$(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $<
+	$(call mkdir_p_parent_template)
+	$(QUIET_MSGFMT)$(MSGFMT) -o $@ $<
 
 LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm)
 LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2761,15 +2762,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS))
 endif
 
 perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN) \
 	sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \
 	    -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \
 	    -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \
 	< $< > $@
 
 perl/build/man/man3/Git.3pm: perl/Git.pm
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
-	pod2man $< $@
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN)pod2man $< $@
 
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
@@ -2903,7 +2905,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(
 all:: $(TEST_PROGRAMS) $(test_bindir_programs)
 
 bin-wrappers/%: wrap-for-bin.sh
-	@mkdir -p bin-wrappers
+	$(call mkdir_p_parent_template)
 	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
 	     -e 's|@@BUILD_DIR@@|$(shell pwd)|' \
 	     -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
shared.mak +17 -0
index 6e0f103b65f..7ba04049c8a 100644
@@ -59,6 +59,8 @@ ifndef V
 	QUIET          = @
 	QUIET_GEN      = @echo $(wspfx_SQ) GEN $@;
 
+	QUIET_MKDIR_P_PARENT  = @echo $(wspfx_SQ) MKDIR -p $(@D);
+
 ## Used in "Makefile"
 	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
 	QUIET_AR       = @echo $(wspfx_SQ) AR $@;
@@ -90,3 +92,18 @@ ifndef V
 	export V
 endif
 endif
+
+### Templates
+
+## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@
+## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
+## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a
+## "a/prefix/dir/file". This can instead be inserted at the start of
+## the "a/prefix/dir/file" rule.
+define mkdir_p_parent_template
+$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D)))
+endef
Signature · 1 line
2.35.1.1175.gf9e1b23ea35
#35 lore raw

v4 cover Makefile: optimize noop runs, add shared.mak

In reply to Ævar Arnfjörð Bjarmason

As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/

This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).

This v4 re-roll (for v3, see
https://lore.kernel.org/git/[email protected]/):

* The "all" boilerplate goes first now, before "include"
* Elaborated on DELETE_ON_ERROR ina commit message.
* No longer change a ":=" to "=" while moving code.
* Rephrased other commit messages (one of which referred to a function from another future series)
* Typo fix in commit message.

Ævar Arnfjörð Bjarmason (9):
scalar Makefile: use "The default target of..." pattern
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" template

 Documentation/Makefile    |  63 ++------------------
 Makefile                  | 118 ++++++++++++--------------------------
 config.mak.uname          |   1 -
 contrib/scalar/Makefile   |  20 ++-----
 contrib/scalar/t/Makefile |   3 +
 shared.mak                | 109 +++++++++++++++++++++++++++++++++++
 t/Makefile                |   3 +
 t/interop/Makefile        |   3 +
 t/perf/Makefile           |   3 +
 templates/Makefile        |   8 +--
 10 files changed, 171 insertions(+), 160 deletions(-)
 create mode 100644 shared.mak

Range-diff against v3:

 1:  2404c4d8b96 <  -:  ----------- scalar Makefile: set the default target after the includes
 -:  ----------- >  1:  26c6bb897cf scalar Makefile: use "The default target of..." pattern
 2:  96a490bec54 !  2:  74692458b70 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

@@ Commit message
.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.

- This does have the potential downside that if e.g. templates/Makefile
- would like to include this "shared.mak" in the future the semantics of
- such a Makefile will change, but as noted in the above commits (and
- GNU make's own documentation) any such change would be for the better,
- so it's safe to do this.
+ I.e. this changes the behavior of existing rules in the altered
+ Makefiles (except "Makefile" & "Documentation/Makefile"). I'm
+ confident that this is safe having read the relevant rules in those
+ Makfiles, and as the GNU make manual notes that it isn't the default
+ behavior is out of an abundance of backwards compatibility
+ caution. From edition 0.75 of its manual, covering GNU make 4.3:

- This also doesn't introduce a bug by e.g. having this
+ [Enabling '.DELETE_ON_ERROR' is] almost always what you want
+ 'make' to do, but it is not historical practice; so for
+ compatibility, you must explicitly request it.
+
+ This doesn't introduce a bug by e.g. having this
".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

+ It does increase the danger that any Makefile without an explicit "The
+ default target of this Makefile is..." snippet to define the default
+ target as "all" could have its default rule changed if our new
+ shared.mak ever defines a "real" rule. In subsequent commits we'll be
+ careful not to do that, and such breakage would be obvious e.g. in the
+ case of "make -C t".
+
+ We might want to make that less fragile still (e.g. by using
+ ".DEFAULT_GOAL" as noted in the preceding commit), but for now let's
+ simply include "shared.mak" without adding that boilerplate to all the
+ Makefiles that don't have it already. Most of those are already
+ exposed to that potential caveat e.g. due to including "config.mak*".
+

## Documentation/Makefile ##
@@ Documentation/Makefile: doc-l10n install-l10n::

## Makefile ##
@@
-+# Import tree-wide shared Makefile behavior and libraries -+include shared.mak
-+
# The default target of this Makefile is... all::

++# Import tree-wide shared Makefile behavior and libraries
++include shared.mak
++
+ # Define V=1 to have a more verbose compile.
+ #
+ # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
@@ Makefile: shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $^
@@ Makefile: shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell

## contrib/scalar/Makefile ##
@@
+ # The default target of this Makefile is...
+ all::
+
+# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak
+
- QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
- QUIET_SUBDIR1 =
-
+ include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak

## contrib/scalar/t/Makefile ##
@@
3: 9392e3c3e97 ! 3: 0fbdeeffc7b Makefile: disable GNU make built-in wildcard rules
@@ Commit message
contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.

- The speed-up may wary. I've seen 1-10% depending on the speed of the
+ The speed-up may vary. I've seen 1-10% depending on the speed of the
local disk, caches, -jN etc. Running:

strace -f -c -S calls make -j1 NO_TCLTK=Y

 4:  07cf9daa9d6 =  4:  ea6b835308a Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
 5:  16f2e3ff35b =  5:  c2339694cf7 Makefile: move ".SUFFIXES" rule to shared.mak
 6:  1b6ecb27f02 !  6:  741fdfd48e2 Makefile: move $(comma), $(empty) and $(space) to shared.mak

@@ Metadata
## Commit message ##
Makefile: move $(comma), $(empty) and $(space) to shared.mak

- Move these variables over to the shared.max, we'll make use of them in
- a subsequent commit. There was no reason for these to be "simply
- expanded variables", so let's use the normal lazy "=" assignment here.
+ Move these variables over to the shared.mak, we'll make use of them in
+ a subsequent commit.
+
+ Note that there's reason for these to be "simply expanded variables",
+ i.e. to use ":=" assignments instead of lazily expanded "="
+ assignments. We could use "=", but let's leave this as-is for now for
+ ease of review.

See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.
@@ shared.mak
+
+## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among other Makefile syntax.
-+comma = ,
-+empty =
-+space = $(empty) $(empty)
++comma := ,
++empty :=
++space := $(empty) $(empty)
7: 471067deefc ! 7: a723cbce270 Makefile: add "$(QUIET)" boilerplate to shared.mak
@@ config.mak.uname: vcxproj:

## contrib/scalar/Makefile ##
-@@
- # Import tree-wide shared Makefile behavior and libraries
- include ../../shared.mak
+@@ contrib/scalar/Makefile: include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak

-QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 =
@@ contrib/scalar/Makefile
-endif
-endif
-
- include ../../config.mak.uname
- -include ../../config.mak.autogen
- -include ../../config.mak
+ TARGETS = scalar$(X) scalar.o
+ GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a +

## shared.mak ##
@@
- comma = ,
- empty =
- space = $(empty) $(empty)
+ comma := ,
+ empty :=
+ space := $(empty) $(empty)
+
+### Quieting
+## common
8: 510306d2219 ! 8: 3733b0c8df1 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
@@ Commit message
Makefile: use $(wspfx) for $(QUIET...) in shared.mak

Change the mostly move-only change in the preceding commit to use the
- $(wspfx) variable for defining the QUIET padding, to guarantee that
- it's consistent with the "TRACK_template" template.
+ $(wspfx) variable for defining the QUIET padding. This refactoring
+ will make it easier to emit that exact amount of padding in functions
+ that we might add to shared.mak in the future.

- $ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
- -> GIT-CFLAGS PARAMETERS (changed)
- -> CC grep.o
+ Such a function is not part of this patch series, but a
+ "TRACK_template" that I'd like to add as a follow-up to it makes use
+ of this. Let's make this change now while modifying these QUIET rules
+ is fresh in our minds.

## shared.mak ##
-@@ shared.mak: comma = ,
- empty =
- space = $(empty) $(empty)
+@@ shared.mak: comma := ,
+ empty :=
+ space := $(empty) $(empty)

+## wspfx: the whitespace prefix padding for $(QUIET...) and similarly +## aligned output.
9: 85bb74aa32f = 9: 4cc4aeabb20 Makefiles: add and use wildcard "mkdir -p" template

Signature · 1 line
2.35.1.1228.g56895c6ee86
#36 lore raw

PATCH v4 1/9 scalar Makefile: use "The default target of..." pattern

Make have the "contrib/scalar/Makefile" be stylistically consistent with the top-level "Makefile" in first declaring "all" to be the default rule, follwed by including other Makefile snippets.

This adjusts code added in 0a43fb22026 (scalar: create a rudimentary executable, 2021-12-03), it's a style-only change, in a subsequent commit the "QUIET" boilerplate at the beginning of this file will be retrieved via an include, and having an "all:" between the two set of "include"'s after that change would look odd.

As noted in [1] using ".DEFAULT_GOAL = all" is another way to do this in more modern GNU make versions, which we already have a hard dependency on, but let's leave any such change for a future improvement and go with using our established pattern consistently for now.

1. https://lore.kernel.org/git/[email protected]/

---
 contrib/scalar/Makefile | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
contrib/scalar/Makefile +7 -6
index 231b1ee1796..1e9b969f3df 100644
@@ -1,3 +1,10 @@
+# The default target of this Makefile is...
+all::
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
 QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
 QUIET_SUBDIR1  =
 
@@ -12,12 +19,6 @@ else
 endif
 endif
 
-all:
-
-include ../../config.mak.uname
--include ../../config.mak.autogen
--include ../../config.mak
-
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
Signature · 1 line
2.35.1.1228.g56895c6ee86
#37 lore raw

Re: [PATCH v4 1/9] scalar Makefile: use "The default target of..." pattern

Quoted 3 lines · Ævar Arnfjörð Bjarmason

Make have the "contrib/scalar/Makefile" be stylistically consistent with the top-level "Makefile" in first declaring "all" to be the default rule, follwed by including other Makefile snippets.

Sorry, but I cannot quite parse it. Perhaps remove "have"?

Quoted 5 lines

This adjusts code added in 0a43fb22026 (scalar: create a rudimentary executable, 2021-12-03), it's a style-only change, in a subsequent commit the "QUIET" boilerplate at the beginning of this file will be retrieved via an include, and having an "all:" between the two set of "include"'s after that change would look odd.

In other words, raising these includes to the top is not style-only but is more significant change. I think these two are logically distinct changes, but I am fine to see both in the same commit, as long as they are explained as such.

Quoted 6 lines

As noted in [1] using ".DEFAULT_GOAL = all" is another way to do this in more modern GNU make versions, which we already have a hard dependency on, but let's leave any such change for a future improvement and go with using our established pattern consistently for now.

I would suggest dropping this paragraph. In the discussion, we didn't even establish that such a change would be an "improvement", and in fact, not everything new is an improvement. If we know .DEFAULT_GOAL is not used, readers of Makefile do not have to run around and look for it to figure out what happens when "make" is run without argument.

#38 lore raw

PATCH v4 2/9 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

In reply to Ævar Arnfjörð Bjarmason

We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.

See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.

I.e. this changes the behavior of existing rules in the altered Makefiles (except "Makefile" & "Documentation/Makefile"). I'm confident that this is safe having read the relevant rules in those Makfiles, and as the GNU make manual notes that it isn't the default behavior is out of an abundance of backwards compatibility caution. From edition 0.75 of its manual, covering GNU make 4.3:

[Enabling '.DELETE_ON_ERROR' is] almost always what you want 'make' to do, but it is not historical practice; so for compatibility, you must explicitly request it.

This doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

It does increase the danger that any Makefile without an explicit "The default target of this Makefile is..." snippet to define the default target as "all" could have its default rule changed if our new shared.mak ever defines a "real" rule. In subsequent commits we'll be careful not to do that, and such breakage would be obvious e.g. in the case of "make -C t".

We might want to make that less fragile still (e.g. by using ".DEFAULT_GOAL" as noted in the preceding commit), but for now let's simply include "shared.mak" without adding that boilerplate to all the Makefiles that don't have it already. Most of those are already exposed to that potential caveat e.g. due to including "config.mak*".

---
 Documentation/Makefile    |  6 +++---
 Makefile                  | 13 +++----------
 contrib/scalar/Makefile   |  3 +++
 contrib/scalar/t/Makefile |  3 +++
 shared.mak                |  9 +++++++++
 t/Makefile                |  3 +++
 t/interop/Makefile        |  3 +++
 t/perf/Makefile           |  3 +++
 templates/Makefile        |  3 +++
 9 files changed, 33 insertions(+), 13 deletions(-)
 create mode 100644 shared.mak
Documentation/Makefile +3 -3
index ed656db2ae9..ba27456c86a 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # Guard against environment variables
 MAN1_TXT =
 MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n::
 	$(MAKE) -C po $@
 endif
 
-# Delete the target file on error
-.DELETE_ON_ERROR:
-
 .PHONY: FORCE
Makefile +3 -10
index 6f0b4b775fe..aa5ff001b6e 100644
@@ -1,6 +1,9 @@
 # The default target of this Makefile is...
 all::
 
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
 # Define V=1 to have a more verbose compile.
 #
 # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
@@ -2194,16 +2197,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
 strip: $(PROGRAMS) git$X
 	$(STRIP) $(STRIP_OPTS) $^
 
-### Flags affecting all rules
-
-# A GNU make extension since gmake 3.72 (released in late 1994) to
-# remove the target of rules if commands in those rules fail. The
-# default is to only do that if make itself receives a signal. Affects
-# all targets, see:
-#
-#    info make --index-search=.DELETE_ON_ERROR
-.DELETE_ON_ERROR:
-
 ### Target-specific flags and dependencies
 
 # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
index 1e9b969f3df..d585b319edc 100644
@@ -1,6 +1,9 @@
 # The default target of this Makefile is...
 all::
 
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 include ../../config.mak.uname
 -include ../../config.mak.autogen
 -include ../../config.mak
contrib/scalar/t/Makefile +3 -0
index 6170672bb37..01e82e56d15 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../../shared.mak
+
 # Run scalar tests
 #
 # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
new file mode 100644
index 00000000000..0170bb397ae
@@ -0,0 +1,9 @@
+### Flags affecting all rules
+
+# A GNU make extension since gmake 3.72 (released in late 1994) to
+# remove the target of rules if commands in those rules fail. The
+# default is to only do that if make itself receives a signal. Affects
+# all targets, see:
+#
+#    info make --index-search=.DELETE_ON_ERROR
+.DELETE_ON_ERROR:
t/Makefile +3 -0
index 46cd5fc5273..056ce55dcc9 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # Run tests
 #
 # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
index 31a4bbc716a..6911c2915a7 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 -include ../../config.mak
 export GIT_TEST_OPTIONS
 
t/perf/Makefile +3 -0
index 2465770a782..e4808aebed0 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 -include ../../config.mak
 export GIT_TEST_OPTIONS
 
templates/Makefile +3 -0
index d22a71a3999..636cee52f51 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # make and install sample templates
 
 ifndef V
Signature · 1 line
2.35.1.1228.g56895c6ee86
#39 lore raw

PATCH v4 3/9 Makefile: disable GNU make built-in wildcard rules

In reply to Ævar Arnfjörð Bjarmason

Override built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.

The speed-up may vary. I've seen 1-10% depending on the speed of the local disk, caches, -jN etc. Running:

strace -f -c -S calls make -j1 NO_TCLTK=Y

Shows that we reduce the number of syscalls we make, mostly in "stat" calls.

We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early. Doing so might make us a bit faster still. But doing so is a much bigger hammer, since it will disable all built-in rules, some (all?) of which can be seen with:

    make -f/dev/null -p | grep -v -e ^# -e ^$

We may have something that relies on them, so let's go for the more isolated optimization here that gives us most or all of the wins.

1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html

---
 shared.mak | 11 +++++++++++
 1 file changed, 11 insertions(+)
shared.mak +11 -0
index 0170bb397ae..29f0e69ecb9 100644
@@ -1,3 +1,14 @@
+### Remove GNU make implicit rules
+
+## This speeds things up since we don't need to look for and stat() a
+## "foo.c,v" every time a rule referring to "foo.c" is in play. See
+## "make -p -f/dev/null | grep ^%::'".
+%:: %,v
+%:: RCS/%,v
+%:: RCS/%
+%:: s.%
+%:: SCCS/s.%
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1228.g56895c6ee86
#40 lore raw

PATCH v4 4/9 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)

In reply to Ævar Arnfjörð Bjarmason

Combine the definitions of $(FIND_SOURCE_FILES) and $(LIB_H) to speed up the Makefile, as these are the two main expensive $(shell) commands that we execute unconditionally.

When see what was in $(FOUND_SOURCE_FILES) that wasn't in $(LIB_H) via the ad-hoc test of:

    $(error $(filter-out $(LIB_H),$(filter %.h,$(ALL_SOURCE_FILES))))
    $(error $(filter-out $(ALL_SOURCE_FILES),$(filter %.h,$(LIB_H))))

We'll get, respectively:

    Makefile:850: *** t/helper/test-tool.h.  Stop.
    Makefile:850: *** .  Stop.

I.e. we only had a discrepancy when it came to t/helper/test-tool.h. In terms of correctness this was broken before, but now works:

    $ make t/helper/test-tool.hco
        HDR t/helper/test-tool.h

This speeds things up a lot:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10
    Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1
      Time (mean ± σ):     159.9 ms ±   6.8 ms    [User: 137.2 ms, System: 28.0 ms]
      Range (min … max):   154.6 ms … 175.9 ms    10 runs
    Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0
      Time (mean ± σ):     100.0 ms ±   1.3 ms    [User: 84.2 ms, System: 20.2 ms]
      Range (min … max):    98.8 ms … 102.8 ms    10 runs

    Summary
      'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran
        1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1'
---
 Makefile | 54 ++++++++++++++++++++++++++----------------------------
 1 file changed, 26 insertions(+), 28 deletions(-)
Makefile +26 -28
index aa5ff001b6e..1ac924bd844 100644
@@ -833,12 +833,33 @@ GENERATED_H += hook-list.h
 .PHONY: generated-hdrs
 generated-hdrs: $(GENERATED_H)
 
-LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \
+## Exhaustive lists of our source files, either dynamically generated,
+## or hardcoded.
+SOURCES_CMD = ( \
+	git ls-files \
+		'*.[hcS]' \
+		'*.sh' \
+		':!*[tp][0-9][0-9][0-9][0-9]*' \
+		':!contrib' \
+		2>/dev/null || \
 	$(FIND) . \
-	-name .git -prune -o \
-	-name t -prune -o \
-	-name Documentation -prune -o \
-	-name '*.h' -print)))
+		\( -name .git -type d -prune \) \
+		-o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
+		-o \( -name contrib -type d -prune \) \
+		-o \( -name build -type d -prune \) \
+		-o \( -name 'trash*' -type d -prune \) \
+		-o \( -name '*.[hcS]' -type f -print \) \
+		-o \( -name '*.sh' -type f -print \) \
+		| sed -e 's|^\./||' \
+	)
+FOUND_SOURCE_FILES := $(shell $(SOURCES_CMD))
+
+FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))
+FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))
+
+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
+
+LIB_H = $(FOUND_H_SOURCES)
 
 LIB_OBJS += abspath.o
 LIB_OBJS += add-interactive.o
@@ -2789,26 +2810,6 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
 	$(QUIET_GEN)mkdir -p $(dir $@) && \
 	pod2man $< $@
 
-FIND_SOURCE_FILES = ( \
-	git ls-files \
-		'*.[hcS]' \
-		'*.sh' \
-		':!*[tp][0-9][0-9][0-9][0-9]*' \
-		':!contrib' \
-		2>/dev/null || \
-	$(FIND) . \
-		\( -name .git -type d -prune \) \
-		-o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
-		-o \( -name contrib -type d -prune \) \
-		-o \( -name build -type d -prune \) \
-		-o \( -name 'trash*' -type d -prune \) \
-		-o \( -name '*.[hcS]' -type f -print \) \
-		-o \( -name '*.sh' -type f -print \) \
-		| sed -e 's|^\./||' \
-	)
-
-FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES))
-
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
 	echo $(FOUND_SOURCE_FILES) | xargs etags -a -o $@+ && \
@@ -3018,9 +3019,6 @@ check: $(GENERATED_H)
 		exit 1; \
 	fi
 
-FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))
-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
-
 %.cocci.patch: %.cocci $(COCCI_SOURCES)
 	$(QUIET_SPATCH) \
 	if test $(SPATCH_BATCH_SIZE) = 0; then \
Signature · 1 line
2.35.1.1228.g56895c6ee86
#41 lore raw

PATCH v4 5/9 Makefile: move ".SUFFIXES" rule to shared.mak

In reply to Ævar Arnfjörð Bjarmason

This was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.

This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
    Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
      Time (mean ± σ):     121.7 ms ±   8.8 ms    [User: 105.8 ms, System: 18.6 ms]
      Range (min … max):   112.8 ms … 148.4 ms    26 runs

Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs

Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.

So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.

1. https://lore.kernel.org/git/[email protected]/

---
 Makefile   | 2 --
 shared.mak | 5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
index 1ac924bd844..ce362720947 100644
@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S))
 ASM_OBJ := $(ASM_SRC:S=o)
 C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS))
 
-.SUFFIXES:
-
 $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
 	$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
 $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +5 -0
index 29f0e69ecb9..1dda948df09 100644
@@ -9,6 +9,11 @@
 %:: s.%
 %:: SCCS/s.%
 
+## Likewise delete default $(SUFFIXES). See:
+##
+##     info make --index-search=.SUFFIXES
+.SUFFIXES:
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1228.g56895c6ee86
#42 lore raw

PATCH v4 6/9 Makefile: move $(comma), $(empty) and $(space) to shared.mak

In reply to Ævar Arnfjörð Bjarmason

Move these variables over to the shared.mak, we'll make use of them in a subsequent commit.

Note that there's reason for these to be "simply expanded variables", i.e. to use ":=" assignments instead of lazily expanded "=" assignments. We could use "=", but let's leave this as-is for now for ease of review.

See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.

---
 Makefile   | 4 ----
 shared.mak | 8 ++++++++
 2 files changed, 8 insertions(+), 4 deletions(-)
Makefile +0 -4
index ce362720947..1e7a1277ce9 100644
@@ -1289,10 +1289,6 @@ endif
 ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS)
 ALL_LDFLAGS = $(LDFLAGS)
 
-comma := ,
-empty :=
-space := $(empty) $(empty)
-
 ifdef SANITIZE
 SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag))
 BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)
shared.mak +8 -0
index 1dda948df09..934bf428936 100644
@@ -23,3 +23,11 @@
 #
 #    info make --index-search=.DELETE_ON_ERROR
 .DELETE_ON_ERROR:
+
+### Global variables
+
+## comma, empty, space: handy variables as these tokens are either
+## special or can be hard to spot among other Makefile syntax.
+comma := ,
+empty :=
+space := $(empty) $(empty)
Signature · 1 line
2.35.1.1228.g56895c6ee86
#43 lore raw

PATCH v4 7/9 Makefile: add "$(QUIET)" boilerplate to shared.mak

In reply to Ævar Arnfjörð Bjarmason

The $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.

Since we're not using the environment to pass these around we don't need to export the "QUIET_GEN" and "QUIET_BUILT_IN" variables anymore. The "QUIET_GEN" variable is used in "git-gui/Makefile" and "gitweb/Makefile", but they've got their own definition for those. The "QUIET_BUILT_IN" variable is only used in the top-level "Makefile". We still need to export the "V" variable.

---
 Documentation/Makefile  | 32 -------------------------
 Makefile                | 33 -------------------------
 config.mak.uname        |  1 -
 contrib/scalar/Makefile | 14 -----------
 shared.mak              | 53 +++++++++++++++++++++++++++++++++++++++++
 templates/Makefile      |  5 ----
 6 files changed, 53 insertions(+), 85 deletions(-)
Documentation/Makefile +0 -32
index ba27456c86a..0f4ebdeda8a 100644
@@ -218,38 +218,6 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR))
 ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)'
 endif
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring $(MAKEFLAGS),w),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring $(MAKEFLAGS),s),s)
-ifndef V
-	QUIET		= @
-	QUIET_ASCIIDOC	= @echo '   ' ASCIIDOC $@;
-	QUIET_XMLTO	= @echo '   ' XMLTO $@;
-	QUIET_DB2TEXI	= @echo '   ' DB2TEXI $@;
-	QUIET_MAKEINFO	= @echo '   ' MAKEINFO $@;
-	QUIET_DBLATEX	= @echo '   ' DBLATEX $@;
-	QUIET_XSLTPROC	= @echo '   ' XSLTPROC $@;
-	QUIET_GEN	= @echo '   ' GEN $@;
-	QUIET_STDERR	= 2> /dev/null
-	QUIET_SUBDIR0	= +@subdir=
-	QUIET_SUBDIR1	= ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			  $(MAKE) $(PRINT_DIR) -C $$subdir
-
-	QUIET_LINT_GITLINK	= @echo '   ' LINT GITLINK $<;
-	QUIET_LINT_MANSEC	= @echo '   ' LINT MAN SEC $<;
-	QUIET_LINT_MANEND	= @echo '   ' LINT MAN END $<;
-
-	export V
-endif
-endif
-
 all: html man
 
 html: $(DOC_HTML)
Makefile +0 -33
index 1e7a1277ce9..d4c0f4ca7b8 100644
@@ -2001,39 +2001,6 @@ ifndef PAGER_ENV
 PAGER_ENV = LESS=FRX LV=-c
 endif
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring w,$(MAKEFLAGS)),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring s,$(MAKEFLAGS)),s)
-ifndef V
-	QUIET_CC       = @echo '   ' CC $@;
-	QUIET_AR       = @echo '   ' AR $@;
-	QUIET_LINK     = @echo '   ' LINK $@;
-	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
-	QUIET_GEN      = @echo '   ' GEN $@;
-	QUIET_LNCP     = @echo '   ' LN/CP $@;
-	QUIET_XGETTEXT = @echo '   ' XGETTEXT $@;
-	QUIET_MSGFMT   = @echo '   ' MSGFMT $@;
-	QUIET_GCOV     = @echo '   ' GCOV $@;
-	QUIET_SP       = @echo '   ' SP $<;
-	QUIET_HDR      = @echo '   ' HDR $(<:hcc=h);
-	QUIET_RC       = @echo '   ' RC $@;
-	QUIET_SPATCH   = @echo '   ' SPATCH $<;
-	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			 $(MAKE) $(PRINT_DIR) -C $$subdir
-	export V
-	export QUIET_GEN
-	export QUIET_BUILT_IN
-endif
-endif
-
 ifdef NO_INSTALL_HARDLINKS
 	export NO_INSTALL_HARDLINKS
 endif
config.mak.uname +0 -1
index 4352ea39e9b..7727b707b74 100644
@@ -727,7 +727,6 @@ vcxproj:
 	git diff-index --cached --quiet HEAD --
 
 	# Make .vcxproj files and add them
-	unset QUIET_GEN QUIET_BUILT_IN; \
 	perl contrib/buildsystems/generate -g Vcxproj
 	git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
 
contrib/scalar/Makefile +0 -14
index d585b319edc..5e86d78e19b 100644
@@ -8,20 +8,6 @@ include ../../config.mak.uname
 -include ../../config.mak.autogen
 -include ../../config.mak
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring s,$(MAKEFLAGS)),s)
-ifndef V
-	QUIET_GEN      = @echo '   ' GEN $@;
-	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			 $(MAKE) $(PRINT_DIR) -C $$subdir
-else
-	export V
-endif
-endif
-
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
shared.mak +53 -0

56 lines not on this page · raw message

templates/Makefile +0 -5
index 636cee52f51..367ad00c24c 100644
@@ -2,11 +2,6 @@
 include ../shared.mak
 
 # make and install sample templates
-
-ifndef V
-	QUIET = @
-endif
-
 INSTALL ?= install
 TAR ?= tar
 RM ?= rm -f
Signature · 1 line
2.35.1.1228.g56895c6ee86
#44 lore raw

PATCH v4 8/9 Makefile: use $(wspfx) for $(QUIET...) in shared.mak

In reply to Ævar Arnfjörð Bjarmason

Change the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding. This refactoring will make it easier to emit that exact amount of padding in functions that we might add to shared.mak in the future.

Such a function is not part of this patch series, but a "TRACK_template" that I'd like to add as a follow-up to it makes use of this. Let's make this change now while modifying these QUIET rules is fresh in our minds.

---
 shared.mak | 54 ++++++++++++++++++++++++++++++------------------------
 1 file changed, 30 insertions(+), 24 deletions(-)
shared.mak +30 -24
index c45b2812eb6..bb910273538 100644
@@ -32,6 +32,12 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
+## wspfx: the whitespace prefix padding for $(QUIET...) and similarly
+## aligned output.
+wspfx = $(space)$(space)$(space)
+wspfx_SQ = '$(subst ','\'',$(wspfx))'
+# ' closing quote to appease Emacs make-mode.elxo
+
 ### Quieting
 ## common
 QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
@@ -47,39 +53,39 @@ ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
 ## common
 	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo $(wspfx_SQ) SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
 
 	QUIET          = @
-	QUIET_GEN      = @echo '   ' GEN $@;
+	QUIET_GEN      = @echo $(wspfx_SQ) GEN $@;
 
 ## Used in "Makefile"
-	QUIET_CC       = @echo '   ' CC $@;
-	QUIET_AR       = @echo '   ' AR $@;
-	QUIET_LINK     = @echo '   ' LINK $@;
-	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
-	QUIET_LNCP     = @echo '   ' LN/CP $@;
-	QUIET_XGETTEXT = @echo '   ' XGETTEXT $@;
-	QUIET_MSGFMT   = @echo '   ' MSGFMT $@;
-	QUIET_GCOV     = @echo '   ' GCOV $@;
-	QUIET_SP       = @echo '   ' SP $<;
-	QUIET_HDR      = @echo '   ' HDR $(<:hcc=h);
-	QUIET_RC       = @echo '   ' RC $@;
-	QUIET_SPATCH   = @echo '   ' SPATCH $<;
+	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
+	QUIET_AR       = @echo $(wspfx_SQ) AR $@;
+	QUIET_LINK     = @echo $(wspfx_SQ) LINK $@;
+	QUIET_BUILT_IN = @echo $(wspfx_SQ) BUILTIN $@;
+	QUIET_LNCP     = @echo $(wspfx_SQ) LN/CP $@;
+	QUIET_XGETTEXT = @echo $(wspfx_SQ) XGETTEXT $@;
+	QUIET_MSGFMT   = @echo $(wspfx_SQ) MSGFMT $@;
+	QUIET_GCOV     = @echo $(wspfx_SQ) GCOV $@;
+	QUIET_SP       = @echo $(wspfx_SQ) SP $<;
+	QUIET_HDR      = @echo $(wspfx_SQ) HDR $(<:hcc=h);
+	QUIET_RC       = @echo $(wspfx_SQ) RC $@;
+	QUIET_SPATCH   = @echo $(wspfx_SQ) SPATCH $<;
 
 ## Used in "Documentation/Makefile"
-	QUIET_ASCIIDOC	= @echo '   ' ASCIIDOC $@;
-	QUIET_XMLTO	= @echo '   ' XMLTO $@;
-	QUIET_DB2TEXI	= @echo '   ' DB2TEXI $@;
-	QUIET_MAKEINFO	= @echo '   ' MAKEINFO $@;
-	QUIET_DBLATEX	= @echo '   ' DBLATEX $@;
-	QUIET_XSLTPROC	= @echo '   ' XSLTPROC $@;
-	QUIET_GEN	= @echo '   ' GEN $@;
+	QUIET_ASCIIDOC	= @echo $(wspfx_SQ) ASCIIDOC $@;
+	QUIET_XMLTO	= @echo $(wspfx_SQ) XMLTO $@;
+	QUIET_DB2TEXI	= @echo $(wspfx_SQ) DB2TEXI $@;
+	QUIET_MAKEINFO	= @echo $(wspfx_SQ) MAKEINFO $@;
+	QUIET_DBLATEX	= @echo $(wspfx_SQ) DBLATEX $@;
+	QUIET_XSLTPROC	= @echo $(wspfx_SQ) XSLTPROC $@;
+	QUIET_GEN	= @echo $(wspfx_SQ) GEN $@;
 	QUIET_STDERR	= 2> /dev/null
 
-	QUIET_LINT_GITLINK	= @echo '   ' LINT GITLINK $<;
-	QUIET_LINT_MANSEC	= @echo '   ' LINT MAN SEC $<;
-	QUIET_LINT_MANEND	= @echo '   ' LINT MAN END $<;
+	QUIET_LINT_GITLINK	= @echo $(wspfx_SQ) LINT GITLINK $<;
+	QUIET_LINT_MANSEC	= @echo $(wspfx_SQ) LINT MAN SEC $<;
+	QUIET_LINT_MANEND	= @echo $(wspfx_SQ) LINT MAN END $<;
 
 	export V
 endif
Signature · 1 line
2.35.1.1228.g56895c6ee86
#45 lore raw

Re: [PATCH v4 8/9] Makefile: use $(wspfx) for $(QUIET...) in shared.mak

Quoted 9 lines · Ævar Arnfjörð Bjarmason

Change the mostly move-only change in the preceding commit to use the $(wspfx) variable for defining the QUIET padding. This refactoring will make it easier to emit that exact amount of padding in functions that we might add to shared.mak in the future.

Such a function is not part of this patch series, but a "TRACK_template" that I'd like to add as a follow-up to it makes use of this. Let's make this change now while modifying these QUIET rules is fresh in our minds.

I do not see why this change is still part of this series, then.

shared.mak ifneq ($(findstring s,$(MAKEFLAGS)),s) · line 64

Quoted 7 lines

## Used in "Makefile"

shared.mak +2 -2
-	QUIET_CC       = @echo '   ' CC $@;
-	QUIET_AR       = @echo '   ' AR $@;
...
+	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
+	QUIET_AR       = @echo $(wspfx_SQ) AR $@;
...

Look how much uglier these lines got.

Without the benefit TRACK_template brings us, whatever that is, this change looks more of an unnecessary churn. Let's leave it to the later series that introduces the user that justifies this change.

#46 lore raw

PATCH v4 9/9 Makefiles: add and use wildcard "mkdir -p" template

In reply to Ævar Arnfjörð Bjarmason

Add a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).

As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.

I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.

But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
    Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
      Time (mean ± σ):      2.914 s ±  0.062 s    [User: 2.449 s, System: 0.489 s]
      Range (min … max):    2.834 s …  3.020 s    10 runs

Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s]
Range (min … max): 2.229 s … 2.397 s 10 runs

Summary
'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'

So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.

This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://gitlab.com/avar/git-hyperfine/

---
 Documentation/Makefile | 25 +++----------------------
 Makefile               | 12 +++++++-----
 shared.mak             | 17 +++++++++++++++++
 3 files changed, 27 insertions(+), 27 deletions(-)
Documentation/Makefile +3 -22
index 0f4ebdeda8a..1eb9192dae8 100644
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo
 print-man1:
 	@for i in $(MAN1_TXT); do echo $$i; done
 
-## Lint: Common
-.build:
-	$(QUIET)mkdir $@
-.build/lint-docs: | .build
-	$(QUIET)mkdir $@
-
 ## Lint: gitlink
-.build/lint-docs/gitlink: | .build/lint-docs
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
 LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT))
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config
 $(LINT_DOCS_GITLINK): lint-gitlink.perl
 $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \
 		$< \
 		$(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
 lint-docs-gitlink: $(LINT_DOCS_GITLINK)
 
 ## Lint: man-end-blurb
-.build/lint-docs/man-end-blurb: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb
 $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl
 $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@
 .PHONY: lint-docs-man-end-blurb
-lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB)
 
 ## Lint: man-section-order
-.build/lint-docs/man-section-order: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order
 $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl
 $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@
 .PHONY: lint-docs-man-section-order
 lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
Makefile +7 -5
index d4c0f4ca7b8..12f08411d2d 100644
@@ -2745,7 +2745,8 @@ all:: $(MOFILES)
 endif
 
 po/build/locale/%/LC_MESSAGES/git.mo: po/%.po
-	$(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $<
+	$(call mkdir_p_parent_template)
+	$(QUIET_MSGFMT)$(MSGFMT) -o $@ $<
 
 LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm)
 LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2761,15 +2762,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS))
 endif
 
 perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN) \
 	sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \
 	    -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \
 	    -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \
 	< $< > $@
 
 perl/build/man/man3/Git.3pm: perl/Git.pm
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
-	pod2man $< $@
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN)pod2man $< $@
 
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
@@ -2903,7 +2905,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(
 all:: $(TEST_PROGRAMS) $(test_bindir_programs)
 
 bin-wrappers/%: wrap-for-bin.sh
-	@mkdir -p bin-wrappers
+	$(call mkdir_p_parent_template)
 	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
 	     -e 's|@@BUILD_DIR@@|$(shell pwd)|' \
 	     -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
shared.mak +17 -0
index bb910273538..83c22d5e7b3 100644
@@ -59,6 +59,8 @@ ifndef V
 	QUIET          = @
 	QUIET_GEN      = @echo $(wspfx_SQ) GEN $@;
 
+	QUIET_MKDIR_P_PARENT  = @echo $(wspfx_SQ) MKDIR -p $(@D);
+
 ## Used in "Makefile"
 	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
 	QUIET_AR       = @echo $(wspfx_SQ) AR $@;
@@ -90,3 +92,18 @@ ifndef V
 	export V
 endif
 endif
+
+### Templates
+
+## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@
+## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
+## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a
+## "a/prefix/dir/file". This can instead be inserted at the start of
+## the "a/prefix/dir/file" rule.
+define mkdir_p_parent_template
+$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D)))
+endef
Signature · 1 line
2.35.1.1228.g56895c6ee86
#47 lore raw

Re: [PATCH v4 9/9] Makefiles: add and use wildcard "mkdir -p" template

Quoted 3 lines · Ævar Arnfjörð Bjarmason

Add a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).

Sounds good.

Quoted 5 lines

But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):

Cute.

Quoted 14 lines
shared.mak +2 -0
index bb910273538..83c22d5e7b3 100644
@@ -59,6 +59,8 @@ ifndef V
 	QUIET          = @
 	QUIET_GEN      = @echo $(wspfx_SQ) GEN $@;
 
+	QUIET_MKDIR_P_PARENT  = @echo $(wspfx_SQ) MKDIR -p $(@D);
+
 ## Used in "Makefile"
 	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
 	QUIET_AR       = @echo $(wspfx_SQ) AR $@;

shared.mak ifndef V · line 94

Quoted 3 lines
shared.mak +0 -0
@@ -90,3 +92,18 @@ ifndef V
 	export V
 endif
 endif

It is unfortunate that this came after [8/9] and needs manual resolution to reorder to leave 8/9 for later series.

#48 lore raw
In reply to Ævar Arnfjörð Bjarmason
Quoted 7 lines · Ævar Arnfjörð Bjarmason

As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/

This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).

Other than a few things I noticed and commented on, 1-7 and 9 looked all sensible.

Thanks.

#49 lore raw
In reply to Ævar Arnfjörð Bjarmason
Quoted 62 lines · Ævar Arnfjörð Bjarmason

As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/

This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).

This v4 re-roll (for v3, see
https://lore.kernel.org/git/[email protected]/):

* The "all" boilerplate goes first now, before "include"
* Elaborated on DELETE_ON_ERROR ina commit message.
* No longer change a ":=" to "=" while moving code.
* Rephrased other commit messages (one of which referred to a function from another future series)
* Typo fix in commit message.

Ævar Arnfjörð Bjarmason (9):
scalar Makefile: use "The default target of..." pattern
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefile: use $(wspfx) for $(QUIET...) in shared.mak
Makefiles: add and use wildcard "mkdir -p" template

  Documentation/Makefile    |  63 ++------------------
  Makefile                  | 118 ++++++++++++--------------------------
  config.mak.uname          |   1 -
  contrib/scalar/Makefile   |  20 ++-----
  contrib/scalar/t/Makefile |   3 +
  shared.mak                | 109 +++++++++++++++++++++++++++++++++++
  t/Makefile                |   3 +
  t/interop/Makefile        |   3 +
  t/perf/Makefile           |   3 +
  templates/Makefile        |   8 +--
  10 files changed, 171 insertions(+), 160 deletions(-)

create mode 100644 shared.mak

Range-diff against v3:

  1:  2404c4d8b96 <  -:  ----------- scalar Makefile: set the default target after the includes
  -:  ----------- >  1:  26c6bb897cf scalar Makefile: use "The default target of..." pattern
  2:  96a490bec54 !  2:  74692458b70 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

@@ Commit message
.DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.

- This does have the potential downside that if e.g. templates/Makefile
- would like to include this "shared.mak" in the future the semantics of
- such a Makefile will change, but as noted in the above commits (and
- GNU make's own documentation) any such change would be for the better,
- so it's safe to do this.
+ I.e. this changes the behavior of existing rules in the altered
+ Makefiles (except "Makefile" & "Documentation/Makefile"). I'm
+ confident that this is safe having read the relevant rules in those
+ Makfiles, and as the GNU make manual notes that it isn't the default
+ behavior is out of an abundance of backwards compatibility
+ caution. From edition 0.75 of its manual, covering GNU make 4.3:

Quoted 8 lines

- This also doesn't introduce a bug by e.g. having this
+ [Enabling '.DELETE_ON_ERROR' is] almost always what you want
+ 'make' to do, but it is not historical practice; so for
+ compatibility, you must explicitly request it.
+
+ This doesn't introduce a bug by e.g. having this
".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

I think this is much clearer now,

Best Wishes

Phillip

Quoted 162 lines
     +    It does increase the danger that any Makefile without an explicit "The
     +    default target of this Makefile is..." snippet to define the default
     +    target as "all" could have its default rule changed if our new
     +    shared.mak ever defines a "real" rule. In subsequent commits we'll be
     +    careful not to do that, and such breakage would be obvious e.g. in the
     +    case of "make -C t".

+
+ We might want to make that less fragile still (e.g. by using
+ ".DEFAULT_GOAL" as noted in the preceding commit), but for now let's
+ simply include "shared.mak" without adding that boilerplate to all the
+ Makefiles that don't have it already. Most of those are already
+ exposed to that potential caveat e.g. due to including "config.mak*".
+

## Documentation/Makefile ##
@@ Documentation/Makefile: doc-l10n install-l10n::

## Makefile ##
@@
-+# Import tree-wide shared Makefile behavior and libraries -+include shared.mak
-+
# The default target of this Makefile is... all::

++# Import tree-wide shared Makefile behavior and libraries
++include shared.mak
++
+ # Define V=1 to have a more verbose compile.
+ #
+ # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
@@ Makefile: shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $^
@@ Makefile: shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell

## contrib/scalar/Makefile ##
@@
+ # The default target of this Makefile is...
+ all::
+
+# Import tree-wide shared Makefile behavior and libraries +include ../../shared.mak
+
- QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
- QUIET_SUBDIR1 =
-
+ include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak

## contrib/scalar/t/Makefile ##
@@
3: 9392e3c3e97 ! 3: 0fbdeeffc7b Makefile: disable GNU make built-in wildcard rules
@@ Commit message
contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.

- The speed-up may wary. I've seen 1-10% depending on the speed of the
+ The speed-up may vary. I've seen 1-10% depending on the speed of the
local disk, caches, -jN etc. Running:

strace -f -c -S calls make -j1 NO_TCLTK=Y

  4:  07cf9daa9d6 =  4:  ea6b835308a Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
  5:  16f2e3ff35b =  5:  c2339694cf7 Makefile: move ".SUFFIXES" rule to shared.mak
  6:  1b6ecb27f02 !  6:  741fdfd48e2 Makefile: move $(comma), $(empty) and $(space) to shared.mak

@@ Metadata
## Commit message ##
Makefile: move $(comma), $(empty) and $(space) to shared.mak

- Move these variables over to the shared.max, we'll make use of them in
- a subsequent commit. There was no reason for these to be "simply
- expanded variables", so let's use the normal lazy "=" assignment here.
+ Move these variables over to the shared.mak, we'll make use of them in
+ a subsequent commit.
+
+ Note that there's reason for these to be "simply expanded variables",
+ i.e. to use ":=" assignments instead of lazily expanded "="
+ assignments. We could use "=", but let's leave this as-is for now for
+ ease of review.

See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.
@@ shared.mak
+
+## comma, empty, space: handy variables as these tokens are either +## special or can be hard to spot among other Makefile syntax.
-+comma = ,
-+empty =
-+space = $(empty) $(empty)
++comma := ,
++empty :=
++space := $(empty) $(empty)
7: 471067deefc ! 7: a723cbce270 Makefile: add "$(QUIET)" boilerplate to shared.mak
@@ config.mak.uname: vcxproj:

## contrib/scalar/Makefile ##
-@@
- # Import tree-wide shared Makefile behavior and libraries
- include ../../shared.mak
+@@ contrib/scalar/Makefile: include ../../config.mak.uname
+ -include ../../config.mak.autogen
+ -include ../../config.mak

-QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir -QUIET_SUBDIR1 =
@@ contrib/scalar/Makefile
-endif
-endif
-
- include ../../config.mak.uname
- -include ../../config.mak.autogen
- -include ../../config.mak
+ TARGETS = scalar$(X) scalar.o
+ GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a +

## shared.mak ##
@@
- comma = ,
- empty =
- space = $(empty) $(empty)
+ comma := ,
+ empty :=
+ space := $(empty) $(empty)
+
+### Quieting
+## common
8: 510306d2219 ! 8: 3733b0c8df1 Makefile: use $(wspfx) for $(QUIET...) in shared.mak
@@ Commit message
Makefile: use $(wspfx) for $(QUIET...) in shared.mak

Change the mostly move-only change in the preceding commit to use the
- $(wspfx) variable for defining the QUIET padding, to guarantee that
- it's consistent with the "TRACK_template" template.
+ $(wspfx) variable for defining the QUIET padding. This refactoring
+ will make it easier to emit that exact amount of padding in functions
+ that we might add to shared.mak in the future.

- $ make CFLAGS=-I$RANDOM grep.o wspfx='$(space)->'
- -> GIT-CFLAGS PARAMETERS (changed)
- -> CC grep.o
+ Such a function is not part of this patch series, but a
+ "TRACK_template" that I'd like to add as a follow-up to it makes use
+ of this. Let's make this change now while modifying these QUIET rules
+ is fresh in our minds.

## shared.mak ##
-@@ shared.mak: comma = ,
- empty =
- space = $(empty) $(empty)
+@@ shared.mak: comma := ,
+ empty :=
+ space := $(empty) $(empty)

+## wspfx: the whitespace prefix padding for $(QUIET...) and similarly +## aligned output.
9: 85bb74aa32f = 9: 4cc4aeabb20 Makefiles: add and use wildcard "mkdir -p" template

#50 lore raw

v5 cover Makefile: optimize noop runs, add shared.mak

In reply to Ævar Arnfjörð Bjarmason

As the v1 notes (among other things):
https://lore.kernel.org/git/[email protected]/

This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter).

This v5 re-roll (for v3, see:
https://lore.kernel.org/git/[email protected]/):

* Drops the "wspfx" patch.
* Rephrases the 1/8 commit message, dropping the mention of .DEFAULT_GOAL etc.

I think this should be ready to advance now per Junio's comment on v4
(https://lore.kernel.org/git/[email protected]/):

Other than a few things I noticed and commented on, 1-7 and 9 looked all sensible.

Ævar Arnfjörð Bjarmason (8):
scalar Makefile: use "The default target of..." pattern
Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
Makefile: disable GNU make built-in wildcard rules
Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
Makefile: move ".SUFFIXES" rule to shared.mak
Makefile: move $(comma), $(empty) and $(space) to shared.mak
Makefile: add "$(QUIET)" boilerplate to shared.mak
Makefiles: add and use wildcard "mkdir -p" template

 Documentation/Makefile    |  63 ++------------------
 Makefile                  | 118 ++++++++++++--------------------------
 config.mak.uname          |   1 -
 contrib/scalar/Makefile   |  20 ++-----
 contrib/scalar/t/Makefile |   3 +
 shared.mak                | 103 +++++++++++++++++++++++++++++++++
 t/Makefile                |   3 +
 t/interop/Makefile        |   3 +
 t/perf/Makefile           |   3 +
 templates/Makefile        |   8 +--
 10 files changed, 165 insertions(+), 160 deletions(-)
 create mode 100644 shared.mak

Range-diff against v4:
1: 26c6bb897cf ! 1: 7547bf3e481 scalar Makefile: use "The default target of..." pattern
@@ Metadata
## Commit message ##
scalar Makefile: use "The default target of..." pattern

- Make have the "contrib/scalar/Makefile" be stylistically consistent
- with the top-level "Makefile" in first declaring "all" to be the
- default rule, follwed by including other Makefile snippets.
+ Make the "contrib/scalar/Makefile" be stylistically consistent with
+ the top-level "Makefile" in first declaring "all" to be the default
+ rule, followed by including other Makefile snippets.

This adjusts code added in 0a43fb22026 (scalar: create a rudimentary
- executable, 2021-12-03), it's a style-only change, in a subsequent
- commit the "QUIET" boilerplate at the beginning of this file will be
- retrieved via an include, and having an "all:" between the two set of
- "include"'s after that change would look odd.
-
- As noted in [1] using ".DEFAULT_GOAL = all" is another way to do this
- in more modern GNU make versions, which we already have a hard
- dependency on, but let's leave any such change for a future
- improvement and go with using our established pattern consistently for
- now.
-
- 1. https://lore.kernel.org/git/[email protected]/
+ executable, 2021-12-03), it further ensures that when we add another
+ "include" file in a subsequent commit that the included file won't be
+ the one to define our default target.

 2:  74692458b70 =  2:  91795eccc32 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it
 3:  0fbdeeffc7b =  3:  9f42f40b518 Makefile: disable GNU make built-in wildcard rules
 4:  ea6b835308a =  4:  034779ff7fb Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)
 5:  c2339694cf7 =  5:  18e0a6985f1 Makefile: move ".SUFFIXES" rule to shared.mak
 6:  741fdfd48e2 =  6:  422dee02ae9 Makefile: move $(comma), $(empty) and $(space) to shared.mak
 7:  a723cbce270 =  7:  21bf1e6e01c Makefile: add "$(QUIET)" boilerplate to shared.mak
 8:  3733b0c8df1 <  -:  ----------- Makefile: use $(wspfx) for $(QUIET...) in shared.mak
 9:  4cc4aeabb20 !  8:  e6a93cae81d Makefiles: add and use wildcard "mkdir -p" template

@@ Makefile: test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_
## shared.mak ##
@@ shared.mak: ifndef V
QUIET = @
- QUIET_GEN = @echo $(wspfx_SQ) GEN $@;
+ QUIET_GEN = @echo ' ' GEN $@;

+ QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D);
+
## Used in "Makefile"
- QUIET_CC = @echo $(wspfx_SQ) CC $@;
- QUIET_AR = @echo $(wspfx_SQ) AR $@;
+ QUIET_CC = @echo ' ' CC $@;
+ QUIET_AR = @echo ' ' AR $@;
@@ shared.mak: ifndef V
export V
endif

Signature · 1 line
2.35.1.1230.ga6e6579e98c
#51 lore raw

PATCH v5 1/8 scalar Makefile: use "The default target of..." pattern

Make the "contrib/scalar/Makefile" be stylistically consistent with the top-level "Makefile" in first declaring "all" to be the default rule, followed by including other Makefile snippets.

This adjusts code added in 0a43fb22026 (scalar: create a rudimentary executable, 2021-12-03), it further ensures that when we add another "include" file in a subsequent commit that the included file won't be the one to define our default target.

---
 contrib/scalar/Makefile | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
contrib/scalar/Makefile +7 -6
index 231b1ee1796..1e9b969f3df 100644
@@ -1,3 +1,10 @@
+# The default target of this Makefile is...
+all::
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
 QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
 QUIET_SUBDIR1  =
 
@@ -12,12 +19,6 @@ else
 endif
 endif
 
-all:
-
-include ../../config.mak.uname
--include ../../config.mak.autogen
--include ../../config.mak
-
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
Signature · 1 line
2.35.1.1230.ga6e6579e98c
#52 lore raw

PATCH v5 2/8 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it

In reply to Ævar Arnfjörð Bjarmason

We have various behavior that's shared across our Makefiles, or that really should be (e.g. via defined templates). Let's create a top-level "shared.mak" to house those sorts of things, and start by adding the ".DELETE_ON_ERROR" flag to it.

See my own 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29) and db10fc6c09f (doc: simplify Makefile using .DELETE_ON_ERROR, 2021-05-21) for the addition and use of the ".DELETE_ON_ERROR" flag.

I.e. this changes the behavior of existing rules in the altered Makefiles (except "Makefile" & "Documentation/Makefile"). I'm confident that this is safe having read the relevant rules in those Makfiles, and as the GNU make manual notes that it isn't the default behavior is out of an abundance of backwards compatibility caution. From edition 0.75 of its manual, covering GNU make 4.3:

[Enabling '.DELETE_ON_ERROR' is] almost always what you want 'make' to do, but it is not historical practice; so for compatibility, you must explicitly request it.

This doesn't introduce a bug by e.g. having this ".DELETE_ON_ERROR" flag only apply to this new shared.mak, Makefiles have no such scoping semantics.

It does increase the danger that any Makefile without an explicit "The default target of this Makefile is..." snippet to define the default target as "all" could have its default rule changed if our new shared.mak ever defines a "real" rule. In subsequent commits we'll be careful not to do that, and such breakage would be obvious e.g. in the case of "make -C t".

We might want to make that less fragile still (e.g. by using ".DEFAULT_GOAL" as noted in the preceding commit), but for now let's simply include "shared.mak" without adding that boilerplate to all the Makefiles that don't have it already. Most of those are already exposed to that potential caveat e.g. due to including "config.mak*".

---
 Documentation/Makefile    |  6 +++---
 Makefile                  | 13 +++----------
 contrib/scalar/Makefile   |  3 +++
 contrib/scalar/t/Makefile |  3 +++
 shared.mak                |  9 +++++++++
 t/Makefile                |  3 +++
 t/interop/Makefile        |  3 +++
 t/perf/Makefile           |  3 +++
 templates/Makefile        |  3 +++
 9 files changed, 33 insertions(+), 13 deletions(-)
 create mode 100644 shared.mak
Documentation/Makefile +3 -3
index ed656db2ae9..ba27456c86a 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # Guard against environment variables
 MAN1_TXT =
 MAN5_TXT =
@@ -524,7 +527,4 @@ doc-l10n install-l10n::
 	$(MAKE) -C po $@
 endif
 
-# Delete the target file on error
-.DELETE_ON_ERROR:
-
 .PHONY: FORCE
Makefile +3 -10
index 6f0b4b775fe..aa5ff001b6e 100644
@@ -1,6 +1,9 @@
 # The default target of this Makefile is...
 all::
 
+# Import tree-wide shared Makefile behavior and libraries
+include shared.mak
+
 # Define V=1 to have a more verbose compile.
 #
 # Define SHELL_PATH to a POSIX shell if your /bin/sh is broken.
@@ -2194,16 +2197,6 @@ shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
 strip: $(PROGRAMS) git$X
 	$(STRIP) $(STRIP_OPTS) $^
 
-### Flags affecting all rules
-
-# A GNU make extension since gmake 3.72 (released in late 1994) to
-# remove the target of rules if commands in those rules fail. The
-# default is to only do that if make itself receives a signal. Affects
-# all targets, see:
-#
-#    info make --index-search=.DELETE_ON_ERROR
-.DELETE_ON_ERROR:
-
 ### Target-specific flags and dependencies
 
 # The generic compilation pattern rule and automatically
contrib/scalar/Makefile +3 -0
index 1e9b969f3df..d585b319edc 100644
@@ -1,6 +1,9 @@
 # The default target of this Makefile is...
 all::
 
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 include ../../config.mak.uname
 -include ../../config.mak.autogen
 -include ../../config.mak
contrib/scalar/t/Makefile +3 -0
index 6170672bb37..01e82e56d15 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../../shared.mak
+
 # Run scalar tests
 #
 # Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
shared.mak new file +9 -0
new file mode 100644
index 00000000000..0170bb397ae
@@ -0,0 +1,9 @@
+### Flags affecting all rules
+
+# A GNU make extension since gmake 3.72 (released in late 1994) to
+# remove the target of rules if commands in those rules fail. The
+# default is to only do that if make itself receives a signal. Affects
+# all targets, see:
+#
+#    info make --index-search=.DELETE_ON_ERROR
+.DELETE_ON_ERROR:
t/Makefile +3 -0
index 46cd5fc5273..056ce55dcc9 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # Run tests
 #
 # Copyright (c) 2005 Junio C Hamano
t/interop/Makefile +3 -0
index 31a4bbc716a..6911c2915a7 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 -include ../../config.mak
 export GIT_TEST_OPTIONS
 
t/perf/Makefile +3 -0
index 2465770a782..e4808aebed0 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../../shared.mak
+
 -include ../../config.mak
 export GIT_TEST_OPTIONS
 
templates/Makefile +3 -0
index d22a71a3999..636cee52f51 100644
@@ -1,3 +1,6 @@
+# Import tree-wide shared Makefile behavior and libraries
+include ../shared.mak
+
 # make and install sample templates
 
 ifndef V
Signature · 1 line
2.35.1.1230.ga6e6579e98c
#53 lore raw

PATCH v5 3/8 Makefile: disable GNU make built-in wildcard rules

In reply to Ævar Arnfjörð Bjarmason

Override built-in rules of GNU make that use a wildcard target. This can speeds things up significantly as we don't need to stat() so many files. GNU make does that by default to see if it can retrieve their contents from RCS or SCCS. See [1] for an old mailing list discussion about how to disable these.

The speed-up may vary. I've seen 1-10% depending on the speed of the local disk, caches, -jN etc. Running:

strace -f -c -S calls make -j1 NO_TCLTK=Y

Shows that we reduce the number of syscalls we make, mostly in "stat" calls.

We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early. Doing so might make us a bit faster still. But doing so is a much bigger hammer, since it will disable all built-in rules, some (all?) of which can be seen with:

    make -f/dev/null -p | grep -v -e ^# -e ^$

We may have something that relies on them, so let's go for the more isolated optimization here that gives us most or all of the wins.

1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html

---
 shared.mak | 11 +++++++++++
 1 file changed, 11 insertions(+)
shared.mak +11 -0
index 0170bb397ae..29f0e69ecb9 100644
@@ -1,3 +1,14 @@
+### Remove GNU make implicit rules
+
+## This speeds things up since we don't need to look for and stat() a
+## "foo.c,v" every time a rule referring to "foo.c" is in play. See
+## "make -p -f/dev/null | grep ^%::'".
+%:: %,v
+%:: RCS/%,v
+%:: RCS/%
+%:: s.%
+%:: SCCS/s.%
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1230.ga6e6579e98c
#54 lore raw

Re: [PATCH v5 3/8] Makefile: disable GNU make built-in wildcard rules

On Thu,  Mar 03, 2022 at 05:04:14PM +0100, Ævar Arnfjörð Bjarmason wrote:

[...]

Quoted 10 lines
---
 shared.mak | 11 +++++++++++
 1 file changed, 11 insertions(+)
shared.mak +0 -0
index 0170bb397ae..29f0e69ecb9 100644

shared.mak line 5

Quoted 5 lines
shared.mak +5 -0
@@ -1,3 +1,14 @@
+### Remove GNU make implicit rules
+
+## This speeds things up since we don't need to look for and stat() a
+## "foo.c,v" every time a rule referring to "foo.c" is in play. See
+## "make -p -f/dev/null | grep ^%::'".
                                  ^

A bit late to the party, but are you missing an opening single quote there?

Quoted 12 lines
shared.mak +6 -0
+%:: %,v
+%:: RCS/%,v
+%:: RCS/%
+%:: s.%
+%:: SCCS/s.%
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1230.ga6e6579e98c
#55 lore raw

Re: [PATCH v5 3/8] Makefile: disable GNU make built-in wildcard rules

Quoted 20 lines · Rene Kita

On Thu,  Mar 03, 2022 at 05:04:14PM +0100, Ævar Arnfjörð Bjarmason wrote:

[...]

Quoted 10 lines
---
 shared.mak | 11 +++++++++++
 1 file changed, 11 insertions(+)
shared.mak +0 -0
index 0170bb397ae..29f0e69ecb9 100644

shared.mak line 5

Quoted 5 lines
shared.mak +5 -0
@@ -1,3 +1,14 @@
+### Remove GNU make implicit rules
+
+## This speeds things up since we don't need to look for and stat() a
+## "foo.c,v" every time a rule referring to "foo.c" is in play. See
+## "make -p -f/dev/null | grep ^%::'".
                                  ^
Quoted 2 lines

A bit late to the party, but are you missing an opening single quote there?

Yes, that's a typo. Oops!

#56 lore raw

PATCH v5 4/8 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)

In reply to Ævar Arnfjörð Bjarmason

Combine the definitions of $(FIND_SOURCE_FILES) and $(LIB_H) to speed up the Makefile, as these are the two main expensive $(shell) commands that we execute unconditionally.

When see what was in $(FOUND_SOURCE_FILES) that wasn't in $(LIB_H) via the ad-hoc test of:

    $(error $(filter-out $(LIB_H),$(filter %.h,$(ALL_SOURCE_FILES))))
    $(error $(filter-out $(ALL_SOURCE_FILES),$(filter %.h,$(LIB_H))))

We'll get, respectively:

    Makefile:850: *** t/helper/test-tool.h.  Stop.
    Makefile:850: *** .  Stop.

I.e. we only had a discrepancy when it came to t/helper/test-tool.h. In terms of correctness this was broken before, but now works:

    $ make t/helper/test-tool.hco
        HDR t/helper/test-tool.h

This speeds things up a lot:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10
    Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1
      Time (mean ± σ):     159.9 ms ±   6.8 ms    [User: 137.2 ms, System: 28.0 ms]
      Range (min … max):   154.6 ms … 175.9 ms    10 runs
    Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0
      Time (mean ± σ):     100.0 ms ±   1.3 ms    [User: 84.2 ms, System: 20.2 ms]
      Range (min … max):    98.8 ms … 102.8 ms    10 runs

    Summary
      'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran
        1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1'
---
 Makefile | 54 ++++++++++++++++++++++++++----------------------------
 1 file changed, 26 insertions(+), 28 deletions(-)
Makefile +26 -28
index aa5ff001b6e..1ac924bd844 100644
@@ -833,12 +833,33 @@ GENERATED_H += hook-list.h
 .PHONY: generated-hdrs
 generated-hdrs: $(GENERATED_H)
 
-LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \
+## Exhaustive lists of our source files, either dynamically generated,
+## or hardcoded.
+SOURCES_CMD = ( \
+	git ls-files \
+		'*.[hcS]' \
+		'*.sh' \
+		':!*[tp][0-9][0-9][0-9][0-9]*' \
+		':!contrib' \
+		2>/dev/null || \
 	$(FIND) . \
-	-name .git -prune -o \
-	-name t -prune -o \
-	-name Documentation -prune -o \
-	-name '*.h' -print)))
+		\( -name .git -type d -prune \) \
+		-o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
+		-o \( -name contrib -type d -prune \) \
+		-o \( -name build -type d -prune \) \
+		-o \( -name 'trash*' -type d -prune \) \
+		-o \( -name '*.[hcS]' -type f -print \) \
+		-o \( -name '*.sh' -type f -print \) \
+		| sed -e 's|^\./||' \
+	)
+FOUND_SOURCE_FILES := $(shell $(SOURCES_CMD))
+
+FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))
+FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))
+
+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
+
+LIB_H = $(FOUND_H_SOURCES)
 
 LIB_OBJS += abspath.o
 LIB_OBJS += add-interactive.o
@@ -2789,26 +2810,6 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
 	$(QUIET_GEN)mkdir -p $(dir $@) && \
 	pod2man $< $@
 
-FIND_SOURCE_FILES = ( \
-	git ls-files \
-		'*.[hcS]' \
-		'*.sh' \
-		':!*[tp][0-9][0-9][0-9][0-9]*' \
-		':!contrib' \
-		2>/dev/null || \
-	$(FIND) . \
-		\( -name .git -type d -prune \) \
-		-o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \
-		-o \( -name contrib -type d -prune \) \
-		-o \( -name build -type d -prune \) \
-		-o \( -name 'trash*' -type d -prune \) \
-		-o \( -name '*.[hcS]' -type f -print \) \
-		-o \( -name '*.sh' -type f -print \) \
-		| sed -e 's|^\./||' \
-	)
-
-FOUND_SOURCE_FILES = $(shell $(FIND_SOURCE_FILES))
-
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
 	echo $(FOUND_SOURCE_FILES) | xargs etags -a -o $@+ && \
@@ -3018,9 +3019,6 @@ check: $(GENERATED_H)
 		exit 1; \
 	fi
 
-FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))
-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
-
 %.cocci.patch: %.cocci $(COCCI_SOURCES)
 	$(QUIET_SPATCH) \
 	if test $(SPATCH_BATCH_SIZE) = 0; then \
Signature · 1 line
2.35.1.1230.ga6e6579e98c
#57 lore raw

PATCH v5 5/8 Makefile: move ".SUFFIXES" rule to shared.mak

In reply to Ævar Arnfjörð Bjarmason

This was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.

This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
    Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
      Time (mean ± σ):     121.7 ms ±   8.8 ms    [User: 105.8 ms, System: 18.6 ms]
      Range (min … max):   112.8 ms … 148.4 ms    26 runs

Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs

Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.

So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.

1. https://lore.kernel.org/git/[email protected]/

---
 Makefile   | 2 --
 shared.mak | 5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
index 1ac924bd844..ce362720947 100644
@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S))
 ASM_OBJ := $(ASM_SRC:S=o)
 C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS))
 
-.SUFFIXES:
-
 $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
 	$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
 $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +5 -0
index 29f0e69ecb9..1dda948df09 100644
@@ -9,6 +9,11 @@
 %:: s.%
 %:: SCCS/s.%
 
+## Likewise delete default $(SUFFIXES). See:
+##
+##     info make --index-search=.SUFFIXES
+.SUFFIXES:
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
Signature · 1 line
2.35.1.1230.ga6e6579e98c
#58 lore raw

Re: [PATCH v5 5/8] Makefile: move ".SUFFIXES" rule to shared.mak

Quoted 59 lines · Ævar Arnfjörð Bjarmason

This was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.

This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
    Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
      Time (mean ± σ):     121.7 ms ±   8.8 ms    [User: 105.8 ms, System: 18.6 ms]
      Range (min … max):   112.8 ms … 148.4 ms    26 runs

Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs

Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.

So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.

1. https://lore.kernel.org/git/[email protected]/

---
 Makefile   | 2 --
 shared.mak | 5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
index 1ac924bd844..ce362720947 100644
@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S))
 ASM_OBJ := $(ASM_SRC:S=o)
 C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS))
 
-.SUFFIXES:
-
 $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
 	$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
 $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +0 -0
index 29f0e69ecb9..1dda948df09 100644
@@ -9,6 +9,11 @@
 %:: s.%

shared.mak line 19

Quoted 10 lines
shared.mak +5 -0
 %:: SCCS/s.%
 
+## Likewise delete default $(SUFFIXES). See:
+##
+##     info make --index-search=.SUFFIXES
+.SUFFIXES:
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to

I confess I really don't understand why, but as part of testing v2.36.0-rc0 on Cygwin, I've started getting errors building the info pages, and bisect points to this commit as the culprit.

Specifically, I've been running

    git clean -dffx && make configure && ./configure && make -j4 info

Without this commit, that gets me a successful build; there's a bunch of noisy warnings that have been hanging around for a long time, and I think are fundamentally due to the slightly mismatched documentation libraries that Cygwin has. With this commit, I get the same noisy warnings, but I also get the error "could not open .texi: No such file or directory".

I have to confess, I don't really understand this aspect of GNU Make, so I'm not sure if this is a problem with Cygwin having a bad toolset or there being something about my environment that means this doesn't work, but regardless, it's currently causing the Cygwin Git builds to fail.

#59 lore raw

Re: [PATCH v5 5/8] Makefile: move ".SUFFIXES" rule to shared.mak

Quoted 85 lines · Adam Dinwoodie
Quoted 59 lines · Ævar Arnfjörð Bjarmason

This was added in 30248886ce8 (Makefile: disable default implicit rules, 2010-01-26), let's move it to the top of "shared.mak" so it'll apply to all our Makefiles.

This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles starts to benefit them. E.g. running the 'man" target is now faster:

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man'
    Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1
      Time (mean ± σ):     121.7 ms ±   8.8 ms    [User: 105.8 ms, System: 18.6 ms]
      Range (min … max):   112.8 ms … 148.4 ms    26 runs

Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0
Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms]
Range (min … max): 89.8 ms … 111.8 ms 32 runs

Summary
'make -C Documentation -j1 man' in 'HEAD~0' ran
1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1'

The reason for that can be seen when comparing that run with "--debug=a". Without this change making a target like "git-status.1" will cause "make" to consider not only "git-status.txt", but "git-status.txt.o", as well as numerous other implicit suffixes such as ".c", ".cc", ".cpp" etc. See [1] for a more detailed before/after example.

So this is causing us to omit a bunch of work we didn't need to do. For making "git-status.1" the "--debug=a" output is reduced from ~140k lines to ~6k.

1. https://lore.kernel.org/git/[email protected]/

---
 Makefile   | 2 --
 shared.mak | 5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)
Makefile +0 -2
index 1ac924bd844..ce362720947 100644
@@ -2580,8 +2580,6 @@ ASM_SRC := $(wildcard $(OBJECTS:o=S))
 ASM_OBJ := $(ASM_SRC:S=o)
 C_OBJ := $(filter-out $(ASM_OBJ),$(OBJECTS))
 
-.SUFFIXES:
-
 $(C_OBJ): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
 	$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
 $(ASM_OBJ): %.o: %.S GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
shared.mak +0 -0
index 29f0e69ecb9..1dda948df09 100644
@@ -9,6 +9,11 @@
 %:: s.%

shared.mak line 19

Quoted 10 lines
shared.mak +5 -0
 %:: SCCS/s.%
 
+## Likewise delete default $(SUFFIXES). See:
+##
+##     info make --index-search=.SUFFIXES
+.SUFFIXES:
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to

I confess I really don't understand why, but as part of testing v2.36.0-rc0 on Cygwin, I've started getting errors building the info pages, and bisect points to this commit as the culprit.

Specifically, I've been running

    git clean -dffx && make configure && ./configure && make -j4 info

Without this commit, that gets me a successful build; there's a bunch of noisy warnings that have been hanging around for a long time, and I think are fundamentally due to the slightly mismatched documentation libraries that Cygwin has. With this commit, I get the same noisy warnings, but I also get the error "could not open .texi: No such file or directory".

Quoted 4 lines

I have to confess, I don't really understand this aspect of GNU Make, so I'm not sure if this is a problem with Cygwin having a bad toolset or there being something about my environment that means this doesn't work, but regardless, it's currently causing the Cygwin Git builds to fail.

Hi. I can reproduce this locally, will look at it and fix it, sorry.

#60 lore raw

PATCH 1/1 Documentation/Makefile: fix "make info" regression in dad9cd7d518

In reply to Adam Dinwoodie

Fix a regression in my dad9cd7d518 (Makefile: move ".SUFFIXES" rule to shared.mak, 2022-03-03). As explained in the GNU make documentation for the $* variable, available at:

	info make --index-search='$*'

This rule relied on ".texi" being in the default list of suffixes, as seen at:

	make -f/dev/null -p | grep -v -e ^# -e ^$|grep -F .SUFFIXES

The documentation explains what was going on here:

In an explicit rule, there is no stem; so '$*' cannot be determined in that way. Instead, if the target name ends with a recognized suffix (*note Old-Fashioned Suffix Rules: Suffix Rules.), '$*' is set to the target name minus the suffix. For example, if the target name is 'foo.c', then '$*' is set to 'foo', since '.c' is a suffix. GNU 'make' does this bizarre thing only for compatibility with other implementations of 'make'. You should generally avoid using '$*' except in implicit rules or static pattern rules.

If the target name in an explicit rule does not end with a recognized suffix, '$*' is set to the empty string for that rule.

I.e. this rule added back in 5cefc33bffd (Documentation: add gitman.info target, 2007-12-10) was resolving gitman.texi from gitman.info. We can instead just use the more obvious $< variable referring to the prerequisite.

This was the only use of $* in our Makefiles in an explicit rule, the three remaining ones are all implicit rules, and therefore didn't depend on the ".SUFFIXES" list.

---

Quoted 2 lines · Adam Dinwoodie

With this commit, I get the same noisy warnings, but I also get the error "could not open .texi: No such file or directory".

Sorry about the regression. This fixes it, and as noted above I'm pretty sure this was the only fallout of the change.

(I didn't have building the full extended documentation as part of my local build, but I'll be adding it now).

 Documentation/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/Makefile +1 -1
index 1eb9192dae8..44c080e3e5b 100644
@@ -390,7 +390,7 @@ gitman.texi: $(MAN_XML) cat-texi.perl texi.xsl
 	$(RM) $@+
 
 gitman.info: gitman.texi
-	$(QUIET_MAKEINFO)$(MAKEINFO) --no-split --no-validate $*.texi
+	$(QUIET_MAKEINFO)$(MAKEINFO) --no-split --no-validate $<
 
 $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
 	$(QUIET_DB2TEXI)$(DOCBOOK2X_TEXI) --to-stdout $*.xml >$@
Signature · 1 line
2.35.1.1604.g35dc6517170
Tested-by on v1 1/1
#61 lore raw

Re: [PATCH] Documentation/Makefile: fix "make info" regression in dad9cd7d518

Quoted 40 lines · Ævar Arnfjörð Bjarmason

Fix a regression in my dad9cd7d518 (Makefile: move ".SUFFIXES" rule to shared.mak, 2022-03-03). As explained in the GNU make documentation for the $* variable, available at:

	info make --index-search='$*'

This rule relied on ".texi" being in the default list of suffixes, as seen at:

	make -f/dev/null -p | grep -v -e ^# -e ^$|grep -F .SUFFIXES

The documentation explains what was going on here:

In an explicit rule, there is no stem; so '$*' cannot be determined in that way. Instead, if the target name ends with a recognized suffix (*note Old-Fashioned Suffix Rules: Suffix Rules.), '$*' is set to the target name minus the suffix. For example, if the target name is 'foo.c', then '$*' is set to 'foo', since '.c' is a suffix. GNU 'make' does this bizarre thing only for compatibility with other implementations of 'make'. You should generally avoid using '$*' except in implicit rules or static pattern rules.

If the target name in an explicit rule does not end with a recognized suffix, '$*' is set to the empty string for that rule.

I.e. this rule added back in 5cefc33bffd (Documentation: add gitman.info target, 2007-12-10) was resolving gitman.texi from gitman.info. We can instead just use the more obvious $< variable referring to the prerequisite.

This was the only use of $* in our Makefiles in an explicit rule, the three remaining ones are all implicit rules, and therefore didn't depend on the ".SUFFIXES" list.

---

On Tue, Apr 05 2022, Adam Dinwoodie wrote:

Quoted 8 lines
Quoted 2 lines

With this commit, I get the same noisy warnings, but I also get the error "could not open .texi: No such file or directory".

Sorry about the regression. This fixes it, and as noted above I'm pretty sure this was the only fallout of the change.

(I didn't have building the full extended documentation as part of my local build, but I'll be adding it now).

Confirmed this patch fixes things for me. Thanks for the quick fix!

#63 lore raw

Re: [PATCH] Documentation/Makefile: fix "make info" regression in dad9cd7d518

Quoted 7 lines · Junio C Hamano
Quoted 3 lines · Adam Dinwoodie

Confirmed this patch fixes things for me. Thanks for the quick fix!

Thanks, both.

Indeed; thanks for the fix, Ævar. This sub-thread was a fun read for me to get to learn about a new "make" quirk. You learn something new every day ;-).

Thanks,
Taylor

#64 lore raw

PATCH v5 6/8 Makefile: move $(comma), $(empty) and $(space) to shared.mak

In reply to Ævar Arnfjörð Bjarmason

Move these variables over to the shared.mak, we'll make use of them in a subsequent commit.

Note that there's reason for these to be "simply expanded variables", i.e. to use ":=" assignments instead of lazily expanded "=" assignments. We could use "=", but let's leave this as-is for now for ease of review.

See 425ca6710b2 (Makefile: allow combining UBSan with other sanitizers, 2017-07-15) for the commit that introduced these.

---
 Makefile   | 4 ----
 shared.mak | 8 ++++++++
 2 files changed, 8 insertions(+), 4 deletions(-)
Makefile +0 -4
index ce362720947..1e7a1277ce9 100644
@@ -1289,10 +1289,6 @@ endif
 ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS)
 ALL_LDFLAGS = $(LDFLAGS)
 
-comma := ,
-empty :=
-space := $(empty) $(empty)
-
 ifdef SANITIZE
 SANITIZERS := $(foreach flag,$(subst $(comma),$(space),$(SANITIZE)),$(flag))
 BASIC_CFLAGS += -fsanitize=$(SANITIZE) -fno-sanitize-recover=$(SANITIZE)
shared.mak +8 -0
index 1dda948df09..934bf428936 100644
@@ -23,3 +23,11 @@
 #
 #    info make --index-search=.DELETE_ON_ERROR
 .DELETE_ON_ERROR:
+
+### Global variables
+
+## comma, empty, space: handy variables as these tokens are either
+## special or can be hard to spot among other Makefile syntax.
+comma := ,
+empty :=
+space := $(empty) $(empty)
Signature · 1 line
2.35.1.1230.ga6e6579e98c
#65 lore raw

PATCH v5 7/8 Makefile: add "$(QUIET)" boilerplate to shared.mak

In reply to Ævar Arnfjörð Bjarmason

The $(QUIET) variables we define are largely duplicated between our various Makefiles, let's define them in the new "shared.mak" instead.

Since we're not using the environment to pass these around we don't need to export the "QUIET_GEN" and "QUIET_BUILT_IN" variables anymore. The "QUIET_GEN" variable is used in "git-gui/Makefile" and "gitweb/Makefile", but they've got their own definition for those. The "QUIET_BUILT_IN" variable is only used in the top-level "Makefile". We still need to export the "V" variable.

---
 Documentation/Makefile  | 32 -------------------------
 Makefile                | 33 -------------------------
 config.mak.uname        |  1 -
 contrib/scalar/Makefile | 14 -----------
 shared.mak              | 53 +++++++++++++++++++++++++++++++++++++++++
 templates/Makefile      |  5 ----
 6 files changed, 53 insertions(+), 85 deletions(-)
Documentation/Makefile +0 -32
index ba27456c86a..0f4ebdeda8a 100644
@@ -218,38 +218,6 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR))
 ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)'
 endif
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring $(MAKEFLAGS),w),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring $(MAKEFLAGS),s),s)
-ifndef V
-	QUIET		= @
-	QUIET_ASCIIDOC	= @echo '   ' ASCIIDOC $@;
-	QUIET_XMLTO	= @echo '   ' XMLTO $@;
-	QUIET_DB2TEXI	= @echo '   ' DB2TEXI $@;
-	QUIET_MAKEINFO	= @echo '   ' MAKEINFO $@;
-	QUIET_DBLATEX	= @echo '   ' DBLATEX $@;
-	QUIET_XSLTPROC	= @echo '   ' XSLTPROC $@;
-	QUIET_GEN	= @echo '   ' GEN $@;
-	QUIET_STDERR	= 2> /dev/null
-	QUIET_SUBDIR0	= +@subdir=
-	QUIET_SUBDIR1	= ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			  $(MAKE) $(PRINT_DIR) -C $$subdir
-
-	QUIET_LINT_GITLINK	= @echo '   ' LINT GITLINK $<;
-	QUIET_LINT_MANSEC	= @echo '   ' LINT MAN SEC $<;
-	QUIET_LINT_MANEND	= @echo '   ' LINT MAN END $<;
-
-	export V
-endif
-endif
-
 all: html man
 
 html: $(DOC_HTML)
Makefile +0 -33
index 1e7a1277ce9..d4c0f4ca7b8 100644
@@ -2001,39 +2001,6 @@ ifndef PAGER_ENV
 PAGER_ENV = LESS=FRX LV=-c
 endif
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring w,$(MAKEFLAGS)),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring s,$(MAKEFLAGS)),s)
-ifndef V
-	QUIET_CC       = @echo '   ' CC $@;
-	QUIET_AR       = @echo '   ' AR $@;
-	QUIET_LINK     = @echo '   ' LINK $@;
-	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
-	QUIET_GEN      = @echo '   ' GEN $@;
-	QUIET_LNCP     = @echo '   ' LN/CP $@;
-	QUIET_XGETTEXT = @echo '   ' XGETTEXT $@;
-	QUIET_MSGFMT   = @echo '   ' MSGFMT $@;
-	QUIET_GCOV     = @echo '   ' GCOV $@;
-	QUIET_SP       = @echo '   ' SP $<;
-	QUIET_HDR      = @echo '   ' HDR $(<:hcc=h);
-	QUIET_RC       = @echo '   ' RC $@;
-	QUIET_SPATCH   = @echo '   ' SPATCH $<;
-	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			 $(MAKE) $(PRINT_DIR) -C $$subdir
-	export V
-	export QUIET_GEN
-	export QUIET_BUILT_IN
-endif
-endif
-
 ifdef NO_INSTALL_HARDLINKS
 	export NO_INSTALL_HARDLINKS
 endif
config.mak.uname +0 -1
index 4352ea39e9b..7727b707b74 100644
@@ -727,7 +727,6 @@ vcxproj:
 	git diff-index --cached --quiet HEAD --
 
 	# Make .vcxproj files and add them
-	unset QUIET_GEN QUIET_BUILT_IN; \
 	perl contrib/buildsystems/generate -g Vcxproj
 	git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj
 
contrib/scalar/Makefile +0 -14
index d585b319edc..5e86d78e19b 100644
@@ -8,20 +8,6 @@ include ../../config.mak.uname
 -include ../../config.mak.autogen
 -include ../../config.mak
 
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring s,$(MAKEFLAGS)),s)
-ifndef V
-	QUIET_GEN      = @echo '   ' GEN $@;
-	QUIET_SUBDIR0  = +@subdir=
-	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
-			 $(MAKE) $(PRINT_DIR) -C $$subdir
-else
-	export V
-endif
-endif
-
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
shared.mak +53 -0

56 lines not on this page · raw message

templates/Makefile +0 -5
index 636cee52f51..367ad00c24c 100644
@@ -2,11 +2,6 @@
 include ../shared.mak
 
 # make and install sample templates
-
-ifndef V
-	QUIET = @
-endif
-
 INSTALL ?= install
 TAR ?= tar
 RM ?= rm -f
Signature · 1 line
2.35.1.1230.ga6e6579e98c
#66 lore raw

PATCH v5 8/8 Makefiles: add and use wildcard "mkdir -p" template

In reply to Ævar Arnfjörð Bjarmason

Add a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).

As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.

I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.

But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
    Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
      Time (mean ± σ):      2.914 s ±  0.062 s    [User: 2.449 s, System: 0.489 s]
      Range (min … max):    2.834 s …  3.020 s    10 runs

Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s]
Range (min … max): 2.229 s … 2.397 s 10 runs

Summary
'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'

So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.

This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://gitlab.com/avar/git-hyperfine/

---
 Documentation/Makefile | 25 +++----------------------
 Makefile               | 12 +++++++-----
 shared.mak             | 17 +++++++++++++++++
 3 files changed, 27 insertions(+), 27 deletions(-)
Documentation/Makefile +3 -22
index 0f4ebdeda8a..1eb9192dae8 100644
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo
 print-man1:
 	@for i in $(MAN1_TXT); do echo $$i; done
 
-## Lint: Common
-.build:
-	$(QUIET)mkdir $@
-.build/lint-docs: | .build
-	$(QUIET)mkdir $@
-
 ## Lint: gitlink
-.build/lint-docs/gitlink: | .build/lint-docs
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
 LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT))
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config
 $(LINT_DOCS_GITLINK): lint-gitlink.perl
 $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \
 		$< \
 		$(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
 lint-docs-gitlink: $(LINT_DOCS_GITLINK)
 
 ## Lint: man-end-blurb
-.build/lint-docs/man-end-blurb: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb
 $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl
 $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@
 .PHONY: lint-docs-man-end-blurb
-lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB)
 
 ## Lint: man-section-order
-.build/lint-docs/man-section-order: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order
 $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl
 $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@
 .PHONY: lint-docs-man-section-order
 lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
Makefile +7 -5
index d4c0f4ca7b8..12f08411d2d 100644
@@ -2745,7 +2745,8 @@ all:: $(MOFILES)
 endif
 
 po/build/locale/%/LC_MESSAGES/git.mo: po/%.po
-	$(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $<
+	$(call mkdir_p_parent_template)
+	$(QUIET_MSGFMT)$(MSGFMT) -o $@ $<
 
 LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm)
 LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2761,15 +2762,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS))
 endif
 
 perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN) \
 	sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \
 	    -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \
 	    -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \
 	< $< > $@
 
 perl/build/man/man3/Git.3pm: perl/Git.pm
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
-	pod2man $< $@
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN)pod2man $< $@
 
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
@@ -2903,7 +2905,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(
 all:: $(TEST_PROGRAMS) $(test_bindir_programs)
 
 bin-wrappers/%: wrap-for-bin.sh
-	@mkdir -p bin-wrappers
+	$(call mkdir_p_parent_template)
 	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
 	     -e 's|@@BUILD_DIR@@|$(shell pwd)|' \
 	     -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
shared.mak +17 -0
index c45b2812eb6..4e1b62ee99e 100644
@@ -53,6 +53,8 @@ ifndef V
 	QUIET          = @
 	QUIET_GEN      = @echo '   ' GEN $@;
 
+	QUIET_MKDIR_P_PARENT  = @echo $(wspfx_SQ) MKDIR -p $(@D);
+
 ## Used in "Makefile"
 	QUIET_CC       = @echo '   ' CC $@;
 	QUIET_AR       = @echo '   ' AR $@;
@@ -84,3 +86,18 @@ ifndef V
 	export V
 endif
 endif
+
+### Templates
+
+## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@
+## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
+## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a
+## "a/prefix/dir/file". This can instead be inserted at the start of
+## the "a/prefix/dir/file" rule.
+define mkdir_p_parent_template
+$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D)))
+endef
Signature · 1 line
2.35.1.1230.ga6e6579e98c