Skip to contentSkip to conversation

test-lib.sh: Use GLIBC_TUNABLES instead of MALLOC_CHECK_ on glibc >= 2.34

PATCH In review Elia Pinto 6 participants 19 messages Mar 4, 2022 – Apr 5, 2022

lore
Share & tools
lore

Elia Pinto

author
#0 lore raw

PATCH v3 1/1 test-lib.sh: Use GLIBC_TUNABLES instead of MALLOC_CHECK_ on glibc >= 2.34

In glibc >= 2.34 MALLOC_CHECK_ and MALLOC_PERTURB_ environment variables have been replaced by GLIBC_TUNABLES. Also the new glibc requires that you preload a library called libc_malloc_debug.so to get these features.

Using the ordinary glibc system variable detect if this is glibc >= 2.34 and use GLIBC_TUNABLES and the new library.

This patch was inspired by a Richard W.M. Jones ndbkit patch

---
This is the third version of the patch.

Compared to the second version[1], the code is further simplified, eliminating a case statement and modifying a string statement.

[1] https://www.spinics.net/lists/git/msg433917.html

 t/test-lib.sh | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
t/test-lib.sh +16 -0
index 9af5fb7674..4d10646015 100644
@@ -550,9 +550,25 @@ else
 	setup_malloc_check () {
 		MALLOC_CHECK_=3	MALLOC_PERTURB_=165
 		export MALLOC_CHECK_ MALLOC_PERTURB_
+		if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
+		_GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
+		expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+		then
+			g=
+			LD_PRELOAD="libc_malloc_debug.so.0"
+			for t in \
+				glibc.malloc.check=1 \
+				glibc.malloc.perturb=165
+			do
+				g="${g#:}:$t"
+			done
+			GLIBC_TUNABLES=$g
+			export LD_PRELOAD GLIBC_TUNABLES
+		fi
 	}
 	teardown_malloc_check () {
 		unset MALLOC_CHECK_ MALLOC_PERTURB_
+		unset LD_PRELOAD GLIBC_TUNABLES
 	}
 fi
 

--
2.35.1

#1 lore raw
Quoted 10 lines · Elia Pinto

In glibc >= 2.34 MALLOC_CHECK_ and MALLOC_PERTURB_ environment variables have been replaced by GLIBC_TUNABLES. Also the new glibc requires that you preload a library called libc_malloc_debug.so to get these features.

Using the ordinary glibc system variable detect if this is glibc >= 2.34 and use GLIBC_TUNABLES and the new library.

This patch was inspired by a Richard W.M. Jones ndbkit patch

Quoted 7 lines

---
This is the third version of the patch.

Compared to the second version[1], the code is further simplified, eliminating a case statement and modifying a string statement.

Thanks; will queue. Let's declare victory and merge it down to 'next' and 'master/main'.

#2 lore raw

PATCH 1/1 test-lib.sh: use awk instead of expr for a POSIX non integer check

In reply to Elia Pinto

Restrict the glibc version to a single version number and compare it arithmetically against the base glibc version to avoid accidentally matching against "2.3" and better supporting versions like "2.34.9000"

---
 t/test-lib.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
t/test-lib.sh +3 -3
index 8e59c58e7e7..f624f87eb81 100644
@@ -518,9 +518,9 @@ else
 	setup_malloc_check () {
 		MALLOC_CHECK_=3	MALLOC_PERTURB_=165
 		export MALLOC_CHECK_ MALLOC_PERTURB_
-		if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
-		   _GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
-		   expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+		local _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null)
+		if echo "$_GLIBC_VERSION" | cut -d. -f1-2 |
+			awk '{ if ($2 - 2.34 < 0) exit 1 }'
 		then
 			g=
 			LD_PRELOAD="libc_malloc_debug.so.0"
Signature · 1 line
2.35.1.505.g27486cd1b2d
#3 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 8 lines · Carlo Marcelo Arenas Belón

Restrict the glibc version to a single version number and compare it arithmetically against the base glibc version to avoid accidentally matching against "2.3" and better supporting versions like "2.34.9000"

---

t/test-lib.sh else · line 520

Quoted 6 lines
t/test-lib.sh +3 -3
@@ -518,9 +518,9 @@ else
-               if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
-                  _GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
-                  expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+               local _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null)
+               if echo "$_GLIBC_VERSION" | cut -d. -f1-2 |
+                       awk '{ if ($2 - 2.34 < 0) exit 1 }'

No need for cut since awk can accomplish the same by itself.

    if echo "$_GLIBC_VERSION" | awk '/^glibc / { if ($2 - 2.34 < 0) exit 1 }'

should work, I would think.

#4 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 19 lines · Eric Sunshine
Quoted 8 lines · Carlo Marcelo Arenas Belón

Restrict the glibc version to a single version number and compare it arithmetically against the base glibc version to avoid accidentally matching against "2.3" and better supporting versions like "2.34.9000"

---

t/test-lib.sh else · line 520

Quoted 6 lines
t/test-lib.sh +3 -3
@@ -518,9 +518,9 @@ else
-               if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
-                  _GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
-                  expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+               local _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null)
+               if echo "$_GLIBC_VERSION" | cut -d. -f1-2 |
+                       awk '{ if ($2 - 2.34 < 0) exit 1 }'

No need for cut since awk can accomplish the same by itself.

Quoted 3 lines
    if echo "$_GLIBC_VERSION" | awk '/^glibc / { if ($2 - 2.34 < 0) exit 1 }'

should work, I would think.

Nevermind, I forgot you want to better support "2.34.9000" matches. Though, awk should still be able to do so on its own, one would expect, but not too important.

#5 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 9 lines · Eric Sunshine
Quoted 8 lines · Eric Sunshine

t/test-lib.sh else · line 523

Quoted 3 lines · Carlo Marcelo Arenas Belón
t/test-lib.sh +3 -0
+               local _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null)
+               if echo "$_GLIBC_VERSION" | cut -d. -f1-2 |
+                       awk '{ if ($2 - 2.34 < 0) exit 1 }'

No need for cut since awk can accomplish the same by itself.

Quoted 7 lines
Quoted 3 lines
    if echo "$_GLIBC_VERSION" | awk '/^glibc / { if ($2 - 2.34 < 0) exit 1 }'

should work, I would think.

Nevermind, I forgot you want to better support "2.34.9000" matches. Though, awk should still be able to do so on its own, one would expect, but not too important.

This seems to work, though it's getting a bit verbose:

    awk '/^glibc / { split($2,v,"."); if (sprintf("%s.%s", v[1], v[2])

- 2.34 < 0) exit 1 }'

#6 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 14 lines · Eric Sunshine
Quoted 13 lines · Eric Sunshine
Quoted 11 lines · Eric Sunshine

t/test-lib.sh else · line 523

Quoted 3 lines · Carlo Marcelo Arenas Belón
t/test-lib.sh +3 -0
+               local _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null)
+               if echo "$_GLIBC_VERSION" | cut -d. -f1-2 |
+                       awk '{ if ($2 - 2.34 < 0) exit 1 }'

No need for cut since awk can accomplish the same by itself.

    if echo "$_GLIBC_VERSION" | awk '/^glibc / { if ($2 - 2.34 < 0) exit 1 }'

should work, I would think.

Quoted 8 lines
Quoted 3 lines

Nevermind, I forgot you want to better support "2.34.9000" matches. Though, awk should still be able to do so on its own, one would expect, but not too important.

This seems to work, though it's getting a bit verbose:

    awk '/^glibc / { split($2,v,"."); if (sprintf("%s.%s", v[1], v[2])

- 2.34 < 0) exit 1 }'

If we are losing "cut" (which I think is a good thing to do), we probably can lose the pipe, too and refer to $_GLIBC_VERSION as an element in ARGV[] and make the command used as "if" condition to a single "awk" script?

In general it is a good discipline to question a pipeline that preprocesses input fed to a script written in a language with full programming power like awk and perl (and to lessor extent, sed) to see if we can come up with a simpler solution without pipeline helping to solve what these languages are invented to solve, and I very much appreciate your exploration ;-)

Thanks.

#7 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 30 lines · Junio C Hamano
Quoted 14 lines · Eric Sunshine
Quoted 13 lines · Eric Sunshine
Quoted 11 lines · Eric Sunshine

t/test-lib.sh else · line 523

Quoted 3 lines · Carlo Marcelo Arenas Belón
t/test-lib.sh +3 -0
+               local _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null)
+               if echo "$_GLIBC_VERSION" | cut -d. -f1-2 |
+                       awk '{ if ($2 - 2.34 < 0) exit 1 }'

No need for cut since awk can accomplish the same by itself.

    if echo "$_GLIBC_VERSION" | awk '/^glibc / { if ($2 - 2.34 < 0) exit 1 }'

should work, I would think.

Quoted 8 lines
Quoted 3 lines

Nevermind, I forgot you want to better support "2.34.9000" matches. Though, awk should still be able to do so on its own, one would expect, but not too important.

This seems to work, though it's getting a bit verbose:

    awk '/^glibc / { split($2,v,"."); if (sprintf("%s.%s", v[1], v[2])

- 2.34 < 0) exit 1 }'

If we are losing "cut" (which I think is a good thing to do), we probably can lose the pipe, too and refer to $_GLIBC_VERSION as an element in ARGV[] and make the command used as "if" condition to a single "awk" script?

Quoted 6 lines

In general it is a good discipline to question a pipeline that preprocesses input fed to a script written in a language with full programming power like awk and perl (and to lessor extent, sed) to see if we can come up with a simpler solution without pipeline helping to solve what these languages are invented to solve, and I very much appreciate your exploration ;-)

I agree :) But the first language we've got here is C. Rather than fiddle around with getconf, awk/sed etc. why not just the rather trivial:

	diff --git a/Makefile b/Makefile
	index 6f0b4b775fe..f566c9c5df2 100644
	--- a/Makefile
	+++ b/Makefile
	@@ -732,6 +732,7 @@ TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
	 TEST_BUILTINS_OBJS += test-partial-clone.o
	 TEST_BUILTINS_OBJS += test-path-utils.o
	 TEST_BUILTINS_OBJS += test-pcre2-config.o
	+TEST_BUILTINS_OBJS += test-glibc-config.o
	 TEST_BUILTINS_OBJS += test-pkt-line.o
	 TEST_BUILTINS_OBJS += test-prio-queue.o
	 TEST_BUILTINS_OBJS += test-proc-receive.o
	diff --git a/t/helper/test-glibc-config.c b/t/helper/test-glibc-config.c
	new file mode 100644
	index 00000000000..3c3cc2a8ba5
	--- /dev/null
	+++ b/t/helper/test-glibc-config.c
	@@ -0,0 +1,12 @@
	+#include "test-tool.h"
	+#include "cache.h"
	+
	+int cmd__glibc_config(int argc, const char **argv)
	+{
	+#ifdef __GNU_LIBRARY__
	+	printf("%d\n%d\n", __GLIBC__, __GLIBC_MINOR__);
	+	return 0;
	+#else
	+	return 1;
	+#endif
	+}
	diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
	index e6ec69cf326..c01422f5cab 100644
	--- a/t/helper/test-tool.c
	+++ b/t/helper/test-tool.c
	@@ -35,6 +35,7 @@ static struct test_cmd cmds[] = {
	 	{ "genrandom", cmd__genrandom },
	 	{ "genzeros", cmd__genzeros },
	 	{ "getcwd", cmd__getcwd },
	+	{ "glibc-config", cmd__glibc_config },
	 	{ "hashmap", cmd__hashmap },
	 	{ "hash-speed", cmd__hash_speed },
	 	{ "index-version", cmd__index_version },
	diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
	index 20756eefdda..dc061bc7833 100644
	--- a/t/helper/test-tool.h
	+++ b/t/helper/test-tool.h
	@@ -26,6 +26,7 @@ int cmd__fast_rebase(int argc, const char **argv);
	 int cmd__genrandom(int argc, const char **argv);
	 int cmd__genzeros(int argc, const char **argv);
	 int cmd__getcwd(int argc, const char **argv);
	+int cmd__glibc_config(int argc, const char **argv);
	 int cmd__hashmap(int argc, const char **argv);
	 int cmd__hash_speed(int argc, const char **argv);
	 int cmd__index_version(int argc, const char **argv);

I mainly copied the pcre2-config template I added in 95ca1f987ed (grep/pcre2: better support invalid UTF-8 haystacks, 2021-01-24), which likewise would have been quite a bit more complex to do from non-C.

#8 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 18 lines · Ævar Arnfjörð Bjarmason
Quoted 6 lines · Junio C Hamano
Quoted 4 lines · Eric Sunshine

This seems to work, though it's getting a bit verbose:

    awk '/^glibc / { split($2,v,"."); if (sprintf("%s.%s", v[1], v[2])

- 2.34 < 0) exit 1 }'

Quoted 6 lines

In general it is a good discipline to question a pipeline that preprocesses input fed to a script written in a language with full programming power like awk and perl (and to lessor extent, sed) to see if we can come up with a simpler solution without pipeline helping to solve what these languages are invented to solve, and I very much appreciate your exploration ;-)

I agree :) But the first language we've got here is C. Rather than fiddle around with getconf, awk/sed etc. why not just the rather trivial:

Quoted 12 lines
        +#include "test-tool.h"
        +#include "cache.h"
        +
        +int cmd__glibc_config(int argc, const char **argv)
        +{
        +#ifdef __GNU_LIBRARY__
        +       printf("%d\n%d\n", __GLIBC__, __GLIBC_MINOR__);
        +       return 0;
        +#else
        +       return 1;
        +#endif
        +}

It feels overkill to add this just for this one case which is otherwise done easily enough with existing shell tools.

That said, perhaps I'm missing something, but I don't see how this solution helps us get away from the need for expr, awk, or perl since one of those languages would be needed to perform the arithmetic comparison (checking if glibc is >= 2.34).

#9 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 32 lines · Eric Sunshine
Quoted 18 lines · Ævar Arnfjörð Bjarmason
Quoted 6 lines · Junio C Hamano
Quoted 4 lines · Eric Sunshine

This seems to work, though it's getting a bit verbose:

    awk '/^glibc / { split($2,v,"."); if (sprintf("%s.%s", v[1], v[2])

- 2.34 < 0) exit 1 }'

Quoted 6 lines

In general it is a good discipline to question a pipeline that preprocesses input fed to a script written in a language with full programming power like awk and perl (and to lessor extent, sed) to see if we can come up with a simpler solution without pipeline helping to solve what these languages are invented to solve, and I very much appreciate your exploration ;-)

I agree :) But the first language we've got here is C. Rather than fiddle around with getconf, awk/sed etc. why not just the rather trivial:

Quoted 12 lines
        +#include "test-tool.h"
        +#include "cache.h"
        +
        +int cmd__glibc_config(int argc, const char **argv)
        +{
        +#ifdef __GNU_LIBRARY__
        +       printf("%d\n%d\n", __GLIBC__, __GLIBC_MINOR__);
        +       return 0;
        +#else
        +       return 1;
        +#endif
        +}
Quoted 7 lines

It feels overkill to add this just for this one case which is otherwise done easily enough with existing shell tools.

That said, perhaps I'm missing something, but I don't see how this solution helps us get away from the need for expr, awk, or perl since one of those languages would be needed to perform the arithmetic comparison (checking if glibc is >= 2.34).

In this case they're \n delimited, so we can use the shell's native whitespace splitting to $(())-compare $1 and $2.

But probably better is to just amend that to call it as "test-tool libc is-glibc-2.34-or-newer" or whatever. Then just do:

	if (__GLIBC__ > 2 || (__GLIBC__ == 2 && 34 >= __GLIBC_MINOR__))
		return 0;
	return 1;
#10 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 6 lines · Ævar Arnfjörð Bjarmason

But probably better is to just amend that to call it as "test-tool libc is-glibc-2.34-or-newer" or whatever. Then just do:

	if (__GLIBC__ > 2 || (__GLIBC__ == 2 && 34 >= __GLIBC_MINOR__))
		return 0;
	return 1;

Yuck. Then we'd have yet another libc-is-glibc-2.36-or-newer option, too, in the future?

#11 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 6 lines · Junio C Hamano
Quoted 3 lines · Ævar Arnfjörð Bjarmason

But probably better is to just amend that to call it as "test-tool libc is-glibc-2.34-or-newer" or whatever. Then just do:

Quoted 6 lines
Quoted 3 lines
      if (__GLIBC__ > 2 || (__GLIBC__ == 2 && 34 >= __GLIBC_MINOR__))
              return 0;
      return 1;

Yuck. Then we'd have yet another libc-is-glibc-2.36-or-newer option, too, in the future?

Luckily that won't be needed, as this the original version (with expr) is practically good enough even if it might be a little odd looking and incorrect for 2.4 <= glibc <= 2.9 (which are over 10 years old).

  $ expr 2.34 \<= "2.34.9000"
  1
  $ expr 2.34 \<= ""
  0

Apologies for the confusion, and feel free to drop this patch

Carlo

#12 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

Quoted 14 lines · Carlo Arenas
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · Ævar Arnfjörð Bjarmason

But probably better is to just amend that to call it as "test-tool libc is-glibc-2.34-or-newer" or whatever. Then just do:

Quoted 6 lines
Quoted 3 lines
      if (__GLIBC__ > 2 || (__GLIBC__ == 2 && 34 >= __GLIBC_MINOR__))
              return 0;
      return 1;

Yuck. Then we'd have yet another libc-is-glibc-2.36-or-newer option, too, in the future?

Quoted 8 lines

Luckily that won't be needed, as this the original version (with expr) is practically good enough even if it might be a little odd looking and incorrect for 2.4 <= glibc <= 2.9 (which are over 10 years old).

  $ expr 2.34 \<= "2.34.9000"
  1
  $ expr 2.34 \<= ""
  0

Yeah, that is good.

What I was trying to get at was to extend Ævar's one trivially to

  $ test-tool libc-is-at-or-later-than 2.34

so that we can deal with

  $ test-tool libc-is-at-or-later-than 2.36

for free, but if we do not have to do anything, that is even better ;-)

#13 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

In reply to Junio C Hamano
Quoted 10 lines · Junio C Hamano
Quoted 9 lines · Eric Sunshine
Quoted 5 lines · Eric Sunshine
Quoted 5 lines

t/test-lib.sh else · line 523

Quoted 3 lines · Carlo Marcelo Arenas Belón
t/test-lib.sh +3 -0
+               local _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null)
+               if echo "$_GLIBC_VERSION" | cut -d. -f1-2 |
+                       awk '{ if ($2 - 2.34 < 0) exit 1 }'

This seems to work, though it's getting a bit verbose:

Quoted 7 lines
Quoted 2 lines
    awk '/^glibc / { split($2,v,"."); if (sprintf("%s.%s", v[1], v[2])

- 2.34 < 0) exit 1 }'

If we are losing "cut" (which I think is a good thing to do), we probably can lose the pipe, too and refer to $_GLIBC_VERSION as an element in ARGV[] and make the command used as "if" condition to a single "awk" script?

Erm, something like this perhaps?

    awk 'BEGIN { split(ARGV[1], v, "[ .]"); if (v[1]=="glibc" &&

sprintf("%s.%s", v[2], v[3]) - 2.34 < 0) exit 1 }'

Quoted 6 lines

In general it is a good discipline to question a pipeline that preprocesses input fed to a script written in a language with full programming power like awk and perl (and to lessor extent, sed) to see if we can come up with a simpler solution without pipeline helping to solve what these languages are invented to solve, and I very much appreciate your exploration ;-)

Yup, although in this case, the pure-awk solution may not convey the intent as easily as the original posted by Carlo which had the benefit of being perhaps easier to digest.

author
#14 lore raw

Re: [PATCH] test-lib.sh: use awk instead of expr for a POSIX non integer check

In reply to Carlo Marcelo Arenas Belón

Il giorno mar 8 mar 2022 alle ore 12:34 Carlo Marcelo Arenas Belón <[email protected]> ha scritto:

Quoted 5 lines

Restrict the glibc version to a single version number and compare it arithmetically against the base glibc version to avoid accidentally matching against "2.3" and better supporting versions like "2.34.9000"

I didn't understand the problem. How glibc names the versions is known:

https://sourceware.org/glibc/wiki/Glibc%20Timeline

What is wrong with the expr statement ?

+ expr 2.34 '<=' 2.35
1
+ expr 2.34 '<=' 2.34
1
+ expr 2.34 '<=' 2.33
0
+ expr 2.34 '<=' 2.32
0
+ expr 2.34 '<=' 2.31
0
+ expr 2.34 '<=' 2.30
0
+ expr 2.34 '<=' 2.29
0
+ expr 2.34 '<=' 2.28
0
+ expr 2.34 '<=' 2.27
0
+ expr 2.34 '<=' 2.26
0
+ expr 2.34 '<=' 2.25
0
+ expr 2.34 '<=' 2.24
0
+ expr 2.34 '<=' 2.23
0
+ expr 2.34 '<=' 2.22
0
+ expr 2.34 '<=' 2.21
0
+ expr 2.34 '<=' 2.20
0
+ expr 2.34 '<=' 2.19
0
+ expr 2.34 '<=' 2.18
0
+ expr 2.34 '<=' 2.17
0
+ expr 2.34 '<=' 2.16
0
+ expr 2.34 '<=' 2.15
0
+ expr 2.34 '<=' 2.13

Quoted 25 lines
---
 t/test-lib.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
t/test-lib.sh +3 -4
index 8e59c58e7e7..f624f87eb81 100644
@@ -518,9 +518,9 @@ else
        setup_malloc_check () {
                MALLOC_CHECK_=3 MALLOC_PERTURB_=165
                export MALLOC_CHECK_ MALLOC_PERTURB_
-               if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
-                  _GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
-                  expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+               local _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null)
+               if echo "$_GLIBC_VERSION" | cut -d. -f1-2 |
+                       awk '{ if ($2 - 2.34 < 0) exit 1 }'
                then
                        g=
                        LD_PRELOAD="libc_malloc_debug.so.0"
--

2.35.1.505.g27486cd1b2d

#15 lore raw
In reply to Elia Pinto
Quoted 28 lines · Elia Pinto

In glibc >= 2.34 MALLOC_CHECK_ and MALLOC_PERTURB_ environment variables have been replaced by GLIBC_TUNABLES. Also the new glibc requires that you preload a library called libc_malloc_debug.so to get these features.

Using the ordinary glibc system variable detect if this is glibc >= 2.34 and use GLIBC_TUNABLES and the new library.

This patch was inspired by a Richard W.M. Jones ndbkit patch

---
This is the third version of the patch.

Compared to the second version[1], the code is further simplified, eliminating a case statement and modifying a string statement.

[1] https://www.spinics.net/lists/git/msg433917.html

  t/test-lib.sh | 16 ++++++++++++++++
  1 file changed, 16 insertions(+)
t/test-lib.sh +0 -0
index 9af5fb7674..4d10646015 100644

t/test-lib.sh else · line 558

Quoted 9 lines
t/test-lib.sh +6 -0
@@ -550,9 +550,25 @@ else
  	setup_malloc_check () {
  		MALLOC_CHECK_=3	MALLOC_PERTURB_=165
  		export MALLOC_CHECK_ MALLOC_PERTURB_
+		if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
+		_GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
+		expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+		then
+			g=
+			LD_PRELOAD="libc_malloc_debug.so.0"

When compiling with "SANITIZE = address,leak" this use of LD_PRELOAD makes the tests fail with

==9750==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

because libc_malloc_debug.so is being loaded before libasan.so. If I set TEST_NO_MALLOC_CHECK=1 when I run the tests then ASAN does not complain but it would be nicer if I did not have to do that. I'm confused as to why the CI leak tests are running fine - am I missing something with my setup?

Best Wishes

Phillip

#16 lore raw

Making the tests ~2.5x faster (was: [PATCH v3] test-lib.sh: Use GLIBC_TUNABLES instead of MALLOC_CHECK_ on glibc >= 2.34)

Quoted 40 lines · Phillip Wood
Quoted 22 lines · Elia Pinto

In glibc >= 2.34 MALLOC_CHECK_ and MALLOC_PERTURB_ environment variables have been replaced by GLIBC_TUNABLES. Also the new glibc requires that you preload a library called libc_malloc_debug.so to get these features.
Using the ordinary glibc system variable detect if this is glibc >= 2.34 and
use GLIBC_TUNABLES and the new library.
This patch was inspired by a Richard W.M. Jones ndbkit patch

---
This is the third version of the patch.
Compared to the second version[1], the code is further simplified, eliminating a case statement and modifying a string statement.
[1] https://www.spinics.net/lists/git/msg433917.html

  t/test-lib.sh | 16 ++++++++++++++++
  1 file changed, 16 insertions(+)
t/test-lib.sh +0 -0
index 9af5fb7674..4d10646015 100644

t/test-lib.sh else · line 558

Quoted 9 lines
t/test-lib.sh +6 -0
@@ -550,9 +550,25 @@ else
  	setup_malloc_check () {
  		MALLOC_CHECK_=3	MALLOC_PERTURB_=165
  		export MALLOC_CHECK_ MALLOC_PERTURB_
+		if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
+		_GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
+		expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+		then
+			g=
+			LD_PRELOAD="libc_malloc_debug.so.0"

When compiling with "SANITIZE = address,leak" this use of LD_PRELOAD makes the tests fail with

==9750==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

Quoted 5 lines

because libc_malloc_debug.so is being loaded before libasan.so. If I set TEST_NO_MALLOC_CHECK=1 when I run the tests then ASAN does not complain but it would be nicer if I did not have to do that. I'm confused as to why the CI leak tests are running fine - am I missing something with my setup?

Perhaps they have an older glibc? They're on Ubunt, and e.g. my Debian version is on 2.33.

But more generally, I'd somehow managed to not notice for all my time in hacking on git (including on SANITIZE=leak, another tracing mode!) that this check was being enabled *by default*, which could have saved me some time waiting for tests...:

	$ git hyperfine -L rev HEAD~0 -L off yes, -s 'make CFLAGS=-O3' '(cd t && TEST_NO_MALLOC_CHECK={off} ./t3070-wildmatch.sh)' --warmup 1 -r 3
	Benchmark 1: (cd t && TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh)' in 'HEAD~0
	  Time (mean ± σ):      4.191 s ±  0.012 s    [User: 3.600 s, System: 0.746 s]
	  Range (min … max):    4.181 s …  4.204 s    3 runs
	Benchmark 2: (cd t && TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh)' in 'HEAD~0
	  Time (mean ± σ):      5.945 s ±  0.101 s    [User: 4.989 s, System: 1.146 s]
	  Range (min … max):    5.878 s …  6.062 s    3 runs
	 
	Summary
	  '(cd t && TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh)' in 'HEAD~0' ran
	    1.42 ± 0.02 times faster than '(cd t && TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh)' in 'HEAD~0'

I.e. I get that it's catching actual issues, but I was also doing runs with SANITIZE=address, which I believe are going to catch a superset of issues that this check does, so...

Whatever we do with this narrow patch it would be a really nice improvement if the test-lib.sh could fold all of these "instrumentations" behind a single flag, and that both it and "make test" would make it clear that you're testing in a slower "tracing" or "instrumentation" mode.

Ditto things like chain lint and the bin-wrappers, e.g.:

    $ git hyperfine -L rev HEAD~0 -L off yes, -L cl 0,1 -L nbw --no-bin-wrappers, -s 'make CFLAGS=-O3' '(cd t && GIT_TEST_CHAIN_LINT={cl} TEST_NO_MALLOC_CHECK={off} ./t3070-wildmatch.sh {nbw})' -r 1
    [...]	
	Summary
	  '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0' ran
	    1.23 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.30 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    1.54 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.63 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    1.87 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.92 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    2.24 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh )' in 'HEAD~0'

I.e. between this, chain lint and bin wrappers we're coming up on our tests running almost 3x as slow as they otherwise could *by default*.

But right now knowing which things you need to chase around to turn off if you're just looking to test the semantics of your code without all this instrumentation is a matter of archane knowledge, I'm not even sure I remembered all the major ones (I didn't know about this one until today).

#17 lore raw

Re: Making the tests ~2.5x faster (was: [PATCH v3] test-lib.sh: Use GLIBC_TUNABLES instead of MALLOC_CHECK_ on glibc >= 2.34)

Quoted 43 lines · Ævar Arnfjörð Bjarmason
Quoted 40 lines · Phillip Wood
Quoted 22 lines · Elia Pinto

In glibc >= 2.34 MALLOC_CHECK_ and MALLOC_PERTURB_ environment variables have been replaced by GLIBC_TUNABLES. Also the new glibc requires that you preload a library called libc_malloc_debug.so to get these features.
Using the ordinary glibc system variable detect if this is glibc >= 2.34 and
use GLIBC_TUNABLES and the new library.
This patch was inspired by a Richard W.M. Jones ndbkit patch

---
This is the third version of the patch.
Compared to the second version[1], the code is further simplified, eliminating a case statement and modifying a string statement.
[1] https://www.spinics.net/lists/git/msg433917.html

   t/test-lib.sh | 16 ++++++++++++++++
   1 file changed, 16 insertions(+)
t/test-lib.sh +0 -0
index 9af5fb7674..4d10646015 100644

t/test-lib.sh else · line 558

Quoted 9 lines
t/test-lib.sh +6 -0
@@ -550,9 +550,25 @@ else
   	setup_malloc_check () {
   		MALLOC_CHECK_=3	MALLOC_PERTURB_=165
   		export MALLOC_CHECK_ MALLOC_PERTURB_
+		if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
+		_GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
+		expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+		then
+			g=
+			LD_PRELOAD="libc_malloc_debug.so.0"

When compiling with "SANITIZE = address,leak" this use of LD_PRELOAD makes the tests fail with

==9750==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

Quoted 8 lines
Quoted 5 lines

because libc_malloc_debug.so is being loaded before libasan.so. If I set TEST_NO_MALLOC_CHECK=1 when I run the tests then ASAN does not complain but it would be nicer if I did not have to do that. I'm confused as to why the CI leak tests are running fine - am I missing something with my setup?

Perhaps they have an older glibc? They're on Ubunt, and e.g. my Debian version is on 2.33.

Good point, I'd not realized quite how new glibc 2.34 was

Quoted 18 lines

But more generally, I'd somehow managed to not notice for all my time in hacking on git (including on SANITIZE=leak, another tracing mode!) that this check was being enabled *by default*, which could have saved me some time waiting for tests...:

	$ git hyperfine -L rev HEAD~0 -L off yes, -s 'make CFLAGS=-O3' '(cd t && TEST_NO_MALLOC_CHECK={off} ./t3070-wildmatch.sh)' --warmup 1 -r 3
	Benchmark 1: (cd t && TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh)' in 'HEAD~0
	  Time (mean ± σ):      4.191 s ±  0.012 s    [User: 3.600 s, System: 0.746 s]
	  Range (min … max):    4.181 s …  4.204 s    3 runs
	Benchmark 2: (cd t && TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh)' in 'HEAD~0
	  Time (mean ± σ):      5.945 s ±  0.101 s    [User: 4.989 s, System: 1.146 s]
	  Range (min … max):    5.878 s …  6.062 s    3 runs
	
	Summary
	  '(cd t && TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh)' in 'HEAD~0' ran
	    1.42 ± 0.02 times faster than '(cd t && TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh)' in 'HEAD~0'
Quoted 3 lines

I.e. I get that it's catching actual issues, but I was also doing runs with SANITIZE=address, which I believe are going to catch a superset of issues that this check does, so...

I assumed SANITIZE=address would catch a superset of issues as well but I haven't actually checked the glibc tunables documentation. We disable MALLOC_PERTURB_ when running under valgrind so perhaps we should do the same when compiling with SANITIZE=address.

I just noticed that setup_malloc_check() is called by test_expect_success() and test_when_finished() so it really should be caching the result of the check rather than forking getconf and expr each time it is called. Overwriting LD_PRELOAD is not very friendly either, it would be better if it appended the debug library if the variable is already set.

Quoted 7 lines

Whatever we do with this narrow patch it would be a really nice improvement if the test-lib.sh could fold all of these "instrumentations" behind a single flag, and that both it and "make test" would make it clear that you're testing in a slower "tracing" or "instrumentation" mode.

Ditto things like chain lint and the bin-wrappers, e.g.:

I sometimes wish there was a way to only chain lint the tests that have changed since the last run.

Quoted 12 lines
     $ git hyperfine -L rev HEAD~0 -L off yes, -L cl 0,1 -L nbw --no-bin-wrappers, -s 'make CFLAGS=-O3' '(cd t && GIT_TEST_CHAIN_LINT={cl} TEST_NO_MALLOC_CHECK={off} ./t3070-wildmatch.sh {nbw})' -r 1
     [...]	
	Summary
	  '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0' ran
	    1.23 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.30 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    1.54 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.63 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    1.87 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.92 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    2.24 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh )' in 'HEAD~0'
Quoted 8 lines

I.e. between this, chain lint and bin wrappers we're coming up on our tests running almost 3x as slow as they otherwise could *by default*.

But right now knowing which things you need to chase around to turn off if you're just looking to test the semantics of your code without all this instrumentation is a matter of archane knowledge, I'm not even sure I remembered all the major ones (I didn't know about this one until today).

That is quite a difference in run time - I wonder how much scope there is for optimizing some of these features like the chain-lint vs disabling them completely.

Best Wishes

Phillip

#18 lore raw

Re: Making the tests ~2.5x faster (was: [PATCH v3] test-lib.sh: Use GLIBC_TUNABLES instead of MALLOC_CHECK_ on glibc >= 2.34)

Quoted 76 lines · Phillip Wood
Quoted 42 lines · Ævar Arnfjörð Bjarmason
Quoted 40 lines · Phillip Wood
Quoted 22 lines · Elia Pinto

In glibc >= 2.34 MALLOC_CHECK_ and MALLOC_PERTURB_ environment variables have been replaced by GLIBC_TUNABLES. Also the new glibc requires that you preload a library called libc_malloc_debug.so to get these features.
Using the ordinary glibc system variable detect if this is glibc >= 2.34 and
use GLIBC_TUNABLES and the new library.
This patch was inspired by a Richard W.M. Jones ndbkit patch

---
This is the third version of the patch.
Compared to the second version[1], the code is further simplified, eliminating a case statement and modifying a string statement.
[1] https://www.spinics.net/lists/git/msg433917.html

   t/test-lib.sh | 16 ++++++++++++++++
   1 file changed, 16 insertions(+)
t/test-lib.sh +0 -0
index 9af5fb7674..4d10646015 100644

t/test-lib.sh else · line 558

Quoted 9 lines
t/test-lib.sh +6 -0
@@ -550,9 +550,25 @@ else
   	setup_malloc_check () {
   		MALLOC_CHECK_=3	MALLOC_PERTURB_=165
   		export MALLOC_CHECK_ MALLOC_PERTURB_
+		if _GLIBC_VERSION=$(getconf GNU_LIBC_VERSION 2>/dev/null) &&
+		_GLIBC_VERSION=${_GLIBC_VERSION#"glibc "} &&
+		expr 2.34 \<= "$_GLIBC_VERSION" >/dev/null
+		then
+			g=
+			LD_PRELOAD="libc_malloc_debug.so.0"

When compiling with "SANITIZE = address,leak" this use of LD_PRELOAD makes the tests fail with

==9750==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

Quoted 8 lines
Quoted 5 lines

because libc_malloc_debug.so is being loaded before libasan.so. If I set TEST_NO_MALLOC_CHECK=1 when I run the tests then ASAN does not complain but it would be nicer if I did not have to do that. I'm confused as to why the CI leak tests are running fine - am I missing something with my setup?

Perhaps they have an older glibc? They're on Ubunt, and e.g. my Debian
version is on 2.33.

Good point, I'd not realized quite how new glibc 2.34 was

Quoted 17 lines

But more generally, I'd somehow managed to not notice for all my time in hacking on git (including on SANITIZE=leak, another tracing mode!) that this check was being enabled *by default*, which could have saved me some time waiting for tests...:

	$ git hyperfine -L rev HEAD~0 -L off yes, -s 'make CFLAGS=-O3' '(cd t && TEST_NO_MALLOC_CHECK={off} ./t3070-wildmatch.sh)' --warmup 1 -r 3
	Benchmark 1: (cd t && TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh)' in 'HEAD~0
	  Time (mean ± σ):      4.191 s ±  0.012 s    [User: 3.600 s, System: 0.746 s]
	  Range (min … max):    4.181 s …  4.204 s    3 runs
	Benchmark 2: (cd t && TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh)' in 'HEAD~0
	  Time (mean ± σ):      5.945 s ±  0.101 s    [User: 4.989 s, System: 1.146 s]
	  Range (min … max):    5.878 s …  6.062 s    3 runs
	
	Summary
	  '(cd t && TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh)' in 'HEAD~0' ran
	    1.42 ± 0.02 times faster than '(cd t && TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh)' in 'HEAD~0'
Quoted 4 lines

I.e. I get that it's catching actual issues, but I was also doing runs
with SANITIZE=address, which I believe are going to catch a superset of issues that this check does, so...

Quoted 4 lines

I assumed SANITIZE=address would catch a superset of issues as well but I haven't actually checked the glibc tunables documentation. We disable MALLOC_PERTURB_ when running under valgrind so perhaps we should do the same when compiling with SANITIZE=address.

I'm not sure either, but given how exhaustive SANITIZE=address is I'd be surprised if not.

Quoted 6 lines

I just noticed that setup_malloc_check() is called by test_expect_success() and test_when_finished() so it really should be caching the result of the check rather than forking getconf and expr each time it is called. Overwriting LD_PRELOAD is not very friendly either, it would be better if it appended the debug library if the variable is already set.

We really should just be checking this when building, we even have C code already that detects glibc and its version, I have some local (but semi-unrelated) patches. Anyway...

Quoted 9 lines
Quoted 6 lines

Whatever we do with this narrow patch it would be a really nice improvement if the test-lib.sh could fold all of these "instrumentations" behind a single flag, and that both it and "make test" would make it clear that you're testing in a slower "tracing" or "instrumentation" mode.
Ditto things like chain lint and the bin-wrappers, e.g.:

I sometimes wish there was a way to only chain lint the tests that have changed since the last run.

Mm, perhaps some make-based solution... :)

I had some experiments to go even further, and have "make test" only run the tests relevant to the code that just changed, which with trace2's filenames and GCC/clang's -MF option you can bridge that gap.

Quoted 21 lines
Quoted 12 lines
     $ git hyperfine -L rev HEAD~0 -L off yes, -L cl 0,1 -L nbw --no-bin-wrappers, -s 'make CFLAGS=-O3' '(cd t && GIT_TEST_CHAIN_LINT={cl} TEST_NO_MALLOC_CHECK={off} ./t3070-wildmatch.sh {nbw})' -r 1
     [...]	
	Summary
	  '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0' ran
	    1.23 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.30 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    1.54 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK=yes ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.63 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    1.87 times faster than '(cd t && GIT_TEST_CHAIN_LINT=0 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh )' in 'HEAD~0'
	    1.92 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh --no-bin-wrappers)' in 'HEAD~0'
	    2.24 times faster than '(cd t && GIT_TEST_CHAIN_LINT=1 TEST_NO_MALLOC_CHECK= ./t3070-wildmatch.sh )' in 'HEAD~0'
I.e. between this, chain lint and bin wrappers we're coming up on
Quoted 8 lines
our
tests running almost 3x as slow as they otherwise could *by default*.
But right now knowing which things you need to chase around to turn
off
if you're just looking to test the semantics of your code without all
this instrumentation is a matter of archane knowledge, I'm not even sure
I remembered all the major ones (I didn't know about this one until
today).
Quoted 3 lines

That is quite a difference in run time - I wonder how much scope there is for optimizing some of these features like the chain-lint vs disabling them completely.

Per my series at
https://lore.kernel.org/git/[email protected]/
I'd much rather see us go in the direction of mainly piggy-backing on CI for such extended testing, and just having easy to use targets for "do exhaustive tests please".

E.g. so you could run "make test-like-ci" or whatever, and it would do all the N permutations we do in CI locally.