Skip to contentSkip to conversation

ci: make sure we build Git parallel

PATCH In review SZEDER Gábor 3 participants 11 messages Feb 7, 2019 – Feb 8, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ci: make sure we build Git parallel

Commit 2c8921db2b (travis-ci: build with the right compiler, 2019-01-17) started to use MAKEFLAGS to specify which compiler to use to build Git. A bit later, and in a different topic branch commit eaa62291ff (ci: inherit --jobs via MAKEFLAGS in run-build-and-tests, 2019-01-27) started to use MAKEFLAGS as well. Unfortunately, there is a semantic conflict between these two commits: both of them set MAKEFLAGS, and since the line adding CC from 2c8921db2b comes later in 'ci/lib.sh', it overwrites the number of parallel jobs added in eaa62291ff.

Consequently, since both commits have been merged all our CI jobs have been building Git, building its documentation, and applying semantic patches sequentially, making all build jobs a bit slower. Running the test suite is unaffected, because the number of test jobs comes from GIT_PROVE_OPTS.

Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved.

---
 ci/lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ci/lib.sh +1 -1
index 16f4ecbc67..cee51a4cc4 100755
@@ -185,4 +185,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="CC=${CC:-cc}"
+export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Signature · 1 line
2.20.1.940.g8404bb2d1a
#1 lore raw
Quoted 13 lines · SZEDER Gábor

Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved.

---
 ci/lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ci/lib.sh +0 -0
index 16f4ecbc67..cee51a4cc4 100755

ci/lib.sh GIT_TEST_GETTEXT_POISON) · line 188

Quoted 5 lines
ci/lib.sh +1 -1
@@ -185,4 +185,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="CC=${CC:-cc}"
+export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"

Makes sense. Thanks.

#2 lore raw
Quoted 18 lines · Junio C Hamano
Quoted 16 lines · SZEDER Gábor

Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved.

---
 ci/lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ci/lib.sh +0 -0
index 16f4ecbc67..cee51a4cc4 100755
@@ -185,4 +185,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
Quoted 4 lines

ci/lib.sh GIT_TEST_GETTEXT_POISON) · line 188

Quoted 2 lines
ci/lib.sh +1 -1
-export MAKEFLAGS="CC=${CC:-cc}"
+export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"

Makes sense. Thanks.

Wouldn't all other hits of "MAKEFLAGS=" in ci/lib.sh also want the same treatment, though? We know that "if travis to this, otherwise if Asure, do that" is the first block to muck with MAKEFLAGS in the script, but a new assignment before that block can be added in the future and cause a similar issue unless we do so.

Of course, at some point we do want to say "we do not want to inherit it from the outside environment", but then such an assignment of empty value should be done at the very beginning with a comment, not with "this happens to be the first one to set, so let's not append but assign to clear any previous value", I would think.

#3 lore raw
Quoted 6 lines · Junio C Hamano

Wouldn't all other hits of "MAKEFLAGS=" in ci/lib.sh also want the same treatment, though? We know that "if travis to this, otherwise if Asure, do that" is the first block to muck with MAKEFLAGS in the script, but a new assignment before that block can be added in the future and cause a similar issue unless we do so.

Quoted 6 lines

Of course, at some point we do want to say "we do not want to inherit it from the outside environment", but then such an assignment of empty value should be done at the very beginning with a comment, not with "this happens to be the first one to set, so let's not append but assign to clear any previous value", I would think.

That is, in a patch form on top of yours, something like this.

 ci/lib.sh | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
ci/lib.sh +6 -3
index cee51a4cc4..288a5b3884 100755
@@ -74,6 +74,9 @@ check_unignored_build_artifacts ()
 	}
 }
 
+# Clear MAKEFLAGS that may come from the outside world.
+export MAKEFLAGS=
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then
 	BREW_INSTALL_PACKAGES="git-lfs gettext"
 	export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --immediate"
-	export MAKEFLAGS="--jobs=2"
+	MAKEFLAGS="$MAKEFLAGS --jobs=2"
 elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
 	CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then
 	BREW_INSTALL_PACKAGES=gcc@8
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
-	export MAKEFLAGS="--jobs=10"
+	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test windows_nt != "$CI_OS_NAME" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
+MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
#4 lore raw

Hi Junio,

Quoted 54 lines · Junio C Hamano
Quoted 6 lines · Junio C Hamano

Wouldn't all other hits of "MAKEFLAGS=" in ci/lib.sh also want the same treatment, though? We know that "if travis to this, otherwise if Asure, do that" is the first block to muck with MAKEFLAGS in the script, but a new assignment before that block can be added in the future and cause a similar issue unless we do so.

Quoted 6 lines

Of course, at some point we do want to say "we do not want to inherit it from the outside environment", but then such an assignment of empty value should be done at the very beginning with a comment, not with "this happens to be the first one to set, so let's not append but assign to clear any previous value", I would think.

That is, in a patch form on top of yours, something like this.

 ci/lib.sh | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
ci/lib.sh +5 -2
index cee51a4cc4..288a5b3884 100755
@@ -74,6 +74,9 @@ check_unignored_build_artifacts ()
 	}
 }
 
+# Clear MAKEFLAGS that may come from the outside world.
+export MAKEFLAGS=
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then
 	BREW_INSTALL_PACKAGES="git-lfs gettext"
 	export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --immediate"
-	export MAKEFLAGS="--jobs=2"
+	MAKEFLAGS="$MAKEFLAGS --jobs=2"
 elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
 	CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then
 	BREW_INSTALL_PACKAGES=gcc@8
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
-	export MAKEFLAGS="--jobs=10"
+	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test windows_nt != "$CI_OS_NAME" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else

ci/lib.sh GIT_TEST_GETTEXT_POISON) · line 191

Quoted 5 lines
ci/lib.sh +1 -1
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
+MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"

Since this is intended to be run in a CI setting, there is not a whole lot of opportunity to set MAKEFLAGS outside of the script. And if there is, that might open a rabbit hole when debugging issues that somehow in the end turn out to come from a hard-coded MAKEFLAGS somewhere in the CI system.

So I'd rather clear MAKEFLAGS at the beginning (i.e. where you export MAKEFLAGS, I'd simply append a =).

Ciao,
Dscho

#5 lore raw
Quoted 38 lines · Johannes Schindelin
Quoted 26 lines
ci/lib.sh +5 -2
+# Clear MAKEFLAGS that may come from the outside world.
+export MAKEFLAGS=
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then
 	BREW_INSTALL_PACKAGES="git-lfs gettext"
 	export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --immediate"
-	export MAKEFLAGS="--jobs=2"
+	MAKEFLAGS="$MAKEFLAGS --jobs=2"
 elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
 	CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then
 	BREW_INSTALL_PACKAGES=gcc@8
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
-	export MAKEFLAGS="--jobs=10"
+	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test windows_nt != "$CI_OS_NAME" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else

ci/lib.sh GIT_TEST_GETTEXT_POISON) · line 191

Quoted 5 lines
ci/lib.sh +1 -1
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
+MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"

Since this is intended to be run in a CI setting, there is not a whole lot of opportunity to set MAKEFLAGS outside of the script. And if there is, that might open a rabbit hole when debugging issues that somehow in the end turn out to come from a hard-coded MAKEFLAGS somewhere in the CI system.

Quoted 2 lines

So I'd rather clear MAKEFLAGS at the beginning (i.e. where you export MAKEFLAGS, I'd simply append a =).

I meant to clear it at the beginning, where I "export MAKEFLAGS=". Did your MUA ate the equal sign at the end, mistaking it with part of text/plain; format=flawed or something?

#6 lore raw

PATCH 1/1 ci: clear and mark MAKEFLAGS exported just once

Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again.

Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly.

---

Quoted 2 lines
Quoted 2 lines

So I'd rather clear MAKEFLAGS at the beginning (i.e. where you export MAKEFLAGS, I'd simply append a =).

This time in proper patch form.

 ci/lib.sh | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
ci/lib.sh +6 -3
index cee51a4cc4..288a5b3884 100755
@@ -74,6 +74,9 @@ check_unignored_build_artifacts ()
 	}
 }
 
+# Clear MAKEFLAGS that may come from the outside world.
+export MAKEFLAGS=
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then
 	BREW_INSTALL_PACKAGES="git-lfs gettext"
 	export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --immediate"
-	export MAKEFLAGS="--jobs=2"
+	MAKEFLAGS="$MAKEFLAGS --jobs=2"
 elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
 	CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then
 	BREW_INSTALL_PACKAGES=gcc@8
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
-	export MAKEFLAGS="--jobs=10"
+	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test windows_nt != "$CI_OS_NAME" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
+MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Signature · 1 line
2.21.0-rc0
#7 lore raw

Re: [PATCH] ci: clear and mark MAKEFLAGS exported just once

Quoted 7 lines · Junio C Hamano

Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again.

Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly.

Quoted 6 lines

---

Quoted 2 lines
Quoted 2 lines

So I'd rather clear MAKEFLAGS at the beginning (i.e. where you export MAKEFLAGS, I'd simply append a =).

This time in proper patch form.

Makes sense, and the patch looks good to me.

Quoted 44 lines
 ci/lib.sh | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
ci/lib.sh +6 -3
index cee51a4cc4..288a5b3884 100755
@@ -74,6 +74,9 @@ check_unignored_build_artifacts ()
 	}
 }
 
+# Clear MAKEFLAGS that may come from the outside world.
+export MAKEFLAGS=
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then
 	BREW_INSTALL_PACKAGES="git-lfs gettext"
 	export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --immediate"
-	export MAKEFLAGS="--jobs=2"
+	MAKEFLAGS="$MAKEFLAGS --jobs=2"
 elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
 	CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then
 	BREW_INSTALL_PACKAGES=gcc@8
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
-	export MAKEFLAGS="--jobs=10"
+	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test windows_nt != "$CI_OS_NAME" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
+MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Signature · 1 line
2.21.0-rc0
#8 lore raw

Re: [PATCH] ci: clear and mark MAKEFLAGS exported just once

Hi,

Quoted 8 lines · SZEDER Gábor
Quoted 7 lines · Junio C Hamano

Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again.

Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly.

Quoted 8 lines
Quoted 6 lines

---

Quoted 2 lines
Quoted 2 lines

So I'd rather clear MAKEFLAGS at the beginning (i.e. where you export MAKEFLAGS, I'd simply append a =).

This time in proper patch form.

Makes sense, and the patch looks good to me.

To me, too.

Thank you,
Dscho

Quoted 46 lines
Quoted 45 lines
 ci/lib.sh | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
ci/lib.sh +6 -3
index cee51a4cc4..288a5b3884 100755
@@ -74,6 +74,9 @@ check_unignored_build_artifacts ()
 	}
 }
 
+# Clear MAKEFLAGS that may come from the outside world.
+export MAKEFLAGS=
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then
 	BREW_INSTALL_PACKAGES="git-lfs gettext"
 	export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --immediate"
-	export MAKEFLAGS="--jobs=2"
+	MAKEFLAGS="$MAKEFLAGS --jobs=2"
 elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
 	CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then
 	BREW_INSTALL_PACKAGES=gcc@8
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
-	export MAKEFLAGS="--jobs=10"
+	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test windows_nt != "$CI_OS_NAME" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
+MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Signature · 1 line
2.21.0-rc0
#9 lore raw
In reply to Junio C Hamano

Hi Junio,

Quoted 40 lines · Junio C Hamano
Quoted 38 lines · Johannes Schindelin
Quoted 26 lines
ci/lib.sh +5 -2
+# Clear MAKEFLAGS that may come from the outside world.
+export MAKEFLAGS=
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then
 	BREW_INSTALL_PACKAGES="git-lfs gettext"
 	export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --immediate"
-	export MAKEFLAGS="--jobs=2"
+	MAKEFLAGS="$MAKEFLAGS --jobs=2"
 elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
 	CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then
 	BREW_INSTALL_PACKAGES=gcc@8
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
-	export MAKEFLAGS="--jobs=10"
+	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test windows_nt != "$CI_OS_NAME" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else

ci/lib.sh GIT_TEST_GETTEXT_POISON) · line 191

Quoted 5 lines
ci/lib.sh +1 -1
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON)
 	;;
 esac
 
-export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
+MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"

Since this is intended to be run in a CI setting, there is not a whole lot of opportunity to set MAKEFLAGS outside of the script. And if there is, that might open a rabbit hole when debugging issues that somehow in the end turn out to come from a hard-coded MAKEFLAGS somewhere in the CI system.

Quoted 6 lines
Quoted 2 lines

So I'd rather clear MAKEFLAGS at the beginning (i.e. where you export MAKEFLAGS, I'd simply append a =).

I meant to clear it at the beginning, where I "export MAKEFLAGS=". Did your MUA ate the equal sign at the end, mistaking it with part of text/plain; format=flawed or something?

I could have sworn that there was no equal sign yesterday.

Sorry for the noise,
Dscho

#10 lore raw
Quoted 15 lines · Johannes Schindelin

Hi Junio,

Quoted 10 lines · Junio C Hamano
Quoted 4 lines · Johannes Schindelin

ci/lib.sh check_unignored_build_artifacts () · line 79

Quoted 4 lines
ci/lib.sh +3 -0
+# Clear MAKEFLAGS that may come from the outside world.
+export MAKEFLAGS=
+

...
I meant to clear it at the beginning, where I "export MAKEFLAGS=". Did your MUA ate the equal sign at the end, mistaking it with part of text/plain; format=flawed or something?

Quoted 3 lines

I could have sworn that there was no equal sign yesterday.

Sorry for the noise,

Don't be. Mistakes happen and watching stupid mistakes for each other is part of the teamwork I very much appreciate.

Will apply on top of Szeder's fix. Thanks.