ci: make sure we build Git parallel
PATCH 1/1 ci: make sure we build Git parallel
Commit 2c8921db2b (travis-ci: build with the right compiler, 2019-01-17) started to use MAKEFLAGS to specify which compiler to use to build Git. A bit later, and in a different topic branch commit eaa62291ff (ci: inherit --jobs via MAKEFLAGS in run-build-and-tests, 2019-01-27) started to use MAKEFLAGS as well. Unfortunately, there is a semantic conflict between these two commits: both of them set MAKEFLAGS, and since the line adding CC from 2c8921db2b comes later in 'ci/lib.sh', it overwrites the number of parallel jobs added in eaa62291ff.
Consequently, since both commits have been merged all our CI jobs have been building Git, building its documentation, and applying semantic patches sequentially, making all build jobs a bit slower. Running the test suite is unaffected, because the number of test jobs comes from GIT_PROVE_OPTS.
Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved.
--- ci/lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ci/lib.sh +1 -1
@@ -185,4 +185,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="CC=${CC:-cc}" +export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Signature · 1 line
2.20.1.940.g8404bb2d1a
Quoted 13 lines · SZEDER Gábor
Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved.
Signed-off-by SZEDER Gábor <[email protected]>--- ci/lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)ci/lib.sh +0
-0
ci/lib.sh GIT_TEST_GETTEXT_POISON) · line 188
Quoted 5 lines
ci/lib.sh +1
-1@@ -185,4 +185,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="CC=${CC:-cc}" +export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Makes sense. Thanks.
Quoted 18 lines · Junio C Hamano
Quoted 16 lines · SZEDER Gábor
Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved.
Signed-off-by SZEDER Gábor <[email protected]>--- ci/lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)ci/lib.sh +0
-0@@ -185,4 +185,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac
Quoted 4 lines
ci/lib.sh GIT_TEST_GETTEXT_POISON) · line 188
Quoted 2 lines
ci/lib.sh +1
-1-export MAKEFLAGS="CC=${CC:-cc}" +export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"Makes sense. Thanks.
Wouldn't all other hits of "MAKEFLAGS=" in ci/lib.sh also want the same treatment, though? We know that "if travis to this, otherwise if Asure, do that" is the first block to muck with MAKEFLAGS in the script, but a new assignment before that block can be added in the future and cause a similar issue unless we do so.
Of course, at some point we do want to say "we do not want to inherit it from the outside environment", but then such an assignment of empty value should be done at the very beginning with a comment, not with "this happens to be the first one to set, so let's not append but assign to clear any previous value", I would think.
Quoted 6 lines · Junio C Hamano
Wouldn't all other hits of "MAKEFLAGS=" in
ci/lib.shalso want the same treatment, though? We know that "if travis to this, otherwise if Asure, do that" is the first block to muck with MAKEFLAGS in the script, but a new assignment before that block can be added in the future and cause a similar issue unless we do so.
Quoted 6 lines
Of course, at some point we do want to say "we do not want to inherit it from the outside environment", but then such an assignment of empty value should be done at the very beginning with a comment, not with "this happens to be the first one to set, so let's not append but assign to clear any previous value", I would think.
That is, in a patch form on top of yours, something like this.
ci/lib.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
ci/lib.sh +6 -3
@@ -74,6 +74,9 @@ check_unignored_build_artifacts () } } +# Clear MAKEFLAGS that may come from the outside world. +export MAKEFLAGS= + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then BREW_INSTALL_PACKAGES="git-lfs gettext" export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --immediate" - export MAKEFLAGS="--jobs=2" + MAKEFLAGS="$MAKEFLAGS --jobs=2" elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then BREW_INSTALL_PACKAGES=gcc@8 export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - export MAKEFLAGS="--jobs=10" + MAKEFLAGS="$MAKEFLAGS --jobs=10" test windows_nt != "$CI_OS_NAME" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" else
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Hi Junio,
Quoted 54 lines · Junio C Hamano
Quoted 6 lines · Junio C Hamano
Wouldn't all other hits of "MAKEFLAGS=" in
ci/lib.shalso want the same treatment, though? We know that "if travis to this, otherwise if Asure, do that" is the first block to muck with MAKEFLAGS in the script, but a new assignment before that block can be added in the future and cause a similar issue unless we do so.Quoted 6 lines
Of course, at some point we do want to say "we do not want to inherit it from the outside environment", but then such an assignment of empty value should be done at the very beginning with a comment, not with "this happens to be the first one to set, so let's not append but assign to clear any previous value", I would think.
That is, in a patch form on top of yours, something like this.
ci/lib.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)ci/lib.sh +5
-2@@ -74,6 +74,9 @@ check_unignored_build_artifacts () } } +# Clear MAKEFLAGS that may come from the outside world. +export MAKEFLAGS= + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # Set tracing executed commands, primarily setting environment variables@@ -101,7 +104,7 @@ then BREW_INSTALL_PACKAGES="git-lfs gettext" export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --immediate" - export MAKEFLAGS="--jobs=2" + MAKEFLAGS="$MAKEFLAGS --jobs=2" elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines@@ -126,7 +129,7 @@ then BREW_INSTALL_PACKAGES=gcc@8 export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - export MAKEFLAGS="--jobs=10" + MAKEFLAGS="$MAKEFLAGS --jobs=10" test windows_nt != "$CI_OS_NAME" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" else
ci/lib.sh GIT_TEST_GETTEXT_POISON) · line 191
Quoted 5 lines
ci/lib.sh +1
-1@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Since this is intended to be run in a CI setting, there is not a whole lot of opportunity to set MAKEFLAGS outside of the script. And if there is, that might open a rabbit hole when debugging issues that somehow in the end turn out to come from a hard-coded MAKEFLAGS somewhere in the CI system.
So I'd rather clear MAKEFLAGS at the beginning (i.e. where you export MAKEFLAGS, I'd simply append a =).
Ciao,
Dscho
Quoted 38 lines · Johannes Schindelin
Quoted 26 lines
ci/lib.sh +5
-2+# Clear MAKEFLAGS that may come from the outside world. +export MAKEFLAGS= + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # Set tracing executed commands, primarily setting environment variables@@ -101,7 +104,7 @@ then BREW_INSTALL_PACKAGES="git-lfs gettext" export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --immediate" - export MAKEFLAGS="--jobs=2" + MAKEFLAGS="$MAKEFLAGS --jobs=2" elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines@@ -126,7 +129,7 @@ then BREW_INSTALL_PACKAGES=gcc@8 export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - export MAKEFLAGS="--jobs=10" + MAKEFLAGS="$MAKEFLAGS --jobs=10" test windows_nt != "$CI_OS_NAME" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" elseci/lib.sh GIT_TEST_GETTEXT_POISON) · line 191
Quoted 5 lines
ci/lib.sh +1
-1@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"Since this is intended to be run in a CI setting, there is not a whole lot of opportunity to set
MAKEFLAGSoutside of the script. And if there is, that might open a rabbit hole when debugging issues that somehow in the end turn out to come from a hard-codedMAKEFLAGSsomewhere in the CI system.
Quoted 2 lines
So I'd rather clear
MAKEFLAGSat the beginning (i.e. where youexport MAKEFLAGS, I'd simply append a=).
I meant to clear it at the beginning, where I "export MAKEFLAGS=". Did your MUA ate the equal sign at the end, mistaking it with part of text/plain; format=flawed or something?
PATCH 1/1 ci: clear and mark MAKEFLAGS exported just once
Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again.
Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly.
---
Quoted 2 lines
Quoted 2 lines
So I'd rather clear
MAKEFLAGSat the beginning (i.e. where youexport MAKEFLAGS, I'd simply append a=).
This time in proper patch form.
ci/lib.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
ci/lib.sh +6 -3
@@ -74,6 +74,9 @@ check_unignored_build_artifacts () } } +# Clear MAKEFLAGS that may come from the outside world. +export MAKEFLAGS= + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # Set tracing executed commands, primarily setting environment variables
@@ -101,7 +104,7 @@ then BREW_INSTALL_PACKAGES="git-lfs gettext" export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --immediate" - export MAKEFLAGS="--jobs=2" + MAKEFLAGS="$MAKEFLAGS --jobs=2" elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines
@@ -126,7 +129,7 @@ then BREW_INSTALL_PACKAGES=gcc@8 export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - export MAKEFLAGS="--jobs=10" + MAKEFLAGS="$MAKEFLAGS --jobs=10" test windows_nt != "$CI_OS_NAME" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" else
@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
Signature · 1 line
2.21.0-rc0
Re: [PATCH] ci: clear and mark MAKEFLAGS exported just once
Quoted 7 lines · Junio C Hamano
Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again.
Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly.
Quoted 6 lines
Signed-off-by Junio C Hamano <[email protected]>---
Quoted 2 lines
Quoted 2 lines
So I'd rather clear
MAKEFLAGSat the beginning (i.e. where youexport MAKEFLAGS, I'd simply append a=).This time in proper patch form.
Makes sense, and the patch looks good to me.
Quoted 44 lines
ci/lib.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)ci/lib.sh +6
-3@@ -74,6 +74,9 @@ check_unignored_build_artifacts () } } +# Clear MAKEFLAGS that may come from the outside world. +export MAKEFLAGS= + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # Set tracing executed commands, primarily setting environment variables@@ -101,7 +104,7 @@ then BREW_INSTALL_PACKAGES="git-lfs gettext" export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --immediate" - export MAKEFLAGS="--jobs=2" + MAKEFLAGS="$MAKEFLAGS --jobs=2" elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines@@ -126,7 +129,7 @@ then BREW_INSTALL_PACKAGES=gcc@8 export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - export MAKEFLAGS="--jobs=10" + MAKEFLAGS="$MAKEFLAGS --jobs=10" test windows_nt != "$CI_OS_NAME" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" else@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"Signature · 1 line
2.21.0-rc0
Re: [PATCH] ci: clear and mark MAKEFLAGS exported just once
Hi,
Quoted 8 lines · SZEDER Gábor
Quoted 7 lines · Junio C Hamano
Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again.
Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly.
Quoted 8 lines
Quoted 6 lines
Signed-off-by Junio C Hamano <[email protected]>---
Quoted 2 lines
Quoted 2 lines
So I'd rather clear
MAKEFLAGSat the beginning (i.e. where youexport MAKEFLAGS, I'd simply append a=).This time in proper patch form.
Makes sense, and the patch looks good to me.
To me, too.
Thank you,
Dscho
Quoted 46 lines
Quoted 45 lines
ci/lib.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)ci/lib.sh +6
-3@@ -74,6 +74,9 @@ check_unignored_build_artifacts () } } +# Clear MAKEFLAGS that may come from the outside world. +export MAKEFLAGS= + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # Set tracing executed commands, primarily setting environment variables@@ -101,7 +104,7 @@ then BREW_INSTALL_PACKAGES="git-lfs gettext" export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --immediate" - export MAKEFLAGS="--jobs=2" + MAKEFLAGS="$MAKEFLAGS --jobs=2" elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines@@ -126,7 +129,7 @@ then BREW_INSTALL_PACKAGES=gcc@8 export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - export MAKEFLAGS="--jobs=10" + MAKEFLAGS="$MAKEFLAGS --jobs=10" test windows_nt != "$CI_OS_NAME" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" else@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"Signature · 1 line
2.21.0-rc0
Hi Junio,
Quoted 40 lines · Junio C Hamano
Quoted 38 lines · Johannes Schindelin
Quoted 26 lines
ci/lib.sh +5
-2+# Clear MAKEFLAGS that may come from the outside world. +export MAKEFLAGS= + # Set 'exit on error' for all CI scripts to let the caller know that # something went wrong. # Set tracing executed commands, primarily setting environment variables@@ -101,7 +104,7 @@ then BREW_INSTALL_PACKAGES="git-lfs gettext" export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --immediate" - export MAKEFLAGS="--jobs=2" + MAKEFLAGS="$MAKEFLAGS --jobs=2" elif test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI" then CI_TYPE=azure-pipelines@@ -126,7 +129,7 @@ then BREW_INSTALL_PACKAGES=gcc@8 export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml" - export MAKEFLAGS="--jobs=10" + MAKEFLAGS="$MAKEFLAGS --jobs=10" test windows_nt != "$CI_OS_NAME" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" elseci/lib.sh GIT_TEST_GETTEXT_POISON) · line 191
Quoted 5 lines
ci/lib.sh +1
-1@@ -185,4 +188,4 @@ GIT_TEST_GETTEXT_POISON) ;; esac -export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}" +MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"Since this is intended to be run in a CI setting, there is not a whole lot of opportunity to set
MAKEFLAGSoutside of the script. And if there is, that might open a rabbit hole when debugging issues that somehow in the end turn out to come from a hard-codedMAKEFLAGSsomewhere in the CI system.
Quoted 6 lines
Quoted 2 lines
So I'd rather clear
MAKEFLAGSat the beginning (i.e. where youexport MAKEFLAGS, I'd simply append a=).I meant to clear it at the beginning, where I "export MAKEFLAGS=". Did your MUA ate the equal sign at the end, mistaking it with part of text/plain; format=flawed or something?
I could have sworn that there was no equal sign yesterday.
Sorry for the noise,
Dscho
Quoted 15 lines · Johannes Schindelin
Hi Junio,
Quoted 10 lines · Junio C Hamano
Quoted 4 lines · Johannes Schindelin
ci/lib.sh check_unignored_build_artifacts () · line 79
Quoted 4 lines
ci/lib.sh +3
-0+# Clear MAKEFLAGS that may come from the outside world. +export MAKEFLAGS= +...
I meant to clear it at the beginning, where I "export MAKEFLAGS=". Did your MUA ate the equal sign at the end, mistaking it with part of text/plain; format=flawed or something?
Quoted 3 lines
I could have sworn that there was no equal sign yesterday.
Sorry for the noise,
Don't be. Mistakes happen and watching stupid mistakes for each other is part of the teamwork I very much appreciate.
Will apply on top of Szeder's fix. Thanks.