2.20.0 - Undocumented change in submodule update wrt # parallel jobs
When switching to 2.20 our `git submodule update' (which clones through ssh) broke because our ssh-server rejected the ~20 simultaneous connections the git-client makes. This seems to be caused by a (possibly unintended) change in
https://github.com/git/git/commit/90efe595c53f4bb1851371344c35eff71f604d2b
which removed the default of max_jobs=1
While this can easily be fixed by configuring submodule.fetchJobs I think this change should be documented - or reverted back to it's previous default of 1
--
Kind regards,
Sjon Hortensius
| Parse Software Development B.V.
| http://www.parse.nl
Quoted 10 lines · Sjon Hortensius
When switching to 2.20 our `git submodule update' (which clones through ssh) broke because our ssh-server rejected the ~20 simultaneous connections the git-client makes. This seems to be caused by a (possibly unintended) change in
https://github.com/git/git/commit/90efe595c53f4bb1851371344c35eff71f604d2b
which removed the default of max_jobs=1While this can easily be fixed by configuring submodule.fetchJobs I think this change should be documented - or reverted back to it's previous default of 1
Just to add to this. SSH-ing with -j<ncores> (which I assume that ~20 is) is going to run into MaxStartups in sshd_config, which is especially annoying as it's pre-ssh-auth, so the server doesn't even get "so and so tried to login", it's just dropped as a potential DoS attack.
Quoted 10 lines · Sjon Hortensius
When switching to 2.20 our `git submodule update' (which clones through ssh) broke because our ssh-server rejected the ~20 simultaneous connections the git-client makes. This seems to be caused by a (possibly unintended) change in
https://github.com/git/git/commit/90efe595c53f4bb1851371344c35eff71f604d2b
which removed the default of max_jobs=1While this can easily be fixed by configuring submodule.fetchJobs I think this change should be documented - or reverted back to it's previous default of 1
The commit in question does not look like it _wanted_ to change the default; rather, it appears to me that it wanted to be bug-to-bug compatible with the original, and any such change of behaviour is entirely unintended.
I think the attached may be sufficient to change the default max_jobs back to 1.
By the way, is there a place where we document that the default value for fetchjobs, when unconfigured, is 1? If we are not making such a concrete promise, then I would think it is OK to update the default without any fanfare, as long as we have good reasons to do so. For this particular one, however, as I already said, I do not think we wanted to change the default to unlimited or anything like that, so...
builtin/submodule--helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
builtin/submodule--helper.c +1 -1
@@ -1552,7 +1552,7 @@ struct submodule_update_clone {#define SUBMODULE_UPDATE_CLONE_INIT {0, MODULE_LIST_INIT, 0, \ SUBMODULE_UPDATE_STRATEGY_INIT, 0, 0, -1, STRING_LIST_INIT_DUP, 0, \ NULL, NULL, NULL, \ - NULL, 0, 0, 0, NULL, 0, 0, 0} + NULL, 0, 0, 0, NULL, 0, 0, 1} static void next_submodule_warn_missing(struct submodule_update_clone *suc,
Quoted 10 lines · Junio C Hamano
Quoted 7 lines · Sjon Hortensius
When switching to 2.20 our `git submodule update' (which clones through ssh) broke because our ssh-server rejected the ~20 simultaneous connections the git-client makes. This seems to be caused by a (possibly unintended) change in
https://github.com/git/git/commit/90efe595c53f4bb1851371344c35eff71f604d2b
which removed the default of max_jobs=1
Quoted 8 lines
Quoted 3 lines
While this can easily be fixed by configuring submodule.fetchJobs I think this change should be documented - or reverted back to it's previous default of 1
The commit in question does not look like it _wanted_ to change the default; rather, it appears to me that it wanted to be bug-to-bug compatible with the original, and any such change of behaviour is entirely unintended.
Indeed.
Quoted 2 lines
I think the attached may be sufficient to change the default
max_jobsback to 1.
I think so, too. I can wrap it into a commit with a proper message.
Quoted 3 lines
By the way, is there a place where we document that the default value for fetchjobs, when unconfigured, is 1?
man git config
submodule.fetchJobs
Specifies how many submodules are fetched/cloned at the same time. A positive integer allows up to that number of submodules fetched in parallel. A value of 0 will give some reasonable default. If unset, it defaults to 1.
and that seems to be the only place, other places only reference this place:
Documentation$ git grep submodule.fetch config/submodule.txt:66:submodule.fetchJobs::
git-clone.txt:259: Defaults to the submodule.fetchJobs option.
git-submodule.txt:408: Defaults to the submodule.fetchJobs option.
The behavior of that seems to have been there since the beginning of a028a1930c (fetching submodules: respect submodule.fetchJobs config option, 2016-02-29)
Quoted 6 lines
If we are not making
such a concrete promise, then I would think it is OK to update the default without any fanfare, as long as we have good reasons to do so. For this particular one, however, as I already said, I do not think we wanted to change the default to unlimited or anything like that, so...
We definitely want the diff below as a proper patch.
Quoted 16 lines
builtin/submodule--helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)builtin/submodule--helper.c +1
-1@@ -1552,7 +1552,7 @@ struct submodule_update_clone {#define SUBMODULE_UPDATE_CLONE_INIT {0, MODULE_LIST_INIT, 0, \ SUBMODULE_UPDATE_STRATEGY_INIT, 0, 0, -1, STRING_LIST_INIT_DUP, 0, \ NULL, NULL, NULL, \ - NULL, 0, 0, 0, NULL, 0, 0, 0} + NULL, 0, 0, 0, NULL, 0, 0, 1} static void next_submodule_warn_missing(struct submodule_update_clone *suc,
PATCH 1/1 submodule update: run at most one fetch job unless otherwise set
From: Junio C Hamano <[email protected]>
In a028a1930c (fetching submodules: respect submodule.fetchJobs config option, 2016-02-29), we made sure to keep the default behavior of a fetching at most one submodule at once when not setting the newly introduced submodule.fetchJobs config.
This regressed in 90efe595c5 (builtin/submodule--helper: factor out submodule updating, 2018-08-03). Fix it.
--- builtin/submodule--helper.c | 2 +- t/t5526-fetch-submodules.sh | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
builtin/submodule--helper.c +1 -1
@@ -1551,7 +1551,7 @@ struct submodule_update_clone {#define SUBMODULE_UPDATE_CLONE_INIT {0, MODULE_LIST_INIT, 0, \ SUBMODULE_UPDATE_STRATEGY_INIT, 0, 0, -1, STRING_LIST_INIT_DUP, 0, \ NULL, NULL, NULL, \ - NULL, 0, 0, 0, NULL, 0, 0, 0} + NULL, 0, 0, 0, NULL, 0, 0, 1} static void next_submodule_warn_missing(struct submodule_update_clone *suc,
t/t5526-fetch-submodules.sh +2 -0
@@ -524,6 +524,8 @@ test_expect_success 'fetching submodules respects parallel settings' ' git config fetch.recurseSubmodules true && ( cd downstream && + GIT_TRACE=$(pwd)/trace.out git fetch && + grep "1 tasks" trace.out && GIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 && grep "7 tasks" trace.out && git config submodule.fetchJobs 8 &&
Signature · 1 line
2.20.0.rc2.230.gc28305e538
Re: [PATCH] submodule update: run at most one fetch job unless otherwise set
Quoted 3 lines · Stefan Beller
In
a028a1930c(fetching submodules: respectsubmodule.fetchJobsconfig option, 2016-02-29), we made sure to keep the default behavior of a fetching at most one submodule at once when not setting the
s/of a/of/
Quoted 8 lines
newly introduced
submodule.fetchJobsconfig.This regressed in
90efe595c5(builtin/submodule--helper: factor out submodule updating, 2018-08-03). Fix it.Reported-by Sjon Hortensius <[email protected]>Signed-off-by Junio C Hamano <[email protected]>Signed-off-by Stefan Beller <[email protected]>
Re: [PATCH] submodule update: run at most one fetch job unless otherwise set
In reply to Stefan BellerQuoted 7 lines · Stefan Beller
From: Junio C Hamano <[email protected]>
In
a028a1930c(fetching submodules: respectsubmodule.fetchJobsconfig option, 2016-02-29), we made sure to keep the default behavior of a fetching at most one submodule at once when not setting the newly introducedsubmodule.fetchJobsconfig.
Quoted 7 lines
This regressed in
90efe595c5(builtin/submodule--helper: factor out submodule updating, 2018-08-03). Fix it.Reported-by Sjon Hortensius <[email protected]>Signed-off-by Junio C Hamano <[email protected]>Signed-off-by Stefan Beller <[email protected]>---
Thanks for tying the loose ends.
We may want to convert the _INIT macro to use the designated initializers; I had to count the fields twice to make sure I was tweaking the right one.
It did not help that I saw, before looking at the current code, that 90efe595c5 added one field at the end to the struct but did not touch _INIT macro at all. That made me guess that max_jobs=0 was due to _missing_ initialization value, leading me to an incorrect fix to append an extra 1 at the end, but that was bogus. The missing initialization left by 90efe595 ("builtin/submodule--helper:
factor out submodule updating", 2018-08-03) was silently fixed by f1d15713 ("builtin/submodule--helper: store update_clone information in a struct", 2018-08-03), I think, so replacing the 0 at the end is 1 happens to be the right fix, but with designated initializers, all these confusions are more easily avoided.
Quoted 30 lines
builtin/submodule--helper.c | 2 +- t/t5526-fetch-submodules.sh | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)builtin/submodule--helper.c +1
-1@@ -1551,7 +1551,7 @@ struct submodule_update_clone {#define SUBMODULE_UPDATE_CLONE_INIT {0, MODULE_LIST_INIT, 0, \ SUBMODULE_UPDATE_STRATEGY_INIT, 0, 0, -1, STRING_LIST_INIT_DUP, 0, \ NULL, NULL, NULL, \ - NULL, 0, 0, 0, NULL, 0, 0, 0} + NULL, 0, 0, 0, NULL, 0, 0, 1} static void next_submodule_warn_missing(struct submodule_update_clone *suc,t/t5526-fetch-submodules.sh +2
-0@@ -524,6 +524,8 @@ test_expect_success 'fetching submodules respects parallel settings' ' git config fetch.recurseSubmodules true && ( cd downstream && + GIT_TRACE=$(pwd)/trace.out git fetch && + grep "1 tasks" trace.out && GIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 && grep "7 tasks" trace.out && git config submodule.fetchJobs 8 &&