BUG in fetching non-checked out submodule
Hi,
I have the current 'master' branch of git installed and get the following error when fetching a submodule that is not checked out.
I've bisected this error down to commit 1b7ac4e6d4 (submodules: fix of regression on fetching of non-init subsub-repo, 2020-11-12)
$ git version git version 2.29.2.435.g72ffeb997e
$ git config --get submodule.recurse true
$ git submodule status -855827c583bc30645ba427885caa40c5b81764d2 sha1collisiondetection
$ git fetch Fetching submodule sha1collisiondetection Fetching submodule sha1collisiondetection/sha1collisiondetection Fetching submodule sha1collisiondetection/sha1collisiondetection/sha1collisiondetection Fetching submodule sha1collisiondetection/sha1collisiondetection/sha1collisiondetection/sha1collisiondetection ...
$ git submodule update --checkout Submodule path 'sha1collisiondetection': checked out '855827c583bc30645ba427885caa40c5b81764d2'
$ git fetch Fetching submodule sha1collisiondetection $
Ralf
Hi Ralf,
Quoted 11 lines · Ralf Thielow
Hi,
I have the current 'master' branch of git installed and get the following error when fetching a submodule that is not checked out.
I've bisected this error down to commit
1b7ac4e6d4(submodules: fix of regression on fetching of non-init subsub-repo, 2020-11-12)
Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).
As you discovered, it breaks other scenarios.
Quoted 6 lines
$ git version git version 2.29.2.435.g72ffeb997e$ git config --get submodule.recurse true
Yeah, I think the test suite could make more efforts to run more tests with that setting turned 'on', but it would require significants efforts since it changes the behaviour of several commands.
Meta question: is there an easy way to run the whole test suite with specific config options turned on ?
Quoted 13 lines
$ git submodule status -855827c583bc30645ba427885caa40c5b81764d2 sha1collisiondetection$ git fetch Fetching submodule sha1collisiondetection Fetching submodule sha1collisiondetection/sha1collisiondetection Fetching submodule sha1collisiondetection/sha1collisiondetection/sha1collisiondetection Fetching submodule sha1collisiondetection/sha1collisiondetection/sha1collisiondetection/sha1collisiondetection ...
Quoted 3 lines
$ git submodule update --checkout Submodule path 'sha1collisiondetection': checked out '855827c583bc30645ba427885caa40c5b81764d2'
Ok, you don't add '--init' but the submodule gets checked out, so it looks like you have 'submodule.active' set to a pathspec that matches 'sha1collisiondetection'. Did you clone the git repo with '--recurse-submodules', which would add '.' as the value of 'submodule.active' ? Or maybe you manually configured that value in your global gitconfig ?
Thanks for the report,
Philippe.
Quoted 5 lines · Philippe Blain
Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).
Quoted 8 lines
As you discovered, it breaks other scenarios.
Quoted 6 lines
$ git version git version 2.29.2.435.g72ffeb997e$ git config --get submodule.recurse true
I think the current situation is probably worse.
As a short-term fix, we should revert 1b7ac4e6d4 until we can come up with a real fix, probably.
Quoted 4 lines
Yeah, I think the test suite could make more efforts to run more tests with that setting turned 'on', but it would require significants efforts since it changes the behaviour of several commands.
I am not sure if the question is about amount of efforts.
A configuration variable is there to change the behaviour of commands, so a test of a command that has been running happily and producing a set of expected outcome with a configuration unset should break the expectation when the configuration is set --- otherwise there is no point in having a configuration variable.
Quoted 2 lines
Meta question: is there an easy way to run the whole test suite with specific config options turned on ?
Hence, I do not think it even makes sense to have such an "easy way". If the "fetch" command, for example, is expected to change behaviour depending on the value of submodule.recurse, a test written for the case where the variable is not set should produce different outcome when the variable is set.
What we need may be a better test coverage. submodule.recurse is a later addition, and all tests written earlier do test how the commands behave without the configuration being set. If one wants to change the behaviour of these commands when the configuration is set, new tests to specify what the expected behaviour need to be added.
Quoted 1 line
Thanks for the report,
Yup, thanks for helping out.
Hi,
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Philippe Blain
Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).As you discovered, it breaks other scenarios.
Quoted 4 lines
$ git version git version 2.29.2.435.g72ffeb997e
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
$ git config --get submodule.recurse trueI think the current situation is probably worse.
As a short-term fix, we should revert
1b7ac4e6d4until we can come up with a real fix, probably.
Junio: This is why I originally intended to commit the test case for the testsuite separated from the revert and wanted to start a discussion about the actual real fix for the issue:
https://public-inbox.org/git/[email protected]/
My proposal would be to revert 1b7ac4e6d4 and isolate the test case "test_expect_success 'setup nested submodule fetch test' '" make it "test_expect_failure" and apply it instead, until we come up with a real solution.
Quoted 22 lines
Quoted 4 lines
Yeah, I think the test suite could make more efforts to run more tests with that setting turned 'on', but it would require significants efforts since it changes the behaviour of several commands.
I am not sure if the question is about amount of efforts.
A configuration variable is there to change the behaviour of commands, so a test of a command that has been running happily and producing a set of expected outcome with a configuration unset should break the expectation when the configuration is set --- otherwise there is no point in having a configuration variable.
Quoted 2 lines
Meta question: is there an easy way to run the whole test suite with specific config options turned on ?
Hence, I do not think it even makes sense to have such an "easy way". If the "fetch" command, for example, is expected to change behaviour depending on the value of submodule.recurse, a test written for the case where the variable is not set should produce different outcome when the variable is set.
Quoted 5 lines
What we need may be a better test coverage. submodule.recurse is a later addition, and all tests written earlier do test how the commands behave without the configuration being set. If one wants to change the behaviour of these commands when the configuration is set, new tests to specify what the expected behaviour need to be added.
Can we start a new test suite for tests on which this variable is set? The case of Ralf can be used as first test case.
Ralf: could you please send a command sequence, which recreates the required repository setup from scratch and is able to trigger your observation? Then we can convert this into a test case.
Thanks.
Signature · 2 lines
best regards --peter;
Hello Peter,
Quoted 30 lines · Peter Kästle
Hi,
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Philippe Blain
Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).As you discovered, it breaks other scenarios.
Quoted 4 lines
$ git version git version 2.29.2.435.g72ffeb997eQuoted 7 lines
Quoted 2 lines
Quoted 2 lines
$ git config --get submodule.recurse trueI think the current situation is probably worse.
As a short-term fix, we should revert
1b7ac4e6d4until we can come up with a real fix, probably.Junio: This is why I originally intended to commit the test case for the testsuite separated from the revert and wanted to start a discussion about the actual real fix for the issue:
https://public-inbox.org/git/[email protected]/
Quoted 4 lines
My proposal would be to revert
1b7ac4e6d4and isolate the test case "test_expect_success'setup nested submodule fetch test' '" make it "test_expect_failure" and apply it instead, until we come up with a real solution.
I think I have the real solution. I did some debugging and I think it is quite easy:
In 'get_next_submodule', 'get_submodule_repo_for(spf->r, task->sub)' fails to get a repo pointer for the submodule repository, since it is not initialized. That
is normal. Then we go in the "else" branch, and hit this code:
/*
* An empty directory is normal,
* the submodule is not initialized
*/
if (S_ISGITLINK(ce->ce_mode) &&
!is_empty_dir(ce->name)) {
'is_empty_dir' receives ce->name, but the current working directory is the Git directory of 'middle', so clearly is_empty_dir returns false, as
/path/to/git/t/trash directory.t5526-fetch-submodules/B/.git/modules/middle/inner
is a non-existent path. The path that we should send is the worktree of inner, ie.
the concatenation of spf->r->worktree and ce->name. This would give
/path/to/git/t/trash directory.t5526-fetch-submodules/B/middle/inner,
which is an empty directory since the inner submodule is not initialized, and we would not get the "Could not access submodule inner" error that you wanted to solve.
Cheers,
Philippe.
Hi Philippe,
Quoted 57 lines · Philippe Blain
Quoted 27 lines · Junio C Hamano
Quoted 13 lines
Quoted 11 lines · Philippe Blain
Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).As you discovered, it breaks other scenarios.
Quoted 4 lines
$ git version git version 2.29.2.435.g72ffeb997eQuoted 7 lines
Quoted 2 lines
Quoted 2 lines
$ git config --get submodule.recurse trueI think the current situation is probably worse.
As a short-term fix, we should revert
1b7ac4e6d4until we can come up with a real fix, probably.Junio: This is why I originally intended to commit the test case for the testsuite separated from the revert and wanted to start a discussion about the actual real fix for the issue:
https://public-inbox.org/git/[email protected]/Quoted 4 lines
My proposal would be to revert
1b7ac4e6d4and isolate the test case "test_expect_success'setup nested submodule fetch test' '" make it "test_expect_failure" and apply it instead, until we come up with a real solution.I think I have the real solution. I did some debugging and I think it is quite easy:
In 'get_next_submodule', 'get_submodule_repo_for(spf->r, task->sub)' fails to get a repo pointer for the submodule repository, since it is not initialized. That
is normal. Then we go in the "else" branch, and hit this code:/*
* An empty directory is normal,
* the submodule is not initialized
*/
if (S_ISGITLINK(ce->ce_mode)&&
!is_empty_dir(ce->name)) {'
is_empty_dir' receivesce->name, but the current working directory is the Git directory of 'middle', so clearlyis_empty_dirreturns false, as/path/to/git/t/trash directory.t5526-fetch-submodules/B/.git/modules/middle/inner
is a non-existent path. The path that we should send is the worktree of inner, ie.
the concatenation ofspf->r->worktreeandce->name. This would give
Quoted 5 lines
/path/to/git/t/trash directory.t5526-fetch-submodules/B/middle/inner,
which is an empty directory since the inner submodule is not initialized, and we would not get the "Could not access submodule inner" error that you wanted to solve.
On quick glance this sounds plausible, but to fully understand it I need to put some effort in reading this code again. I hope to do so tomorrow. We can then compile a new set of patches including this real fix and Ralf's and my test case.
Thanks for digging into it.
Signature · 2 lines
kind regards --peter;
Quoted 6 lines · Peter Kästle
On quick glance this sounds plausible, but to fully understand it I need to put some effort in reading this code again. I hope to do so tomorrow. We can then compile a new set of patches including this real fix and Ralf's and my test case.
Thanks for digging into it.
Yeah, thanks, both for noticing problem so quickly and started digging to find the real solution.
By the way, as to the "if this were originally two patches, we could have saved the test that expects failure" you raised, I do not think it is a good idea to optimize for cases where changes turn out problematic and have to get reverted.
PATCH 1/1 submodules: fix of regression on fetching of non-init subsub-repo
In reply to Peter KästleA regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).
The scenario in which it triggers is when one has a remote repository with a subrepository inside a subrepository like this:
superproject/middle_repo/inner_repo
Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.
Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.
Once person A pushed the changes and person B wants to fetch them using "git fetch" on superproject level, B's git call will return with error saying:
Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo
Expectation is that in this case the inner submodule will be recognized as uninitialized subrepository and skipped by the git fetch command.
This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.
Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.
This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.
--- submodule.c | 7 ++- t/t5526-fetch-submodules.sh | 94 +++++++++++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+), 1 deletion(-)
submodule.c +6 -1
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&submodule_prefix);return 1 ; }else { +struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);free (task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,* An empty directory is normal, * the submodule is not initialized */ +strbuf_addf (&empty_submodule_path,"%s/%s/" , + spf->r->worktree, + ce->name);if (S_ISGITLINK (ce->ce_mode) && - !is_empty_dir (ce->name)) { + !is_empty_dir (empty_submodule_path.buf)) { spf->result =1 ;strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), ce->name); } +strbuf_release (&empty_submodule_path); } }
t/t5526-fetch-submodules.sh +94 -0
98 lines not on this page · raw message
Signature · 1 line
2.29.2
Re: [PATCH] submodules: fix of regression on fetching of non-init subsub-repo
Quoted 4 lines · Peter Kaestle
[...]
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.
Just a few small comments (nothing comprehensive) from a quick scan of the patch...
Mostly they are just minor style issues, not necessarily worth a re-roll, but there is one actionable item.
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 722
Quoted 8 lines
Signed-off-by Peter Kaestle <[email protected]>---
t/t5526-fetch-submodules.sh +4
-0@@ -719,4 +719,98 @@ test_expect_success 'fetch new submodule commit intermittently referenced by sup +add_commit_push () { + dir="$1" + msg="$2" + shift 2
We typically recommend including these assignments in the &&-chain to future-proof against someone later inserting code above them and not realizing that that code is not part of the &&-chain, in which case if the new code fails, the failure might go unnoticed.
Quoted 6 lines
t/t5526-fetch-submodules.sh +6
-0+ git -C "$dir" add "$@" && + git -C "$dir" commit -a -m "$msg" && + git -C "$dir" push +} + +compare_refs_in_dir () {
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 741
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ fail= && + if test "x$1" = 'x!' + then + fail='!' && + shift + fi && + git -C "$1" rev-parse --verify "$2" >expect && + git -C "$3" rev-parse --verify "$4" >actual && + eval $fail test_cmp expect actual +}
We have a test_cmp_rev() similar to this but it doesn't support -C as some of our other test functions do. I briefly wondered if it would make sense to extend it to understand -C, but even that wouldn't help this case since compare_refs_in_dir() introduced here involves two distinct directories. The need here is so special-purpose that it likely would not make sense to upgrade test_cmp_rev() to accommodate it. Okay.
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 755
Quoted 12 lines
t/t5526-fetch-submodules.sh +12
-0+test_expect_success 'setup nested submodule fetch test' ' + # does not depend on any previous test setups + + for repo in outer middle inner + do + ( + git init --bare $repo && + git clone $repo ${repo}_content && + echo "$repo" >"${repo}_content/file" && + add_commit_push ${repo}_content "initial" file + ) || return 1 + done &&
What is the purpose of the subshell here? Is it to ensure that commits in each repo have identical timestamps? Or is it just for making the && and || expression more clear? If the latter, we normally don't bother with the parentheses.
Quoted 4 lines
t/t5526-fetch-submodules.sh +4
-0+ git clone outer A && + git -C A submodule add "$pwd/middle" && + git -C A/middle/ submodule add "$pwd/inner" && + add_commit_push A/middle/ "adding inner sub" .gitmodules inner &&
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 770
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ add_commit_push A/ "adding middle sub" .gitmodules middle && + + git clone outer B && + git -C B/ submodule update --init middle && + + compare_refs_in_dir A HEAD B HEAD && + compare_refs_in_dir A/middle HEAD B/middle HEAD && + test -f B/file && + test -f B/middle/file && + ! test -f B/middle/inner/file &&
These days we typically use test_path_exists() (or test_path_is_file()) and test_path_is_missing() rather than bare test.
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 792
Quoted 7 lines
t/t5526-fetch-submodules.sh +7
-0+test_expect_success 'setup recursive fetch with uninit submodule' ' + # does not depend on any previous test setups + + git init main && + git init sub && + + touch sub/file &&
Unless the timestamp of the file is significant to the test, in which case touch is used, we normally create empty files like this:
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 810
Quoted 6 lines
sub/file &&
t/t5526-fetch-submodules.sh +4
-0+test_expect_success 'recursive fetch with uninit submodule' ' + git -C main submodule deinit -f sub && + ! git -C main fetch --recurse-submodules |& + grep -v -m1 "Fetching submodule sub$" &&
We want the test scripts to be portable, thus avoid Bashisms such as |&.
We also avoid placing a Git command upstream in a pipe since doing so causes the exit code of the Git command to be lost. Instead, we would normally send the Git output to a file and then send that file to whatever would be downstream of the Git command in the pipe. So, a mechanical rewrite of the above (without thinking too hard about it) might be:
git -C main fetch --recurse-submodules >out 2>&1 &&
! grep -v -m1 "Fetching submodule sub$" &&t/t5526-fetch-submodules.sh
Quoted 2 lines
t/t5526-fetch-submodules.sh +2
-0+ git -C main submodule status | + sed -e "s/^-//" -e "s/ sub$//" >actual &&
Same comment about avoiding Git upstream in a pipe, so perhaps:
git -C main submodule status >out &&
sed -e "s/^-//" -e "s/ sub$//" out >actual &&Quoted 2 lines
+
test_cmpexpect actual
+'
Re: [PATCH] submodules: fix of regression on fetching of non-init subsub-repo
Hi Eric,
Quoted 11 lines · Eric Sunshine
Quoted 4 lines · Peter Kaestle
[...]
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.Just a few small comments (nothing comprehensive) from a quick scan of the patch...
Mostly they are just minor style issues, not necessarily worth a re-roll, but there is one actionable item.
thanks for your comments. A new patch will follow soon.
Quoted 5 lines
Quoted 5 lines
Signed-off-by Peter Kaestle <[email protected]>---
t/t5526-fetch-submodules.sh +1
-0@@ -719,4 +719,98 @@ test_expect_success 'fetch new submodule commit intermittently referenced by sup +add_commit_push () {
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 725
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ dir="$1" + msg="$2" + shift 2We typically recommend including these assignments in the &&-chain to future-proof against someone later inserting code above them and not realizing that that code is not part of the &&-chain, in which case if the new code fails, the failure might go unnoticed.
ok
Quoted 18 lines
Quoted 7 lines
t/t5526-fetch-submodules.sh +6
-0+ git -C "$dir" add "$@" && + git -C "$dir" commit -a -m "$msg" && + git -C "$dir" push +} + +compare_refs_in_dir () {t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 741
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ fail= && + if test "x$1" = 'x!' + then + fail='!' && + shift + fi && + git -C "$1" rev-parse --verify "$2" >expect && + git -C "$3" rev-parse --verify "$4" >actual && + eval $fail test_cmp expect actual +}
Quoted 7 lines
We have a
test_cmp_rev()similar to this but it doesn't support -C as some of our other test functions do. I briefly wondered if it would make sense to extend it to understand -C, but even that wouldn't help this case sincecompare_refs_in_dir()introduced here involves two distinct directories. The need here is so special-purpose that it likely would not make sense to upgradetest_cmp_rev()to accommodate it. Okay.
Yes, I saw that there's a similar function and I tried to modify this one first. Unfortunately this didn't work without touching much unaffected test code. So I propose to continue with this additional function.
Quoted 9 lines
Quoted 9 lines
t/t5526-fetch-submodules.sh +9
-0+test_expect_success 'setup nested submodule fetch test' ' + # does not depend on any previous test setups + + for repo in outer middle inner + do + ( + git init --bare $repo && + git clone $repo ${repo}_content && + echo "$repo" >"${repo}_content/file" &&
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 755
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ add_commit_push ${repo}_content "initial" file + ) || return 1 + done &&What is the purpose of the subshell here? Is it to ensure that commits in each repo have identical timestamps? Or is it just for making the && and || expression more clear? If the latter, we normally don't bother with the parentheses.
It was intended to make the correlation of && and || clear. I have experienced many cases in the past where things were screwed because it was not clearly understood by everybody. I'll propose next patch without this subshell.
Quoted 10 lines
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ git clone outer A && + git -C A submodule add "$pwd/middle" && + git -C A/middle/ submodule add "$pwd/inner" && + add_commit_push A/middle/ "adding inner sub" .gitmodules inner && + add_commit_push A/ "adding middle sub" .gitmodules middle && + + git clone outer B && + git -C B/ submodule update --init middle && + + compare_refs_in_dir A HEAD B HEAD &&
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 770
Quoted 4 lines
t/t5526-fetch-submodules.sh +4
-0+ compare_refs_in_dir A/middle HEAD B/middle HEAD && + test -f B/file && + test -f B/middle/file && + ! test -f B/middle/inner/file &&These days we typically use
test_path_exists()(ortest_path_is_file()) andtest_path_is_missing()rather than baretest.
ok.
Quoted 11 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 792
Quoted 7 lines
t/t5526-fetch-submodules.sh +7
-0+test_expect_success 'setup recursive fetch with uninit submodule' ' + # does not depend on any previous test setups + + git init main && + git init sub && + + touch sub/file &&Unless the timestamp of the file is significant to the test, in which case
touchis used, we normally create empty files like this:
Quoted 1 line
>sub/file &&
ok.
Quoted 13 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 810
Quoted 5 lines
t/t5526-fetch-submodules.sh +4
-0+test_expect_success 'recursive fetch with uninit submodule' ' + git -C main submodule deinit -f sub && + ! git -C main fetch --recurse-submodules |& + grep -v -m1 "Fetching submodule sub$" &&We want the test scripts to be portable, thus avoid Bashisms such as
|&. > We also avoid placing a Git command upstream in a pipe since doing so causes the exit code of the Git command to be lost. Instead, we would
normally send the Git output to a file and then send that file to
whatever would be downstream of the Git command in the pipe. So, a
mechanical rewrite of the above (without thinking too hard about it)
might be:
Quoted 2 lines
git -C main fetch --recurse-submodules >out 2>&1 &&
! grep -v -m1 "Fetching submodule sub$" &&In general I agree, but for this special test case, it's required to have the two commands connected by a pipe, as the grep needs to kill the git call in error case. Otherwise for this regression git would go for an infinite recursion loop.
Of course, we can go for a "git 2>&1 | grep" solution.
Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 777
Quoted 2 lines
t/t5526-fetch-submodules.sh +2
-0+ git -C main submodule status | + sed -e "s/^-//" -e "s/ sub$//" >actual &&Same comment about avoiding Git upstream in a pipe, so perhaps:
git -C main submodule status >out && sed -e "s/^-//" -e "s/ sub$//" out >actual &&Quoted 2 lines
t/t5526-fetch-submodules.sh +2
-0+ test_cmp expect actual +'
ok.
Signature · 2 lines
kind regards --peter;
Re: [PATCH] submodules: fix of regression on fetching of non-init subsub-repo
Quoted 9 lines · Peter Kästle
Quoted 7 lines · Eric Sunshine
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 810
Quoted 2 lines · Peter Kaestle
t/t5526-fetch-submodules.sh +2
-0+ ! git -C main fetch --recurse-submodules |& + grep -v -m1 "Fetching submodule sub$" &&We want the test scripts to be portable, thus avoid Bashisms such as
|&.git -C main fetch --recurse-submodules >out 2>&1 && ! grep -v -m1 "Fetching submodule sub$" &&
Quoted 6 lines
In general I agree, but for this special test case, it's required to have the two commands connected by a pipe, as the grep needs to kill the git call in error case. Otherwise for this regression git would go for an infinite recursion loop.
Of course, we can go for a "git 2>&1 | grep" solution.
In that case, an in-code comment explaining why the output of git must be piped to grep would be helpful since it is not uncommon for people to "modernize" test code as they are working in the vicinity, and removing such pipes is one of the modernizations often made. A comment would help avoid such a change.
PATCH v2 1/1 submodules: fix of regression on fetching of non-init subsub-repo
A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).
The scenario in which it triggers is when one has a remote repository with a subrepository inside a subrepository like this:
superproject/middle_repo/inner_repo
Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.
Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.
Once person A pushed the changes and person B wants to fetch them using "git fetch" on superproject level, B's git call will return with error saying:
Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo
Expectation is that in this case the inner submodule will be recognized as uninitialized subrepository and skipped by the git fetch command.
This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.
Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.
This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.
--- submodule.c | 7 ++- t/t5526-fetch-submodules.sh | 104 ++++++++++++++++++++++++++++++++++++ 2 files changed, 110 insertions(+), 1 deletion(-)
submodule.c +6 -1
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&submodule_prefix);return 1 ; }else { +struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);free (task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,* An empty directory is normal, * the submodule is not initialized */ +strbuf_addf (&empty_submodule_path,"%s/%s/" , + spf->r->worktree, + ce->name);if (S_ISGITLINK (ce->ce_mode) && - !is_empty_dir (ce->name)) { + !is_empty_dir (empty_submodule_path.buf)) { spf->result =1 ;strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), ce->name); } +strbuf_release (&empty_submodule_path); } }
t/t5526-fetch-submodules.sh +104 -0
108 lines not on this page · raw message
Signature · 1 line
2.29.2
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
Hi Peter,
Quoted 8 lines · Peter Kaestle
A regression has been introduced by a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).The scenario in which it triggers is when one has a remote repository with a subrepository inside a subrepository like this:
superproject/middle_repo/inner_repo
The correct terminology is "submodule", not "subrepository".
Also, (minor point) I would just write "when one has a repository", as its simpler (the repository by itself is not "remote", it is only "remote" in relation the repositories that are cloned from it).
Quoted 9 lines
Person A and B have both a clone of it, while Person B is not working with the
inner_repoand thus does not have it initialized in his working copy.Now person A introduces a change to the
inner_repoand propagates it through themiddle_repoand the superproject.Once person A pushed the changes and person B wants to fetch them using "git fetch" on superproject level,
s/on/at the/
Quoted 9 lines
B's git call will return with error
saying:Could not access submodule '
inner_repo'
Errors during submodule fetch:
middle_repoExpectation is that in this case the inner submodule will be recognized as uninitialized subrepository and skipped by the git fetch command.
here again, terminology: "as an uninitialized submodule"
Quoted 7 lines
This used to work correctly before 'a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.Starting with a62387b the code wants to evaluate "
is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.
Quoted 8 lines
This patch ensures
is_empty_dir()is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories.
This
issue was leading to an infinite loop when doing a revert of a62387b.
I would maybe add more details here, something like the following (we can cite your previous attempt, because it was merged to 'master'):
The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in
an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.
Quoted 31 lines
Signed-off-by Peter Kaestle <[email protected]>CC Junio C Hamano <[email protected]>CC Philippe Blain <[email protected]>CC Ralf Thielow <[email protected]>CC Eric Sunshine <[email protected]>---
submodule.c| 7 ++-t/t5526-fetch-submodules.sh| 104 ++++++++++++++++++++++++++++++++++++2 files changed, 110 insertions(+), 1 deletion(-)submodule.c +4
-0@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&submodule_prefix);return 1 ; }else { +struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);free (task);@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,* An empty directory is normal, * the submodule is not initialized */ +strbuf_addf (&empty_submodule_path,"%s/%s/" , + spf->r->worktree, + ce->name); if (S_ISGITLINK(ce->ce_mode) &&
submodule.c static int get_next_submodule(struct child_process *cp, · line 1501
Quoted 10 lines
submodule.c +2
-1- !is_empty_dir(ce->name)) { + !is_empty_dir(empty_submodule_path.buf)) { spf->result =1 ;strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), ce->name); } +strbuf_release (&empty_submodule_path); } }
Maybe a personal preference, but I would have gone for something a little simpler, like the following:
submodule.c +1 -1
@@ -1486,7 +1486,7 @@ static int get_next_submodule(struct child_process *cp,* the submodule is not initialized */ if (S_ISGITLINK (ce->ce_mode) && - !is_empty_dir (ce->name)) { + !is_empty_dir (repo_worktree_path (spf->r,"%s" , ce->name))) { spf->result =1 ; strbuf_addf(err, _("Could not access submodule '%s'\n"),
Quoted 75 lines
t/t5526-fetch-submodules.sh +0
-0@@ -719,4 +719,108 @@ test_expect_success 'fetch new submodule commit intermittently referenced by sup )'
t/t5526-fetch-submodules.sh +67
-0+add_commit_push () { + dir="$1" && + msg="$2" && + shift 2 && + git -C "$dir" add "$@" && + git -C "$dir" commit -a -m "$msg" && + git -C "$dir" push +} + +compare_refs_in_dir () { + fail= && + if test "x$1" = 'x!' + then + fail='!' && + shift + fi && + git -C "$1" rev-parse --verify "$2" >expect && + git -C "$3" rev-parse --verify "$4" >actual && + eval $fail test_cmp expect actual +} + + +test_expect_success 'setup nested submodule fetch test' ' + # does not depend on any previous test setups + + for repo in outer middle inner + do + git init --bare $repo && + git clone $repo ${repo}_content && + echo "$repo" >"${repo}_content/file" && + add_commit_push ${repo}_content "initial" file || + return 1 + done && + + git clone outer A && + git -C A submodule add "$pwd/middle" && + git -C A/middle/ submodule add "$pwd/inner" && + add_commit_push A/middle/ "adding inner sub" .gitmodules inner && + add_commit_push A/ "adding middle sub" .gitmodules middle && + + git clone outer B && + git -C B/ submodule update --init middle && + + compare_refs_in_dir A HEAD B HEAD && + compare_refs_in_dir A/middle HEAD B/middle HEAD && + test_path_is_file B/file && + test_path_is_file B/middle/file && + test_path_is_missing B/middle/inner/file && + + echo "change on inner repo of A" >"A/middle/inner/file" && + add_commit_push A/middle/inner "change on inner" file && + add_commit_push A/middle "change on inner" inner && + add_commit_push A "change on inner" middle +' + +test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' ' + # depends on previous test for setup + + git -C B/ fetch && + compare_refs_in_dir A origin/master B origin/master +' + + +test_expect_success 'setup recursive fetch with uninit submodule' ' + # does not depend on any previous test setups + + git init main &&
t/t5526-fetch-submodules.sh
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ git init sub && + + >sub/file && + git -C sub add file && + git -C sub commit -m "add file" && + git -C sub rev-parse HEAD >expect && + + git -C main submodule add ../sub && + git -C main submodule init && + git -C main submodule update --checkout &&
These two steps are unnecessary as they are implicitly done by 'git submodule add'. I think we could reflect real life a little bit more by cloning the superproject, and running the 'recursive fetch with uninit submodule' test below in the clone.
t/t5526-fetch-submodules.sh
Quoted 9 lines
t/t5526-fetch-submodules.sh +9
-0+ git -C main submodule status >out && + sed -e "s/^ //" -e "s/ sub .*$//" out >actual && + test_cmp expect actual +' + +test_expect_success 'recursive fetch with uninit submodule' ' + # depends on previous test for setup + + git -C main submodule deinit -f sub &&
Here you are deiniting the submodule, such that
the Git directory will stay in .git/modules/sub. This is not the same thing as a submodule that was never initialized ("uninitialized"), for which .git/modules/sub will not yet exist. So maybe we could harden the tests by also testing for that scenario ? I don't know... maybe the infinite loop only happens if .git/modules/sub actually already exists. If so, the test name should be "recursive fetch with deinitialized submodule", I think.
Quoted 18 lines
t/t5526-fetch-submodules.sh +18
-0+ + # In a regression the following git call will run into infinite recursion. + # To handle that, we connect the grep command to the git call by a pipe + # so that grep can kill the infinite recusion when detected. + # The recursion creates git output like: + # Fetching submodule sub + # Fetching submodule sub/sub <-- [1] + # Fetching submodule sub/sub/sub + # ... + # [1] grep will trigger here and kill git by exiting and closing its stdin + + ! git -C main fetch --recurse-submodules 2>&1 | + grep -v -m1 "Fetching submodule sub$" && + git -C main submodule status >out && + sed -e "s/^-//" -e "s/ sub$//" out >actual && + test_cmp expect actual +' +
Quoted 1 line
test_done
Thanks for working on that, and sorry for not having the time to comment before you sent v2.
Cheers,
Philippe.
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
submodule.c static int get_next_submodule(struct child_process *cp, · line 1489
Quoted 13 lines · Philippe Blain
Maybe a personal preference, but I would have gone for something a little simpler, like the following:
submodule.c +1
-1@@ -1486,7 +1486,7 @@ static int get_next_submodule(struct child_process *cp,* the submodule is not initialized */ if (S_ISGITLINK (ce->ce_mode) && - !is_empty_dir (ce->name)) { + !is_empty_dir (repo_worktree_path (spf->r,"%s" , ce->name))) {
But then you leak the return value from repo_worktree_path(), no?
Quoted 5 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+test_expect_success 'recursive fetch with uninit submodule' ' + # depends on previous test for setup + + git -C main submodule deinit -f sub &&
Quoted 7 lines
Here you are deiniting the submodule, such that
the Git directory will stay in .git/modules/sub. This is not the same thing as a submodule that was never initialized ("uninitialized"), for which .git/modules/sub will not yet exist. So maybe we could harden the tests by also testing for that scenario ? I don't know... maybe the infinite loop only happens if .git/modules/sub actually already exists. If so, the test name should be "recursive fetch with deinitialized submodule", I think.
Even if the original breakage happens only for deinitialized case, it would be sensible to test uninitialized one as well, I would think.
Quoted 2 lines
Thanks for working on that, and sorry for not having the time to comment before you sent v2.
Thanks, all.
It's not like we corral everybody to a single place on a single day to work on a single thing. Reviews ov v2 by reviewers who have not seen v1 is totally expected and very much appreciated.
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
Quoted 10 lines · Junio C Hamano
Quoted 2 lines · Philippe Blain
Thanks for working on that, and sorry for not having the time to comment before you sent v2.
Thanks, all.
It's not like we corral everybody to a single place on a single day to work on a single thing. Reviews ov v2 by reviewers who have not seen v1 is totally expected and very much appreciated.
Thank you very much for all your comments, they're much appreciated. I prefer spending more time on it now for getting it right, than to take another revert-rethink-refactor round. It will take me some time today to work on them.
Signature · 2 lines
kind regards --peter;
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
In reply to Philippe BlainQuoted 12 lines · Philippe Blain
I would maybe add more details here, something like the following (we can cite your previous attempt, because it was merged to 'master'):
The first attempt to fix this regression, in
1b7ac4e6d4(submodules: fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in7091499bc0(Revert "submodules:
fix of regression on fetching of non-init subsub-repo", 2020-12-02). To prevent future breakages, also add a regression test for this scenario.
Forgot to mention in my other response, but I do find this a very sensible addition.
Thanks.
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
In reply to Philippe BlainQuoted 11 lines · Philippe Blain
Hi Peter,
Quoted 8 lines · Peter Kaestle
A regression has been introduced by a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).The scenario in which it triggers is when one has a remote repository with a subrepository inside a subrepository like this:
superproject/middle_repo/inner_repo
Quoted 5 lines
The correct terminology is "submodule", not "subrepository".
Also, (minor point) I would just write "when one has a repository", as its simpler (the repository by itself is not "remote", it is only "remote" in relation the repositories that are cloned from it).
ok.
Quoted 11 lines
Quoted 9 lines
Person A and B have both a clone of it, while Person B is not working with the
inner_repoand thus does not have it initialized in his working copy.Now person A introduces a change to the
inner_repoand propagates it through themiddle_repoand the superproject.Once person A pushed the changes and person B wants to fetch them using "git fetch" on superproject level,
s/on/at the/
ok.
Quoted 11 lines
Quoted 9 lines
B's git call will return with error
saying:Could not access submodule '
inner_repo'
Errors during submodule fetch:
middle_repoExpectation is that in this case the inner submodule will be recognized as uninitialized subrepository and skipped by the git fetch command.
here again, terminology: "as an uninitialized submodule"
ok.
Quoted 19 lines
Quoted 7 lines
This used to work correctly before 'a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.Starting with a62387b the code wants to evaluate "
is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.Quoted 8 lines
This patch ensures
is_empty_dir()is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories.
This
issue was leading to an infinite loop when doing a revert of a62387b.I would maybe add more details here, something like the following (we can cite your previous attempt, because it was merged to 'master'):
Quoted 7 lines
The first attempt to fix this regression, in
1b7ac4e6d4(submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in
an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in7091499bc0(Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.
Jip, I like that.
Quoted 51 lines
Quoted 31 lines
Signed-off-by Peter Kaestle <[email protected]>CC Junio C Hamano <[email protected]>CC Philippe Blain <[email protected]>CC Ralf Thielow <[email protected]>CC Eric Sunshine <[email protected]>---
submodule.c| 7 ++-t/t5526-fetch-submodules.sh| 104 ++++++++++++++++++++++++++++++++++++2 files changed, 110 insertions(+), 1 deletion(-)submodule.c +4
-0@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&submodule_prefix);return 1 ; }else { +struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);free (task);@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,* An empty directory is normal, * the submodule is not initialized */ +strbuf_addf (&empty_submodule_path,"%s/%s/" , + spf->r->worktree, + ce->name); if (S_ISGITLINK(ce->ce_mode) &&submodule.c static int get_next_submodule(struct child_process *cp, · line 1501
Quoted 10 lines
submodule.c +2
-1- !is_empty_dir(ce->name)) { + !is_empty_dir(empty_submodule_path.buf)) { spf->result =1 ;strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), ce->name); } +strbuf_release (&empty_submodule_path); } }Maybe a personal preference, but I would have gone for something a little simpler, like the following:
submodule.c +0
-0
submodule.c static int get_next_submodule(struct child_process *cp, · line 1489
Quoted 5 lines
submodule.c +1
-1@@ -1486,7 +1486,7 @@ static int get_next_submodule(struct child_process *cp,* the submodule is not initialized */ if (S_ISGITLINK (ce->ce_mode) && - !is_empty_dir (ce->name)) { + !is_empty_dir (repo_worktree_path (spf->r,"%s" , ce->name))) {
I'm not deep enough into the git code to judge which approach is the better one. From my perspective, being a foreigner to the git code, I like my proposed code more, as for me it's much easier to understand what's happening by having a meaningful variable name and without being forced to dig into outer functions first. Also Junio C Hamano <[email protected]> is having some concerns, which I can't judge:
Quoted 1 line
But then you leak the return value from
repo_worktree_path(), no?
Thus for v3 I'll stick to my proposal and when you'll review it, please discuss with each other whether I should go for a v4 using repo_worktree_path().
[...]
Quoted 11 lines
Quoted 11 lines
submodule.c +11
-0+ +test_expect_success 'setup recursive fetch with uninit submodule' ' +# does not depend on any previous test setups + + git init main && + git init sub && + + >sub/file && + git -C sub add file && + git -C sub commit -m "add file" && + git -C sub rev-parse HEAD >expect &&
Quoted 8 lines
submodule.c
Quoted 4 lines
submodule.c +4
-0+ + git -C main submodule add ../sub && + git -C main submodule init && + git -C main submodule update --checkout &&These two steps are unnecessary as they are implicitly done by 'git submodule add'. I think we could reflect real life a little bit more by cloning the superproject, and running the 'recursive fetch with uninit submodule' test below in the clone.
Yes, you're right, "...init" and "...update..." can be removed.
Quoted 10 lines
submodule.c
Quoted 9 lines
submodule.c +9
-0+ git -C main submodule status >out && + sed -e"s/^ //" -e"s/ sub .*$//" out >actual && + test_cmp expect actual +' + +test_expect_success 'recursive fetch with uninit submodule' ' +# depends on previous test for setup + + git -C main submodule deinit -f sub &&
Quoted 7 lines
Here you are deiniting the submodule, such that
the Git directory will stay in .git/modules/sub. This is not the same thing as a submodule that was never initialized ("uninitialized"), for which .git/modules/sub will not yet exist. So maybe we could harden the tests by also testing for that scenario ? I don't know... maybe the infinite loop only happens if .git/modules/sub actually already exists. If so, the test name should be "recursive fetch with deinitialized submodule", I think.
I added another test case for v3, which checks for this in case of never initialized submodule. When executing the test, I can see that the infinite loop regression only occurs after doing the init followed by a deinit. Thus renaming the test accordingly.
Signature · 2 lines
best regards --peter;
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
In reply to Peter KaestleQuoted 18 lines · Peter Kaestle
t/t5526-fetch-submodules.sh +18
-0+add_commit_push () { + dir="$1" && + msg="$2" && + shift 2 && + git -C "$dir" add "$@" && + git -C "$dir" commit -a -m "$msg" && + git -C "$dir" push +} + +compare_refs_in_dir () { + fail= && + if test "x$1" = 'x!' + then + fail='!' && + shift + fi && + git -C "$1" rev-parse --verify "$2" >expect && + git -C "$3" rev-parse --verify "$4" >actual &&
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781
Quoted 10 lines
t/t5526-fetch-submodules.sh +7
-0+ eval $fail test_cmp expect actual +} +test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' ' + # depends on previous test for setup + + git -C B/ fetch && + compare_refs_in_dir A origin/master B origin/master
Can we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 789
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+test_expect_success 'setup recursive fetch with uninit submodule' ' + # does not depend on any previous test setups + + git init main && + git init sub &&
"super vs sub" would give us a better contrast than "main vs sub", and it would help reduce mistakes in the mechanical conversion of "master" to "main" happening in another topic.
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 817
Quoted 9 lines
t/t5526-fetch-submodules.sh +9
-0+ # In a regression the following git call will run into infinite recursion. + # To handle that, we connect the grep command to the git call by a pipe + # so that grep can kill the infinite recusion when detected. + # The recursion creates git output like: + # Fetching submodule sub + # Fetching submodule sub/sub <-- [1] + # Fetching submodule sub/sub/sub + # ... + # [1] grep will trigger here and kill git by exiting and closing its stdin
"trigger here and kill..." -> "stop reading and cause git to eventually stop and die"
But we probably cannot use 'grep -m1' so it is a moot point.
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 820
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ + ! git -C main fetch --recurse-submodules 2>&1 | + grep -v -m1 "Fetching submodule sub$" &&
Unfortunately, "grep -m<count>" is not even in POSIX, I would think.
What do we expect to happen in the correct case?
- A line "Fetching submodule sub" and nothing else is given? That feels a bit brittle (how are we making sure, in the presence of "2>&1", that we will not get any other output, like progress?)
- "sub" is the only thing that appears on lines that begin with "Fetching submodule" (i.e. "Fetching submodule $something" where $something is not 'sub' is an error), and we allow other garbage in the output? That would be a bit more robust than the above.
As you seem to be comfortable using "sed" below, perhaps use it to extract the first few lines that say "^Fetching submodule " from the output and stop, and check that the output has only one such line about 'sub' and nothing else?
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ git -C main submodule status >out && + sed -e "s/^-//" -e "s/ sub$//" out >actual && + test_cmp expect actual
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
Hi Junio,
Quoted 27 lines · Junio C Hamano
Quoted 20 lines · Peter Kaestle
t/t5526-fetch-submodules.sh +20
-0+add_commit_push () { + dir="$1" && + msg="$2" && + shift 2 && + git -C "$dir" add "$@" && + git -C "$dir" commit -a -m "$msg" && + git -C "$dir" push +} + +compare_refs_in_dir () { + fail= && + if test "x$1" = 'x!' + then + fail='!' && + shift + fi && + git -C "$1" rev-parse --verify "$2" >expect && + git -C "$3" rev-parse --verify "$4" >actual && + eval $fail test_cmp expect actual +}
Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' ' + # depends on previous test for setup + + git -C B/ fetch && + compare_refs_in_dir A origin/master B origin/masterCan we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?
Or, simpler, we could call "git remote set-head -a' in A and B in the setup script, which would make origin/HEAD in A and B point to the default branch, such that the call here could be :
compare_refs_in_dir A origin/HEAD B origin/HEAD
Philippe.
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
Quoted 11 lines · Philippe Blain
Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' ' + # depends on previous test for setup + + git -C B/ fetch && + compare_refs_in_dir A origin/master B origin/masterCan we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?
Quoted 4 lines
Or, simpler, we could call "git remote set-head -a' in A and B in the setup script, which would make origin/HEAD in A and B point to the default branch, such that the call here could be :
The set-up prepares A and B by cloning from elsewhere, no? Should we even need a set-head call?
Quoted 1 line
compare_refs_in_dirA origin/HEAD B origin/HEAD
Yes, using HEAD would be another simple way to avoid having to rely on the default behaviour.
THanks.
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Philippe Blain
Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' ' + # depends on previous test for setup + + git -C B/ fetch && + compare_refs_in_dir A origin/master B origin/masterCan we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?
Quoted 7 lines
Quoted 4 lines
Or, simpler, we could call "git remote set-head -a' in A and B in the setup script, which would make origin/HEAD in A and B point to the default branch, such that the call here could be :
The set-up prepares A and B by cloning from elsewhere, no? Should we even need a set-head call?
Ah, they are created by cloning an empty repository. That explains why. Thanks.
Quoted 6 lines
Quoted 1 line
compare_refs_in_dirA origin/HEAD B origin/HEADYes, using HEAD would be another simple way to avoid having to rely on the default behaviour.
THanks.
Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo
In reply to Junio C HamanoQuoted 25 lines · Junio C Hamano
Quoted 20 lines · Peter Kaestle
t/t5526-fetch-submodules.sh +20
-0+add_commit_push () { + dir="$1" && + msg="$2" && + shift 2 && + git -C "$dir" add "$@" && + git -C "$dir" commit -a -m "$msg" && + git -C "$dir" push +} + +compare_refs_in_dir () { + fail= && + if test "x$1" = 'x!' + then + fail='!' && + shift + fi && + git -C "$1" rev-parse --verify "$2" >expect && + git -C "$3" rev-parse --verify "$4" >actual && + eval $fail test_cmp expect actual +}
Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' ' + # depends on previous test for setup + + git -C B/ fetch && + compare_refs_in_dir A origin/master B origin/masterCan we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?
Using origin/HEAD for compare_refs_in_dir should be fine without additional setup, as for the regression the "git -C B/ fetch" will fail and return with false (see description of the patch). This compare_refs_in_dir is just for additional checking as you proposed in the mail:
https://public-inbox.org/git/[email protected]/
------------8<-------------
Quoted 7 lines
And from B that was an original copy of A with only the top and middle layer instantiated, you run "git fetch". Are you happy as long as "git fetch" does not exit with non-zero status? That is hard to believe---it may be a necessary condition for the command to exit with zero status, but you have other expectations, like what commit the remote tracking branch refs/remotes/origin/HEAD ought to be pointing at. I think we should check that, too.
----------->8-------------
Quoted 11 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 789
Quoted 6 lines
t/t5526-fetch-submodules.sh +5
-0+test_expect_success 'setup recursive fetch with uninit submodule' ' + # does not depend on any previous test setups + + git init main && + git init sub &&"super vs sub" would give us a better contrast than "main vs sub", and it would help reduce mistakes in the mechanical conversion of "master" to "main" happening in another topic.
ok.
Quoted 15 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 817
Quoted 9 lines
t/t5526-fetch-submodules.sh +9
-0+ # In a regression the following git call will run into infinite recursion. + # To handle that, we connect the grep command to the git call by a pipe + # so that grep can kill the infinite recusion when detected. + # The recursion creates git output like: + # Fetching submodule sub + # Fetching submodule sub/sub <-- [1] + # Fetching submodule sub/sub/sub + # ... + # [1] grep will trigger here and kill git by exiting and closing its stdin"trigger here and kill..." -> "stop reading and cause git to eventually stop and die"
But we probably cannot use 'grep -m1' so it is a moot point.
Quoted 7 lines
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 820
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ + ! git -C main fetch --recurse-submodules 2>&1 | + grep -v -m1 "Fetching submodule sub$" &&Unfortunately, "grep -m<count>" is not even in POSIX, I would think.
What do we expect to happen in the correct case?
sigh, we can't use grep -m1. Too bad, it was such a nice solution.
Quoted 10 lines
- A line "Fetching submodule sub" and nothing else is given? That feels a bit brittle (how are we making sure, in the presence of "2>&1", that we will not get any other output, like progress?)
- "sub" is the only thing that appears on lines that begin with "Fetching submodule" (i.e. "Fetching submodule $something" where $something is not 'sub' is an error), and we allow other garbage in the output? That would be a bit more robust than the above.
Quoted 4 lines
As you seem to be comfortable using "sed" below, perhaps use it to extract the first few lines that say "^Fetching submodule " from the output and stop, and check that the output has only one such line about 'sub' and nothing else?
According to [1] posix sed offers equal possibility to quit like grep -m1 and I'll adopt:
$> yes posixgrepisnogoodforus | sed "/posix/q"
[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html
Looking at the other mails I think I processed over all open comments and will prepare for v3. Thanks again.
Signature · 2 lines
kind regards --peter;
PATCH v3 1/1 submodules: fix of regression on fetching of non-init subsub-repo
A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).
The scenario in which it triggers is when one has a repository with a submodule inside a submodule like this:
superproject/middle_repo/inner_repo
Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.
Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.
Once person A pushed the changes and person B wants to fetch them using "git fetch" at the superproject level, B's git call will return with error saying:
Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo
Expectation is that in this case the inner submodule will be recognized as uninitialized submodule and skipped by the git fetch command.
This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.
Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.
This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.
The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.
--- submodule.c | 7 ++- t/t5526-fetch-submodules.sh | 124 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+), 1 deletion(-)
submodule.c +6 -1
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&submodule_prefix);return 1 ; }else { +struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);free (task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,* An empty directory is normal, * the submodule is not initialized */ +strbuf_addf (&empty_submodule_path,"%s/%s/" , + spf->r->worktree, + ce->name);if (S_ISGITLINK (ce->ce_mode) && - !is_empty_dir (ce->name)) { + !is_empty_dir (empty_submodule_path.buf)) { spf->result =1 ;strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), ce->name); } +strbuf_release (&empty_submodule_path); } }
t/t5526-fetch-submodules.sh +124 -0
128 lines not on this page · raw message
Signature · 1 line
2.6.2
Re: [PATCH v3] submodules: fix of regression on fetching of non-init subsub-repo
submodule.c
Quoted 64 lines · Peter Kaestle
A regression has been introduced by a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).The scenario in which it triggers is when one has a repository with a submodule inside a submodule like this:
superproject/middle_repo/inner_repoPerson A and B have both a clone of it, while Person B is not working with the
inner_repoand thus does not have it initialized in his working copy.Now person A introduces a change to the
inner_repoand propagates it through themiddle_repoand the superproject.Once person A pushed the changes and person B wants to fetch them using "git fetch" at the superproject level, B's git call will return with error saying:
Could not access submodule '
inner_repo'
Errors during submodule fetch:
middle_repoExpectation is that in this case the inner submodule will be recognized as uninitialized submodule and skipped by the git fetch command.
This used to work correctly before 'a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.Starting with a62387b the code wants to evaluate "
is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.This patch ensures
is_empty_dir()is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.
The first attempt to fix this regression, in
1b7ac4e6d4(submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in7091499bc0(Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.Signed-off-by Peter Kaestle <[email protected]>CC Junio C Hamano <[email protected]>CC Philippe Blain <[email protected]>CC Ralf Thielow <[email protected]>CC Philippe Blain <[email protected]>--- submodule.c | 7 ++- t/t5526-fetch-submodules.sh | 124 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+), 1 deletion(-)submodule.c +0
-0
[...]
These are my test run outputs:final_fix [PATCH v3]
==========
# passed all 41 test(s)
recursion_regression [1b7ac4e6d4]
==========
not ok 41 - recursive fetch after deinit a submodule
no_fix [3a0b884c - origin/master]
==========
not ok 38 - fetching a superproject containing an uninitialized sub/sub project
Signature · 2 lines
kind regards --peter;
Re: [PATCH v3] submodules: fix of regression on fetching of non-init subsub-repo
In reply to Peter KaestleQuoted 50 lines · Peter Kaestle
A regression has been introduced by a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).The scenario in which it triggers is when one has a repository with a submodule inside a submodule like this:
superproject/middle_repo/inner_repoPerson A and B have both a clone of it, while Person B is not working with the
inner_repoand thus does not have it initialized in his working copy.Now person A introduces a change to the
inner_repoand propagates it through themiddle_repoand the superproject.Once person A pushed the changes and person B wants to fetch them using "git fetch" at the superproject level, B's git call will return with error saying:
Could not access submodule '
inner_repo'
Errors during submodule fetch:
middle_repoExpectation is that in this case the inner submodule will be recognized as uninitialized submodule and skipped by the git fetch command.
This used to work correctly before 'a62387b (
submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.Starting with a62387b the code wants to evaluate "
is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.This patch ensures
is_empty_dir()is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.
The first attempt to fix this regression, in
1b7ac4e6d4(submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in7091499bc0(Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.
Quoted 6 lines
Signed-off-by Peter Kaestle <[email protected]>CC Junio C Hamano <[email protected]>CC Philippe Blain <[email protected]>CC Ralf Thielow <[email protected]>CC Philippe Blain <[email protected]>---
Thanks, will replace.
Re: [PATCH v3] submodules: fix of regression on fetching of non-init subsub-repo
In reply to Peter KaestleHi Peter,
Quoted 6 lines · Peter Kaestle
-- 8< --
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.
I think this paragraph could be removed as it's saying the same thing as the one below.
Quoted 113 lines
The first attempt to fix this regression, in
1b7ac4e6d4(submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in7091499bc0(Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.Signed-off-by Peter Kaestle <[email protected]>CC Junio C Hamano <[email protected]>CC Philippe Blain <[email protected]>CC Ralf Thielow <[email protected]>CC Philippe Blain <[email protected]>--- submodule.c | 7 ++- t/t5526-fetch-submodules.sh | 124 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+), 1 deletion(-)submodule.c +6
-1@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&submodule_prefix);return 1 ; }else { +struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);free (task);@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,* An empty directory is normal, * the submodule is not initialized */ +strbuf_addf (&empty_submodule_path,"%s/%s/" , + spf->r->worktree, + ce->name);if (S_ISGITLINK (ce->ce_mode) && - !is_empty_dir (ce->name)) { + !is_empty_dir (empty_submodule_path.buf)) { spf->result =1 ;strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), ce->name); } +strbuf_release (&empty_submodule_path); } }t/t5526-fetch-submodules.sh +53
-0@@ -719,4 +719,128 @@ test_expect_success 'fetch new submodule commit intermittently referenced by sup ) ' +add_commit_push () { + dir="$1" && + msg="$2" && + shift 2 && + git -C "$dir" add "$@" && + git -C "$dir" commit -a -m "$msg" && + git -C "$dir" push +} + +compare_refs_in_dir () { + fail= && + if test "x$1" = 'x!' + then + fail='!' && + shift + fi && + git -C "$1" rev-parse --verify "$2" >expect && + git -C "$3" rev-parse --verify "$4" >actual && + eval $fail test_cmp expect actual +} + + +test_expect_success 'setup nested submodule fetch test' ' + # does not depend on any previous test setups + + for repo in outer middle inner + do + git init --bare $repo && + git clone $repo ${repo}_content && + echo "$repo" >"${repo}_content/file" && + add_commit_push ${repo}_content "initial" file || + return 1 + done && + + git clone outer A && + git -C A submodule add "$pwd/middle" && + git -C A/middle/ submodule add "$pwd/inner" && + add_commit_push A/middle/ "adding inner sub" .gitmodules inner && + add_commit_push A/ "adding middle sub" .gitmodules middle && + + git clone outer B && + git -C B/ submodule update --init middle && + + compare_refs_in_dir A HEAD B HEAD && + compare_refs_in_dir A/middle HEAD B/middle HEAD && + test_path_is_file B/file && + test_path_is_file B/middle/file && + test_path_is_missing B/middle/inner/file && + + echo "change on inner repo of A" >"A/middle/inner/file" && + add_commit_push A/middle/inner "change on inner" file && + add_commit_push A/middle "change on inner" inner && + add_commit_push A "change on inner" middle
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 784
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+' + +test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' ' + # depends on previous test for setup + + git -C B/ fetch && + compare_refs_in_dir A origin/HEAD B origin/HEAD +' + +fetch_with_recusion_abort () {
s/recusion/recursion/
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 787
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ # In a regression the following git call will run into infinite recursion. + # To handle that, we connect the sed command to the git call by a pipe + # so that sed can kill the infinite recusion when detected.
s/recusion/recursion/
Quoted 26 lines
t/t5526-fetch-submodules.sh +26
-0+ # The recursion creates git output like: + # Fetching submodule sub + # Fetching submodule sub/sub <-- [1] + # Fetching submodule sub/sub/sub + # ... + # [1] sed will stop reading and cause git to eventually stop and die + + git -C "$1" fetch --recurse-submodules 2>&1 | + sed "/Fetching submodule $2[^$]/q" >out && + ! grep "Fetching submodule $2[^$]" out +} + +test_expect_success 'setup recursive fetch with uninit submodule' ' + # does not depend on any previous test setups + + # setup a remote superproject to make git fetch work with an uninit submodule + git init --bare super_bare && + git clone super_bare super && + git init sub && + + >sub/file && + git -C sub add file && + git -C sub commit -m "add file" && + git -C sub rev-parse HEAD >expect && + + # adding submodule without cloning
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 823
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ echo "[submodule \"sub\"]" >super/.gitmodules && + echo "path = sub" >>super/.gitmodules && + echo "url = ../sub" >>super/.gitmodules && + git -C super update-index --add --cacheinfo 160000 $(cat expect) sub && + mkdir super/sub && + + git -C super submodule status >out && + sed -e "s/^-//" -e "s/ sub.*$//" out >actual && + test_cmp expect actual +'
I think this is overly complicated, what I was hinting at was adding the submodule
in the superproject before cloning it, so something along these lines:
test_create_repo super &&test_commit -C super initial &&test_create_repo sub &&test_commit -C sub initial &&
git -C sub rev-parse HEAD >expect &&
git -C super submodule add ../sub &&
git -C super commit -m "add sub" &&
git clone super superclone &&
git -C superclone submodule status >out &&
sed -e "s/^-//" -e "s/ sub.*$//" out >actual &&test_cmp expect actual
And then running the two tests below in "superclone".
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 828
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+ +test_expect_success 'recursive fetch with uninit submodule' ' + # depends on previous test for setup + + fetch_with_recusion_abort super sub &&
s/recusion/recursion/
t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 840
Quoted 12 lines
t/t5526-fetch-submodules.sh +12
-0+ git -C super submodule status >out && + sed -e "s/^-//" -e "s/ sub$//" out >actual && + test_cmp expect actual +' + +test_expect_success 'recursive fetch after deinit a submodule' ' + # depends on previous test for setup + + git -C super submodule update --init sub && + git -C super submodule deinit -f sub && + + fetch_with_recusion_abort super sub &&
s/recusion/recursion/
Quoted 9 lines
t/t5526-fetch-submodules.sh +5
-1+ git -C super submodule status >out && + sed -e "s/^-//" -e "s/ sub$//" out >actual && + test_cmp expect actual +' + test_done --2.6.2
Philippe.
Re: [PATCH v3] submodules: fix of regression on fetching of non-init subsub-repo
Hi Philippe,
when sending the patch yesterday, I had the gut feeling to hold it back to double check it a day later. Should have done so, some of your findings were really stupid mistakes.
Quoted 11 lines · Philippe Blain
Quoted 6 lines · Peter Kaestle
-- 8< --
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.
I think this paragraph could be removed as it's saying the same thing as the one below.
jip.
[...]
Quoted 4 lines
Quoted 1 line
+fetch_with_recusion_abort () {
s/recusion/recursion/
yes.
[...]
Quoted 38 lines
Quoted 14 lines
t/t5526-fetch-submodules.sh +14
-0+test_expect_success 'setup recursive fetch with uninit submodule' ' + # does not depend on any previous test setups + + # setup a remote superproject to make git fetch work with an uninit submodule + git init --bare super_bare && + git clone super_bare super && + git init sub && + + >sub/file && + git -C sub add file && + git -C sub commit -m "add file" && + git -C sub rev-parse HEAD >expect && + + # adding submodule without cloningt/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 823
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ echo "[submodule \"sub\"]" >super/.gitmodules && + echo "path = sub" >>super/.gitmodules && + echo "url = ../sub" >>super/.gitmodules && + git -C super update-index --add --cacheinfo 160000 $(cat expect) sub && + mkdir super/sub && + + git -C super submodule status >out && + sed -e "s/^-//" -e "s/ sub.*$//" out >actual && + test_cmp expect actual +'I think this is overly complicated, what I was hinting at was adding the submodule
in the superproject before cloning it, so something along these lines:
test_create_reposuper &&test_commit-C super initial &&test_create_reposub &&test_commit-C sub initial &&
git -C sub rev-parse HEAD >expect &&git -C super submodule add ../sub &&
git -C super commit -m "add sub" &&
Quoted 6 lines
git clone super superclone &&
git -C superclone submodule status >out &&
sed -e "s/^-//" -e "s/ sub.*$//" out >actual &&test_cmpexpect actualAnd then running the two tests below in "superclone".
Indeed, looks much simpler, I gave it a try run and it resulted in same behavior, will double check the logic of it and then send v4. Thanks a lot.
[...]
Signature · 2 lines
kind regards --peter;
PATCH v4 1/1 submodules: fix of regression on fetching of non-init subsub-repo
A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).
The scenario in which it triggers is when one has a repository with a submodule inside a submodule like this:
superproject/middle_repo/inner_repo
Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.
Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.
Once person A pushed the changes and person B wants to fetch them using "git fetch" at the superproject level, B's git call will return with error saying:
Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo
Expectation is that in this case the inner submodule will be recognized as uninitialized submodule and skipped by the git fetch command.
This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.
Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.
This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.
The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.
--- submodule.c | 7 ++- t/t5526-fetch-submodules.sh | 117 ++++++++++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+), 1 deletion(-)
submodule.c +6 -1
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&submodule_prefix);return 1 ; }else { +struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);free (task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,* An empty directory is normal, * the submodule is not initialized */ +strbuf_addf (&empty_submodule_path,"%s/%s/" , + spf->r->worktree, + ce->name);if (S_ISGITLINK (ce->ce_mode) && - !is_empty_dir (ce->name)) { + !is_empty_dir (empty_submodule_path.buf)) { spf->result =1 ;strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), ce->name); } +strbuf_release (&empty_submodule_path); } }
t/t5526-fetch-submodules.sh +117 -0
121 lines not on this page · raw message
Signature · 1 line
2.29.2.457.gf179bca
Re: [PATCH v4] submodules: fix of regression on fetching of non-init subsub-repo
Quoted 3 lines · Peter Kaestle
---8<---
Thanks again for working on this.
Philippe.P.S. I wouldn't call anything you wrote in v3 "stupid mistakes" :) we all have a lot on our minds
- especially these days!
Hi Philippe,
Am Mi., 2. Dez. 2020 um 18:20 Uhr schrieb Philippe Blain <[email protected]>:
Quoted 7 lines
Ok, you don't add '--init' but the submodule gets checked out, so it looks like you have 'submodule.active' set to a pathspec that matches 'sha1collisiondetection'. Did you clone the git repo with '--recurse-submodules', which would add '.' as the value of 'submodule.active' ? Or maybe you manually configured that value in your global gitconfig ?
Sry, I did not mention that I started with the submodule already initialized and checked out. IIRC I deinitialized the submodule with 'git submodule deinit -f sha1collisiondetection/' and ran 'git fetch' to see if it happens on the git repo, too, since I encountered the issue on another repo first.
I can see the same misbehaviour when I run
$ git -c submodule.recurse=false fetch --recurse-submodules too, so I think it doesn't have necessarily something to do with the 'submodule.recursive' config, but with 'recurse submodules' in general.
Ralf
Hi Ralf,
Quoted 16 lines · Ralf Thielow
Am Mi., 2. Dez. 2020 um 18:20 Uhr schrieb Philippe Blain <[email protected]>:
Quoted 7 lines
Ok, you don't add '--init' but the submodule gets checked out, so it looks like you have 'submodule.active' set to a pathspec that matches 'sha1collisiondetection'. Did you clone the git repo with '--recurse-submodules', which would add '.' as the value of 'submodule.active' ? Or maybe you manually configured that value in your global gitconfig ?
Sry, I did not mention that I started with the submodule already initialized and checked out. IIRC I deinitialized the submodule with 'git submodule deinit -f sha1collisiondetection/' and ran 'git fetch' to see if it happens on the git repo, too, since I encountered the issue on another repo first.
Quoted 5 lines
I can see the same misbehaviour when I run
$ git -c submodule.recurse=false fetch --recurse-submodules too, so I think it doesn't have necessarily something to do with the 'submodule.recursive' config, but with 'recurse submodules' in general.
This is interesting. There are several testcases using "--recurse-submodules". Have you tried, whether any of these trigger?
Signature · 2 lines
kind regards --peter;
Hi Peter,
Am Do., 3. Dez. 2020 um 09:20 Uhr schrieb Peter Kästle <[email protected]>:
Quoted 8 lines
Quoted 5 lines
I can see the same misbehaviour when I run
$ git -c submodule.recurse=false fetch --recurse-submodules too, so I think it doesn't have necessarily something to do with the 'submodule.recursive' config, but with 'recurse submodules' in general.This is interesting. There are several testcases using "--recurse-submodules". Have you tried, whether any of these trigger?
The test suite passes.
Ralf
Hi Ralf,
Quoted 9 lines · Ralf Thielow
Hi Peter,
Am Do., 3. Dez. 2020 um 09:20 Uhr schrieb Peter Kästle <[email protected]>:
Quoted 5 lines
Quoted 5 lines
I can see the same misbehaviour when I run
$ git -c submodule.recurse=false fetch --recurse-submodules too, so I think it doesn't have necessarily something to do with the 'submodule.recursive' config, but with 'recurse submodules' in general.
Quoted 5 lines
Quoted 3 lines
This is interesting. There are several testcases using "--recurse-submodules". Have you tried, whether any of these trigger?
The test suite passes.
This is what I experienced. Thus it is important to get your case into the test suite. Could you please send us a command sequence how to create your repository structure from scratch, so that a test case can be created? - Of course, if you want, you can also create a test case on your own.
Signature · 2 lines
kind regards --peter;
Hi Peter,
Am Do., 3. Dez. 2020 um 10:43 Uhr schrieb Peter Kästle <[email protected]>:
Quoted 9 lines
Quoted 2 lines
The test suite passes.
This is what I experienced. Thus it is important to get your case into the test suite. Could you please send us a command sequence how to create your repository structure from scratch, so that a test case can be created? - Of course, if you want, you can also create a test case on your own.
It can be reproduced with the following sequence of commands:
git init sub
cd sub
touch file
git add file
git commit -m "add file"
cd ..
git init main
cd main
git submodule add ../sub
git submodule init
git submodule update --checkout
git submodule deinit -f sub/
git fetch --recurse-submodules
Ralf
Hi Ralf,
Quoted 15 lines · Ralf Thielow
It can be reproduced with the following sequence of commands:
git init sub
cd sub
touch file
git add file
git commit -m "add file"
cd ..
git init main
cd main
git submodule add ../sub
git submodule init
git submodule update --checkout
git submodule deinit -f sub/
git fetch --recurse-submodules
With git from master state the "git fetch --recurse-submodules" results in an infinite recurse call.
I translated your sequence into a bash script, which can then be easily converted into a test case for git. Problematic was the infinite recurse loop of the git fetch command, which I solved by grep'ing for the second recursion output and abort using -m1. Could you please confirm, that you see "passed" for the good git versions and "failed" for the bad ones?
#!/bin/bash
testcase () {
rm -Rf main sub &&
git init main &&
git init sub &&
touch sub/file &&
git -C sub add file &&
git -C sub commit -m "add file" &&
git -C sub rev-parse HEAD >expect &&
git -C main submodule add ../sub &&
git -C main submodule init &&
git -C main submodule update --checkout &&
git -C main submodule deinit -f sub &&
! git -C main fetch --recurse-submodules |&
grep -v -m1 "Fetching submodule sub$" &&
git -C main submodule status |
sed -e "s/^-//" -e "s/ sub$//" >actual &&
cmp expect actual
}if testcase
then
echo "passed"
else
echo "failed"
fi
Signature · 1 line
--peter;
Hi Peter,
Am Do., 3. Dez. 2020 um 16:10 Uhr schrieb Peter Kästle <[email protected]>:
Quoted 9 lines
I translated your sequence into a bash script, which can then be easily converted into a test case for git. Problematic was the infinite recurse loop of the git fetch command, which I solved by grep'ing for the second recursion output and abort using -m1.
Could you please confirm, that you see "passed" for the good git versions and "failed" for the bad ones?
I can confirm that. It "failed" on current master version 2.29.2.435.g72ffeb997e
and "passed" on the latest tag 2.29.2.
Thanks,
Ralf