Skip to contentSkip to conversation

BUG in fetching non-checked out submodule

PATCH In review Ralf Thielow 5 participants 36 messages Dec 2, 2020 – Dec 9, 2020

lore
Share & tools
lore
#0 lore raw

Hi,

I have the current 'master' branch of git installed and get the following error when fetching a submodule that is not checked out.

I've bisected this error down to commit 1b7ac4e6d4 (submodules: fix of regression on fetching of non-init subsub-repo, 2020-11-12)

$ git version
git version 2.29.2.435.g72ffeb997e
$ git config --get submodule.recurse
true
$ git submodule status
-855827c583bc30645ba427885caa40c5b81764d2 sha1collisiondetection
$ git fetch
Fetching submodule sha1collisiondetection
Fetching submodule sha1collisiondetection/sha1collisiondetection
Fetching submodule
sha1collisiondetection/sha1collisiondetection/sha1collisiondetection
Fetching submodule
sha1collisiondetection/sha1collisiondetection/sha1collisiondetection/sha1collisiondetection
...
$ git submodule update --checkout
Submodule path 'sha1collisiondetection': checked out
'855827c583bc30645ba427885caa40c5b81764d2'
$ git fetch
Fetching submodule sha1collisiondetection
$

Ralf

#1 lore raw

Hi Ralf,

Quoted 11 lines · Ralf Thielow

Hi,

I have the current 'master' branch of git installed and get the following error when fetching a submodule that is not checked out.

I've bisected this error down to commit 1b7ac4e6d4 (submodules: fix of regression on fetching of non-init subsub-repo, 2020-11-12)

Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

As you discovered, it breaks other scenarios.

Quoted 6 lines
$ git version
git version 2.29.2.435.g72ffeb997e
$ git config --get submodule.recurse
true

Yeah, I think the test suite could make more efforts to run more tests with that setting turned 'on', but it would require significants efforts since it changes the behaviour of several commands.

Meta question: is there an easy way to run the whole test suite with specific config options turned on ?

Quoted 13 lines
$ git submodule status
-855827c583bc30645ba427885caa40c5b81764d2 sha1collisiondetection
$ git fetch
Fetching submodule sha1collisiondetection
Fetching submodule sha1collisiondetection/sha1collisiondetection
Fetching submodule
sha1collisiondetection/sha1collisiondetection/sha1collisiondetection
Fetching submodule
sha1collisiondetection/sha1collisiondetection/sha1collisiondetection/sha1collisiondetection
...
Quoted 3 lines
$ git submodule update --checkout
Submodule path 'sha1collisiondetection': checked out
'855827c583bc30645ba427885caa40c5b81764d2'

Ok, you don't add '--init' but the submodule gets checked out, so it looks like you have 'submodule.active' set to a pathspec that matches 'sha1collisiondetection'. Did you clone the git repo with '--recurse-submodules', which would add '.' as the value of 'submodule.active' ? Or maybe you manually configured that value in your global gitconfig ?

Thanks for the report,

Philippe.

#2 lore raw
Quoted 5 lines · Philippe Blain

Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

Quoted 8 lines

As you discovered, it breaks other scenarios.

Quoted 6 lines
$ git version
git version 2.29.2.435.g72ffeb997e
$ git config --get submodule.recurse
true

I think the current situation is probably worse.

As a short-term fix, we should revert 1b7ac4e6d4 until we can come up with a real fix, probably.

Quoted 4 lines

Yeah, I think the test suite could make more efforts to run more tests with that setting turned 'on', but it would require significants efforts since it changes the behaviour of several commands.

I am not sure if the question is about amount of efforts.

A configuration variable is there to change the behaviour of commands, so a test of a command that has been running happily and producing a set of expected outcome with a configuration unset should break the expectation when the configuration is set --- otherwise there is no point in having a configuration variable.

Quoted 2 lines

Meta question: is there an easy way to run the whole test suite with specific config options turned on ?

Hence, I do not think it even makes sense to have such an "easy way". If the "fetch" command, for example, is expected to change behaviour depending on the value of submodule.recurse, a test written for the case where the variable is not set should produce different outcome when the variable is set.

What we need may be a better test coverage. submodule.recurse is a later addition, and all tests written earlier do test how the commands behave without the configuration being set. If one wants to change the behaviour of these commands when the configuration is set, new tests to specify what the expected behaviour need to be added.

Quoted 1 line

Thanks for the report,

Yup, thanks for helping out.

#3 lore raw

Hi,

Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Philippe Blain

Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

As you discovered, it breaks other scenarios.

Quoted 4 lines
$ git version
git version 2.29.2.435.g72ffeb997e
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
$ git config --get submodule.recurse
true

I think the current situation is probably worse.

As a short-term fix, we should revert 1b7ac4e6d4 until we can come up with a real fix, probably.

Junio: This is why I originally intended to commit the test case for the testsuite separated from the revert and wanted to start a discussion about the actual real fix for the issue:
https://public-inbox.org/git/[email protected]/

My proposal would be to revert 1b7ac4e6d4 and isolate the test case "test_expect_success 'setup nested submodule fetch test' '" make it "test_expect_failure" and apply it instead, until we come up with a real solution.

Quoted 22 lines
Quoted 4 lines

Yeah, I think the test suite could make more efforts to run more tests with that setting turned 'on', but it would require significants efforts since it changes the behaviour of several commands.

I am not sure if the question is about amount of efforts.

A configuration variable is there to change the behaviour of commands, so a test of a command that has been running happily and producing a set of expected outcome with a configuration unset should break the expectation when the configuration is set --- otherwise there is no point in having a configuration variable.

Quoted 2 lines

Meta question: is there an easy way to run the whole test suite with specific config options turned on ?

Hence, I do not think it even makes sense to have such an "easy way". If the "fetch" command, for example, is expected to change behaviour depending on the value of submodule.recurse, a test written for the case where the variable is not set should produce different outcome when the variable is set.

Quoted 5 lines

What we need may be a better test coverage. submodule.recurse is a later addition, and all tests written earlier do test how the commands behave without the configuration being set. If one wants to change the behaviour of these commands when the configuration is set, new tests to specify what the expected behaviour need to be added.

Can we start a new test suite for tests on which this variable is set? The case of Ralf can be used as first test case.

Ralf: could you please send a command sequence, which recreates the required repository setup from scratch and is able to trigger your observation? Then we can convert this into a test case.

Thanks.

Signature · 2 lines
best regards
--peter;
#4 lore raw

Hello Peter,

Quoted 30 lines · Peter Kästle

Hi,

Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Philippe Blain

Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

As you discovered, it breaks other scenarios.

Quoted 4 lines
$ git version
git version 2.29.2.435.g72ffeb997e
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
$ git config --get submodule.recurse
true

I think the current situation is probably worse.

As a short-term fix, we should revert 1b7ac4e6d4 until we can come up with a real fix, probably.

Junio: This is why I originally intended to commit the test case for the testsuite separated from the revert and wanted to start a discussion about the actual real fix for the issue:
https://public-inbox.org/git/[email protected]/

Quoted 4 lines

My proposal would be to revert 1b7ac4e6d4 and isolate the test case "test_expect_success 'setup nested submodule fetch test' '" make it "test_expect_failure" and apply it instead, until we come up with a real solution.

I think I have the real solution. I did some debugging and I think it is quite easy:
In 'get_next_submodule', 'get_submodule_repo_for(spf->r, task->sub)' fails to get a repo pointer for the submodule repository, since it is not initialized. That
is normal. Then we go in the "else" branch, and hit this code:

/*
* An empty directory is normal,
* the submodule is not initialized
*/
if (S_ISGITLINK(ce->ce_mode) &&
!is_empty_dir(ce->name)) {

'is_empty_dir' receives ce->name, but the current working directory is the Git directory of 'middle', so clearly is_empty_dir returns false, as

/path/to/git/t/trash directory.t5526-fetch-submodules/B/.git/modules/middle/inner

is a non-existent path. The path that we should send is the worktree of inner, ie.
the concatenation of spf->r->worktree and ce->name. This would give

/path/to/git/t/trash directory.t5526-fetch-submodules/B/middle/inner,

which is an empty directory since the inner submodule is not initialized, and we would not get the "Could not access submodule inner" error that you wanted to solve.

Cheers,

Philippe.

#5 lore raw

Hi Philippe,

Quoted 57 lines · Philippe Blain
Quoted 27 lines · Junio C Hamano
Quoted 13 lines
Quoted 11 lines · Philippe Blain

Thanks for bisecting it. That commit wanted to fix a different bug related to nested submodules, and the route taken was simply reverting an earlier commit (a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

As you discovered, it breaks other scenarios.

Quoted 4 lines
$ git version
git version 2.29.2.435.g72ffeb997e
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
$ git config --get submodule.recurse
true

I think the current situation is probably worse.

As a short-term fix, we should revert 1b7ac4e6d4 until we can come up with a real fix, probably.

Junio: This is why I originally intended to commit the test case for the testsuite separated from the revert and wanted to start a discussion about the actual real fix for the issue:
https://public-inbox.org/git/[email protected]/

Quoted 4 lines

My proposal would be to revert 1b7ac4e6d4 and isolate the test case "test_expect_success 'setup nested submodule fetch test' '" make it "test_expect_failure" and apply it instead, until we come up with a real solution.

I think I have the real solution. I did some debugging and I think it is quite easy:
In 'get_next_submodule', 'get_submodule_repo_for(spf->r, task->sub)' fails to get a repo pointer for the submodule repository, since it is not initialized. That
is normal. Then we go in the "else" branch, and hit this code:

/*
* An empty directory is normal,
* the submodule is not initialized
*/
if (S_ISGITLINK(ce->ce_mode) &&
!is_empty_dir(ce->name)) {

'is_empty_dir' receives ce->name, but the current working directory is the Git directory of 'middle', so clearly is_empty_dir returns false, as

/path/to/git/t/trash directory.t5526-fetch-submodules/B/.git/modules/middle/inner

is a non-existent path. The path that we should send is the worktree of inner, ie.
the concatenation of spf->r->worktree and ce->name. This would give

Quoted 5 lines

/path/to/git/t/trash directory.t5526-fetch-submodules/B/middle/inner,

which is an empty directory since the inner submodule is not initialized, and we would not get the "Could not access submodule inner" error that you wanted to solve.

On quick glance this sounds plausible, but to fully understand it I need to put some effort in reading this code again. I hope to do so tomorrow. We can then compile a new set of patches including this real fix and Ralf's and my test case.

Thanks for digging into it.

Signature · 2 lines
kind regards
--peter;
#6 lore raw
Quoted 6 lines · Peter Kästle

On quick glance this sounds plausible, but to fully understand it I need to put some effort in reading this code again. I hope to do so tomorrow. We can then compile a new set of patches including this real fix and Ralf's and my test case.

Thanks for digging into it.

Yeah, thanks, both for noticing problem so quickly and started digging to find the real solution.

By the way, as to the "if this were originally two patches, we could have saved the test that expects failure" you raised, I do not think it is a good idea to optimize for cases where changes turn out problematic and have to get reverted.

#7 lore raw

PATCH 1/1 submodules: fix of regression on fetching of non-init subsub-repo

In reply to Peter Kästle

A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

The scenario in which it triggers is when one has a remote repository with a subrepository inside a subrepository like this:
superproject/middle_repo/inner_repo

Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.

Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.

Once person A pushed the changes and person B wants to fetch them using "git fetch" on superproject level, B's git call will return with error saying:

Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo

Expectation is that in this case the inner submodule will be recognized as uninitialized subrepository and skipped by the git fetch command.

This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.

Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.

This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
---
 submodule.c                 |  7 ++-
 t/t5526-fetch-submodules.sh | 94 +++++++++++++++++++++++++++++++++++++
 2 files changed, 100 insertions(+), 1 deletion(-)
submodule.c +6 -1
index b3bb59f066..b561445329 100644
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,
 			strbuf_release(&submodule_prefix);
 			return 1;
 		} else {
+			struct strbuf empty_submodule_path = STRBUF_INIT;
 
 			fetch_task_release(task);
 			free(task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,
 			 * An empty directory is normal,
 			 * the submodule is not initialized
 			 */
+			strbuf_addf(&empty_submodule_path, "%s/%s/",
+							spf->r->worktree,
+							ce->name);
 			if (S_ISGITLINK(ce->ce_mode) &&
-			    !is_empty_dir(ce->name)) {
+			    !is_empty_dir(empty_submodule_path.buf)) {
 				spf->result = 1;
 				strbuf_addf(err,
 					    _("Could not access submodule '%s'\n"),
 					    ce->name);
 			}
+			strbuf_release(&empty_submodule_path);
 		}
 	}
 
t/t5526-fetch-submodules.sh +94 -0

98 lines not on this page · raw message

Signature · 1 line
2.29.2
#8 lore raw

Re: [PATCH] submodules: fix of regression on fetching of non-init subsub-repo

Quoted 4 lines · Peter Kaestle

[...]
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

Just a few small comments (nothing comprehensive) from a quick scan of the patch...

Mostly they are just minor style issues, not necessarily worth a re-roll, but there is one actionable item.

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 722

Quoted 8 lines

---

t/t5526-fetch-submodules.sh +4 -0
@@ -719,4 +719,98 @@ test_expect_success 'fetch new submodule commit intermittently referenced by sup
+add_commit_push () {
+       dir="$1"
+       msg="$2"
+       shift 2

We typically recommend including these assignments in the &&-chain to future-proof against someone later inserting code above them and not realizing that that code is not part of the &&-chain, in which case if the new code fails, the failure might go unnoticed.

Quoted 6 lines
t/t5526-fetch-submodules.sh +6 -0
+       git -C "$dir" add "$@" &&
+       git -C "$dir" commit -a -m "$msg" &&
+       git -C "$dir" push
+}
+
+compare_refs_in_dir () {

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 741

Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+       fail= &&
+       if test "x$1" = 'x!'
+       then
+               fail='!' &&
+               shift
+       fi &&
+       git -C "$1" rev-parse --verify "$2" >expect &&
+       git -C "$3" rev-parse --verify "$4" >actual &&
+       eval $fail test_cmp expect actual
+}

We have a test_cmp_rev() similar to this but it doesn't support -C as some of our other test functions do. I briefly wondered if it would make sense to extend it to understand -C, but even that wouldn't help this case since compare_refs_in_dir() introduced here involves two distinct directories. The need here is so special-purpose that it likely would not make sense to upgrade test_cmp_rev() to accommodate it. Okay.

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 755

Quoted 12 lines
t/t5526-fetch-submodules.sh +12 -0
+test_expect_success 'setup nested submodule fetch test' '
+       # does not depend on any previous test setups
+
+       for repo in outer middle inner
+       do
+               (
+                       git init --bare $repo &&
+                       git clone $repo ${repo}_content &&
+                       echo "$repo" >"${repo}_content/file" &&
+                       add_commit_push ${repo}_content "initial" file
+               ) || return 1
+       done &&

What is the purpose of the subshell here? Is it to ensure that commits in each repo have identical timestamps? Or is it just for making the && and || expression more clear? If the latter, we normally don't bother with the parentheses.

Quoted 4 lines
t/t5526-fetch-submodules.sh +4 -0
+       git clone outer A &&
+       git -C A submodule add "$pwd/middle" &&
+       git -C A/middle/ submodule add "$pwd/inner" &&
+       add_commit_push A/middle/ "adding inner sub" .gitmodules inner &&

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 770

Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+       add_commit_push A/ "adding middle sub" .gitmodules middle &&
+
+       git clone outer B &&
+       git -C B/ submodule update --init middle &&
+
+       compare_refs_in_dir A HEAD B HEAD &&
+       compare_refs_in_dir A/middle HEAD B/middle HEAD &&
+       test -f B/file &&
+       test -f B/middle/file &&
+       ! test -f B/middle/inner/file &&

These days we typically use test_path_exists() (or test_path_is_file()) and test_path_is_missing() rather than bare test.

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 792

Quoted 7 lines
t/t5526-fetch-submodules.sh +7 -0
+test_expect_success 'setup recursive fetch with uninit submodule' '
+       # does not depend on any previous test setups
+
+       git init main &&
+       git init sub &&
+
+       touch sub/file &&

Unless the timestamp of the file is significant to the test, in which case touch is used, we normally create empty files like this:

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 810

Quoted 6 lines

sub/file &&

t/t5526-fetch-submodules.sh +4 -0
+test_expect_success 'recursive fetch with uninit submodule' '
+       git -C main submodule deinit -f sub &&
+       ! git -C main fetch --recurse-submodules |&
+               grep -v -m1 "Fetching submodule sub$" &&

We want the test scripts to be portable, thus avoid Bashisms such as |&.

We also avoid placing a Git command upstream in a pipe since doing so causes the exit code of the Git command to be lost. Instead, we would normally send the Git output to a file and then send that file to whatever would be downstream of the Git command in the pipe. So, a mechanical rewrite of the above (without thinking too hard about it) might be:

    git -C main fetch --recurse-submodules >out 2>&1 &&
    ! grep -v -m1 "Fetching submodule sub$" &&

t/t5526-fetch-submodules.sh

Quoted 2 lines
t/t5526-fetch-submodules.sh +2 -0
+       git -C main submodule status |
+               sed -e "s/^-//" -e "s/ sub$//" >actual &&

Same comment about avoiding Git upstream in a pipe, so perhaps:

    git -C main submodule status >out &&
    sed -e "s/^-//" -e "s/ sub$//" out >actual &&
Quoted 2 lines

+ test_cmp expect actual
+'

#9 lore raw

Re: [PATCH] submodules: fix of regression on fetching of non-init subsub-repo

Hi Eric,

Quoted 11 lines · Eric Sunshine
Quoted 4 lines · Peter Kaestle

[...]
Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

Just a few small comments (nothing comprehensive) from a quick scan of the patch...

Mostly they are just minor style issues, not necessarily worth a re-roll, but there is one actionable item.

thanks for your comments. A new patch will follow soon.

Quoted 5 lines
Quoted 5 lines

---

t/t5526-fetch-submodules.sh +1 -0
@@ -719,4 +719,98 @@ test_expect_success 'fetch new submodule commit intermittently referenced by sup
+add_commit_push () {
Quoted 8 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 725

Quoted 3 lines
t/t5526-fetch-submodules.sh +3 -0
+       dir="$1"
+       msg="$2"
+       shift 2

We typically recommend including these assignments in the &&-chain to future-proof against someone later inserting code above them and not realizing that that code is not part of the &&-chain, in which case if the new code fails, the failure might go unnoticed.

ok

Quoted 18 lines
Quoted 7 lines
t/t5526-fetch-submodules.sh +6 -0
+       git -C "$dir" add "$@" &&
+       git -C "$dir" commit -a -m "$msg" &&
+       git -C "$dir" push
+}
+
+compare_refs_in_dir () {

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 741

Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+       fail= &&
+       if test "x$1" = 'x!'
+       then
+               fail='!' &&
+               shift
+       fi &&
+       git -C "$1" rev-parse --verify "$2" >expect &&
+       git -C "$3" rev-parse --verify "$4" >actual &&
+       eval $fail test_cmp expect actual
+}
Quoted 7 lines

We have a test_cmp_rev() similar to this but it doesn't support -C as some of our other test functions do. I briefly wondered if it would make sense to extend it to understand -C, but even that wouldn't help this case since compare_refs_in_dir() introduced here involves two distinct directories. The need here is so special-purpose that it likely would not make sense to upgrade test_cmp_rev() to accommodate it. Okay.

Yes, I saw that there's a similar function and I tried to modify this one first. Unfortunately this didn't work without touching much unaffected test code. So I propose to continue with this additional function.

Quoted 9 lines
Quoted 9 lines
t/t5526-fetch-submodules.sh +9 -0
+test_expect_success 'setup nested submodule fetch test' '
+       # does not depend on any previous test setups
+
+       for repo in outer middle inner
+       do
+               (
+                       git init --bare $repo &&
+                       git clone $repo ${repo}_content &&
+                       echo "$repo" >"${repo}_content/file" &&
Quoted 8 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 755

Quoted 3 lines
t/t5526-fetch-submodules.sh +3 -0
+                       add_commit_push ${repo}_content "initial" file
+               ) || return 1
+       done &&

What is the purpose of the subshell here? Is it to ensure that commits in each repo have identical timestamps? Or is it just for making the && and || expression more clear? If the latter, we normally don't bother with the parentheses.

It was intended to make the correlation of && and || clear. I have experienced many cases in the past where things were screwed because it was not clearly understood by everybody. I'll propose next patch without this subshell.

Quoted 10 lines
Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+       git clone outer A &&
+       git -C A submodule add "$pwd/middle" &&
+       git -C A/middle/ submodule add "$pwd/inner" &&
+       add_commit_push A/middle/ "adding inner sub" .gitmodules inner &&
+       add_commit_push A/ "adding middle sub" .gitmodules middle &&
+
+       git clone outer B &&
+       git -C B/ submodule update --init middle &&
+
+       compare_refs_in_dir A HEAD B HEAD &&
Quoted 8 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 770

Quoted 4 lines
t/t5526-fetch-submodules.sh +4 -0
+       compare_refs_in_dir A/middle HEAD B/middle HEAD &&
+       test -f B/file &&
+       test -f B/middle/file &&
+       ! test -f B/middle/inner/file &&

These days we typically use test_path_exists() (or test_path_is_file()) and test_path_is_missing() rather than bare test.

ok.

Quoted 11 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 792

Quoted 7 lines
t/t5526-fetch-submodules.sh +7 -0
+test_expect_success 'setup recursive fetch with uninit submodule' '
+       # does not depend on any previous test setups
+
+       git init main &&
+       git init sub &&
+
+       touch sub/file &&

Unless the timestamp of the file is significant to the test, in which case touch is used, we normally create empty files like this:

Quoted 1 line
     >sub/file &&

ok.

Quoted 13 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 810

Quoted 5 lines
t/t5526-fetch-submodules.sh +4 -0
+test_expect_success 'recursive fetch with uninit submodule' '
+       git -C main submodule deinit -f sub &&
+       ! git -C main fetch --recurse-submodules |&
+               grep -v -m1 "Fetching submodule sub$" &&

We want the test scripts to be portable, thus avoid Bashisms such as |&. > We also avoid placing a Git command upstream in a pipe since doing so causes the exit code of the Git command to be lost. Instead, we would
normally send the Git output to a file and then send that file to
whatever would be downstream of the Git command in the pipe. So, a
mechanical rewrite of the above (without thinking too hard about it)
might be:

Quoted 2 lines
     git -C main fetch --recurse-submodules >out 2>&1 &&
     ! grep -v -m1 "Fetching submodule sub$" &&

In general I agree, but for this special test case, it's required to have the two commands connected by a pipe, as the grep needs to kill the git call in error case. Otherwise for this regression git would go for an infinite recursion loop.

Of course, we can go for a "git 2>&1 | grep" solution.

Quoted 10 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 777

Quoted 2 lines
t/t5526-fetch-submodules.sh +2 -0
+       git -C main submodule status |
+               sed -e "s/^-//" -e "s/ sub$//" >actual &&

Same comment about avoiding Git upstream in a pipe, so perhaps:

     git -C main submodule status >out &&
     sed -e "s/^-//" -e "s/ sub$//" out >actual &&
Quoted 2 lines
t/t5526-fetch-submodules.sh +2 -0
+       test_cmp expect actual
+'

ok.

Signature · 2 lines
kind regards
--peter;
#10 lore raw

Re: [PATCH] submodules: fix of regression on fetching of non-init subsub-repo

Quoted 9 lines · Peter Kästle
Quoted 7 lines · Eric Sunshine

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 810

Quoted 2 lines · Peter Kaestle
t/t5526-fetch-submodules.sh +2 -0
+       ! git -C main fetch --recurse-submodules |&
+               grep -v -m1 "Fetching submodule sub$" &&

We want the test scripts to be portable, thus avoid Bashisms such as |&.

     git -C main fetch --recurse-submodules >out 2>&1 &&
     ! grep -v -m1 "Fetching submodule sub$" &&
Quoted 6 lines

In general I agree, but for this special test case, it's required to have the two commands connected by a pipe, as the grep needs to kill the git call in error case. Otherwise for this regression git would go for an infinite recursion loop.

Of course, we can go for a "git 2>&1 | grep" solution.

In that case, an in-code comment explaining why the output of git must be piped to grep would be helpful since it is not uncommon for people to "modernize" test code as they are working in the vicinity, and removing such pipes is one of the modernizations often made. A comment would help avoid such a change.

#11 lore raw

PATCH v2 1/1 submodules: fix of regression on fetching of non-init subsub-repo

A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

The scenario in which it triggers is when one has a remote repository with a subrepository inside a subrepository like this:
superproject/middle_repo/inner_repo

Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.

Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.

Once person A pushed the changes and person B wants to fetch them using "git fetch" on superproject level, B's git call will return with error saying:

Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo

Expectation is that in this case the inner submodule will be recognized as uninitialized subrepository and skipped by the git fetch command.

This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.

Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.

This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
CC Eric Sunshine <[email protected]>
---
 submodule.c                 |   7 ++-
 t/t5526-fetch-submodules.sh | 104 ++++++++++++++++++++++++++++++++++++
 2 files changed, 110 insertions(+), 1 deletion(-)
submodule.c +6 -1
index b3bb59f066..b561445329 100644
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,
 			strbuf_release(&submodule_prefix);
 			return 1;
 		} else {
+			struct strbuf empty_submodule_path = STRBUF_INIT;
 
 			fetch_task_release(task);
 			free(task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,
 			 * An empty directory is normal,
 			 * the submodule is not initialized
 			 */
+			strbuf_addf(&empty_submodule_path, "%s/%s/",
+							spf->r->worktree,
+							ce->name);
 			if (S_ISGITLINK(ce->ce_mode) &&
-			    !is_empty_dir(ce->name)) {
+			    !is_empty_dir(empty_submodule_path.buf)) {
 				spf->result = 1;
 				strbuf_addf(err,
 					    _("Could not access submodule '%s'\n"),
 					    ce->name);
 			}
+			strbuf_release(&empty_submodule_path);
 		}
 	}
 
t/t5526-fetch-submodules.sh +104 -0

108 lines not on this page · raw message

Signature · 1 line
2.29.2
#12 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

Hi Peter,

Quoted 8 lines · Peter Kaestle

A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

The scenario in which it triggers is when one has a remote repository with a subrepository inside a subrepository like this:
superproject/middle_repo/inner_repo

The correct terminology is "submodule", not "subrepository".

Also, (minor point) I would just write "when one has a repository", as its simpler (the repository by itself is not "remote", it is only "remote" in relation the repositories that are cloned from it).

Quoted 9 lines

Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.

Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.

Once person A pushed the changes and person B wants to fetch them using "git fetch" on superproject level,

s/on/at the/

Quoted 9 lines

B's git call will return with error
saying:

Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo

Expectation is that in this case the inner submodule will be recognized as uninitialized subrepository and skipped by the git fetch command.

here again, terminology: "as an uninitialized submodule"

Quoted 7 lines

This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.

Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.

Quoted 8 lines

This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories.
This
issue was leading to an infinite loop when doing a revert of a62387b.

I would maybe add more details here, something like the following (we can cite your previous attempt, because it was merged to 'master'):

The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in
an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.

Quoted 31 lines
CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
CC Eric Sunshine <[email protected]>

---
submodule.c | 7 ++-
t/t5526-fetch-submodules.sh | 104 ++++++++++++++++++++++++++++++++++++

2 files changed, 110 insertions(+), 1 deletion(-)
submodule.c +4 -0
index b3bb59f066..b561445329 100644
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,
 			strbuf_release(&submodule_prefix);
 			return 1;
 		} else {
+			struct strbuf empty_submodule_path = STRBUF_INIT;
 
 			fetch_task_release(task);
 			free(task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,
 			 * An empty directory is normal,
 			 * the submodule is not initialized
 			 */
+			strbuf_addf(&empty_submodule_path, "%s/%s/",
+							spf->r->worktree,
+							ce->name);
 			if (S_ISGITLINK(ce->ce_mode) &&

submodule.c static int get_next_submodule(struct child_process *cp, · line 1501

Quoted 10 lines
submodule.c +2 -1
-			    !is_empty_dir(ce->name)) {
+			    !is_empty_dir(empty_submodule_path.buf)) {
 				spf->result = 1;
 				strbuf_addf(err,
 					    _("Could not access submodule '%s'\n"),
 					    ce->name);
 			}
+			strbuf_release(&empty_submodule_path);
 		}
 	}

Maybe a personal preference, but I would have gone for something a little simpler, like the following:

submodule.c +1 -1
index b3bb59f066..4200865174 100644
@@ -1486,7 +1486,7 @@ static int get_next_submodule(struct child_process *cp,
                         * the submodule is not initialized
                         */
                        if (S_ISGITLINK(ce->ce_mode) &&
-                           !is_empty_dir(ce->name)) {
+                           !is_empty_dir(repo_worktree_path(spf->r, "%s", ce->name))) {
                                spf->result = 1;
                                strbuf_addf(err,
                                            _("Could not access submodule '%s'\n"),
Quoted 75 lines
t/t5526-fetch-submodules.sh +0 -0
index dd8e423d25..666dd1e2b7 100755
@@ -719,4 +719,108 @@ test_expect_success 'fetch new submodule commit intermittently referenced by sup
 	)

'

t/t5526-fetch-submodules.sh +67 -0
+add_commit_push () {
+	dir="$1" &&
+	msg="$2" &&
+	shift 2 &&
+	git -C "$dir" add "$@" &&
+	git -C "$dir" commit -a -m "$msg" &&
+	git -C "$dir" push
+}
+
+compare_refs_in_dir () {
+	fail= &&
+	if test "x$1" = 'x!'
+	then
+		fail='!' &&
+		shift
+	fi &&
+	git -C "$1" rev-parse --verify "$2" >expect &&
+	git -C "$3" rev-parse --verify "$4" >actual &&
+	eval $fail test_cmp expect actual
+}
+
+
+test_expect_success 'setup nested submodule fetch test' '
+	# does not depend on any previous test setups
+
+	for repo in outer middle inner
+	do
+		git init --bare $repo &&
+		git clone $repo ${repo}_content &&
+		echo "$repo" >"${repo}_content/file" &&
+		add_commit_push ${repo}_content "initial" file ||
+		return 1
+	done &&
+
+	git clone outer A &&
+	git -C A submodule add "$pwd/middle" &&
+	git -C A/middle/ submodule add "$pwd/inner" &&
+	add_commit_push A/middle/ "adding inner sub" .gitmodules inner &&
+	add_commit_push A/ "adding middle sub" .gitmodules middle &&
+
+	git clone outer B &&
+	git -C B/ submodule update --init middle &&
+
+	compare_refs_in_dir A HEAD B HEAD &&
+	compare_refs_in_dir A/middle HEAD B/middle HEAD &&
+	test_path_is_file B/file &&
+	test_path_is_file B/middle/file &&
+	test_path_is_missing B/middle/inner/file &&
+
+	echo "change on inner repo of A" >"A/middle/inner/file" &&
+	add_commit_push A/middle/inner "change on inner" file &&
+	add_commit_push A/middle "change on inner" inner &&
+	add_commit_push A "change on inner" middle
+'
+
+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' '
+	# depends on previous test for setup
+
+	git -C B/ fetch &&
+	compare_refs_in_dir A origin/master B origin/master
+'
+
+
+test_expect_success 'setup recursive fetch with uninit submodule' '
+	# does not depend on any previous test setups
+
+	git init main &&

t/t5526-fetch-submodules.sh

Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+	git init sub &&
+
+	>sub/file &&
+	git -C sub add file &&
+	git -C sub commit -m "add file" &&
+	git -C sub rev-parse HEAD >expect &&
+
+	git -C main submodule add ../sub &&
+	git -C main submodule init &&
+	git -C main submodule update --checkout &&

These two steps are unnecessary as they are implicitly done by 'git submodule add'. I think we could reflect real life a little bit more by cloning the superproject, and running the 'recursive fetch with uninit submodule' test below in the clone.

t/t5526-fetch-submodules.sh

Quoted 9 lines
t/t5526-fetch-submodules.sh +9 -0
+	git -C main submodule status >out &&
+	sed -e "s/^ //" -e "s/ sub .*$//" out >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'recursive fetch with uninit submodule' '
+	# depends on previous test for setup
+
+	git -C main submodule deinit -f sub &&

Here you are deiniting the submodule, such that
the Git directory will stay in .git/modules/sub. This is not the same thing as a submodule that was never initialized ("uninitialized"), for which .git/modules/sub will not yet exist. So maybe we could harden the tests by also testing for that scenario ? I don't know... maybe the infinite loop only happens if .git/modules/sub actually already exists. If so, the test name should be "recursive fetch with deinitialized submodule", I think.

Quoted 18 lines
t/t5526-fetch-submodules.sh +18 -0
+
+	# In a regression the following git call will run into infinite recursion.
+	# To handle that, we connect the grep command to the git call by a pipe
+	# so that grep can kill the infinite recusion when detected.
+	# The recursion creates git output like:
+	# Fetching submodule sub
+	# Fetching submodule sub/sub              <-- [1]
+	# Fetching submodule sub/sub/sub
+	# ...
+	# [1] grep will trigger here and kill git by exiting and closing its stdin
+
+	! git -C main fetch --recurse-submodules 2>&1 |
+		grep -v -m1 "Fetching submodule sub$" &&
+	git -C main submodule status >out &&
+	sed -e "s/^-//" -e "s/ sub$//" out >actual &&
+	test_cmp expect actual
+'
+
Quoted 1 line

test_done

Thanks for working on that, and sorry for not having the time to comment before you sent v2.

Cheers,

Philippe.

#13 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

submodule.c static int get_next_submodule(struct child_process *cp, · line 1489

Quoted 13 lines · Philippe Blain

Maybe a personal preference, but I would have gone for something a little simpler, like the following:

submodule.c +1 -1
index b3bb59f066..4200865174 100644
@@ -1486,7 +1486,7 @@ static int get_next_submodule(struct child_process *cp,
                         * the submodule is not initialized
                         */
                        if (S_ISGITLINK(ce->ce_mode) &&
-                           !is_empty_dir(ce->name)) {
+                           !is_empty_dir(repo_worktree_path(spf->r, "%s", ce->name))) {

But then you leak the return value from repo_worktree_path(), no?

Quoted 5 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+test_expect_success 'recursive fetch with uninit submodule' '
+	# depends on previous test for setup
+
+	git -C main submodule deinit -f sub &&
Quoted 7 lines

Here you are deiniting the submodule, such that
the Git directory will stay in .git/modules/sub. This is not the same thing as a submodule that was never initialized ("uninitialized"), for which .git/modules/sub will not yet exist. So maybe we could harden the tests by also testing for that scenario ? I don't know... maybe the infinite loop only happens if .git/modules/sub actually already exists. If so, the test name should be "recursive fetch with deinitialized submodule", I think.

Even if the original breakage happens only for deinitialized case, it would be sensible to test uninitialized one as well, I would think.

Quoted 2 lines

Thanks for working on that, and sorry for not having the time to comment before you sent v2.

Thanks, all.

It's not like we corral everybody to a single place on a single day to work on a single thing. Reviews ov v2 by reviewers who have not seen v1 is totally expected and very much appreciated.

#14 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

Quoted 10 lines · Junio C Hamano
Quoted 2 lines · Philippe Blain

Thanks for working on that, and sorry for not having the time to comment before you sent v2.

Thanks, all.

It's not like we corral everybody to a single place on a single day to work on a single thing. Reviews ov v2 by reviewers who have not seen v1 is totally expected and very much appreciated.

Thank you very much for all your comments, they're much appreciated. I prefer spending more time on it now for getting it right, than to take another revert-rethink-refactor round. It will take me some time today to work on them.

Signature · 2 lines
kind regards
--peter;
#15 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

In reply to Philippe Blain
Quoted 12 lines · Philippe Blain

I would maybe add more details here, something like the following (we can cite your previous attempt, because it was merged to 'master'):

The first attempt to fix this regression, in 1b7ac4e6d4 (submodules: fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules:
fix of regression on fetching of non-init subsub-repo", 2020-12-02). To prevent future breakages, also add a regression test for this scenario.

Forgot to mention in my other response, but I do find this a very sensible addition.

Thanks.

#16 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

In reply to Philippe Blain
Quoted 11 lines · Philippe Blain

Hi Peter,

Quoted 8 lines · Peter Kaestle

A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

The scenario in which it triggers is when one has a remote repository with a subrepository inside a subrepository like this:
superproject/middle_repo/inner_repo

Quoted 5 lines

The correct terminology is "submodule", not "subrepository".

Also, (minor point) I would just write "when one has a repository", as its simpler (the repository by itself is not "remote", it is only "remote" in relation the repositories that are cloned from it).

ok.

Quoted 11 lines
Quoted 9 lines

Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.

Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.

Once person A pushed the changes and person B wants to fetch them using "git fetch" on superproject level,

s/on/at the/

ok.

Quoted 11 lines
Quoted 9 lines

B's git call will return with error
saying:

Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo

Expectation is that in this case the inner submodule will be recognized as uninitialized subrepository and skipped by the git fetch command.

here again, terminology: "as an uninitialized submodule"

ok.

Quoted 19 lines
Quoted 7 lines

This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.

Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.

Quoted 8 lines

This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories.
This
issue was leading to an infinite loop when doing a revert of a62387b.

I would maybe add more details here, something like the following (we can cite your previous attempt, because it was merged to 'master'):

Quoted 7 lines

The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in
an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.

Jip, I like that.

Quoted 51 lines
Quoted 31 lines
CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
CC Eric Sunshine <[email protected]>

---
submodule.c | 7 ++-
t/t5526-fetch-submodules.sh | 104 ++++++++++++++++++++++++++++++++++++

2 files changed, 110 insertions(+), 1 deletion(-)
submodule.c +4 -0
index b3bb59f066..b561445329 100644
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,
 			strbuf_release(&submodule_prefix);
 			return 1;
 		} else {
+			struct strbuf empty_submodule_path = STRBUF_INIT;
 
 			fetch_task_release(task);
 			free(task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,
 			 * An empty directory is normal,
 			 * the submodule is not initialized
 			 */
+			strbuf_addf(&empty_submodule_path, "%s/%s/",
+							spf->r->worktree,
+							ce->name);
 			if (S_ISGITLINK(ce->ce_mode) &&

submodule.c static int get_next_submodule(struct child_process *cp, · line 1501

Quoted 10 lines
submodule.c +2 -1
-			    !is_empty_dir(ce->name)) {
+			    !is_empty_dir(empty_submodule_path.buf)) {
 				spf->result = 1;
 				strbuf_addf(err,
 					    _("Could not access submodule '%s'\n"),
 					    ce->name);
 			}
+			strbuf_release(&empty_submodule_path);
 		}
 	}

Maybe a personal preference, but I would have gone for something a little simpler, like the following:

submodule.c +0 -0
index b3bb59f066..4200865174 100644

submodule.c static int get_next_submodule(struct child_process *cp, · line 1489

Quoted 5 lines
submodule.c +1 -1
@@ -1486,7 +1486,7 @@ static int get_next_submodule(struct child_process *cp,
                          * the submodule is not initialized
                          */
                         if (S_ISGITLINK(ce->ce_mode) &&
-                           !is_empty_dir(ce->name)) {
+                           !is_empty_dir(repo_worktree_path(spf->r, "%s", ce->name))) {

I'm not deep enough into the git code to judge which approach is the better one. From my perspective, being a foreigner to the git code, I like my proposed code more, as for me it's much easier to understand what's happening by having a meaningful variable name and without being forced to dig into outer functions first. Also Junio C Hamano <[email protected]> is having some concerns, which I can't judge:

Quoted 1 line

But then you leak the return value from repo_worktree_path(), no?

Thus for v3 I'll stick to my proposal and when you'll review it, please discuss with each other whether I should go for a v4 using repo_worktree_path().

[...]

Quoted 11 lines
Quoted 11 lines
submodule.c +11 -0
+
+test_expect_success 'setup recursive fetch with uninit submodule' '
+	# does not depend on any previous test setups
+
+	git init main &&
+	git init sub &&
+
+	>sub/file &&
+	git -C sub add file &&
+	git -C sub commit -m "add file" &&
+	git -C sub rev-parse HEAD >expect &&
Quoted 8 lines

submodule.c

Quoted 4 lines
submodule.c +4 -0
+
+	git -C main submodule add ../sub &&
+	git -C main submodule init &&
+	git -C main submodule update --checkout &&

These two steps are unnecessary as they are implicitly done by 'git submodule add'. I think we could reflect real life a little bit more by cloning the superproject, and running the 'recursive fetch with uninit submodule' test below in the clone.

Yes, you're right, "...init" and "...update..." can be removed.

Quoted 10 lines

submodule.c

Quoted 9 lines
submodule.c +9 -0
+	git -C main submodule status >out &&
+	sed -e "s/^ //" -e "s/ sub .*$//" out >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'recursive fetch with uninit submodule' '
+	# depends on previous test for setup
+
+	git -C main submodule deinit -f sub &&
Quoted 7 lines

Here you are deiniting the submodule, such that
the Git directory will stay in .git/modules/sub. This is not the same thing as a submodule that was never initialized ("uninitialized"), for which .git/modules/sub will not yet exist. So maybe we could harden the tests by also testing for that scenario ? I don't know... maybe the infinite loop only happens if .git/modules/sub actually already exists. If so, the test name should be "recursive fetch with deinitialized submodule", I think.

I added another test case for v3, which checks for this in case of never initialized submodule. When executing the test, I can see that the infinite loop regression only occurs after doing the init followed by a deinit. Thus renaming the test accordingly.

Signature · 2 lines
best regards
--peter;
#17 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

In reply to Peter Kaestle
Quoted 18 lines · Peter Kaestle
t/t5526-fetch-submodules.sh +18 -0
+add_commit_push () {
+	dir="$1" &&
+	msg="$2" &&
+	shift 2 &&
+	git -C "$dir" add "$@" &&
+	git -C "$dir" commit -a -m "$msg" &&
+	git -C "$dir" push
+}
+
+compare_refs_in_dir () {
+	fail= &&
+	if test "x$1" = 'x!'
+	then
+		fail='!' &&
+		shift
+	fi &&
+	git -C "$1" rev-parse --verify "$2" >expect &&
+	git -C "$3" rev-parse --verify "$4" >actual &&

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781

Quoted 10 lines
t/t5526-fetch-submodules.sh +7 -0
+	eval $fail test_cmp expect actual
+}
 
 
 
+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' '
+	# depends on previous test for setup
+
+	git -C B/ fetch &&
+	compare_refs_in_dir A origin/master B origin/master

Can we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 789

Quoted 5 lines
t/t5526-fetch-submodules.sh +5 -0
+test_expect_success 'setup recursive fetch with uninit submodule' '
+	# does not depend on any previous test setups
+
+	git init main &&
+	git init sub &&

"super vs sub" would give us a better contrast than "main vs sub", and it would help reduce mistakes in the mechanical conversion of "master" to "main" happening in another topic.

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 817

Quoted 9 lines
t/t5526-fetch-submodules.sh +9 -0
+	# In a regression the following git call will run into infinite recursion.
+	# To handle that, we connect the grep command to the git call by a pipe
+	# so that grep can kill the infinite recusion when detected.
+	# The recursion creates git output like:
+	# Fetching submodule sub
+	# Fetching submodule sub/sub              <-- [1]
+	# Fetching submodule sub/sub/sub
+	# ...
+	# [1] grep will trigger here and kill git by exiting and closing its stdin

"trigger here and kill..." -> "stop reading and cause git to eventually stop and die"

But we probably cannot use 'grep -m1' so it is a moot point.

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 820

Quoted 3 lines
t/t5526-fetch-submodules.sh +3 -0
+
+	! git -C main fetch --recurse-submodules 2>&1 |
+		grep -v -m1 "Fetching submodule sub$" &&

Unfortunately, "grep -m<count>" is not even in POSIX, I would think.

What do we expect to happen in the correct case?

- A line "Fetching submodule sub" and nothing else is given? That feels a bit brittle (how are we making sure, in the presence of "2>&1", that we will not get any other output, like progress?)

- "sub" is the only thing that appears on lines that begin with "Fetching submodule" (i.e. "Fetching submodule $something" where $something is not 'sub' is an error), and we allow other garbage in the output? That would be a bit more robust than the above.

As you seem to be comfortable using "sed" below, perhaps use it to extract the first few lines that say "^Fetching submodule " from the output and stop, and check that the output has only one such line about 'sub' and nothing else?

Quoted 3 lines
t/t5526-fetch-submodules.sh +3 -0
+	git -C main submodule status >out &&
+	sed -e "s/^-//" -e "s/ sub$//" out >actual &&
+	test_cmp expect actual
#18 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

Hi Junio,

Quoted 27 lines · Junio C Hamano
Quoted 20 lines · Peter Kaestle
t/t5526-fetch-submodules.sh +20 -0
+add_commit_push () {
+	dir="$1" &&
+	msg="$2" &&
+	shift 2 &&
+	git -C "$dir" add "$@" &&
+	git -C "$dir" commit -a -m "$msg" &&
+	git -C "$dir" push
+}
+
+compare_refs_in_dir () {
+	fail= &&
+	if test "x$1" = 'x!'
+	then
+		fail='!' &&
+		shift
+	fi &&
+	git -C "$1" rev-parse --verify "$2" >expect &&
+	git -C "$3" rev-parse --verify "$4" >actual &&
+	eval $fail test_cmp expect actual
+}
Quoted 10 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781

Quoted 5 lines
t/t5526-fetch-submodules.sh +5 -0
+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' '
+	# depends on previous test for setup
+
+	git -C B/ fetch &&
+	compare_refs_in_dir A origin/master B origin/master

Can we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?

Or, simpler, we could call "git remote set-head -a' in A and B in the setup script, which would make origin/HEAD in A and B point to the default branch, such that the call here could be :

compare_refs_in_dir A origin/HEAD B origin/HEAD

Philippe.

#19 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

Quoted 11 lines · Philippe Blain
Quoted 10 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781

Quoted 5 lines
t/t5526-fetch-submodules.sh +5 -0
+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' '
+	# depends on previous test for setup
+
+	git -C B/ fetch &&
+	compare_refs_in_dir A origin/master B origin/master

Can we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?

Quoted 4 lines

Or, simpler, we could call "git remote set-head -a' in A and B in the setup script, which would make origin/HEAD in A and B point to the default branch, such that the call here could be :

The set-up prepares A and B by cloning from elsewhere, no? Should we even need a set-head call?

Quoted 1 line

compare_refs_in_dir A origin/HEAD B origin/HEAD

Yes, using HEAD would be another simple way to avoid having to rely on the default behaviour.

THanks.

#20 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Philippe Blain
Quoted 10 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781

Quoted 5 lines
t/t5526-fetch-submodules.sh +5 -0
+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' '
+	# depends on previous test for setup
+
+	git -C B/ fetch &&
+	compare_refs_in_dir A origin/master B origin/master

Can we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?

Quoted 7 lines
Quoted 4 lines

Or, simpler, we could call "git remote set-head -a' in A and B in the setup script, which would make origin/HEAD in A and B point to the default branch, such that the call here could be :

The set-up prepares A and B by cloning from elsewhere, no? Should we even need a set-head call?

Ah, they are created by cloning an empty repository. That explains why. Thanks.

Quoted 6 lines
Quoted 1 line

compare_refs_in_dir A origin/HEAD B origin/HEAD

Yes, using HEAD would be another simple way to avoid having to rely on the default behaviour.

THanks.

#21 lore raw

Re: [PATCH v2] submodules: fix of regression on fetching of non-init subsub-repo

In reply to Junio C Hamano
Quoted 25 lines · Junio C Hamano
Quoted 20 lines · Peter Kaestle
t/t5526-fetch-submodules.sh +20 -0
+add_commit_push () {
+	dir="$1" &&
+	msg="$2" &&
+	shift 2 &&
+	git -C "$dir" add "$@" &&
+	git -C "$dir" commit -a -m "$msg" &&
+	git -C "$dir" push
+}
+
+compare_refs_in_dir () {
+	fail= &&
+	if test "x$1" = 'x!'
+	then
+		fail='!' &&
+		shift
+	fi &&
+	git -C "$1" rev-parse --verify "$2" >expect &&
+	git -C "$3" rev-parse --verify "$4" >actual &&
+	eval $fail test_cmp expect actual
+}
Quoted 10 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 781

Quoted 5 lines
t/t5526-fetch-submodules.sh +5 -0
+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' '
+	# depends on previous test for setup
+
+	git -C B/ fetch &&
+	compare_refs_in_dir A origin/master B origin/master

Can we do this without relying on the name of the default branch? Perhaps when outer, middle and inner are prepared, they can be forced to be on the 'sample' (not 'master' nor 'main') branch, or something like that?

Using origin/HEAD for compare_refs_in_dir should be fine without additional setup, as for the regression the "git -C B/ fetch" will fail and return with false (see description of the patch). This compare_refs_in_dir is just for additional checking as you proposed in the mail:
https://public-inbox.org/git/[email protected]/
------------8<-------------

Quoted 7 lines

And from B that was an original copy of A with only the top and middle layer instantiated, you run "git fetch". Are you happy as long as "git fetch" does not exit with non-zero status? That is hard to believe---it may be a necessary condition for the command to exit with zero status, but you have other expectations, like what commit the remote tracking branch refs/remotes/origin/HEAD ought to be pointing at. I think we should check that, too.

----------->8-------------

Quoted 11 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 789

Quoted 6 lines
t/t5526-fetch-submodules.sh +5 -0
+test_expect_success 'setup recursive fetch with uninit submodule' '
+	# does not depend on any previous test setups
+
+	git init main &&
+	git init sub &&

"super vs sub" would give us a better contrast than "main vs sub", and it would help reduce mistakes in the mechanical conversion of "master" to "main" happening in another topic.

ok.

Quoted 15 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 817

Quoted 9 lines
t/t5526-fetch-submodules.sh +9 -0
+	# In a regression the following git call will run into infinite recursion.
+	# To handle that, we connect the grep command to the git call by a pipe
+	# so that grep can kill the infinite recusion when detected.
+	# The recursion creates git output like:
+	# Fetching submodule sub
+	# Fetching submodule sub/sub              <-- [1]
+	# Fetching submodule sub/sub/sub
+	# ...
+	# [1] grep will trigger here and kill git by exiting and closing its stdin

"trigger here and kill..." -> "stop reading and cause git to eventually stop and die"

But we probably cannot use 'grep -m1' so it is a moot point.

Quoted 7 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 820

Quoted 3 lines
t/t5526-fetch-submodules.sh +3 -0
+
+	! git -C main fetch --recurse-submodules 2>&1 |
+		grep -v -m1 "Fetching submodule sub$" &&

Unfortunately, "grep -m<count>" is not even in POSIX, I would think.

What do we expect to happen in the correct case?

sigh, we can't use grep -m1. Too bad, it was such a nice solution.

Quoted 10 lines

- A line "Fetching submodule sub" and nothing else is given? That feels a bit brittle (how are we making sure, in the presence of "2>&1", that we will not get any other output, like progress?)

- "sub" is the only thing that appears on lines that begin with "Fetching submodule" (i.e. "Fetching submodule $something" where $something is not 'sub' is an error), and we allow other garbage in the output? That would be a bit more robust than the above.

Quoted 4 lines

As you seem to be comfortable using "sed" below, perhaps use it to extract the first few lines that say "^Fetching submodule " from the output and stop, and check that the output has only one such line about 'sub' and nothing else?

According to [1] posix sed offers equal possibility to quit like grep -m1 and I'll adopt:
$> yes posixgrepisnogoodforus | sed "/posix/q"

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html

Looking at the other mails I think I processed over all open comments and will prepare for v3. Thanks again.

Signature · 2 lines
kind regards
--peter;
#22 lore raw

PATCH v3 1/1 submodules: fix of regression on fetching of non-init subsub-repo

A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

The scenario in which it triggers is when one has a repository with a submodule inside a submodule like this:
superproject/middle_repo/inner_repo

Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.

Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.

Once person A pushed the changes and person B wants to fetch them using "git fetch" at the superproject level, B's git call will return with error saying:

Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo

Expectation is that in this case the inner submodule will be recognized as uninitialized submodule and skipped by the git fetch command.

This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.

Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.

This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.

CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
CC Philippe Blain <[email protected]>
---
 submodule.c                 |   7 ++-
 t/t5526-fetch-submodules.sh | 124 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 130 insertions(+), 1 deletion(-)
submodule.c +6 -1
index b3bb59f..b561445 100644
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,
 			strbuf_release(&submodule_prefix);
 			return 1;
 		} else {
+			struct strbuf empty_submodule_path = STRBUF_INIT;
 
 			fetch_task_release(task);
 			free(task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,
 			 * An empty directory is normal,
 			 * the submodule is not initialized
 			 */
+			strbuf_addf(&empty_submodule_path, "%s/%s/",
+							spf->r->worktree,
+							ce->name);
 			if (S_ISGITLINK(ce->ce_mode) &&
-			    !is_empty_dir(ce->name)) {
+			    !is_empty_dir(empty_submodule_path.buf)) {
 				spf->result = 1;
 				strbuf_addf(err,
 					    _("Could not access submodule '%s'\n"),
 					    ce->name);
 			}
+			strbuf_release(&empty_submodule_path);
 		}
 	}
 
t/t5526-fetch-submodules.sh +124 -0

128 lines not on this page · raw message

Signature · 1 line
2.6.2
#23 lore raw

Re: [PATCH v3] submodules: fix of regression on fetching of non-init subsub-repo

submodule.c

Quoted 64 lines · Peter Kaestle

A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

The scenario in which it triggers is when one has a repository with a submodule inside a submodule like this:
superproject/middle_repo/inner_repo

Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.

Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.

Once person A pushed the changes and person B wants to fetch them using "git fetch" at the superproject level, B's git call will return with error saying:

Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo

Expectation is that in this case the inner submodule will be recognized as uninitialized submodule and skipped by the git fetch command.

This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.

Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.

This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.

CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
CC Philippe Blain <[email protected]>
---
  submodule.c                 |   7 ++-
  t/t5526-fetch-submodules.sh | 124 ++++++++++++++++++++++++++++++++++++++++++++
  2 files changed, 130 insertions(+), 1 deletion(-)
submodule.c +0 -0
index b3bb59f..b561445 100644

[...]

These are my test run outputs:
final_fix [PATCH v3]
==========

# passed all 41 test(s)

recursion_regression [1b7ac4e6d4]
==========
not ok 41 - recursive fetch after deinit a submodule

no_fix [3a0b884c - origin/master]
==========
not ok 38 - fetching a superproject containing an uninitialized sub/sub project

Signature · 2 lines
kind regards
--peter;
#24 lore raw

Re: [PATCH v3] submodules: fix of regression on fetching of non-init subsub-repo

In reply to Peter Kaestle
Quoted 50 lines · Peter Kaestle

A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

The scenario in which it triggers is when one has a repository with a submodule inside a submodule like this:
superproject/middle_repo/inner_repo

Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.

Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.

Once person A pushed the changes and person B wants to fetch them using "git fetch" at the superproject level, B's git call will return with error saying:

Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo

Expectation is that in this case the inner submodule will be recognized as uninitialized submodule and skipped by the git fetch command.

This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.

Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.

This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.

Quoted 6 lines
CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
CC Philippe Blain <[email protected]>

---

Thanks, will replace.

#25 lore raw

Re: [PATCH v3] submodules: fix of regression on fetching of non-init subsub-repo

In reply to Peter Kaestle

Hi Peter,

Quoted 6 lines · Peter Kaestle

-- 8< --

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

I think this paragraph could be removed as it's saying the same thing as the one below.

Quoted 113 lines

The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.

CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
CC Philippe Blain <[email protected]>
---
 submodule.c                 |   7 ++-
 t/t5526-fetch-submodules.sh | 124 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 130 insertions(+), 1 deletion(-)
submodule.c +6 -1
index b3bb59f..b561445 100644
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,
                        strbuf_release(&submodule_prefix);
                        return 1;
                } else {
+                       struct strbuf empty_submodule_path = STRBUF_INIT;
 
                        fetch_task_release(task);
                        free(task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,
                         * An empty directory is normal,
                         * the submodule is not initialized
                         */
+                       strbuf_addf(&empty_submodule_path, "%s/%s/",
+                                                       spf->r->worktree,
+                                                       ce->name);
                        if (S_ISGITLINK(ce->ce_mode) &&
-                           !is_empty_dir(ce->name)) {
+                           !is_empty_dir(empty_submodule_path.buf)) {
                                spf->result = 1;
                                strbuf_addf(err,
                                            _("Could not access submodule '%s'\n"),
                                            ce->name);
                        }
+                       strbuf_release(&empty_submodule_path);
                }
        }
 
t/t5526-fetch-submodules.sh +53 -0
index dd8e423..495348a 100755
@@ -719,4 +719,128 @@ test_expect_success 'fetch new submodule commit intermittently referenced by sup
        )
 '
 
+add_commit_push () {
+       dir="$1" &&
+       msg="$2" &&
+       shift 2 &&
+       git -C "$dir" add "$@" &&
+       git -C "$dir" commit -a -m "$msg" &&
+       git -C "$dir" push
+}
+
+compare_refs_in_dir () {
+       fail= &&
+       if test "x$1" = 'x!'
+       then
+               fail='!' &&
+               shift
+       fi &&
+       git -C "$1" rev-parse --verify "$2" >expect &&
+       git -C "$3" rev-parse --verify "$4" >actual &&
+       eval $fail test_cmp expect actual
+}
+
+
+test_expect_success 'setup nested submodule fetch test' '
+       # does not depend on any previous test setups
+
+       for repo in outer middle inner
+       do
+               git init --bare $repo &&
+               git clone $repo ${repo}_content &&
+               echo "$repo" >"${repo}_content/file" &&
+               add_commit_push ${repo}_content "initial" file ||
+               return 1
+       done &&
+
+       git clone outer A &&
+       git -C A submodule add "$pwd/middle" &&
+       git -C A/middle/ submodule add "$pwd/inner" &&
+       add_commit_push A/middle/ "adding inner sub" .gitmodules inner &&
+       add_commit_push A/ "adding middle sub" .gitmodules middle &&
+
+       git clone outer B &&
+       git -C B/ submodule update --init middle &&
+
+       compare_refs_in_dir A HEAD B HEAD &&
+       compare_refs_in_dir A/middle HEAD B/middle HEAD &&
+       test_path_is_file B/file &&
+       test_path_is_file B/middle/file &&
+       test_path_is_missing B/middle/inner/file &&
+
+       echo "change on inner repo of A" >"A/middle/inner/file" &&
+       add_commit_push A/middle/inner "change on inner" file &&
+       add_commit_push A/middle "change on inner" inner &&
+       add_commit_push A "change on inner" middle

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 784

Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+'
+
+test_expect_success 'fetching a superproject containing an uninitialized sub/sub project' '
+       # depends on previous test for setup
+
+       git -C B/ fetch &&
+       compare_refs_in_dir A origin/HEAD B origin/HEAD
+'
+
+fetch_with_recusion_abort () {

s/recusion/recursion/

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 787

Quoted 3 lines
t/t5526-fetch-submodules.sh +3 -0
+       # In a regression the following git call will run into infinite recursion.
+       # To handle that, we connect the sed command to the git call by a pipe
+       # so that sed can kill the infinite recusion when detected.

s/recusion/recursion/

Quoted 26 lines
t/t5526-fetch-submodules.sh +26 -0
+       # The recursion creates git output like:
+       # Fetching submodule sub
+       # Fetching submodule sub/sub              <-- [1]
+       # Fetching submodule sub/sub/sub
+       # ...
+       # [1] sed will stop reading and cause git to eventually stop and die
+
+       git -C "$1" fetch --recurse-submodules 2>&1 |
+               sed "/Fetching submodule $2[^$]/q" >out &&
+       ! grep "Fetching submodule $2[^$]" out
+}
+
+test_expect_success 'setup recursive fetch with uninit submodule' '
+       # does not depend on any previous test setups
+
+       # setup a remote superproject to make git fetch work with an uninit submodule
+       git init --bare super_bare &&
+       git clone super_bare super &&
+       git init sub &&
+
+       >sub/file &&
+       git -C sub add file &&
+       git -C sub commit -m "add file" &&
+       git -C sub rev-parse HEAD >expect &&
+
+       # adding submodule without cloning

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 823

Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+       echo "[submodule \"sub\"]" >super/.gitmodules &&
+       echo "path = sub" >>super/.gitmodules &&
+       echo "url = ../sub" >>super/.gitmodules &&
+       git -C super update-index --add --cacheinfo 160000 $(cat expect) sub &&
+       mkdir super/sub &&
+
+       git -C super submodule status >out &&
+       sed -e "s/^-//" -e "s/ sub.*$//" out >actual &&
+       test_cmp expect actual
+'

I think this is overly complicated, what I was hinting at was adding the submodule
in the superproject before cloning it, so something along these lines:

test_create_repo super &&
test_commit -C super initial &&
test_create_repo sub &&
test_commit -C sub initial &&
git -C sub rev-parse HEAD >expect &&

git -C super submodule add ../sub &&
git -C super commit -m "add sub" &&

git clone super superclone &&
git -C superclone submodule status >out &&
sed -e "s/^-//" -e "s/ sub.*$//" out >actual &&
test_cmp expect actual

And then running the two tests below in "superclone".

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 828

Quoted 5 lines
t/t5526-fetch-submodules.sh +5 -0
+
+test_expect_success 'recursive fetch with uninit submodule' '
+       # depends on previous test for setup
+
+       fetch_with_recusion_abort super sub &&

s/recusion/recursion/

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 840

Quoted 12 lines
t/t5526-fetch-submodules.sh +12 -0
+       git -C super submodule status >out &&
+       sed -e "s/^-//" -e "s/ sub$//" out >actual &&
+       test_cmp expect actual
+'
+
+test_expect_success 'recursive fetch after deinit a submodule' '
+       # depends on previous test for setup
+
+       git -C super submodule update --init sub &&
+       git -C super submodule deinit -f sub &&
+
+       fetch_with_recusion_abort super sub &&

s/recusion/recursion/

Quoted 9 lines
t/t5526-fetch-submodules.sh +5 -1
+       git -C super submodule status >out &&
+       sed -e "s/^-//" -e "s/ sub$//" out >actual &&
+       test_cmp expect actual
+'
+
 test_done
--

2.6.2

Philippe.

#26 lore raw

Re: [PATCH v3] submodules: fix of regression on fetching of non-init subsub-repo

Hi Philippe,

when sending the patch yesterday, I had the gut feeling to hold it back to double check it a day later. Should have done so, some of your findings were really stupid mistakes.

Quoted 11 lines · Philippe Blain
Quoted 6 lines · Peter Kaestle

-- 8< --

Furthermore a regression test case is added, which tests for recursive fetches on a superproject with uninitialized sub repositories. This issue was leading to an infinite loop when doing a revert of a62387b.

I think this paragraph could be removed as it's saying the same thing as the one below.

jip.

[...]

Quoted 4 lines
Quoted 1 line

+fetch_with_recusion_abort () {

s/recusion/recursion/

yes.

[...]

Quoted 38 lines
Quoted 14 lines
t/t5526-fetch-submodules.sh +14 -0
+test_expect_success 'setup recursive fetch with uninit submodule' '
+       # does not depend on any previous test setups
+
+       # setup a remote superproject to make git fetch work with an uninit submodule
+       git init --bare super_bare &&
+       git clone super_bare super &&
+       git init sub &&
+
+       >sub/file &&
+       git -C sub add file &&
+       git -C sub commit -m "add file" &&
+       git -C sub rev-parse HEAD >expect &&
+
+       # adding submodule without cloning

t/t5526-fetch-submodules.sh test_expect_success 'fetch new submodule commit intermittently referenced by sup · line 823

Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+       echo "[submodule \"sub\"]" >super/.gitmodules &&
+       echo "path = sub" >>super/.gitmodules &&
+       echo "url = ../sub" >>super/.gitmodules &&
+       git -C super update-index --add --cacheinfo 160000 $(cat expect) sub &&
+       mkdir super/sub &&
+
+       git -C super submodule status >out &&
+       sed -e "s/^-//" -e "s/ sub.*$//" out >actual &&
+       test_cmp expect actual
+'

I think this is overly complicated, what I was hinting at was adding the submodule
in the superproject before cloning it, so something along these lines:

test_create_repo super &&
test_commit -C super initial &&
test_create_repo sub &&
test_commit -C sub initial &&
git -C sub rev-parse HEAD >expect &&

git -C super submodule add ../sub &&
git -C super commit -m "add sub" &&

Quoted 6 lines

git clone super superclone &&
git -C superclone submodule status >out &&
sed -e "s/^-//" -e "s/ sub.*$//" out >actual &&
test_cmp expect actual

And then running the two tests below in "superclone".

Indeed, looks much simpler, I gave it a try run and it resulted in same behavior, will double check the logic of it and then send v4. Thanks a lot.

[...]

Signature · 2 lines
kind regards
--peter;
#27 lore raw

PATCH v4 1/1 submodules: fix of regression on fetching of non-init subsub-repo

A regression has been introduced by a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28).

The scenario in which it triggers is when one has a repository with a submodule inside a submodule like this:
superproject/middle_repo/inner_repo

Person A and B have both a clone of it, while Person B is not working with the inner_repo and thus does not have it initialized in his working copy.

Now person A introduces a change to the inner_repo and propagates it through the middle_repo and the superproject.

Once person A pushed the changes and person B wants to fetch them using "git fetch" at the superproject level, B's git call will return with error saying:

Could not access submodule 'inner_repo'
Errors during submodule fetch:
middle_repo

Expectation is that in this case the inner submodule will be recognized as uninitialized submodule and skipped by the git fetch command.

This used to work correctly before 'a62387b (submodule.c: fetch in submodules git directory instead of in worktree, 2018-11-28)'.

Starting with a62387b the code wants to evaluate "is_empty_dir()" inside .git/modules for a directory only existing in the worktree, delivering then of course wrong return value.

This patch ensures is_empty_dir() is getting the correct path of the uninitialized submodule by concatenation of the actual worktree and the name of the uninitialized submodule.

The first attempt to fix this regression, in 1b7ac4e6d4 (submodules:
fix of regression on fetching of non-init subsub-repo, 2020-11-12), by simply reverting a62387b, resulted in an infinite loop of submodule fetches in the simpler case of a recursive fetch of a superproject with uninitialized submodules, and so this commit was reverted in 7091499bc0 (Revert "submodules: fix of regression on fetching of non-init subsub-repo", 2020-12-02).
To prevent future breakages, also add a regression test for this scenario.

CC Junio C Hamano <[email protected]>
CC Philippe Blain <[email protected]>
CC Ralf Thielow <[email protected]>
CC Eric Sunshine <[email protected]>
---
 submodule.c                 |   7 ++-
 t/t5526-fetch-submodules.sh | 117 ++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+), 1 deletion(-)
submodule.c +6 -1
index b3bb59f066..b561445329 100644
@@ -1477,6 +1477,7 @@ static int get_next_submodule(struct child_process *cp,
 			strbuf_release(&submodule_prefix);
 			return 1;
 		} else {
+			struct strbuf empty_submodule_path = STRBUF_INIT;
 
 			fetch_task_release(task);
 			free(task);
@@ -1485,13 +1486,17 @@ static int get_next_submodule(struct child_process *cp,
 			 * An empty directory is normal,
 			 * the submodule is not initialized
 			 */
+			strbuf_addf(&empty_submodule_path, "%s/%s/",
+							spf->r->worktree,
+							ce->name);
 			if (S_ISGITLINK(ce->ce_mode) &&
-			    !is_empty_dir(ce->name)) {
+			    !is_empty_dir(empty_submodule_path.buf)) {
 				spf->result = 1;
 				strbuf_addf(err,
 					    _("Could not access submodule '%s'\n"),
 					    ce->name);
 			}
+			strbuf_release(&empty_submodule_path);
 		}
 	}
 
t/t5526-fetch-submodules.sh +117 -0

121 lines not on this page · raw message

Signature · 1 line
2.29.2.457.gf179bca
Reviewed-by on v4 1/1
#28 lore raw

Re: [PATCH v4] submodules: fix of regression on fetching of non-init subsub-repo

Quoted 3 lines · Peter Kaestle

---8<---

Thanks again for working on this.

Philippe.
P.S. I wouldn't call anything you wrote in v3 "stupid mistakes" :) we all have a lot on our minds
- especially these days!

#29 lore raw
In reply to Philippe Blain

Hi Philippe,

Am Mi., 2. Dez. 2020 um 18:20 Uhr schrieb Philippe Blain <[email protected]>:

Quoted 7 lines

Ok, you don't add '--init' but the submodule gets checked out, so it looks like you have 'submodule.active' set to a pathspec that matches 'sha1collisiondetection'. Did you clone the git repo with '--recurse-submodules', which would add '.' as the value of 'submodule.active' ? Or maybe you manually configured that value in your global gitconfig ?

Sry, I did not mention that I started with the submodule already initialized and checked out. IIRC I deinitialized the submodule with 'git submodule deinit -f sha1collisiondetection/' and ran 'git fetch' to see if it happens on the git repo, too, since I encountered the issue on another repo first.

I can see the same misbehaviour when I run

$ git -c submodule.recurse=false fetch --recurse-submodules
too, so I think it doesn't have necessarily something to do with
the 'submodule.recursive' config, but with 'recurse submodules'
in general.

Ralf

#30 lore raw

Hi Ralf,

Quoted 16 lines · Ralf Thielow

Am Mi., 2. Dez. 2020 um 18:20 Uhr schrieb Philippe Blain <[email protected]>:

Quoted 7 lines

Ok, you don't add '--init' but the submodule gets checked out, so it looks like you have 'submodule.active' set to a pathspec that matches 'sha1collisiondetection'. Did you clone the git repo with '--recurse-submodules', which would add '.' as the value of 'submodule.active' ? Or maybe you manually configured that value in your global gitconfig ?

Sry, I did not mention that I started with the submodule already initialized and checked out. IIRC I deinitialized the submodule with 'git submodule deinit -f sha1collisiondetection/' and ran 'git fetch' to see if it happens on the git repo, too, since I encountered the issue on another repo first.

Quoted 5 lines

I can see the same misbehaviour when I run

$ git -c submodule.recurse=false fetch --recurse-submodules
too, so I think it doesn't have necessarily something to do with
the 'submodule.recursive' config, but with 'recurse submodules'
in general.

This is interesting. There are several testcases using "--recurse-submodules". Have you tried, whether any of these trigger?

Signature · 2 lines
kind regards
--peter;
#31 lore raw

Hi Peter,

Am Do., 3. Dez. 2020 um 09:20 Uhr schrieb Peter Kästle <[email protected]>:

Quoted 8 lines
Quoted 5 lines

I can see the same misbehaviour when I run

$ git -c submodule.recurse=false fetch --recurse-submodules
too, so I think it doesn't have necessarily something to do with
the 'submodule.recursive' config, but with 'recurse submodules'
in general.

This is interesting. There are several testcases using "--recurse-submodules". Have you tried, whether any of these trigger?

The test suite passes.

Ralf

#32 lore raw

Hi Ralf,

Quoted 9 lines · Ralf Thielow

Hi Peter,

Am Do., 3. Dez. 2020 um 09:20 Uhr schrieb Peter Kästle <[email protected]>:

Quoted 5 lines
Quoted 5 lines

I can see the same misbehaviour when I run

$ git -c submodule.recurse=false fetch --recurse-submodules
too, so I think it doesn't have necessarily something to do with
the 'submodule.recursive' config, but with 'recurse submodules'
in general.
Quoted 5 lines
Quoted 3 lines

This is interesting. There are several testcases using "--recurse-submodules". Have you tried, whether any of these trigger?

The test suite passes.

This is what I experienced. Thus it is important to get your case into the test suite. Could you please send us a command sequence how to create your repository structure from scratch, so that a test case can be created? - Of course, if you want, you can also create a test case on your own.

Signature · 2 lines
kind regards
--peter;
#33 lore raw

Hi Peter,

Am Do., 3. Dez. 2020 um 10:43 Uhr schrieb Peter Kästle <[email protected]>:

Quoted 9 lines
Quoted 2 lines

The test suite passes.

This is what I experienced. Thus it is important to get your case into the test suite. Could you please send us a command sequence how to create your repository structure from scratch, so that a test case can be created? - Of course, if you want, you can also create a test case on your own.

It can be reproduced with the following sequence of commands:

git init sub
cd sub
touch file
git add file
git commit -m "add file"
cd ..
git init main
cd main
git submodule add ../sub
git submodule init
git submodule update --checkout
git submodule deinit -f sub/
git fetch --recurse-submodules

Ralf

#34 lore raw

Hi Ralf,

Quoted 15 lines · Ralf Thielow

It can be reproduced with the following sequence of commands:

git init sub
cd sub
touch file
git add file
git commit -m "add file"
cd ..
git init main
cd main
git submodule add ../sub
git submodule init
git submodule update --checkout
git submodule deinit -f sub/
git fetch --recurse-submodules

With git from master state the "git fetch --recurse-submodules" results in an infinite recurse call.

I translated your sequence into a bash script, which can then be easily converted into a test case for git. Problematic was the infinite recurse loop of the git fetch command, which I solved by grep'ing for the second recursion output and abort using -m1. Could you please confirm, that you see "passed" for the good git versions and "failed" for the bad ones?

#!/bin/bash

testcase () {
        rm -Rf main sub &&

        git init main &&
        git init sub &&

        touch sub/file &&
        git -C sub add file &&
        git -C sub commit -m "add file" &&
        git -C sub rev-parse HEAD >expect &&

        git -C main submodule add ../sub &&
        git -C main submodule init &&
        git -C main submodule update --checkout &&

        git -C main submodule deinit -f sub &&
        ! git -C main fetch --recurse-submodules |&
                grep -v -m1 "Fetching submodule sub$" &&
        git -C main submodule status |
                sed -e "s/^-//" -e "s/ sub$//" >actual &&
        cmp expect actual
}

if testcase
then
echo "passed"
else
echo "failed"
fi

Signature · 1 line
--peter;
#35 lore raw

Hi Peter,

Am Do., 3. Dez. 2020 um 16:10 Uhr schrieb Peter Kästle <[email protected]>:

Quoted 9 lines

I translated your sequence into a bash script, which can then be easily converted into a test case for git. Problematic was the infinite recurse loop of the git fetch command, which I solved by grep'ing for the second recursion output and abort using -m1.
Could you please confirm, that you see "passed" for the good git versions and "failed" for the bad ones?

I can confirm that. It "failed" on current master version 2.29.2.435.g72ffeb997e
and "passed" on the latest tag 2.29.2.

Thanks,
Ralf