clone: shallow-submodules should be single-branch by default

PATCH In review Bruce Perry via GitGitGadget 3 participants 7 messages Jul 2, 2024 – Jul 25, 2024 Documentation

Share & tools

Bruce Perry via GitGitGadget

author

2024-07-02 19:07 UTC #0 lore raw

cover clone: shallow-submodules should be single-branch by default

I noticed a couple places where the behavior of recursive clones for shallow submodules does not match what is implied by the documentation. Shallow submodules should be, but aren't, single branch by default. It would also be useful to allow users to override the shallow specification in gitmodules on the command line for clones as they can for submodule update. The modification here for the former is a bit ugly, but hopefully at least gets the point across to start a discussion. First time submitting a patch here, hopefully I'm getting the process right.

Bruce Perry (2):
clone: shallow-submodules should be single-branch by default
clone: no-shallow-submodules clone overrides option in gitmodules

 Documentation/git-clone.txt         |  3 ++
 Documentation/gitmodules.txt        |  4 +--
 builtin/clone.c                     | 10 ++++--
 t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
 4 files changed, 53 insertions(+), 16 deletions(-)

base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1740%2Fbaperry2%2Fsubmods-clone-bug-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1740/baperry2/submods-clone-bug-v1
Pull-Request: https://github.com/git/git/pull/1740

Signature · 1 line

gitgitgadget

Bruce Perry via GitGitGadget

author

2024-07-02 19:07 UTC #1 lore raw

PATCH 1/2 clone: shallow-submodules should be single-branch by default

From: Bruce Perry <[email protected]>

Recursive cloning with with the --shallow-submodules command line option (or the shallow = true option in .gitmodules) and no specification of --[no-]single-branch does result in a shallow submodule clone, but obtains all available branches. This behavior is unexpected because in other circumstances shallow clones imply single-branch clones. It also violates the documented behavior: the documentation states that performing a clone with --recurse-submodules is equivalent to doing a nonrecursive clone then immediately running git submodule update --init --recursive. However, with the shallow = true option in .gitmodules, the former results in a shallow but not single-branch clone of the submodules, while the latter results submodules that are both shallow and single-branch.

Modify the logic for git clone with --recurse-submodules so that if no option is specified for --[no-]single-branch, then no --[no-]single-branch option is passed to the internal call to git submodule update --init --recursive. As a result, the default will be used, which is to make shallow clones also single-branch and non-shallow clones multi-branch.

Modify the tests for shallow-submodules so that the submodule has multiple branches and the expected behavior in terms of obtaining branches is validated: shallow submodules must also be single-branch, while non-shallow submodules must obtain both branches.

Make a slight clarification to the documentation regarding the above behavior, although primarily the effect is to bring the behavior in line with the present documentation.

Signed-off-by Bruce Perry <[email protected]>

---
 Documentation/git-clone.txt         |  3 +++
 Documentation/gitmodules.txt        |  4 ++--
 builtin/clone.c                     |  6 +++--
 t/t5614-clone-submodules-shallow.sh | 36 ++++++++++++++++++++---------
 4 files changed, 34 insertions(+), 15 deletions(-)

Documentation/git-clone.txt +3 -0

index 5de18de2ab8..69248421390 100644
@@ -297,6 +297,9 @@ or `--mirror` is given)
 
 `--`[`no-`]`shallow-submodules`::
 	All submodules which are cloned will be shallow with a depth of 1.
+	Also implies `--single-branch` for the submodule clones
+	unless `--no-single-branch` is passed, in which case histories near
+	the tips of all branches of the cloned submodules are fetched.
 
 `--`[`no-`]`remote-submodules`::
 	All submodules which are cloned will use the status of the submodule's

Documentation/gitmodules.txt +2 -2

index d9bec8b1875..c8160cd1df7 100644
@@ -95,8 +95,8 @@ affected by this setting.
 
 submodule.<name>.shallow::
 	When set to true, a clone of this submodule will be performed as a
-	shallow clone (with a history depth of 1) unless the user explicitly
-	asks for a non-shallow clone.
+	shallow clone (single branch with a history depth of 1) unless the
+	user explicitly asks for a non-shallow clone.
 
 NOTES
 -----

builtin/clone.c +4 -2

index b28f88eb43d..50ccce8902d 100644
@@ -58,6 +58,7 @@ static const char * const builtin_clone_usage[] = {
 };
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
+static int option_single_branch_submodules;
 static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_no_tags;
 static int option_shallow_submodules;

@@ -816,8 +817,8 @@ static int checkout(int submodule_progress, int filter_submodules)
 			strvec_pushf(&cmd.args, "--filter=%s",
 				     expand_list_objects_filter_spec(&filter_options));
 
-		if (option_single_branch >= 0)
-			strvec_push(&cmd.args, option_single_branch ?
+		if (option_single_branch_submodules >= 0)
+			strvec_push(&cmd.args, option_single_branch_submodules ?
 					       "--single-branch" :
 					       "--no-single-branch");

@@ -997,6 +998,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 
 	if (option_depth || option_since || option_not.nr)
 		deepen = 1;
+	option_single_branch_submodules = option_single_branch;
 	if (option_single_branch == -1)
 		option_single_branch = deepen ? 1 : 0;

t/t5614-clone-submodules-shallow.sh +25 ~~-11~~

index c2a2bb453ee..b23c7d085aa 100755
@@ -14,10 +14,14 @@ test_expect_success 'setup' '
 	mkdir sub &&
 	(
 		cd sub &&
-		git init &&
+		git init -b main &&
 		test_commit subcommit1 &&
 		test_commit subcommit2 &&
-		test_commit subcommit3
+		test_commit subcommit3 &&
+		git checkout -b branch &&
+		test_commit branchcommit1 &&
+		test_commit branchcommit2 &&
+		git checkout main
 	) &&
 	git submodule add "file://$pwd/sub" sub &&
 	git commit -m "add submodule"

@@ -30,16 +34,18 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 3 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'shallow clone with shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	test_config_global protocol.file.allow always &&
 	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
-	git -C super_clone log --oneline >lines &&
+	git -C super_clone log --oneline --all >lines &&
 	test_line_count = 2 lines &&
-	git -C super_clone/sub log --oneline >lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
 	test_line_count = 1 lines
 '

@@ -50,7 +56,9 @@ test_expect_success 'shallow clone does not imply shallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 2 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines  &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'shallow clone with non shallow submodule' '

@@ -60,7 +68,9 @@ test_expect_success 'shallow clone with non shallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 2 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines  &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'non shallow clone with shallow submodule' '

@@ -69,7 +79,7 @@ test_expect_success 'non shallow clone with shallow submodule' '
 	git clone --recurse-submodules --no-local --shallow-submodules "file://$pwd/." super_clone &&
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 3 lines &&
-	git -C super_clone/sub log --oneline >lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
 	test_line_count = 1 lines
 '

@@ -87,7 +97,7 @@ test_expect_success 'clone follows shallow recommendation' '
 	) &&
 	(
 		cd super_clone/sub &&
-		git log --oneline >lines &&
+		git log --oneline --all >lines &&
 		test_line_count = 1 lines
 	)
 '

@@ -105,7 +115,9 @@ test_expect_success 'get unshallow recommended shallow submodule' '
 	(
 		cd super_clone/sub &&
 		git log --oneline >lines &&
-		test_line_count = 3 lines
+		test_line_count = 3 lines &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
 	)
 '

@@ -124,7 +136,9 @@ test_expect_success 'clone follows non shallow recommendation' '
 	(
 		cd super_clone/sub &&
 		git log --oneline >lines &&
-		test_line_count = 3 lines
+		test_line_count = 3 lines &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
 	)
 '

Signature · 1 line

gitgitgadget

Bruce Perry via GitGitGadget

author

2024-07-02 19:07 UTC #2 lore raw

PATCH 2/2 clone: no-shallow-submodules clone overrides option in gitmodules

In reply to Bruce Perry via GitGitGadget

From: Bruce Perry <[email protected]>

When the --no-shallow-submodules option is used for cloning, pass the --no-recommend-shallow option to the internal submodule update command, which will override an option for shallow in .gitmodules if present. This allows the user to force a non-shallow clone of the submodules if desired. It brings the behavior in line with the present documentation, which states the option in gitmodules is used unless explitcitly overridden by the user. Add a test to confirm that the override is done when using the clone command (similar to a current test that validates similar behavior for the submodule update command).

Signed-off-by Bruce Perry <[email protected]>

---
 builtin/clone.c                     |  4 +++-
 t/t5614-clone-submodules-shallow.sh | 16 ++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

builtin/clone.c +3 -1

index 50ccce8902d..63cfb48484c 100644
@@ -61,7 +61,7 @@ static int option_no_checkout, option_bare, option_mirror, option_single_branch
 static int option_single_branch_submodules;
 static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_no_tags;
-static int option_shallow_submodules;
+static int option_shallow_submodules = -1;
 static int option_reject_shallow = -1;    /* unspecified */
 static int config_reject_shallow = -1;    /* unspecified */
 static int deepen;

@@ -798,6 +798,8 @@ static int checkout(int submodule_progress, int filter_submodules)
 
 		if (option_shallow_submodules == 1)
 			strvec_push(&cmd.args, "--depth=1");
+		else if (option_shallow_submodules == 0)
+			strvec_push(&cmd.args, "--no-recommend-shallow");
 
 		if (max_jobs != -1)
 			strvec_pushf(&cmd.args, "--jobs=%d", max_jobs);

t/t5614-clone-submodules-shallow.sh +16 -0

index b23c7d085aa..8e2965248a2 100755
@@ -102,6 +102,22 @@ test_expect_success 'clone follows shallow recommendation' '
 	)
 '
 
+test_expect_success 'no-shallow-submodules clone option overrides gitmodules' '
+	test_when_finished "rm -rf super_clone" &&
+	test_config_global protocol.file.allow always &&
+	git clone --recurse-submodules --no-shallow-submodules --no-local "file://$pwd/." super_clone &&
+	(
+		cd super_clone &&
+		git log --oneline >lines &&
+		test_line_count = 4 lines
+	) &&
+	(
+		cd super_clone/sub &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
+	)
+'
+
 test_expect_success 'get unshallow recommended shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	test_config_global protocol.file.allow always &&

Signature · 1 line

gitgitgadget

[email protected]

2024-07-02 19:12 UTC #3 lore raw

In reply to Bruce Perry via GitGitGadget

Quoted 19 lines · Bruce Perry

I noticed a couple places where the behavior of recursive clones for shallow submodules does not match what is implied by the documentation. Shallow submodules should be, but aren't, single branch by default. It would also be useful to allow users to override the shallow specification in gitmodules on the command line for clones as they can for submodule update. The modification here for the former is a bit ugly, but hopefully at least gets the point across to start a discussion. First time submitting a patch here, hopefully I'm getting the process right.
Bruce Perry (2):
clone: shallow-submodules should be single-branch by default
clone: no-shallow-submodules clone overrides option in gitmodules
Documentation/git-clone.txt         |  3 ++
Documentation/gitmodules.txt        |  4 +--
builtin/clone.c                     | 10 ++++--
t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
4 files changed, 53 insertions(+), 16 deletions(-)

Quoted 6 lines

base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-
1740%2Fbaperry2%2Fsubmods-clone-bug-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git- 1740/baperry2/submods-clone-bug-v1
Pull-Request: https://github.com/git/git/pull/1740

I am concerned about this one. Many CI systems (including Jenkins GitSCM) assume a detached head for submodule clone/checkout. Adding a branch to the mix will change the expected semantics. Am I missing something? --Randall

Bruce Perry

2024-07-03 05:00 UTC #4 lore raw

Randall,

Perhaps I was using imprecise terminology. This change should not impact whether submodule clones land in a detached head state, so it should not impact anything that assumes submodule clones are detached head.

The change being made is this: "git clone --recurse-submodules --shallow-submodules" currently gives you a submodule with a detached head at the desired state, but also downloads data for the tips of all branches in the remote being cloned (potentially a lot of unneeded data as in my use case). The modification means the same command would give you a detached head at the desired state, plus the tip of only the default branch in the remote. The modified behavior matches the current behavior for a simple "git clone" followed by "git submodule update --init --recurse--submodules --depth=1".

Thanks,
Bruce

(Resent due to a formatting failure)

Quoted 31 lines · On Tue, Jul 2, 2024, 1:12 PM <[email protected]> wrote:

Quoted 19 lines · Bruce Perry
I noticed a couple places where the behavior of recursive clones for shallow submodules does not match what is implied by the documentation. Shallow submodules should be, but aren't, single branch by default. It would also be useful to allow users to override the shallow specification in gitmodules on the command line for clones as they can for submodule update. The modification here for the former is a bit ugly, but hopefully at least gets the point across to start a discussion. First time submitting a patch here, hopefully I'm getting the process right.
Bruce Perry (2):
clone: shallow-submodules should be single-branch by default
clone: no-shallow-submodules clone overrides option in gitmodules
Documentation/git-clone.txt         |  3 ++
Documentation/gitmodules.txt        |  4 +--
builtin/clone.c                     | 10 ++++--
t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
4 files changed, 53 insertions(+), 16 deletions(-)
Quoted 6 lines
base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-
1740%2Fbaperry2%2Fsubmods-clone-bug-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git- 1740/baperry2/submods-clone-bug-v1
Pull-Request: https://github.com/git/git/pull/1740
I am concerned about this one. Many CI systems (including Jenkins GitSCM) assume a detached head for submodule clone/checkout. Adding a branch to the mix will change the expected semantics. Am I missing something? --Randall

[email protected]

2024-07-03 14:05 UTC #5 lore raw

Quoted 50 lines · Bruce Perry

Perhaps I was using imprecise terminology. This change should not impact whether submodule clones land in a detached head state, so it should not impact anything that assumes submodule clones are detached head.
The change being made is this: "git clone --recurse-submodules --shallow- submodules" currently gives you a submodule with a detached head at the desired state, but also downloads data for the tips of all branches in the remote being cloned (potentially a lot of unneeded data as in my use case). The modification means the same command would give you a detached head at the desired state, plus the tip of only the default branch in the remote. The modified behavior matches the current behavior for a simple "git clone" followed by "git submodule update --init --recurse--submodules --depth=1".
Thanks,
Bruce
(Resent due to a formatting failure)
Quoted 8 lines · On Tue, Jul 2, 2024, 1:12 PM <[email protected]> wrote:
Quoted 6 lines · Bruce Perry
I noticed a couple places where the behavior of recursive clones for shallow submodules does not match what is implied by the documentation. Shallow submodules should be, but aren't, single branch by default. It would also be useful to allow users to override the shallow specification in gitmodules on the command line for clones as they can for submodule update. The modification here for the former is
a bit ugly, but hopefully at least gets the point across to start a discussion.
Quoted 22 lines
Quoted 21 lines
First time submitting a patch here, hopefully I'm getting the process right.
Bruce Perry (2):
clone: shallow-submodules should be single-branch by default
clone: no-shallow-submodules clone overrides option in gitmodules
Documentation/git-clone.txt         |  3 ++
Documentation/gitmodules.txt        |  4 +--
builtin/clone.c                     | 10 ++++--
t/t5614-clone-submodules-shallow.sh | 52
+++++++++++++++++++++++------
4 files changed, 53 insertions(+), 16 deletions(-)
base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1 Published-As:
https://github.com/gitgitgadget/git/releases/tag/pr-git-
1740%2Fbaperry2%2Fsubmods-clone-bug-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git- 1740/baperry2/submods-clone-bug-v1
Pull-Request: https://github.com/git/git/pull/1740

Quoted 3 lines

Quoted 1 line
I am concerned about this one. Many CI systems (including Jenkins GitSCM)
assume a detached head for submodule clone/checkout. Adding a branch to the mix will change the expected semantics. Am I missing something?

Thanks for clearing that up. Fine with me.
Regards
Randall

Bruce Perry

2024-07-25 17:10 UTC #6 lore raw

Any further comments on this or changes I should make so it can be merged?

The current code behavior does not match the documentation, so something should be changed. If these changes are not desired, I could submit an alternative patch that changes the documentation to clarify what currently happens instead.

Quoted 60 lines · On Wed, Jul 3, 2024 at 8:05 AM <[email protected]> wrote:

Quoted 50 lines · Bruce Perry
Perhaps I was using imprecise terminology. This change should not impact whether submodule clones land in a detached head state, so it should not impact anything that assumes submodule clones are detached head.
The change being made is this: "git clone --recurse-submodules --shallow- submodules" currently gives you a submodule with a detached head at the desired state, but also downloads data for the tips of all branches in the remote being cloned (potentially a lot of unneeded data as in my use case). The modification means the same command would give you a detached head at the desired state, plus the tip of only the default branch in the remote. The modified behavior matches the current behavior for a simple "git clone" followed by "git submodule update --init --recurse--submodules --depth=1".
Thanks,
Bruce
(Resent due to a formatting failure)
Quoted 8 lines · On Tue, Jul 2, 2024, 1:12 PM <[email protected]> wrote:
Quoted 6 lines · Bruce Perry
I noticed a couple places where the behavior of recursive clones for shallow submodules does not match what is implied by the documentation. Shallow submodules should be, but aren't, single branch by default. It would also be useful to allow users to override the shallow specification in gitmodules on the command line for clones as they can for submodule update. The modification here for the former is
a bit ugly, but hopefully at least gets the point across to start a discussion.
Quoted 22 lines
Quoted 21 lines
First time submitting a patch here, hopefully I'm getting the process right.
Bruce Perry (2):
clone: shallow-submodules should be single-branch by default
clone: no-shallow-submodules clone overrides option in gitmodules
Documentation/git-clone.txt         |  3 ++
Documentation/gitmodules.txt        |  4 +--
builtin/clone.c                     | 10 ++++--
t/t5614-clone-submodules-shallow.sh | 52
+++++++++++++++++++++++------
4 files changed, 53 insertions(+), 16 deletions(-)
base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1 Published-As:
https://github.com/gitgitgadget/git/releases/tag/pr-git-
1740%2Fbaperry2%2Fsubmods-clone-bug-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git- 1740/baperry2/submods-clone-bug-v1
Pull-Request: https://github.com/git/git/pull/1740
Quoted 3 lines
Quoted 1 line
I am concerned about this one. Many CI systems (including Jenkins GitSCM)
assume a detached head for submodule clone/checkout. Adding a branch to the mix will change the expected semantics. Am I missing something?
Thanks for clearing that up. Fine with me.
Regards
Randall