''git submodule sync'' should not add uninitialized submodules to .git/config

When I initialize 2/3 submodules of my git repository and do git submodule update, all is fine: Only the 2 submodules that I need are updated.

When I run a git submodule sync to update the URLs that may have been changed in .gitmodules, it ADDS the URL of the submodule that was NOT initialized, thus "initializing" it.

Now, when I run git submodule update, it starts checking out the third module and my workflow is broken.

git submodule sync should not add entries to .git/config, only update existing ones.

Maarten,

Is it enough to similarly limit 'submodule sync' the same way you did with 'submodule init'? Like this:
git submodule sync -- A B

From 'git help submodule':
"git submodule sync" synchronizes all submodules while "git submodule sync — A" synchronizes submodule "A" only.

Phil

See 33f072f (submodule sync: Update "submodule.<name>.url" for empty directories, 2010-10-08), which introduced this behaviour.

cmd_update considers anything that has submodule.<name>.url defined as "the user is interested", so I suspect "git submodule sync" should not do this.

The situation 33f072f cites as needing this behaviour can easily fixed by running 'submodule sync' after switching to the branch to which the submodule _matters_, no?

Jens, what do you think?

Am 23.06.2011 16:35, schrieb Junio C Hamano:

I agree that 33f072f introduced a regression. One could argue if it was a good idea to let "git submodule init" not do the clone itself but defer it to "git submodule update" by setting the url in .git/config, but that's the way things are done now (and maybe there was a very good reason to do it that way I'm not aware of, because I didn't follow the list that closely back then).

So while I think 33f072f solved a problem for a valid use case, it breaks other use cases that worked so far. So unless we want to change init to do the clone itself (which would be a pretty invasive change in behavior), I'd vote for a revert.

Actually, shouldn't the fix be more like this patch, which is directly on top of 33f072f? I think this is more in line with what the end user wants to tell the system with "submodule init", namely "I am interested in this submodule".

-- >8 --
Subject: submodule sync: do not auto-vivify uninteresting submodule

Earlier 33f072f (submodule sync: Update "submodule.<name>.url" for empty directories, 2010-10-08) attempted to fix a bug where "git submodule sync" command does not update the URL if the current superproject does not have a checkout of the submodule.

However, it did so by unconditionally registering submodule.$name.url to every submodule in the project, even the ones that the user has never showed interest in at all by running 'git submodule init' command. This caused subsequent 'git submodule update' to start cloning/updating submodules that are not interesting to the user at all.

Update the code so that the URL is updated from the .gitmodules file only for submodules that already have submodule.$name.url entries, i.e. the ones the user has showed interested in having a checkout.

---

 git-submodule.sh          |   23 +++++++++++++----------
 t/t7403-submodule-sync.sh |   13 +++++++++++--
 2 files changed, 24 insertions(+), 12 deletions(-)

index c291eed..ce65971 100755
@@ -836,17 +836,20 @@ cmd_sync()
 			;;
 		esac
 
-		say "Synchronizing submodule url for '$name'"
-		git config submodule."$name".url "$url"
-
-		if test -e "$path"/.git
+		if git config "submodule.$name.url" >/dev/null 2>/dev/null
 		then
-		(
-			clear_local_git_env
-			cd "$path"
-			remote=$(get_default_remote)
-			git config remote."$remote".url "$url"
-		)
+			say "Synchronizing submodule url for '$name'"
+			git config submodule."$name".url "$url"
+
+			if test -e "$path"/.git
+			then
+			(
+				clear_local_git_env
+				cd "$path"
+				remote=$(get_default_remote)
+				git config remote."$remote".url "$url"
+			)
+			fi
 		fi
 	done
 }

index e5b1953..597a06f 100755
@@ -25,7 +25,8 @@ test_expect_success setup '
 	git clone super super-clone &&
 	(cd super-clone && git submodule update --init) &&
 	git clone super empty-clone &&
-	(cd empty-clone && git submodule init)
+	(cd empty-clone && git submodule init) &&
+	git clone super top-only-clone
 '
 
 test_expect_success 'change submodule' '

@@ -66,7 +67,7 @@ test_expect_success '"git submodule sync" should update submodule URLs' '
 	)
 '
 
-test_expect_success '"git submodule sync" should update submodule URLs if not yet cloned' '
+test_expect_success '"git submodule sync" should update known submodule URLs' '
 	(cd empty-clone &&
 	 git pull &&
 	 git submodule sync &&

@@ -74,4 +75,12 @@ test_expect_success '"git submodule sync" should update submodule URLs if not ye
 	)
 '
 
+test_expect_success '"git submodule sync" should not vivify uninteresting submodule' '
+	(cd top-only-clone &&
+	 git pull &&
+	 git submodule sync &&
+	 test -z "$(git config submodule.submodule.url)"
+	)
+'
+
 test_done

Am 24.06.2011 00:28, schrieb Junio C Hamano:
---8<---

---8<---

I think that describes the situation pretty well and the patch looks good. I will not be able to test it before next week though, do not wait for me.

There are two minor issues for me left.
(1) The man page of submodule sync talks about "all submodules" and not "registered submodules". I treated it as "subsequent init" and personally do not restrict the list of submodules, simply because my build would not work without them.

(2) It is confusing to have registered submodules in .git/config without a matching gitlink. Say, you switch between branch a1 with a submodule s with url u1, to a branch without s, you git clean -xdff (url u1 is still in .git/config, right?), and then to a branch a2 with s pointing to url u2. This bugged me.

Ciao,
-- andi5

Am 24.06.2011 01:10, schrieb Andreas Köhler:

While investigating this I stumbled across that too, will post a patch.

I think that is ok, because that way git remembers that the user cares about submodule s even when he switches to a branch where s doesn't exist. And the documentation is pretty clear that you'll have to use "git submodule sync" to update the url itself from u1 to u2 when needed.

From: Junio C Hamano <[email protected]>

Earlier 33f072f (submodule sync: Update "submodule.<name>.url" for empty directories, 2010-10-08) attempted to fix a bug where "git submodule sync" command does not update the URL if the current superproject does not have a checkout of the submodule.

However, it did so by unconditionally registering submodule.$name.url to every submodule in the project, even the ones that the user has never showed interest in at all by running 'git submodule init' command. This caused subsequent 'git submodule update' to start cloning/updating submodules that are not interesting to the user at all.

Update the code so that the URL is updated from the .gitmodules file only for submodules that already have submodule.$name.url entries, i.e. the ones the user has showed interested in having a checkout.

---

Am 24.06.2011 00:28, schrieb Junio C Hamano:

Yes, I am convinced your patch is the doing Right Thing. I squashed in a change to the git submodule sync documentation (explicitly stating that a sync only affects submodules which have an url entry in .git/config) and two more lines in the test you added to make sure that explicitly giving a submodule on the command line uses the same logic.

What do you think?

(I'll send another patch shortly addressing the failing tests when jl/submodule-add-relurl-wo-upstream is merged into this one)

 Documentation/git-submodule.txt |    4 +++-
 git-submodule.sh                |   23 +++++++++++++----------
 t/t7403-submodule-sync.sh       |   15 +++++++++++++--
 3 files changed, 29 insertions(+), 13 deletions(-)

index 5e7a413..2b31d5f 100644
@@ -167,7 +167,9 @@ commit for each submodule.
 
 sync::
 	Synchronizes submodules' remote URL configuration setting
-	to the value specified in .gitmodules.  This is useful when
+	to the value specified in .gitmodules. It will only affect those
+	submodules which already have an url entry in .git/config (that is the
+	case when they are initialized or freshly added). This is useful when
 	submodule URLs change upstream and you need to update your local
 	repositories accordingly.
 +

index d189a24..543f1d0 100755
@@ -874,17 +874,20 @@ cmd_sync()
 			;;
 		esac
 
-		say "Synchronizing submodule url for '$name'"
-		git config submodule."$name".url "$url"
-
-		if test -e "$path"/.git
+		if git config "submodule.$name.url" >/dev/null 2>/dev/null
 		then
-		(
-			clear_local_git_env
-			cd "$path"
-			remote=$(get_default_remote)
-			git config remote."$remote".url "$url"
-		)
+			say "Synchronizing submodule url for '$name'"
+			git config submodule."$name".url "$url"
+
+			if test -e "$path"/.git
+			then
+			(
+				clear_local_git_env
+				cd "$path"
+				remote=$(get_default_remote)
+				git config remote."$remote".url "$url"
+			)
+			fi
 		fi
 	done
 }

index d600583..95ffe34 100755
@@ -25,7 +25,8 @@ test_expect_success setup '
 	git clone super super-clone &&
 	(cd super-clone && git submodule update --init) &&
 	git clone super empty-clone &&
-	(cd empty-clone && git submodule init)
+	(cd empty-clone && git submodule init) &&
+	git clone super top-only-clone
 '
 
 test_expect_success 'change submodule' '

@@ -66,7 +67,7 @@ test_expect_success '"git submodule sync" should update submodule URLs' '
 	)
 '
 
-test_expect_success '"git submodule sync" should update submodule URLs if not yet cloned' '
+test_expect_success '"git submodule sync" should update known submodule URLs' '
 	(cd empty-clone &&
 	 git pull &&
 	 git submodule sync &&

@@ -74,4 +75,14 @@ test_expect_success '"git submodule sync" should update submodule URLs if not ye
 	)
 '
 
+test_expect_success '"git submodule sync" should not vivify uninteresting submodule' '
+	(cd top-only-clone &&
+	 git pull &&
+	 git submodule sync &&
+	 test -z "$(git config submodule.submodule.url)" &&
+	 git submodule sync submodule &&
+	 test -z "$(git config submodule.submodule.url)"
+	)
+'
+
 test_done

1.7.6.rc3.1.g8fd6

This, when merged with jl/submodule-add-relurl-wo-upstream topic in "pu", breaks a test in t7400, which was introduced by 4d68932 (submodule add:
allow relative repository path even when no url is set, 2011-06-06).

A workaround patch (shown offset below) to make it "pass" exposes that the test was relying on the too aggressive "submodule sync" behaviour I just fixed with the previous patch.

And I do not think the particular workaround is correct.

Shouldn't "submodule add" add an entry for .git/config even when it cloned from elsewhere?

When adding a local subdirectory that houses the subproject with "submodule add", the code does add an entry to .git/config, but when the moule is cloned from elsewhere (by calling module_clone), nobody adds the corresponding entry to .git/config. "submodule add $URL $path" should result in exactly the same state as if the user said "submodule init $path" and then "submodule update $path" to instantiate the path in a superproject that was cloned from elsewhere, no?

I suspect this fix will cascade to breakage elsewhere, but I've run out of energy and inclination to look at the submodule code tonight, so I'll let the list to take it further from here.

-- >8 --
Subject: [RFC] submodule add: initialize .git/config entry

When "git submodule add $path" is run to add a subdirectory $path to the superproject, and $path is already the top of the working tree of the submodule repository, the command created submodule.$path.url entry in the configuration file in the superproject. However, when adding a repository $URL that is outside the respository of the superproject to $path that does not exist (yet) with "git submodule add $URL $path", the command forgot to set it up.

The user is expressing the interest in the submodule and wants to keep a checkout; the "submodule add" command should consistently set up the submodule.$path.url entry in either case.

---

    (A workaround patch that may not be a correct fix for the test breakage)

        diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
        index 9099e80..cfe81e3 100755
        --- a/t/t7400-submodule-basic.sh
        +++ b/t/t7400-submodule-basic.sh
        @@ -446,16 +446,17 @@ test_expect_success 'add should fail when pat...
                )
         '

         test_expect_success 'use superproject as upstream when path is rel...
                (
                        cd addtest &&
                        git submodule add ../repo relative &&
                        test "$(git config -f .gitmodules submodule.relativ...
        +		git submodule init relative &&
                        git submodule sync relative &&
                        test "$(git config submodule.relative.url)" = "$sub...
                )
         '

         test_expect_success 'set up for relative path tests' '
                mkdir reltest &&
                (

 git-submodule.sh |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

index b0a8e63..8a16ad1 100755
@@ -238,7 +238,6 @@ cmd_add()
 			die "'$path' already exists and is not a valid git repo"
 		fi
 
-		git config submodule."$path".url "$realrepo"
 	else
 
 		module_clone "$path" "$realrepo" "$reference" || exit

@@ -253,6 +252,8 @@ cmd_add()
 		) || die "Unable to checkout submodule '$path'"
 	fi
 
+	git config submodule."$path".url "$realrepo"
+
 	git add $force "$path" ||
 	die "Failed to add submodule '$path'"
 

Am 24.06.2011 06:13, schrieb Junio C Hamano:

I'll won't have much git time today but I'll look into that over the weekend.

When "git submodule add $path" is run to add a subdirectory $path to the superproject, and $path is already the top of the working tree of the submodule repository, the command created submodule.$path.url entry in the configuration file in the superproject. However, when adding a repository $URL that is outside the respository of the superproject to $path that does not exist (yet) with "git submodule add $URL $path", the command forgot to set it up.

The user is expressing the interest in the submodule and wants to keep a checkout, the "submodule add" command should consistently set up the submodule.$path.url entry in either case.

As a result "git submodule init" can't simply skip the initialization of those submodules for which it finds an url entry in the git./config anymore. That lead to problems when adding a submodule (which now sets the url), add the "update" setting to .gitmodules and expect init to copy that into .git/config like it is done in t7406. So change init to only then copy the "url" and "update" entries when they don't exist yet in the .git/config and do nothing otherwise.

---

Am 24.06.2011 06:13, schrieb Junio C Hamano:

Yes, we should be consistent here.

Ok, t7406 expected "git submodule init" to copy the new update setting into .git/config for a newly added submodule, which it didn't do anymore because it already found the url set. I solved that by teaching init to only then copy the url and update settings if they aren't present yet. Now all tests are running fine and your change to the test I added in jl/submodule-add-relurl-wo-upstream isn't necessary anymore.

When I cherry pick that onto cbd0a3c6bc in your current pu branch and resolve the conflicts all tests run fine (if you want me to resend this patch based on that commit to avoid the conflicts with i18n and the "submodule add: clean up duplicated code" patch please just say so).

 git-submodule.sh |   33 +++++++++++++++++----------------
 1 files changed, 17 insertions(+), 16 deletions(-)

index 543f1d0..7e39c97 100755
@@ -246,7 +246,6 @@ cmd_add()
 			url="$repo"
 			;;
 		esac
-		git config submodule."$path".url "$url"
 	else
 
 		module_clone "$path" "$realrepo" "$reference" || exit

@@ -260,6 +259,7 @@ cmd_add()
 			esac
 		) || die "Unable to checkout submodule '$path'"
 	fi
+	git config submodule."$path".url "$url"
 
 	git add $force "$path" ||
 	die "Failed to add submodule '$path'"

@@ -355,25 +355,26 @@ cmd_init()
 	do
 		# Skip already registered paths
 		name=$(module_name "$path") || exit
-		url=$(git config submodule."$name".url)
-		test -z "$url" || continue
-
-		url=$(git config -f .gitmodules submodule."$name".url)
-		test -z "$url" &&
-		die "No url found for submodule path '$path' in .gitmodules"
-
-		# Possibly a url relative to parent
-		case "$url" in
-		./*|../*)
-			url=$(resolve_relative_url "$url") || exit
-			;;
-		esac
-
-		git config submodule."$name".url "$url" ||
-		die "Failed to register url for submodule path '$path'"
+		if test -z "$(git config "submodule.$name.url")"
+		then
+			url=$(git config -f .gitmodules submodule."$name".url)
+			test -z "$url" &&
+			die "No url found for submodule path '$path' in .gitmodules"
+
+			# Possibly a url relative to parent
+			case "$url" in
+			./*|../*)
+				url=$(resolve_relative_url "$url") || exit
+				;;
+			esac
+			git config submodule."$name".url "$url" ||
+			die "Failed to register url for submodule path '$path'"
+		fi
 
+		# Copy "update" setting when it is not set yet
 		upd="$(git config -f .gitmodules submodule."$name".update)"
 		test -z "$upd" ||
+		test -n "$(git config submodule."$name".update)" ||
 		git config submodule."$name".update "$upd" ||
 		die "Failed to register update mode for submodule path '$path'"
 

1.7.6.rc3.2.gcfa18

Hmm, now 7610 seems to expect somewhat different behaviour and fails.

What about a compromise? Change git-submodule-sync to skip submodules which are not already initialized.

I have a patch to do this, but I need to test it still. If it sounds right, I'll try to submit it later today.

Phil

You completely lost me. How's that different from reverting 33f072f? I do not see a room for "compromise" here.

Your patch does what I meant, so I assume you figured out my compromise.

My patch was simpler, but I haven't tested it properly yet.

Phil