Teach fetch/pull the on-demand mode and make it the default

PATCH Superseded Jens Lehmann 2 participants 16 messages Jun 15, 2016 Documentation

lore

Share & tools

lore

Jens Lehmann

author

2016-06-15 22:50 UTC #0 lore raw

v2 cover Teach fetch/pull the on-demand mode and make it the default

So here is version 2 of the on-demand recursive fetch for submodules.

Changes since the last version:

*) Add a name parameter to parse_fetch_recurse_submodules_arg() *) Enhance description of the "--submodule-default" fetch option *) Free the changed_submodule_paths string_list at the end of
fetch_populated_submodules() to aid libification and to make valgrind happy
*) Add a comment for the while loop in check_for_new_submodule_commits() *) Use "--all" instead of "--branches" and "--remotes" for rev-list to
include tags too
*) Compute the argc value properly instead of hard coding it in
check_for_new_submodule_commits()
*) Change size_t casts to intptr_t where appropriate
*) Use "rev-list --not --all" when checking if a submodule commit exists
instead of a simple lookup/rev-parse to make sure they are not only present but are reachable from a ref
*) Describe that fetch and pull only work on populated submodules for
now in the BUGS section of their man pages (done in a separate commit because this doesn't belong to a special commit in this series but also to the functionality that has already been merged)

Thanks to Jonathan an Junio for the feedback that lead to these changes.

Issues that were raised but not addressed yet:

*) Should this option be called "on-demand" or "changed" or maybe even
"updated"?
I have no strong feelings about this, me just thinks the name should tell somehow that only submodules where new commits are recorded in the superproject which aren't already present locally will be fetched. Opinions?
*) Check the remote tracking refs in the submodule in tests instead of
the output of the fetch command

Jens Lehmann (7):
fetch/pull: recurse into submodules when necessary fetch/pull: Add the 'on-demand' value to the --recurse-submodules
option
config: teach the fetch.recurseSubmodules option the 'on-demand'
value
Submodules: Add 'on-demand' value for the 'fetchRecurseSubmodule'
option
fetch/pull: Don't recurse into a submodule when commits are already
present
submodule update: Don't fetch when the submodule commit is already
present
fetch/pull: Describe --recurse-submodule restrictions in the BUGS
section

 Documentation/config.txt        |   12 ++-
 Documentation/fetch-options.txt |   27 ++++-
 Documentation/git-fetch.txt     |    9 ++
 Documentation/git-pull.txt      |   11 ++-
 Documentation/gitmodules.txt    |    4 +-
 builtin/fetch.c                 |   47 ++++++--
 git-pull.sh                     |    3 +
 git-submodule.sh                |    5 +-
 submodule.c                     |  152 +++++++++++++++++++++--
 submodule.h                     |   11 ++-
 t/t5526-fetch-submodules.sh     |  255 +++++++++++++++++++++++++++++++++++++++
 t/t7403-submodule-sync.sh       |    2 +-
 t/t7406-submodule-update.sh     |   20 +++
 13 files changed, 521 insertions(+), 37 deletions(-)

Signature · 1 line

1.7.4.1.291.g6060

Jens Lehmann

author

2016-06-15 22:50 UTC #1 lore raw

PATCH v2 1/7 fetch/pull: recurse into submodules when necessary

To be able to access all commits of populated submodules referenced by the superproject it is sufficient to only then let "git fetch" recurse into a submodule when the new commits fetched in the superproject record new commits for it. Having these commits present is extremely useful when using the "--submodule" option to "git diff" (which is what "git gui" and "gitk" do since 1.6.6), as all submodule commits needed for creating a descriptive output can be accessed. Also merging submodule commits (added in 1.7.3) depends on the submodule commits in question being present to work. Last but not least this enables disconnected operation when using submodules, as all commits necessary for a successful "git submodule update -N" will have been fetched automatically. So we choose this mode as the default for fetch and pull.

Before a new or changed ref from upstream is updated in update_local_ref() "git rev-list <new-sha1> --not --branches --remotes" is used to determine all newly fetched commits. These are then walked and diffed against their parent(s) to see if a submodule has been changed. If that is the case, its path is stored to be fetched after the superproject fetch is completed.

Using the "--recurse-submodules" or the "--no-recurse-submodules" option disables the examination of the fetched refs because the result will be ignored anyway.

There is currently no infrastructure for storing deleted and new submodules in the .git directory of the superproject. Thats why fetch and pull for now only fetch submodules that are already checked out and are not renamed.

In t7403 the "--no-recurse-submodules" argument had to be added to "git pull" to avoid failure because of the moved upstream submodule repo.

Thanks-to: Jonathan Nieder <[email protected]>
Thanks-to: Heiko Voigt <[email protected]>

Signed-off-by Jens Lehmann <[email protected]>

---
 Documentation/fetch-options.txt |   11 ++++
 builtin/fetch.c                 |   25 ++++++---
 submodule.c                     |  106 +++++++++++++++++++++++++++++++++++---
 submodule.h                     |    9 +++
 t/t5526-fetch-submodules.sh     |  109 +++++++++++++++++++++++++++++++++++++++
 t/t7403-submodule-sync.sh       |    2 +-
 6 files changed, 245 insertions(+), 17 deletions(-)

Documentation/fetch-options.txt +11 -0

index f37276e..ae22f75 100644
@@ -73,6 +73,17 @@ ifndef::git-pull[]
 	Prepend <path> to paths printed in informative messages
 	such as "Fetching submodule foo".  This option is used
 	internally when recursing over submodules.
+
+--submodule-default=[yes|on-demand]::
+	This option is used internally to set the submodule recursion default
+	to either a boolean configuration value representing "true" (for
+	unconditonal recursion) or to "on-demand" (when only those submodules
+	should be fetched of which new commits have been fetched in its
+	superproject).
+	This option will be overridden by the 'fetch.recurseSubmodules' and
+	'submodule.<name>.fetchRecurseSubmodules' settings in
+	linkgit:gitmodules[5] and linkgit:git-config[1] while all of them
+	are ignored when the "--[no-]recurse-submodules" option is given.
 endif::git-pull[]
 
 -u::

builtin/fetch.c +16 -9

index 7efecfe..55d6b1a 100644
@@ -28,12 +28,6 @@ enum {
 	TAGS_SET = 2
 };
 
-enum {
-	RECURSE_SUBMODULES_OFF = 0,
-	RECURSE_SUBMODULES_DEFAULT = 1,
-	RECURSE_SUBMODULES_ON = 2
-};
-
 static int all, append, dry_run, force, keep, multiple, prune, update_head_ok, verbosity;
 static int progress, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
 static int tags = TAGS_DEFAULT;

@@ -42,6 +36,7 @@ static const char *upload_pack;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *transport;
 static const char *submodule_prefix = "";
+static const char *submodule_default;
 
 static struct option builtin_fetch_options[] = {
 	OPT__VERBOSITY(&verbosity),

@@ -73,6 +68,8 @@ static struct option builtin_fetch_options[] = {
 		   "deepen history of shallow clone"),
 	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, "dir",
 		   "prepend this to submodule path output", PARSE_OPT_HIDDEN },
+	{ OPTION_STRING, 0, "submodule-default", &submodule_default, NULL,
+		   "default mode for recursion", PARSE_OPT_HIDDEN },
 	OPT_END()
 };

@@ -284,6 +281,9 @@ static int update_local_ref(struct ref *ref,
 		else {
 			msg = "storing head";
 			what = "[new branch]";
+			if ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&
+			    (recurse_submodules != RECURSE_SUBMODULES_ON))
+				check_for_new_submodule_commits(ref->new_sha1);
 		}
 
 		r = s_update_ref(msg, ref, 0);

@@ -299,6 +299,9 @@ static int update_local_ref(struct ref *ref,
 		strcpy(quickref, find_unique_abbrev(current->object.sha1, DEFAULT_ABBREV));
 		strcat(quickref, "..");
 		strcat(quickref, find_unique_abbrev(ref->new_sha1, DEFAULT_ABBREV));
+		if ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&
+		    (recurse_submodules != RECURSE_SUBMODULES_ON))
+			check_for_new_submodule_commits(ref->new_sha1);
 		r = s_update_ref("fast-forward", ref, 1);
 		sprintf(display, "%c %-*s %-*s -> %s%s", r ? '!' : ' ',
 			TRANSPORT_SUMMARY_WIDTH, quickref, REFCOL_WIDTH, remote,

@@ -310,6 +313,9 @@ static int update_local_ref(struct ref *ref,
 		strcpy(quickref, find_unique_abbrev(current->object.sha1, DEFAULT_ABBREV));
 		strcat(quickref, "...");
 		strcat(quickref, find_unique_abbrev(ref->new_sha1, DEFAULT_ABBREV));
+		if ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&
+		    (recurse_submodules != RECURSE_SUBMODULES_ON))
+			check_for_new_submodule_commits(ref->new_sha1);
 		r = s_update_ref("forced-update", ref, 1);
 		sprintf(display, "%c %-*s %-*s -> %s  (%s)", r ? '!' : '+',
 			TRANSPORT_SUMMARY_WIDTH, quickref, REFCOL_WIDTH, remote,

@@ -949,9 +955,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	if (!result && (recurse_submodules != RECURSE_SUBMODULES_OFF)) {
 		const char *options[10];
 		int num_options = 0;
-		/* Set recursion as default when we already are recursing */
-		if (submodule_prefix[0])
-			set_config_fetch_recurse_submodules(1);
+		if (submodule_default) {
+			int arg = parse_fetch_recurse_submodules_arg("--submodule-default", submodule_default);
+			set_config_fetch_recurse_submodules(arg);
+		}
 		gitmodules_config();
 		git_config(submodule_config, NULL);
 		add_options_to_argv(&num_options, options);

submodule.c +99 -7

172 lines not on this page · raw message

submodule.h +9 -0

index 4729023..3434a8e 100644
@@ -3,17 +3,26 @@
 
 struct diff_options;
 
+enum {
+	RECURSE_SUBMODULES_ON_DEMAND = -1,
+	RECURSE_SUBMODULES_OFF = 0,
+	RECURSE_SUBMODULES_DEFAULT = 1,
+	RECURSE_SUBMODULES_ON = 2
+};
+
 void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,
 		const char *path);
 int submodule_config(const char *var, const char *value, void *cb);
 void gitmodules_config();
 int parse_submodule_config_option(const char *var, const char *value);
 void handle_ignore_submodules_arg(struct diff_options *diffopt, const char *);
+int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);
 void show_submodule_summary(FILE *f, const char *path,
 		unsigned char one[20], unsigned char two[20],
 		unsigned dirty_submodule,
 		const char *del, const char *add, const char *reset);
 void set_config_fetch_recurse_submodules(int value);
+void check_for_new_submodule_commits(unsigned char new_sha1[20]);
 int fetch_populated_submodules(int num_options, const char **options,
 			       const char *prefix, int ignore_config,
 			       int quiet);

t/t5526-fetch-submodules.sh +109 -0

113 lines not on this page · raw message

t/t7403-submodule-sync.sh +1 -1

index e5b1953..d600583 100755
@@ -52,7 +52,7 @@ test_expect_success 'change submodule url' '
 
 test_expect_success '"git submodule sync" should update submodule URLs' '
 	(cd super-clone &&
-	 git pull &&
+	 git pull --no-recurse-submodules &&
 	 git submodule sync
 	) &&
 	test -d "$(git config -f super-clone/submodule/.git/config \

Signature · 1 line

1.7.4.1.291.g6060

Marc Branchaud

2016-06-15 22:50 UTC #2 lore raw

Re: [PATCH v2 1/7] fetch/pull: recurse into submodules when necessary

Quoted 53 lines · Jens Lehmann

To be able to access all commits of populated submodules referenced by the superproject it is sufficient to only then let "git fetch" recurse into a submodule when the new commits fetched in the superproject record new commits for it. Having these commits present is extremely useful when using the "--submodule" option to "git diff" (which is what "git gui" and "gitk" do since 1.6.6), as all submodule commits needed for creating a descriptive output can be accessed. Also merging submodule commits (added in 1.7.3) depends on the submodule commits in question being present to work. Last but not least this enables disconnected operation when using submodules, as all commits necessary for a successful "git submodule update -N" will have been fetched automatically. So we choose this mode as the default for fetch and pull.
Before a new or changed ref from upstream is updated in update_local_ref() "git rev-list <new-sha1> --not --branches --remotes" is used to determine all newly fetched commits. These are then walked and diffed against their parent(s) to see if a submodule has been changed. If that is the case, its path is stored to be fetched after the superproject fetch is completed.
Using the "--recurse-submodules" or the "--no-recurse-submodules" option disables the examination of the fetched refs because the result will be ignored anyway.
There is currently no infrastructure for storing deleted and new submodules in the .git directory of the superproject. Thats why fetch and pull for now only fetch submodules that are already checked out and are not renamed.
In t7403 the "--no-recurse-submodules" argument had to be added to "git pull" to avoid failure because of the moved upstream submodule repo.
Thanks-to: Jonathan Nieder <[email protected]>
Thanks-to: Heiko Voigt <[email protected]>
Signed-off-by Jens Lehmann <[email protected]>
---
 Documentation/fetch-options.txt |   11 ++++
 builtin/fetch.c                 |   25 ++++++---
 submodule.c                     |  106 +++++++++++++++++++++++++++++++++++---
 submodule.h                     |    9 +++
 t/t5526-fetch-submodules.sh     |  109 +++++++++++++++++++++++++++++++++++++++
 t/t7403-submodule-sync.sh       |    2 +-
 6 files changed, 245 insertions(+), 17 deletions(-)
Documentation/fetch-options.txt +2 -0
index f37276e..ae22f75 100644
@@ -73,6 +73,17 @@ ifndef::git-pull[]
 	Prepend <path> to paths printed in informative messages
 	such as "Fetching submodule foo".  This option is used
 	internally when recursing over submodules.
+
+--submodule-default=[yes|on-demand]::

Documentation/fetch-options.txt ifndef::git-pull[] · line 87

Quoted 10 lines

Documentation/fetch-options.txt +9 -0

+	This option is used internally to set the submodule recursion default
+	to either a boolean configuration value representing "true" (for
+	unconditonal recursion) or to "on-demand" (when only those submodules
+	should be fetched of which new commits have been fetched in its
+	superproject).
+	This option will be overridden by the 'fetch.recurseSubmodules' and
+	'submodule.<name>.fetchRecurseSubmodules' settings in
+	linkgit:gitmodules[5] and linkgit:git-config[1] while all of them
+	are ignored when the "--[no-]recurse-submodules" option is given.
 endif::git-pull[]

Rolling out the bike...

I think the option name and it's explanation need to be a little clearer:

--recurse-submodules-default=[yes|on-demand]::
This option is used internally to temporarily provide a non-negative default value for the --recurse-submodules option. All other methods of configuring fetch's submodule recursion (such as settings in linkgit:gitmodules[5] and linkgit:git-config[1]) override this option, as does specifying --[no-]recurse-submodules directly.

Jens Lehmann

author

2016-06-15 22:50 UTC #3 lore raw

PATCH v2 2/7 fetch/pull: Add the 'on-demand' value to the --recurse-submodules option

In reply to Jens Lehmann

Until now the --recurse-submodules option could only be used to either fetch all populated submodules recursively or to disable recursion completely. As fetch and pull now by default just fetch those submodules for which new commits have been fetched in the superproject, a command line option to enforce that behavior is needed to be able to override configuration settings.

Signed-off-by Jens Lehmann <[email protected]>

---
 Documentation/fetch-options.txt |   16 +++++++--
 Documentation/git-pull.txt      |    2 +-
 builtin/fetch.c                 |   22 ++++++++++--
 git-pull.sh                     |    3 ++
 submodule.c                     |    8 +++-
 submodule.h                     |    2 +-
 t/t5526-fetch-submodules.sh     |   71 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 114 insertions(+), 10 deletions(-)

Documentation/fetch-options.txt +13 -3

index ae22f75..f28c0b1 100644
@@ -65,9 +65,19 @@ ifndef::git-pull[]
 	specified with the remote.<name>.tagopt setting. See
 	linkgit:git-config[1].
 
---[no-]recurse-submodules::
-	This option controls if new commits of all populated submodules should
-	be fetched too (see linkgit:git-config[1] and linkgit:gitmodules[5]).
+--recurse-submodules[=yes|on-demand|no]::
+	This option controls if and under what conditions new commits of all
+	populated submodules should be fetched too. It can be used as a
+	boolean option to completely disable recursion when set to 'no' or to
+	unconditionally recurse into all populated submodules when set to
+	'yes', which is the default when this option is used without any
+	value. If 'on-demand' is used, it will only recurse into those
+	submodules where new commits have been fetched in the superproject
+	(also see linkgit:git-config[1] and linkgit:gitmodules[5]).
+
+--no-recurse-submodules::
+	Disable recursive fetching of submodules (this has the same effect as
+	using the '--recurse-submodules=no' option).
 
 --submodule-prefix=<path>::
 	Prepend <path> to paths printed in informative messages

Documentation/git-pull.txt +1 -1

index b33e6be..c45efb3 100644
@@ -84,7 +84,7 @@ must be given before the options meant for 'git fetch'.
 --verbose::
 	Pass --verbose to git-fetch and git-merge.
 
---[no-]recurse-submodules::
+--[no-]recurse-submodules[=yes|on-demand|no]::
 	This option controls if new commits of all populated submodules should
 	be fetched too (see linkgit:git-config[1] and linkgit:gitmodules[5]).
 	That might be necessary to get the data needed for merging submodule

builtin/fetch.c +19 -3

index 55d6b1a..7d24e47 100644
@@ -38,6 +38,20 @@ static struct transport *transport;
 static const char *submodule_prefix = "";
 static const char *submodule_default;
 
+static int option_parse_recurse_submodules(const struct option *opt,
+				   const char *arg, int unset)
+{
+	if (unset) {
+		recurse_submodules = RECURSE_SUBMODULES_OFF;
+	} else {
+		if (arg)
+			recurse_submodules = parse_fetch_recurse_submodules_arg(opt->long_name, arg);
+		else
+			recurse_submodules = RECURSE_SUBMODULES_ON;
+	}
+	return 0;
+}
+
 static struct option builtin_fetch_options[] = {
 	OPT__VERBOSITY(&verbosity),
 	OPT_BOOLEAN(0, "all", &all,

@@ -55,9 +69,9 @@ static struct option builtin_fetch_options[] = {
 		    "do not fetch all tags (--no-tags)", TAGS_UNSET),
 	OPT_BOOLEAN('p', "prune", &prune,
 		    "prune remote-tracking branches no longer on remote"),
-	OPT_SET_INT(0, "recurse-submodules", &recurse_submodules,
+	{ OPTION_CALLBACK, 0, "recurse-submodules", NULL, "on-demand",
 		    "control recursive fetching of submodules",
-		    RECURSE_SUBMODULES_ON),
+		    PARSE_OPT_OPTARG, option_parse_recurse_submodules },
 	OPT_BOOLEAN(0, "dry-run", &dry_run,
 		    "dry run"),
 	OPT_BOOLEAN('k', "keep", &keep, "keep downloaded pack"),

@@ -816,6 +830,8 @@ static void add_options_to_argv(int *argc, const char **argv)
 		argv[(*argc)++] = "--keep";
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		argv[(*argc)++] = "--recurse-submodules";
+	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
+		argv[(*argc)++] = "--recurse-submodules=on-demand";
 	if (verbosity >= 2)
 		argv[(*argc)++] = "-v";
 	if (verbosity >= 1)

@@ -964,7 +980,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		add_options_to_argv(&num_options, options);
 		result = fetch_populated_submodules(num_options, options,
 						    submodule_prefix,
-						    recurse_submodules == RECURSE_SUBMODULES_ON,
+						    recurse_submodules,
 						    verbosity < 0);
 	}

git-pull.sh +3 -0

index 86517e9..7a97d8e 100755
@@ -108,6 +108,9 @@ do
 	--recurse-submodules)
 		recurse_submodules=--recurse-submodules
 		;;
+	--recurse-submodules=*)
+		recurse_submodules="$1"
+		;;
 	--no-recurse-submodules)
 		recurse_submodules=--no-recurse-submodules
 		;;

submodule.c +6 -2

index 8f991ce..86a99af 100644
@@ -330,7 +330,7 @@ void check_for_new_submodule_commits(unsigned char new_sha1[20])
 }
 
 int fetch_populated_submodules(int num_options, const char **options,
-			       const char *prefix, int ignore_config,
+			       const char *prefix, int command_line_option,
 			       int quiet)
 {
 	int i, result = 0, argc = 0, default_argc;

@@ -376,7 +376,7 @@ int fetch_populated_submodules(int num_options, const char **options,
 			name = name_for_path->util;
 
 		default_argv = "yes";
-		if (!ignore_config) {
+		if (command_line_option == RECURSE_SUBMODULES_DEFAULT) {
 			struct string_list_item *fetch_recurse_submodules_option;
 			fetch_recurse_submodules_option = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name);
 			if (fetch_recurse_submodules_option) {

@@ -391,6 +391,10 @@ int fetch_populated_submodules(int num_options, const char **options,
 					default_argv = "on-demand";
 				}
 			}
+		} else if (command_line_option == RECURSE_SUBMODULES_ON_DEMAND) {
+			if (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))
+				continue;
+			default_argv = "on-demand";
 		}
 
 		strbuf_addf(&submodule_path, "%s/%s", work_tree, ce->name);

submodule.h +1 -1

index 3434a8e..5350b0d 100644
@@ -24,7 +24,7 @@ void show_submodule_summary(FILE *f, const char *path,
 void set_config_fetch_recurse_submodules(int value);
 void check_for_new_submodule_commits(unsigned char new_sha1[20]);
 int fetch_populated_submodules(int num_options, const char **options,
-			       const char *prefix, int ignore_config,
+			       const char *prefix, int command_line_option,
 			       int quiet);
 unsigned is_submodule_modified(const char *path, int ignore_untracked);
 int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],

t/t5526-fetch-submodules.sh +71 -0

75 lines not on this page · raw message

Signature · 1 line

1.7.4.1.291.g6060

Marc Branchaud

2016-06-15 22:50 UTC #4 lore raw

Re: [PATCH v2 2/7] fetch/pull: Add the 'on-demand' value to the --recurse-submodules option

Quoted 6 lines · Jens Lehmann

Until now the --recurse-submodules option could only be used to either fetch all populated submodules recursively or to disable recursion completely. As fetch and pull now by default just fetch those submodules for which new commits have been fetched in the superproject, a command line option to enforce that behavior is needed to be able to override configuration settings.

Strictly speaking, this patch should add the 'on-demand' value to the first patch's --submodule-default option. Perhaps I'm being a bit too uptight though...

Quoted 16 lines

Signed-off-by Jens Lehmann <[email protected]>

---
 Documentation/fetch-options.txt |   16 +++++++--
 Documentation/git-pull.txt      |    2 +-
 builtin/fetch.c                 |   22 ++++++++++--
 git-pull.sh                     |    3 ++
 submodule.c                     |    8 +++-
 submodule.h                     |    2 +-
 t/t5526-fetch-submodules.sh     |   71 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 114 insertions(+), 10 deletions(-)

Documentation/fetch-options.txt +0 -0

index ae22f75..f28c0b1 100644

Documentation/fetch-options.txt ifndef::git-pull[] · line 69

Quoted 8 lines

Documentation/fetch-options.txt +2 -3

@@ -65,9 +65,19 @@ ifndef::git-pull[]
 	specified with the remote.<name>.tagopt setting. See
 	linkgit:git-config[1].
 
---[no-]recurse-submodules::
-	This option controls if new commits of all populated submodules should
-	be fetched too (see linkgit:git-config[1] and linkgit:gitmodules[5]).
+--recurse-submodules[=yes|on-demand|no]::
+	This option controls if and under what conditions new commits of all

Nit: Remove "all" from the above line.

Documentation/fetch-options.txt ifndef::git-pull[] · line 76

Quoted 7 lines

Documentation/fetch-options.txt +7 -0

+	populated submodules should be fetched too. It can be used as a
+	boolean option to completely disable recursion when set to 'no' or to
+	unconditionally recurse into all populated submodules when set to
+	'yes', which is the default when this option is used without any
+	value. If 'on-demand' is used, it will only recurse into those
+	submodules where new commits have been fetched in the superproject
+	(also see linkgit:git-config[1] and linkgit:gitmodules[5]).

Nit: Replace that last sentence with

Use 'on-demand' to only recurse into a submodule when the superproject retrieves a commit that updates the submodule's reference.

Also, this has to change again with patch 5 (Don't recurse into a submodule when commits are already present).

M.

Marc Branchaud

2016-06-15 22:50 UTC #5 lore raw

Re: [PATCH v2 2/7] fetch/pull: Add the 'on-demand' value to the --recurse-submodules option

Quoted 49 lines · Marc Branchaud

Quoted 6 lines · Jens Lehmann
Until now the --recurse-submodules option could only be used to either fetch all populated submodules recursively or to disable recursion completely. As fetch and pull now by default just fetch those submodules for which new commits have been fetched in the superproject, a command line option to enforce that behavior is needed to be able to override configuration settings.
Strictly speaking, this patch should add the 'on-demand' value to the first patch's --submodule-default option. Perhaps I'm being a bit too uptight though...
Quoted 16 lines
Signed-off-by Jens Lehmann <[email protected]>
---
 Documentation/fetch-options.txt |   16 +++++++--
 Documentation/git-pull.txt      |    2 +-
 builtin/fetch.c                 |   22 ++++++++++--
 git-pull.sh                     |    3 ++
 submodule.c                     |    8 +++-
 submodule.h                     |    2 +-
 t/t5526-fetch-submodules.sh     |   71 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 114 insertions(+), 10 deletions(-)
Documentation/fetch-options.txt +0 -0
index ae22f75..f28c0b1 100644
Documentation/fetch-options.txt ifndef::git-pull[] · line 69
Quoted 8 lines
Documentation/fetch-options.txt +2 -3
@@ -65,9 +65,19 @@ ifndef::git-pull[]
 	specified with the remote.<name>.tagopt setting. See
 	linkgit:git-config[1].
 
---[no-]recurse-submodules::
-	This option controls if new commits of all populated submodules should
-	be fetched too (see linkgit:git-config[1] and linkgit:gitmodules[5]).
+--recurse-submodules[=yes|on-demand|no]::
+	This option controls if and under what conditions new commits of all
Nit: Remove "all" from the above line.
Documentation/fetch-options.txt ifndef::git-pull[] · line 76
Quoted 7 lines
Documentation/fetch-options.txt +7 -0
+	populated submodules should be fetched too. It can be used as a
+	boolean option to completely disable recursion when set to 'no' or to
+	unconditionally recurse into all populated submodules when set to
+	'yes', which is the default when this option is used without any
+	value. If 'on-demand' is used, it will only recurse into those
+	submodules where new commits have been fetched in the superproject
+	(also see linkgit:git-config[1] and linkgit:gitmodules[5]).
Nit: Replace that last sentence with

Quoted 3 lines

Use 'on-demand' to only recurse into a submodule when the superproject retrieves a commit that updates the submodule's reference.

Doh! That should read:

Use 'on-demand' to only recurse into a populated submodule when the superproject retrieves a commit that updates the submodule's reference.

Jens Lehmann

author

2016-06-15 22:50 UTC #6 lore raw

PATCH v3 3/7 config: teach the fetch.recurseSubmodules option the 'on-demand' value

In reply to Jens Lehmann

To enable the user to change the default behavior of "git fetch" and "git pull" regarding submodule recursion add the new "on-demand" value which has just been added to the "--recurse-submodules" command line option.

Signed-off-by Jens Lehmann <[email protected]>

---
 Documentation/config.txt    |   10 +++++++---
 submodule.c                 |    2 +-
 t/t5526-fetch-submodules.sh |   28 ++++++++++++++++++++++++++++
 3 files changed, 36 insertions(+), 4 deletions(-)

Documentation/config.txt +7 -3

index 96ce272..7b06fe5 100644
@@ -900,9 +900,13 @@ diff.wordRegex::
 	characters are *ignorable* whitespace.
 
 fetch.recurseSubmodules::
-	A boolean value which changes the behavior for fetch and pull, the
-	default is to not recursively fetch populated submodules unless
-	configured otherwise.
+	This option can be either set to a boolean value or to 'on-demand'.
+	Setting it to a boolean changes the behavior of fetch and pull to
+	unconditionally recurse into submodules when set to true or to not
+	recurse at all when set to false. When set to 'on-demand' (the default
+	value), it tells fetch and pull to recurse only into those submodules
+	where new commits are recorded in the commmits fetched for the
+	superproject.
 
 fetch.unpackLimit::
 	If the number of objects fetched over the git native

submodule.c +1 -1

index 86a99af..a1753f0 100644
@@ -71,7 +71,7 @@ int submodule_config(const char *var, const char *value, void *cb)
 	if (!prefixcmp(var, "submodule."))
 		return parse_submodule_config_option(var, value);
 	else if (!strcmp(var, "fetch.recursesubmodules")) {
-		config_fetch_recurse_submodules = git_config_bool(var, value);
+		config_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(var, value);
 		return 0;
 	}
 	return 0;

t/t5526-fetch-submodules.sh +28 -0

index 4cd723c..e6d873a 100755
@@ -372,4 +372,32 @@ test_expect_success "'--recurse-submodules=on-demand' stops when no new submodul
 	test_cmp expect.err.file actual.err
 '
 
+test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config" '
+	(
+		cd downstream &&
+		git fetch --recurse-submodules
+	) &&
+	add_upstream_commit &&
+	git config --global fetch.recurseSubmodules false &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "new submodule" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err.2 &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.2
+	head -2 expect.err >> expect.err.2 &&
+	(
+		cd downstream &&
+		git config fetch.recurseSubmodules on-demand &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	git config --global --unset fetch.recurseSubmodules &&
+	(
+		cd downstream &&
+		git config --unset fetch.recurseSubmodules
+	) &&
+	test_cmp expect.out.sub actual.out &&
+	test_cmp expect.err.2 actual.err
+'
+
 test_done

Signature · 1 line

1.7.4.1.291.g6060

Marc Branchaud

2016-06-15 22:50 UTC #7 lore raw

Re: [PATCH v3 3/7] config: teach the fetch.recurseSubmodules option the 'on-demand' value

Quoted 19 lines · Jens Lehmann

To enable the user to change the default behavior of "git fetch" and "git pull" regarding submodule recursion add the new "on-demand" value which has just been added to the "--recurse-submodules" command line option.
Signed-off-by Jens Lehmann <[email protected]>
---
 Documentation/config.txt    |   10 +++++++---
 submodule.c                 |    2 +-
 t/t5526-fetch-submodules.sh |   28 ++++++++++++++++++++++++++++
 3 files changed, 36 insertions(+), 4 deletions(-)
Documentation/config.txt +0 -0
index 96ce272..7b06fe5 100644
@@ -900,9 +900,13 @@ diff.wordRegex::
 	characters are *ignorable* whitespace.
 
 fetch.recurseSubmodules::

Documentation/config.txt diff.wordRegex:: · line 909

Quoted 10 lines

Documentation/config.txt +7 -3

-	A boolean value which changes the behavior for fetch and pull, the
-	default is to not recursively fetch populated submodules unless
-	configured otherwise.
+	This option can be either set to a boolean value or to 'on-demand'.
+	Setting it to a boolean changes the behavior of fetch and pull to
+	unconditionally recurse into submodules when set to true or to not
+	recurse at all when set to false. When set to 'on-demand' (the default
+	value), it tells fetch and pull to recurse only into those submodules
+	where new commits are recorded in the commmits fetched for the
+	superproject.

Nit: Replace the last sentence with

When set to 'on-demand' (the default value), fetch and pull will only recurse into a populated submodule when its superproject retrieves a commit that updates the submodule's reference.

Jens Lehmann

author

2016-06-15 22:50 UTC #8 lore raw

PATCH v2 4/7 Submodules: Add 'on-demand' value for the 'fetchRecurseSubmodule' option

In reply to Jens Lehmann

Now the behavior of fetch and pull can be configured to the recently added 'on-demand' mode separately for each submodule too.

Signed-off-by Jens Lehmann <[email protected]>

---
 Documentation/config.txt     |    2 +-
 Documentation/gitmodules.txt |    4 ++--
 submodule.c                  |    9 +++++++--
 t/t5526-fetch-submodules.sh  |   28 ++++++++++++++++++++++++++++
 4 files changed, 38 insertions(+), 5 deletions(-)

Documentation/config.txt +1 -1

index 7b06fe5..d47a95d 100644
@@ -1824,7 +1824,7 @@ submodule.<name>.update::
 	linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
 
 submodule.<name>.fetchRecurseSubmodules::
-	This option can be used to enable/disable recursive fetching of this
+	This option can be used to control recursive fetching of this
 	submodule. It can be overridden by using the --[no-]recurse-submodules
 	command line option to "git fetch" and "git pull".
 	This setting will override that from in the linkgit:gitmodules[5]

Documentation/gitmodules.txt +2 -2

index 6897794..25daee2 100644
@@ -45,12 +45,12 @@ submodule.<name>.update::
 	the '--merge' or '--rebase' options.
 
 submodule.<name>.fetchRecurseSubmodules::
-	This option can be used to enable/disable recursive fetching of this
+	This option can be used to control recursive fetching of this
 	submodule. If this option is also present in the submodules entry in
 	.git/config of the superproject, the setting there will override the
 	one found in .gitmodules.
 	Both settings can be overridden on the command line by using the
-	"--[no-]recurse-submodules" option to "git fetch" and "git pull"..
+	"--[no-]recurse-submodules" option to "git fetch" and "git pull".
 
 submodule.<name>.ignore::
 	Defines under what circumstances "git status" and the diff family show

submodule.c +7 -2

index a1753f0..3be6654 100644
@@ -113,7 +113,7 @@ int parse_submodule_config_option(const char *var, const char *value)
 		if (!config)
 			config = string_list_append(&config_fetch_recurse_submodules_for_name,
 						    strbuf_detach(&submodname, NULL));
-		config->util = git_config_bool(var, value) ? (void *)1 : NULL;
+		config->util = (void *)(intptr_t)parse_fetch_recurse_submodules_arg(var, value);
 		strbuf_release(&submodname);
 	} else if ((len > 7) && !strcmp(var + len - 7, ".ignore")) {
 		if (strcmp(value, "untracked") && strcmp(value, "dirty") &&

@@ -380,8 +380,13 @@ int fetch_populated_submodules(int num_options, const char **options,
 			struct string_list_item *fetch_recurse_submodules_option;
 			fetch_recurse_submodules_option = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name);
 			if (fetch_recurse_submodules_option) {
-				if (!fetch_recurse_submodules_option->util)
+				if ((intptr_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_OFF)
 					continue;
+				if ((intptr_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_ON_DEMAND) {
+					if (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))
+						continue;
+					default_argv = "on-demand";
+				}
 			} else {
 				if (config_fetch_recurse_submodules == RECURSE_SUBMODULES_OFF)
 					continue;

t/t5526-fetch-submodules.sh +28 -0

index e6d873a..09701aa 100755
@@ -400,4 +400,32 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config
 	test_cmp expect.err.2 actual.err
 '
 
+test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules" '
+	(
+		cd downstream &&
+		git fetch --recurse-submodules
+	) &&
+	add_upstream_commit &&
+	git config fetch.recurseSubmodules false &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "new submodule" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err.2 &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err.2
+	head -2 expect.err >> expect.err.2 &&
+	(
+		cd downstream &&
+		git config submodule.submodule.fetchRecurseSubmodules on-demand &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	git config --unset fetch.recurseSubmodules &&
+	(
+		cd downstream &&
+		git config --unset submodule.submodule.fetchRecurseSubmodules
+	) &&
+	test_cmp expect.out.sub actual.out &&
+	test_cmp expect.err.2 actual.err
+'
+
 test_done

Signature · 1 line

1.7.4.1.291.g6060

Jens Lehmann

author

2016-06-15 22:50 UTC #9 lore raw

PATCH v2 5/7 fetch/pull: Don't recurse into a submodule when commits are already present

In reply to Jens Lehmann

When looking for submodules where new commits have been recorded in the superproject ignore those cases where the submodules commits are already present locally. This can happen e.g. when the submodule has been rewound to an earlier state. Then there is no need to fetch the submodule again as the commit recorded in the newly fetched superproject commit has already been fetched earlier into the submodule.

Signed-off-by Jens Lehmann <[email protected]>

---
 submodule.c                 |   29 ++++++++++++++++++++++++++++-
 t/t5526-fetch-submodules.sh |   19 +++++++++++++++++++
 2 files changed, 47 insertions(+), 1 deletions(-)

submodule.c +28 -1

index 3be6654..11e0ef5 100644
@@ -263,6 +263,33 @@ void set_config_fetch_recurse_submodules(int value)
 	config_fetch_recurse_submodules = value;
 }
 
+static int is_submodule_commit_present(const char *path, unsigned char sha1[20])
+{
+	int is_present = 0;
+	if (!add_submodule_odb(path) && lookup_commit_reference(sha1)) {
+		/* Even if the submodule is checked out and the commit is
+		 * present, make sure it is reachable from a ref. */
+		struct child_process cp;
+		const char *argv[] = {"rev-list", "-n", "1", NULL, "--not", "--all", NULL};
+		struct strbuf buf = STRBUF_INIT;
+
+		argv[3] = sha1_to_hex(sha1);
+		memset(&cp, 0, sizeof(cp));
+		cp.argv = argv;
+		cp.env = local_repo_env;
+		cp.git_cmd = 1;
+		cp.no_stdin = 1;
+		cp.out = -1;
+		cp.dir = path;
+		if (!run_command(&cp) && !strbuf_read(&buf, cp.out, 1024))
+			is_present = 1;
+
+		close(cp.out);
+		strbuf_release(&buf);
+	}
+	return is_present;
+}
+
 static void submodule_collect_changed_cb(struct diff_queue_struct *q,
 					 struct diff_options *options,
 					 void *data)

@@ -280,7 +307,7 @@ static void submodule_collect_changed_cb(struct diff_queue_struct *q,
 			 * being moved around. */
 			struct string_list_item *path;
 			path = unsorted_string_list_lookup(&changed_submodule_paths, p->two->path);
-			if (!path)
+			if (!path && !is_submodule_commit_present(p->two->path, p->two->sha1))
 				string_list_append(&changed_submodule_paths, xstrdup(p->two->path));
 		} else {
 			/* Submodule is new or was moved here */

t/t5526-fetch-submodules.sh +19 -0

index 09701aa..3decfae 100755
@@ -428,4 +428,23 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override
 	test_cmp expect.err.2 actual.err
 '
 
+test_expect_success "don't fetch submodule when newly recorded commits are already present" '
+	(
+		cd submodule &&
+		git checkout -q HEAD^^
+	) &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "submodule rewound" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err &&
+	(
+		cd downstream &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	! test -s actual.out &&
+	test_cmp expect.err actual.err
+'
+
 test_done

Signature · 1 line

1.7.4.1.291.g6060

Marc Branchaud

2016-06-15 22:50 UTC #10 lore raw

Re: [PATCH v2 5/7] fetch/pull: Don't recurse into a submodule when commits are already present

As I said earlier, this patch should also tweak the documentation in fetch-options.txt and config.txt.

In my suggested wording I used the phrase

... only recurse into a populated submodule when its superproject retrieves a commit that updates the submodule's reference.

With this patch, that should now be

... only recurse into a populated submodule when its superproject retrieves a commit that updates the submodule's reference to a commit that isn't already in the local submodule clone.

Quoted 88 lines · Jens Lehmann

Signed-off-by Jens Lehmann <[email protected]>

---
 submodule.c                 |   29 ++++++++++++++++++++++++++++-
 t/t5526-fetch-submodules.sh |   19 +++++++++++++++++++
 2 files changed, 47 insertions(+), 1 deletions(-)

submodule.c +28 -1

index 3be6654..11e0ef5 100644
@@ -263,6 +263,33 @@ void set_config_fetch_recurse_submodules(int value)
 	config_fetch_recurse_submodules = value;
 }
 
+static int is_submodule_commit_present(const char *path, unsigned char sha1[20])
+{
+	int is_present = 0;
+	if (!add_submodule_odb(path) && lookup_commit_reference(sha1)) {
+		/* Even if the submodule is checked out and the commit is
+		 * present, make sure it is reachable from a ref. */
+		struct child_process cp;
+		const char *argv[] = {"rev-list", "-n", "1", NULL, "--not", "--all", NULL};
+		struct strbuf buf = STRBUF_INIT;
+
+		argv[3] = sha1_to_hex(sha1);
+		memset(&cp, 0, sizeof(cp));
+		cp.argv = argv;
+		cp.env = local_repo_env;
+		cp.git_cmd = 1;
+		cp.no_stdin = 1;
+		cp.out = -1;
+		cp.dir = path;
+		if (!run_command(&cp) && !strbuf_read(&buf, cp.out, 1024))
+			is_present = 1;
+
+		close(cp.out);
+		strbuf_release(&buf);
+	}
+	return is_present;
+}
+
 static void submodule_collect_changed_cb(struct diff_queue_struct *q,
 					 struct diff_options *options,
 					 void *data)

@@ -280,7 +307,7 @@ static void submodule_collect_changed_cb(struct diff_queue_struct *q,
 			 * being moved around. */
 			struct string_list_item *path;
 			path = unsorted_string_list_lookup(&changed_submodule_paths, p->two->path);
-			if (!path)
+			if (!path && !is_submodule_commit_present(p->two->path, p->two->sha1))
 				string_list_append(&changed_submodule_paths, xstrdup(p->two->path));
 		} else {
 			/* Submodule is new or was moved here */

t/t5526-fetch-submodules.sh +19 -0

index 09701aa..3decfae 100755
@@ -428,4 +428,23 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override
 	test_cmp expect.err.2 actual.err
 '
 
+test_expect_success "don't fetch submodule when newly recorded commits are already present" '
+	(
+		cd submodule &&
+		git checkout -q HEAD^^
+	) &&
+	head1=$(git rev-parse --short HEAD) &&
+	git add submodule &&
+	git commit -m "submodule rewound" &&
+	head2=$(git rev-parse --short HEAD) &&
+	echo "From $pwd/." > expect.err &&
+	echo "   $head1..$head2  master     -> origin/master" >> expect.err &&
+	(
+		cd downstream &&
+		git fetch >../actual.out 2>../actual.err
+	) &&
+	! test -s actual.out &&
+	test_cmp expect.err actual.err
+'
+
 test_done

Jens Lehmann

author

2016-06-15 22:50 UTC #11 lore raw

PATCH v2 6/7 submodule update: Don't fetch when the submodule commit is already present

In reply to Jens Lehmann

If the commit to be checked out on "git submodule update" has already been fetched in the submodule there is no need to run "git fetch" again. Since "git fetch" recently learned recursion (and the new on-demand mode to fetch commits recorded in the superproject is enabled by default) this will happen pretty often, thereby making the unconditional fetch during "git submodule update" unnecessary.

If the commit is not present in the submodule (e.g. the user disabled the fetch on-demand mode) the fetch will be run as before.

Signed-off-by Jens Lehmann <[email protected]>

---
 git-submodule.sh            |    5 ++++-
 t/t7406-submodule-update.sh |   20 ++++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletions(-)

git-submodule.sh +4 -1

index 3a13397..c16e275 100755
@@ -466,8 +466,11 @@ cmd_update()
 
 			if test -z "$nofetch"
 			then
+				# Run fetch only if $sha1 isn't present or it
+				# is not reachable from a ref.
 				(clear_local_git_env; cd "$path" &&
-					git-fetch) ||
+					((rev=$(git rev-list -n 1 $sha1 --not --all 2>/dev/null) &&
+					 test -z "$rev") || git-fetch)) ||
 				die "Unable to fetch in submodule path '$path'"
 			fi

t/t7406-submodule-update.sh +20 -0

index fa9d23a..bf7c788 100755
@@ -74,6 +74,26 @@ test_expect_success 'submodule update detaching the HEAD ' '
 	)
 '
 
+apos="'";
+test_expect_success 'submodule update does not fetch already present commits' '
+	(cd submodule &&
+	  echo line3 >> file &&
+	  git add file &&
+	  test_tick &&
+	  git commit -m "upstream line3"
+	) &&
+	(cd super/submodule &&
+	  head=$(git rev-parse --verify HEAD) &&
+	  echo "Submodule path ${apos}submodule$apos: checked out $apos$head$apos" > ../../expected &&
+	  git reset --hard HEAD~1
+	) &&
+	(cd super &&
+	  git submodule update > ../actual 2> ../actual.err
+	) &&
+	test_cmp expected actual &&
+	! test -s actual.err
+'
+
 test_expect_success 'submodule update --rebase staying on master' '
 	(cd super/submodule &&
 	  git checkout master

Signature · 1 line

1.7.4.1.291.g6060

Jens Lehmann

author

2016-06-15 22:50 UTC #12 lore raw

PATCH v2 7/7 fetch/pull: Describe --recurse-submodule restrictions in the BUGS section

In reply to Jens Lehmann

Using the --recurse-submodules option with fetch and pull might not always fetch all the submodule commits the user expects, as this will only work when the submodule is already checked out. Document that and warn that this is expected to change in the future.

Signed-off-by Jens Lehmann <[email protected]>

---
 Documentation/git-fetch.txt |    9 +++++++++
 Documentation/git-pull.txt  |    9 +++++++++
 2 files changed, 18 insertions(+), 0 deletions(-)

Documentation/git-fetch.txt +9 -0

index c76e313..f907ef5 100644
@@ -76,6 +76,15 @@ The `pu` branch will be updated even if it is does not fast-forward,
 because it is prefixed with a plus sign; `tmp` will not be.
 
 
+BUGS
+----
+Using --recurse-submodules can only fetch new commits in alrady checked
+out submodules right now. When e.g. upstream added a new submodule in the
+just fetched commits of the superproject the submodule itself can not be
+fetched, making it impossible to check out that submodule later whithout
+having to do a fetch again. This is expected to be fixed in a future git
+version.
+
 SEE ALSO
 --------
 linkgit:git-pull[1]

Documentation/git-pull.txt +9 -0

index c45efb3..1596d2b 100644
@@ -220,6 +220,15 @@ If you tried a pull which resulted in a complex conflicts and
 would want to start over, you can recover with 'git reset'.
 
 
+BUGS
+----
+Using --recurse-submodules can only fetch new commits in alrady checked
+out submodules right now. When e.g. upstream added a new submodule in the
+just fetched commits of the superproject the submodule itself can not be
+fetched, making it impossible to check out that submodule later whithout
+having to do a fetch again. This is expected to be fixed in a future git
+version.
+
 SEE ALSO
 --------
 linkgit:git-fetch[1], linkgit:git-merge[1], linkgit:git-config[1]

Signature · 1 line

1.7.4.1.291.g6060

Jens Lehmann

author

2016-06-15 22:50 UTC #13 lore raw

Re: [PATCH v2 7/7] fetch/pull: Describe --recurse-submodule restrictions in the BUGS section

Am 02.03.2011 00:04, schrieb Jens Lehmann:

Quoted 4 lines

Using the --recurse-submodules option with fetch and pull might not always fetch all the submodule commits the user expects, as this will only work when the submodule is already checked out. Document that and warn that this is expected to change in the future.

This patch needs the following diff to get rid of some typos. Will be fixed in v3.

--------8<--------

Documentation/git-fetch.txt +2 -2

index f907ef5..8669227 100644
@@ -78,10 +78,10 @@ because it is prefixed with a plus sign; `tmp` will not be.
 
 BUGS
 ----
-Using --recurse-submodules can only fetch new commits in alrady checked
+Using --recurse-submodules can only fetch new commits in already checked
 out submodules right now. When e.g. upstream added a new submodule in the
 just fetched commits of the superproject the submodule itself can not be
-fetched, making it impossible to check out that submodule later whithout
+fetched, making it impossible to check out that submodule later without
 having to do a fetch again. This is expected to be fixed in a future git
 version.

Documentation/git-pull.txt +2 -2

index 1596d2b..1aad8bf 100644
@@ -222,10 +222,10 @@ would want to start over, you can recover with 'git reset'
 
 BUGS
 ----
-Using --recurse-submodules can only fetch new commits in alrady checked
+Using --recurse-submodules can only fetch new commits in already checked
 out submodules right now. When e.g. upstream added a new submodule in the
 just fetched commits of the superproject the submodule itself can not be
-fetched, making it impossible to check out that submodule later whithout
+fetched, making it impossible to check out that submodule later without
 having to do a fetch again. This is expected to be fixed in a future git
 version.

Marc Branchaud

2016-06-15 22:50 UTC #14 lore raw

In reply to Jens Lehmann

Quoted 25 lines · Jens Lehmann

So here is version 2 of the on-demand recursive fetch for submodules.
Changes since the last version:
*) Add a name parameter to parse_fetch_recurse_submodules_arg() *) Enhance description of the "--submodule-default" fetch option *) Free the changed_submodule_paths string_list at the end of
fetch_populated_submodules() to aid libification and to make valgrind happy
*) Add a comment for the while loop in check_for_new_submodule_commits() *) Use "--all" instead of "--branches" and "--remotes" for rev-list to
include tags too
*) Compute the argc value properly instead of hard coding it in
check_for_new_submodule_commits()
*) Change size_t casts to intptr_t where appropriate
*) Use "rev-list --not --all" when checking if a submodule commit exists
instead of a simple lookup/rev-parse to make sure they are not only present but are reachable from a ref
*) Describe that fetch and pull only work on populated submodules for
now in the BUGS section of their man pages (done in a separate commit because this doesn't belong to a special commit in this series but also to the functionality that has already been merged)
Thanks to Jonathan an Junio for the feedback that lead to these changes.

Quoted 8 lines

Issues that were raised but not addressed yet:
*) Should this option be called "on-demand" or "changed" or maybe even
"updated"?
I have no strong feelings about this, me just thinks the name should tell somehow that only submodules where new commits are recorded in the superproject which aren't already present locally will be fetched. Opinions?

First, thanks again for this work!

I have a slight preference for "updated" or "updated-only", but no really strong feelings either way.

M.

Jens Lehmann

author

2016-06-15 22:50 UTC #15 lore raw

Am 02.03.2011 17:09, schrieb Marc Branchaud:

Quoted 9 lines · Jens Lehmann

Quoted 6 lines
*) Should this option be called "on-demand" or "changed" or maybe even
"updated"?
I have no strong feelings about this, me just thinks the name should tell somehow that only submodules where new commits are recorded in the superproject which aren't already present locally will be fetched. Opinions?
First, thanks again for this work!

Thanks for your feedback on this series, I'll include that in v3.

Quoted 2 lines

I have a slight preference for "updated" or "updated-only", but no really strong feelings either way.

I'm looking forward to hear what others think ...