fetch: add config to avoid fetching every branch in shallow repo

From: Harald Nordgren <[email protected]>

In a shallow, sparsely checked out clone of a repository with many branches, plain git pull can take minutes or hang outright, even though only one branch is actually being worked on.

Add fetch.shallow, off by default. When enabled, a fetch or pull for a shallow repository that isn't already scoped to specific refs fetches only the current branch's tracked upstream, instead of every branch the remote has. git pull ultimately runs such a fetch under the hood, so this fixes pull the same way. It has no effect once the repository is no longer shallow, and no effect on a fetch of a remote the current branch doesn't track, both fall back to the existing behavior.

This is opt-in rather than automatic because it changes what a plain fetch or pull leaves in refs/remotes/<name>/ for anyone who currently relies on it syncing every branch of a shallow remote, not just the one they are on. Scoping remote.<name>.fetch by hand already covers this for a single remote, but that requires knowing the config exists and applies it permanently, even to branches that are not currently checked out.

The remote's recorded default branch (remotes/<name>/HEAD) is kept up to date the same way it always is, only the other branches are skipped.

---
fetch: add fetch.shallow so pull doesn't fetch every branch on shallow repo

Add fetch.shallow config for big shallow repo, so git fetch/pull doesn't hang by fetching every branch.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v1
Pull-Request: https://github.com/git/git/pull/2412

 Documentation/config/fetch.adoc |  15 ++++
 builtin/fetch.c                 |  18 +++-
 t/t5537-fetch-shallow.sh        | 146 ++++++++++++++++++++++++++++++++
 3 files changed, 175 insertions(+), 4 deletions(-)

index 00435e9a16..e63e54a463 100644
@@ -145,3 +145,18 @@ remove the value for the `fetch.bundleCreationToken` value before fetching.
 `never`;;
 	Never create or modify the `remotes/<name>/HEAD` symbolic-ref.
 --
+
+`fetch.shallow`::
+	If true, and the repository is a shallow repository (see
+	linkgit:git-clone[1] `--depth`), a fetch or `git pull` that names no
+	explicit refspec and would otherwise fall back to the remote's
+	configured `remote.<name>.fetch` refspec instead fetches only the
+	current branch's upstream, when that upstream is on the remote being
+	fetched. This avoids negotiating history for every branch the remote
+	advertises, which can be slow on a shallow repository that tracks
+	many disjoint shallow histories. It has no effect on a fetch that
+	names an explicit remote or refspec, and no effect on a repository
+	that is not shallow. Defaults to false.
++
+`remotes/<name>/HEAD` is still kept up to date per `fetch.followRemoteHEAD`
+while this is in effect, only the other branches are skipped.

index 533fdfe7d8..b22f7fe5f4 100644
@@ -111,6 +111,7 @@ struct fetch_config {
 	int recurse_submodules;
 	int parallel;
 	int submodule_fetch_jobs;
+	int shallow;
 };
 
 static int git_fetch_config(const char *k, const char *v,

@@ -175,6 +176,11 @@ static int git_fetch_config(const char *k, const char *v,
 		return 0;
 	}
 
+	if (!strcmp(k, "fetch.shallow")) {
+		fetch_config->shallow = git_config_bool(k, v);
+		return 0;
+	}
+
 	if (!strcmp(k, "fetch.followremotehead")) {
 		if (!v)
 			return config_error_nonbool(k);

@@ -1958,15 +1964,19 @@ static int do_fetch(struct transport *transport,
 		refspec_ref_prefixes(rs, &transport_ls_refs_options.ref_prefixes);
 	} else {
 		struct branch *branch = branch_get(NULL);
+		int tracks_this_remote = branch && branch_has_merge_config(branch) &&
+			!strcmp(branch->remote_name, transport->remote->name);
+		int narrow_to_tracked_ref = config->shallow &&
+			is_repository_shallow(the_repository) && tracks_this_remote;
 
 		if (transport->remote->fetch.nr) {
-			refspec_ref_prefixes(&transport->remote->fetch,
-					     &transport_ls_refs_options.ref_prefixes);
+			if (!narrow_to_tracked_ref)
+				refspec_ref_prefixes(&transport->remote->fetch,
+						     &transport_ls_refs_options.ref_prefixes);
 			if (follow_remote_head != FOLLOW_REMOTE_NEVER)
 				do_set_head = 1;
 		}
-		if (branch && branch_has_merge_config(branch) &&
-		    !strcmp(branch->remote_name, transport->remote->name)) {
+		if (tracks_this_remote) {
 			int i;
 			for (i = 0; i < branch->merge_nr; i++) {
 				strvec_push(&transport_ls_refs_options.ref_prefixes,

base-commit: d38352cd43ab9745686d697872408bc3249a153f

gitgitgadget

Hi Harald

I think the sparse checkout is irrelevant? It is unclear to me if this is talking about a case where there are many branches in the remote repository and only one of them was cloned, then adding a second remote created a wildcard fetch refspec; or if there are intentionally lots of remote tracking branches in the local repository and you don't want to wait for them all to update. If it is the former then we should think how we can improve the behavior of "git remote add" in a sparse repository to prevent it adding a wildcard fetch refspec and instead setup the new remote to fetch only the branch(es) we're interested in.

Thanks

Phillip

Yeah likely. I cloned my fork of homebrew-core with --depth=1 and then added real upstream as a remote. Then I made my work on a feature branch.

Then tried to set my branch to track upstream, but it didn't work.

Then 'git pull' and it hung.

I didn't even consider that, that sounds like an interesting idea!

Harald

Would you suggest we do that instead of this, or in addition to this fix?

Seems maybe we only need the fix on "git remote add".

Harald

I'd say instead - if we can fix "git remote add" then the problem shouldn't arise in the first place which is better than having an opt-in config option that most users are probably unaware of.

Thanks

Phillip

That makes a lot of sense.

However, I think there is info missing on the local side that we can't get without making a network call to the newly added remote, like its branches (and default branch). How do we handle that?

Harald

Hi Harald

There is a "--track" option for "git remote add" to set up the branches that are fetched. Without "--fetch" we could refuse to add a new remote in a shallow repository unless the user tells us which branches to track. That is a bit of a pain, but I think it is better than adding a wildcard refspec that makes "git fetch" unusably slow. We could also perhaps look at the refspecs for the existing remotes and, if they are all fetching the same branch(es?) copy those.

With "--fetch" we could lookup the default branch on the remote and fetch that if "--track" isn't given.

Thanks

Phillip

Very good suggestions. "Avoid wildcards" is easy, but designing a suitable alternative ("only the ones we are interested in") may be harder.

Perhaps we want to have something similar in spirit to the "matching" mode 'git push' has, where the set of local branches we have defines the set of branches we are interested in? That is, when 'remote.*.fetch' is configured to signal that special mode, 'git fetch' would:

- Find each local branch that has its '@{upstream}' set to a branch at the remote we are fetching from.

- Fetch these branches at the remote that our local branches care about.

I said "in spirit" above, and I find it tempting to use ':' and '+:' as the special '<refspec>' to trigger this mode, to mimic what 'git push' does when using the local branches we have as the set of branches we care about. But there are important differences:

(1) The correspondence between local and remote-tracking branches is not one-to-one, as you can fork multiple local topics out of the same upstream branch. Maybe our 7 local branches build on top of only 2 branches we fetch from the remote, for example.

(2) Corollary. Unlike the matching mode in 'git push' where local branch 'B' is used to update branch 'B' at the remote (if it exists), this new mode in 'git fetch' only uses local branches as a guide to determine which branches to fetch from the remote. If our local branch 'B' builds on top of branch 'U' at the remote, it is branch 'U' we fetch and store as the 'refs/remotes/R/U' remote-tracking branch, where 'R' is the remote, and 'B' as the name does not get anywhere in the picture.

In other words, this is not "matching" at all, even though it takes inspiration from it. I do not know what it should be called, but I think it would be a useful addition.

Thanks.

Very interesting idea! I would like to suggest that the default branch of the upstream is always included as well, even if it's not the upstream of any branch yet.

I run this on every repo I work with

    git branch --set-upstream-to=upstream # upstream's default branch

and noticed on the shallow repo that it didn't work unless I first ran this

    git remote set-head upstream --auto

which was very annoying.

Harald

I can see that being useful fetch mode for a remote that we've already fetched from, but for a newly added remote there will be no local branches with their upstream set to it because "git branch --set-upstream-to" fails if the upstream does not already exist. So I like the idea for fetching from existing remotes, but it leaves us with a chicken-and-egg problem when adding new remotes, so I'm not sure how it would work in practice.

Thanks

Phillip

Just like with "git push there :", you prime the pump by explicitly doing something (for "push", you do "git push there mine" to express your preference to work with branch 'mine' and share it with the remote).

So if we are to allow customizing the refspec used for fetch with "git remote add", you might do:

    $ git remote add --fetch=: second https://ho.st/git/second

which creates:

[remote "second"]
url = https://ho.st/git/second
fetch = :

(Note: I am not sure if ":" is a good special token to express this mode of fetching, as I said earlier).

Your initial "git fetch second" without any other arguments will be a no-op. You may decide to work on top of their 'main' branch by running:

    $ git fetch second main:refs/remotes/second/main
    $ git checkout -b topic -t second/main

At that point, the local branch 'topic' is built on top of their 'main' branch by having its @{upstream} set to that remote-tracking branch. After that, running "git fetch" will update 'second/main' and no other remote-tracking branch.

I've wanted something similar for notes, so allow me to interject from the sidelines: it would be even nicer to still have the ability to map fetches (so that "git fetch second main" did the right thing, creating a useful remote tracking branch with the usual hierarchy; configurable, of course) on top of this. So, the first "git fetch second" does nothing, but "git fetch second main" + creating the branch does as you describe.

D. Ben Knoble

The thing is, the command line

    $ git fetch second main

has been used for the past 20 years as a "single-shot fetch" syntax that expresses that the user does not intend to keep interacting with the same 'main' branch or even the same 'second' repository, and for the "single-shot fetch", it is absolutely the wrong thing to create a remote-tracking branch.

It would be even worse if we created a remote 'second' and remote-tracking branch 'refs/remotes/second/main' when you ran

    $ git fetch https://ho.st/second main

Having said that, I suspect that the fact that you have the shorthand 'second' (i.e., you have "[remote "second"] url = ..." defined) may be a good enough sign that you expect to keep interacting with that repository, and some people might appreciate it if

    $ git fetch second main

created a remote-tracking branch "refs/remotes/second/main" automatically.

But we cannot suddenly start doing so without breaking people's expectations, and without a good transition plan. We need at least an escape hatch for users to say "No, this is a single-shot fetch; do not write the object anywhere other than FETCH_HEAD as we have always done".

Yes, perhaps I was a bit cavalier about that particular case; but in your other message you suggested allowing configurable refmaps, and that's what I was getting at previously without having the vocabulary for it. Thanks!

So anyway, I think fetch.<remote>.refmap is the right thing here, thanks!

D. Ben Knoble

Ahh, yes, if we introduce fetch.second.refmap that makes it pretend as if you gave a --refmap argument on the command line, your

	$ git fetch second main

will automatically do what you want it to do. I didn't realize that it was the implication of what I was outlining while preparing the response to Harald ;-)

Thanks.

From: Harald Nordgren <[email protected]>

git remote add sets a new remote up to fetch every branch by default. In an already shallow repository, that turns the next plain fetch or pull into a slow or hanging one, even when only one or two branches are ever used.

Add a special fetch refspec, "+:", that fetches only the branches our local branches are built on, plus the remote's default branch so a brand new remote is usable right away, without needing to first set anything up to track it. git remote add now uses it instead of the usual wildcard refspec whenever the repository is already shallow.

---
fetch: avoid fetching every branch of a new remote in a shallow repo

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v2:

* Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v2
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v1:

1: ec05d3e2d6 ! 1: f04f5ce219 fetch: add config to avoid fetching every branch in shallow repo

     @@ Metadata
      Author: Harald Nordgren <[email protected]>
      
       ## Commit message ##
     -    fetch: add config to avoid fetching every branch in shallow repo
     +    fetch: avoid fetching every branch of a new remote in a shallow repo
      
     -    In a shallow, sparsely checked out clone of a repository with many
     -    branches, plain git pull can take minutes or hang outright, even
     -    though only one branch is actually being worked on.
     +    git remote add sets a new remote up to fetch every branch by default.
     +    In an already shallow repository, that turns the next plain fetch or
     +    pull into a slow or hanging one, even when only one or two branches
     +    are ever used.
      
     -    Add fetch.shallow, off by default. When enabled, a fetch or pull for
     -    a shallow repository that isn't already scoped to specific refs
     -    fetches only the current branch's tracked upstream, instead of every
     -    branch the remote has. git pull ultimately runs such a fetch under
     -    the hood, so this fixes pull the same way. It has no effect once the
     -    repository is no longer shallow, and no effect on a fetch of a remote
     -    the current branch doesn't track, both fall back to the existing
     -    behavior.
     -
     -    This is opt-in rather than automatic because it changes what a plain
     -    fetch or pull leaves in refs/remotes/<name>/ for anyone who currently
     -    relies on it syncing every branch of a shallow remote, not just the
     -    one they are on. Scoping remote.<name>.fetch by hand already covers
     -    this for a single remote, but that requires knowing the config exists
     -    and applies it permanently, even to branches that are not currently
     -    checked out.
     -
     -    The remote's recorded default branch (remotes/<name>/HEAD) is kept up
     -    to date the same way it always is, only the other branches are
     -    skipped.
     +    Add a special fetch refspec, "+:", that fetches only the branches our
     +    local branches are built on, plus the remote's default branch so a
     +    brand new remote is usable right away, without needing to first set
     +    anything up to track it. git remote add now uses it instead of the
     +    usual wildcard refspec whenever the repository is already shallow.
      
          Signed-off-by: Harald Nordgren <[email protected]>
      
     - ## Documentation/config/fetch.adoc ##
     -@@ Documentation/config/fetch.adoc: remove the value for the `fetch.bundleCreationToken` value before fetching.
     - `never`;;
     - 	Never create or modify the `remotes/<name>/HEAD` symbolic-ref.
     - --
     -+
     -+`fetch.shallow`::
     -+	If true, and the repository is a shallow repository (see
     -+	linkgit:git-clone[1] `--depth`), a fetch or `git pull` that names no
     -+	explicit refspec and would otherwise fall back to the remote's
     -+	configured `remote.<name>.fetch` refspec instead fetches only the
     -+	current branch's upstream, when that upstream is on the remote being
     -+	fetched. This avoids negotiating history for every branch the remote
     -+	advertises, which can be slow on a shallow repository that tracks
     -+	many disjoint shallow histories. It has no effect on a fetch that
     -+	names an explicit remote or refspec, and no effect on a repository
     -+	that is not shallow. Defaults to false.
     + ## Documentation/git-remote.adoc ##
     +@@ Documentation/git-remote.adoc: Add a remote named _<name>_ for the repository at
     + _<URL>_.  The command `git fetch <name>` can then be used to create and
     + update remote-tracking branches `<name>/<branch>`.
     + +
     ++If the repository is already a shallow repository (see linkgit:git-clone[1]
     ++`--depth`) and neither `-t` nor `--mirror` is given, the remote is set up
     ++with the special tracking refspec (see the _<refspec>_ entry in
     ++linkgit:git-fetch[1]) instead of the default glob refspec, so that
     ++`git fetch <name>` does not need to negotiate history for every branch
     ++the remote has.
     +++
     + With `-f` option, `git fetch <name>` is run immediately after
     + the remote information is set up.
     + +
     +
     + ## Documentation/pull-fetch-param.adoc ##
     +@@ Documentation/pull-fetch-param.adoc: Negative refspecs can themselves be pattern refspecs. However, they may only
     + contain a _<src>_ and do not specify a _<dst>_. Fully spelled out hex object
     + names are also not supported.
     + +
     ++The special refspec `:` (or `+:` to allow non-fast-forward updates)
     ++directs Git to fetch whichever branches at the remote our local branches
     ++have as their `@{upstream}`, plus the remote's default branch, which is
     ++always included so that it is available even before anything is set up
     ++to track it. Unlike the same syntax when pushing, the local branch name
     ++plays no role here beyond identifying which remote branch to fetch: the
     ++result is written to `refs/remotes/<remote>/<branch>` using the remote's
     ++own branch name, and several local branches built on the same remote
     ++branch only cause it to be fetched once.
      ++
     -+`remotes/<name>/HEAD` is still kept up to date per `fetch.followRemoteHEAD`
     -+while this is in effect, only the other branches are skipped.
     + `tag <tag>` means the same as `refs/tags/<tag>:refs/tags/<tag>`;
     + it requests fetching everything up to the given tag.
     + +
      
       ## builtin/fetch.c ##
     -@@ builtin/fetch.c: struct fetch_config {
     - 	int recurse_submodules;
     - 	int parallel;
     - 	int submodule_fetch_jobs;
     -+	int shallow;
     - };
     - 
     - static int git_fetch_config(const char *k, const char *v,
     -@@ builtin/fetch.c: static int git_fetch_config(const char *k, const char *v,
     - 		return 0;
     +@@ builtin/fetch.c: static void filter_prefetch_refspec(struct refspec *rs)
       	}
     + }
       
     -+	if (!strcmp(k, "fetch.shallow")) {
     -+		fetch_config->shallow = git_config_bool(k, v);
     -+		return 0;
     ++static void fetch_tracked_branches(struct remote *remote,
     ++				   const struct ref *remote_refs,
     ++				   struct ref ***tail,
     ++				   char **default_branch_out,
     ++				   int force)
     ++{
     ++	struct string_list tracked = STRING_LIST_INIT_DUP;
     ++	struct string_list_item *item;
     ++	struct ref *head;
     ++	char *default_branch = NULL;
     ++
     ++	branches_tracking_remote(remote, &tracked);
     ++
     ++	head = get_remote_ref(remote_refs, "HEAD");
     ++	if (head && head->symref && *head->symref) {
     ++		default_branch = xstrdup(head->symref);
     ++		string_list_insert(&tracked, default_branch);
     ++	}
     ++	free_one_ref(head);
     ++
     ++	for_each_string_list_item(item, &tracked) {
     ++		struct refspec_item ref = { .force = force };
     ++		const char *branch_name = item->string;
     ++		int is_default_branch = default_branch &&
     ++			!strcmp(item->string, default_branch);
     ++
     ++		skip_prefix(branch_name, "refs/heads/", &branch_name);
     ++		ref.src = xstrdup(is_default_branch ? "HEAD" : item->string);
     ++		ref.dst = xstrfmt("refs/remotes/%s/%s", remote->name, branch_name);
     ++		get_fetch_map(remote_refs, &ref, tail, 1);
     ++		free(ref.src);
     ++		free(ref.dst);
     ++	}
     ++
     ++	if (default_branch && default_branch_out) {
     ++		const char *branch_name = default_branch;
     ++		skip_prefix(branch_name, "refs/heads/", &branch_name);
     ++		*default_branch_out = xstrdup(branch_name);
      +	}
      +
     - 	if (!strcmp(k, "fetch.followremotehead")) {
     - 		if (!v)
     - 			return config_error_nonbool(k);
     ++	free(default_branch);
     ++	string_list_clear(&tracked, 0);
     ++}
     ++
     + static struct ref *get_ref_map(struct remote *remote,
     + 			       const struct ref *remote_refs,
     + 			       struct refspec *rs,
     +-			       int tags, int *autotags)
     ++			       int tags, int *autotags,
     ++			       char **bootstrap_head_branch)
     + {
     + 	int i;
     + 	struct ref *rm;
     +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
     + 		     /* Note: has_merge implies non-NULL branch->remote_name */
     + 		     (has_merge && !strcmp(branch->remote_name, remote->name)))) {
     + 			for (i = 0; i < remote->fetch.nr; i++) {
     ++				if (remote->fetch.items[i].tracking) {
     ++					fetch_tracked_branches(remote, remote_refs, &tail,
     ++								bootstrap_head_branch,
     ++								remote->fetch.items[i].force);
     ++					continue;
     ++				}
     + 				get_fetch_map(remote_refs, &remote->fetch.items[i], &tail, 0);
     + 				if (remote->fetch.items[i].dst &&
     + 				    remote->fetch.items[i].dst[0])
     +@@ builtin/fetch.c: static void warn_set_head(const char *remote, const char *head_name,
     + }
     + 
     + static int set_head(const struct ref *remote_refs, struct remote *remote,
     +-			int follow_remote_head)
     ++			int follow_remote_head, const char *known_head_branch)
     + {
     + 	int result = 0, create_only, baremirror, was_detached;
     + 	struct strbuf b_head = STRBUF_INIT, b_remote_head = STRBUF_INIT,
     + 		      b_local_head = STRBUF_INIT;
     + 	const char *no_warn_branch = remote->no_warn_branch;
     + 	char *head_name = NULL;
     +-	struct ref *ref, *matches;
     ++	struct ref *ref, *matches = NULL;
     + 	struct ref *fetch_map = NULL, **fetch_map_tail = &fetch_map;
     + 	struct refspec_item refspec = {
     + 		.force = 0,
     +@@ builtin/fetch.c: static int set_head(const struct ref *remote_refs, struct remote *remote,
     + 	struct string_list heads = STRING_LIST_INIT_DUP;
     + 	struct ref_store *refs = get_main_ref_store(the_repository);
     + 
     +-	get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);
     +-	matches = guess_remote_head(find_ref_by_name(remote_refs, "HEAD"),
     +-				    fetch_map, REMOTE_GUESS_HEAD_ALL);
     +-	for (ref = matches; ref; ref = ref->next) {
     +-		string_list_append(&heads, strip_refshead(ref->name));
     +-	}
     ++	if (known_head_branch) {
     ++		head_name = xstrdup(known_head_branch);
     ++	} else {
     ++		get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);
     ++		matches = guess_remote_head(find_ref_by_name(remote_refs, "HEAD"),
     ++					    fetch_map, REMOTE_GUESS_HEAD_ALL);
     ++		for (ref = matches; ref; ref = ref->next) {
     ++			string_list_append(&heads, strip_refshead(ref->name));
     ++		}
     + 
     +-	if (!heads.nr)
     +-		result = 1;
     +-	else if (heads.nr > 1)
     +-		result = 1;
     +-	else
     +-		head_name = xstrdup(heads.items[0].string);
     ++		if (!heads.nr)
     ++			result = 1;
     ++		else if (heads.nr > 1)
     ++			result = 1;
     ++		else
     ++			head_name = xstrdup(heads.items[0].string);
     ++	}
     + 
     + 	if (!head_name)
     + 		goto cleanup;
     +@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     + 	struct strmap rejected_refs = STRMAP_INIT;
     + 	int summary_width = 0;
     + 	int follow_remote_head;
     ++	char *bootstrap_head_branch = NULL;
     + 
     + 	if (tags == TAGS_DEFAULT) {
     + 		if (transport->remote->fetch_tags == 2)
      @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
       		refspec_ref_prefixes(rs, &transport_ls_refs_options.ref_prefixes);
       	} else {
       		struct branch *branch = branch_get(NULL);
      +		int tracks_this_remote = branch && branch_has_merge_config(branch) &&
      +			!strcmp(branch->remote_name, transport->remote->name);
     -+		int narrow_to_tracked_ref = config->shallow &&
     -+			is_repository_shallow(the_repository) && tracks_this_remote;
     ++		int tracking_refspec = transport->remote->fetch.nr == 1 &&
     ++			transport->remote->fetch.items[0].tracking;
       
     - 		if (transport->remote->fetch.nr) {
     --			refspec_ref_prefixes(&transport->remote->fetch,
     --					     &transport_ls_refs_options.ref_prefixes);
     -+			if (!narrow_to_tracked_ref)
     -+				refspec_ref_prefixes(&transport->remote->fetch,
     -+						     &transport_ls_refs_options.ref_prefixes);
     - 			if (follow_remote_head != FOLLOW_REMOTE_NEVER)
     - 				do_set_head = 1;
     +-		if (transport->remote->fetch.nr) {
     ++		if (tracking_refspec) {
     ++			struct string_list tracked = STRING_LIST_INIT_DUP;
     ++			struct string_list_item *item;
     ++
     ++			branches_tracking_remote(transport->remote, &tracked);
     ++			for_each_string_list_item(item, &tracked)
     ++				strvec_push(&transport_ls_refs_options.ref_prefixes,
     ++					    item->string);
     ++			string_list_clear(&tracked, 0);
     ++			strvec_push(&transport_ls_refs_options.ref_prefixes, "HEAD");
     ++		} else if (transport->remote->fetch.nr) {
     + 			refspec_ref_prefixes(&transport->remote->fetch,
     + 					     &transport_ls_refs_options.ref_prefixes);
     +-			if (follow_remote_head != FOLLOW_REMOTE_NEVER)
     +-				do_set_head = 1;
       		}
      -		if (branch && branch_has_merge_config(branch) &&
      -		    !strcmp(branch->remote_name, transport->remote->name)) {
     ++		if (transport->remote->fetch.nr &&
     ++		    follow_remote_head != FOLLOW_REMOTE_NEVER)
     ++			do_set_head = 1;
      +		if (tracks_this_remote) {
       			int i;
       			for (i = 0; i < branch->merge_nr; i++) {
       				strvec_push(&transport_ls_refs_options.ref_prefixes,
     +@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     + 	transport_ls_refs_options_release(&transport_ls_refs_options);
     + 
     + 	ref_map = get_ref_map(transport->remote, remote_refs, rs,
     +-			      tags, &autotags);
     ++			      tags, &autotags, &bootstrap_head_branch);
     ++
     + 	if (!update_head_ok)
     + 		check_not_current_branch(ref_map);
     + 
     +@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     + 		 * Way too many cases where this can go wrong so let's just
     + 		 * ignore errors and fail silently for now.
     + 		 */
     +-		set_head(remote_refs, transport->remote, follow_remote_head);
     ++		set_head(remote_refs, transport->remote, follow_remote_head,
     ++			 bootstrap_head_branch);
     + 	}
     + 
     + cleanup:
     ++	free(bootstrap_head_branch);
     + 	/*
     + 	 * When using batched updates, we want to commit the non-rejected
     + 	 * updates and also handle the rejections.
      
     - ## t/t5537-fetch-shallow.sh ##
     -@@ t/t5537-fetch-shallow.sh: commit() {
     - 	git commit -m "$1"
     + ## builtin/remote.c ##
     +@@
     + #include "rebase.h"
     + #include "refs.h"
     + #include "refspec.h"
     ++#include "shallow.h"
     + #include "odb.h"
     + #include "strvec.h"
     + #include "commit-reach.h"
     +@@ builtin/remote.c: static void add_branch(const char *key, const char *branchname,
     + 		       const char *remotename, int mirror, struct strbuf *tmp)
     + {
     + 	strbuf_reset(tmp);
     +-	strbuf_addch(tmp, '+');
     +-	if (mirror)
     +-		strbuf_addf(tmp, "refs/%s:refs/%s",
     +-				branchname, branchname);
     +-	else
     +-		strbuf_addf(tmp, "refs/heads/%s:refs/remotes/%s/%s",
     +-				branchname, remotename, branchname);
     ++	if (!strcmp(branchname, ":")) {
     ++		strbuf_addstr(tmp, "+:");
     ++	} else {
     ++		strbuf_addch(tmp, '+');
     ++		if (mirror)
     ++			strbuf_addf(tmp, "refs/%s:refs/%s",
     ++					branchname, branchname);
     ++		else
     ++			strbuf_addf(tmp, "refs/heads/%s:refs/remotes/%s/%s",
     ++					branchname, remotename, branchname);
     ++	}
     + 	repo_config_set_multivar(the_repository, key, tmp->buf, "^$", 0);
       }
       
     -+check_upstream_refs () {
     -+	git for-each-ref --format="%(refname)" refs/remotes/upstream/ >actual &&
     -+	cat >expect &&
     -+	test_cmp expect actual
     -+}
     +@@ builtin/remote.c: static int add(int argc, const char **argv, const char *prefix,
     + 		strbuf_reset(&buf);
     + 		strbuf_addf(&buf, "remote.%s.fetch", name);
     + 		if (track.nr == 0)
     +-			string_list_append(&track, "*");
     +-		for (size_t i = 0; i < track.nr; i++) {
     ++			string_list_append(&track,
     ++				mirror == MIRROR_NONE &&
     ++				is_repository_shallow(the_repository) ?
     ++				":" : "*");
     ++		for (size_t i = 0; i < track.nr; i++)
     + 			add_branch(buf.buf, track.items[i].string,
     + 				   name, mirror, &buf2);
     +-		}
     + 	}
     + 
     + 	if (mirror & MIRROR_PUSH) {
     +
     + ## refspec.c ##
     +@@ refspec.c: static int parse_refspec(struct refspec_item *item, const char *refspec,
     + 		return 1;
     + 	}
     + 
     ++	if (fetch && rhs == lhs && rhs[1] == '\0') {
     ++		item->tracking = 1;
     ++		return 1;
     ++	}
     ++
     + 	if (rhs) {
     + 		size_t rlen = strlen(++rhs);
     + 		is_glob = (1 <= rlen && strchr(rhs, '*'));
     +@@ refspec.c: void refspec_item_clear(struct refspec_item *item)
     + 	item->force = 0;
     + 	item->pattern = 0;
     + 	item->matching = 0;
     ++	item->tracking = 0;
     + 	item->exact_sha1 = 0;
     + }
     + 
     +@@ refspec.c: static int refspec_find_negative_match(struct refspec *rs, struct refspec_item *
     + 		struct refspec_item *refspec = &rs->items[i];
     + 		char *expn_name;
     + 
     +-		if (refspec->negative)
     ++		if (refspec->negative || refspec->tracking)
     + 			continue;
     + 
     + 		/* Note the reversal of src and dst */
     +
     + ## refspec.h ##
     +@@ refspec.h: struct refspec_item {
     + 	unsigned matching : 1;
     + 	unsigned exact_sha1 : 1;
     + 	unsigned negative : 1;
     ++	unsigned tracking : 1;
     + 
     + 	char *src;
     + 	char *dst;
     +
     + ## remote.c ##
     +@@ remote.c: void ref_push_report_free(struct ref_push_report *report)
     + 
     + int remote_find_tracking(struct remote *remote, struct refspec_item *refspec)
     + {
     ++	if (remote->fetch.nr == 1 && remote->fetch.items[0].tracking) {
     ++		struct strbuf prefix = STRBUF_INIT;
     ++		const char *branch_name;
     ++		int ret = -1;
     ++
     ++		strbuf_addf(&prefix, "refs/remotes/%s/", remote->name);
     ++		if (!refspec->src) {
     ++			if (refspec->dst &&
     ++			    skip_prefix(refspec->dst, prefix.buf, &branch_name)) {
     ++				refspec->src = xstrfmt("refs/heads/%s", branch_name);
     ++				refspec->force = 1;
     ++				ret = 0;
     ++			}
     ++		} else if (skip_prefix(refspec->src, "refs/heads/", &branch_name)) {
     ++			free(refspec->dst);
     ++			refspec->dst = xstrfmt("%s%s", prefix.buf, branch_name);
     ++			refspec->force = 1;
     ++			ret = 0;
     ++		}
     ++		strbuf_release(&prefix);
     ++		return ret;
     ++	}
     ++
     + 	return refspec_find_match(&remote->fetch, refspec);
     + }
     + 
     +@@ remote.c: int branch_merge_matches(struct branch *branch,
     + 	return refname_match(branch->merge[i]->src, refname);
     + }
     + 
     ++struct branches_tracking_remote_cb_data {
     ++	struct remote *remote;
     ++	struct string_list *tracked;
     ++};
     ++
     ++static int add_if_tracking_remote(const struct reference *ref, void *cb_data)
     ++{
     ++	struct branches_tracking_remote_cb_data *data = cb_data;
     ++	struct branch *branch;
     ++
     ++	branch = branch_get(ref->name);
     ++	if (!branch_has_merge_config(branch) ||
     ++	    strcmp(branch->remote_name, data->remote->name))
     ++		return 0;
      +
     -+check_upstream_head () {
     -+	git symbolic-ref refs/remotes/upstream/HEAD >actual &&
     -+	echo "refs/remotes/upstream/$1" >expect &&
     -+	test_cmp expect actual
     ++	for (int i = 0; i < branch->merge_nr; i++)
     ++		string_list_insert(data->tracked, branch->merge[i]->src);
     ++
     ++	return 0;
      +}
      +
     -+check_same_tip () {
     -+	git log --oneline -1 "$1" >expect &&
     -+	git -C "$2" log --oneline -1 "$3" >actual &&
     -+	test_cmp expect actual
     ++void branches_tracking_remote(struct remote *remote, struct string_list *tracked)
     ++{
     ++	struct branches_tracking_remote_cb_data data = { remote, tracked };
     ++
     ++	refs_for_each_branch_ref(get_main_ref_store(the_repository),
     ++				  add_if_tracking_remote, &data);
      +}
      +
     - test_expect_success 'setup' '
     - 	commit 1 &&
     - 	commit 2 &&
     -@@ t/t5537-fetch-shallow.sh: test_expect_success 'fetch --deepen does not truncate' '
     - 	test_cmp expect actual
     + __attribute__((format (printf,2,3)))
     + static char *error_buf(struct strbuf *err, const char *fmt, ...)
     + {
     +@@ remote.c: int get_fetch_map(const struct ref *remote_refs,
     + {
     + 	struct ref *ref_map, **rmp;
     + 
     +-	if (refspec->negative)
     ++	if (refspec->negative || refspec->tracking)
     + 		return 0;
     + 
     + 	if (refspec->pattern) {
     +
     + ## remote.h ##
     +@@ remote.h: int branch_has_merge_config(struct branch *branch);
     + 
     + int branch_merge_matches(struct branch *, int n, const char *);
     + 
     ++/* fills tracked with the refname of every local branch's upstream on remote */
     ++void branches_tracking_remote(struct remote *remote, struct string_list *tracked);
     ++
     + /* list of the remote in a group as configured */
     + struct remote_group_data {
     + 	const char *name;
     +
     + ## t/meson.build ##
     +@@ t/meson.build: integration_tests = [
     +   't5582-fetch-negative-refspec.sh',
     +   't5583-push-branches.sh',
     +   't5584-http-429-retry.sh',
     ++  't5585-fetch-tracking-refspec.sh',
     +   't5600-clone-fail-cleanup.sh',
     +   't5601-clone.sh',
     +   't5602-clone-remote-exec.sh',
     +
     + ## t/t5505-remote.sh ##
     +@@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only' '
     + 	test_grep ! "\[blob:none\]" out
       '
       
     -+test_expect_success 'fetch.shallow setup' '
     -+	git branch narrow-side &&
     -+	git clone --no-local --depth=1 --branch main --single-branch \
     -+		.git narrow-default &&
     ++test_expect_success 'add remote -t keeps an explicit refspec in a shallow repository' '
     ++	test_when_finished "rm -rf shallow-add" &&
      +	git clone --no-local --depth=1 --branch main --single-branch \
     -+		.git narrow-enabled &&
     -+	(
     -+		cd narrow-default &&
     -+		git remote add upstream ../.git &&
     -+		git fetch --depth=1 upstream main:refs/remotes/upstream/main &&
     -+		git branch --set-upstream-to=upstream/main main
     -+	) &&
     ++		one shallow-add &&
      +	(
     -+		cd narrow-enabled &&
     -+		git remote add upstream ../.git &&
     -+		git fetch --depth=1 upstream main:refs/remotes/upstream/main &&
     -+		git branch --set-upstream-to=upstream/main main &&
     -+		git config fetch.shallow true
     ++		cd shallow-add &&
     ++		git remote add -t main upstream ../two &&
     ++		test_cmp_config "+refs/heads/main:refs/remotes/upstream/main" \
     ++			remote.upstream.fetch
      +	)
      +'
      +
     -+test_expect_success 'a refspec-less fetch expands to the configured refspec by default' '
     ++test_expect_success 'add remote keeps the wildcard refspec in a full repository' '
     ++	test_when_finished "rm -rf full-add" &&
     ++	git clone --no-local one full-add &&
      +	(
     -+		cd narrow-default &&
     -+		git fetch upstream &&
     -+		check_upstream_refs <<-\EOF
     -+		refs/remotes/upstream/HEAD
     -+		refs/remotes/upstream/main
     -+		refs/remotes/upstream/narrow-side
     -+		EOF
     ++		cd full-add &&
     ++		git remote add upstream ../two &&
     ++		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
     ++			remote.upstream.fetch
      +	)
      +'
      +
     -+test_expect_success 'fetch.shallow=true limits a refspec-less fetch to the tracked branch' '
     ++test_expect_success 'a remote added in a shallow repository defaults to the tracking refspec' '
     ++	test_when_finished "rm -rf shallow-add" &&
     ++	git clone --no-local --depth=1 --branch main --single-branch \
     ++		one shallow-add &&
      +	(
     -+		cd narrow-enabled &&
     ++		cd shallow-add &&
     ++		git remote add upstream ../two &&
     ++		test_cmp_config "+:" remote.upstream.fetch &&
      +		git fetch upstream &&
     -+		check_upstream_refs <<-\EOF
     ++		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
     ++		cat >expect <<-\EOF &&
      +		refs/remotes/upstream/HEAD
      +		refs/remotes/upstream/main
      +		EOF
     ++		test_cmp expect actual &&
     ++		git branch --set-upstream-to=upstream &&
     ++		test_cmp_config upstream branch.main.remote &&
     ++		test_cmp_config refs/heads/main branch.main.merge
      +	)
      +'
      +
     -+test_expect_success 'fetch.shallow=true still creates refs/remotes/<remote>/HEAD' '
     + test_expect_success 'check remote-tracking' '
     + 	(
     + 		cd test &&
     +
     + ## t/t5585-fetch-tracking-refspec.sh (new) ##
     +@@
     ++#!/bin/sh
     ++
     ++test_description='"git fetch" with the tracking refspec ":" (or "+:")
     ++
     ++The special refspec fetches whichever branches at the remote our local
     ++branches have as their @{upstream}, plus the default branch of that
     ++remote, which is always included so that it is available even before
     ++anything is set up to track it. git remote add uses it by default for
     ++a new remote in an already shallow repository, so that adding a
     ++second remote there does not turn the next plain fetch or pull into
     ++one that negotiates history for every branch the remote has.
     ++'
     ++
     ++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
     ++export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
     ++
     ++. ./test-lib.sh
     ++
     ++test_expect_success 'setup' '
     ++	test_commit main-1 &&
     ++	test_commit main-2 &&
     ++	git checkout -b side main-1 &&
     ++	test_commit side-1 &&
     ++	git checkout -b next main-1 &&
     ++	test_commit next-1 &&
     ++	git checkout main
     ++'
     ++
     ++test_expect_success 'clone shallow and single-branch, then add a second remote' '
     ++	git clone --no-local --depth=1 --branch main --single-branch . client &&
      +	(
     -+		cd narrow-enabled &&
     -+		git symbolic-ref -d refs/remotes/upstream/HEAD &&
     -+		git fetch upstream &&
     -+		check_upstream_head main
     ++		cd client &&
     ++		git remote add upstream .. &&
     ++		test_cmp_config "+:" remote.upstream.fetch
      +	)
      +'
      +
     -+test_expect_success 'fetch.shallow=true with followRemoteHEAD=always corrects a stale HEAD' '
     -+	test_when_finished \
     -+		"git -C narrow-enabled update-ref -d refs/remotes/upstream/stale-branch" &&
     ++test_expect_success 'fetching the new remote does not need every branch it has' '
      +	(
     -+		cd narrow-enabled &&
     -+		git update-ref refs/remotes/upstream/stale-branch refs/remotes/upstream/main &&
     -+		git symbolic-ref refs/remotes/upstream/HEAD refs/remotes/upstream/stale-branch &&
     -+		git -c fetch.followRemoteHEAD=always fetch upstream &&
     -+		check_upstream_head main
     ++		cd client &&
     ++		git fetch upstream &&
     ++		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
     ++		cat >expect <<-\EOF &&
     ++		refs/remotes/upstream/HEAD
     ++		refs/remotes/upstream/main
     ++		EOF
     ++		test_cmp expect actual
      +	)
      +'
      +
     -+test_expect_success 'fetch.shallow=true still updates the tracked branch' '
     -+	commit 5 &&
     -+	git -C narrow-enabled fetch upstream &&
     -+	check_same_tip main narrow-enabled refs/remotes/upstream/main
     ++test_expect_success 'set-upstream-to now resolves right after that first fetch' '
     ++	(
     ++		cd client &&
     ++		git branch --set-upstream-to=upstream &&
     ++		test_cmp_config upstream branch.main.remote &&
     ++		test_cmp_config refs/heads/main branch.main.merge
     ++	)
      +'
      +
     -+test_expect_success 'fetch.shallow=true keeps git pull narrowed too' '
     -+	test_when_finished "git branch -D narrow-side" &&
     -+	commit 6 &&
     ++test_expect_success 'a branch checked out from a one-time fetch is kept updated by later plain fetches' '
     ++	(
     ++		cd client &&
     ++		git fetch upstream side:refs/remotes/upstream/side &&
     ++		git branch side-topic upstream/side
     ++	) &&
     ++	git checkout side &&
     ++	test_commit side-2 &&
     ++	git checkout main &&
      +	(
     -+		cd narrow-enabled &&
     -+		git pull &&
     -+		check_upstream_refs <<-\EOF
     ++		cd client &&
     ++		git fetch upstream &&
     ++		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
     ++		cat >expect <<-\EOF &&
      +		refs/remotes/upstream/HEAD
      +		refs/remotes/upstream/main
     ++		refs/remotes/upstream/side
      +		EOF
     -+	) &&
     -+	check_same_tip main narrow-enabled HEAD
     ++		test_cmp expect actual &&
     ++		git rev-parse refs/remotes/upstream/side >actual-oid &&
     ++		git -C .. rev-parse side >expect-oid &&
     ++		test_cmp expect-oid actual-oid
     ++	)
      +'
      +
     -+test_expect_success 'fetch.shallow=true has no effect on a non-shallow repository' '
     -+	git clone --no-local --branch main --single-branch .git narrow-full &&
     -+	(
     -+		cd narrow-full &&
     -+		git rev-parse --is-shallow-repository >actual &&
     -+		echo false >expect &&
     -+		test_cmp expect actual &&
     -+		git remote add upstream ../.git &&
     -+		git fetch upstream &&
     -+		git branch --set-upstream-to=upstream/main main &&
     -+		git config fetch.shallow true
     -+	) &&
     -+	test_when_finished "git branch -D narrow-full-side" &&
     -+	git branch narrow-full-side &&
     ++test_expect_success 'a second branch tracking the same upstream branch does not fetch it twice' '
      +	(
     -+		cd narrow-full &&
     ++		cd client &&
     ++		git branch side-topic-2 upstream/side &&
      +		git fetch upstream &&
     -+		check_upstream_refs <<-\EOF
     ++		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
     ++		cat >expect <<-\EOF &&
      +		refs/remotes/upstream/HEAD
      +		refs/remotes/upstream/main
     -+		refs/remotes/upstream/narrow-full-side
     ++		refs/remotes/upstream/side
      +		EOF
     ++		test_cmp expect actual
      +	)
      +'
      +
     -+test_expect_success 'fetch.shallow=true only narrows a fetch of the tracked remote' '
     -+	test_when_finished "git branch -D other-side" &&
     -+	git branch other-side &&
     -+	git clone --no-local --depth=1 --branch main --single-branch \
     -+		.git narrow-other-remote &&
     ++test_expect_success 'a branch tracking a different remote is not fetched from upstream' '
      +	(
     -+		cd narrow-other-remote &&
     -+		git remote add upstream ../.git &&
     -+		git config fetch.shallow true &&
     ++		cd client &&
     ++		git remote add other .. &&
     ++		git fetch other next:refs/remotes/other/next &&
     ++		git branch next-topic other/next &&
      +		git fetch upstream &&
     -+		check_upstream_refs <<-\EOF
     ++		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
     ++		cat >expect <<-\EOF &&
      +		refs/remotes/upstream/HEAD
      +		refs/remotes/upstream/main
     -+		refs/remotes/upstream/other-side
     ++		refs/remotes/upstream/side
      +		EOF
     ++		test_cmp expect actual
      +	)
      +'
      +
     - . "$TEST_DIRECTORY"/lib-httpd.sh
     - start_httpd
     - 
     ++test_expect_success 'git remote show does not choke on the tracking refspec' '
     ++	(
     ++		cd client &&
     ++		git remote show upstream
     ++	)
     ++'
     ++
     ++test_done

 Documentation/git-remote.adoc       |   7 ++
 Documentation/pull-fetch-param.adoc |  10 +++
 builtin/fetch.c                     | 117 +++++++++++++++++++++-----
 builtin/remote.c                    |  27 +++---
 refspec.c                           |   8 +-
 refspec.h                           |   1 +
 remote.c                            |  54 +++++++++++-
 remote.h                            |   3 +
 t/meson.build                       |   1 +
 t/t5505-remote.sh                   |  44 ++++++++++
 t/t5585-fetch-tracking-refspec.sh   | 124 ++++++++++++++++++++++++++++
 11 files changed, 362 insertions(+), 34 deletions(-)
 create mode 100755 t/t5585-fetch-tracking-refspec.sh

index eaae30aa88..7856ccf54f 100644
@@ -52,6 +52,13 @@ Add a remote named _<name>_ for the repository at
 _<URL>_.  The command `git fetch <name>` can then be used to create and
 update remote-tracking branches `<name>/<branch>`.
 +
+If the repository is already a shallow repository (see linkgit:git-clone[1]
+`--depth`) and neither `-t` nor `--mirror` is given, the remote is set up
+with the special tracking refspec (see the _<refspec>_ entry in
+linkgit:git-fetch[1]) instead of the default glob refspec, so that
+`git fetch <name>` does not need to negotiate history for every branch
+the remote has.
++
 With `-f` option, `git fetch <name>` is run immediately after
 the remote information is set up.
 +

index d903dc8900..e1a42152d9 100644
@@ -48,6 +48,16 @@ Negative refspecs can themselves be pattern refspecs. However, they may only
 contain a _<src>_ and do not specify a _<dst>_. Fully spelled out hex object
 names are also not supported.
 +
+The special refspec `:` (or `+:` to allow non-fast-forward updates)
+directs Git to fetch whichever branches at the remote our local branches
+have as their `@{upstream}`, plus the remote's default branch, which is
+always included so that it is available even before anything is set up
+to track it. Unlike the same syntax when pushing, the local branch name
+plays no role here beyond identifying which remote branch to fetch: the
+result is written to `refs/remotes/<remote>/<branch>` using the remote's
+own branch name, and several local branches built on the same remote
+branch only cause it to be fetched once.
++
 `tag <tag>` means the same as `refs/tags/<tag>:refs/tags/<tag>`;
 it requests fetching everything up to the given tag.
 +

index de989ea3ba..e407d67037 100644
@@ -16,6 +16,7 @@
 #include "rebase.h"
 #include "refs.h"
 #include "refspec.h"
+#include "shallow.h"
 #include "odb.h"
 #include "strvec.h"
 #include "commit-reach.h"

@@ -127,13 +128,17 @@ static void add_branch(const char *key, const char *branchname,
 		       const char *remotename, int mirror, struct strbuf *tmp)
 {
 	strbuf_reset(tmp);
-	strbuf_addch(tmp, '+');
-	if (mirror)
-		strbuf_addf(tmp, "refs/%s:refs/%s",
-				branchname, branchname);
-	else
-		strbuf_addf(tmp, "refs/heads/%s:refs/remotes/%s/%s",
-				branchname, remotename, branchname);
+	if (!strcmp(branchname, ":")) {
+		strbuf_addstr(tmp, "+:");
+	} else {
+		strbuf_addch(tmp, '+');
+		if (mirror)
+			strbuf_addf(tmp, "refs/%s:refs/%s",
+					branchname, branchname);
+		else
+			strbuf_addf(tmp, "refs/heads/%s:refs/remotes/%s/%s",
+					branchname, remotename, branchname);
+	}
 	repo_config_set_multivar(the_repository, key, tmp->buf, "^$", 0);
 }
 

@@ -233,11 +238,13 @@ static int add(int argc, const char **argv, const char *prefix,
 		strbuf_reset(&buf);
 		strbuf_addf(&buf, "remote.%s.fetch", name);
 		if (track.nr == 0)
-			string_list_append(&track, "*");
-		for (size_t i = 0; i < track.nr; i++) {
+			string_list_append(&track,
+				mirror == MIRROR_NONE &&
+				is_repository_shallow(the_repository) ?
+				":" : "*");
+		for (size_t i = 0; i < track.nr; i++)
 			add_branch(buf.buf, track.items[i].string,
 				   name, mirror, &buf2);
-		}
 	}
 
 	if (mirror & MIRROR_PUSH) {

index 7cb479983b..84a19b5b5c 100644
@@ -49,6 +49,11 @@ static int parse_refspec(struct refspec_item *item, const char *refspec,
 		return 1;
 	}
 
+	if (fetch && rhs == lhs && rhs[1] == '\0') {
+		item->tracking = 1;
+		return 1;
+	}
+
 	if (rhs) {
 		size_t rlen = strlen(++rhs);
 		is_glob = (1 <= rlen && strchr(rhs, '*'));

@@ -181,6 +186,7 @@ void refspec_item_clear(struct refspec_item *item)
 	item->force = 0;
 	item->pattern = 0;
 	item->matching = 0;
+	item->tracking = 0;
 	item->exact_sha1 = 0;
 }
 

@@ -373,7 +379,7 @@ static int refspec_find_negative_match(struct refspec *rs, struct refspec_item *
 		struct refspec_item *refspec = &rs->items[i];
 		char *expn_name;
 
-		if (refspec->negative)
+		if (refspec->negative || refspec->tracking)
 			continue;
 
 		/* Note the reversal of src and dst */

index fadef67933..97bd6ffef5 100644
@@ -27,6 +27,7 @@ struct refspec_item {
 	unsigned matching : 1;
 	unsigned exact_sha1 : 1;
 	unsigned negative : 1;
+	unsigned tracking : 1;
 
 	char *src;
 	char *dst;

index cca02033b9..5ab187e00a 100644
@@ -353,6 +353,9 @@ int branch_has_merge_config(struct branch *branch);
 
 int branch_merge_matches(struct branch *, int n, const char *);
 
+/* fills tracked with the refname of every local branch's upstream on remote */
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked);
+
 /* list of the remote in a group as configured */
 struct remote_group_data {
 	const char *name;

index 3ca7b27104..4c13048d46 100644
@@ -728,6 +728,7 @@ integration_tests = [
   't5582-fetch-negative-refspec.sh',
   't5583-push-branches.sh',
   't5584-http-429-retry.sh',
+  't5585-fetch-tracking-refspec.sh',
   't5600-clone-fail-cleanup.sh',
   't5601-clone.sh',
   't5602-clone-remote-exec.sh',

base-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7

gitgitgadget

That's way too much for a single patch. It needs to be split into digestible chunks, but I offhand do not know how many pieces are appropriate, so let's think aloud together and try to refine the design while we do so.

The outline of our design so far should give something like this in our configuration file.

[remote "second"]
fetch = :+

[branch "topic1"]
remote = second
merge = refs/heads/main

[branch "topic2"]
remote = second
merge = refs/heads/next

In the "fetch only what we build on" mode, we would collect local branches $X where branch.$X.remote == second and then collect branch.$X.merge for these local branches. In this case, we would decide to fetch 'main' and 'next' branches in the end.

But notice that this does not give us sufficient information. There is no explicit clue that tells that the remote-tracking branches for this remote 'second' should be stored under refs/remotes/second/ hierarchy. A normal remote that is defined like so:

[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*

does not have such a problem, as it makes it crystal clear that their branches go under refs/remotes/origin/ hierarchy.

So using "fetch = :+" is *not* a good idea, as I said. Let's scrap that syntax.

One thing we could do is probably to introduce

[remote "second"]
refmap = +refs/heads/*:refs/remotes/second/*

instead to give this clue (see "git fetch --help" for what a refmap is; it looks similar to refspec but only defines how their refs are mapped to our namespace without specifying what to be fetched, which is exactly what we need here). We do not use remote.second.fetch at all.

It would be an easy first step to teach that an explicit

	$ git fetch second main next

with such a remote.second.refmap should behave the same way as

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second \
		main next

in a repository without the refmote.second.refmap configuration. As "git fetch --refmap=... second main next" should already work, it would be only the matter of supplementing the command line argument with configured default.

Then teach "git fetch" to further treat

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second

i.e., fetch with refmap but without specifying what exactly to fetch, as a request to fetch their branches we build on (and nothing else), using the refmap, in other words, the lack of "what to fetch" in the above command line signals "git fetch" to rewrite the above to

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second \
		main next

internally. Since we have the previous remote.X.refmap step already, it means that with remote.second.refmap configured properly, the user can only say

	$ git fetch second

and it would do the right thing in our scenario.

Another and final step would be to teach "git remote add" to add

[remote "second"]
refmap = +refs/heads/*:refs/remotes/second/*

when you want to fetch only what you build on. I am not sure what should trigger the decision. Your initial message said something about shallow and sparse and an earlier review refuted one of them (I do not recall which offhand, but probably sparse). It probably is a good idea to start with an explicit command line option to "git remote add --limited-fetch" in a single commit.

And then add heuristics (like "in a shallow clone, this mode is turned on by default, but an explicit '--no-limited-fetch' can countermand it") in another commit.

So far, we identified four distinct commits, each bite sized.

- remote.X.refmap configuration acts as if --refmap=... command line argument was passed.

- passing refmap without saying what to fetch enumerates what their branches we build on, and pretend as if the user listed these branches on the command line to fetch.

- "git remote add --limited-fetch" creates remote.X.refmap instead of remote.X.fetch as necessary.

- "git remote add" without explicit "--[no-]limited-fetch" uses heuristics to enable it.

Or something like that, perhaps?

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v3:

* Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap, reusing git's existing --refmap mechanism instead of inventing new refspec syntax.
* Split the change into 4 commits.

Changes in v2:

* Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (4):
fetch: add remote.<name>.refmap
fetch: infer branches to fetch from a refmap-only remote
remote: add "git remote add --limited-fetch"
remote: default to --limited-fetch in a shallow repository

 Documentation/config/remote.adoc |   7 ++
 Documentation/fetch-options.adoc |   7 ++
 Documentation/git-remote.adoc    |  14 ++-
 builtin/fetch.c                  | 142 +++++++++++++++++++++++++------
 builtin/remote.c                 |  31 +++++--
 remote.c                         |  41 ++++++++-
 remote.h                         |   9 ++
 t/meson.build                    |   1 +
 t/t5505-remote.sh                |  75 ++++++++++++++++
 t/t5510-fetch.sh                 |  17 ++++
 t/t5585-fetch-refmap.sh          | 124 +++++++++++++++++++++++++++
 11 files changed, 433 insertions(+), 35 deletions(-)
 create mode 100755 t/t5585-fetch-refmap.sh

base-commit: 0f8e75abebff0877cae681a3d5ff31ac47f54220
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v3
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v2:

 -:  ---------- > 1:  b04c00b974 fetch: add remote.<name>.refmap
 1:  f04f5ce219 ! 2:  4ec508a223 fetch: avoid fetching every branch of a new remote in a shallow repo
     @@ Metadata
      Author: Harald Nordgren <[email protected]>
      
       ## Commit message ##
     -    fetch: avoid fetching every branch of a new remote in a shallow repo
     +    fetch: infer branches to fetch from a refmap-only remote
      
     -    git remote add sets a new remote up to fetch every branch by default.
     -    In an already shallow repository, that turns the next plain fetch or
     -    pull into a slow or hanging one, even when only one or two branches
     -    are ever used.
     +    Configuring remote.<name>.refmap without remote.<name>.fetch used to
     +    make a refspec-less "git fetch <name>" fail with "--refmap option is
     +    only meaningful with command-line refspec(s)", since a refmap only
     +    says where to put fetched refs, not what to fetch.
      
     -    Add a special fetch refspec, "+:", that fetches only the branches our
     -    local branches are built on, plus the remote's default branch so a
     -    brand new remote is usable right away, without needing to first set
     -    anything up to track it. git remote add now uses it instead of the
     -    usual wildcard refspec whenever the repository is already shallow.
     +    Make that case infer what to fetch: the local branches whose
     +    @{upstream} is already on that remote, plus the remote's default
     +    branch, which is always included so it is available even before
     +    anything is set up to track it. This lets a remote be configured to
     +    fetch only the branches actually in use, without listing them by
     +    hand in remote.<name>.fetch, and without needing to touch the
     +    command line every time.
      
          Signed-off-by: Harald Nordgren <[email protected]>
      
     - ## Documentation/git-remote.adoc ##
     -@@ Documentation/git-remote.adoc: Add a remote named _<name>_ for the repository at
     - _<URL>_.  The command `git fetch <name>` can then be used to create and
     - update remote-tracking branches `<name>/<branch>`.
     - +
     -+If the repository is already a shallow repository (see linkgit:git-clone[1]
     -+`--depth`) and neither `-t` nor `--mirror` is given, the remote is set up
     -+with the special tracking refspec (see the _<refspec>_ entry in
     -+linkgit:git-fetch[1]) instead of the default glob refspec, so that
     -+`git fetch <name>` does not need to negotiate history for every branch
     -+the remote has.
     -++
     - With `-f` option, `git fetch <name>` is run immediately after
     - the remote information is set up.
     - +
     + ## Documentation/config/remote.adoc ##
     +@@ Documentation/config/remote.adoc: remote.<name>.fetch::
     + 
     + remote.<name>.refmap::
     + 	The default value of the `--refmap` option for linkgit:git-fetch[1].
     +-	Only takes effect when the fetch names what to fetch explicitly,
     +-	either on the command line or via `remote.<name>.fetch`. See the
     +-	`--refmap` entry in linkgit:git-fetch[1].
     ++	If `remote.<name>.fetch` is not set either, a refspec-less fetch
     ++	infers what to fetch from local branches built on this remote,
     ++	instead of fetching every branch it has. See the `--refmap` entry
     ++	in linkgit:git-fetch[1].
     + 
     + remote.<name>.push::
     + 	The default set of "refspec" for linkgit:git-push[1]. See
      
     - ## Documentation/pull-fetch-param.adoc ##
     -@@ Documentation/pull-fetch-param.adoc: Negative refspecs can themselves be pattern refspecs. However, they may only
     - contain a _<src>_ and do not specify a _<dst>_. Fully spelled out hex object
     - names are also not supported.
     - +
     -+The special refspec `:` (or `+:` to allow non-fast-forward updates)
     -+directs Git to fetch whichever branches at the remote our local branches
     -+have as their `@{upstream}`, plus the remote's default branch, which is
     -+always included so that it is available even before anything is set up
     -+to track it. Unlike the same syntax when pushing, the local branch name
     -+plays no role here beyond identifying which remote branch to fetch: the
     -+result is written to `refs/remotes/<remote>/<branch>` using the remote's
     -+own branch name, and several local branches built on the same remote
     -+branch only cause it to be fetched once.
     -++
     - `tag <tag>` means the same as `refs/tags/<tag>:refs/tags/<tag>`;
     - it requests fetching everything up to the given tag.
     + ## Documentation/fetch-options.adoc ##
     +@@ Documentation/fetch-options.adoc: endif::git-pull[]
     + 	command-line arguments. See section on "Configured Remote-tracking
     + 	Branches" for details.
       +
     +-`remote.<name>.refmap` provides the default value for this option, the
     +-same way `remote.<name>.fetch` provides the default refspecs to fetch.
     ++When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
     ++there is nothing to fetch, neither on the command line nor from
     ++`remote.<name>.fetch`, Git infers what to fetch from the local branches
     ++whose `@{upstream}` is on that remote, plus the remote's default branch,
     ++which is always included so that it is available even before anything
     ++is set up to track it.
     + 
     + `-t`::
     + `--tags`::
      
       ## builtin/fetch.c ##
      @@ builtin/fetch.c: static void filter_prefetch_refspec(struct refspec *rs)
     - 	}
     - }
     - 
     -+static void fetch_tracked_branches(struct remote *remote,
     -+				   const struct ref *remote_refs,
     -+				   struct ref ***tail,
     -+				   char **default_branch_out,
     -+				   int force)
     -+{
     -+	struct string_list tracked = STRING_LIST_INIT_DUP;
     -+	struct string_list_item *item;
     -+	struct ref *head;
     -+	char *default_branch = NULL;
     -+
     -+	branches_tracking_remote(remote, &tracked);
     -+
     -+	head = get_remote_ref(remote_refs, "HEAD");
     -+	if (head && head->symref && *head->symref) {
     -+		default_branch = xstrdup(head->symref);
     -+		string_list_insert(&tracked, default_branch);
     -+	}
     -+	free_one_ref(head);
     -+
     -+	for_each_string_list_item(item, &tracked) {
     -+		struct refspec_item ref = { .force = force };
     -+		const char *branch_name = item->string;
     -+		int is_default_branch = default_branch &&
     -+			!strcmp(item->string, default_branch);
     -+
     -+		skip_prefix(branch_name, "refs/heads/", &branch_name);
     -+		ref.src = xstrdup(is_default_branch ? "HEAD" : item->string);
     -+		ref.dst = xstrfmt("refs/remotes/%s/%s", remote->name, branch_name);
     -+		get_fetch_map(remote_refs, &ref, tail, 1);
     -+		free(ref.src);
     -+		free(ref.dst);
     -+	}
     -+
     -+	if (default_branch && default_branch_out) {
     -+		const char *branch_name = default_branch;
     -+		skip_prefix(branch_name, "refs/heads/", &branch_name);
     -+		*default_branch_out = xstrdup(branch_name);
     -+	}
     -+
     -+	free(default_branch);
     -+	string_list_clear(&tracked, 0);
     -+}
     -+
       static struct ref *get_ref_map(struct remote *remote,
       			       const struct ref *remote_refs,
       			       struct refspec *rs,
     @@ builtin/fetch.c: static void filter_prefetch_refspec(struct refspec *rs)
       	int i;
       	struct ref *rm;
      @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
     - 		     /* Note: has_merge implies non-NULL branch->remote_name */
     - 		     (has_merge && !strcmp(branch->remote_name, remote->name)))) {
     - 			for (i = 0; i < remote->fetch.nr; i++) {
     -+				if (remote->fetch.items[i].tracking) {
     -+					fetch_tracked_branches(remote, remote_refs, &tail,
     -+								bootstrap_head_branch,
     -+								remote->fetch.items[i].force);
     -+					continue;
     -+				}
     - 				get_fetch_map(remote_refs, &remote->fetch.items[i], &tail, 0);
     - 				if (remote->fetch.items[i].dst &&
     - 				    remote->fetch.items[i].dst[0])
     + 	struct ref **tail = &ref_map;
     + 	struct refspec *effective_refmap =
     + 		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;
     ++	struct refspec inferred_rs;
     ++	int inferred_branches = 0;
     + 
     + 	/* opportunistically-updated references: */
     + 	struct ref *orefs = NULL, **oref_tail = &orefs;
     +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
     + 	struct hashmap existing_refs;
     + 	int existing_refs_populated = 0;
     + 
     ++	refspec_init_fetch(&inferred_rs, the_hash_algo);
     ++
     + 	filter_prefetch_refspec(rs);
     + 	if (remote)
     + 		filter_prefetch_refspec(&remote->fetch);
     + 
     ++	if (!rs->nr && remote && !remote->fetch.nr &&
     ++	    effective_refmap && effective_refmap->nr) {
     ++		struct string_list tracked = STRING_LIST_INIT_DUP;
     ++		struct string_list_item *item;
     ++		struct ref *head;
     ++		char *default_branch = NULL;
     ++		char *default_branch_dst = NULL;
     ++
     ++		branches_tracking_remote(remote, &tracked);
     ++		for_each_string_list_item(item, &tracked)
     ++			refspec_append(&inferred_rs, item->string);
     ++
     ++		/* Always fetch the default branch too, as "HEAD". */
     ++		head = get_remote_ref(remote_refs, "HEAD");
     ++		if (head && head->symref && *head->symref)
     ++			default_branch = xstrdup(head->symref);
     ++		free_one_ref(head);
     ++
     ++		for (i = 0; default_branch && !default_branch_dst &&
     ++			    !string_list_has_string(&tracked, default_branch) &&
     ++			    i < effective_refmap->nr; i++) {
     ++			struct refspec_item *map = &effective_refmap->items[i];
     ++
     ++			if (map->pattern)
     ++				match_refname_with_pattern(map->src, default_branch,
     ++							    map->dst, &default_branch_dst);
     ++			else if (!strcmp(map->src, default_branch))
     ++				default_branch_dst = xstrdup(map->dst);
     ++		}
     ++
     ++		if (default_branch_dst) {
     ++			struct refspec_item head_item = { .force = 1 };
     ++
     ++			head_item.src = xstrdup("HEAD");
     ++			head_item.dst = default_branch_dst;
     ++			get_fetch_map(remote_refs, &head_item, &tail, 1);
     ++			free(head_item.src);
     ++		}
     ++
     ++		if (default_branch && bootstrap_head_branch) {
     ++			const char *branch_name = default_branch;
     ++			skip_prefix(branch_name, "refs/heads/", &branch_name);
     ++			*bootstrap_head_branch = xstrdup(branch_name);
     ++		}
     ++
     ++		free(default_branch);
     ++		free(default_branch_dst);
     ++		string_list_clear(&tracked, 0);
     ++
     ++		rs = &inferred_rs;
     ++		inferred_branches = 1;
     ++	}
     ++
     + 	if (rs->nr) {
     + 		struct refspec *fetch_refspec;
     + 
     + 		for (i = 0; i < rs->nr; i++) {
     +-			get_fetch_map(remote_refs, &rs->items[i], &tail, 0);
     ++			get_fetch_map(remote_refs, &rs->items[i], &tail,
     ++				      inferred_branches);
     + 			if (rs->items[i].dst && rs->items[i].dst[0])
     + 				*autotags = 1;
     + 		}
     +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
     + 
     + 		for (i = 0; i < fetch_refspec->nr; i++)
     + 			get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1);
     ++	} else if (inferred_branches) {
     ++		/* Already fully handled above. */
     + 	} else if (effective_refmap && effective_refmap->nr) {
     + 		die("--refmap option is only meaningful with command-line refspec(s)");
     + 	} else {
     +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
     + 	if (existing_refs_populated)
     + 		hashmap_clear_and_free(&existing_refs, struct refname_hash_entry, ent);
     + 
     ++	refspec_clear(&inferred_rs);
     + 	return ref_map;
     + }
     + 
      @@ builtin/fetch.c: static void warn_set_head(const char *remote, const char *head_name,
       }
       
     @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
       		refspec_ref_prefixes(rs, &transport_ls_refs_options.ref_prefixes);
       	} else {
       		struct branch *branch = branch_get(NULL);
     +-
     +-		if (transport->remote->fetch.nr) {
      +		int tracks_this_remote = branch && branch_has_merge_config(branch) &&
      +			!strcmp(branch->remote_name, transport->remote->name);
     -+		int tracking_refspec = transport->remote->fetch.nr == 1 &&
     -+			transport->remote->fetch.items[0].tracking;
     - 
     --		if (transport->remote->fetch.nr) {
     -+		if (tracking_refspec) {
     ++		struct refspec *effective_refmap = refmap.nr ? &refmap :
     ++			&transport->remote->refmap;
     ++		int inferred_branches = !transport->remote->fetch.nr &&
     ++			effective_refmap->nr;
     ++
     ++		if (inferred_branches) {
      +			struct string_list tracked = STRING_LIST_INIT_DUP;
      +			struct string_list_item *item;
      +
     @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
       		}
      -		if (branch && branch_has_merge_config(branch) &&
      -		    !strcmp(branch->remote_name, transport->remote->name)) {
     -+		if (transport->remote->fetch.nr &&
     ++		if ((transport->remote->fetch.nr || inferred_branches) &&
      +		    follow_remote_head != FOLLOW_REMOTE_NEVER)
      +			do_set_head = 1;
      +		if (tracks_this_remote) {
     @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
       	 * When using batched updates, we want to commit the non-rejected
       	 * updates and also handle the rejections.
      
     - ## builtin/remote.c ##
     -@@
     - #include "rebase.h"
     - #include "refs.h"
     - #include "refspec.h"
     -+#include "shallow.h"
     - #include "odb.h"
     - #include "strvec.h"
     - #include "commit-reach.h"
     -@@ builtin/remote.c: static void add_branch(const char *key, const char *branchname,
     - 		       const char *remotename, int mirror, struct strbuf *tmp)
     - {
     - 	strbuf_reset(tmp);
     --	strbuf_addch(tmp, '+');
     --	if (mirror)
     --		strbuf_addf(tmp, "refs/%s:refs/%s",
     --				branchname, branchname);
     --	else
     --		strbuf_addf(tmp, "refs/heads/%s:refs/remotes/%s/%s",
     --				branchname, remotename, branchname);
     -+	if (!strcmp(branchname, ":")) {
     -+		strbuf_addstr(tmp, "+:");
     -+	} else {
     -+		strbuf_addch(tmp, '+');
     -+		if (mirror)
     -+			strbuf_addf(tmp, "refs/%s:refs/%s",
     -+					branchname, branchname);
     -+		else
     -+			strbuf_addf(tmp, "refs/heads/%s:refs/remotes/%s/%s",
     -+					branchname, remotename, branchname);
     -+	}
     - 	repo_config_set_multivar(the_repository, key, tmp->buf, "^$", 0);
     - }
     - 
     -@@ builtin/remote.c: static int add(int argc, const char **argv, const char *prefix,
     - 		strbuf_reset(&buf);
     - 		strbuf_addf(&buf, "remote.%s.fetch", name);
     - 		if (track.nr == 0)
     --			string_list_append(&track, "*");
     --		for (size_t i = 0; i < track.nr; i++) {
     -+			string_list_append(&track,
     -+				mirror == MIRROR_NONE &&
     -+				is_repository_shallow(the_repository) ?
     -+				":" : "*");
     -+		for (size_t i = 0; i < track.nr; i++)
     - 			add_branch(buf.buf, track.items[i].string,
     - 				   name, mirror, &buf2);
     --		}
     - 	}
     - 
     - 	if (mirror & MIRROR_PUSH) {
     -
     - ## refspec.c ##
     -@@ refspec.c: static int parse_refspec(struct refspec_item *item, const char *refspec,
     - 		return 1;
     - 	}
     - 
     -+	if (fetch && rhs == lhs && rhs[1] == '\0') {
     -+		item->tracking = 1;
     -+		return 1;
     -+	}
     -+
     - 	if (rhs) {
     - 		size_t rlen = strlen(++rhs);
     - 		is_glob = (1 <= rlen && strchr(rhs, '*'));
     -@@ refspec.c: void refspec_item_clear(struct refspec_item *item)
     - 	item->force = 0;
     - 	item->pattern = 0;
     - 	item->matching = 0;
     -+	item->tracking = 0;
     - 	item->exact_sha1 = 0;
     - }
     - 
     -@@ refspec.c: static int refspec_find_negative_match(struct refspec *rs, struct refspec_item *
     - 		struct refspec_item *refspec = &rs->items[i];
     - 		char *expn_name;
     - 
     --		if (refspec->negative)
     -+		if (refspec->negative || refspec->tracking)
     - 			continue;
     - 
     - 		/* Note the reversal of src and dst */
     -
     - ## refspec.h ##
     -@@ refspec.h: struct refspec_item {
     - 	unsigned matching : 1;
     - 	unsigned exact_sha1 : 1;
     - 	unsigned negative : 1;
     -+	unsigned tracking : 1;
     - 
     - 	char *src;
     - 	char *dst;
     -
       ## remote.c ##
     -@@ remote.c: void ref_push_report_free(struct ref_push_report *report)
     - 
     - int remote_find_tracking(struct remote *remote, struct refspec_item *refspec)
     - {
     -+	if (remote->fetch.nr == 1 && remote->fetch.items[0].tracking) {
     -+		struct strbuf prefix = STRBUF_INIT;
     -+		const char *branch_name;
     -+		int ret = -1;
     -+
     -+		strbuf_addf(&prefix, "refs/remotes/%s/", remote->name);
     -+		if (!refspec->src) {
     -+			if (refspec->dst &&
     -+			    skip_prefix(refspec->dst, prefix.buf, &branch_name)) {
     -+				refspec->src = xstrfmt("refs/heads/%s", branch_name);
     -+				refspec->force = 1;
     -+				ret = 0;
     -+			}
     -+		} else if (skip_prefix(refspec->src, "refs/heads/", &branch_name)) {
     -+			free(refspec->dst);
     -+			refspec->dst = xstrfmt("%s%s", prefix.buf, branch_name);
     -+			refspec->force = 1;
     -+			ret = 0;
     -+		}
     -+		strbuf_release(&prefix);
     -+		return ret;
     -+	}
     -+
     - 	return refspec_find_match(&remote->fetch, refspec);
     - }
     - 
      @@ remote.c: int branch_merge_matches(struct branch *branch,
       	return refname_match(branch->merge[i]->src, refname);
       }
     @@ remote.c: int branch_merge_matches(struct branch *branch,
       __attribute__((format (printf,2,3)))
       static char *error_buf(struct strbuf *err, const char *fmt, ...)
       {
     -@@ remote.c: int get_fetch_map(const struct ref *remote_refs,
     - {
     - 	struct ref *ref_map, **rmp;
     - 
     --	if (refspec->negative)
     -+	if (refspec->negative || refspec->tracking)
     - 		return 0;
     - 
     - 	if (refspec->pattern) {
      
       ## remote.h ##
      @@ remote.h: int branch_has_merge_config(struct branch *branch);
     @@ t/meson.build: integration_tests = [
         't5582-fetch-negative-refspec.sh',
         't5583-push-branches.sh',
         't5584-http-429-retry.sh',
     -+  't5585-fetch-tracking-refspec.sh',
     ++  't5585-fetch-refmap.sh',
         't5600-clone-fail-cleanup.sh',
         't5601-clone.sh',
         't5602-clone-remote-exec.sh',
      
     - ## t/t5505-remote.sh ##
     -@@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only' '
     - 	test_grep ! "\[blob:none\]" out
     - '
     - 
     -+test_expect_success 'add remote -t keeps an explicit refspec in a shallow repository' '
     -+	test_when_finished "rm -rf shallow-add" &&
     -+	git clone --no-local --depth=1 --branch main --single-branch \
     -+		one shallow-add &&
     -+	(
     -+		cd shallow-add &&
     -+		git remote add -t main upstream ../two &&
     -+		test_cmp_config "+refs/heads/main:refs/remotes/upstream/main" \
     -+			remote.upstream.fetch
     -+	)
     -+'
     -+
     -+test_expect_success 'add remote keeps the wildcard refspec in a full repository' '
     -+	test_when_finished "rm -rf full-add" &&
     -+	git clone --no-local one full-add &&
     -+	(
     -+		cd full-add &&
     -+		git remote add upstream ../two &&
     -+		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
     -+			remote.upstream.fetch
     -+	)
     -+'
     -+
     -+test_expect_success 'a remote added in a shallow repository defaults to the tracking refspec' '
     -+	test_when_finished "rm -rf shallow-add" &&
     -+	git clone --no-local --depth=1 --branch main --single-branch \
     -+		one shallow-add &&
     -+	(
     -+		cd shallow-add &&
     -+		git remote add upstream ../two &&
     -+		test_cmp_config "+:" remote.upstream.fetch &&
     -+		git fetch upstream &&
     -+		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
     -+		cat >expect <<-\EOF &&
     -+		refs/remotes/upstream/HEAD
     -+		refs/remotes/upstream/main
     -+		EOF
     -+		test_cmp expect actual &&
     -+		git branch --set-upstream-to=upstream &&
     -+		test_cmp_config upstream branch.main.remote &&
     -+		test_cmp_config refs/heads/main branch.main.merge
     -+	)
     -+'
     -+
     - test_expect_success 'check remote-tracking' '
     - 	(
     - 		cd test &&
     -
     - ## t/t5585-fetch-tracking-refspec.sh (new) ##
     + ## t/t5585-fetch-refmap.sh (new) ##
      @@
      +#!/bin/sh
      +
     -+test_description='"git fetch" with the tracking refspec ":" (or "+:")
     ++test_description='"git fetch" with a remote.<name>.refmap but no remote.<name>.fetch
      +
     -+The special refspec fetches whichever branches at the remote our local
     -+branches have as their @{upstream}, plus the default branch of that
     -+remote, which is always included so that it is available even before
     -+anything is set up to track it. git remote add uses it by default for
     -+a new remote in an already shallow repository, so that adding a
     -+second remote there does not turn the next plain fetch or pull into
     -+one that negotiates history for every branch the remote has.
     ++When a remote has a refmap configured but no fetch refspec, a
     ++refspec-less fetch infers what to fetch from the local branches whose
     ++@{upstream} is on that remote, plus the default branch of that remote,
     ++which is always included so that it is available even before anything
     ++is set up to track it.
      +'
      +
      +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
     @@ t/t5585-fetch-tracking-refspec.sh (new)
      +	(
      +		cd client &&
      +		git remote add upstream .. &&
     -+		test_cmp_config "+:" remote.upstream.fetch
     ++		test_might_fail git config unset remote.upstream.fetch &&
     ++		git config remote.upstream.refmap \
     ++			"+refs/heads/*:refs/remotes/upstream/*"
      +	)
      +'
      +
     @@ t/t5585-fetch-tracking-refspec.sh (new)
      +	)
      +'
      +
     -+test_expect_success 'git remote show does not choke on the tracking refspec' '
     ++test_expect_success 'git remote show does not choke on a refmap-only remote' '
      +	(
      +		cd client &&
      +		git remote show upstream
 -:  ---------- > 3:  604e584956 remote: add "git remote add --limited-fetch"
 -:  ---------- > 4:  31462e9445 remote: default to --limited-fetch in a shallow repository

gitgitgadget

From: Harald Nordgren <[email protected]>

Add a per-remote config variable, remote.<name>.refmap, that provides the default value for --refmap the same way remote.<name>.fetch already provides the default refspecs to fetch. It only takes effect when there is something explicit to fetch, on the command line or via remote.<name>.fetch, matching how --refmap itself already behaves.

---
 Documentation/config/remote.adoc |  6 ++++++
 Documentation/fetch-options.adoc |  3 +++
 builtin/fetch.c                  |  8 +++++---
 remote.c                         | 12 +++++++++++-
 remote.h                         |  6 ++++++
 t/t5510-fetch.sh                 | 17 +++++++++++++++++
 6 files changed, 48 insertions(+), 4 deletions(-)

index 3a20d0f752..00f9ed8d3d 100644
@@ -33,6 +33,12 @@ remote.<name>.fetch::
 	The default set of "refspec" for linkgit:git-fetch[1]. See
 	linkgit:git-fetch[1].
 
+remote.<name>.refmap::
+	The default value of the `--refmap` option for linkgit:git-fetch[1].
+	Only takes effect when the fetch names what to fetch explicitly,
+	either on the command line or via `remote.<name>.fetch`. See the
+	`--refmap` entry in linkgit:git-fetch[1].
+
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See
 	linkgit:git-push[1].

index 035f780e58..538914bc6e 100644
@@ -244,6 +244,9 @@ endif::git-pull[]
 	refspecs and rely entirely on the refspecs supplied as
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
++
+`remote.<name>.refmap` provides the default value for this option, the
+same way `remote.<name>.fetch` provides the default refspecs to fetch.
 
 `-t`::
 `--tags`::

index 533fdfe7d8..7651b41139 100644
@@ -509,6 +509,8 @@ static struct ref *get_ref_map(struct remote *remote,
 	struct ref *rm;
 	struct ref *ref_map = NULL;
 	struct ref **tail = &ref_map;
+	struct refspec *effective_refmap =
+		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;
 
 	/* opportunistically-updated references: */
 	struct ref *orefs = NULL, **oref_tail = &orefs;

@@ -552,14 +554,14 @@ static struct ref *get_ref_map(struct remote *remote,
 		 * by ref_remove_duplicates() in favor of one of these
 		 * opportunistic entries with FETCH_HEAD_IGNORE.
 		 */
-		if (refmap.nr)
-			fetch_refspec = &refmap;
+		if (effective_refmap && effective_refmap->nr)
+			fetch_refspec = effective_refmap;
 		else
 			fetch_refspec = &remote->fetch;
 
 		for (i = 0; i < fetch_refspec->nr; i++)
 			get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1);
-	} else if (refmap.nr) {
+	} else if (effective_refmap && effective_refmap->nr) {
 		die("--refmap option is only meaningful with command-line refspec(s)");
 	} else {
 		/* Use the defaults */

index fe62068463..017cd9d13e 100644
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,
 	ret->name = xstrndup(name, len);
 	refspec_init_push(&ret->push, the_hash_algo);
 	refspec_init_fetch(&ret->fetch, the_hash_algo);
+	refspec_init_fetch(&ret->refmap, the_hash_algo);
 	string_list_init_dup(&ret->server_options);
 	string_list_init_dup(&ret->negotiation_restrict);
 	string_list_init_dup(&ret->negotiation_include);

@@ -176,6 +177,7 @@ static void remote_clear(struct remote *remote)
 
 	refspec_clear(&remote->push);
 	refspec_clear(&remote->fetch);
+	refspec_clear(&remote->refmap);
 
 	free((char *)remote->receivepack);
 	free((char *)remote->uploadpack);

@@ -539,6 +541,12 @@ static int handle_config(const char *key, const char *value,
 			return -1;
 		refspec_append(&remote->fetch, v);
 		free(v);
+	} else if (!strcmp(subkey, "refmap")) {
+		char *v;
+		if (git_config_string(&v, key, value))
+			return -1;
+		refspec_append(&remote->refmap, v);
+		free(v);
 	} else if (!strcmp(subkey, "receivepack")) {
 		char *v;
 		if (git_config_string(&v, key, value))

@@ -988,7 +996,9 @@ void ref_push_report_free(struct ref_push_report *report)
 
 int remote_find_tracking(struct remote *remote, struct refspec_item *refspec)
 {
-	return refspec_find_match(&remote->fetch, refspec);
+	if (remote->fetch.nr)
+		return refspec_find_match(&remote->fetch, refspec);
+	return refspec_find_match(&remote->refmap, refspec);
 }
 
 static struct ref *alloc_ref_with_prefix(const char *prefix, size_t prefixlen,

index cca02033b9..ac485a584d 100644
@@ -90,6 +90,12 @@ struct remote {
 
 	struct refspec fetch;
 
+	/*
+	 * How to map refs fetched without an explicit destination into our
+	 * own namespace, the same as the --refmap command line option.
+	 */
+	struct refspec refmap;
+
 	/*
 	 * The setting for whether to fetch tags (as a separate rule from the
 	 * configured refspecs);

index a8d38d9176..bd853d0a84 100755
@@ -927,6 +927,23 @@ test_expect_success 'explicit --refmap option overrides remote.*.fetch' '
 	)
 '
 
+test_expect_success 'remote.*.refmap acts like --refmap on the command line' '
+	test_when_finished "git -C three config --unset remote.origin.refmap" &&
+	git branch -f side &&
+	git -C three config remote.origin.refmap \
+		"refs/heads/*:refs/remotes/other/*" &&
+	(
+		cd three &&
+		git update-ref refs/remotes/origin/main base-origin-main &&
+		o=$(git rev-parse --verify refs/remotes/origin/main) &&
+		git fetch origin main &&
+		n=$(git rev-parse --verify refs/remotes/origin/main) &&
+		test "$o" = "$n" &&
+		test_must_fail git rev-parse --verify refs/remotes/origin/side &&
+		git rev-parse --verify refs/remotes/other/main
+	)
+'
+
 test_expect_success 'explicitly empty --refmap option disables remote.*.fetch' '
 	git branch -f side &&
 	(

gitgitgadget

This ...

From: Harald Nordgren <[email protected]>

Configuring remote.<name>.refmap without remote.<name>.fetch used to make a refspec-less "git fetch <name>" fail with "--refmap option is only meaningful with command-line refspec(s)", since a refmap only says where to put fetched refs, not what to fetch.

Make that case infer what to fetch: the local branches whose @{upstream} is already on that remote, plus the remote's default branch, which is always included so it is available even before anything is set up to track it. This lets a remote be configured to fetch only the branches actually in use, without listing them by hand in remote.<name>.fetch, and without needing to touch the command line every time.

---
 Documentation/config/remote.adoc |   7 +-
 Documentation/fetch-options.adoc |   8 +-
 builtin/fetch.c                  | 134 +++++++++++++++++++++++++------
 remote.c                         |  29 +++++++
 remote.h                         |   3 +
 t/meson.build                    |   1 +
 t/t5585-fetch-refmap.sh          | 124 ++++++++++++++++++++++++++++
 7 files changed, 277 insertions(+), 29 deletions(-)
 create mode 100755 t/t5585-fetch-refmap.sh

index 00f9ed8d3d..5b68af3af1 100644
@@ -35,9 +35,10 @@ remote.<name>.fetch::
 
 remote.<name>.refmap::
 	The default value of the `--refmap` option for linkgit:git-fetch[1].
-	Only takes effect when the fetch names what to fetch explicitly,
-	either on the command line or via `remote.<name>.fetch`. See the
-	`--refmap` entry in linkgit:git-fetch[1].
+	If `remote.<name>.fetch` is not set either, a refspec-less fetch
+	infers what to fetch from local branches built on this remote,
+	instead of fetching every branch it has. See the `--refmap` entry
+	in linkgit:git-fetch[1].
 
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See

index 538914bc6e..c973a07caf 100644
@@ -245,8 +245,12 @@ endif::git-pull[]
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
 +
-`remote.<name>.refmap` provides the default value for this option, the
-same way `remote.<name>.fetch` provides the default refspecs to fetch.
+When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
+there is nothing to fetch, neither on the command line nor from
+`remote.<name>.fetch`, Git infers what to fetch from the local branches
+whose `@{upstream}` is on that remote, plus the remote's default branch,
+which is always included so that it is available even before anything
+is set up to track it.
 
 `-t`::
 `--tags`::

index 017cd9d13e..a5deb9b929 100644
@@ -1884,6 +1884,35 @@ int branch_merge_matches(struct branch *branch,
 	return refname_match(branch->merge[i]->src, refname);
 }
 
+struct branches_tracking_remote_cb_data {
+	struct remote *remote;
+	struct string_list *tracked;
+};
+
+static int add_if_tracking_remote(const struct reference *ref, void *cb_data)
+{
+	struct branches_tracking_remote_cb_data *data = cb_data;
+	struct branch *branch;
+
+	branch = branch_get(ref->name);
+	if (!branch_has_merge_config(branch) ||
+	    strcmp(branch->remote_name, data->remote->name))
+		return 0;
+
+	for (int i = 0; i < branch->merge_nr; i++)
+		string_list_insert(data->tracked, branch->merge[i]->src);
+
+	return 0;
+}
+
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked)
+{
+	struct branches_tracking_remote_cb_data data = { remote, tracked };
+
+	refs_for_each_branch_ref(get_main_ref_store(the_repository),
+				  add_if_tracking_remote, &data);
+}
+
 __attribute__((format (printf,2,3)))
 static char *error_buf(struct strbuf *err, const char *fmt, ...)
 {

index ac485a584d..da6c3ef53a 100644
@@ -359,6 +359,9 @@ int branch_has_merge_config(struct branch *branch);
 
 int branch_merge_matches(struct branch *, int n, const char *);
 
+/* fills tracked with the refname of every local branch's upstream on remote */
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked);
+
 /* list of the remote in a group as configured */
 struct remote_group_data {
 	const char *name;

index 3ca7b27104..ab83d5409c 100644
@@ -728,6 +728,7 @@ integration_tests = [
   't5582-fetch-negative-refspec.sh',
   't5583-push-branches.sh',
   't5584-http-429-retry.sh',
+  't5585-fetch-refmap.sh',
   't5600-clone-fail-cleanup.sh',
   't5601-clone.sh',
   't5602-clone-remote-exec.sh',

gitgitgadget

You may not necessarily be interested in what the remote's HEAD is pointing at.

- You may only be working on top of their 'maint' and their 'next' without ever touching their 'master'. It would be rude for Git to decide for you to fetch their 'master', which you have no use for, without being asked at all.

- Or you may be very much interested in their 'master' that is pointed at by their HEAD. In such a case, you would have your local branch that is forked from their 'master' *anyway*, and you'll get one, without special casing their HEAD.

Especially since we are about to add the knob to trigger this mechanism to "git remote add", presumably this is to be used for second and later remotes. The primary remote may already be supplying "refs/heads/master -> refs/remotes/origin/master" when the repository was made with "git clone --shallow --single-branch" or something like that. That makes it doubly dubious to assume that the branch the HEAD of this secondary remote points at is so special and everybody that does "git remote add" wants to interact with it.

From: Harald Nordgren <[email protected]>

A remote added the ordinary way tracks every branch it has, via a wildcard remote.<name>.fetch refspec. That is wasteful for a remote whose history is only worth following for the branches actually in use locally, and it can make "git fetch" negotiate history for branches nobody asked for.

Give "git remote add" a --limited-fetch option that sets up remote.<name>.refmap instead of remote.<name>.fetch, so that "git fetch <name>" only fetches the branches already tracked, plus the remote's default branch, as described in the previous commit. It is rejected together with -t/--track or --mirror, since those already say explicitly what to fetch.

---
 Documentation/git-remote.adoc |  8 +++++++-
 builtin/remote.c              | 28 ++++++++++++++++++++++------
 t/t5505-remote.sh             | 16 ++++++++++++++++
 3 files changed, 45 insertions(+), 7 deletions(-)

index eaae30aa88..74fbce6e0e 100644
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [synopsis]
 git remote [-v | --verbose]
-git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] <name> <URL>
+git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] [--[no-]limited-fetch] <name> <URL>
 git remote rename [--[no-]progress] <old> <new>
 git remote remove <name>
 git remote set-head <name> (-a | --auto | -d | --delete | <branch>)

@@ -70,6 +70,12 @@ the `refs/remotes/<name>/` namespace, a refspec to track only _<branch>_
 is created.  You can give more than one `-t <branch>` to track
 multiple branches without grabbing all branches.
 +
+With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
+that tracks all branches, `remote.<name>.refmap` is set up so that a
+refspec-less `git fetch <name>` only fetches branches our local branches
+are built on, plus the remote's default branch. See the `--refmap` entry
+in linkgit:git-fetch[1] for details.
++
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.
 +

index de989ea3ba..f036dd5d6f 100644
@@ -179,6 +179,7 @@ static int add(int argc, const char **argv, const char *prefix,
 {
 	int fetch = 0, fetch_tags = TAGS_DEFAULT;
 	unsigned mirror = MIRROR_NONE;
+	int limited_fetch = -1; /* unspecified */
 	struct string_list track = STRING_LIST_INIT_NODUP;
 	const char *master = NULL;
 	struct remote *remote;

@@ -198,6 +199,8 @@ static int add(int argc, const char **argv, const char *prefix,
 		OPT_CALLBACK_F(0, "mirror", &mirror, "(push|fetch)",
 			N_("set up remote as a mirror to push to or fetch from"),
 			PARSE_OPT_OPTARG | PARSE_OPT_COMP_ARG, parse_mirror_opt),
+		OPT_BOOL(0, "limited-fetch", &limited_fetch,
+			N_("fetch only the branches we build on, instead of every branch")),
 		OPT_END()
 	};
 

@@ -211,6 +214,10 @@ static int add(int argc, const char **argv, const char *prefix,
 		die(_("specifying a master branch makes no sense with --mirror"));
 	if (mirror && !(mirror & MIRROR_FETCH) && track.nr)
 		die(_("specifying branches to track makes sense only with fetch mirrors"));
+	if (limited_fetch == 1 && track.nr)
+		die(_("--limited-fetch does not make sense with -t/--track"));
+	if (limited_fetch == 1 && mirror)
+		die(_("--limited-fetch does not make sense with --mirror"));
 
 	name = argv[0];
 	url = argv[1];

@@ -230,13 +237,22 @@ static int add(int argc, const char **argv, const char *prefix,
 	repo_config_set(the_repository, buf.buf, url);
 
 	if (!mirror || mirror & MIRROR_FETCH) {
+		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
+			limited_fetch == 1;
+
 		strbuf_reset(&buf);
-		strbuf_addf(&buf, "remote.%s.fetch", name);
-		if (track.nr == 0)
-			string_list_append(&track, "*");
-		for (size_t i = 0; i < track.nr; i++) {
-			add_branch(buf.buf, track.items[i].string,
-				   name, mirror, &buf2);
+		if (use_limited_fetch) {
+			strbuf_addf(&buf, "remote.%s.refmap", name);
+			strbuf_reset(&buf2);
+			strbuf_addf(&buf2, "+refs/heads/*:refs/remotes/%s/*", name);
+			repo_config_set(the_repository, buf.buf, buf2.buf);
+		} else {
+			strbuf_addf(&buf, "remote.%s.fetch", name);
+			if (track.nr == 0)
+				string_list_append(&track, "*");
+			for (size_t i = 0; i < track.nr; i++)
+				add_branch(buf.buf, track.items[i].string,
+					   name, mirror, &buf2);
 		}
 	}
 

index 5fbfcb0848..0168d5abfe 100755
@@ -137,6 +137,22 @@ test_expect_success 'filters are listed by git remote -v only' '
 	test_grep ! "\[blob:none\]" out
 '
 
+test_expect_success '--limited-fetch works in a full repository too' '
+	test_when_finished "rm -rf full-add" &&
+	git clone --no-local one full-add &&
+	(
+		cd full-add &&
+		git remote add --limited-fetch upstream ../two &&
+		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
+			remote.upstream.refmap &&
+		test_must_fail git config get remote.upstream.fetch
+	)
+'
+
+test_expect_success '--limited-fetch conflicts with -t' '
+	test_must_fail git remote add --limited-fetch -t main upstream ../two
+'
+
 test_expect_success 'check remote-tracking' '
 	(
 		cd test &&

gitgitgadget

From: Harald Nordgren <[email protected]>

Adding a second remote to a shallow, single-branch clone used to still fetch every branch that remote has, since "git remote add" always set up a wildcard remote.<name>.fetch refspec regardless of how shallow the repository already was. That defeats the purpose of having cloned shallow and single-branch in the first place, and can make a plain "git fetch" on that remote hang or take a very long time on a repository with many branches.

Turn --limited-fetch on by default when the repository is already shallow and neither -t/--track nor --mirror was given, so that adding a remote there does not by itself commit to following every branch it has. --no-limited-fetch keeps the previous behavior for whoever wants it.

---
 Documentation/git-remote.adoc |  8 ++++-
 builtin/remote.c              |  5 ++-
 t/t5505-remote.sh             | 59 +++++++++++++++++++++++++++++++++++
 3 files changed, 70 insertions(+), 2 deletions(-)

index 74fbce6e0e..82b21eafa7 100644
@@ -52,6 +52,11 @@ Add a remote named _<name>_ for the repository at
 _<URL>_.  The command `git fetch <name>` can then be used to create and
 update remote-tracking branches `<name>/<branch>`.
 +
+If the repository is already a shallow repository (see linkgit:git-clone[1]
+`--depth`) and neither `-t`, `--mirror` nor `--no-limited-fetch` is given,
+`--limited-fetch` is turned on by default, so that `git fetch <name>` does
+not need to negotiate history for every branch the remote has.
++
 With `-f` option, `git fetch <name>` is run immediately after
 the remote information is set up.
 +

@@ -74,7 +79,8 @@ With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
 that tracks all branches, `remote.<name>.refmap` is set up so that a
 refspec-less `git fetch <name>` only fetches branches our local branches
 are built on, plus the remote's default branch. See the `--refmap` entry
-in linkgit:git-fetch[1] for details.
+in linkgit:git-fetch[1] for details. `--no-limited-fetch` explicitly
+disables this, overriding the shallow-repository default described above.
 +
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.

index f036dd5d6f..0cf1126976 100644
@@ -16,6 +16,7 @@
 #include "rebase.h"
 #include "refs.h"
 #include "refspec.h"
+#include "shallow.h"
 #include "odb.h"
 #include "strvec.h"
 #include "commit-reach.h"

@@ -238,7 +239,9 @@ static int add(int argc, const char **argv, const char *prefix,
 
 	if (!mirror || mirror & MIRROR_FETCH) {
 		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
-			limited_fetch == 1;
+			(limited_fetch == 1 ||
+			 (limited_fetch == -1 &&
+			  is_repository_shallow(the_repository)));
 
 		strbuf_reset(&buf);
 		if (use_limited_fetch) {

gitgitgadget

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v4:

* Removed the automatic default-branch fetch. A fresh remote fetches nothing until you track a branch explicitly.
* Fixed fetch report showing "new ref HEAD" instead of "new branch "
* Reworded remote..refmap docs and commit message.

Changes in v3:

* Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap, reusing git's existing --refmap mechanism instead of inventing new refspec syntax.
* Split the change into 4 commits.

Changes in v2:

* Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (4):
fetch: add remote.<name>.refmap
fetch: infer branches to fetch from a refmap-only remote
remote: add "git remote add --limited-fetch"
remote: default to --limited-fetch in a shallow repository

 Documentation/config/remote.adoc |   8 +++
 Documentation/fetch-options.adoc |   5 ++
 Documentation/git-remote.adoc    |  14 +++-
 builtin/fetch.c                  |  60 +++++++++++++---
 builtin/remote.c                 |  31 ++++++--
 remote.c                         |  41 ++++++++++-
 remote.h                         |   9 +++
 t/meson.build                    |   1 +
 t/t5505-remote.sh                |  76 ++++++++++++++++++++
 t/t5510-fetch.sh                 |  17 +++++
 t/t5585-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 11 files changed, 363 insertions(+), 18 deletions(-)
 create mode 100755 t/t5585-fetch-refmap.sh

base-commit: a018953688f1b10bddf91bff8747068f5f4746a4
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v4
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v3:

1: b04c00b974 ! 1: d48a7004e4 fetch: add remote.<name>.refmap
@@ Commit message

Add a per-remote config variable, remote.<name>.refmap, that provides the default value for --refmap the same way remote.<name>.fetch
- already provides the default refspecs to fetch. It only takes effect
- when there is something explicit to fetch, on the command line or via
- remote.<name>.fetch, matching how --refmap itself already behaves.
+ already provides the default refspecs to fetch. Like --refmap itself,
+ it only maps refs that are actually being fetched, so it has nothing
+ to do when there is nothing explicit to fetch, on the command line or
+ via remote.<name>.fetch.

@@ Documentation/config/remote.adoc: remote.<name>.fetch::

+remote.<name>.refmap::
+ The default value of the --refmap option for linkgit:git-fetch[1].
-+ Only takes effect when the fetch names what to fetch explicitly,
-+ either on the command line or via remote.<name>.fetch. See the
-+ --refmap entry in linkgit:git-fetch[1].
++ Used to map remote refs being fetched to remote-tracking refs to
++ store. See the --refmap entry in linkgit:git-fetch[1].
+
remote.<name>.push::
The default set of "refspec" for linkgit:git-push[1]. See
2: 4ec508a223 ! 2: 45b26e2bb2 fetch: infer branches to fetch from a refmap-only remote

     @@ Commit message
          says where to put fetched refs, not what to fetch.
      
          Make that case infer what to fetch: the local branches whose
     -    @{upstream} is already on that remote, plus the remote's default
     -    branch, which is always included so it is available even before
     -    anything is set up to track it. This lets a remote be configured to
     -    fetch only the branches actually in use, without listing them by
     -    hand in remote.<name>.fetch, and without needing to touch the
     -    command line every time.
     +    @{upstream} is already on that remote. This lets a remote be
     +    configured to fetch only the branches actually in use, without
     +    listing them by hand in remote.<name>.fetch, and without needing to
     +    touch the command line every time.
      
          Signed-off-by: Harald Nordgren <[email protected]>
      
       ## Documentation/config/remote.adoc ##
      @@ Documentation/config/remote.adoc: remote.<name>.fetch::
     - 
       remote.<name>.refmap::
       	The default value of the `--refmap` option for linkgit:git-fetch[1].
     --	Only takes effect when the fetch names what to fetch explicitly,
     --	either on the command line or via `remote.<name>.fetch`. See the
     --	`--refmap` entry in linkgit:git-fetch[1].
     -+	If `remote.<name>.fetch` is not set either, a refspec-less fetch
     -+	infers what to fetch from local branches built on this remote,
     -+	instead of fetching every branch it has. See the `--refmap` entry
     -+	in linkgit:git-fetch[1].
     + 	Used to map remote refs being fetched to remote-tracking refs to
     +-	store. See the `--refmap` entry in linkgit:git-fetch[1].
     ++	store. If `remote.<name>.fetch` is not set either, a refspec-less
     ++	fetch infers what to fetch from local branches built on this
     ++	remote, instead of fetching every branch it has. See the
     ++	`--refmap` entry in linkgit:git-fetch[1].
       
       remote.<name>.push::
       	The default set of "refspec" for linkgit:git-push[1]. See
     @@ Documentation/fetch-options.adoc: endif::git-pull[]
      +When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
      +there is nothing to fetch, neither on the command line nor from
      +`remote.<name>.fetch`, Git infers what to fetch from the local branches
     -+whose `@{upstream}` is on that remote, plus the remote's default branch,
     -+which is always included so that it is available even before anything
     -+is set up to track it.
     ++whose `@{upstream}` is on that remote.
       
       `-t`::
       `--tags`::
      
       ## builtin/fetch.c ##
     -@@ builtin/fetch.c: static void filter_prefetch_refspec(struct refspec *rs)
     - static struct ref *get_ref_map(struct remote *remote,
     - 			       const struct ref *remote_refs,
     - 			       struct refspec *rs,
     --			       int tags, int *autotags)
     -+			       int tags, int *autotags,
     -+			       char **bootstrap_head_branch)
     - {
     - 	int i;
     - 	struct ref *rm;
      @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       	struct ref **tail = &ref_map;
       	struct refspec *effective_refmap =
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
      +	    effective_refmap && effective_refmap->nr) {
      +		struct string_list tracked = STRING_LIST_INIT_DUP;
      +		struct string_list_item *item;
     -+		struct ref *head;
     -+		char *default_branch = NULL;
     -+		char *default_branch_dst = NULL;
      +
      +		branches_tracking_remote(remote, &tracked);
      +		for_each_string_list_item(item, &tracked)
      +			refspec_append(&inferred_rs, item->string);
     -+
     -+		/* Always fetch the default branch too, as "HEAD". */
     -+		head = get_remote_ref(remote_refs, "HEAD");
     -+		if (head && head->symref && *head->symref)
     -+			default_branch = xstrdup(head->symref);
     -+		free_one_ref(head);
     -+
     -+		for (i = 0; default_branch && !default_branch_dst &&
     -+			    !string_list_has_string(&tracked, default_branch) &&
     -+			    i < effective_refmap->nr; i++) {
     -+			struct refspec_item *map = &effective_refmap->items[i];
     -+
     -+			if (map->pattern)
     -+				match_refname_with_pattern(map->src, default_branch,
     -+							    map->dst, &default_branch_dst);
     -+			else if (!strcmp(map->src, default_branch))
     -+				default_branch_dst = xstrdup(map->dst);
     -+		}
     -+
     -+		if (default_branch_dst) {
     -+			struct refspec_item head_item = { .force = 1 };
     -+
     -+			head_item.src = xstrdup("HEAD");
     -+			head_item.dst = default_branch_dst;
     -+			get_fetch_map(remote_refs, &head_item, &tail, 1);
     -+			free(head_item.src);
     -+		}
     -+
     -+		if (default_branch && bootstrap_head_branch) {
     -+			const char *branch_name = default_branch;
     -+			skip_prefix(branch_name, "refs/heads/", &branch_name);
     -+			*bootstrap_head_branch = xstrdup(branch_name);
     -+		}
     -+
     -+		free(default_branch);
     -+		free(default_branch_dst);
      +		string_list_clear(&tracked, 0);
      +
      +		rs = &inferred_rs;
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       	return ref_map;
       }
       
     -@@ builtin/fetch.c: static void warn_set_head(const char *remote, const char *head_name,
     - }
     - 
     - static int set_head(const struct ref *remote_refs, struct remote *remote,
     --			int follow_remote_head)
     -+			int follow_remote_head, const char *known_head_branch)
     - {
     - 	int result = 0, create_only, baremirror, was_detached;
     - 	struct strbuf b_head = STRBUF_INIT, b_remote_head = STRBUF_INIT,
     - 		      b_local_head = STRBUF_INIT;
     - 	const char *no_warn_branch = remote->no_warn_branch;
     - 	char *head_name = NULL;
     --	struct ref *ref, *matches;
     -+	struct ref *ref, *matches = NULL;
     - 	struct ref *fetch_map = NULL, **fetch_map_tail = &fetch_map;
     - 	struct refspec_item refspec = {
     - 		.force = 0,
     -@@ builtin/fetch.c: static int set_head(const struct ref *remote_refs, struct remote *remote,
     - 	struct string_list heads = STRING_LIST_INIT_DUP;
     - 	struct ref_store *refs = get_main_ref_store(the_repository);
     - 
     --	get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);
     --	matches = guess_remote_head(find_ref_by_name(remote_refs, "HEAD"),
     --				    fetch_map, REMOTE_GUESS_HEAD_ALL);
     --	for (ref = matches; ref; ref = ref->next) {
     --		string_list_append(&heads, strip_refshead(ref->name));
     --	}
     -+	if (known_head_branch) {
     -+		head_name = xstrdup(known_head_branch);
     -+	} else {
     -+		get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);
     -+		matches = guess_remote_head(find_ref_by_name(remote_refs, "HEAD"),
     -+					    fetch_map, REMOTE_GUESS_HEAD_ALL);
     -+		for (ref = matches; ref; ref = ref->next) {
     -+			string_list_append(&heads, strip_refshead(ref->name));
     -+		}
     - 
     --	if (!heads.nr)
     --		result = 1;
     --	else if (heads.nr > 1)
     --		result = 1;
     --	else
     --		head_name = xstrdup(heads.items[0].string);
     -+		if (!heads.nr)
     -+			result = 1;
     -+		else if (heads.nr > 1)
     -+			result = 1;
     -+		else
     -+			head_name = xstrdup(heads.items[0].string);
     -+	}
     - 
     - 	if (!head_name)
     - 		goto cleanup;
     -@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     - 	struct strmap rejected_refs = STRMAP_INIT;
     - 	int summary_width = 0;
     - 	int follow_remote_head;
     -+	char *bootstrap_head_branch = NULL;
     - 
     - 	if (tags == TAGS_DEFAULT) {
     - 		if (transport->remote->fetch_tags == 2)
      @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
       		refspec_ref_prefixes(rs, &transport_ls_refs_options.ref_prefixes);
       	} else {
     @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
      +				strvec_push(&transport_ls_refs_options.ref_prefixes,
      +					    item->string);
      +			string_list_clear(&tracked, 0);
     -+			strvec_push(&transport_ls_refs_options.ref_prefixes, "HEAD");
      +		} else if (transport->remote->fetch.nr) {
       			refspec_ref_prefixes(&transport->remote->fetch,
       					     &transport_ls_refs_options.ref_prefixes);
     @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
       			for (i = 0; i < branch->merge_nr; i++) {
       				strvec_push(&transport_ls_refs_options.ref_prefixes,
      @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     - 	transport_ls_refs_options_release(&transport_ls_refs_options);
       
       	ref_map = get_ref_map(transport->remote, remote_refs, rs,
     --			      tags, &autotags);
     -+			      tags, &autotags, &bootstrap_head_branch);
     + 			      tags, &autotags);
      +
       	if (!update_head_ok)
       		check_not_current_branch(ref_map);
       
     -@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     - 		 * Way too many cases where this can go wrong so let's just
     - 		 * ignore errors and fail silently for now.
     - 		 */
     --		set_head(remote_refs, transport->remote, follow_remote_head);
     -+		set_head(remote_refs, transport->remote, follow_remote_head,
     -+			 bootstrap_head_branch);
     - 	}
     - 
     - cleanup:
     -+	free(bootstrap_head_branch);
     - 	/*
     - 	 * When using batched updates, we want to commit the non-rejected
     - 	 * updates and also handle the rejections.
      
       ## remote.c ##
      @@ remote.c: int branch_merge_matches(struct branch *branch,
     @@ t/t5585-fetch-refmap.sh (new)
      +
      +When a remote has a refmap configured but no fetch refspec, a
      +refspec-less fetch infers what to fetch from the local branches whose
     -+@{upstream} is on that remote, plus the default branch of that remote,
     -+which is always included so that it is available even before anything
     -+is set up to track it.
     ++@{upstream} is on that remote.
      +'
      +
      +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
     @@ t/t5585-fetch-refmap.sh (new)
      +	)
      +'
      +
     -+test_expect_success 'fetching the new remote does not need every branch it has' '
     ++test_expect_success 'a bare fetch needs nothing until a branch is tracked' '
      +	(
      +		cd client &&
      +		git fetch upstream &&
      +		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
     -+		cat >expect <<-\EOF &&
     -+		refs/remotes/upstream/HEAD
     -+		refs/remotes/upstream/main
     -+		EOF
     -+		test_cmp expect actual
     ++		test_must_be_empty actual
      +	)
      +'
      +
     -+test_expect_success 'set-upstream-to now resolves right after that first fetch' '
     ++test_expect_success 'an explicit one-time fetch lets a branch be tracked' '
      +	(
      +		cd client &&
     -+		git branch --set-upstream-to=upstream &&
     ++		git fetch upstream main &&
     ++		git branch --set-upstream-to=upstream/main &&
      +		test_cmp_config upstream branch.main.remote &&
      +		test_cmp_config refs/heads/main branch.main.merge
      +	)
 3:  604e584956 ! 3:  e2f072c254 remote: add "git remote add --limited-fetch"
     @@ Documentation/git-remote.adoc: the `refs/remotes/<name>/` namespace, a refspec t
      +With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
      +that tracks all branches, `remote.<name>.refmap` is set up so that a
      +refspec-less `git fetch <name>` only fetches branches our local branches
     -+are built on, plus the remote's default branch. See the `--refmap` entry
     -+in linkgit:git-fetch[1] for details.
     ++are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
     ++details.
      ++
       With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
       up to point at remote's _<master>_ branch. See also the set-head command.
 4:  31462e9445 ! 4:  43b9711a2c remote: default to --limited-fetch in a shallow repository
     @@ Documentation/git-remote.adoc: Add a remote named _<name>_ for the repository at
      @@ Documentation/git-remote.adoc: With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
       that tracks all branches, `remote.<name>.refmap` is set up so that a
       refspec-less `git fetch <name>` only fetches branches our local branches
     - are built on, plus the remote's default branch. See the `--refmap` entry
     --in linkgit:git-fetch[1] for details.
     -+in linkgit:git-fetch[1] for details. `--no-limited-fetch` explicitly
     -+disables this, overriding the shallow-repository default described above.
     + are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
     +-details.
     ++details. `--no-limited-fetch` explicitly disables this, overriding the
     ++shallow-repository default described above.
       +
       With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
       up to point at remote's _<master>_ branch. See also the set-head command.
     @@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only
      +		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
      +			remote.upstream.refmap &&
      +		test_must_fail git config get remote.upstream.fetch &&
     ++		git fetch upstream main &&
     ++		git branch --set-upstream-to=upstream/main &&
     ++		test_cmp_config upstream branch.main.remote &&
     ++		test_cmp_config refs/heads/main branch.main.merge &&
      +		git fetch upstream &&
      +		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
      +		cat >expect <<-\EOF &&
      +		refs/remotes/upstream/HEAD
      +		refs/remotes/upstream/main
      +		EOF
     -+		test_cmp expect actual &&
     -+		git branch --set-upstream-to=upstream &&
     -+		test_cmp_config upstream branch.main.remote &&
     -+		test_cmp_config refs/heads/main branch.main.merge
     ++		test_cmp expect actual
      +	)
      +'
      +

gitgitgadget

From: Harald Nordgren <[email protected]>

Add a per-remote config variable, remote.<name>.refmap, that provides the default value for --refmap the same way remote.<name>.fetch already provides the default refspecs to fetch. Like --refmap itself, it only maps refs that are actually being fetched, so it has nothing to do when there is nothing explicit to fetch, on the command line or via remote.<name>.fetch.

---
 Documentation/config/remote.adoc |  5 +++++
 Documentation/fetch-options.adoc |  3 +++
 builtin/fetch.c                  |  8 +++++---
 remote.c                         | 12 +++++++++++-
 remote.h                         |  6 ++++++
 t/t5510-fetch.sh                 | 17 +++++++++++++++++
 6 files changed, 47 insertions(+), 4 deletions(-)

index 3a20d0f752..103eada406 100644
@@ -33,6 +33,11 @@ remote.<name>.fetch::
 	The default set of "refspec" for linkgit:git-fetch[1]. See
 	linkgit:git-fetch[1].
 
+remote.<name>.refmap::
+	The default value of the `--refmap` option for linkgit:git-fetch[1].
+	Used to map remote refs being fetched to remote-tracking refs to
+	store. See the `--refmap` entry in linkgit:git-fetch[1].
+
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See
 	linkgit:git-push[1].

index 035f780e58..538914bc6e 100644
@@ -244,6 +244,9 @@ endif::git-pull[]
 	refspecs and rely entirely on the refspecs supplied as
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
++
+`remote.<name>.refmap` provides the default value for this option, the
+same way `remote.<name>.fetch` provides the default refspecs to fetch.
 
 `-t`::
 `--tags`::

index 533fdfe7d8..7651b41139 100644
@@ -509,6 +509,8 @@ static struct ref *get_ref_map(struct remote *remote,
 	struct ref *rm;
 	struct ref *ref_map = NULL;
 	struct ref **tail = &ref_map;
+	struct refspec *effective_refmap =
+		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;
 
 	/* opportunistically-updated references: */
 	struct ref *orefs = NULL, **oref_tail = &orefs;

@@ -552,14 +554,14 @@ static struct ref *get_ref_map(struct remote *remote,
 		 * by ref_remove_duplicates() in favor of one of these
 		 * opportunistic entries with FETCH_HEAD_IGNORE.
 		 */
-		if (refmap.nr)
-			fetch_refspec = &refmap;
+		if (effective_refmap && effective_refmap->nr)
+			fetch_refspec = effective_refmap;
 		else
 			fetch_refspec = &remote->fetch;
 
 		for (i = 0; i < fetch_refspec->nr; i++)
 			get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1);
-	} else if (refmap.nr) {
+	} else if (effective_refmap && effective_refmap->nr) {
 		die("--refmap option is only meaningful with command-line refspec(s)");
 	} else {
 		/* Use the defaults */

index fe62068463..017cd9d13e 100644
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,
 	ret->name = xstrndup(name, len);
 	refspec_init_push(&ret->push, the_hash_algo);
 	refspec_init_fetch(&ret->fetch, the_hash_algo);
+	refspec_init_fetch(&ret->refmap, the_hash_algo);
 	string_list_init_dup(&ret->server_options);
 	string_list_init_dup(&ret->negotiation_restrict);
 	string_list_init_dup(&ret->negotiation_include);

@@ -176,6 +177,7 @@ static void remote_clear(struct remote *remote)
 
 	refspec_clear(&remote->push);
 	refspec_clear(&remote->fetch);
+	refspec_clear(&remote->refmap);
 
 	free((char *)remote->receivepack);
 	free((char *)remote->uploadpack);

@@ -539,6 +541,12 @@ static int handle_config(const char *key, const char *value,
 			return -1;
 		refspec_append(&remote->fetch, v);
 		free(v);
+	} else if (!strcmp(subkey, "refmap")) {
+		char *v;
+		if (git_config_string(&v, key, value))
+			return -1;
+		refspec_append(&remote->refmap, v);
+		free(v);
 	} else if (!strcmp(subkey, "receivepack")) {
 		char *v;
 		if (git_config_string(&v, key, value))

@@ -988,7 +996,9 @@ void ref_push_report_free(struct ref_push_report *report)
 
 int remote_find_tracking(struct remote *remote, struct refspec_item *refspec)
 {
-	return refspec_find_match(&remote->fetch, refspec);
+	if (remote->fetch.nr)
+		return refspec_find_match(&remote->fetch, refspec);
+	return refspec_find_match(&remote->refmap, refspec);
 }
 
 static struct ref *alloc_ref_with_prefix(const char *prefix, size_t prefixlen,

index cca02033b9..ac485a584d 100644
@@ -90,6 +90,12 @@ struct remote {
 
 	struct refspec fetch;
 
+	/*
+	 * How to map refs fetched without an explicit destination into our
+	 * own namespace, the same as the --refmap command line option.
+	 */
+	struct refspec refmap;
+
 	/*
 	 * The setting for whether to fetch tags (as a separate rule from the
 	 * configured refspecs);

index a8d38d9176..bd853d0a84 100755
@@ -927,6 +927,23 @@ test_expect_success 'explicit --refmap option overrides remote.*.fetch' '
 	)
 '
 
+test_expect_success 'remote.*.refmap acts like --refmap on the command line' '
+	test_when_finished "git -C three config --unset remote.origin.refmap" &&
+	git branch -f side &&
+	git -C three config remote.origin.refmap \
+		"refs/heads/*:refs/remotes/other/*" &&
+	(
+		cd three &&
+		git update-ref refs/remotes/origin/main base-origin-main &&
+		o=$(git rev-parse --verify refs/remotes/origin/main) &&
+		git fetch origin main &&
+		n=$(git rev-parse --verify refs/remotes/origin/main) &&
+		test "$o" = "$n" &&
+		test_must_fail git rev-parse --verify refs/remotes/origin/side &&
+		git rev-parse --verify refs/remotes/other/main
+	)
+'
+
 test_expect_success 'explicitly empty --refmap option disables remote.*.fetch' '
 	git branch -f side &&
 	(

gitgitgadget

From: Harald Nordgren <[email protected]>

Configuring remote.<name>.refmap without remote.<name>.fetch used to make a refspec-less "git fetch <name>" fail with "--refmap option is only meaningful with command-line refspec(s)", since a refmap only says where to put fetched refs, not what to fetch.

Make that case infer what to fetch: the local branches whose @{upstream} is already on that remote. This lets a remote be configured to fetch only the branches actually in use, without listing them by hand in remote.<name>.fetch, and without needing to touch the command line every time.

---
 Documentation/config/remote.adoc |   5 +-
 Documentation/fetch-options.adoc |   6 +-
 builtin/fetch.c                  |  52 ++++++++++++--
 remote.c                         |  29 ++++++++
 remote.h                         |   3 +
 t/meson.build                    |   1 +
 t/t5585-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 7 files changed, 205 insertions(+), 10 deletions(-)
 create mode 100755 t/t5585-fetch-refmap.sh

index 103eada406..7c52e443a5 100644
@@ -36,7 +36,10 @@ remote.<name>.fetch::
 remote.<name>.refmap::
 	The default value of the `--refmap` option for linkgit:git-fetch[1].
 	Used to map remote refs being fetched to remote-tracking refs to
-	store. See the `--refmap` entry in linkgit:git-fetch[1].
+	store. If `remote.<name>.fetch` is not set either, a refspec-less
+	fetch infers what to fetch from local branches built on this
+	remote, instead of fetching every branch it has. See the
+	`--refmap` entry in linkgit:git-fetch[1].
 
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See

index 538914bc6e..315daf97d7 100644
@@ -245,8 +245,10 @@ endif::git-pull[]
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
 +
-`remote.<name>.refmap` provides the default value for this option, the
-same way `remote.<name>.fetch` provides the default refspecs to fetch.
+When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
+there is nothing to fetch, neither on the command line nor from
+`remote.<name>.fetch`, Git infers what to fetch from the local branches
+whose `@{upstream}` is on that remote.
 
 `-t`::
 `--tags`::

index 017cd9d13e..a5deb9b929 100644
@@ -1884,6 +1884,35 @@ int branch_merge_matches(struct branch *branch,
 	return refname_match(branch->merge[i]->src, refname);
 }
 
+struct branches_tracking_remote_cb_data {
+	struct remote *remote;
+	struct string_list *tracked;
+};
+
+static int add_if_tracking_remote(const struct reference *ref, void *cb_data)
+{
+	struct branches_tracking_remote_cb_data *data = cb_data;
+	struct branch *branch;
+
+	branch = branch_get(ref->name);
+	if (!branch_has_merge_config(branch) ||
+	    strcmp(branch->remote_name, data->remote->name))
+		return 0;
+
+	for (int i = 0; i < branch->merge_nr; i++)
+		string_list_insert(data->tracked, branch->merge[i]->src);
+
+	return 0;
+}
+
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked)
+{
+	struct branches_tracking_remote_cb_data data = { remote, tracked };
+
+	refs_for_each_branch_ref(get_main_ref_store(the_repository),
+				  add_if_tracking_remote, &data);
+}
+
 __attribute__((format (printf,2,3)))
 static char *error_buf(struct strbuf *err, const char *fmt, ...)
 {

index ac485a584d..da6c3ef53a 100644
@@ -359,6 +359,9 @@ int branch_has_merge_config(struct branch *branch);
 
 int branch_merge_matches(struct branch *, int n, const char *);
 
+/* fills tracked with the refname of every local branch's upstream on remote */
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked);
+
 /* list of the remote in a group as configured */
 struct remote_group_data {
 	const char *name;

index 3ca7b27104..ab83d5409c 100644
@@ -728,6 +728,7 @@ integration_tests = [
   't5582-fetch-negative-refspec.sh',
   't5583-push-branches.sh',
   't5584-http-429-retry.sh',
+  't5585-fetch-refmap.sh',
   't5600-clone-fail-cleanup.sh',
   't5601-clone.sh',
   't5602-clone-remote-exec.sh',

gitgitgadget

Please renumber. This number is taken by another topic already in flight, I think.

From: Harald Nordgren <[email protected]>

A remote added the ordinary way tracks every branch it has, via a wildcard remote.<name>.fetch refspec. That is wasteful for a remote whose history is only worth following for the branches actually in use locally, and it can make "git fetch" negotiate history for branches nobody asked for.

Give "git remote add" a --limited-fetch option that sets up remote.<name>.refmap instead of remote.<name>.fetch, so that "git fetch <name>" only fetches the branches already tracked, plus the remote's default branch, as described in the previous commit. It is rejected together with -t/--track or --mirror, since those already say explicitly what to fetch.

---
 Documentation/git-remote.adoc |  8 +++++++-
 builtin/remote.c              | 28 ++++++++++++++++++++++------
 t/t5505-remote.sh             | 16 ++++++++++++++++
 3 files changed, 45 insertions(+), 7 deletions(-)

index eaae30aa88..4255f8b3e6 100644
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [synopsis]
 git remote [-v | --verbose]
-git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] <name> <URL>
+git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] [--[no-]limited-fetch] <name> <URL>
 git remote rename [--[no-]progress] <old> <new>
 git remote remove <name>
 git remote set-head <name> (-a | --auto | -d | --delete | <branch>)

@@ -70,6 +70,12 @@ the `refs/remotes/<name>/` namespace, a refspec to track only _<branch>_
 is created.  You can give more than one `-t <branch>` to track
 multiple branches without grabbing all branches.
 +
+With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
+that tracks all branches, `remote.<name>.refmap` is set up so that a
+refspec-less `git fetch <name>` only fetches branches our local branches
+are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
+details.
++
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.
 +

index de989ea3ba..f036dd5d6f 100644
@@ -179,6 +179,7 @@ static int add(int argc, const char **argv, const char *prefix,
 {
 	int fetch = 0, fetch_tags = TAGS_DEFAULT;
 	unsigned mirror = MIRROR_NONE;
+	int limited_fetch = -1; /* unspecified */
 	struct string_list track = STRING_LIST_INIT_NODUP;
 	const char *master = NULL;
 	struct remote *remote;

@@ -198,6 +199,8 @@ static int add(int argc, const char **argv, const char *prefix,
 		OPT_CALLBACK_F(0, "mirror", &mirror, "(push|fetch)",
 			N_("set up remote as a mirror to push to or fetch from"),
 			PARSE_OPT_OPTARG | PARSE_OPT_COMP_ARG, parse_mirror_opt),
+		OPT_BOOL(0, "limited-fetch", &limited_fetch,
+			N_("fetch only the branches we build on, instead of every branch")),
 		OPT_END()
 	};
 

@@ -211,6 +214,10 @@ static int add(int argc, const char **argv, const char *prefix,
 		die(_("specifying a master branch makes no sense with --mirror"));
 	if (mirror && !(mirror & MIRROR_FETCH) && track.nr)
 		die(_("specifying branches to track makes sense only with fetch mirrors"));
+	if (limited_fetch == 1 && track.nr)
+		die(_("--limited-fetch does not make sense with -t/--track"));
+	if (limited_fetch == 1 && mirror)
+		die(_("--limited-fetch does not make sense with --mirror"));
 
 	name = argv[0];
 	url = argv[1];

@@ -230,13 +237,22 @@ static int add(int argc, const char **argv, const char *prefix,
 	repo_config_set(the_repository, buf.buf, url);
 
 	if (!mirror || mirror & MIRROR_FETCH) {
+		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
+			limited_fetch == 1;
+
 		strbuf_reset(&buf);
-		strbuf_addf(&buf, "remote.%s.fetch", name);
-		if (track.nr == 0)
-			string_list_append(&track, "*");
-		for (size_t i = 0; i < track.nr; i++) {
-			add_branch(buf.buf, track.items[i].string,
-				   name, mirror, &buf2);
+		if (use_limited_fetch) {
+			strbuf_addf(&buf, "remote.%s.refmap", name);
+			strbuf_reset(&buf2);
+			strbuf_addf(&buf2, "+refs/heads/*:refs/remotes/%s/*", name);
+			repo_config_set(the_repository, buf.buf, buf2.buf);
+		} else {
+			strbuf_addf(&buf, "remote.%s.fetch", name);
+			if (track.nr == 0)
+				string_list_append(&track, "*");
+			for (size_t i = 0; i < track.nr; i++)
+				add_branch(buf.buf, track.items[i].string,
+					   name, mirror, &buf2);
 		}
 	}
 

index 5fbfcb0848..0168d5abfe 100755
@@ -137,6 +137,22 @@ test_expect_success 'filters are listed by git remote -v only' '
 	test_grep ! "\[blob:none\]" out
 '
 
+test_expect_success '--limited-fetch works in a full repository too' '
+	test_when_finished "rm -rf full-add" &&
+	git clone --no-local one full-add &&
+	(
+		cd full-add &&
+		git remote add --limited-fetch upstream ../two &&
+		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
+			remote.upstream.refmap &&
+		test_must_fail git config get remote.upstream.fetch
+	)
+'
+
+test_expect_success '--limited-fetch conflicts with -t' '
+	test_must_fail git remote add --limited-fetch -t main upstream ../two
+'
+
 test_expect_success 'check remote-tracking' '
 	(
 		cd test &&

gitgitgadget

From: Harald Nordgren <[email protected]>

Adding a second remote to a shallow, single-branch clone used to still fetch every branch that remote has, since "git remote add" always set up a wildcard remote.<name>.fetch refspec regardless of how shallow the repository already was. That defeats the purpose of having cloned shallow and single-branch in the first place, and can make a plain "git fetch" on that remote hang or take a very long time on a repository with many branches.

Turn --limited-fetch on by default when the repository is already shallow and neither -t/--track nor --mirror was given, so that adding a remote there does not by itself commit to following every branch it has. --no-limited-fetch keeps the previous behavior for whoever wants it.

---
 Documentation/git-remote.adoc |  8 ++++-
 builtin/remote.c              |  5 ++-
 t/t5505-remote.sh             | 60 +++++++++++++++++++++++++++++++++++
 3 files changed, 71 insertions(+), 2 deletions(-)

index 4255f8b3e6..80f8a4a183 100644
@@ -52,6 +52,11 @@ Add a remote named _<name>_ for the repository at
 _<URL>_.  The command `git fetch <name>` can then be used to create and
 update remote-tracking branches `<name>/<branch>`.
 +
+If the repository is already a shallow repository (see linkgit:git-clone[1]
+`--depth`) and neither `-t`, `--mirror` nor `--no-limited-fetch` is given,
+`--limited-fetch` is turned on by default, so that `git fetch <name>` does
+not need to negotiate history for every branch the remote has.
++
 With `-f` option, `git fetch <name>` is run immediately after
 the remote information is set up.
 +

@@ -74,7 +79,8 @@ With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
 that tracks all branches, `remote.<name>.refmap` is set up so that a
 refspec-less `git fetch <name>` only fetches branches our local branches
 are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
-details.
+details. `--no-limited-fetch` explicitly disables this, overriding the
+shallow-repository default described above.
 +
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.

index f036dd5d6f..0cf1126976 100644
@@ -16,6 +16,7 @@
 #include "rebase.h"
 #include "refs.h"
 #include "refspec.h"
+#include "shallow.h"
 #include "odb.h"
 #include "strvec.h"
 #include "commit-reach.h"

@@ -238,7 +239,9 @@ static int add(int argc, const char **argv, const char *prefix,
 
 	if (!mirror || mirror & MIRROR_FETCH) {
 		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
-			limited_fetch == 1;
+			(limited_fetch == 1 ||
+			 (limited_fetch == -1 &&
+			  is_repository_shallow(the_repository)));
 
 		strbuf_reset(&buf);
 		if (use_limited_fetch) {

gitgitgadget

Will replace.

As my eyes have been contaminated by and biased for this topic, I'll spend my time on other topics first to let my eyes and brain "forget" about it before I revisit these patches. I'd really welcome reviews with fresh eyes to happen in the meantime.

Thanks.

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v5:

* Renumber test file from t5585 to t5586.

Changes in v4:

* Removed the automatic default-branch fetch. A fresh remote fetches nothing until you track a branch explicitly.
* Fixed fetch report showing "new ref HEAD" instead of "new branch "
* Reworded remote..refmap docs and commit message.

Changes in v3:

* Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap, reusing git's existing --refmap mechanism instead of inventing new refspec syntax.
* Split the change into 4 commits.

Changes in v2:

* Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (4):
fetch: add remote.<name>.refmap
fetch: infer branches to fetch from a refmap-only remote
remote: add "git remote add --limited-fetch"
remote: default to --limited-fetch in a shallow repository

 Documentation/config/remote.adoc |   8 +++
 Documentation/fetch-options.adoc |   5 ++
 Documentation/git-remote.adoc    |  14 +++-
 builtin/fetch.c                  |  60 +++++++++++++---
 builtin/remote.c                 |  31 ++++++--
 remote.c                         |  41 ++++++++++-
 remote.h                         |   9 +++
 t/meson.build                    |   1 +
 t/t5505-remote.sh                |  76 ++++++++++++++++++++
 t/t5510-fetch.sh                 |  17 +++++
 t/t5586-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 11 files changed, 363 insertions(+), 18 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh

base-commit: a018953688f1b10bddf91bff8747068f5f4746a4
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v5
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v4:

 1:  d48a7004e4 = 1:  d48a7004e4 fetch: add remote.<name>.refmap
 2:  45b26e2bb2 ! 2:  fecdbc43b8 fetch: infer branches to fetch from a refmap-only remote
     @@ t/meson.build: integration_tests = [
         't5582-fetch-negative-refspec.sh',
         't5583-push-branches.sh',
         't5584-http-429-retry.sh',
     -+  't5585-fetch-refmap.sh',
     ++  't5586-fetch-refmap.sh',
         't5600-clone-fail-cleanup.sh',
         't5601-clone.sh',
         't5602-clone-remote-exec.sh',
      
     - ## t/t5585-fetch-refmap.sh (new) ##
     + ## t/t5586-fetch-refmap.sh (new) ##
      @@
      +#!/bin/sh
      +
 3:  e2f072c254 = 3:  b45f3cf0ab remote: add "git remote add --limited-fetch"
 4:  43b9711a2c = 4:  8c2a144eb7 remote: default to --limited-fetch in a shallow repository

gitgitgadget

From: Harald Nordgren <[email protected]>

Add a per-remote config variable, remote.<name>.refmap, that provides the default value for --refmap the same way remote.<name>.fetch already provides the default refspecs to fetch. Like --refmap itself, it only maps refs that are actually being fetched, so it has nothing to do when there is nothing explicit to fetch, on the command line or via remote.<name>.fetch.

---
 Documentation/config/remote.adoc |  5 +++++
 Documentation/fetch-options.adoc |  3 +++
 builtin/fetch.c                  |  8 +++++---
 remote.c                         | 12 +++++++++++-
 remote.h                         |  6 ++++++
 t/t5510-fetch.sh                 | 17 +++++++++++++++++
 6 files changed, 47 insertions(+), 4 deletions(-)

index 3a20d0f752..103eada406 100644
@@ -33,6 +33,11 @@ remote.<name>.fetch::
 	The default set of "refspec" for linkgit:git-fetch[1]. See
 	linkgit:git-fetch[1].
 
+remote.<name>.refmap::
+	The default value of the `--refmap` option for linkgit:git-fetch[1].
+	Used to map remote refs being fetched to remote-tracking refs to
+	store. See the `--refmap` entry in linkgit:git-fetch[1].
+
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See
 	linkgit:git-push[1].

index 035f780e58..538914bc6e 100644
@@ -244,6 +244,9 @@ endif::git-pull[]
 	refspecs and rely entirely on the refspecs supplied as
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
++
+`remote.<name>.refmap` provides the default value for this option, the
+same way `remote.<name>.fetch` provides the default refspecs to fetch.
 
 `-t`::
 `--tags`::

index 533fdfe7d8..7651b41139 100644
@@ -509,6 +509,8 @@ static struct ref *get_ref_map(struct remote *remote,
 	struct ref *rm;
 	struct ref *ref_map = NULL;
 	struct ref **tail = &ref_map;
+	struct refspec *effective_refmap =
+		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;
 
 	/* opportunistically-updated references: */
 	struct ref *orefs = NULL, **oref_tail = &orefs;

@@ -552,14 +554,14 @@ static struct ref *get_ref_map(struct remote *remote,
 		 * by ref_remove_duplicates() in favor of one of these
 		 * opportunistic entries with FETCH_HEAD_IGNORE.
 		 */
-		if (refmap.nr)
-			fetch_refspec = &refmap;
+		if (effective_refmap && effective_refmap->nr)
+			fetch_refspec = effective_refmap;
 		else
 			fetch_refspec = &remote->fetch;
 
 		for (i = 0; i < fetch_refspec->nr; i++)
 			get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1);
-	} else if (refmap.nr) {
+	} else if (effective_refmap && effective_refmap->nr) {
 		die("--refmap option is only meaningful with command-line refspec(s)");
 	} else {
 		/* Use the defaults */

index fe62068463..017cd9d13e 100644
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,
 	ret->name = xstrndup(name, len);
 	refspec_init_push(&ret->push, the_hash_algo);
 	refspec_init_fetch(&ret->fetch, the_hash_algo);
+	refspec_init_fetch(&ret->refmap, the_hash_algo);
 	string_list_init_dup(&ret->server_options);
 	string_list_init_dup(&ret->negotiation_restrict);
 	string_list_init_dup(&ret->negotiation_include);

@@ -176,6 +177,7 @@ static void remote_clear(struct remote *remote)
 
 	refspec_clear(&remote->push);
 	refspec_clear(&remote->fetch);
+	refspec_clear(&remote->refmap);
 
 	free((char *)remote->receivepack);
 	free((char *)remote->uploadpack);

@@ -539,6 +541,12 @@ static int handle_config(const char *key, const char *value,
 			return -1;
 		refspec_append(&remote->fetch, v);
 		free(v);
+	} else if (!strcmp(subkey, "refmap")) {
+		char *v;
+		if (git_config_string(&v, key, value))
+			return -1;
+		refspec_append(&remote->refmap, v);
+		free(v);
 	} else if (!strcmp(subkey, "receivepack")) {
 		char *v;
 		if (git_config_string(&v, key, value))

@@ -988,7 +996,9 @@ void ref_push_report_free(struct ref_push_report *report)
 
 int remote_find_tracking(struct remote *remote, struct refspec_item *refspec)
 {
-	return refspec_find_match(&remote->fetch, refspec);
+	if (remote->fetch.nr)
+		return refspec_find_match(&remote->fetch, refspec);
+	return refspec_find_match(&remote->refmap, refspec);
 }
 
 static struct ref *alloc_ref_with_prefix(const char *prefix, size_t prefixlen,

index cca02033b9..ac485a584d 100644
@@ -90,6 +90,12 @@ struct remote {
 
 	struct refspec fetch;
 
+	/*
+	 * How to map refs fetched without an explicit destination into our
+	 * own namespace, the same as the --refmap command line option.
+	 */
+	struct refspec refmap;
+
 	/*
 	 * The setting for whether to fetch tags (as a separate rule from the
 	 * configured refspecs);

index a8d38d9176..bd853d0a84 100755
@@ -927,6 +927,23 @@ test_expect_success 'explicit --refmap option overrides remote.*.fetch' '
 	)
 '
 
+test_expect_success 'remote.*.refmap acts like --refmap on the command line' '
+	test_when_finished "git -C three config --unset remote.origin.refmap" &&
+	git branch -f side &&
+	git -C three config remote.origin.refmap \
+		"refs/heads/*:refs/remotes/other/*" &&
+	(
+		cd three &&
+		git update-ref refs/remotes/origin/main base-origin-main &&
+		o=$(git rev-parse --verify refs/remotes/origin/main) &&
+		git fetch origin main &&
+		n=$(git rev-parse --verify refs/remotes/origin/main) &&
+		test "$o" = "$n" &&
+		test_must_fail git rev-parse --verify refs/remotes/origin/side &&
+		git rev-parse --verify refs/remotes/other/main
+	)
+'
+
 test_expect_success 'explicitly empty --refmap option disables remote.*.fetch' '
 	git branch -f side &&
 	(

gitgitgadget

From: Harald Nordgren <[email protected]>

Configuring remote.<name>.refmap without remote.<name>.fetch used to make a refspec-less "git fetch <name>" fail with "--refmap option is only meaningful with command-line refspec(s)", since a refmap only says where to put fetched refs, not what to fetch.

Make that case infer what to fetch: the local branches whose @{upstream} is already on that remote. This lets a remote be configured to fetch only the branches actually in use, without listing them by hand in remote.<name>.fetch, and without needing to touch the command line every time.

---
 Documentation/config/remote.adoc |   5 +-
 Documentation/fetch-options.adoc |   6 +-
 builtin/fetch.c                  |  52 ++++++++++++--
 remote.c                         |  29 ++++++++
 remote.h                         |   3 +
 t/meson.build                    |   1 +
 t/t5586-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 7 files changed, 205 insertions(+), 10 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh

index 103eada406..7c52e443a5 100644
@@ -36,7 +36,10 @@ remote.<name>.fetch::
 remote.<name>.refmap::
 	The default value of the `--refmap` option for linkgit:git-fetch[1].
 	Used to map remote refs being fetched to remote-tracking refs to
-	store. See the `--refmap` entry in linkgit:git-fetch[1].
+	store. If `remote.<name>.fetch` is not set either, a refspec-less
+	fetch infers what to fetch from local branches built on this
+	remote, instead of fetching every branch it has. See the
+	`--refmap` entry in linkgit:git-fetch[1].
 
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See

index 538914bc6e..315daf97d7 100644
@@ -245,8 +245,10 @@ endif::git-pull[]
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
 +
-`remote.<name>.refmap` provides the default value for this option, the
-same way `remote.<name>.fetch` provides the default refspecs to fetch.
+When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
+there is nothing to fetch, neither on the command line nor from
+`remote.<name>.fetch`, Git infers what to fetch from the local branches
+whose `@{upstream}` is on that remote.
 
 `-t`::
 `--tags`::

index 017cd9d13e..a5deb9b929 100644
@@ -1884,6 +1884,35 @@ int branch_merge_matches(struct branch *branch,
 	return refname_match(branch->merge[i]->src, refname);
 }
 
+struct branches_tracking_remote_cb_data {
+	struct remote *remote;
+	struct string_list *tracked;
+};
+
+static int add_if_tracking_remote(const struct reference *ref, void *cb_data)
+{
+	struct branches_tracking_remote_cb_data *data = cb_data;
+	struct branch *branch;
+
+	branch = branch_get(ref->name);
+	if (!branch_has_merge_config(branch) ||
+	    strcmp(branch->remote_name, data->remote->name))
+		return 0;
+
+	for (int i = 0; i < branch->merge_nr; i++)
+		string_list_insert(data->tracked, branch->merge[i]->src);
+
+	return 0;
+}
+
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked)
+{
+	struct branches_tracking_remote_cb_data data = { remote, tracked };
+
+	refs_for_each_branch_ref(get_main_ref_store(the_repository),
+				  add_if_tracking_remote, &data);
+}
+
 __attribute__((format (printf,2,3)))
 static char *error_buf(struct strbuf *err, const char *fmt, ...)
 {

index ac485a584d..da6c3ef53a 100644
@@ -359,6 +359,9 @@ int branch_has_merge_config(struct branch *branch);
 
 int branch_merge_matches(struct branch *, int n, const char *);
 
+/* fills tracked with the refname of every local branch's upstream on remote */
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked);
+
 /* list of the remote in a group as configured */
 struct remote_group_data {
 	const char *name;

index 3ca7b27104..a1c7465c2b 100644
@@ -728,6 +728,7 @@ integration_tests = [
   't5582-fetch-negative-refspec.sh',
   't5583-push-branches.sh',
   't5584-http-429-retry.sh',
+  't5586-fetch-refmap.sh',
   't5600-clone-fail-cleanup.sh',
   't5601-clone.sh',
   't5602-clone-remote-exec.sh',

gitgitgadget

From: Harald Nordgren <[email protected]>

A remote added the ordinary way tracks every branch it has, via a wildcard remote.<name>.fetch refspec. That is wasteful for a remote whose history is only worth following for the branches actually in use locally, and it can make "git fetch" negotiate history for branches nobody asked for.

Give "git remote add" a --limited-fetch option that sets up remote.<name>.refmap instead of remote.<name>.fetch, so that "git fetch <name>" only fetches the branches already tracked, plus the remote's default branch, as described in the previous commit. It is rejected together with -t/--track or --mirror, since those already say explicitly what to fetch.

---
 Documentation/git-remote.adoc |  8 +++++++-
 builtin/remote.c              | 28 ++++++++++++++++++++++------
 t/t5505-remote.sh             | 16 ++++++++++++++++
 3 files changed, 45 insertions(+), 7 deletions(-)

index eaae30aa88..4255f8b3e6 100644
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [synopsis]
 git remote [-v | --verbose]
-git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] <name> <URL>
+git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] [--[no-]limited-fetch] <name> <URL>
 git remote rename [--[no-]progress] <old> <new>
 git remote remove <name>
 git remote set-head <name> (-a | --auto | -d | --delete | <branch>)

@@ -70,6 +70,12 @@ the `refs/remotes/<name>/` namespace, a refspec to track only _<branch>_
 is created.  You can give more than one `-t <branch>` to track
 multiple branches without grabbing all branches.
 +
+With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
+that tracks all branches, `remote.<name>.refmap` is set up so that a
+refspec-less `git fetch <name>` only fetches branches our local branches
+are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
+details.
++
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.
 +

index de989ea3ba..f036dd5d6f 100644
@@ -179,6 +179,7 @@ static int add(int argc, const char **argv, const char *prefix,
 {
 	int fetch = 0, fetch_tags = TAGS_DEFAULT;
 	unsigned mirror = MIRROR_NONE;
+	int limited_fetch = -1; /* unspecified */
 	struct string_list track = STRING_LIST_INIT_NODUP;
 	const char *master = NULL;
 	struct remote *remote;

@@ -198,6 +199,8 @@ static int add(int argc, const char **argv, const char *prefix,
 		OPT_CALLBACK_F(0, "mirror", &mirror, "(push|fetch)",
 			N_("set up remote as a mirror to push to or fetch from"),
 			PARSE_OPT_OPTARG | PARSE_OPT_COMP_ARG, parse_mirror_opt),
+		OPT_BOOL(0, "limited-fetch", &limited_fetch,
+			N_("fetch only the branches we build on, instead of every branch")),
 		OPT_END()
 	};
 

@@ -211,6 +214,10 @@ static int add(int argc, const char **argv, const char *prefix,
 		die(_("specifying a master branch makes no sense with --mirror"));
 	if (mirror && !(mirror & MIRROR_FETCH) && track.nr)
 		die(_("specifying branches to track makes sense only with fetch mirrors"));
+	if (limited_fetch == 1 && track.nr)
+		die(_("--limited-fetch does not make sense with -t/--track"));
+	if (limited_fetch == 1 && mirror)
+		die(_("--limited-fetch does not make sense with --mirror"));
 
 	name = argv[0];
 	url = argv[1];

@@ -230,13 +237,22 @@ static int add(int argc, const char **argv, const char *prefix,
 	repo_config_set(the_repository, buf.buf, url);
 
 	if (!mirror || mirror & MIRROR_FETCH) {
+		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
+			limited_fetch == 1;
+
 		strbuf_reset(&buf);
-		strbuf_addf(&buf, "remote.%s.fetch", name);
-		if (track.nr == 0)
-			string_list_append(&track, "*");
-		for (size_t i = 0; i < track.nr; i++) {
-			add_branch(buf.buf, track.items[i].string,
-				   name, mirror, &buf2);
+		if (use_limited_fetch) {
+			strbuf_addf(&buf, "remote.%s.refmap", name);
+			strbuf_reset(&buf2);
+			strbuf_addf(&buf2, "+refs/heads/*:refs/remotes/%s/*", name);
+			repo_config_set(the_repository, buf.buf, buf2.buf);
+		} else {
+			strbuf_addf(&buf, "remote.%s.fetch", name);
+			if (track.nr == 0)
+				string_list_append(&track, "*");
+			for (size_t i = 0; i < track.nr; i++)
+				add_branch(buf.buf, track.items[i].string,
+					   name, mirror, &buf2);
 		}
 	}
 

index 5fbfcb0848..0168d5abfe 100755
@@ -137,6 +137,22 @@ test_expect_success 'filters are listed by git remote -v only' '
 	test_grep ! "\[blob:none\]" out
 '
 
+test_expect_success '--limited-fetch works in a full repository too' '
+	test_when_finished "rm -rf full-add" &&
+	git clone --no-local one full-add &&
+	(
+		cd full-add &&
+		git remote add --limited-fetch upstream ../two &&
+		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
+			remote.upstream.refmap &&
+		test_must_fail git config get remote.upstream.fetch
+	)
+'
+
+test_expect_success '--limited-fetch conflicts with -t' '
+	test_must_fail git remote add --limited-fetch -t main upstream ../two
+'
+
 test_expect_success 'check remote-tracking' '
 	(
 		cd test &&

gitgitgadget

You forgot to remove ", plus the ... in the previous commit", probably, when you updated the series to stop doing so in the step [2/4].

From: Harald Nordgren <[email protected]>

Adding a second remote to a shallow, single-branch clone used to still fetch every branch that remote has, since "git remote add" always set up a wildcard remote.<name>.fetch refspec regardless of how shallow the repository already was. That defeats the purpose of having cloned shallow and single-branch in the first place, and can make a plain "git fetch" on that remote hang or take a very long time on a repository with many branches.

Turn --limited-fetch on by default when the repository is already shallow and neither -t/--track nor --mirror was given, so that adding a remote there does not by itself commit to following every branch it has. --no-limited-fetch keeps the previous behavior for whoever wants it.

---
 Documentation/git-remote.adoc |  8 ++++-
 builtin/remote.c              |  5 ++-
 t/t5505-remote.sh             | 60 +++++++++++++++++++++++++++++++++++
 3 files changed, 71 insertions(+), 2 deletions(-)

index 4255f8b3e6..80f8a4a183 100644
@@ -52,6 +52,11 @@ Add a remote named _<name>_ for the repository at
 _<URL>_.  The command `git fetch <name>` can then be used to create and
 update remote-tracking branches `<name>/<branch>`.
 +
+If the repository is already a shallow repository (see linkgit:git-clone[1]
+`--depth`) and neither `-t`, `--mirror` nor `--no-limited-fetch` is given,
+`--limited-fetch` is turned on by default, so that `git fetch <name>` does
+not need to negotiate history for every branch the remote has.
++
 With `-f` option, `git fetch <name>` is run immediately after
 the remote information is set up.
 +

@@ -74,7 +79,8 @@ With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
 that tracks all branches, `remote.<name>.refmap` is set up so that a
 refspec-less `git fetch <name>` only fetches branches our local branches
 are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
-details.
+details. `--no-limited-fetch` explicitly disables this, overriding the
+shallow-repository default described above.
 +
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.

index f036dd5d6f..0cf1126976 100644
@@ -16,6 +16,7 @@
 #include "rebase.h"
 #include "refs.h"
 #include "refspec.h"
+#include "shallow.h"
 #include "odb.h"
 #include "strvec.h"
 #include "commit-reach.h"

@@ -238,7 +239,9 @@ static int add(int argc, const char **argv, const char *prefix,
 
 	if (!mirror || mirror & MIRROR_FETCH) {
 		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
-			limited_fetch == 1;
+			(limited_fetch == 1 ||
+			 (limited_fetch == -1 &&
+			  is_repository_shallow(the_repository)));
 
 		strbuf_reset(&buf);
 		if (use_limited_fetch) {

gitgitgadget

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v6:

* Remove leftover reference to deleted default-branch logic in commit message.

Changes in v5:

* Renumber test file from t5585 to t5586.

Changes in v4:

* Removed the automatic default-branch fetch. A fresh remote fetches nothing until you track a branch explicitly.
* Fixed fetch report showing "new ref HEAD" instead of "new branch "
* Reworded remote..refmap docs and commit message.

Changes in v3:

* Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap, reusing git's existing --refmap mechanism instead of inventing new refspec syntax.
* Split the change into 4 commits.

Changes in v2:

* Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (4):
fetch: add remote.<name>.refmap
fetch: infer branches to fetch from a refmap-only remote
remote: add "git remote add --limited-fetch"
remote: default to --limited-fetch in a shallow repository

 Documentation/config/remote.adoc |   8 +++
 Documentation/fetch-options.adoc |   5 ++
 Documentation/git-remote.adoc    |  14 +++-
 builtin/fetch.c                  |  60 +++++++++++++---
 builtin/remote.c                 |  31 ++++++--
 remote.c                         |  41 ++++++++++-
 remote.h                         |   9 +++
 t/meson.build                    |   1 +
 t/t5505-remote.sh                |  76 ++++++++++++++++++++
 t/t5510-fetch.sh                 |  17 +++++
 t/t5586-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 11 files changed, 363 insertions(+), 18 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh

base-commit: c46c1e37724f0478939de636ab8ea5a89086d532
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v6
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v5:

 1:  d48a7004e4 = 1:  5c31a51bb7 fetch: add remote.<name>.refmap
 2:  fecdbc43b8 = 2:  74c177e3e2 fetch: infer branches to fetch from a refmap-only remote
 3:  b45f3cf0ab ! 3:  3abcc8915b remote: add "git remote add --limited-fetch"

@@ Commit message

Give "git remote add" a --limited-fetch option that sets up remote.<name>.refmap instead of remote.<name>.fetch, so that
- "git fetch <name>" only fetches the branches already tracked, plus
- the remote's default branch, as described in the previous commit.
- It is rejected together with -t/--track or --mirror, since those
- already say explicitly what to fetch.
+ "git fetch <name>" only fetches the branches already tracked, as
+ described in the previous commit. It is rejected together with
+ -t/--track or --mirror, since those already say explicitly what to
+ fetch.

4: 8c2a144eb7 = 4: 84d192445c remote: default to --limited-fetch in a shallow repository

gitgitgadget

From: Harald Nordgren <[email protected]>

Add a per-remote config variable, remote.<name>.refmap, that provides the default value for --refmap the same way remote.<name>.fetch already provides the default refspecs to fetch. Like --refmap itself, it only maps refs that are actually being fetched, so it has nothing to do when there is nothing explicit to fetch, on the command line or via remote.<name>.fetch.

---
 Documentation/config/remote.adoc |  5 +++++
 Documentation/fetch-options.adoc |  3 +++
 builtin/fetch.c                  |  8 +++++---
 remote.c                         | 12 +++++++++++-
 remote.h                         |  6 ++++++
 t/t5510-fetch.sh                 | 17 +++++++++++++++++
 6 files changed, 47 insertions(+), 4 deletions(-)

index 3a20d0f752..103eada406 100644
@@ -33,6 +33,11 @@ remote.<name>.fetch::
 	The default set of "refspec" for linkgit:git-fetch[1]. See
 	linkgit:git-fetch[1].
 
+remote.<name>.refmap::
+	The default value of the `--refmap` option for linkgit:git-fetch[1].
+	Used to map remote refs being fetched to remote-tracking refs to
+	store. See the `--refmap` entry in linkgit:git-fetch[1].
+
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See
 	linkgit:git-push[1].

index 035f780e58..538914bc6e 100644
@@ -244,6 +244,9 @@ endif::git-pull[]
 	refspecs and rely entirely on the refspecs supplied as
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
++
+`remote.<name>.refmap` provides the default value for this option, the
+same way `remote.<name>.fetch` provides the default refspecs to fetch.
 
 `-t`::
 `--tags`::

index b2decc6cfd..04d0a78ecf 100644
@@ -509,6 +509,8 @@ static struct ref *get_ref_map(struct remote *remote,
 	struct ref *rm;
 	struct ref *ref_map = NULL;
 	struct ref **tail = &ref_map;
+	struct refspec *effective_refmap =
+		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;
 
 	/* opportunistically-updated references: */
 	struct ref *orefs = NULL, **oref_tail = &orefs;

@@ -552,14 +554,14 @@ static struct ref *get_ref_map(struct remote *remote,
 		 * by ref_remove_duplicates() in favor of one of these
 		 * opportunistic entries with FETCH_HEAD_IGNORE.
 		 */
-		if (refmap.nr)
-			fetch_refspec = &refmap;
+		if (effective_refmap && effective_refmap->nr)
+			fetch_refspec = effective_refmap;
 		else
 			fetch_refspec = &remote->fetch;
 
 		for (i = 0; i < fetch_refspec->nr; i++)
 			get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1);
-	} else if (refmap.nr) {
+	} else if (effective_refmap && effective_refmap->nr) {
 		die("--refmap option is only meaningful with command-line refspec(s)");
 	} else {
 		/* Use the defaults */

index fe62068463..017cd9d13e 100644
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,
 	ret->name = xstrndup(name, len);
 	refspec_init_push(&ret->push, the_hash_algo);
 	refspec_init_fetch(&ret->fetch, the_hash_algo);
+	refspec_init_fetch(&ret->refmap, the_hash_algo);
 	string_list_init_dup(&ret->server_options);
 	string_list_init_dup(&ret->negotiation_restrict);
 	string_list_init_dup(&ret->negotiation_include);

@@ -176,6 +177,7 @@ static void remote_clear(struct remote *remote)
 
 	refspec_clear(&remote->push);
 	refspec_clear(&remote->fetch);
+	refspec_clear(&remote->refmap);
 
 	free((char *)remote->receivepack);
 	free((char *)remote->uploadpack);

@@ -539,6 +541,12 @@ static int handle_config(const char *key, const char *value,
 			return -1;
 		refspec_append(&remote->fetch, v);
 		free(v);
+	} else if (!strcmp(subkey, "refmap")) {
+		char *v;
+		if (git_config_string(&v, key, value))
+			return -1;
+		refspec_append(&remote->refmap, v);
+		free(v);
 	} else if (!strcmp(subkey, "receivepack")) {
 		char *v;
 		if (git_config_string(&v, key, value))

@@ -988,7 +996,9 @@ void ref_push_report_free(struct ref_push_report *report)
 
 int remote_find_tracking(struct remote *remote, struct refspec_item *refspec)
 {
-	return refspec_find_match(&remote->fetch, refspec);
+	if (remote->fetch.nr)
+		return refspec_find_match(&remote->fetch, refspec);
+	return refspec_find_match(&remote->refmap, refspec);
 }
 
 static struct ref *alloc_ref_with_prefix(const char *prefix, size_t prefixlen,

index cca02033b9..ac485a584d 100644
@@ -90,6 +90,12 @@ struct remote {
 
 	struct refspec fetch;
 
+	/*
+	 * How to map refs fetched without an explicit destination into our
+	 * own namespace, the same as the --refmap command line option.
+	 */
+	struct refspec refmap;
+
 	/*
 	 * The setting for whether to fetch tags (as a separate rule from the
 	 * configured refspecs);

index 300bd5396d..36fa9319e1 100755
@@ -927,6 +927,23 @@ test_expect_success 'explicit --refmap option overrides remote.*.fetch' '
 	)
 '
 
+test_expect_success 'remote.*.refmap acts like --refmap on the command line' '
+	test_when_finished "git -C three config --unset remote.origin.refmap" &&
+	git branch -f side &&
+	git -C three config remote.origin.refmap \
+		"refs/heads/*:refs/remotes/other/*" &&
+	(
+		cd three &&
+		git update-ref refs/remotes/origin/main base-origin-main &&
+		o=$(git rev-parse --verify refs/remotes/origin/main) &&
+		git fetch origin main &&
+		n=$(git rev-parse --verify refs/remotes/origin/main) &&
+		test "$o" = "$n" &&
+		test_must_fail git rev-parse --verify refs/remotes/origin/side &&
+		git rev-parse --verify refs/remotes/other/main
+	)
+'
+
 test_expect_success 'explicitly empty --refmap option disables remote.*.fetch' '
 	git branch -f side &&
 	(

gitgitgadget

From: Harald Nordgren <[email protected]>

Configuring remote.<name>.refmap without remote.<name>.fetch used to make a refspec-less "git fetch <name>" fail with "--refmap option is only meaningful with command-line refspec(s)", since a refmap only says where to put fetched refs, not what to fetch.

Make that case infer what to fetch: the local branches whose @{upstream} is already on that remote. This lets a remote be configured to fetch only the branches actually in use, without listing them by hand in remote.<name>.fetch, and without needing to touch the command line every time.

---
 Documentation/config/remote.adoc |   5 +-
 Documentation/fetch-options.adoc |   6 +-
 builtin/fetch.c                  |  52 ++++++++++++--
 remote.c                         |  29 ++++++++
 remote.h                         |   3 +
 t/meson.build                    |   1 +
 t/t5586-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 7 files changed, 205 insertions(+), 10 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh

index 103eada406..7c52e443a5 100644
@@ -36,7 +36,10 @@ remote.<name>.fetch::
 remote.<name>.refmap::
 	The default value of the `--refmap` option for linkgit:git-fetch[1].
 	Used to map remote refs being fetched to remote-tracking refs to
-	store. See the `--refmap` entry in linkgit:git-fetch[1].
+	store. If `remote.<name>.fetch` is not set either, a refspec-less
+	fetch infers what to fetch from local branches built on this
+	remote, instead of fetching every branch it has. See the
+	`--refmap` entry in linkgit:git-fetch[1].
 
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See

index 538914bc6e..315daf97d7 100644
@@ -245,8 +245,10 @@ endif::git-pull[]
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
 +
-`remote.<name>.refmap` provides the default value for this option, the
-same way `remote.<name>.fetch` provides the default refspecs to fetch.
+When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
+there is nothing to fetch, neither on the command line nor from
+`remote.<name>.fetch`, Git infers what to fetch from the local branches
+whose `@{upstream}` is on that remote.
 
 `-t`::
 `--tags`::

index 017cd9d13e..a5deb9b929 100644
@@ -1884,6 +1884,35 @@ int branch_merge_matches(struct branch *branch,
 	return refname_match(branch->merge[i]->src, refname);
 }
 
+struct branches_tracking_remote_cb_data {
+	struct remote *remote;
+	struct string_list *tracked;
+};
+
+static int add_if_tracking_remote(const struct reference *ref, void *cb_data)
+{
+	struct branches_tracking_remote_cb_data *data = cb_data;
+	struct branch *branch;
+
+	branch = branch_get(ref->name);
+	if (!branch_has_merge_config(branch) ||
+	    strcmp(branch->remote_name, data->remote->name))
+		return 0;
+
+	for (int i = 0; i < branch->merge_nr; i++)
+		string_list_insert(data->tracked, branch->merge[i]->src);
+
+	return 0;
+}
+
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked)
+{
+	struct branches_tracking_remote_cb_data data = { remote, tracked };
+
+	refs_for_each_branch_ref(get_main_ref_store(the_repository),
+				  add_if_tracking_remote, &data);
+}
+
 __attribute__((format (printf,2,3)))
 static char *error_buf(struct strbuf *err, const char *fmt, ...)
 {

index ac485a584d..da6c3ef53a 100644
@@ -359,6 +359,9 @@ int branch_has_merge_config(struct branch *branch);
 
 int branch_merge_matches(struct branch *, int n, const char *);
 
+/* fills tracked with the refname of every local branch's upstream on remote */
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked);
+
 /* list of the remote in a group as configured */
 struct remote_group_data {
 	const char *name;

index f65eb04684..e287a6b947 100644
@@ -730,6 +730,7 @@ integration_tests = [
   't5582-fetch-negative-refspec.sh',
   't5583-push-branches.sh',
   't5584-http-429-retry.sh',
+  't5586-fetch-refmap.sh',
   't5600-clone-fail-cleanup.sh',
   't5601-clone.sh',
   't5602-clone-remote-exec.sh',

gitgitgadget

From: Harald Nordgren <[email protected]>

A remote added the ordinary way tracks every branch it has, via a wildcard remote.<name>.fetch refspec. That is wasteful for a remote whose history is only worth following for the branches actually in use locally, and it can make "git fetch" negotiate history for branches nobody asked for.

Give "git remote add" a --limited-fetch option that sets up remote.<name>.refmap instead of remote.<name>.fetch, so that "git fetch <name>" only fetches the branches already tracked, as described in the previous commit. It is rejected together with -t/--track or --mirror, since those already say explicitly what to fetch.

---
 Documentation/git-remote.adoc |  8 +++++++-
 builtin/remote.c              | 28 ++++++++++++++++++++++------
 t/t5505-remote.sh             | 16 ++++++++++++++++
 3 files changed, 45 insertions(+), 7 deletions(-)

index eaae30aa88..4255f8b3e6 100644
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [synopsis]
 git remote [-v | --verbose]
-git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] <name> <URL>
+git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] [--[no-]limited-fetch] <name> <URL>
 git remote rename [--[no-]progress] <old> <new>
 git remote remove <name>
 git remote set-head <name> (-a | --auto | -d | --delete | <branch>)

@@ -70,6 +70,12 @@ the `refs/remotes/<name>/` namespace, a refspec to track only _<branch>_
 is created.  You can give more than one `-t <branch>` to track
 multiple branches without grabbing all branches.
 +
+With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
+that tracks all branches, `remote.<name>.refmap` is set up so that a
+refspec-less `git fetch <name>` only fetches branches our local branches
+are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
+details.
++
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.
 +

index de989ea3ba..f036dd5d6f 100644
@@ -179,6 +179,7 @@ static int add(int argc, const char **argv, const char *prefix,
 {
 	int fetch = 0, fetch_tags = TAGS_DEFAULT;
 	unsigned mirror = MIRROR_NONE;
+	int limited_fetch = -1; /* unspecified */
 	struct string_list track = STRING_LIST_INIT_NODUP;
 	const char *master = NULL;
 	struct remote *remote;

@@ -198,6 +199,8 @@ static int add(int argc, const char **argv, const char *prefix,
 		OPT_CALLBACK_F(0, "mirror", &mirror, "(push|fetch)",
 			N_("set up remote as a mirror to push to or fetch from"),
 			PARSE_OPT_OPTARG | PARSE_OPT_COMP_ARG, parse_mirror_opt),
+		OPT_BOOL(0, "limited-fetch", &limited_fetch,
+			N_("fetch only the branches we build on, instead of every branch")),
 		OPT_END()
 	};
 

@@ -211,6 +214,10 @@ static int add(int argc, const char **argv, const char *prefix,
 		die(_("specifying a master branch makes no sense with --mirror"));
 	if (mirror && !(mirror & MIRROR_FETCH) && track.nr)
 		die(_("specifying branches to track makes sense only with fetch mirrors"));
+	if (limited_fetch == 1 && track.nr)
+		die(_("--limited-fetch does not make sense with -t/--track"));
+	if (limited_fetch == 1 && mirror)
+		die(_("--limited-fetch does not make sense with --mirror"));
 
 	name = argv[0];
 	url = argv[1];

@@ -230,13 +237,22 @@ static int add(int argc, const char **argv, const char *prefix,
 	repo_config_set(the_repository, buf.buf, url);
 
 	if (!mirror || mirror & MIRROR_FETCH) {
+		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
+			limited_fetch == 1;
+
 		strbuf_reset(&buf);
-		strbuf_addf(&buf, "remote.%s.fetch", name);
-		if (track.nr == 0)
-			string_list_append(&track, "*");
-		for (size_t i = 0; i < track.nr; i++) {
-			add_branch(buf.buf, track.items[i].string,
-				   name, mirror, &buf2);
+		if (use_limited_fetch) {
+			strbuf_addf(&buf, "remote.%s.refmap", name);
+			strbuf_reset(&buf2);
+			strbuf_addf(&buf2, "+refs/heads/*:refs/remotes/%s/*", name);
+			repo_config_set(the_repository, buf.buf, buf2.buf);
+		} else {
+			strbuf_addf(&buf, "remote.%s.fetch", name);
+			if (track.nr == 0)
+				string_list_append(&track, "*");
+			for (size_t i = 0; i < track.nr; i++)
+				add_branch(buf.buf, track.items[i].string,
+					   name, mirror, &buf2);
 		}
 	}
 

index 5fbfcb0848..0168d5abfe 100755
@@ -137,6 +137,22 @@ test_expect_success 'filters are listed by git remote -v only' '
 	test_grep ! "\[blob:none\]" out
 '
 
+test_expect_success '--limited-fetch works in a full repository too' '
+	test_when_finished "rm -rf full-add" &&
+	git clone --no-local one full-add &&
+	(
+		cd full-add &&
+		git remote add --limited-fetch upstream ../two &&
+		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
+			remote.upstream.refmap &&
+		test_must_fail git config get remote.upstream.fetch
+	)
+'
+
+test_expect_success '--limited-fetch conflicts with -t' '
+	test_must_fail git remote add --limited-fetch -t main upstream ../two
+'
+
 test_expect_success 'check remote-tracking' '
 	(
 		cd test &&

gitgitgadget

From: Harald Nordgren <[email protected]>

Adding a second remote to a shallow, single-branch clone used to still fetch every branch that remote has, since "git remote add" always set up a wildcard remote.<name>.fetch refspec regardless of how shallow the repository already was. That defeats the purpose of having cloned shallow and single-branch in the first place, and can make a plain "git fetch" on that remote hang or take a very long time on a repository with many branches.

Turn --limited-fetch on by default when the repository is already shallow and neither -t/--track nor --mirror was given, so that adding a remote there does not by itself commit to following every branch it has. --no-limited-fetch keeps the previous behavior for whoever wants it.

---
 Documentation/git-remote.adoc |  8 ++++-
 builtin/remote.c              |  5 ++-
 t/t5505-remote.sh             | 60 +++++++++++++++++++++++++++++++++++
 3 files changed, 71 insertions(+), 2 deletions(-)

index 4255f8b3e6..80f8a4a183 100644
@@ -52,6 +52,11 @@ Add a remote named _<name>_ for the repository at
 _<URL>_.  The command `git fetch <name>` can then be used to create and
 update remote-tracking branches `<name>/<branch>`.
 +
+If the repository is already a shallow repository (see linkgit:git-clone[1]
+`--depth`) and neither `-t`, `--mirror` nor `--no-limited-fetch` is given,
+`--limited-fetch` is turned on by default, so that `git fetch <name>` does
+not need to negotiate history for every branch the remote has.
++
 With `-f` option, `git fetch <name>` is run immediately after
 the remote information is set up.
 +

@@ -74,7 +79,8 @@ With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
 that tracks all branches, `remote.<name>.refmap` is set up so that a
 refspec-less `git fetch <name>` only fetches branches our local branches
 are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
-details.
+details. `--no-limited-fetch` explicitly disables this, overriding the
+shallow-repository default described above.
 +
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.

index f036dd5d6f..0cf1126976 100644
@@ -16,6 +16,7 @@
 #include "rebase.h"
 #include "refs.h"
 #include "refspec.h"
+#include "shallow.h"
 #include "odb.h"
 #include "strvec.h"
 #include "commit-reach.h"

@@ -238,7 +239,9 @@ static int add(int argc, const char **argv, const char *prefix,
 
 	if (!mirror || mirror & MIRROR_FETCH) {
 		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
-			limited_fetch == 1;
+			(limited_fetch == 1 ||
+			 (limited_fetch == -1 &&
+			  is_repository_shallow(the_repository)));
 
 		strbuf_reset(&buf);
 		if (use_limited_fetch) {

gitgitgadget

Thanks. I think this is becoming much better, but I see one glitch and one design question, for which I do not yet know the right answer.

Before going there, since one of the test scripts added by this series is called 'fetch refmap', we should have a test or two to check its more basic use.

When the user configures remote.origin.refmap, the command should behave as if --refmap were given on the command line, even when the repository does not yet have a local branch that builds on anything from the remote. Attached is my attempt to do so. It does multiple things in a single block, which we may want to split up, but I am sending it here to illustrate what we might want to test and, more importantly, to present a scenario that exposes both the design question and the glitch.

The early part of the scenario goes like this:

* We create a new repository and add ".." as a remote.
* We remove remote.origin.fetch and set up remote.origin.refmap.
* When we run "git fetch origin", nothing is fetched because nothing yet builds on what we would fetch from them.

If you try to run this with [1/4] alone, however, it errors out with "fatal: --refmap option is only meaningful with command-line refspec", which is suboptimal when triggered by a configuration variable. Even though our design says that remote.*.refmap makes the command behave as if the user gave '--refmap' on the command line, applying that rule here is a bit too strict.

Fortunately, this is rectified later in the series when we begin tracking which of our local branches build on what we get from them. Even when the number of branches to fetch is zero, meaning we should pretend no command-line refspec was given with --refmap, we no longer get the same error, which is good.

The second part of the scenario explicitly specifies what to fetch on the command line and verifies that we fetch exactly that.

Then there is the last part, where the desired behavior is unclear. What should happen if the remote.origin.* configuration defines both fetch and refmap? How would we explain our choice to the users? I do not have a good answer to this design question.

As for the glitch, the last part of the test below dies with the "fatal: --refmap option is only meaningful..." message when run with the current patchset. We might decide to error out if both are set. Alternatively, we could ignore .refmap and use .fetch, or ignore .fetch and use .refmap. Whatever we decide, the "fatal: --refmap option is only meaningful..." error is not the right message to show in this situation.

Thoughts?

 t/t5586-fetch-refmap.sh | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

index b81fc48cbe..30a8d79c16 100755
@@ -22,6 +22,35 @@ test_expect_success 'setup' '
 	git checkout main
 '
 
+test_expect_success 'remote.<name>.refmap without tracking (baseline)' '
+	test_when_finished "rm -fr fetch-refmap-baseline" &&
+	git init fetch-refmap-baseline &&
+	(
+		cd fetch-refmap-baseline &&
+		git remote add origin ../ &&
+
+		# without fetch refspec, but with fetch refmap
+		git config --unset-all remote.origin.fetch &&
+		git config remote.origin.refmap "+refs/heads/*:refs/remotes/origin/*" &&
+
+		# nothing tracked, nothing fetched, no error
+		git fetch origin 2>error &&
+		test_grep ! "fatal: --refmap option is only meaningful" error &&
+		git for-each-ref --format="%(refname)" refs/remotes/ >actual &&
+		test_line_count = 0 actual &&
+
+		# nothing tracked, explicit ref on the command line
+		git fetch origin main &&
+		git for-each-ref --format="%(refname)" refs/remotes/ >actual &&
+		echo refs/remotes/origin/main >expect &&
+		test_cmp expect actual &&
+
+		# what should happen when we have both refmap and refspec?
+		git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" &&
+		git fetch origin
+	)
+'
+
 test_expect_success 'clone shallow and single-branch, then add a second remote' '
 	git clone --no-local --depth=1 --branch main --single-branch . client &&
 	(

Thanks for pointing that out. How did you find that?

I think that when both remote.<name>.fetch and remote.<name>.refmap are configured, remote.<name>.fetch can win.

By thinking the process through while writing these very basic tests.

I think this topic textually interacts with Collin's followRemoteHAD work. I tried to be careful while resolving the conflicts when I merged both to 'seen', but please double check the result when I push it out.

I range-diffed against 'seen' and everything looks normal.

I have a new version ready to send out, do you want me to hold off?

Harald

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v7:

* Fixed fatal: --refmap option is only meaningful with command-line refspec(s) firing just from configuring remote.<name>.refmap alone, with nothing else to fetch.
* When both remote.<name>.fetch and remote.<name>.refmap are configured, a refspec-less fetch follows .fetch as usual, .refmap only remaps something already being fetched by name. Added tests.

Changes in v6:

* Remove leftover reference to deleted default-branch logic in commit message.

Changes in v5:

* Renumber test file from t5585 to t5586.

Changes in v4:

* Removed the automatic default-branch fetch. A fresh remote fetches nothing until you track a branch explicitly.
* Fixed fetch report showing "new ref HEAD" instead of "new branch "
* Reworded remote..refmap docs and commit message.

Changes in v3:

* Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap, reusing git's existing --refmap mechanism instead of inventing new refspec syntax.
* Split the change into 4 commits.

Changes in v2:

* Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (4):
fetch: add remote.<name>.refmap
fetch: infer branches to fetch from a refmap-only remote
remote: add "git remote add --limited-fetch"
remote: default to --limited-fetch in a shallow repository

 Documentation/config/remote.adoc |   8 +++
 Documentation/fetch-options.adoc |   5 ++
 Documentation/git-remote.adoc    |  14 +++-
 builtin/fetch.c                  |  58 ++++++++++++---
 builtin/remote.c                 |  31 ++++++--
 remote.c                         |  41 ++++++++++-
 remote.h                         |   9 +++
 t/meson.build                    |   1 +
 t/t5505-remote.sh                |  76 ++++++++++++++++++++
 t/t5510-fetch.sh                 |  63 ++++++++++++++++
 t/t5586-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 11 files changed, 408 insertions(+), 17 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh

base-commit: 6de20f6092dcf9bdb1c8efe03db4b70c82b423dd
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v7
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v7
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v6:

1: 5c31a51bb7 ! 1: b5db64d56f fetch: add remote.<name>.refmap

     @@ Commit message
          to do when there is nothing explicit to fetch, on the command line or
          via remote.<name>.fetch.
      
     +    A refspec-less fetch with a refmap configured but nothing else to say
     +    what to fetch falls back to the same defaults as a remote with no
     +    refspec or refmap at all. Only an explicit --refmap given on the
     +    command line with no command-line refspec to go with it is still
     +    rejected, since that combination is a plain mistake to type, unlike a
     +    remote.<name>.refmap configured on its own. A remote.<name>.fetch
     +    that is also configured is used as before.
     +
          Signed-off-by: Harald Nordgren <[email protected]>
      
       ## Documentation/config/remote.adoc ##
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       		else
       			fetch_refspec = &remote->fetch;
       
     - 		for (i = 0; i < fetch_refspec->nr; i++)
     - 			get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1);
     --	} else if (refmap.nr) {
     -+	} else if (effective_refmap && effective_refmap->nr) {
     - 		die("--refmap option is only meaningful with command-line refspec(s)");
     - 	} else {
     - 		/* Use the defaults */
      
       ## remote.c ##
      @@ remote.c: static struct remote *make_remote(struct remote_state *remote_state,
     @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote
      +		git rev-parse --verify refs/remotes/other/main
      +	)
      +'
     ++
     ++check_fetched_refs () {
     ++	git for-each-ref --format="%(refname)" refs/remotes/ >actual &&
     ++	cat >expect &&
     ++	test_cmp expect actual
     ++}
     ++
     ++test_expect_success 'remote.<name>.refmap without tracking (baseline)' '
     ++	test_when_finished "rm -fr fetch-refmap-baseline fetch-refmap-upstream" &&
     ++	git init -b main fetch-refmap-upstream &&
     ++	test_commit -C fetch-refmap-upstream base &&
     ++	git -C fetch-refmap-upstream branch other &&
     ++	git init fetch-refmap-baseline &&
     ++	(
     ++		cd fetch-refmap-baseline &&
     ++		git remote add origin ../fetch-refmap-upstream &&
     ++
     ++		# Without fetch refspec, but with fetch refmap.
     ++		git config --unset-all remote.origin.fetch &&
     ++		git config remote.origin.refmap "+refs/heads/*:refs/remotes/origin/*" &&
     ++
     ++		# Nothing tracked, nothing fetched, no error.
     ++		git fetch origin &&
     ++		check_fetched_refs <<-\EOF &&
     ++		EOF
     ++
     ++		# Nothing tracked, explicit ref on the command line.
     ++		git fetch origin main &&
     ++		check_fetched_refs <<-\EOF &&
     ++		refs/remotes/origin/main
     ++		EOF
     ++
     ++		# With both refmap and fetch configured, remote.<name>.fetch
     ++		# wins: a refspec-less fetch follows it as usual, and
     ++		# remote.<name>.refmap plays no part in deciding what to
     ++		# fetch, only in remapping something that is already being
     ++		# fetched by name.
     ++		git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" &&
     ++		git fetch origin &&
     ++		check_fetched_refs <<-\EOF
     ++		refs/remotes/origin/HEAD
     ++		refs/remotes/origin/main
     ++		refs/remotes/origin/other
     ++		EOF
     ++	)
     ++'
      +
       test_expect_success 'explicitly empty --refmap option disables remote.*.fetch' '
       	git branch -f side &&
 2:  74c177e3e2 ! 2:  fd6864daaf fetch: infer branches to fetch from a refmap-only remote
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       			get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1);
      +	} else if (inferred_branches) {
      +		/* Already fully handled above. */
     - 	} else if (effective_refmap && effective_refmap->nr) {
     + 	} else if (refmap.nr) {
       		die("--refmap option is only meaningful with command-line refspec(s)");
       	} else {
      @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
 3:  3abcc8915b = 3:  a2208875b6 remote: add "git remote add --limited-fetch"
 4:  84d192445c = 4:  c8fd073de3 remote: default to --limited-fetch in a shallow repository

gitgitgadget

From: Harald Nordgren <[email protected]>

Add a per-remote config variable, remote.<name>.refmap, that provides the default value for --refmap the same way remote.<name>.fetch already provides the default refspecs to fetch. Like --refmap itself, it only maps refs that are actually being fetched, so it has nothing to do when there is nothing explicit to fetch, on the command line or via remote.<name>.fetch.

A refspec-less fetch with a refmap configured but nothing else to say what to fetch falls back to the same defaults as a remote with no refspec or refmap at all. Only an explicit --refmap given on the command line with no command-line refspec to go with it is still rejected, since that combination is a plain mistake to type, unlike a remote.<name>.refmap configured on its own. A remote.<name>.fetch that is also configured is used as before.

---
 Documentation/config/remote.adoc |  5 +++
 Documentation/fetch-options.adoc |  3 ++
 builtin/fetch.c                  |  6 ++-
 remote.c                         | 12 +++++-
 remote.h                         |  6 +++
 t/t5510-fetch.sh                 | 63 ++++++++++++++++++++++++++++++++
 6 files changed, 92 insertions(+), 3 deletions(-)

index 3a20d0f752..103eada406 100644
@@ -33,6 +33,11 @@ remote.<name>.fetch::
 	The default set of "refspec" for linkgit:git-fetch[1]. See
 	linkgit:git-fetch[1].
 
+remote.<name>.refmap::
+	The default value of the `--refmap` option for linkgit:git-fetch[1].
+	Used to map remote refs being fetched to remote-tracking refs to
+	store. See the `--refmap` entry in linkgit:git-fetch[1].
+
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See
 	linkgit:git-push[1].

index 47dea1de8e..c2101a7b39 100644
@@ -244,6 +244,9 @@ endif::git-pull[]
 	refspecs and rely entirely on the refspecs supplied as
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
++
+`remote.<name>.refmap` provides the default value for this option, the
+same way `remote.<name>.fetch` provides the default refspecs to fetch.
 
 `-t`::
 `--tags`::

index b2decc6cfd..b0ad8c0b5f 100644
@@ -509,6 +509,8 @@ static struct ref *get_ref_map(struct remote *remote,
 	struct ref *rm;
 	struct ref *ref_map = NULL;
 	struct ref **tail = &ref_map;
+	struct refspec *effective_refmap =
+		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;
 
 	/* opportunistically-updated references: */
 	struct ref *orefs = NULL, **oref_tail = &orefs;

@@ -552,8 +554,8 @@ static struct ref *get_ref_map(struct remote *remote,
 		 * by ref_remove_duplicates() in favor of one of these
 		 * opportunistic entries with FETCH_HEAD_IGNORE.
 		 */
-		if (refmap.nr)
-			fetch_refspec = &refmap;
+		if (effective_refmap && effective_refmap->nr)
+			fetch_refspec = effective_refmap;
 		else
 			fetch_refspec = &remote->fetch;
 

index 71170f36a9..99a086ea5a 100644
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,
 	ret->name = xstrndup(name, len);
 	refspec_init_push(&ret->push, the_hash_algo);
 	refspec_init_fetch(&ret->fetch, the_hash_algo);
+	refspec_init_fetch(&ret->refmap, the_hash_algo);
 	string_list_init_dup(&ret->server_options);
 	string_list_init_dup(&ret->negotiation_restrict);
 	string_list_init_dup(&ret->negotiation_include);

@@ -176,6 +177,7 @@ static void remote_clear(struct remote *remote)
 
 	refspec_clear(&remote->push);
 	refspec_clear(&remote->fetch);
+	refspec_clear(&remote->refmap);
 
 	free((char *)remote->receivepack);
 	free((char *)remote->uploadpack);

@@ -539,6 +541,12 @@ static int handle_config(const char *key, const char *value,
 			return -1;
 		refspec_append(&remote->fetch, v);
 		free(v);
+	} else if (!strcmp(subkey, "refmap")) {
+		char *v;
+		if (git_config_string(&v, key, value))
+			return -1;
+		refspec_append(&remote->refmap, v);
+		free(v);
 	} else if (!strcmp(subkey, "receivepack")) {
 		char *v;
 		if (git_config_string(&v, key, value))

@@ -988,7 +996,9 @@ void ref_push_report_free(struct ref_push_report *report)
 
 int remote_find_tracking(struct remote *remote, struct refspec_item *refspec)
 {
-	return refspec_find_match(&remote->fetch, refspec);
+	if (remote->fetch.nr)
+		return refspec_find_match(&remote->fetch, refspec);
+	return refspec_find_match(&remote->refmap, refspec);
 }
 
 static struct ref *alloc_ref_with_prefix(const char *prefix, size_t prefixlen,

index cca02033b9..ac485a584d 100644
@@ -90,6 +90,12 @@ struct remote {
 
 	struct refspec fetch;
 
+	/*
+	 * How to map refs fetched without an explicit destination into our
+	 * own namespace, the same as the --refmap command line option.
+	 */
+	struct refspec refmap;
+
 	/*
 	 * The setting for whether to fetch tags (as a separate rule from the
 	 * configured refspecs);

gitgitgadget

From: Harald Nordgren <[email protected]>

Configuring remote.<name>.refmap without remote.<name>.fetch used to make a refspec-less "git fetch <name>" fail with "--refmap option is only meaningful with command-line refspec(s)", since a refmap only says where to put fetched refs, not what to fetch.

Make that case infer what to fetch: the local branches whose @{upstream} is already on that remote. This lets a remote be configured to fetch only the branches actually in use, without listing them by hand in remote.<name>.fetch, and without needing to touch the command line every time.

---
 Documentation/config/remote.adoc |   5 +-
 Documentation/fetch-options.adoc |   6 +-
 builtin/fetch.c                  |  52 ++++++++++++--
 remote.c                         |  29 ++++++++
 remote.h                         |   3 +
 t/meson.build                    |   1 +
 t/t5586-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 7 files changed, 205 insertions(+), 10 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh

index 103eada406..7c52e443a5 100644
@@ -36,7 +36,10 @@ remote.<name>.fetch::
 remote.<name>.refmap::
 	The default value of the `--refmap` option for linkgit:git-fetch[1].
 	Used to map remote refs being fetched to remote-tracking refs to
-	store. See the `--refmap` entry in linkgit:git-fetch[1].
+	store. If `remote.<name>.fetch` is not set either, a refspec-less
+	fetch infers what to fetch from local branches built on this
+	remote, instead of fetching every branch it has. See the
+	`--refmap` entry in linkgit:git-fetch[1].
 
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See

index c2101a7b39..75899d91cc 100644
@@ -245,8 +245,10 @@ endif::git-pull[]
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
 +
-`remote.<name>.refmap` provides the default value for this option, the
-same way `remote.<name>.fetch` provides the default refspecs to fetch.
+When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
+there is nothing to fetch, neither on the command line nor from
+`remote.<name>.fetch`, Git infers what to fetch from the local branches
+whose `@{upstream}` is on that remote.
 
 `-t`::
 `--tags`::

index 99a086ea5a..c26312beea 100644
@@ -1884,6 +1884,35 @@ int branch_merge_matches(struct branch *branch,
 	return refname_match(branch->merge[i]->src, refname);
 }
 
+struct branches_tracking_remote_cb_data {
+	struct remote *remote;
+	struct string_list *tracked;
+};
+
+static int add_if_tracking_remote(const struct reference *ref, void *cb_data)
+{
+	struct branches_tracking_remote_cb_data *data = cb_data;
+	struct branch *branch;
+
+	branch = branch_get(ref->name);
+	if (!branch_has_merge_config(branch) ||
+	    strcmp(branch->remote_name, data->remote->name))
+		return 0;
+
+	for (int i = 0; i < branch->merge_nr; i++)
+		string_list_insert(data->tracked, branch->merge[i]->src);
+
+	return 0;
+}
+
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked)
+{
+	struct branches_tracking_remote_cb_data data = { remote, tracked };
+
+	refs_for_each_branch_ref(get_main_ref_store(the_repository),
+				  add_if_tracking_remote, &data);
+}
+
 __attribute__((format (printf,2,3)))
 static char *error_buf(struct strbuf *err, const char *fmt, ...)
 {

index ac485a584d..da6c3ef53a 100644
@@ -359,6 +359,9 @@ int branch_has_merge_config(struct branch *branch);
 
 int branch_merge_matches(struct branch *, int n, const char *);
 
+/* fills tracked with the refname of every local branch's upstream on remote */
+void branches_tracking_remote(struct remote *remote, struct string_list *tracked);
+
 /* list of the remote in a group as configured */
 struct remote_group_data {
 	const char *name;

index f65eb04684..e287a6b947 100644
@@ -730,6 +730,7 @@ integration_tests = [
   't5582-fetch-negative-refspec.sh',
   't5583-push-branches.sh',
   't5584-http-429-retry.sh',
+  't5586-fetch-refmap.sh',
   't5600-clone-fail-cleanup.sh',
   't5601-clone.sh',
   't5602-clone-remote-exec.sh',

gitgitgadget

Nicely explained what we want out of this new feature.

	if (rs->nr) {
		... use command line refspec ...
	} else if (transport->remote->fetch.nr) {
		... use remote.*.fetch refspec ...
	} else if (effective_refmap->nr) {
		... your new logic ...
	} else {
		struct string_list list = STRING_LIST_INIT;
		collect_upstream_from_remote(&list, remote, NULL);
		for_each_string_list_item(item, &list)
			strvec_push(&transport_ls_refs_options.ref_prefixes,
					item->string);
	}

	static int add_if_tracking_remote(...)
	{
		struct branches_tracking_remote_cb_data *data = cb_data;

		collect_upstream_from_remote(data->tracked, data->remote, ref->name);
	}

Yes, there is a conflict between 'ch/fetch-followremotehead-lazy-validation's' and my v7.

But that way it's looking now, it will go away when pushing out v8.

Harald

Thanks, will update.

Yes, this can be improved.

I'll rename it to 'infer_from_refmap'.

Good points.

Fixed.

I'll look into unifying.

The 0 return value is there so it can be used with 'refs_for_each_branch_ref' without stopping the iteration.

Yes, good idea.

Good point.

Harald

From: Harald Nordgren <[email protected]>

A remote added the ordinary way tracks every branch it has, via a wildcard remote.<name>.fetch refspec. That is wasteful for a remote whose history is only worth following for the branches actually in use locally, and it can make "git fetch" negotiate history for branches nobody asked for.

Give "git remote add" a --limited-fetch option that sets up remote.<name>.refmap instead of remote.<name>.fetch, so that "git fetch <name>" only fetches the branches already tracked, as described in the previous commit. It is rejected together with -t/--track or --mirror, since those already say explicitly what to fetch.

---
 Documentation/git-remote.adoc |  8 +++++++-
 builtin/remote.c              | 28 ++++++++++++++++++++++------
 t/t5505-remote.sh             | 16 ++++++++++++++++
 3 files changed, 45 insertions(+), 7 deletions(-)

index eaae30aa88..4255f8b3e6 100644
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [synopsis]
 git remote [-v | --verbose]
-git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] <name> <URL>
+git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=(fetch|push)] [--[no-]limited-fetch] <name> <URL>
 git remote rename [--[no-]progress] <old> <new>
 git remote remove <name>
 git remote set-head <name> (-a | --auto | -d | --delete | <branch>)

@@ -70,6 +70,12 @@ the `refs/remotes/<name>/` namespace, a refspec to track only _<branch>_
 is created.  You can give more than one `-t <branch>` to track
 multiple branches without grabbing all branches.
 +
+With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
+that tracks all branches, `remote.<name>.refmap` is set up so that a
+refspec-less `git fetch <name>` only fetches branches our local branches
+are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
+details.
++
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.
 +

index de989ea3ba..f036dd5d6f 100644
@@ -179,6 +179,7 @@ static int add(int argc, const char **argv, const char *prefix,
 {
 	int fetch = 0, fetch_tags = TAGS_DEFAULT;
 	unsigned mirror = MIRROR_NONE;
+	int limited_fetch = -1; /* unspecified */
 	struct string_list track = STRING_LIST_INIT_NODUP;
 	const char *master = NULL;
 	struct remote *remote;

@@ -198,6 +199,8 @@ static int add(int argc, const char **argv, const char *prefix,
 		OPT_CALLBACK_F(0, "mirror", &mirror, "(push|fetch)",
 			N_("set up remote as a mirror to push to or fetch from"),
 			PARSE_OPT_OPTARG | PARSE_OPT_COMP_ARG, parse_mirror_opt),
+		OPT_BOOL(0, "limited-fetch", &limited_fetch,
+			N_("fetch only the branches we build on, instead of every branch")),
 		OPT_END()
 	};
 

@@ -211,6 +214,10 @@ static int add(int argc, const char **argv, const char *prefix,
 		die(_("specifying a master branch makes no sense with --mirror"));
 	if (mirror && !(mirror & MIRROR_FETCH) && track.nr)
 		die(_("specifying branches to track makes sense only with fetch mirrors"));
+	if (limited_fetch == 1 && track.nr)
+		die(_("--limited-fetch does not make sense with -t/--track"));
+	if (limited_fetch == 1 && mirror)
+		die(_("--limited-fetch does not make sense with --mirror"));
 
 	name = argv[0];
 	url = argv[1];

@@ -230,13 +237,22 @@ static int add(int argc, const char **argv, const char *prefix,
 	repo_config_set(the_repository, buf.buf, url);
 
 	if (!mirror || mirror & MIRROR_FETCH) {
+		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
+			limited_fetch == 1;
+
 		strbuf_reset(&buf);
-		strbuf_addf(&buf, "remote.%s.fetch", name);
-		if (track.nr == 0)
-			string_list_append(&track, "*");
-		for (size_t i = 0; i < track.nr; i++) {
-			add_branch(buf.buf, track.items[i].string,
-				   name, mirror, &buf2);
+		if (use_limited_fetch) {
+			strbuf_addf(&buf, "remote.%s.refmap", name);
+			strbuf_reset(&buf2);
+			strbuf_addf(&buf2, "+refs/heads/*:refs/remotes/%s/*", name);
+			repo_config_set(the_repository, buf.buf, buf2.buf);
+		} else {
+			strbuf_addf(&buf, "remote.%s.fetch", name);
+			if (track.nr == 0)
+				string_list_append(&track, "*");
+			for (size_t i = 0; i < track.nr; i++)
+				add_branch(buf.buf, track.items[i].string,
+					   name, mirror, &buf2);
 		}
 	}
 

index 5fbfcb0848..0168d5abfe 100755
@@ -137,6 +137,22 @@ test_expect_success 'filters are listed by git remote -v only' '
 	test_grep ! "\[blob:none\]" out
 '
 
+test_expect_success '--limited-fetch works in a full repository too' '
+	test_when_finished "rm -rf full-add" &&
+	git clone --no-local one full-add &&
+	(
+		cd full-add &&
+		git remote add --limited-fetch upstream ../two &&
+		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
+			remote.upstream.refmap &&
+		test_must_fail git config get remote.upstream.fetch
+	)
+'
+
+test_expect_success '--limited-fetch conflicts with -t' '
+	test_must_fail git remote add --limited-fetch -t main upstream ../two
+'
+
 test_expect_success 'check remote-tracking' '
 	(
 		cd test &&

gitgitgadget

From: Harald Nordgren <[email protected]>

Adding a second remote to a shallow, single-branch clone used to still fetch every branch that remote has, since "git remote add" always set up a wildcard remote.<name>.fetch refspec regardless of how shallow the repository already was. That defeats the purpose of having cloned shallow and single-branch in the first place, and can make a plain "git fetch" on that remote hang or take a very long time on a repository with many branches.

Turn --limited-fetch on by default when the repository is already shallow and neither -t/--track nor --mirror was given, so that adding a remote there does not by itself commit to following every branch it has. --no-limited-fetch keeps the previous behavior for whoever wants it.

---
 Documentation/git-remote.adoc |  8 ++++-
 builtin/remote.c              |  5 ++-
 t/t5505-remote.sh             | 60 +++++++++++++++++++++++++++++++++++
 3 files changed, 71 insertions(+), 2 deletions(-)

index 4255f8b3e6..80f8a4a183 100644
@@ -52,6 +52,11 @@ Add a remote named _<name>_ for the repository at
 _<URL>_.  The command `git fetch <name>` can then be used to create and
 update remote-tracking branches `<name>/<branch>`.
 +
+If the repository is already a shallow repository (see linkgit:git-clone[1]
+`--depth`) and neither `-t`, `--mirror` nor `--no-limited-fetch` is given,
+`--limited-fetch` is turned on by default, so that `git fetch <name>` does
+not need to negotiate history for every branch the remote has.
++
 With `-f` option, `git fetch <name>` is run immediately after
 the remote information is set up.
 +

@@ -74,7 +79,8 @@ With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
 that tracks all branches, `remote.<name>.refmap` is set up so that a
 refspec-less `git fetch <name>` only fetches branches our local branches
 are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
-details.
+details. `--no-limited-fetch` explicitly disables this, overriding the
+shallow-repository default described above.
 +
 With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
 up to point at remote's _<master>_ branch. See also the set-head command.

index f036dd5d6f..0cf1126976 100644
@@ -16,6 +16,7 @@
 #include "rebase.h"
 #include "refs.h"
 #include "refspec.h"
+#include "shallow.h"
 #include "odb.h"
 #include "strvec.h"
 #include "commit-reach.h"

@@ -238,7 +239,9 @@ static int add(int argc, const char **argv, const char *prefix,
 
 	if (!mirror || mirror & MIRROR_FETCH) {
 		int use_limited_fetch = mirror == MIRROR_NONE && track.nr == 0 &&
-			limited_fetch == 1;
+			(limited_fetch == 1 ||
+			 (limited_fetch == -1 &&
+			  is_repository_shallow(the_repository)));
 
 		strbuf_reset(&buf);
 		if (use_limited_fetch) {

gitgitgadget

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v8:

* Split into a preliminary collect_upstream_from_remote() helper commit plus a simplified refmap-inference commit.
* Fixed a regression that could drop the current branch's @{upstream} when remote.<name>.fetch was narrowed.
* Threaded struct repository * through instead of hardcoding the_repository.
* Minor naming and comment cleanups.

Changes in v7:

* Fixed fatal: --refmap option is only meaningful with command-line refspec(s) firing just from configuring remote.<name>.refmap alone, with nothing else to fetch.
* When both remote.<name>.fetch and remote.<name>.refmap are configured, a refspec-less fetch follows .fetch as usual, .refmap only remaps something already being fetched by name. Added tests.

Changes in v6:

* Remove leftover reference to deleted default-branch logic in commit message.

Changes in v5:

* Renumber test file from t5585 to t5586.

Changes in v4:

* Removed the automatic default-branch fetch. A fresh remote fetches nothing until you track a branch explicitly.
* Fixed fetch report showing "new ref HEAD" instead of "new branch "
* Reworded remote..refmap docs and commit message.

Changes in v3:

* Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap, reusing git's existing --refmap mechanism instead of inventing new refspec syntax.
* Split the change into 4 commits.

Changes in v2:

* Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (5):
fetch: add remote.<name>.refmap
fetch: extract collect_upstream_from_remote() helper
fetch: infer branches to fetch from a refmap-only remote
remote: add "git remote add --limited-fetch"
remote: default to --limited-fetch in a shallow repository

 Documentation/config/remote.adoc |   8 +++
 Documentation/fetch-options.adoc |   5 ++
 Documentation/git-remote.adoc    |  14 +++-
 builtin/fetch.c                  |  84 ++++++++++++++++++----
 builtin/remote.c                 |  31 ++++++--
 remote.c                         |  51 ++++++++++++-
 remote.h                         |  19 +++++
 t/meson.build                    |   1 +
 t/t5505-remote.sh                |  76 ++++++++++++++++++++
 t/t5510-fetch.sh                 |  63 ++++++++++++++++
 t/t5586-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 11 files changed, 448 insertions(+), 23 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh

base-commit: 6de20f6092dcf9bdb1c8efe03db4b70c82b423dd
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v8
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v8
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v7:

1: b5db64d56f ! 1: 2a48c21bc8 fetch: add remote.<name>.refmap

     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       	struct ref *rm;
       	struct ref *ref_map = NULL;
       	struct ref **tail = &ref_map;
     ++	/*
     ++	 * The --refmap command line option, if given, takes precedence
     ++	 * over remote.<name>.refmap.
     ++	 */
      +	struct refspec *effective_refmap =
      +		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;

 -:  ---------- > 2:  08fea07a8b fetch: extract collect_upstream_from_remote() helper
 2:  fd6864daaf ! 3:  a39316d233 fetch: infer branches to fetch from a refmap-only remote
     @@ Documentation/fetch-options.adoc: endif::git-pull[]
      -`remote.<name>.refmap` provides the default value for this option, the
      -same way `remote.<name>.fetch` provides the default refspecs to fetch.
      +When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
     -+there is nothing to fetch, neither on the command line nor from
     -+`remote.<name>.fetch`, Git infers what to fetch from the local branches
     -+whose `@{upstream}` is on that remote.
     ++nothing to fetch is specified on the command line, nor is there a
     ++`remote.<name>.fetch`, branches from the remote that are used as the
     ++`@{upstream}` of our local branches are fetched.
       
       `-t`::
       `--tags`::
      
       ## builtin/fetch.c ##
      @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
     - 	struct ref **tail = &ref_map;
     + 	 */
       	struct refspec *effective_refmap =
       		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;
      +	struct refspec inferred_rs;
     -+	int inferred_branches = 0;
     ++	int infer_from_refmap = 0;
       
       	/* opportunistically-updated references: */
       	struct ref *orefs = NULL, **oref_tail = &orefs;
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
      +		struct string_list tracked = STRING_LIST_INIT_DUP;
      +		struct string_list_item *item;
      +
     -+		branches_tracking_remote(remote, &tracked);
     ++		branches_tracking_remote(the_repository, remote, &tracked);
      +		for_each_string_list_item(item, &tracked)
      +			refspec_append(&inferred_rs, item->string);
      +		string_list_clear(&tracked, 0);
      +
      +		rs = &inferred_rs;
     -+		inferred_branches = 1;
     ++		infer_from_refmap = 1;
      +	}
      +
       	if (rs->nr) {
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       		for (i = 0; i < rs->nr; i++) {
      -			get_fetch_map(remote_refs, &rs->items[i], &tail, 0);
      +			get_fetch_map(remote_refs, &rs->items[i], &tail,
     -+				      inferred_branches);
     ++				      infer_from_refmap);
       			if (rs->items[i].dst && rs->items[i].dst[0])
       				*autotags = 1;
       		}
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       
       		for (i = 0; i < fetch_refspec->nr; i++)
       			get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1);
     -+	} else if (inferred_branches) {
     ++	} else if (infer_from_refmap) {
      +		/* Already fully handled above. */
       	} else if (refmap.nr) {
       		die("--refmap option is only meaningful with command-line refspec(s)");
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       }
       
      @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     - 		refspec_ref_prefixes(rs, &transport_ls_refs_options.ref_prefixes);
     + 				    item->string);
     + 		string_list_clear(&tracked, 0);
       	} else {
     - 		struct branch *branch = branch_get(NULL);
     --
     --		if (transport->remote->fetch.nr) {
     -+		int tracks_this_remote = branch && branch_has_merge_config(branch) &&
     -+			!strcmp(branch->remote_name, transport->remote->name);
     ++		/*
     ++		 * The --refmap command line option, if given, takes
     ++		 * precedence over remote.<name>.refmap.
     ++		 */
      +		struct refspec *effective_refmap = refmap.nr ? &refmap :
      +			&transport->remote->refmap;
     -+		int inferred_branches = !transport->remote->fetch.nr &&
     -+			effective_refmap->nr;
     -+
     -+		if (inferred_branches) {
     -+			struct string_list tracked = STRING_LIST_INIT_DUP;
     -+			struct string_list_item *item;
     -+
     -+			branches_tracking_remote(transport->remote, &tracked);
     -+			for_each_string_list_item(item, &tracked)
     -+				strvec_push(&transport_ls_refs_options.ref_prefixes,
     -+					    item->string);
     -+			string_list_clear(&tracked, 0);
     -+		} else if (transport->remote->fetch.nr) {
     - 			refspec_ref_prefixes(&transport->remote->fetch,
     - 					     &transport_ls_refs_options.ref_prefixes);
     --			if (follow_remote_head != FOLLOW_REMOTE_NEVER)
     --				do_set_head = 1;
     - 		}
     --		if (branch && branch_has_merge_config(branch) &&
     --		    !strcmp(branch->remote_name, transport->remote->name)) {
     -+		if ((transport->remote->fetch.nr || inferred_branches) &&
     -+		    follow_remote_head != FOLLOW_REMOTE_NEVER)
     -+			do_set_head = 1;
     -+		if (tracks_this_remote) {
     - 			int i;
     - 			for (i = 0; i < branch->merge_nr; i++) {
     - 				strvec_push(&transport_ls_refs_options.ref_prefixes,
     -@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     - 
     - 	ref_map = get_ref_map(transport->remote, remote_refs, rs,
     - 			      tags, &autotags);
     -+
     - 	if (!update_head_ok)
     - 		check_not_current_branch(ref_map);
     + 		struct string_list tracked = STRING_LIST_INIT_DUP;
     + 		struct string_list_item *item;
       
     +-		collect_upstream_from_remote(the_repository, &tracked,
     +-					      transport->remote, NULL);
     ++		if (effective_refmap->nr) {
     ++			branches_tracking_remote(the_repository,
     ++						  transport->remote, &tracked);
     ++			if (follow_remote_head != FOLLOW_REMOTE_NEVER)
     ++				do_set_head = 1;
     ++		} else {
     ++			collect_upstream_from_remote(the_repository, &tracked,
     ++						      transport->remote, NULL);
     ++		}
     + 		for_each_string_list_item(item, &tracked)
     + 			strvec_push(&transport_ls_refs_options.ref_prefixes,
     + 				    item->string);
      
       ## remote.c ##
     -@@ remote.c: int branch_merge_matches(struct branch *branch,
     - 	return refname_match(branch->merge[i]->src, refname);
     +@@ remote.c: void collect_upstream_from_remote(struct repository *repo,
     + 		string_list_insert(tracked, branch->merge[i]->src);
       }
       
      +struct branches_tracking_remote_cb_data {
     ++	struct repository *repo;
      +	struct remote *remote;
      +	struct string_list *tracked;
      +};
     @@ remote.c: int branch_merge_matches(struct branch *branch,
      +static int add_if_tracking_remote(const struct reference *ref, void *cb_data)
      +{
      +	struct branches_tracking_remote_cb_data *data = cb_data;
     -+	struct branch *branch;
     -+
     -+	branch = branch_get(ref->name);
     -+	if (!branch_has_merge_config(branch) ||
     -+	    strcmp(branch->remote_name, data->remote->name))
     -+		return 0;
     -+
     -+	for (int i = 0; i < branch->merge_nr; i++)
     -+		string_list_insert(data->tracked, branch->merge[i]->src);
      +
     ++	collect_upstream_from_remote(data->repo, data->tracked, data->remote,
     ++				      ref->name);
      +	return 0;
      +}
      +
     -+void branches_tracking_remote(struct remote *remote, struct string_list *tracked)
     ++void branches_tracking_remote(struct repository *repo, struct remote *remote,
     ++			       struct string_list *tracked)
      +{
     -+	struct branches_tracking_remote_cb_data data = { remote, tracked };
     ++	struct branches_tracking_remote_cb_data data = { repo, remote, tracked };
      +
     -+	refs_for_each_branch_ref(get_main_ref_store(the_repository),
     ++	refs_for_each_branch_ref(get_main_ref_store(repo),
      +				  add_if_tracking_remote, &data);
      +}
      +
     @@ remote.c: int branch_merge_matches(struct branch *branch,
       {
      
       ## remote.h ##
     -@@ remote.h: int branch_has_merge_config(struct branch *branch);
     - 
     - int branch_merge_matches(struct branch *, int n, const char *);
     +@@ remote.h: void collect_upstream_from_remote(struct repository *repo,
     + 				   struct remote *remote,
     + 				   const char *refname);
       
      +/* fills tracked with the refname of every local branch's upstream on remote */
     -+void branches_tracking_remote(struct remote *remote, struct string_list *tracked);
     ++void branches_tracking_remote(struct repository *repo, struct remote *remote,
     ++			       struct string_list *tracked);
      +
       /* list of the remote in a group as configured */
       struct remote_group_data {
 3:  a2208875b6 = 4:  2b974bbe0a remote: add "git remote add --limited-fetch"
 4:  c8fd073de3 = 5:  db27c29086 remote: default to --limited-fetch in a shallow repository

gitgitgadget

From: Harald Nordgren <[email protected]>

Add a per-remote config variable, remote.<name>.refmap, that provides the default value for --refmap the same way remote.<name>.fetch already provides the default refspecs to fetch. Like --refmap itself, it only maps refs that are actually being fetched, so it has nothing to do when there is nothing explicit to fetch, on the command line or via remote.<name>.fetch.

A refspec-less fetch with a refmap configured but nothing else to say what to fetch falls back to the same defaults as a remote with no refspec or refmap at all. Only an explicit --refmap given on the command line with no command-line refspec to go with it is still rejected, since that combination is a plain mistake to type, unlike a remote.<name>.refmap configured on its own. A remote.<name>.fetch that is also configured is used as before.

---
 Documentation/config/remote.adoc |  5 +++
 Documentation/fetch-options.adoc |  3 ++
 builtin/fetch.c                  | 10 ++++-
 remote.c                         | 12 +++++-
 remote.h                         |  6 +++
 t/t5510-fetch.sh                 | 63 ++++++++++++++++++++++++++++++++
 6 files changed, 96 insertions(+), 3 deletions(-)

index 3a20d0f752..103eada406 100644
@@ -33,6 +33,11 @@ remote.<name>.fetch::
 	The default set of "refspec" for linkgit:git-fetch[1]. See
 	linkgit:git-fetch[1].
 
+remote.<name>.refmap::
+	The default value of the `--refmap` option for linkgit:git-fetch[1].
+	Used to map remote refs being fetched to remote-tracking refs to
+	store. See the `--refmap` entry in linkgit:git-fetch[1].
+
 remote.<name>.push::
 	The default set of "refspec" for linkgit:git-push[1]. See
 	linkgit:git-push[1].

index 47dea1de8e..c2101a7b39 100644
@@ -244,6 +244,9 @@ endif::git-pull[]
 	refspecs and rely entirely on the refspecs supplied as
 	command-line arguments. See section on "Configured Remote-tracking
 	Branches" for details.
++
+`remote.<name>.refmap` provides the default value for this option, the
+same way `remote.<name>.fetch` provides the default refspecs to fetch.
 
 `-t`::
 `--tags`::

index b2decc6cfd..c1c65c7528 100644
@@ -509,6 +509,12 @@ static struct ref *get_ref_map(struct remote *remote,
 	struct ref *rm;
 	struct ref *ref_map = NULL;
 	struct ref **tail = &ref_map;
+	/*
+	 * The --refmap command line option, if given, takes precedence
+	 * over remote.<name>.refmap.
+	 */
+	struct refspec *effective_refmap =
+		refmap.nr ? &refmap : remote ? &remote->refmap : NULL;
 
 	/* opportunistically-updated references: */
 	struct ref *orefs = NULL, **oref_tail = &orefs;