fetch: add --must-have and remote.*.mustHave
cover fetch: add --must-have and remote.*.mustHave
Fetch negotiation aims to find enough information from haves and wants such that the server can be reasonably confident that it will send all necessary objects and not too many "extra" objects that the client already has. However, this can break down if there are too many references, since Git truncates the list of haves based on a few factors (a 256 count limit or the server sending an ACK at the right time).
We already have the --negotiation-tip feature to focus the set of references that are used in negotiation, but I feel like this is designed backwards. I'd rather that we have a way to say "this is an important set of refs, but feel free to add more refs if needed" than "only use these refs for negotiation".
Here's an example that demonstrates the problem. In an internal monorepo, developers work off of the 'main' branch so there are thousands of user branches that each add a few commits different from the 'main' branch. However, there is also a long-lived 'release' branch. This branch has a first-parent history that is parallel to 'main' and each of those commits is a merge whose second parent is a commit from 'main' that had a successful CI run. There are additional changes in the 'release' branch merge commits that add some changelog data, so there is a nontrivial set of novel blob content in that branch and not just a different set of commits.
The problem we had was that our georeplication system was regularly fetching from the origin and trying to get all data from all reachable branches. When the 'release' branch updated, the client would run out of haves before advertising its copy of the 'release' branch, but it would still list the new 'release' tip as a want. The server would then think that the client had never fetched that branch before and would send all of the changelog data from the whole history of the repo. (This led to a lot of downstream problems; we mitigated by setting a refspec that stopped fetching the 'release' branch, but this is not ideal.)
What I'd like is a mechanism to say "always advertise the client's version of 'main' and 'release' but also opportunistically include some user branches".
Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.
There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.
This patch series adds a new '--must-have' argument in the same places that '--negotiation-tip' exists. This adds a set of references that will always be included in the negotiation as haves, but then the rest of the negotiation can proceed as normal. It also adds a new 'remote.<name>.mustHave' config option to enable this behavior without CLI arguments.
The series is organized into four changes:
1. The first change updates a test that started failing after I added some tests before it. This fix should be valuable on its own, but I need it for my tests to pass.
2. The second change adds the --must-have option for fetch and tests it
relative to --negotiation-tip.
3. The third change adds the config option to enable --must-have by default. Any use in the CLI completely overrides the config option.
4. The fourth change adds capabilities for the config to update negotiation during git push.
During development, I had briefly considered only using config values, but that required some strange changes to care about the remote name in the transport layer. This was most different in the 'git push' integration. When I discovered the '--negotiation-tip' feature during the process, that gave me a clear pattern to follow with the addition of a config on top.
I've CC'd some folks who have been working on or near the '--negotiation-tip' feature for feedback as to how these things fit together in the bigger picture.
Big picture questions to think about:
* Is this a valuable addition to the fetch negotiation?
* Is the interaction between --must-have and --negotiation-tip correct?
* Is the "must have" name sensical to users? I expect that this only matters to experts, but I'm open to better names that could be more self-documenting.
* Should we add a similar config key for --negotiation-tip?
Thanks, -Stolee
Derrick Stolee (4):
t5516: fix test order flakiness
fetch: add --must-have option for negotiation
remote: add mustHave config as default for --must-have
send-pack: pass --must-have for push negotiation
Documentation/config/remote.adoc | 23 ++++++ Documentation/fetch-options.adoc | 22 ++++++ builtin/fetch.c | 14 +++- fetch-pack.c | 92 +++++++++++++++++++++-- fetch-pack.h | 10 ++- remote.c | 6 ++ remote.h | 1 + send-pack.c | 12 ++- send-pack.h | 1 + t/t5510-fetch.sh | 121 +++++++++++++++++++++++++++++++ t/t5516-fetch-push.sh | 17 ++++- transport.c | 5 +- transport.h | 6 ++ 13 files changed, 319 insertions(+), 11 deletions(-)
base-commit: 6e8d538aab8fe4dd07ba9fb87b5c7edcfa5706ad
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2085%2Fderrickstolee%2Fmust-have-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2085/derrickstolee/must-have-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2085
Signature · 1 line
gitgitgadget
PATCH 1/4 t5516: fix test order flakiness
From: Derrick Stolee <[email protected]>
The 'fetch follows tags by default' test sorts using 'sort -k 4', but for-each-ref output only has 3 columns. This relies on sort treating records with fewer fields as having an empty fourth field, which may produce unstable results depending on locale. Use 'sort -k 3' to match the actual number of columns in the output.
--- t/t5516-fetch-push.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5516-fetch-push.sh +1 -1
@@ -1349,7 +1349,7 @@ test_expect_success 'fetch follows tags by default' ' git for-each-ref >tmp1 && sed -n "p; s|refs/heads/main$|refs/remotes/origin/main|p" tmp1 | sed -n "p; s|refs/heads/main$|refs/remotes/origin/HEAD|p" | - sort -k 4 >../expect + sort -k 3 >../expect ) && test_when_finished "rm -rf dst" && git init dst &&
Signature · 1 line
gitgitgadget
PATCH 2/4 fetch: add --must-have option for negotiation
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a --must-have option to git fetch that specifies ref patterns whose tips should always be sent as "have" commits during negotiation, regardless of what the negotiation algorithm selects.
Each value is either an exact ref name (e.g. refs/heads/release) or a glob pattern (e.g. refs/heads/release/*). The pattern syntax is the same as for --negotiation-tip.
This is useful when certain references are important for negotiation efficiency but might be skipped by the negotiation algorithm or excluded by --negotiation-tip. Unlike --negotiation-tip which restricts the have set, --must-have is additive: the negotiation algorithm still runs and advertises its own selected commits, but the refs matching --must-have are sent unconditionally on top of those.
If --negotiation-tip is used, the have set is first restricted by that option and then increased to include the tips specified by --must-have.
Due to the comparision with --negotiation-tip, a previously untranslated warning around --negotiation-tip is converted into a translatable string with a swap for which option that is relevant.
Getting this functionality to work requires moving these options through the transport API layer.
--- Documentation/fetch-options.adoc | 18 +++++++ builtin/fetch.c | 11 +++- fetch-pack.c | 92 +++++++++++++++++++++++++++++--- fetch-pack.h | 10 +++- t/t5510-fetch.sh | 73 +++++++++++++++++++++++++ transport.c | 4 +- transport.h | 6 +++ 7 files changed, 205 insertions(+), 9 deletions(-)
Documentation/fetch-options.adoc +18 -0
@@ -69,6 +69,24 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate` configuration variables documented in linkgit:git-config[1], and the `--negotiate-only` option below. +`--must-have=<revision>`:: + Ensure that the given ref tip is always sent as a "have" line + during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-tip` restricts the set of tips or when the + negotiation algorithm would otherwise skip them. ++ +This option may be specified more than once; if so, each ref is sent +unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax +is the same as for `--negotiation-tip`. ++ +If `--negotiation-tip` is used, the have set is first restricted by that +option and then increased to include the tips specified by `--must-have`. + `--negotiate-only`:: Do not fetch anything from the server, and instead print the ancestors of the provided `--negotiation-tip=` arguments,
builtin/fetch.c +10 -1
@@ -99,6 +99,7 @@ static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ;static struct string_list negotiation_tip =STRING_LIST_INIT_NODUP ; +static struct string_list must_have =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;
@@ -1599,7 +1600,13 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,if (transport->smart_options)add_negotiation_tips (transport->smart_options);else -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-tip" ); + } +if (must_have.nr) { +if (transport->smart_options) + transport->smart_options->must_have = &must_have; +else +warning (_ ("ignoring %s because the protocol does not support it" ),"--must-have" ); }return transport; }
@@ -2567,6 +2574,8 @@ int cmd_fetch(int argc,OPT_IPVERSION (&family),OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_STRING_LIST (0 ,"must-have" , &must_have,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" )),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
fetch-pack.c +86 -6
199 lines not on this page · raw message
fetch-pack.h +9 -1
@@ -23,6 +23,13 @@ struct fetch_pack_args {*/ const struct oid_array *negotiation_tips; +/* +* If non-empty, ref patterns whose tips should always be sent +* as "have" lines during negotiation, regardless of what the +* negotiation algorithm selects. +*/ +const struct string_list *must_have; + unsigned deepen_relative:1; unsigned quiet:1; unsigned keep_pack:1;
@@ -93,7 +100,8 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,const struct string_list *server_options, int stateless_rpc, int fd[], - struct oidset *acked_commits); + struct oidset *acked_commits, + const struct string_list *must_have);/* * Print an appropriate error message for each sought ref that wasn't
t/t5510-fetch.sh +73 -0
79 lines not on this page · raw message
transport.c +3 -1
@@ -464,6 +464,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; args.negotiation_tips = data->options.negotiation_tips; + args.must_have = data->options.must_have; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -495,7 +496,8 @@ static int fetch_refs_via_pack(struct transport *transport, transport->server_options, transport->stateless_rpc, data->fd, - data->options.acked_commits); + data->options.acked_commits, + data->options.must_have); ret =0 ; } goto cleanup;
transport.h +6 -0
@@ -48,6 +48,12 @@ struct git_transport_options {*/ struct oid_array *negotiation_tips; +/* +* If non-empty, ref patterns whose tips should always be sent +* as "have" lines during negotiation. +*/ +const struct string_list *must_have; +/* * If allocated, whenever transport_fetch_refs() is called, add known * common commits to this oidset instead of fetching any packfiles.
Signature · 1 line
gitgitgadget
PATCH 3/4 remote: add mustHave config as default for --must-have
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new multi-valued config option remote.<name>.mustHave that specifies ref patterns whose tips should always be sent as "have" commits during fetch negotiation with that remote.
Parse the option in handle_config() following the same pattern as remote.<name>.serverOption. Store the values in a string_list on struct remote so they are available per-remote.
In builtin/fetch.c, when no --must-have options are given on the command line, use the remote.<name>.mustHave config values as the default. If the user explicitly provides --must-have on the CLI, the config is not used, giving CLI precedence.
--- Documentation/config/remote.adoc | 23 +++++++++++++++ Documentation/fetch-options.adoc | 4 +++ builtin/fetch.c | 3 ++ remote.c | 6 ++++ remote.h | 1 + t/t5510-fetch.sh | 48 ++++++++++++++++++++++++++++++++ 6 files changed, 85 insertions(+)
Documentation/config/remote.adoc +23 -0
@@ -107,6 +107,29 @@ priority configuration file (e.g. `.git/config` in a repository) to clear the values inherited from a lower priority configuration files (e.g. `$HOME/.gitconfig`). +remote.<name>.mustHave:: + When negotiating with this remote during `git fetch` and `git push`, + the client advertises a list of commits that exist locally. In + repos with many references, this list of "haves" can be truncated. + Depending on data shape, dropping certain references may be + expensive. This multi-valued config option specifies ref patterns + whose tips should always be sent as "have" commits during fetch + negotiation with this remote. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the same +as for `--negotiation-tip`. ++ +These config values are used as defaults for the `--must-have` command-line +option. If `--must-have` is specified on the command line, then the config +values are not used. ++ +This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits, +but the refs matching `remote.<name>.mustHave` are sent unconditionally on +top of those heuristically selected commits. This option is also used +during push negotiation when `push.negotiate` is enabled. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
Documentation/fetch-options.adoc +4 -0
@@ -86,6 +86,10 @@ is the same as for `--negotiation-tip`. + If `--negotiation-tip` is used, the have set is first restricted by that option and then increased to include the tips specified by `--must-have`. ++ +If this option is not specified on the command line, then any +`remote.<name>.mustHave` config values for the current remote are used +instead. `--negotiate-only`:: Do not fetch anything from the server, and instead print the
builtin/fetch.c +3 -0
@@ -1607,6 +1607,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen, transport->smart_options->must_have = &must_have; else warning(_("ignoring %s because the protocol does not support it"), "--must-have"); + }else if (remote->must_have.nr) { +if (transport->smart_options) + transport->smart_options->must_have = &remote->must_have; }return transport; }
remote.c +6 -0
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_push (&ret->push);refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options); +string_list_init_dup (&ret->must_have);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -179,6 +180,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy);FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 ); +string_list_clear (&remote->must_have,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -562,6 +564,10 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"serveroption" )) {return parse_transport_option (key, value, &remote->server_options); + }else if (!strcmp (subkey,"musthave" )) { +if (!value) +return config_error_nonbool (key); +string_list_append (&remote->must_have, value); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -117,6 +117,7 @@ struct remote {char *http_proxy_authmethod;struct string_list server_options; +struct string_list must_have;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +48 -0
54 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH 4/4 send-pack: pass --must-have for push negotiation
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
When push.negotiate is enabled, send-pack spawns a 'git fetch --negotiate-only' subprocess to discover common commits. Previously this subprocess had no way to include must-have refs in the negotiation.
Add a must_have field to send_pack_args, set it from the transport layer where the remote struct is available, and pass explicit --must-have arguments to the negotiation subprocess. This approach directly passes the resolved config values rather than relying on the subprocess to read remote config, which is more robust when the URL alone is used as the remote identifier.
--- send-pack.c | 12 +++++++++++- send-pack.h | 1 + t/t5516-fetch-push.sh | 15 +++++++++++++++ transport.c | 1 + 4 files changed, 28 insertions(+), 1 deletion(-)
send-pack.c +11 -1
@@ -433,6 +433,7 @@ static void reject_invalid_nonce(const char *nonce, int len)static void get_commons_through_negotiation (struct repository *r,const char *url, +const struct string_list *must_have,const struct ref *remote_refs,struct oid_array *commons) {
@@ -452,6 +453,14 @@ static void get_commons_through_negotiation(struct repository *r, nr_negotiation_tip++; } } + +if (must_have) { +struct string_list_item *item; +for_each_string_list_item (item, must_have) +strvec_pushf (&child.args,"--must-have=%s" , + item->string); + } +strvec_push (&child.args, url);if (!nr_negotiation_tip) {
@@ -528,7 +537,8 @@ int send_pack(struct repository *r,repo_config_get_bool (r,"push.negotiate" , &push_negotiate);if (push_negotiate) {trace2_region_enter ("send_pack" ,"push_negotiate" , r); -get_commons_through_negotiation (r, args->url, remote_refs, &commons); +get_commons_through_negotiation (r, args->url, args->must_have, + remote_refs, &commons);trace2_region_leave ("send_pack" ,"push_negotiate" , r); }
send-pack.h +1 -0
@@ -18,6 +18,7 @@ struct repository;struct send_pack_args {const char *url; +const struct string_list *must_have;unsigned verbose:1 , quiet:1 , porcelain:1,
t/t5516-fetch-push.sh +15 -0
@@ -254,6 +254,21 @@ test_expect_success 'push with negotiation does not attempt to fetch submodules' ! grep "Fetching submodule" err ' +test_expect_success 'push with negotiation and remote.<name>.mustHave' ' + test_when_finished rm -rf musthave && + mk_empty musthave && + git push musthave $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C musthave unrelated_commit && + git -C musthave config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.musthave.mustHave=refs/heads/main \ + push musthave refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + test_expect_success 'push without wildcard' ' mk_empty testrepo &&
transport.c +1 -0
@@ -921,6 +921,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re args.atomic = !!(flags &TRANSPORT_PUSH_ATOMIC ); args.push_options = transport->push_options; args.url = transport->url; + args.must_have = &transport->remote->must_have;if (flags &TRANSPORT_PUSH_CERT_ALWAYS ) args.push_cert =SEND_PACK_PUSH_CERT_ALWAYS ;
Signature · 1 line
gitgitgadget
Quoted 13 lines · Derrick Stolee via GitGitGadget
Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.
There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.
...
Big picture questions to think about:
Quoted 6 lines
* Is this a valuable addition to the fetch negotiation?
* Is the interaction between --must-have and --negotiation-tip correct?
* Is the "must have" name sensical to users? I expect that this only matters to experts, but I'm open to better names that could be more self-documenting.
* Should we add a similar config key for --negotiation-tip?
Just like you, I hate the name "must have", but stepping back a bit, would it work if we add a single boolean option that says "use the negotiation tips as the primary source of 'have's you'd send, but unlike the way how the original negotiation-tip feature worked without this bit enabled, which did not send anything other than the ones reachable by negotiation tips, do advertise opportunistically other tips", essentially turning the existing negotiation-tips feature into your must-have feature? You could even call the option "--negotiate-better(=(yes|no))" or something, perhaps?
Quoted 22 lines · Junio C Hamano
Quoted 13 lines · Derrick Stolee via GitGitGadget
Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.
There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.
...
Big picture questions to think about:Quoted 6 lines
* Is this a valuable addition to the fetch negotiation?
* Is the interaction between --must-have and --negotiation-tip correct?
* Is the "must have" name sensical to users? I expect that this only matters to experts, but I'm open to better names that could be more self-documenting.
* Should we add a similar config key for --negotiation-tip?
Quoted 9 lines
Just like you, I hate the name "must have", but stepping back a bit, would it work if we add a single boolean option that says "use the negotiation tips as the primary source of 'have's you'd send, but unlike the way how the original negotiation-tip feature worked without this bit enabled, which did not send anything other than the ones reachable by negotiation tips, do advertise opportunistically other tips", essentially turning the existing negotiation-tips feature into your must-have feature? You could even call the option "--negotiate-better(=(yes|no))" or something, perhaps?
I like this line of thought. You essentially want to use the existing scaffolding of the --negotiate-tip option but change it from being a _maximum set_ to being a _minimum set_.
## Considering --negotiation-tip-mode=<mode>
With that in mind, we could have an option like --negotiation-tip-mode that takes one of a few options. Here are some word choices that I immediately thought about:
* maximum|minimum: Are these sets a maximum set to choose from or a minimum set to include?
* restrict|include: Are we restricting the haves to this set, or are we including these tips by default?
* v1|v2: Use numerical versions to indicate the mode without commentary so it could be extended in the future to v3 or more.
None of these jump out as a clear winner in my head. I'm interested in more exploration of this space before rerolling.
## To mix modes, or not to mix modes?
One downside of this approach is that it disables the ability to use both modes, at least in its most obvious implementation. What if someone wants to force a minimum set of wants but also wants to focus the set of additional wants to a specific ref space?
Theoretically, we could implement the option to toggle with multiple options, using
--negotiation-tip-mode=minimum --negotiation-tip=refs/remotes/origin/main \ --negotiation-tip-mode=maximum --negotiation-tip=refs/remotes/origin/*
and as we process the --negotiation-tip options we'd put the input data into different lists. Would this complexity be worth it compared to making a new set of options?
This also becomes more complicated how to describe the interaction of these options and any config options that enable them by default. When exactly does the config get ignored in favor of CLI options?
## Considering --negotiation-(required|restricted)
We could alternatively create two new types of options that are clearly related:
* --negotiation-restricted works exactly like --negotiation-tips and would be a synonym (with the old one being "deprecated" in favor of the newer one).
* --negotiation-required works like the --must-have in this series.
---
Thanks for considering these options with me. There is a lot of room for creativity here. This series isn't even my first attempt at this functionality because there are so many possible ways to accomplish this goal.
Thanks,
-Stolee
v2 cover fetch: rework negotiation tip options
In reply to Derrick Stolee via GitGitGadgetFetch negotiation aims to find enough information from haves and wants such that the server can be reasonably confident that it will send all necessary objects and not too many "extra" objects that the client already has. However, this can break down if there are too many references, since Git truncates the list of haves based on a few factors (a 256 count limit or the server sending an ACK at the right time).
We already have the --negotiation-tip feature to focus the set of references that are used in negotiation, but I feel like this is designed backwards. I'd rather that we have a way to say "this is an important set of refs, but feel free to add more refs if needed" than "only use these refs for negotiation".
Here's an example that demonstrates the problem. In an internal monorepo, developers work off of the 'main' branch so there are thousands of user branches that each add a few commits different from the 'main' branch. However, there is also a long-lived 'release' branch. This branch has a first-parent history that is parallel to 'main' and each of those commits is a merge whose second parent is a commit from 'main' that had a successful CI run. There are additional changes in the 'release' branch merge commits that add some changelog data, so there is a nontrivial set of novel blob content in that branch and not just a different set of commits.
The problem we had was that our georeplication system was regularly fetching from the origin and trying to get all data from all reachable branches. When the 'release' branch updated, the client would run out of haves before advertising its copy of the 'release' branch, but it would still list the new 'release' tip as a want. The server would then think that the client had never fetched that branch before and would send all of the changelog data from the whole history of the repo. (This led to a lot of downstream problems; we mitigated by setting a refspec that stopped fetching the 'release' branch, but this is not ideal.)
What I'd like is a mechanism to say "always advertise the client's version of 'main' and 'release' but also opportunistically include some user branches".
Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.
There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.
This patch series adds a new '--negotiation-require' option that does what I
want: it makes sure that these references are used for 'have's during negotiation. In order to help clarify the difference between this and '--negotiation-tip', I first create a synonym called '--negotiation-restrict'.
Both of these options get 'remote.*.negotiation(Require|Restrict)' config options that enable their behavior by default.
During development, I had briefly considered only using config values, but that required some strange changes to care about the remote name in the transport layer. This was most different in the 'git push' integration. When I discovered the '--negotiation-tip' feature during the process, that gave me a clear pattern to follow with the addition of a config on top.
Updates in v2
=============
This version is a near-complete rewrite based on feedback around the names of the previous option and config. The --negotiation-restrict option is new and the ability to set it via config is also new.
I did try to be more careful around translatable error messages, too.
Thanks, -Stolee
Derrick Stolee (7):
t5516: fix test order flakiness
fetch: add --negotiation-restrict option
transport: rename negotiation_tips
remote: add remote.*.negotiationRestrict config
fetch: add --negotiation-require option for negotiation
remote: add negotiationRequire config as default for
--negotiation-require
send-pack: pass negotiation config in push
Documentation/config/remote.adoc | 40 ++++++++ Documentation/fetch-options.adoc | 27 ++++++ builtin/fetch.c | 59 ++++++++++-- builtin/pull.c | 6 ++ fetch-pack.c | 114 +++++++++++++++++++--- fetch-pack.h | 14 ++- remote.c | 12 +++ remote.h | 2 + send-pack.c | 34 +++++-- send-pack.h | 2 + t/t5510-fetch.sh | 159 +++++++++++++++++++++++++++++++ t/t5516-fetch-push.sh | 32 ++++++- t/t5702-protocol-v2.sh | 4 +- transport-helper.c | 2 +- transport.c | 16 ++-- transport.h | 10 +- 16 files changed, 489 insertions(+), 44 deletions(-)
base-commit: 6e8d538aab8fe4dd07ba9fb87b5c7edcfa5706ad
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2085%2Fderrickstolee%2Fmust-have-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2085/derrickstolee/must-have-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2085
Range-diff vs v1:
1: 540a5682b4 = 1: 466c56abe0 t5516: fix test order flakiness
-: ---------- > 2: 9a25b0fade fetch: add --negotiation-restrict option
-: ---------- > 3: 0f89665aee transport: rename negotiation_tips
-: ---------- > 4: a731f4fc87 remote: add remote.*.negotiationRestrict config
2: 8b39eb9e6c ! 5: 49c80cef2e fetch: add --must-have option for negotiation
@@ Metadata
Author: Derrick Stolee <[email protected]>
## Commit message ##
- fetch: add --must-have option for negotiation
+ fetch: add --negotiation-require option for negotiation
- Add a --must-have option to git fetch that specifies ref patterns whose
- tips should always be sent as "have" commits during negotiation,
- regardless of what the negotiation algorithm selects.
+ Add a new --negotiation-require option to 'git fetch', which ensures
+ that certain ref tips are always sent as 'have' lines during fetch
+ negotiation, regardless of what the negotiation algorithm selects.
- Each value is either an exact ref name (e.g. refs/heads/release) or a
- glob pattern (e.g. refs/heads/release/*). The pattern syntax is the same
- as for --negotiation-tip.
+ This is useful when the repository has a large number of references, so
+ the normal negotiation algorithm truncates the list. This is especially
+ important in repositories with long parallel commit histories. For
+ example, a repo could have a 'dev' branch for development and a
+ 'release' branch for released versions. If the 'dev' branch isn't
+ selected for negotiation, then it's not a big deal because there are
+ many in-progress development branches with a shared history. However, if
+ 'release' is not selected for negotiation, then the server may think
+ that this is the first time the client has asked for that reference,
+ causing a full download of its parallel commit history (and any extra
+ data that may be unique to that branch). This is based on a real example
+ where certain fetches would grow to 60+ GB when a release branch
+ updated.
- This is useful when certain references are important for negotiation
- efficiency but might be skipped by the negotiation algorithm or excluded
- by --negotiation-tip. Unlike --negotiation-tip which restricts the have
- set, --must-have is additive: the negotiation algorithm still runs and
- advertises its own selected commits, but the refs matching --must-have
- are sent unconditionally on top of those.
+ This option is a complement to --negotiation-restrict, which reduces the
+ negotiation ref set to a specific list. In the earlier example, using
+ --negotiation-restrict to focus the negotiation to 'dev' and 'release'
+ would avoid those problematic downloads, but would still not allow
+ advertising potentially-relevant user brances. In this way, the
+ 'require' version solves the problem I mention while allowing
+ negotiation to pick other references opportunistically. The two options
+ can also be combined to allow the best of both worlds.
- If --negotiation-tip is used, the have set is first restricted by that
- option and then increased to include the tips specified by --must-have.
+ The argument may be an exact ref name or a glob pattern. Non-existent
+ refs are silently ignored.
- Due to the comparision with --negotiation-tip, a previously untranslated
- warning around --negotiation-tip is converted into a translatable string
- with a swap for which option that is relevant.
-
- Getting this functionality to work requires moving these options through
- the transport API layer.
+ Also add --negotiation-require to 'git pull' passthrough options.
Signed-off-by: Derrick Stolee <[email protected]>
@@ Documentation/fetch-options.adoc: See also the `fetch.negotiationAlgorithm` and
configuration variables documented in linkgit:git-config[1], and the
`--negotiate-only` option below.
-+`--must-have=<revision>`::
++`--negotiation-require=<revision>`::
+ Ensure that the given ref tip is always sent as a "have" line
+ during fetch negotiation, regardless of what the negotiation
+ algorithm selects. This is useful to guarantee that common
+ history reachable from specific refs is always considered, even
-+ when `--negotiation-tip` restricts the set of tips or when the
-+ negotiation algorithm would otherwise skip them.
++ when `--negotiation-restrict` restricts the set of tips or when
++ the negotiation algorithm would otherwise skip them.
++
+This option may be specified more than once; if so, each ref is sent
+unconditionally.
++
+The argument may be an exact ref name (e.g. `refs/heads/release`) or a
+glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax
-+is the same as for `--negotiation-tip`.
++is the same as for `--negotiation-restrict`.
++
-+If `--negotiation-tip` is used, the have set is first restricted by that
-+option and then increased to include the tips specified by `--must-have`.
++If `--negotiation-restrict` is used, the have set is first restricted by
++that option and then increased to include the tips specified by
++`--negotiation-require`.
+
`--negotiate-only`::
Do not fetch anything from the server, and instead print the
@@ builtin/fetch.c: static struct transport *gsecondary;
static struct refspec refmap = REFSPEC_INIT_FETCH;
static struct string_list server_options = STRING_LIST_INIT_DUP;
static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-+static struct string_list must_have = STRING_LIST_INIT_NODUP;
++static struct string_list negotiation_require = STRING_LIST_INIT_NODUP;
struct fetch_config {
enum display_format display_format;
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remote, int deepen,
- if (transport->smart_options)
- add_negotiation_tips(transport->smart_options);
- else
-- warning("ignoring --negotiation-tip because the protocol does not support it");
-+ warning(_("ignoring %s because the protocol does not support it"), "--negotiation-tip");
-+ }
-+ if (must_have.nr) {
+ strbuf_release(&config_name);
+ }
+ }
++ if (negotiation_require.nr) {
+ if (transport->smart_options)
-+ transport->smart_options->must_have = &must_have;
++ transport->smart_options->negotiation_require = &negotiation_require;
+ else
-+ warning(_("ignoring %s because the protocol does not support it"), "--must-have");
- }
++ warning(_("ignoring %s because the protocol does not support it"),
++ "--negotiation-require");
++ }
return transport;
}
+
@@ builtin/fetch.c: int cmd_fetch(int argc,
- OPT_IPVERSION(&family),
- OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
N_("report that we have only objects reachable from this object")),
-+ OPT_STRING_LIST(0, "must-have", &must_have, N_("revision"),
+ OPT_STRING_LIST(0, "negotiation-restrict", &negotiation_tip, N_("revision"),
+ N_("report that we have only objects reachable from this object")),
++ OPT_STRING_LIST(0, "negotiation-require", &negotiation_require, N_("revision"),
+ N_("ensure this ref is always sent as a negotiation have")),
OPT_BOOL(0, "negotiate-only", &negotiate_only,
N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+ ## builtin/pull.c ##
+@@ builtin/pull.c: int cmd_pull(int argc,
+ OPT_PASSTHRU_ARGV(0, "negotiation-restrict", &opt_fetch, N_("revision"),
+ N_("report that we have only objects reachable from this object"),
+ 0),
++ OPT_PASSTHRU_ARGV(0, "negotiation-require", &opt_fetch, N_("revision"),
++ N_("ensure this ref is always sent as a negotiation have"),
++ 0),
+ OPT_BOOL(0, "show-forced-updates", &opt_show_forced_updates,
+ N_("check for forced-updates on all updated branches")),
+ OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
+
## fetch-pack.c ##
@@
#include "oidset.h"
@@ fetch-pack.c: static void send_filter(struct fetch_pack_args *args,
+static int add_oid_to_oidset(const struct reference *ref, void *cb_data)
+{
+ struct oidset *set = cb_data;
-+ oidset_insert(set, ref->oid);
++ if (odb_has_object(the_repository->objects, ref->oid, 0))
++ oidset_insert(set, ref->oid);
+ return 0;
+}
+
-+static void resolve_must_have(const struct string_list *must_have,
-+ struct oidset *result)
++static void resolve_negotiation_require(const struct string_list *negotiation_require,
++ struct oidset *result)
+{
+ struct string_list_item *item;
+
-+ if (!must_have || !must_have->nr)
++ if (!negotiation_require || !negotiation_require->nr)
+ return;
+
-+ for_each_string_list_item(item, must_have) {
++ for_each_string_list_item(item, negotiation_require) {
+ if (!has_glob_specials(item->string)) {
+ struct object_id oid;
+ if (repo_get_oid(the_repository, item->string, &oid))
@@ fetch-pack.c: static int find_common(struct fetch_negotiator *negotiator,
struct strbuf req_buf = STRBUF_INIT;
size_t state_len = 0;
struct packet_reader reader;
-+ struct oidset must_have_oids = OIDSET_INIT;
++ struct oidset negotiation_require_oids = OIDSET_INIT;
if (args->stateless_rpc && multi_ack == 1)
die(_("the option '%s' requires '%s'"), "--stateless-rpc", "multi_ack_detailed");
@@ fetch-pack.c: static int find_common(struct fetch_negotiator *negotiator,
flushes = 0;
retval = -1;
+
-+ /* Send unconditional haves from --must-have */
-+ resolve_must_have(args->must_have, &must_have_oids);
-+ if (oidset_size(&must_have_oids)) {
++ /* Send unconditional haves from --negotiation-require */
++ resolve_negotiation_require(args->negotiation_require,
++ &negotiation_require_oids);
++ if (oidset_size(&negotiation_require_oids)) {
+ struct oidset_iter iter;
-+ oidset_iter_init(&must_have_oids, &iter);
++ oidset_iter_init(&negotiation_require_oids, &iter);
+
+ while ((oid = oidset_iter_next(&iter))) {
+ packet_buf_write(&req_buf, "have %s\n",
@@ fetch-pack.c: static int find_common(struct fetch_negotiator *negotiator,
+ }
+
while ((oid = negotiator->next(negotiator))) {
-+ /* avoid duplicate oids from --must-have */
-+ if (oidset_contains(&must_have_oids, oid))
++ /* avoid duplicate oids from --negotiation-require */
++ if (oidset_contains(&negotiation_require_oids, oid))
+ continue;
packet_buf_write(&req_buf, "have %s\n", oid_to_hex(oid));
print_verbose(args, "have %s", oid_to_hex(oid));
@@ fetch-pack.c: done:
flushes++;
}
strbuf_release(&req_buf);
-+ oidset_clear(&must_have_oids);
++ oidset_clear(&negotiation_require_oids);
if (!got_ready || !no_done)
consume_shallow_list(args, &reader);
@@ fetch-pack.c: static void add_common(struct strbuf *req_buf, struct oidset *comm
struct strbuf *req_buf,
- int *haves_to_send)
+ int *haves_to_send,
-+ struct oidset *must_have_oids)
++ struct oidset *negotiation_require_oids)
{
int haves_added = 0;
const struct object_id *oid;
-+ /* Send unconditional haves from --must-have */
-+ if (must_have_oids) {
++ /* Send unconditional haves from --negotiation-require */
++ if (negotiation_require_oids) {
+ struct oidset_iter iter;
-+ oidset_iter_init(must_have_oids, &iter);
++ oidset_iter_init(negotiation_require_oids, &iter);
+
+ while ((oid = oidset_iter_next(&iter)))
+ packet_buf_write(req_buf, "have %s\n",
@@ fetch-pack.c: static void add_common(struct strbuf *req_buf, struct oidset *comm
+ }
+
while ((oid = negotiator->next(negotiator))) {
-+ if (must_have_oids && oidset_contains(must_have_oids, oid))
++ if (negotiation_require_oids &&
++ oidset_contains(negotiation_require_oids, oid))
+ continue;
packet_buf_write(req_buf, "have %s\n", oid_to_hex(oid));
if (++haves_added >= *haves_to_send)
@@ fetch-pack.c: static int send_fetch_request(struct fetch_negotiator *negotiator,
int *haves_to_send, int *in_vain,
- int sideband_all, int seen_ack)
+ int sideband_all, int seen_ack,
-+ struct oidset *must_have_oids)
++ struct oidset *negotiation_require_oids)
{
int haves_added;
int done_sent = 0;
@@ fetch-pack.c: static int send_fetch_request(struct fetch_negotiator *negotiator,
- haves_added = add_haves(negotiator, &req_buf, haves_to_send);
+ haves_added = add_haves(negotiator, &req_buf, haves_to_send,
-+ must_have_oids);
++ negotiation_require_oids);
*in_vain += haves_added;
trace2_data_intmax("negotiation_v2", the_repository, "haves_added", haves_added);
trace2_data_intmax("negotiation_v2", the_repository, "in_vain", *in_vain);
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
struct ref *ref = copy_ref_list(orig_ref);
enum fetch_state state = FETCH_CHECK_LOCAL;
struct oidset common = OIDSET_INIT;
-+ struct oidset must_have_oids = OIDSET_INIT;
++ struct oidset negotiation_require_oids = OIDSET_INIT;
struct packet_reader reader;
int in_vain = 0, negotiation_started = 0;
int negotiation_round = 0;
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
- reader.me = "fetch-pack";
- }
+ state = FETCH_SEND_REQUEST;
-+ resolve_must_have(args->must_have, &must_have_oids);
-+
- while (state != FETCH_DONE) {
- switch (state) {
- case FETCH_CHECK_LOCAL:
+ mark_tips(negotiator, args->negotiation_restrict_tips);
++ resolve_negotiation_require(args->negotiation_require,
++ &negotiation_require_oids);
+ for_each_cached_alternate(negotiator,
+ insert_one_alternate_object);
+ break;
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
&common,
&haves_to_send, &in_vain,
reader.use_sideband,
- seen_ack)) {
+ seen_ack,
-+ &must_have_oids)) {
++ &negotiation_require_oids)) {
trace2_region_leave_printf("negotiation_v2", "round",
the_repository, "%d",
negotiation_round);
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
negotiator->release(negotiator);
oidset_clear(&common);
-+ oidset_clear(&must_have_oids);
++ oidset_clear(&negotiation_require_oids);
return ref;
}
-@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_tips,
+@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,
const struct string_list *server_options,
int stateless_rpc,
int fd[],
- struct oidset *acked_commits)
+ struct oidset *acked_commits,
-+ const struct string_list *must_have)
++ const struct string_list *negotiation_require)
{
struct fetch_negotiator negotiator;
struct packet_reader reader;
struct object_array nt_object_array = OBJECT_ARRAY_INIT;
struct strbuf req_buf = STRBUF_INIT;
-+ struct oidset must_have_oids = OIDSET_INIT;
++ struct oidset negotiation_require_oids = OIDSET_INIT;
int haves_to_send = INITIAL_FLUSH;
int in_vain = 0;
int seen_ack = 0;
-@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_tips,
- add_to_object_array,
- &nt_object_array);
+@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,
+ fetch_negotiator_init(the_repository, &negotiator);
+ mark_tips(&negotiator, negotiation_restrict_tips);
-+ resolve_must_have(must_have, &must_have_oids);
++ resolve_negotiation_require(negotiation_require,
++ &negotiation_require_oids);
+
- trace2_region_enter("fetch-pack", "negotiate_using_fetch", the_repository);
- while (!last_iteration) {
- int haves_added;
-@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_tips,
+ packet_reader_init(&reader, fd[0], NULL, 0,
+ PACKET_READ_CHOMP_NEWLINE |
+ PACKET_READ_DIE_ON_ERR_PACKET);
+@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,
packet_buf_write(&req_buf, "wait-for-done");
- haves_added = add_haves(&negotiator, &req_buf, &haves_to_send);
+ haves_added = add_haves(&negotiator, &req_buf, &haves_to_send,
-+ &must_have_oids);
++ &negotiation_require_oids);
in_vain += haves_added;
if (!haves_added || (seen_ack && in_vain >= MAX_IN_VAIN))
last_iteration = 1;
-@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_tips,
+@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,
clear_common_flag(acked_commits);
object_array_clear(&nt_object_array);
-+ oidset_clear(&must_have_oids);
++ oidset_clear(&negotiation_require_oids);
negotiator.release(&negotiator);
strbuf_release(&req_buf);
}
@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_tip
## fetch-pack.h ##
@@ fetch-pack.h: struct fetch_pack_args {
*/
- const struct oid_array *negotiation_tips;
+ const struct oid_array *negotiation_restrict_tips;
+ /*
+ * If non-empty, ref patterns whose tips should always be sent
+ * as "have" lines during negotiation, regardless of what the
+ * negotiation algorithm selects.
+ */
-+ const struct string_list *must_have;
++ const struct string_list *negotiation_require;
+
unsigned deepen_relative:1;
unsigned quiet:1;
unsigned keep_pack:1;
-@@ fetch-pack.h: void negotiate_using_fetch(const struct oid_array *negotiation_tips,
+@@ fetch-pack.h: void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,
const struct string_list *server_options,
int stateless_rpc,
int fd[],
- struct oidset *acked_commits);
+ struct oidset *acked_commits,
-+ const struct string_list *must_have);
++ const struct string_list *negotiation_require);
/*
* Print an appropriate error message for each sought ref that wasn't
## t/t5510-fetch.sh ##
-@@ t/t5510-fetch.sh: test_expect_success REFFILES "HEAD is updated even with conflicts" '
- )
+@@ t/t5510-fetch.sh: test_expect_success 'CLI --negotiation-restrict overrides remote config' '
+ test_grep ! "fetch> have $BETA_1" trace
'
-+test_expect_success '--must-have includes configured refs as haves' '
++test_expect_success '--negotiation-require includes configured refs as haves' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ # With --negotiation-tip restricting tips, only alpha_1 is
-+ # normally sent. --must-have should also include beta_1.
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --negotiation-tip=alpha_1 \
-+ --must-have=refs/tags/beta_1 \
++ --negotiation-restrict=alpha_1 \
++ --negotiation-require=refs/tags/beta_1 \
+ origin alpha_s beta_s &&
+
+ ALPHA_1=$(git -C client rev-parse alpha_1) &&
@@ t/t5510-fetch.sh: test_expect_success REFFILES "HEAD is updated even with confli
+ test_grep "fetch> have $BETA_1" trace
+'
+
-+test_expect_success '--must-have works with glob patterns' '
++test_expect_success '--negotiation-require works with glob patterns' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --negotiation-tip=alpha_1 \
-+ --must-have="refs/tags/beta_*" \
++ --negotiation-restrict=alpha_1 \
++ --negotiation-require="refs/tags/beta_*" \
+ origin alpha_s beta_s &&
+
+ BETA_1=$(git -C client rev-parse beta_1) &&
@@ t/t5510-fetch.sh: test_expect_success REFFILES "HEAD is updated even with confli
+ test_grep "fetch> have $BETA_2" trace
+'
+
-+test_expect_success '--must-have is additive with negotiation' '
++test_expect_success '--negotiation-require is additive with negotiation' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ # Without --negotiation-tip, all local refs are used as tips.
-+ # --must-have should add its refs unconditionally on top.
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --must-have=refs/tags/beta_1 \
++ --negotiation-require=refs/tags/beta_1 \
+ origin alpha_s beta_s &&
+
+ BETA_1=$(git -C client rev-parse beta_1) &&
+ test_grep "fetch> have $BETA_1" trace
+'
+
-+test_expect_success '--must-have ignores non-existent refs silently' '
++test_expect_success '--negotiation-require ignores non-existent refs silently' '
+ setup_negotiation_tip server server 0 &&
+
+ git -C client fetch --quiet \
-+ --negotiation-tip=alpha_1 \
-+ --must-have=refs/tags/nonexistent \
++ --negotiation-restrict=alpha_1 \
++ --negotiation-require=refs/tags/nonexistent \
+ origin alpha_s beta_s 2>err &&
+ test_must_be_empty err
+'
+
-+test_expect_success '--must-have avoids duplicates with negotiator' '
++test_expect_success '--negotiation-require avoids duplicates with negotiator' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ # Configure a ref that will also be a negotiation tip.
-+ # fetch should still complete successfully.
+ ALPHA_1=$(git -C client rev-parse alpha_1) &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --negotiation-tip=alpha_1 \
-+ --must-have=refs/tags/alpha_1 \
++ --negotiation-restrict=alpha_1 \
++ --negotiation-require=refs/tags/alpha_1 \
+ origin alpha_s beta_s &&
+
-+ # alpha_1 should appear as a have
+ test_grep "fetch> have $ALPHA_1" trace >matches &&
+ test_line_count = 1 matches
+'
+
- . "$TEST_DIRECTORY"/lib-httpd.sh
- start_httpd
-
+ test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' '
+ git init df-conflict &&
+ (
## transport.c ##
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
args.stateless_rpc = transport->stateless_rpc;
args.server_options = transport->server_options;
- args.negotiation_tips = data->options.negotiation_tips;
-+ args.must_have = data->options.must_have;
+ args.negotiation_restrict_tips = data->options.negotiation_restrict_tips;
++ args.negotiation_require = data->options.negotiation_require;
args.reject_shallow_remote = transport->smart_options->reject_shallow;
if (!data->finished_handshake) {
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
data->fd,
- data->options.acked_commits);
+ data->options.acked_commits,
-+ data->options.must_have);
++ data->options.negotiation_require);
ret = 0;
}
goto cleanup;
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
## transport.h ##
@@ transport.h: struct git_transport_options {
*/
- struct oid_array *negotiation_tips;
+ struct oid_array *negotiation_restrict_tips;
+ /*
+ * If non-empty, ref patterns whose tips should always be sent
+ * as "have" lines during negotiation.
+ */
-+ const struct string_list *must_have;
++ const struct string_list *negotiation_require;
+
/*
* If allocated, whenever transport_fetch_refs() is called, add known 3: fbc98b0cbb ! 6: 081f904c07 remote: add mustHave config as default for --must-have
@@ Metadata
Author: Derrick Stolee <[email protected]>
## Commit message ##
- remote: add mustHave config as default for --must-have
+ remote: add negotiationRequire config as default for --negotiation-require
- Add a new multi-valued config option remote.<name>.mustHave that
- specifies ref patterns whose tips should always be sent as "have"
- commits during fetch negotiation with that remote.
+ Add a new 'remote.<name>.negotiationRequire' multi-valued config option
+ that provides default values for --negotiation-require when no
+ --negotiation-require arguments are specified over the command line.
+ This is a mirror of how 'remote.<name>.negotiationRestrict' specifies
+ defaults for the --negotiation-restrict arguments.
- Parse the option in handle_config() following the same pattern as
- remote.<name>.serverOption. Store the values in a string_list on struct
- remote so they are available per-remote.
+ Each value is either an exact ref name or a glob pattern whose tips
+ should always be sent as 'have' lines during negotiation. The config
+ values are resolved through the same resolve_negotiation_require()
+ codepath as the CLI options.
- In builtin/fetch.c, when no --must-have options are given on the command
- line, use the remote.<name>.mustHave config values as the default. If
- the user explicitly provides --must-have on the CLI, the config is not
- used, giving CLI precedence.
+ This option is additive with the normal negotiation process: the
+ negotiation algorithm still runs and advertises its own selected
+ commits, but the refs matching the config are sent unconditionally
+ on top of those heuristically selected commits.
## Documentation/config/remote.adoc ##
-@@ Documentation/config/remote.adoc: priority configuration file (e.g. .git/config in a repository) to clear
- the values inherited from a lower priority configuration files (e.g.
- $HOME/.gitconfig).
+@@ Documentation/config/remote.adoc: command-line option. If --negotiation-restrict (or its synonym
+ --negotiation-tip) is specified on the command line, then the config
+ values are not used.
-+remote.<name>.mustHave::
++remote.<name>.negotiationRequire::
+ When negotiating with this remote during git fetch and git push,
+ the client advertises a list of commits that exist locally. In
+ repos with many references, this list of "haves" can be truncated.
@@ Documentation/config/remote.adoc: priority configuration file (e.g. `.git/config
++
+Each value is either an exact ref name (e.g. refs/heads/release) or a +glob pattern (e.g. refs/heads/release/*). The pattern syntax is the same
-+as for --negotiation-tip.
++as for --negotiation-restrict.
++
-+These config values are used as defaults for the --must-have command-line -+option. If --must-have is specified on the command line, then the config -+values are not used.
++These config values are used as defaults for the --negotiation-require ++command-line option. If --negotiation-require is specified on the ++command line, then the config values are not used.
++
+This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits,
-+but the refs matching remote.<name>.mustHave are sent unconditionally on -+top of those heuristically selected commits. This option is also used -+during push negotiation when push.negotiate is enabled.
++but the refs matching remote.<name>.negotiationRequire are sent ++unconditionally on top of those heuristically selected commits. This ++option is also used during push negotiation when push.negotiate is ++enabled.
+
remote.<name>.followRemoteHEAD::
How linkgit:git-fetch[1] should handle updates to remotes/<name>/HEAD when fetching using the configured refspecs of a remote.
## Documentation/fetch-options.adoc ##
-@@ Documentation/fetch-options.adoc: is the same as for --negotiation-tip.
- +
- If --negotiation-tip is used, the have set is first restricted by that
- option and then increased to include the tips specified by --must-have. +@@ Documentation/fetch-options.adoc: is the same as for --negotiation-restrict.
+ If --negotiation-restrict is used, the have set is first restricted by
+ that option and then increased to include the tips specified by
+ --negotiation-require.
++
+If this option is not specified on the command line, then any
-+remote.<name>.mustHave config values for the current remote are used -+instead.
++remote.<name>.negotiationRequire config values for the current remote ++are used instead.
--negotiate-only::
Do not fetch anything from the server, and instead print the
## builtin/fetch.c ##
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remote, int deepen,
- transport->smart_options->must_have = &must_have;
else
- warning(_("ignoring %s because the protocol does not support it"), "--must-have");
-+ } else if (remote->must_have.nr) {
-+ if (transport->smart_options)
-+ transport->smart_options->must_have = &remote->must_have;
+ warning(_("ignoring %s because the protocol does not support it"),
+ "--negotiation-require");
++ } else if (remote->negotiation_require.nr) {
++ if (transport->smart_options) {
++ transport->smart_options->negotiation_require = &remote->negotiation_require;
++ } else {
++ struct strbuf config_name = STRBUF_INIT;
++ strbuf_addf(&config_name, "remote.%s.negotiationRequire", remote->name);
++ warning(_("ignoring %s because the protocol does not support it"),
++ config_name.buf);
++ strbuf_release(&config_name);
++ }
}
return transport;
}
## remote.c ##
@@ remote.c: static struct remote *make_remote(struct remote_state *remote_state,
- refspec_init_push(&ret->push);
refspec_init_fetch(&ret->fetch); string_list_init_dup(&ret->server_options);
-+ string_list_init_dup(&ret->must_have);
+ string_list_init_dup(&ret->negotiation_restrict);
++ string_list_init_dup(&ret->negotiation_require);
ALLOC_GROW(remote_state->remotes, remote_state->remotes_nr + 1,
remote_state->remotes_alloc);
@@ remote.c: static void remote_clear(struct remote *remote)
- FREE_AND_NULL(remote->http_proxy);
FREE_AND_NULL(remote->http_proxy_authmethod); string_list_clear(&remote->server_options, 0);
-+ string_list_clear(&remote->must_have, 0);
+ string_list_clear(&remote->negotiation_restrict, 0);
++ string_list_clear(&remote->negotiation_require, 0);
}
static void add_merge(struct branch *branch, const char *name)
@@ remote.c: static int handle_config(const char *key, const char *value,
- } else if (!strcmp(subkey, "serveroption")) {
- return parse_transport_option(key, value,
- &remote->server_options);
-+ } else if (!strcmp(subkey, "musthave")) {
+ if (!value)
+ return config_error_nonbool(key);
+ string_list_append(&remote->negotiation_restrict, value);
++ } else if (!strcmp(subkey, "negotiationrequire")) {
+ if (!value)
+ return config_error_nonbool(key);
-+ string_list_append(&remote->must_have, value);
++ string_list_append(&remote->negotiation_require, value);
} else if (!strcmp(subkey, "followremotehead")) {
const char *no_warn_branch;
if (!strcmp(value, "never"))
## remote.h ##
@@ remote.h: struct remote {
- char *http_proxy_authmethod;
struct string_list server_options;
-+ struct string_list must_have;
+ struct string_list negotiation_restrict;
++ struct string_list negotiation_require;
enum follow_remote_head_settings follow_remote_head; const char *no_warn_branch;
## t/t5510-fetch.sh ##
-@@ t/t5510-fetch.sh: test_expect_success '--must-have avoids duplicates with negotiator' ' +@@ t/t5510-fetch.sh: test_expect_success '--negotiation-require avoids duplicates with negotiator' '
test_line_count = 1 matches
'
-+test_expect_success 'remote.<name>.mustHave used as default for --must-have' ' ++test_expect_success 'remote.<name>.negotiationRequire used as default for --negotiation-require' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ # No --must-have on CLI; config should be used as default.
-+ git -C client config --add remote.origin.mustHave refs/tags/beta_1 &&
++ git -C client config --add remote.origin.negotiationRequire refs/tags/beta_1 &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --negotiation-tip=alpha_1 \
++ --negotiation-restrict=alpha_1 \
+ origin alpha_s beta_s &&
+
+ ALPHA_1=$(git -C client rev-parse alpha_1) &&
@@ t/t5510-fetch.sh: test_expect_success '--must-have avoids duplicates with negoti
+ test_grep "fetch> have $BETA_1" trace
+'
+
-+test_expect_success 'remote.<name>.mustHave works with glob patterns' ' ++test_expect_success 'remote.<name>.negotiationRequire works with glob patterns' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ git -C client config --add remote.origin.mustHave "refs/tags/beta_*" &&
++ git -C client config --add remote.origin.negotiationRequire "refs/tags/beta_*" &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --negotiation-tip=alpha_1 \
++ --negotiation-restrict=alpha_1 \
+ origin alpha_s beta_s &&
+
+ BETA_1=$(git -C client rev-parse beta_1) &&
@@ t/t5510-fetch.sh: test_expect_success '--must-have avoids duplicates with negoti
+ test_grep "fetch> have $BETA_2" trace
+'
+
-+test_expect_success 'CLI --must-have overrides remote.<name>.mustHave' ' ++test_expect_success 'CLI --negotiation-require overrides remote.<name>.negotiationRequire' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ # Config says beta_2, CLI says beta_1; only CLI should be used.
-+ git -C client config --add remote.origin.mustHave refs/tags/beta_2 &&
++ git -C client config --add remote.origin.negotiationRequire refs/tags/beta_2 &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --negotiation-tip=alpha_1 \
-+ --must-have=refs/tags/beta_1 \
++ --negotiation-restrict=alpha_1 \
++ --negotiation-require=refs/tags/beta_1 \
+ origin alpha_s beta_s &&
+
+ BETA_1=$(git -C client rev-parse beta_1) &&
@@ t/t5510-fetch.sh: test_expect_success '--must-have avoids duplicates with negoti
+ test_grep ! "fetch> have $BETA_2" trace
+'
+
- . "$TEST_DIRECTORY"/lib-httpd.sh
- start_httpd
-
+ test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' '
+ git init df-conflict &&
+ (
4: 6c227f18ab ! 7: 7cccf59beb send-pack: pass --must-have for push negotiation
@@ Metadata
Author: Derrick Stolee <[email protected]>
## Commit message ##
- send-pack: pass --must-have for push negotiation
+ send-pack: pass negotiation config in push
- When push.negotiate is enabled, send-pack spawns a 'git fetch
- --negotiate-only' subprocess to discover common commits. Previously
- this subprocess had no way to include must-have refs in the
- negotiation.
+ When push.negotiate is enabled, 'git push' spawns a child 'git fetch
+ --negotiate-only' process to find common commits. Pass
+ --negotiation-require and --negotiation-restrict options from the
+ 'remote.<name>.negotiationRequire' and
+ 'remote.<name>.negotiationRestrict' config keys to this child process.
- Add a must_have field to send_pack_args, set it from the transport
- layer where the remote struct is available, and pass explicit
- --must-have arguments to the negotiation subprocess. This approach
- directly passes the resolved config values rather than relying on the
- subprocess to read remote config, which is more robust when the URL
- alone is used as the remote identifier.
+ When negotiationRestrict is configured, it replaces the default
+ behavior of using all remote refs as negotiation tips. This allows
+ the user to control which local refs are used for push negotiation.
+
+ When negotiationRequire is configured, the specified ref patterns
+ are passed as --negotiation-require to ensure their tips are always
+ sent as 'have' lines during push negotiation.
+
+ This change also updates the use of --negotiation-tip into
+ --negotiation-restrict now that the new synonym exists.
Signed-off-by: Derrick Stolee <[email protected]>
@@ send-pack.c: static void reject_invalid_nonce(const char *nonce, int len)
static void get_commons_through_negotiation(struct repository *r,
const char *url,
-+ const struct string_list *must_have,
++ const struct string_list *negotiation_require,
++ const struct string_list *negotiation_restrict,
const struct ref *remote_refs,
struct oid_array *commons)
{
@@ send-pack.c: static void get_commons_through_negotiation(struct repository *r,
- nr_negotiation_tip++;
+ child.no_stdin = 1;
+ child.out = -1;
+ strvec_pushl(&child.args, "fetch", "--negotiate-only", NULL);
+- for (ref = remote_refs; ref; ref = ref->next) {
+- if (!is_null_oid(&ref->new_oid)) {
+- strvec_pushf(&child.args, "--negotiation-tip=%s",
+- oid_to_hex(&ref->new_oid));
+- nr_negotiation_tip++;
++
++ if (negotiation_restrict && negotiation_restrict->nr) {
++ struct string_list_item *item;
++ for_each_string_list_item(item, negotiation_restrict)
++ strvec_pushf(&child.args, "--negotiation-restrict=%s",
++ item->string);
++ nr_negotiation_tip = negotiation_restrict->nr;
++ } else {
++ for (ref = remote_refs; ref; ref = ref->next) {
++ if (!is_null_oid(&ref->new_oid)) {
++ strvec_pushf(&child.args, "--negotiation-tip=%s",
++ oid_to_hex(&ref->new_oid));
++ nr_negotiation_tip++;
++ }
}
}
+
-+ if (must_have) {
++ if (negotiation_require && negotiation_require->nr) {
+ struct string_list_item *item;
-+ for_each_string_list_item(item, must_have)
-+ strvec_pushf(&child.args, "--must-have=%s",
++ for_each_string_list_item(item, negotiation_require)
++ strvec_pushf(&child.args, "--negotiation-require=%s",
+ item->string);
+ }
+
@@ send-pack.c: int send_pack(struct repository *r,
if (push_negotiate) {
trace2_region_enter("send_pack", "push_negotiate", r);
- get_commons_through_negotiation(r, args->url, remote_refs, &commons);
-+ get_commons_through_negotiation(r, args->url, args->must_have,
++ get_commons_through_negotiation(r, args->url,
++ args->negotiation_require,
++ args->negotiation_restrict,
+ remote_refs, &commons);
trace2_region_leave("send_pack", "push_negotiate", r);
}
@@ send-pack.h: struct repository;
struct send_pack_args {
const char *url;
-+ const struct string_list *must_have;
++ const struct string_list *negotiation_require;
++ const struct string_list *negotiation_restrict;
unsigned verbose:1,
quiet:1,
porcelain:1,
@@ t/t5516-fetch-push.sh: test_expect_success 'push with negotiation does not attem
! grep "Fetching submodule" err
'
-+test_expect_success 'push with negotiation and remote.<name>.mustHave' '
-+ test_when_finished rm -rf musthave &&
-+ mk_empty musthave &&
-+ git push musthave $the_first_commit:refs/remotes/origin/first_commit &&
-+ test_commit -C musthave unrelated_commit &&
-+ git -C musthave config receive.hideRefs refs/remotes/origin/first_commit &&
++test_expect_success 'push with negotiation and remote.<name>.negotiationRequire' '
++ test_when_finished rm -rf negotiation_require &&
++ mk_empty negotiation_require &&
++ git push negotiation_require $the_first_commit:refs/remotes/origin/first_commit &&
++ test_commit -C negotiation_require unrelated_commit &&
++ git -C negotiation_require config receive.hideRefs refs/remotes/origin/first_commit &&
++ test_when_finished "rm event" &&
++ GIT_TRACE2_EVENT="$(pwd)/event" \
++ git -c protocol.version=2 -c push.negotiate=1 \
++ -c remote.negotiation_require.negotiationRequire=refs/heads/main \
++ push negotiation_require refs/heads/main:refs/remotes/origin/main &&
++ test_grep \"key\":\"total_rounds\" event &&
++ grep_wrote 2 event # 1 commit, 1 tree
++'
++
++test_expect_success 'push with negotiation and remote.<name>.negotiationRestrict' '
++ test_when_finished rm -rf negotiation_restrict &&
++ mk_empty negotiation_restrict &&
++ git push negotiation_restrict $the_first_commit:refs/remotes/origin/first_commit &&
++ test_commit -C negotiation_restrict unrelated_commit &&
++ git -C negotiation_restrict config receive.hideRefs refs/remotes/origin/first_commit &&
+ test_when_finished "rm event" &&
+ GIT_TRACE2_EVENT="$(pwd)/event" \
+ git -c protocol.version=2 -c push.negotiate=1 \
-+ -c remote.musthave.mustHave=refs/heads/main \
-+ push musthave refs/heads/main:refs/remotes/origin/main &&
++ -c remote.negotiation_restrict.negotiationRestrict=refs/heads/main \
++ push negotiation_restrict refs/heads/main:refs/remotes/origin/main &&
+ test_grep \"key\":\"total_rounds\" event &&
+ grep_wrote 2 event # 1 commit, 1 tree
+'
@@ transport.c: static int git_transport_push(struct transport *transport, struct r
args.atomic = !!(flags & TRANSPORT_PUSH_ATOMIC);
args.push_options = transport->push_options;
args.url = transport->url;
-+ args.must_have = &transport->remote->must_have;
++ args.negotiation_require = &transport->remote->negotiation_require;
++ args.negotiation_restrict = &transport->remote->negotiation_restrict;
if (flags & TRANSPORT_PUSH_CERT_ALWAYS)
args.push_cert = SEND_PACK_PUSH_CERT_ALWAYS;Signature · 1 line
gitgitgadget
PATCH v2 1/7 t5516: fix test order flakiness
From: Derrick Stolee <[email protected]>
The 'fetch follows tags by default' test sorts using 'sort -k 4', but for-each-ref output only has 3 columns. This relies on sort treating records with fewer fields as having an empty fourth field, which may produce unstable results depending on locale. Use 'sort -k 3' to match the actual number of columns in the output.
--- t/t5516-fetch-push.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5516-fetch-push.sh +1 -1
@@ -1349,7 +1349,7 @@ test_expect_success 'fetch follows tags by default' ' git for-each-ref >tmp1 && sed -n "p; s|refs/heads/main$|refs/remotes/origin/main|p" tmp1 | sed -n "p; s|refs/heads/main$|refs/remotes/origin/HEAD|p" | - sort -k 4 >../expect + sort -k 3 >../expect ) && test_when_finished "rm -rf dst" && git init dst &&
Signature · 1 line
gitgitgadget
PATCH v2 2/7 fetch: add --negotiation-restrict option
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
The --negotiation-tip option to 'git fetch' and 'git pull' allows users to specify that they want to focus negotiation on a small set of references. This is a _restriction_ on the negotiation set, helping to focus the negotiation when the ref count is high. However, it doesn't allow for the ability to opportunistically select references beyond that list.
This subtle detail that this is a 'maximum set' and not a 'minimum set' is not immediately clear from the option name. This makes it more complicated to add a new option that provides the complementary behavior of a minimum set.
For now, create a new synonym option, --negotiation-restrict, that behaves identically to --negotiation-tip. Update the documentation to make it clear that this new name is the preferred option, but we keep the old name for compatibility.
Update a few warning messages with the new option, but also make them translatable with the option name inserted by formatting. At least one of these messages will be reused later for a new option.
--- Documentation/fetch-options.adoc | 4 ++++ builtin/fetch.c | 11 +++++++---- builtin/pull.c | 3 +++ t/t5510-fetch.sh | 25 +++++++++++++++++++++++++ t/t5702-protocol-v2.sh | 4 ++-- 5 files changed, 41 insertions(+), 6 deletions(-)
Documentation/fetch-options.adoc +4 -0
@@ -49,6 +49,7 @@ the current repository has the same history as the source repository. `.git/shallow`. This option updates `.git/shallow` and accepts such refs. +`--negotiation-restrict=(<commit>|<glob>)`:: `--negotiation-tip=(<commit>|<glob>)`:: By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt to
@@ -58,6 +59,9 @@ the current repository has the same history as the source repository. local ref is likely to have commits in common with the upstream ref being fetched. + +`--negotiation-restrict` is the preferred name for this option; +`--negotiation-tip` is accepted as a synonym. ++ This option may be specified more than once; if so, Git will report commits reachable from any of the given commits. +
builtin/fetch.c +7 -4
@@ -1558,8 +1558,8 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)refs_for_each_ref_ext (get_main_ref_store (the_repository), add_oid, oids, &opts);if (old_nr == oids->nr) -warning ("ignoring --negotiation-tip=%s because it does not match any refs" , - s); +warning (_ ("ignoring %s=%s because it does not match any refs" ), +"--negotiation-restrict" , s); } smart_options->negotiation_tips = oids; }
@@ -1599,7 +1599,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,if (transport->smart_options)add_negotiation_tips (transport->smart_options);else -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-restrict" ); }return transport; }
@@ -2567,6 +2568,8 @@ int cmd_fetch(int argc,OPT_IPVERSION (&family),OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), +N_ ("report that we have only objects reachable from this object" )),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
@@ -2657,7 +2660,7 @@ int cmd_fetch(int argc, }if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-tip=*" )); +die (_ ("--negotiate-only needs one or more --negotiation-restrict=*" ));if (deepen_relative) {if (deepen_relative <0 )
builtin/pull.c +3 -0
@@ -999,6 +999,9 @@ int cmd_pull(int argc,OPT_PASSTHRU_ARGV (0 ,"negotiation-tip" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ), +N_ ("report that we have only objects reachable from this object" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
t/t5510-fetch.sh +25 -0
@@ -1460,6 +1460,31 @@ EOF test_cmp fatal-expect fatal-actual ' +test_expect_success '--negotiation-restrict limits "have" lines sent' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 --negotiation-restrict=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict understands globs' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=*_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-tip=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
t/t5702-protocol-v2.sh +2 -2
@@ -869,14 +869,14 @@ setup_negotiate_only () { test_commit -C client three } -test_expect_success 'usage: --negotiate-only without --negotiation-tip' ' +test_expect_success 'usage: --negotiate-only without --negotiation-restrict' ' SERVER="server" && URI="file://$(pwd)/server" && setup_negotiate_only "$SERVER" "$URI" && cat >err.expect <<-\EOF && - fatal: --negotiate-only needs one or more --negotiation-tip=* + fatal: --negotiate-only needs one or more --negotiation-restrict=* EOF test_must_fail git -c protocol.version=2 -C client fetch \
Signature · 1 line
gitgitgadget
PATCH v2 3/7 transport: rename negotiation_tips
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
The previous change added the --negotiation-restrict synonym for the --negotiation-tips option for 'git fetch'. In anticipation of adding a new option that behaves similarly but with distinct changes to its behavior, rename the internal representation of this data from 'negotiation_tips' to 'negotiation_restrict_tips'.
The 'tips' part is kept because this is an oid_array in the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data.
--- builtin/fetch.c | 6 +++--- fetch-pack.c | 18 +++++++++--------- fetch-pack.h | 4 ++-- transport-helper.c | 2 +- transport.c | 10 +++++----- transport.h | 4 ++-- 6 files changed, 22 insertions(+), 22 deletions(-)
builtin/fetch.c +3 -3
@@ -1534,7 +1534,7 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_tips (struct git_transport_options *smart_options) +static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) {struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i;
@@ -1561,7 +1561,7 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)warning (_ ("ignoring %s=%s because it does not match any refs" ),"--negotiation-restrict" , s); } - smart_options->negotiation_tips = oids; + smart_options->negotiation_restrict_tips = oids; } static struct transport *prepare_transport(struct remote *remote, int deepen,
@@ -1597,7 +1597,7 @@ static struct transport *prepare_transport(struct remote *remote, int deepen, }if (negotiation_tip.nr) {if (transport->smart_options) -add_negotiation_tips (transport->smart_options); +add_negotiation_restrict_tips (transport->smart_options);else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );
fetch-pack.c +9 -9
@@ -291,21 +291,21 @@ static int next_flush(int stateless_rpc, int count) }static void mark_tips (struct fetch_negotiator *negotiator, -const struct oid_array *negotiation_tips) +const struct oid_array *negotiation_restrict_tips) {struct refs_for_each_ref_options opts = { .flags =REFS_FOR_EACH_INCLUDE_BROKEN , };int i; -if (!negotiation_tips) { +if (!negotiation_restrict_tips) {refs_for_each_ref_ext (get_main_ref_store (the_repository), rev_list_insert_ref_oid, negotiator, &opts);return ; } -for (i =0 ; i < negotiation_tips->nr; i++) -rev_list_insert_ref (negotiator, &negotiation_tips->oid[i]); +for (i =0 ; i < negotiation_restrict_tips->nr; i++) +rev_list_insert_ref (negotiator, &negotiation_restrict_tips->oid[i]);return ; }
@@ -355,7 +355,7 @@ static int find_common(struct fetch_negotiator *negotiator,PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object); fetching =0 ;
@@ -1728,7 +1728,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,else state =FETCH_SEND_REQUEST ; -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object);break ;
@@ -2177,7 +2177,7 @@ static void clear_common_flag(struct oidset *s) } } -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
@@ -2195,13 +2195,13 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,timestamp_t min_generation =GENERATION_NUMBER_INFINITY ;fetch_negotiator_init (the_repository, &negotiator); -mark_tips (&negotiator, negotiation_tips); +mark_tips (&negotiator, negotiation_restrict_tips);packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE |PACKET_READ_DIE_ON_ERR_PACKET ); -oid_array_for_each ((struct oid_array *) negotiation_tips, +oid_array_for_each ((struct oid_array *) negotiation_restrict_tips, add_to_object_array, &nt_object_array);
fetch-pack.h +2 -2
@@ -21,7 +21,7 @@ struct fetch_pack_args {* If not NULL, during packfile negotiation, fetch-pack will send "have" * lines only with these tips and their ancestors. */ -const struct oid_array *negotiation_tips; +const struct oid_array *negotiation_restrict_tips; unsigned deepen_relative:1; unsigned quiet:1;
@@ -89,7 +89,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,* In the capability advertisement that has happened prior to invoking this * function, the "wait-for-done" capability must be present. */ -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
transport-helper.c +1 -1
@@ -754,7 +754,7 @@ static int fetch_refs(struct transport *transport,set_helper_option (transport,"filter" , spec); } -if (data->transport_options.negotiation_tips) +if (data->transport_options.negotiation_restrict_tips)warning ("Ignoring --negotiation-tip because the protocol does not support it." );if (data->fetch)
transport.c +5 -5
@@ -463,7 +463,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.refetch = data->options.refetch; args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; - args.negotiation_tips = data->options.negotiation_tips; + args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -491,7 +491,7 @@ static int fetch_refs_via_pack(struct transport *transport,warning (_ ("server does not support wait-for-done" )); ret =-1 ; }else { - negotiate_using_fetch(data->options.negotiation_tips, + negotiate_using_fetch(data->options.negotiation_restrict_tips, transport->server_options, transport->stateless_rpc, data->fd,
@@ -979,9 +979,9 @@ static int disconnect_git(struct transport *transport)finish_connect (data->conn); } -if (data->options.negotiation_tips) { -oid_array_clear (data->options.negotiation_tips); -free (data->options.negotiation_tips); +if (data->options.negotiation_restrict_tips) { +oid_array_clear (data->options.negotiation_restrict_tips); +free (data->options.negotiation_restrict_tips); }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);
transport.h +2 -2
@@ -40,13 +40,13 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_tips in struct fetch_pack_args. +* negotiation_restrict_tips in struct fetch_pack_args. * * This field is only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ -struct oid_array *negotiation_tips; +struct oid_array *negotiation_restrict_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
Signature · 1 line
gitgitgadget
PATCH v2 4/7 remote: add remote.*.negotiationRestrict config
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tips. Both of these options restrict the set of 'haves' the client can send as part of negotiation.
This was previously not available via a configuration option. Add a new 'remote.<name>.negotiationRestrict' multi-valued config option that updates 'git fetch <name>' to use these restrictions by default.
If the user provides even one --negotiation-restrict argument, then the config is ignored.
--- Documentation/config/remote.adoc | 16 ++++++++++++++++ builtin/fetch.c | 24 ++++++++++++++++++++++-- remote.c | 6 ++++++ remote.h | 1 + t/t5510-fetch.sh | 22 ++++++++++++++++++++++ 5 files changed, 67 insertions(+), 2 deletions(-)
Documentation/config/remote.adoc +16 -0
@@ -107,6 +107,22 @@ priority configuration file (e.g. `.git/config` in a repository) to clear the values inherited from a lower priority configuration files (e.g. `$HOME/.gitconfig`). +remote.<name>.negotiationRestrict:: + When negotiating with this remote during `git fetch` and `git push`, + restrict the commits advertised as "have" lines to only those + reachable from refs matching the given patterns. This multi-valued + config option behaves like `--negotiation-restrict` on the command + line. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the +same as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-restrict` +command-line option. If `--negotiation-restrict` (or its synonym +`--negotiation-tip`) is specified on the command line, then the config +values are not used. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
builtin/fetch.c +22 -2
@@ -1601,6 +1601,19 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" ); + }else if (remote->negotiation_restrict.nr) { +struct string_list_item *item; +for_each_string_list_item (item, &remote->negotiation_restrict) +string_list_append (&negotiation_tip, item->string); +if (transport->smart_options) +add_negotiation_restrict_tips (transport->smart_options); +else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
@@ -2659,8 +2672,12 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-restrict=*" )); +if (negotiate_only && !negotiation_tip.nr) { +/* +* Defer this check: remote.<name>.negotiationRestrict may +* provide defaults in prepare_transport(). +*/ + }if (deepen_relative) {if (deepen_relative <0 )
@@ -2749,6 +2766,9 @@ int cmd_fetch(int argc,if (!remote)die (_ ("must supply remote when using --negotiate-only" )); gtransport =prepare_transport (remote,1 , &filter_options); +if (!gtransport->smart_options || + !gtransport->smart_options->negotiation_restrict_tips) +die (_ ("--negotiate-only needs one or more --negotiation-restrict=*" ));if (gtransport->smart_options) { gtransport->smart_options->acked_commits = &acked_commits; }else {
remote.c +6 -0
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_push (&ret->push);refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options); +string_list_init_dup (&ret->negotiation_restrict);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -179,6 +180,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy);FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 ); +string_list_clear (&remote->negotiation_restrict,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -562,6 +564,10 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"serveroption" )) {return parse_transport_option (key, value, &remote->server_options); + }else if (!strcmp (subkey,"negotiationrestrict" )) { +if (!value) +return config_error_nonbool (key); +string_list_append (&remote->negotiation_restrict, value); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -117,6 +117,7 @@ struct remote {char *http_proxy_authmethod;struct string_list server_options; +struct string_list negotiation_restrict;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +22 -0
@@ -1485,6 +1485,28 @@ test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' check_negotiation_tip ' +test_expect_success 'remote.<name>.negotiationRestrict used as default' ' + setup_negotiation_tip server server 0 && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success 'CLI --negotiation-restrict overrides remote config' ' + setup_negotiation_tip server server 0 && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + ALPHA_1=$(git -C client rev-parse alpha_1) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + origin alpha_s beta_s && + test_grep "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep ! "fetch> have $BETA_1" trace +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
Signature · 1 line
gitgitgadget
PATCH v2 5/7 fetch: add --negotiation-require option for negotiation
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new --negotiation-require option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
This is useful when the repository has a large number of references, so the normal negotiation algorithm truncates the list. This is especially important in repositories with long parallel commit histories. For example, a repo could have a 'dev' branch for development and a 'release' branch for released versions. If the 'dev' branch isn't selected for negotiation, then it's not a big deal because there are many in-progress development branches with a shared history. However, if 'release' is not selected for negotiation, then the server may think that this is the first time the client has asked for that reference, causing a full download of its parallel commit history (and any extra data that may be unique to that branch). This is based on a real example where certain fetches would grow to 60+ GB when a release branch updated.
This option is a complement to --negotiation-restrict, which reduces the negotiation ref set to a specific list. In the earlier example, using --negotiation-restrict to focus the negotiation to 'dev' and 'release' would avoid those problematic downloads, but would still not allow advertising potentially-relevant user brances. In this way, the 'require' version solves the problem I mention while allowing negotiation to pick other references opportunistically. The two options can also be combined to allow the best of both worlds.
The argument may be an exact ref name or a glob pattern. Non-existent refs are silently ignored.
Also add --negotiation-require to 'git pull' passthrough options.
--- Documentation/fetch-options.adoc | 19 +++++++ builtin/fetch.c | 10 ++++ builtin/pull.c | 3 + fetch-pack.c | 96 ++++++++++++++++++++++++++++++-- fetch-pack.h | 10 +++- t/t5510-fetch.sh | 66 ++++++++++++++++++++++ transport.c | 4 +- transport.h | 6 ++ 8 files changed, 206 insertions(+), 8 deletions(-)
Documentation/fetch-options.adoc +19 -0
@@ -73,6 +73,25 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate` configuration variables documented in linkgit:git-config[1], and the `--negotiate-only` option below. +`--negotiation-require=<revision>`:: + Ensure that the given ref tip is always sent as a "have" line + during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-restrict` restricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++ +This option may be specified more than once; if so, each ref is sent +unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax +is the same as for `--negotiation-restrict`. ++ +If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-require`. + `--negotiate-only`:: Do not fetch anything from the server, and instead print the ancestors of the provided `--negotiation-tip=` arguments,
builtin/fetch.c +10 -0
@@ -99,6 +99,7 @@ static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ;static struct string_list negotiation_tip =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_require =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;
@@ -1615,6 +1616,13 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,strbuf_release (&config_name); } } +if (negotiation_require.nr) { +if (transport->smart_options) + transport->smart_options->negotiation_require = &negotiation_require; +else +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-require" ); + }return transport; }
@@ -2583,6 +2591,8 @@ int cmd_fetch(int argc, N_("report that we have only objects reachable from this object")),OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_STRING_LIST (0 ,"negotiation-require" , &negotiation_require,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" )),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
builtin/pull.c +3 -0
@@ -1002,6 +1002,9 @@ int cmd_pull(int argc,OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-require" , &opt_fetch,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
fetch-pack.c +90 -6
203 lines not on this page · raw message
fetch-pack.h +9 -1
@@ -23,6 +23,13 @@ struct fetch_pack_args {*/ const struct oid_array *negotiation_restrict_tips; +/* +* If non-empty, ref patterns whose tips should always be sent +* as "have" lines during negotiation, regardless of what the +* negotiation algorithm selects. +*/ +const struct string_list *negotiation_require; + unsigned deepen_relative:1; unsigned quiet:1; unsigned keep_pack:1;
@@ -93,7 +100,8 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,const struct string_list *server_options, int stateless_rpc, int fd[], - struct oidset *acked_commits); + struct oidset *acked_commits, + const struct string_list *negotiation_require);/* * Print an appropriate error message for each sought ref that wasn't
t/t5510-fetch.sh +66 -0
72 lines not on this page · raw message
transport.c +3 -1
@@ -464,6 +464,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; + args.negotiation_require = data->options.negotiation_require; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -495,7 +496,8 @@ static int fetch_refs_via_pack(struct transport *transport, transport->server_options, transport->stateless_rpc, data->fd, - data->options.acked_commits); + data->options.acked_commits, + data->options.negotiation_require); ret =0 ; } goto cleanup;
transport.h +6 -0
@@ -48,6 +48,12 @@ struct git_transport_options {*/ struct oid_array *negotiation_restrict_tips; +/* +* If non-empty, ref patterns whose tips should always be sent +* as "have" lines during negotiation. +*/ +const struct string_list *negotiation_require; +/* * If allocated, whenever transport_fetch_refs() is called, add known * common commits to this oidset instead of fetching any packfiles.
Signature · 1 line
gitgitgadget
PATCH v2 6/7 remote: add negotiationRequire config as default for --negotiation-require
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new 'remote.<name>.negotiationRequire' multi-valued config option that provides default values for --negotiation-require when no --negotiation-require arguments are specified over the command line. This is a mirror of how 'remote.<name>.negotiationRestrict' specifies defaults for the --negotiation-restrict arguments.
Each value is either an exact ref name or a glob pattern whose tips should always be sent as 'have' lines during negotiation. The config values are resolved through the same resolve_negotiation_require() codepath as the CLI options.
This option is additive with the normal negotiation process: the negotiation algorithm still runs and advertises its own selected commits, but the refs matching the config are sent unconditionally on top of those heuristically selected commits.
--- Documentation/config/remote.adoc | 24 +++++++++++++++++ Documentation/fetch-options.adoc | 4 +++ builtin/fetch.c | 10 +++++++ remote.c | 6 +++++ remote.h | 1 + t/t5510-fetch.sh | 46 ++++++++++++++++++++++++++++++++ 6 files changed, 91 insertions(+)
Documentation/config/remote.adoc +24 -0
@@ -123,6 +123,30 @@ command-line option. If `--negotiation-restrict` (or its synonym `--negotiation-tip`) is specified on the command line, then the config values are not used. +remote.<name>.negotiationRequire:: + When negotiating with this remote during `git fetch` and `git push`, + the client advertises a list of commits that exist locally. In + repos with many references, this list of "haves" can be truncated. + Depending on data shape, dropping certain references may be + expensive. This multi-valued config option specifies ref patterns + whose tips should always be sent as "have" commits during fetch + negotiation with this remote. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the same +as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-require` +command-line option. If `--negotiation-require` is specified on the +command line, then the config values are not used. ++ +This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits, +but the refs matching `remote.<name>.negotiationRequire` are sent +unconditionally on top of those heuristically selected commits. This +option is also used during push negotiation when `push.negotiate` is +enabled. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
Documentation/fetch-options.adoc +4 -0
@@ -91,6 +91,10 @@ is the same as for `--negotiation-restrict`. If `--negotiation-restrict` is used, the have set is first restricted by that option and then increased to include the tips specified by `--negotiation-require`. ++ +If this option is not specified on the command line, then any +`remote.<name>.negotiationRequire` config values for the current remote +are used instead. `--negotiate-only`:: Do not fetch anything from the server, and instead print the
builtin/fetch.c +10 -0
@@ -1622,6 +1622,16 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-require" ); + }else if (remote->negotiation_require.nr) { +if (transport->smart_options) { + transport->smart_options->negotiation_require = &remote->negotiation_require; + }else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationRequire" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
remote.c +6 -0
@@ -153,6 +153,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options);string_list_init_dup (&ret->negotiation_restrict); +string_list_init_dup (&ret->negotiation_require);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -181,6 +182,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 );string_list_clear (&remote->negotiation_restrict,0 ); +string_list_clear (&remote->negotiation_require,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -568,6 +570,10 @@ static int handle_config(const char *key, const char *value,if (!value)return config_error_nonbool (key);string_list_append (&remote->negotiation_restrict, value); + }else if (!strcmp (subkey,"negotiationrequire" )) { +if (!value) +return config_error_nonbool (key); +string_list_append (&remote->negotiation_require, value); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -118,6 +118,7 @@ struct remote {struct string_list server_options;struct string_list negotiation_restrict; +struct string_list negotiation_require;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +46 -0
52 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v2 7/7 send-pack: pass negotiation config in push
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
When push.negotiate is enabled, 'git push' spawns a child 'git fetch --negotiate-only' process to find common commits. Pass --negotiation-require and --negotiation-restrict options from the 'remote.<name>.negotiationRequire' and 'remote.<name>.negotiationRestrict' config keys to this child process.
When negotiationRestrict is configured, it replaces the default behavior of using all remote refs as negotiation tips. This allows the user to control which local refs are used for push negotiation.
When negotiationRequire is configured, the specified ref patterns are passed as --negotiation-require to ensure their tips are always sent as 'have' lines during push negotiation.
This change also updates the use of --negotiation-tip into --negotiation-restrict now that the new synonym exists.
--- send-pack.c | 34 ++++++++++++++++++++++++++++------ send-pack.h | 2 ++ t/t5516-fetch-push.sh | 30 ++++++++++++++++++++++++++++++ transport.c | 2 ++ 4 files changed, 62 insertions(+), 6 deletions(-)
send-pack.c +28 -6
@@ -433,6 +433,8 @@ static void reject_invalid_nonce(const char *nonce, int len)static void get_commons_through_negotiation (struct repository *r,const char *url, +const struct string_list *negotiation_require, +const struct string_list *negotiation_restrict,const struct ref *remote_refs,struct oid_array *commons) {
@@ -445,13 +447,30 @@ static void get_commons_through_negotiation(struct repository *r, child.no_stdin =1 ; child.out =-1 ;strvec_pushl (&child.args,"fetch" ,"--negotiate-only" ,NULL ); -for (ref = remote_refs; ref; ref = ref->next) { -if (!is_null_oid (&ref->new_oid)) { -strvec_pushf (&child.args,"--negotiation-tip=%s" , -oid_to_hex (&ref->new_oid)); - nr_negotiation_tip++; + +if (negotiation_restrict && negotiation_restrict->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_restrict) +strvec_pushf (&child.args,"--negotiation-restrict=%s" , + item->string); + nr_negotiation_tip = negotiation_restrict->nr; + }else { +for (ref = remote_refs; ref; ref = ref->next) { +if (!is_null_oid (&ref->new_oid)) { +strvec_pushf (&child.args,"--negotiation-tip=%s" , +oid_to_hex (&ref->new_oid)); + nr_negotiation_tip++; + } } } + +if (negotiation_require && negotiation_require->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_require) +strvec_pushf (&child.args,"--negotiation-require=%s" , + item->string); + } +strvec_push (&child.args, url);if (!nr_negotiation_tip) {
@@ -528,7 +547,10 @@ int send_pack(struct repository *r,repo_config_get_bool (r,"push.negotiate" , &push_negotiate);if (push_negotiate) {trace2_region_enter ("send_pack" ,"push_negotiate" , r); -get_commons_through_negotiation (r, args->url, remote_refs, &commons); +get_commons_through_negotiation (r, args->url, + args->negotiation_require, + args->negotiation_restrict, + remote_refs, &commons);trace2_region_leave ("send_pack" ,"push_negotiate" , r); }
send-pack.h +2 -0
@@ -18,6 +18,8 @@ struct repository;struct send_pack_args {const char *url; +const struct string_list *negotiation_require; +const struct string_list *negotiation_restrict;unsigned verbose:1 , quiet:1 , porcelain:1,
t/t5516-fetch-push.sh +30 -0
@@ -254,6 +254,36 @@ test_expect_success 'push with negotiation does not attempt to fetch submodules' ! grep "Fetching submodule" err ' +test_expect_success 'push with negotiation and remote.<name>.negotiationRequire' ' + test_when_finished rm -rf negotiation_require && + mk_empty negotiation_require && + git push negotiation_require $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_require unrelated_commit && + git -C negotiation_require config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_require.negotiationRequire=refs/heads/main \ + push negotiation_require refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + +test_expect_success 'push with negotiation and remote.<name>.negotiationRestrict' ' + test_when_finished rm -rf negotiation_restrict && + mk_empty negotiation_restrict && + git push negotiation_restrict $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_restrict unrelated_commit && + git -C negotiation_restrict config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_restrict.negotiationRestrict=refs/heads/main \ + push negotiation_restrict refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + test_expect_success 'push without wildcard' ' mk_empty testrepo &&
transport.c +2 -0
@@ -921,6 +921,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re args.atomic = !!(flags &TRANSPORT_PUSH_ATOMIC ); args.push_options = transport->push_options; args.url = transport->url; + args.negotiation_require = &transport->remote->negotiation_require; + args.negotiation_restrict = &transport->remote->negotiation_restrict;if (flags &TRANSPORT_PUSH_CERT_ALWAYS ) args.push_cert =SEND_PACK_PUSH_CERT_ALWAYS ;
Signature · 1 line
gitgitgadget
Re: [PATCH v2 4/7] remote: add remote.*.negotiationRestrict config
In reply to Derrick Stolee via GitGitGadgetQuoted 36 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tips. Both of these options restrict the set of 'haves' the client can send as part of negotiation.
This was previously not available via a configuration option. Add a new 'remote.<name>.negotiationRestrict' multi-valued config option that updates 'git fetch <name>' to use these restrictions by default.
If the user provides even one --negotiation-restrict argument, then the config is ignored.
Signed-off-by Derrick Stolee <[email protected]>--- Documentation/config/remote.adoc | 16 ++++++++++++++++ builtin/fetch.c | 24 ++++++++++++++++++++++-- remote.c | 6 ++++++ remote.h | 1 + t/t5510-fetch.sh | 22 ++++++++++++++++++++++ 5 files changed, 67 insertions(+), 2 deletions(-)Documentation/config/remote.adoc +5
-0@@ -107,6 +107,22 @@ priority configuration file (e.g. `.git/config` in a repository) to clear the values inherited from a lower priority configuration files (e.g. `$HOME/.gitconfig`). +remote.<name>.negotiationRestrict:: + When negotiating with this remote during `git fetch` and `git push`, + restrict the commits advertised as "have" lines to only those + reachable from refs matching the given patterns. This multi-valued + config option behaves like `--negotiation-restrict` on the command
Documentation/config/remote.adoc priority configuration file (e.g. `.git/config` in a repository) to clear · line 124
Quoted 10 lines
Documentation/config/remote.adoc +10
-0+ line. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the +same as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-restrict` +command-line option. If `--negotiation-restrict` (or its synonym +`--negotiation-tip`) is specified on the command line, then the config +values are not used.
This is a tangent, but I wonder what happens when this is set in /etc/gitconfig or ~/.gitconfig by mistake. I personally do not think of any good reason to set it in either of these two places, so it might be fine to declare that we read this only from local configuration file or "git -c var=val" command line, but alternative that is easier to implement would be to allow for a variable definition syntax that allows you to say "forget everything you read so far, clear this multi-valued variable", e.g.
== in /etc/gitconfig ==
[remote "origin"]
negotiationRestrict = refs/pull/*
== in .git/config ==
[remote "origin"]
# clear them
negotiationRestrict =
negotiationRestrict = refs/heads/*
negotiationRestrict = refs/tags/*
or something like that, perhaps?
It is a shame that our configuration framework do not allow specifying their meanings and semantics to variables like parse-options do (where OPT_STRING_LIST naturally allows --no-negotiation-restrict to act as a way to clear the deck). Because there is no official way to programatically declare that remote.<name>.negotiationRestrict is a multi-valued variable whose values are stored in a string-list, the config callback needs to be coded to implement the behaviour for each variable X-<.
Re: [PATCH v2 5/7] fetch: add --negotiation-require option for negotiation
In reply to Derrick Stolee via GitGitGadgetQuoted 8 lines · Derrick Stolee via GitGitGadget
+
--negotiation-require=<revision>::Documentation/fetch-options.adoc +7
-0+ Ensure that the given ref tip is always sent as a "have" line + during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-restrict` restricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++
Documentation/fetch-options.adoc See also the `fetch.negotiationAlgorithm` and `push.negotiate` · line 93
Quoted 10 lines
Documentation/fetch-options.adoc +10
-0+This option may be specified more than once; if so, each ref is sent +unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax +is the same as for `--negotiation-restrict`. ++ +If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-require`.
Very readable. Nice.
builtin/fetch.c static struct transport *gsecondary; · line 102
Quoted 9 lines
builtin/fetch.c +1
-0@@ -99,6 +99,7 @@ static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ;static struct string_list negotiation_tip =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_require =STRING_LIST_INIT_NODUP ;
I thought _tip was renamed to _restrict in an earlier step, but that was only in the transport in [3/7]. Perhaps we want to rename the file-scope static variable negotiation_tip to negotiation_restrict in an earlier step, like in [2/7]?
builtin/fetch.c static struct transport *gsecondary;
Quoted 5 lines
builtin/fetch.c +5
-0+for_each_string_list_item (item, negotiation_require) { +if (!has_glob_specials (item->string)) { +struct object_id oid; +if (repo_get_oid (the_repository, item->string, &oid)) +continue ;
The configuration (or command line) says --nego-require=refs/heads/main but this old repository only has refs/heads/master; we do not want to error out in such a case.
Is it true, though? nego-{require,restrict} feels quite tied to each project and unless the configuration or command line options are applied blindly regardless of the project, such an error should not happen. Perhaps the user who gives a command line option "--nego-require=refs/heads/naster" may want to be reminded of a possible typo?
builtin/fetch.c
Quoted 2 lines
builtin/fetch.c +2
-0+if (!odb_has_object (the_repository->objects, &oid,0 )) +continue ;
This is a bit curious. When does the first condition holds but not the second? A lazy clone whose ref-tip contains a missing commit promised by somebody else?
In the presense of "promised objects are allowed to be missing" rule, silently skipping a missing object here is certainly conservative, but this is not an object that is buried deep in a tree hierarchy, but the top-level commit or tag that is directly pointed at by a ref, isn't it? I am a bit uneasy that we ignore such potential repository corruption (i.e., a missing object may not be something a promisor remote promised but simply missing).
Quoted 8 lines
builtin/fetch.c +4
-0@@ -474,7 +511,25 @@ static int find_common(struct fetch_negotiator *negotiator,trace2_region_enter ("fetch-pack" ,"negotiation_v0_v1" , the_repository); flushes =0 ; retval =-1 ; + +/* Send unconditional haves from --negotiation-require */ +resolve_negotiation_require (args->negotiation_require, + &negotiation_require_oids);
builtin/fetch.c static int find_common(struct fetch_negotiator *negotiator, · line 527
Quoted 10 lines
builtin/fetch.c +10
-0+if (oidset_size (&negotiation_require_oids)) { +struct oidset_iter iter; +oidset_iter_init (&negotiation_require_oids, &iter); + +while ((oid =oidset_iter_next (&iter))) { +packet_buf_write (&req_buf,"have %s\n" , +oid_to_hex (oid)); +print_verbose (args,"have %s" ,oid_to_hex (oid)); + } + }
OK. I think it makes sense to send these early. We have already dealt with the usual "tips" by calling mark_tips() way earlier, but that hasn't produced any "have" yet, and these will go before the ones from traversal. We do not traverse from these "require" and that may be why these are not called "_tips"?
And sending these early means the other side has much less chance to say "we've heard enough, stop!", so in a sense they are of much higher priority "have"s (I wonder what happens when they do want to say "stop!" while we are giving a lot of "have" from this loop, though).
builtin/fetch.c static int find_common(struct fetch_negotiator *negotiator, · line 531
Quoted 4 lines
builtin/fetch.c +3
-0while ((oid = negotiator->next (negotiator))) { +/* avoid duplicate oids from --negotiation-require */ +if (oidset_contains (&negotiation_require_oids, oid)) +continue ;
If objects rechable from "require" are traversed like others, then this "avoid duplicate" would become unnecessary, right?
Re: [PATCH v2 2/7] fetch: add --negotiation-restrict option
In reply to Derrick Stolee via GitGitGadgetbuiltin/fetch.c static void add_negotiation_tips(struct git_transport_options *smart_options)
Quoted 7 lines · Derrick Stolee via GitGitGadget
builtin/fetch.c +4
-3-warning ("ignoring --negotiation-tip=%s because it does not match any refs" , - s); +warning (_ ("ignoring %s=%s because it does not match any refs" ), +"--negotiation-restrict" , s); -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-restrict" );
These are nice touches to make sure translators cannot possibly botch these option names that must be given verbatim.
builtin/fetch.c int cmd_fetch(int argc, · line 2572
Quoted 9 lines
builtin/fetch.c +2
-0}return transport; }@@ -2567,6 +2568,8 @@ int cmd_fetch(int argc,OPT_IPVERSION (&family),OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), + N_("report that we have only objects reachable from this object")),
Is OPT_ALIAS() suitable for this?
builtin/fetch.c int cmd_fetch(int argc, · line 2663
Quoted 6 lines
builtin/fetch.c +1
-1@@ -2657,7 +2660,7 @@ int cmd_fetch(int argc, }if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-tip=*" )); +die (_ ("--negotiate-only needs one or more --negotiation-restrict=*" ));
OK. Shouldn't this also do the "%s" thing?
Re: [PATCH v2 2/7] fetch: add --negotiation-restrict option
Quoted 13 lines · Junio C Hamano
builtin/fetch.c static void add_negotiation_tips(struct git_transport_options *smart_options)
Quoted 7 lines · Derrick Stolee via GitGitGadget
builtin/fetch.c +4
-3-warning ("ignoring --negotiation-tip=%s because it does not match any refs" , - s); +warning (_ ("ignoring %s=%s because it does not match any refs" ), +"--negotiation-restrict" , s); -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-restrict" );These are nice touches to make sure translators cannot possibly botch these option names that must be given verbatim.
Quoted 8 lines
builtin/fetch.c int cmd_fetch(int argc, · line 2663
Quoted 6 lines
builtin/fetch.c +1
-1@@ -2657,7 +2660,7 @@ int cmd_fetch(int argc, }if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-tip=*" )); +die (_ ("--negotiate-only needs one or more --negotiation-restrict=*" ));OK. Shouldn't this also do the "%s" thing?
I think I had focused on adding "%s" to strings that were not previously translated, but adjusting the string under translation is enough to require retranslation. I should make it easier to translate, too.
Quoted 11 lines
builtin/fetch.c int cmd_fetch(int argc, · line 2572
Quoted 9 lines
builtin/fetch.c +2
-0}return transport; }@@ -2567,6 +2568,8 @@ int cmd_fetch(int argc,OPT_IPVERSION (&family),OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), + N_("report that we have only objects reachable from this object")),Is
OPT_ALIAS()suitable for this?
I was not aware of this. Thanks for the pointer!
I do plan to make "negotiation-tip" an alias for "negotiation-restrict" based on the new preference for *-restrict as the "real" option now. Is that the right way to do this?
Thanks,
-Stolee
Re: [PATCH v2 3/7] transport: rename negotiation_tips
In reply to Derrick Stolee via GitGitGadgetbuiltin/fetch.c
Quoted 4 lines · Derrick Stolee via GitGitGadget
builtin/fetch.c +0
-0
Don't we want to also rename the local negotiation_tip variable in cmd_fetch()?
Patrick
Re: [PATCH v2 5/7] fetch: add --negotiation-require option for negotiation
In reply to Derrick Stolee via GitGitGadgetQuoted 5 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
Add a new --negotiation-require option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
When reading "--negotiation-require" my mind immediately shifts towards a mode where we require the remote to have a specific reference, and if not we'll abort. That's of course not what you're proposing here, but I would think that I may not be the only person making that connection.
Would an alternative like "--negotiation-include" or "--negotiation-expand" be better?
Quoted 16 lines
Documentation/fetch-options.adoc +8
-0@@ -73,6 +73,25 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate` configuration variables documented in linkgit:git-config[1], and the `--negotiate-only` option below. +`--negotiation-require=<revision>`:: + Ensure that the given ref tip is always sent as a "have" line + during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-restrict` restricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++
Documentation/fetch-options.adoc See also the `fetch.negotiationAlgorithm` and `push.negotiate` · line 93
Quoted 10 lines
Documentation/fetch-options.adoc +10
-0+This option may be specified more than once; if so, each ref is sent +unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax +is the same as for `--negotiation-restrict`. ++ +If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-require`.
This interaction makes sense. You can basically say "send only local branches, but please _also_ send that one particular ref over there".
Quoted 12 lines
fetch-pack.c +4
-0@@ -474,7 +511,25 @@ static int find_common(struct fetch_negotiator *negotiator,trace2_region_enter ("fetch-pack" ,"negotiation_v0_v1" , the_repository); flushes =0 ; retval =-1 ; + +/* Send unconditional haves from --negotiation-require */ +resolve_negotiation_require (args->negotiation_require, + &negotiation_require_oids);
fetch-pack.c static int find_common(struct fetch_negotiator *negotiator, · line 527
Quoted 10 lines
fetch-pack.c +10
-0+if (oidset_size (&negotiation_require_oids)) { +struct oidset_iter iter; +oidset_iter_init (&negotiation_require_oids, &iter); + +while ((oid =oidset_iter_next (&iter))) { +packet_buf_write (&req_buf,"have %s\n" , +oid_to_hex (oid)); +print_verbose (args,"have %s" ,oid_to_hex (oid)); + } + }
Okay, so here we now unconditionally send our requested object IDs.
One thing I was wondering is whether we need to flush eventually. It can happen that the user specifies millions of refs, either intentionally or by accident. But I guess the answer might be "no", as the intent of the feature is that we indeed want to send all of those to the remote side, and the remote is being asked to consider all of those OIDs.
Patrick
Re: [PATCH v2 2/7] fetch: add --negotiation-restrict option
In reply to Derrick StoleeQuoted 7 lines · Derrick Stolee
Quoted 6 lines
builtin/fetch.c int cmd_fetch(int argc, · line 2572
Quoted 4 lines
OPT_STRING_LIST(0, "negotiation-tip",
&negotiation_tip,N_("revision"),
N_("report that we have only objects reachable from this object")),builtin/fetch.c +2
-0+OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), + N_("report that we have only objects reachable from this object")),Is
OPT_ALIAS()suitable for this?
Quoted 5 lines
I was not aware of this. Thanks for the pointer!
I do plan to make "negotiation-tip" an alias for "negotiation-restrict" based on the new preference for *-restrict as the "real" option now. Is that the right way to do this?
Let's see.
$ git grep OPT_ALIAS builtin/clone.c builtin/clone.c: OPT_ALIAS(0, "recursive", "recurse-submodules"), $ git clone -h usage: git clone [<options>] [--] <repo> [<dir>]
-v, --[no-]verbose be more verbose
-q, --[no-]quiet be more quiet
...
--[no-]recurse-submodules[=<pathspec>]
initialize submodules in the clone
--[no-]recursive[=<pathspec>]
alias of --recurse-submodules
...I think we gave the operation the name "recursive", with a common short sightedness that anything we are adding "recursive" for is the only kind of recursiveness, and then prepared for a future where things other than submodules can also be sources of recursiveness by making "recurse-submodules" the official name, while still allowing historical name as the synonym.
In this case, if "-restrict" will become the official name, it should be listed first, and then the historical name should be made its alias.
So
OPT_STRING_LIST(0, "negotiation-restrict", &negotiation_tip, ...), OPT_ALIAS(0, "negotiation-tip", "negotiation-restrict"),
would be the right combination in the correct order, I think. Mention the official thing first, and then tell that another thing is an alias to what the readers have already seen after that (e.g., c28b036f (clone: reorder --recursive/--recurse-submodules, 2020-03-16)).
Re: [PATCH v2 2/7] fetch: add --negotiation-restrict option
Quoted 16 lines · Junio C Hamano
Quoted 7 lines · Derrick Stolee
Quoted 6 lines
builtin/fetch.c int cmd_fetch(int argc, · line 2572
Quoted 4 lines
OPT_STRING_LIST(0, "negotiation-tip",
&negotiation_tip,N_("revision"),
N_("report that we have only objects reachable from this object")),builtin/fetch.c +2
-0+OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), + N_("report that we have only objects reachable from this object")),Is
OPT_ALIAS()suitable for this?Quoted 5 lines
I was not aware of this. Thanks for the pointer!
I do plan to make "negotiation-tip" an alias for "negotiation-restrict" based on the new preference for *-restrict as the "real" option now. Is that the right way to do this?
Let's see.
...
Quoted 10 lines
So
OPT_STRING_LIST(0, "negotiation-restrict", &negotiation_tip, ...), OPT_ALIAS(0, "negotiation-tip", "negotiation-restrict"),would be the right combination in the correct order, I think. Mention the official thing first, and then tell that another thing is an alias to what the readers have already seen after that (e.g.,
c28b036f(clone: reorder --recursive/--recurse-submodules, 2020-03-16)).
Thanks! This is indeed what I have in my local copy in preparation for v3. It helps to have early confirmation about this.
-Stolee
Re: [PATCH v2 5/7] fetch: add --negotiation-require option for negotiation
In reply to Patrick SteinhardtQuoted 7 lines · Patrick Steinhardt
Quoted 5 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
Add a new --negotiation-require option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
Quoted 7 lines
When reading "--negotiation-require" my mind immediately shifts towards a mode where we require the remote to have a specific reference, and if not we'll abort. That's of course not what you're proposing here, but I would think that I may not be the only person making that connection.
Would an alternative like "--negotiation-include" or "--negotiation-expand" be better?
"include" does sound good to me. I'm open to it. I'll let this idea stew and try prepping my local branch in this direction.
Quoted 16 lines
fetch-pack.c static int find_common(struct fetch_negotiator *negotiator, · line 527
Quoted 13 lines
fetch-pack.c +13
-0+/* Send unconditional haves from --negotiation-require */ +resolve_negotiation_require (args->negotiation_require, + &negotiation_require_oids); +if (oidset_size (&negotiation_require_oids)) { +struct oidset_iter iter; +oidset_iter_init (&negotiation_require_oids, &iter); + +while ((oid =oidset_iter_next (&iter))) { +packet_buf_write (&req_buf,"have %s\n" , +oid_to_hex (oid)); +print_verbose (args,"have %s" ,oid_to_hex (oid)); + } + }Okay, so here we now unconditionally send our requested object IDs.
Quoted 5 lines
One thing I was wondering is whether we need to flush eventually. It can happen that the user specifies millions of refs, either intentionally or by accident. But I guess the answer might be "no", as the intent of the feature is that we indeed want to send all of those to the remote side, and the remote is being asked to consider all of those OIDs.
The idea is indeed to send all of the requested OIDs, but this does present an interesting behavior where the Git client can allow the user to misconfigure themselves to send larger-than-normal negotiation requests. Previously, the client would protect the negotiation with a maximum set of haves.
Is there any concern about this becoming a vector for increased load on servers?
Would it be good to have some kind of advice message when the config matches a set of haves that we think is too large? That would maybe be a way to help users get out of a self-made problem.
Thanks,
-Stolee
Re: [PATCH v2 5/7] fetch: add --negotiation-require option for negotiation
In reply to Junio C HamanoQuoted 1 line · Junio C Hamano
"Derrick Stolee via GitGitGadget" <[email protected]> writes:
...
Quoted 6 lines
Quoted 1 line
+static struct
string_listnegotiation_require=STRING_LIST_INIT_NODUP;I thought
_tipwas renamed to_restrictin an earlier step, but that was only in the transport in [3/7]. Perhaps we want to rename the file-scope static variablenegotiation_tiptonegotiation_restrictin an earlier step, like in [2/7]?
This one was missed but will be fixed in v3.
Quoted 10 lines
fetch-pack.c static void send_filter(struct fetch_pack_args *args, · line 356
Quoted 5 lines
fetch-pack.c +5
-0+for_each_string_list_item (item, negotiation_require) { +if (!has_glob_specials (item->string)) { +struct object_id oid; +if (repo_get_oid (the_repository, item->string, &oid)) +continue ;The configuration (or command line) says --nego-require=refs/heads/main but this old repository only has refs/heads/master; we do not want to error out in such a case.
Quoted 6 lines
Is it true, though? nego-{require,restrict} feels quite tied to each project and unless the configuration or command line options are applied blindly regardless of the project, such an error should not happen. Perhaps the user who gives a command line option "--nego-require=refs/heads/naster" may want to be reminded of a possible typo?
You're right here. We shouldn't die() on a bad ref passed this way.
This should be a best-effort attempt to include a "have" and continue normally if it isn't found.
Quoted 6 lines
fetch-pack.c static void send_filter(struct fetch_pack_args *args, · line 358
Quoted 2 lines
fetch-pack.c +2
-0+if (!odb_has_object (the_repository->objects, &oid,0 )) +continue ;This is a bit curious. When does the first condition holds but not the second? A lazy clone whose ref-tip contains a missing commit promised by somebody else?
Good point. This would occur if a ref exists but points to a missing object, which should mean the repo is corrupt and we can't trust that the fetch will succeed (or should).
Quoted 7 lines
In the presense of "promised objects are allowed to be missing" rule, silently skipping a missing object here is certainly conservative, but this is not an object that is buried deep in a tree hierarchy, but the top-level commit or tag that is directly pointed at by a ref, isn't it? I am a bit uneasy that we ignore such potential repository corruption (i.e., a missing object may not be something a promisor remote promised but simply missing).
I'll update this to be an error.
Quoted 16 lines
Quoted 16 lines
fetch-pack.c +12
-0@@ -474,7 +511,25 @@ static int find_common(struct fetch_negotiator *negotiator,trace2_region_enter ("fetch-pack" ,"negotiation_v0_v1" , the_repository); flushes =0 ; retval =-1 ; + +/* Send unconditional haves from --negotiation-require */ +resolve_negotiation_require (args->negotiation_require, + &negotiation_require_oids); +if (oidset_size (&negotiation_require_oids)) { +struct oidset_iter iter; +oidset_iter_init (&negotiation_require_oids, &iter); + +while ((oid =oidset_iter_next (&iter))) { +packet_buf_write (&req_buf,"have %s\n" , +oid_to_hex (oid)); +print_verbose (args,"have %s" ,oid_to_hex (oid));
Quoted 8 lines
Review static int find_common(struct fetch_negotiator *negotiator, · line 527
Quoted 2 lines
(quoted diff) +2
-0+ } + }OK. I think it makes sense to send these early. We have already dealt with the usual "tips" by calling
mark_tips()way earlier, but that hasn't produced any "have" yet, and these will go before the ones from traversal. We do not traverse from these "require" and that may be why these are not called "_tips"?
It is correct that these required haves are not plugged into the walk.
Quoted 5 lines
And sending these early means the other side has much less chance to say "we've heard enough, stop!", so in a sense they are of much higher priority "have"s (I wonder what happens when they do want to say "stop!" while we are giving a lot of "have" from this loop, though).
I believe that we don't give the server an opportunity to say "stop" until we've completed a "round" (see the 'if (flush_at <= ++count)' case).
With this in mind, I should be incrementing 'count' while sending the required haves.
Quoted 7 lines
fetch-pack.c static int find_common(struct fetch_negotiator *negotiator, · line 532
Quoted 4 lines
fetch-pack.c +3
-0while ((oid = negotiator->next (negotiator))) { +/* avoid duplicate oids from --negotiation-require */ +if (oidset_contains (&negotiation_require_oids, oid)) +continue ;If objects rechable from "require" are traversed like others, then this "avoid duplicate" would become unnecessary, right?
Yes, if we add the required things to the traversal, then we wouldn't worry about duplicates. We'd also need to do things in a different way:
1. The negotiator has a next() method that could do a number of things, but is responsible for walking history and ignoring IDs that are reachable from already-emitted haves.
2. This _could_ help us avoid sending required ID if we have emitted a have that could reach that ID.
3. However, this requires waiting until we flush a round of haves before determining if we should send the required IDs based on the walk to this point.
Perhaps the better way to incorporate things together would be to mark the required IDs as COMMON as we emit them, which would signal to the negotiation walks that they should not re-emit it as a have (but since we don't add SEEN, they still walk through the commit to its later ancestors).
Thanks,
-Stolee
v3 cover fetch: rework negotiation tip options
In reply to Derrick Stolee via GitGitGadgetFetch negotiation aims to find enough information from haves and wants such that the server can be reasonably confident that it will send all necessary objects and not too many "extra" objects that the client already has. However, this can break down if there are too many references, since Git truncates the list of haves based on a few factors (a 256 count limit or the server sending an ACK at the right time).
We already have the --negotiation-tip feature to focus the set of references that are used in negotiation, but I feel like this is designed backwards. I'd rather that we have a way to say "this is an important set of refs, but feel free to add more refs if needed" than "only use these refs for negotiation".
Here's an example that demonstrates the problem. In an internal monorepo, developers work off of the 'main' branch so there are thousands of user branches that each add a few commits different from the 'main' branch. However, there is also a long-lived 'release' branch. This branch has a first-parent history that is parallel to 'main' and each of those commits is a merge whose second parent is a commit from 'main' that had a successful CI run. There are additional changes in the 'release' branch merge commits that add some changelog data, so there is a nontrivial set of novel blob content in that branch and not just a different set of commits.
The problem we had was that our georeplication system was regularly fetching from the origin and trying to get all data from all reachable branches. When the 'release' branch updated, the client would run out of haves before advertising its copy of the 'release' branch, but it would still list the new 'release' tip as a want. The server would then think that the client had never fetched that branch before and would send all of the changelog data from the whole history of the repo. (This led to a lot of downstream problems; we mitigated by setting a refspec that stopped fetching the 'release' branch, but this is not ideal.)
What I'd like is a mechanism to say "always advertise the client's version of 'main' and 'release' but also opportunistically include some user branches".
Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.
There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.
This patch series adds a new '--negotiation-include' option that does what I
want: it makes sure that these references are included as 'have's during negotiation. In order to help clarify the difference between this and '--negotiation-tip', I first create a synonym called '--negotiation-restrict'.
Both of these options get 'remote.*.negotiation(Include|Restrict)' config options that enable their behavior by default.
During development, I had briefly considered only using config values, but that required some strange changes to care about the remote name in the transport layer. This was most different in the 'git push' integration. When I discovered the '--negotiation-tip' feature during the process, that gave me a clear pattern to follow with the addition of a config on top.
Updates in v2
=============
This version is a near-complete rewrite based on feedback around the names of the previous option and config. The --negotiation-restrict option is new and the ability to set it via config is also new.
I did try to be more careful around translatable error messages, too.
Updates in v3
=============
* --negotiation-tip is now an alias of --negotiation-restrict.
* More translatable strings use %s to isolate non-translatable options from translatable words.
* The string_list named negotiation_tip is now renamed to negotiation_restrict.
* The config options now allow an empty value to reset the list.
* The --negotiation-require option is now called --negotiation-include.
* Similarly, the config option is renamed and all code references.
* The included haves now mark their commits as COMMON so commits that they can reach are not included in the negotiation walk if they are reached from the restricted commits.
* The ref iterators are more careful about failing on bad references (ref exists but object doesn't) and ignoring missing references (perhaps config is erroneous?).
* When sending tips during push negotiation, use the --negotiation-restrict option instead of -tip.
Thanks, -Stolee
Derrick Stolee (7):
t5516: fix test order flakiness
fetch: add --negotiation-restrict option
transport: rename negotiation_tips
remote: add remote.*.negotiationRestrict config
fetch: add --negotiation-include option for negotiation
remote: add remote.*.negotiationInclude config
send-pack: pass negotiation config in push
Documentation/config/remote.adoc | 46 +++++++++ Documentation/fetch-options.adoc | 27 +++++ builtin/fetch.c | 70 ++++++++++--- builtin/pull.c | 6 ++ fetch-pack.c | 130 +++++++++++++++++++++--- fetch-pack.h | 14 ++- remote.c | 16 +++ remote.h | 2 + send-pack.c | 39 ++++++-- send-pack.h | 2 + t/t5510-fetch.sh | 166 +++++++++++++++++++++++++++++++ t/t5516-fetch-push.sh | 32 +++++- t/t5702-protocol-v2.sh | 4 +- transport-helper.c | 2 +- transport.c | 16 +-- transport.h | 10 +- 16 files changed, 529 insertions(+), 53 deletions(-)
base-commit: 6e8d538aab8fe4dd07ba9fb87b5c7edcfa5706ad
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2085%2Fderrickstolee%2Fmust-have-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2085/derrickstolee/must-have-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/2085
Range-diff vs v2:
1: 466c56abe0 = 1: 466c56abe0 t5516: fix test order flakiness 2: 9a25b0fade ! 2: fe875399a8 fetch: add --negotiation-restrict option
@@ Commit message
For now, create a new synonym option, --negotiation-restrict, that behaves identically to --negotiation-tip. Update the documentation to make it clear that this new name is the preferred option, but we keep
- the old name for compatibility.
+ the old name for compatibility. Mark --negotiation-tip as an alias of the
+ new, preferred option.
Update a few warning messages with the new option, but also make them translatable with the option name inserted by formatting. At least one
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remot
return transport;
}
@@ builtin/fetch.c: int cmd_fetch(int argc,
+ N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+ OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
OPT_IPVERSION(&family),
- OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
- N_("report that we have only objects reachable from this object")),
+- OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+ OPT_STRING_LIST(0, "negotiation-restrict", &negotiation_tip, N_("revision"),
-+ N_("report that we have only objects reachable from this object")),
+ N_("report that we have only objects reachable from this object")),
++ OPT_ALIAS(0, "negotiation-tip", "negotiation-restrict"),
OPT_BOOL(0, "negotiate-only", &negotiate_only,
N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
@@ builtin/fetch.c: int cmd_fetch(int argc,
if (negotiate_only && !negotiation_tip.nr)
- die(_("--negotiate-only needs one or more --negotiation-tip=*"));
-+ die(_("--negotiate-only needs one or more --negotiation-restrict=*"));
++ die(_("%s needs one or more %s"), "--negotiate-only",
++ "--negotiation-restrict=*");
if (deepen_relative) {
if (deepen_relative < 0)
3: 0f89665aee ! 3: 4332cbf266 transport: rename negotiation_tips
@@ Commit message
layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data.
+ Also update the string_list used to store the inputs from command-line
+ options.
+
## builtin/fetch.c ##
+@@ builtin/fetch.c: static struct transport *gtransport;
+ static struct transport *gsecondary;
+ static struct refspec refmap = REFSPEC_INIT_FETCH;
+ static struct string_list server_options = STRING_LIST_INIT_DUP;
+-static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
++static struct string_list negotiation_restrict = STRING_LIST_INIT_NODUP;
+
+ struct fetch_config {
+ enum display_format display_format;
@@ builtin/fetch.c: static int add_oid(const struct reference *ref, void *cb_data)
return 0;
}
@@ builtin/fetch.c: static int add_oid(const struct reference *ref, void *cb_data)
{
struct oid_array *oids = xcalloc(1, sizeof(*oids));
int i;
+
+- for (i = 0; i < negotiation_tip.nr; i++) {
+- const char *s = negotiation_tip.items[i].string;
++ for (i = 0; i < negotiation_restrict.nr; i++) {
++ const char *s = negotiation_restrict.items[i].string;
+ struct refs_for_each_ref_options opts = {
+ .pattern = s,
+ };
@@ builtin/fetch.c: static void add_negotiation_tips(struct git_transport_options *smart_options)
warning(_("ignoring %s=%s because it does not match any refs"),
"--negotiation-restrict", s);
@@ builtin/fetch.c: static void add_negotiation_tips(struct git_transport_options *
static struct transport *prepare_transport(struct remote *remote, int deepen,
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remote, int deepen,
+ set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, spec);
+ set_option(transport, TRANS_OPT_FROM_PROMISOR, "1");
}
- if (negotiation_tip.nr) {
+- if (negotiation_tip.nr) {
++ if (negotiation_restrict.nr) {
if (transport->smart_options)
- add_negotiation_tips(transport->smart_options);
+ add_negotiation_restrict_tips(transport->smart_options);
else
warning(_("ignoring %s because the protocol does not support it"),
"--negotiation-restrict");
+@@ builtin/fetch.c: int cmd_fetch(int argc,
+ N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+ OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+ OPT_IPVERSION(&family),
+- OPT_STRING_LIST(0, "negotiation-restrict", &negotiation_tip, N_("revision"),
++ OPT_STRING_LIST(0, "negotiation-restrict", &negotiation_restrict, N_("revision"),
+ N_("report that we have only objects reachable from this object")),
+ OPT_ALIAS(0, "negotiation-tip", "negotiation-restrict"),
+ OPT_BOOL(0, "negotiate-only", &negotiate_only,
+@@ builtin/fetch.c: int cmd_fetch(int argc,
+ config.display_format = DISPLAY_FORMAT_PORCELAIN;
+ }
+
+- if (negotiate_only && !negotiation_tip.nr)
++ if (negotiate_only && !negotiation_restrict.nr)
+ die(_("%s needs one or more %s"), "--negotiate-only",
+ "--negotiation-restrict=*");
+
## fetch-pack.c ##
@@ fetch-pack.c: static int next_flush(int stateless_rpc, int count)
4: a731f4fc87 ! 4: d2f48b78b5 remote: add remote.*.negotiationRestrict config
@@ Commit message
If the user provides even one --negotiation-restrict argument, then the
config is ignored.
+ An empty value resets the value list to allow ignoring earlier config
+ values, such as those that might be set in system or global config.
+
Signed-off-by: Derrick Stolee <[email protected]>
## Documentation/config/remote.adoc ##
@@ Documentation/config/remote.adoc: priority configuration file (e.g. `.git/config
+command-line option. If `--negotiation-restrict` (or its synonym
+`--negotiation-tip`) is specified on the command line, then the config
+values are not used.
+++
++Blank values signal to ignore all previous values, allowing a reset of
++the list from broader config scenarios.
+
remote.<name>.followRemoteHEAD::
How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD`
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remot
+ } else if (remote->negotiation_restrict.nr) {
+ struct string_list_item *item;
+ for_each_string_list_item(item, &remote->negotiation_restrict)
-+ string_list_append(&negotiation_tip, item->string);
++ string_list_append(&negotiation_restrict, item->string);
+ if (transport->smart_options)
+ add_negotiation_restrict_tips(transport->smart_options);
+ else {
@@ builtin/fetch.c: int cmd_fetch(int argc,
config.display_format = DISPLAY_FORMAT_PORCELAIN;
}
-- if (negotiate_only && !negotiation_tip.nr)
-- die(_("--negotiate-only needs one or more --negotiation-restrict=*"));
-+ if (negotiate_only && !negotiation_tip.nr) {
-+ /*
-+ * Defer this check: remote.<name>.negotiationRestrict may
-+ * provide defaults in prepare_transport().
-+ */
-+ }
-
+- if (negotiate_only && !negotiation_restrict.nr)
+- die(_("%s needs one or more %s"), "--negotiate-only",
+- "--negotiation-restrict=*");
+-
if (deepen_relative) {
if (deepen_relative < 0)
+ die(_("negative depth in --deepen is not supported"));
@@ builtin/fetch.c: int cmd_fetch(int argc,
if (!remote)
die(_("must supply remote when using --negotiate-only"));
gtransport = prepare_transport(remote, 1, &filter_options);
+ if (!gtransport->smart_options ||
+ !gtransport->smart_options->negotiation_restrict_tips)
-+ die(_("--negotiate-only needs one or more --negotiation-restrict=*"));
++ die(_("%s needs one or more %s"), "--negotiate-only",
++ "--negotiation-restrict=*");
if (gtransport->smart_options) {
gtransport->smart_options->acked_commits = &acked_commits;
} else {
@@ remote.c: static int handle_config(const char *key, const char *value,
return parse_transport_option(key, value,
&remote->server_options);
+ } else if (!strcmp(subkey, "negotiationrestrict")) {
-+ if (!value)
-+ return config_error_nonbool(key);
-+ string_list_append(&remote->negotiation_restrict, value);
++ /* reset list on empty value. */
++ if (!value || !*value)
++ string_list_clear(&remote->negotiation_restrict, 0);
++ else
++ string_list_append(&remote->negotiation_restrict, value);
} else if (!strcmp(subkey, "followremotehead")) {
const char *no_warn_branch;
if (!strcmp(value, "never"))
@@ t/t5510-fetch.sh: test_expect_success '--negotiation-restrict and --negotiation-
+test_expect_success 'remote.<name>.negotiationRestrict used as default' '
+ setup_negotiation_tip server server 0 &&
++
++ # test the reset of the list on an empty value
++ git -C client config --add remote.origin.negotiationRestrict alpha_2 &&
++ git -C client config --add remote.origin.negotiationRestrict "" &&
+ git -C client config --add remote.origin.negotiationRestrict alpha_1 &&
+ git -C client config --add remote.origin.negotiationRestrict beta_1 &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
5: 49c80cef2e ! 5: ae81ef36a1 fetch: add --negotiation-require option for negotiation
@@ Metadata
Author: Derrick Stolee <[email protected]>
## Commit message ##
- fetch: add --negotiation-require option for negotiation
+ fetch: add --negotiation-include option for negotiation
- Add a new --negotiation-require option to 'git fetch', which ensures
+ Add a new --negotiation-include option to 'git fetch', which ensures
that certain ref tips are always sent as 'have' lines during fetch
negotiation, regardless of what the negotiation algorithm selects.
@@ Commit message
--negotiation-restrict to focus the negotiation to 'dev' and 'release'
would avoid those problematic downloads, but would still not allow
advertising potentially-relevant user brances. In this way, the
- 'require' version solves the problem I mention while allowing
+ 'include' version solves the problem I mention while allowing
negotiation to pick other references opportunistically. The two options
can also be combined to allow the best of both worlds.
The argument may be an exact ref name or a glob pattern. Non-existent
- refs are silently ignored.
+ refs are silently ignored. This behavior is also updated in the ref matching
+ logic for the related --negotiation-restrict option to match.
- Also add --negotiation-require to 'git pull' passthrough options.
+ The implementation outputs the requested objects as haves before the
+ negotiation algorithm kicks in and performs a priority-queue walk from the
+ tip commits. In order to avoid duplicates, we mark the requested objects as
+ COMMON so they (and their descendants) are not output by the negotiator. The
+ negotiator still outputs at least one have before a round is flushed, when
+ the server could ACK to stop the negotiation.
+
+ Also add --negotiation-include to 'git pull' passthrough options.
Signed-off-by: Derrick Stolee <[email protected]>
@@ Documentation/fetch-options.adoc: See also the `fetch.negotiationAlgorithm` and
configuration variables documented in linkgit:git-config[1], and the
`--negotiate-only` option below.
-+`--negotiation-require=<revision>`::
++`--negotiation-include=<revision>`::
+ Ensure that the given ref tip is always sent as a "have" line
+ during fetch negotiation, regardless of what the negotiation
+ algorithm selects. This is useful to guarantee that common
@@ Documentation/fetch-options.adoc: See also the `fetch.negotiationAlgorithm` and
++
+If `--negotiation-restrict` is used, the have set is first restricted by
+that option and then increased to include the tips specified by
-+`--negotiation-require`.
++`--negotiation-include`.
+
`--negotiate-only`::
Do not fetch anything from the server, and instead print the
@@ builtin/fetch.c
@@ builtin/fetch.c: static struct transport *gsecondary;
static struct refspec refmap = REFSPEC_INIT_FETCH;
static struct string_list server_options = STRING_LIST_INIT_DUP;
- static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-+static struct string_list negotiation_require = STRING_LIST_INIT_NODUP;
+ static struct string_list negotiation_restrict = STRING_LIST_INIT_NODUP;
++static struct string_list negotiation_include = STRING_LIST_INIT_NODUP;
struct fetch_config {
enum display_format display_format;
+@@ builtin/fetch.c: static void add_negotiation_restrict_tips(struct git_transport_options *smart_op
+ int old_nr;
+ if (!has_glob_specials(s)) {
+ struct object_id oid;
++
++ /* Ignore missing reference. */
+ if (repo_get_oid(the_repository, s, &oid))
+- die(_("%s is not a valid object"), s);
++ continue;
++ /* Fail on missing object pointed by ref. */
+ if (!odb_has_object(the_repository->objects, &oid, 0))
+ die(_("the object %s does not exist"), s);
++
+ oid_array_append(oids, &oid);
+ continue;
+ }
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remote, int deepen,
strbuf_release(&config_name);
}
}
-+ if (negotiation_require.nr) {
++ if (negotiation_include.nr) {
+ if (transport->smart_options)
-+ transport->smart_options->negotiation_require = &negotiation_require;
++ transport->smart_options->negotiation_include = &negotiation_include;
+ else
+ warning(_("ignoring %s because the protocol does not support it"),
-+ "--negotiation-require");
++ "--negotiation-include");
+ }
return transport;
}
@@ builtin/fetch.c: int cmd_fetch(int argc,
+ OPT_STRING_LIST(0, "negotiation-restrict", &negotiation_restrict, N_("revision"),
N_("report that we have only objects reachable from this object")),
- OPT_STRING_LIST(0, "negotiation-restrict", &negotiation_tip, N_("revision"),
- N_("report that we have only objects reachable from this object")),
-+ OPT_STRING_LIST(0, "negotiation-require", &negotiation_require, N_("revision"),
+ OPT_ALIAS(0, "negotiation-tip", "negotiation-restrict"),
++ OPT_STRING_LIST(0, "negotiation-include", &negotiation_include, N_("revision"),
+ N_("ensure this ref is always sent as a negotiation have")),
OPT_BOOL(0, "negotiate-only", &negotiate_only,
N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
@@ builtin/pull.c: int cmd_pull(int argc,
OPT_PASSTHRU_ARGV(0, "negotiation-restrict", &opt_fetch, N_("revision"),
N_("report that we have only objects reachable from this object"),
0),
-+ OPT_PASSTHRU_ARGV(0, "negotiation-require", &opt_fetch, N_("revision"),
++ OPT_PASSTHRU_ARGV(0, "negotiation-include", &opt_fetch, N_("revision"),
+ N_("ensure this ref is always sent as a negotiation have"),
+ 0),
OPT_BOOL(0, "show-forced-updates", &opt_show_forced_updates,
@@ fetch-pack.c: static void send_filter(struct fetch_pack_args *args,
+static int add_oid_to_oidset(const struct reference *ref, void *cb_data)
+{
+ struct oidset *set = cb_data;
-+ if (odb_has_object(the_repository->objects, ref->oid, 0))
-+ oidset_insert(set, ref->oid);
++ if (!odb_has_object(the_repository->objects, ref->oid, 0))
++ die(_("the object %s does not exist"), oid_to_hex(ref->oid));
++ oidset_insert(set, ref->oid);
+ return 0;
+}
+
-+static void resolve_negotiation_require(const struct string_list *negotiation_require,
++static void resolve_negotiation_include(const struct string_list *negotiation_include,
+ struct oidset *result)
+{
+ struct string_list_item *item;
+
-+ if (!negotiation_require || !negotiation_require->nr)
++ if (!negotiation_include || !negotiation_include->nr)
+ return;
+
-+ for_each_string_list_item(item, negotiation_require) {
++ for_each_string_list_item(item, negotiation_include) {
+ if (!has_glob_specials(item->string)) {
+ struct object_id oid;
++
++ /* Ignore missing reference. */
+ if (repo_get_oid(the_repository, item->string, &oid))
+ continue;
++
++ /* Fail on missing object pointed by ref. */
+ if (!odb_has_object(the_repository->objects, &oid, 0))
-+ continue;
++ die(_("the object %s does not exist"),
++ item->string);
++
+ oidset_insert(result, &oid);
+ } else {
+ struct refs_for_each_ref_options opts = {
@@ fetch-pack.c: static int find_common(struct fetch_negotiator *negotiator,
struct strbuf req_buf = STRBUF_INIT;
size_t state_len = 0;
struct packet_reader reader;
-+ struct oidset negotiation_require_oids = OIDSET_INIT;
++ struct oidset negotiation_include_oids = OIDSET_INIT;
if (args->stateless_rpc && multi_ack == 1)
die(_("the option '%s' requires '%s'"), "--stateless-rpc", "multi_ack_detailed");
@@ fetch-pack.c: static int find_common(struct fetch_negotiator *negotiator,
flushes = 0;
retval = -1;
+
-+ /* Send unconditional haves from --negotiation-require */
-+ resolve_negotiation_require(args->negotiation_require,
-+ &negotiation_require_oids);
-+ if (oidset_size(&negotiation_require_oids)) {
++ /* Send unconditional haves from --negotiation-include */
++ resolve_negotiation_include(args->negotiation_include,
++ &negotiation_include_oids);
++ if (oidset_size(&negotiation_include_oids)) {
+ struct oidset_iter iter;
-+ oidset_iter_init(&negotiation_require_oids, &iter);
++ oidset_iter_init(&negotiation_include_oids, &iter);
+
+ while ((oid = oidset_iter_next(&iter))) {
++ struct commit *commit;
+ packet_buf_write(&req_buf, "have %s\n",
+ oid_to_hex(oid));
+ print_verbose(args, "have %s", oid_to_hex(oid));
++ count++;
++
++ /*
++ * If this is a commit, then mark as COMMON to
++ * avoid the negotiator also outputting it as
++ * a have.
++ */
++ commit = lookup_commit(the_repository, oid);
++ if (commit &&
++ !repo_parse_commit(the_repository, commit))
++ commit->object.flags |= COMMON;
+ }
+ }
+
while ((oid = negotiator->next(negotiator))) {
-+ /* avoid duplicate oids from --negotiation-require */
-+ if (oidset_contains(&negotiation_require_oids, oid))
-+ continue;
packet_buf_write(&req_buf, "have %s\n", oid_to_hex(oid));
print_verbose(args, "have %s", oid_to_hex(oid));
- in_vain++;
@@ fetch-pack.c: done:
flushes++;
}
strbuf_release(&req_buf);
-+ oidset_clear(&negotiation_require_oids);
++ oidset_clear(&negotiation_include_oids);
if (!got_ready || !no_done)
consume_shallow_list(args, &reader);
@@ fetch-pack.c: static void add_common(struct strbuf *req_buf, struct oidset *comm
struct strbuf *req_buf,
- int *haves_to_send)
+ int *haves_to_send,
-+ struct oidset *negotiation_require_oids)
++ struct oidset *negotiation_include_oids)
{
int haves_added = 0;
const struct object_id *oid;
-+ /* Send unconditional haves from --negotiation-require */
-+ if (negotiation_require_oids) {
++ /* Send unconditional haves from --negotiation-include */
++ if (negotiation_include_oids) {
+ struct oidset_iter iter;
-+ oidset_iter_init(negotiation_require_oids, &iter);
++ oidset_iter_init(negotiation_include_oids, &iter);
+
+ while ((oid = oidset_iter_next(&iter)))
+ packet_buf_write(req_buf, "have %s\n",
@@ fetch-pack.c: static void add_common(struct strbuf *req_buf, struct oidset *comm
+ }
+
while ((oid = negotiator->next(negotiator))) {
-+ if (negotiation_require_oids &&
-+ oidset_contains(negotiation_require_oids, oid))
++ if (negotiation_include_oids &&
++ oidset_contains(negotiation_include_oids, oid))
+ continue;
packet_buf_write(req_buf, "have %s\n", oid_to_hex(oid));
if (++haves_added >= *haves_to_send)
@@ fetch-pack.c: static int send_fetch_request(struct fetch_negotiator *negotiator,
int *haves_to_send, int *in_vain,
- int sideband_all, int seen_ack)
+ int sideband_all, int seen_ack,
-+ struct oidset *negotiation_require_oids)
++ struct oidset *negotiation_include_oids)
{
int haves_added;
int done_sent = 0;
@@ fetch-pack.c: static int send_fetch_request(struct fetch_negotiator *negotiator,
- haves_added = add_haves(negotiator, &req_buf, haves_to_send);
+ haves_added = add_haves(negotiator, &req_buf, haves_to_send,
-+ negotiation_require_oids);
++ negotiation_include_oids);
*in_vain += haves_added;
trace2_data_intmax("negotiation_v2", the_repository, "haves_added", haves_added);
trace2_data_intmax("negotiation_v2", the_repository, "in_vain", *in_vain);
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
struct ref *ref = copy_ref_list(orig_ref);
enum fetch_state state = FETCH_CHECK_LOCAL;
struct oidset common = OIDSET_INIT;
-+ struct oidset negotiation_require_oids = OIDSET_INIT;
++ struct oidset negotiation_include_oids = OIDSET_INIT;
struct packet_reader reader;
int in_vain = 0, negotiation_started = 0;
int negotiation_round = 0;
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
state = FETCH_SEND_REQUEST;
mark_tips(negotiator, args->negotiation_restrict_tips);
-+ resolve_negotiation_require(args->negotiation_require,
-+ &negotiation_require_oids);
++ resolve_negotiation_include(args->negotiation_include,
++ &negotiation_include_oids);
for_each_cached_alternate(negotiator,
insert_one_alternate_object);
break;
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
reader.use_sideband,
- seen_ack)) {
+ seen_ack,
-+ &negotiation_require_oids)) {
++ &negotiation_include_oids)) {
trace2_region_leave_printf("negotiation_v2", "round",
the_repository, "%d",
negotiation_round);
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
negotiator->release(negotiator);
oidset_clear(&common);
-+ oidset_clear(&negotiation_require_oids);
++ oidset_clear(&negotiation_include_oids);
return ref;
}
@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_res
int fd[],
- struct oidset *acked_commits)
+ struct oidset *acked_commits,
-+ const struct string_list *negotiation_require)
++ const struct string_list *negotiation_include)
{
struct fetch_negotiator negotiator;
struct packet_reader reader;
struct object_array nt_object_array = OBJECT_ARRAY_INIT;
struct strbuf req_buf = STRBUF_INIT;
-+ struct oidset negotiation_require_oids = OIDSET_INIT;
++ struct oidset negotiation_include_oids = OIDSET_INIT;
int haves_to_send = INITIAL_FLUSH;
int in_vain = 0;
int seen_ack = 0;
@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_res
fetch_negotiator_init(the_repository, &negotiator);
mark_tips(&negotiator, negotiation_restrict_tips);
-+ resolve_negotiation_require(negotiation_require,
-+ &negotiation_require_oids);
++ resolve_negotiation_include(negotiation_include,
++ &negotiation_include_oids);
+
packet_reader_init(&reader, fd[0], NULL, 0,
PACKET_READ_CHOMP_NEWLINE |
@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_res
- haves_added = add_haves(&negotiator, &req_buf, &haves_to_send);
+ haves_added = add_haves(&negotiator, &req_buf, &haves_to_send,
-+ &negotiation_require_oids);
++ &negotiation_include_oids);
in_vain += haves_added;
if (!haves_added || (seen_ack && in_vain >= MAX_IN_VAIN))
last_iteration = 1;
@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_res
clear_common_flag(acked_commits);
object_array_clear(&nt_object_array);
-+ oidset_clear(&negotiation_require_oids);
++ oidset_clear(&negotiation_include_oids);
negotiator.release(&negotiator);
strbuf_release(&req_buf);
}
@@ fetch-pack.h: struct fetch_pack_args {
+ * as "have" lines during negotiation, regardless of what the
+ * negotiation algorithm selects.
+ */
-+ const struct string_list *negotiation_require;
++ const struct string_list *negotiation_include;
+
unsigned deepen_relative:1;
unsigned quiet:1;
@@ fetch-pack.h: void negotiate_using_fetch(const struct oid_array *negotiation_res
int fd[],
- struct oidset *acked_commits);
+ struct oidset *acked_commits,
-+ const struct string_list *negotiation_require);
++ const struct string_list *negotiation_include);
/*
* Print an appropriate error message for each sought ref that wasn't
@@ t/t5510-fetch.sh: test_expect_success 'CLI --negotiation-restrict overrides remo
test_grep ! "fetch> have $BETA_1" trace
'
-+test_expect_success '--negotiation-require includes configured refs as haves' '
++test_expect_success '--negotiation-include includes configured refs as haves' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
+ --negotiation-restrict=alpha_1 \
-+ --negotiation-require=refs/tags/beta_1 \
++ --negotiation-include=refs/tags/beta_1 \
+ origin alpha_s beta_s &&
+
+ ALPHA_1=$(git -C client rev-parse alpha_1) &&
@@ t/t5510-fetch.sh: test_expect_success 'CLI --negotiation-restrict overrides remo
+ test_grep "fetch> have $BETA_1" trace
+'
+
-+test_expect_success '--negotiation-require works with glob patterns' '
++test_expect_success '--negotiation-include works with glob patterns' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
+ --negotiation-restrict=alpha_1 \
-+ --negotiation-require="refs/tags/beta_*" \
++ --negotiation-include="refs/tags/beta_*" \
+ origin alpha_s beta_s &&
+
+ BETA_1=$(git -C client rev-parse beta_1) &&
@@ t/t5510-fetch.sh: test_expect_success 'CLI --negotiation-restrict overrides remo
+ test_grep "fetch> have $BETA_2" trace
+'
+
-+test_expect_success '--negotiation-require is additive with negotiation' '
++test_expect_success '--negotiation-include is additive with negotiation' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --negotiation-require=refs/tags/beta_1 \
++ --negotiation-include=refs/tags/beta_1 \
+ origin alpha_s beta_s &&
+
+ BETA_1=$(git -C client rev-parse beta_1) &&
+ test_grep "fetch> have $BETA_1" trace
+'
+
-+test_expect_success '--negotiation-require ignores non-existent refs silently' '
++test_expect_success '--negotiation-include ignores non-existent refs silently' '
+ setup_negotiation_tip server server 0 &&
+
+ git -C client fetch --quiet \
+ --negotiation-restrict=alpha_1 \
-+ --negotiation-require=refs/tags/nonexistent \
++ --negotiation-include=refs/tags/nonexistent \
+ origin alpha_s beta_s 2>err &&
+ test_must_be_empty err
+'
+
-+test_expect_success '--negotiation-require avoids duplicates with negotiator' '
++test_expect_success '--negotiation-include avoids duplicates with negotiator' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
+ ALPHA_1=$(git -C client rev-parse alpha_1) &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
+ --negotiation-restrict=alpha_1 \
-+ --negotiation-require=refs/tags/alpha_1 \
++ --negotiation-include=refs/tags/alpha_1 \
+ origin alpha_s beta_s &&
+
+ test_grep "fetch> have $ALPHA_1" trace >matches &&
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
args.stateless_rpc = transport->stateless_rpc;
args.server_options = transport->server_options;
args.negotiation_restrict_tips = data->options.negotiation_restrict_tips;
-+ args.negotiation_require = data->options.negotiation_require;
++ args.negotiation_include = data->options.negotiation_include;
args.reject_shallow_remote = transport->smart_options->reject_shallow;
if (!data->finished_handshake) {
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
data->fd,
- data->options.acked_commits);
+ data->options.acked_commits,
-+ data->options.negotiation_require);
++ data->options.negotiation_include);
ret = 0;
}
goto cleanup;
@@ transport.h: struct git_transport_options {
+ * If non-empty, ref patterns whose tips should always be sent
+ * as "have" lines during negotiation.
+ */
-+ const struct string_list *negotiation_require;
++ const struct string_list *negotiation_include;
+
/*
* If allocated, whenever transport_fetch_refs() is called, add known 6: 081f904c07 ! 6: a2d15fa12a remote: add negotiationRequire config as default for --negotiation-require
@@ Metadata
Author: Derrick Stolee <[email protected]>
## Commit message ##
- remote: add negotiationRequire config as default for --negotiation-require
+ remote: add remote.*.negotiationInclude config
- Add a new 'remote.<name>.negotiationRequire' multi-valued config option
- that provides default values for --negotiation-require when no
- --negotiation-require arguments are specified over the command line.
- This is a mirror of how 'remote.<name>.negotiationRestrict' specifies
- defaults for the --negotiation-restrict arguments.
+ Add a new 'remote.<name>.negotiationInclude' multi-valued config option that
+ provides default values for --negotiation-include when no
+ --negotiation-include arguments are specified over the command line. This
+ is a mirror of how 'remote.<name>.negotiationRestrict' specifies defaults
+ for the --negotiation-restrict arguments.
- Each value is either an exact ref name or a glob pattern whose tips
- should always be sent as 'have' lines during negotiation. The config
- values are resolved through the same resolve_negotiation_require()
- codepath as the CLI options.
+ Each value is either an exact ref name or a glob pattern whose tips should
+ always be sent as 'have' lines during negotiation. The config values are
+ resolved through the same resolve_negotiation_include() codepath as the CLI
+ options.
- This option is additive with the normal negotiation process: the
- negotiation algorithm still runs and advertises its own selected
- commits, but the refs matching the config are sent unconditionally
- on top of those heuristically selected commits.
+ This option is additive with the normal negotiation process: the negotiation
+ algorithm still runs and advertises its own selected commits, but the refs
+ matching the config are sent unconditionally on top of those heuristically
+ selected commits.
+
+ Similar to the negotiationRestrict config, an empty value resets the value
+ list to allow ignoring earlier config values, such as those that might be
+ set in system or global config.
## Documentation/config/remote.adoc ##
-@@ Documentation/config/remote.adoc: command-line option. If --negotiation-restrict (or its synonym
- --negotiation-tip) is specified on the command line, then the config
- values are not used.
+@@ Documentation/config/remote.adoc: values are not used.
+ Blank values signal to ignore all previous values, allowing a reset of
+ the list from broader config scenarios.
-+remote.<name>.negotiationRequire::
++remote.<name>.negotiationInclude::
+ When negotiating with this remote during git fetch and git push,
+ the client advertises a list of commits that exist locally. In
+ repos with many references, this list of "haves" can be truncated.
@@ Documentation/config/remote.adoc: command-line option. If `--negotiation-restri
+glob pattern (e.g. refs/heads/release/*). The pattern syntax is the same +as for --negotiation-restrict.
++
-+These config values are used as defaults for the --negotiation-require -+command-line option. If --negotiation-require is specified on the ++These config values are used as defaults for the --negotiation-include ++command-line option. If --negotiation-include is specified on the
+command line, then the config values are not used.
++
+This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits,
-+but the refs matching remote.<name>.negotiationRequire are sent ++but the refs matching remote.<name>.negotiationInclude are sent
+unconditionally on top of those heuristically selected commits. This +option is also used during push negotiation when push.negotiate is +enabled.
+++
++Blank values signal to ignore all previous values, allowing a reset of ++the list from broader config scenarios.
+
remote.<name>.followRemoteHEAD::
How linkgit:git-fetch[1] should handle updates to remotes/<name>/HEAD
@@ Documentation/fetch-options.adoc
@@ Documentation/fetch-options.adoc: is the same as for --negotiation-restrict.
If --negotiation-restrict is used, the have set is first restricted by that option and then increased to include the tips specified by
- --negotiation-require.
+ --negotiation-include.
++
+If this option is not specified on the command line, then any
-+remote.<name>.negotiationRequire config values for the current remote ++remote.<name>.negotiationInclude config values for the current remote
+are used instead.
--negotiate-only::
@@ builtin/fetch.c
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remote, int deepen,
else
warning(_("ignoring %s because the protocol does not support it"),
- "--negotiation-require");
-+ } else if (remote->negotiation_require.nr) {
+ "--negotiation-include");
++ } else if (remote->negotiation_include.nr) {
+ if (transport->smart_options) {
-+ transport->smart_options->negotiation_require = &remote->negotiation_require;
++ transport->smart_options->negotiation_include = &remote->negotiation_include;
+ } else {
+ struct strbuf config_name = STRBUF_INIT;
-+ strbuf_addf(&config_name, "remote.%s.negotiationRequire", remote->name);
++ strbuf_addf(&config_name, "remote.%s.negotiationInclude", remote->name);
+ warning(_("ignoring %s because the protocol does not support it"),
+ config_name.buf);
+ strbuf_release(&config_name);
@@ remote.c: static struct remote *make_remote(struct remote_state *remote_state,
refspec_init_fetch(&ret->fetch);
string_list_init_dup(&ret->server_options); string_list_init_dup(&ret->negotiation_restrict);
-+ string_list_init_dup(&ret->negotiation_require);
++ string_list_init_dup(&ret->negotiation_include);
ALLOC_GROW(remote_state->remotes, remote_state->remotes_nr + 1,
remote_state->remotes_alloc);
@@ remote.c: static void remote_clear(struct remote *remote)
FREE_AND_NULL(remote->http_proxy_authmethod); string_list_clear(&remote->server_options, 0); string_list_clear(&remote->negotiation_restrict, 0);
-+ string_list_clear(&remote->negotiation_require, 0);
++ string_list_clear(&remote->negotiation_include, 0);
}
static void add_merge(struct branch *branch, const char *name)
@@ remote.c: static int handle_config(const char *key, const char *value,
- if (!value)
- return config_error_nonbool(key);
- string_list_append(&remote->negotiation_restrict, value);
-+ } else if (!strcmp(subkey, "negotiationrequire")) {
-+ if (!value)
-+ return config_error_nonbool(key);
-+ string_list_append(&remote->negotiation_require, value);
+ string_list_clear(&remote->negotiation_restrict, 0);
+ else
+ string_list_append(&remote->negotiation_restrict, value);
++ } else if (!strcmp(subkey, "negotiationinclude")) {
++ /* reset list on empty value. */
++ if (!value || !*value)
++ string_list_clear(&remote->negotiation_include, 0);
++ else
++ string_list_append(&remote->negotiation_include, value);
} else if (!strcmp(subkey, "followremotehead")) {
const char *no_warn_branch;
if (!strcmp(value, "never"))
@@ remote.h: struct remote {
struct string_list server_options;
struct string_list negotiation_restrict;
-+ struct string_list negotiation_require;
++ struct string_list negotiation_include;
enum follow_remote_head_settings follow_remote_head; const char *no_warn_branch;
## t/t5510-fetch.sh ##
-@@ t/t5510-fetch.sh: test_expect_success '--negotiation-require avoids duplicates with negotiator' ' +@@ t/t5510-fetch.sh: test_expect_success '--negotiation-include avoids duplicates with negotiator' '
test_line_count = 1 matches
'
-+test_expect_success 'remote.<name>.negotiationRequire used as default for --negotiation-require' ' ++test_expect_success 'remote.<name>.negotiationInclude used as default for --negotiation-include' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ git -C client config --add remote.origin.negotiationRequire refs/tags/beta_1 &&
++ # test the reset of the list on an empty value
++ git -C client config --add remote.origin.negotiationInclude refs/tags/alpha_1 &&
++ git -C client config --add remote.origin.negotiationInclude "" &&
++ git -C client config --add remote.origin.negotiationInclude refs/tags/beta_1 &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
+ --negotiation-restrict=alpha_1 \
+ origin alpha_s beta_s &&
@@ t/t5510-fetch.sh: test_expect_success '--negotiation-require avoids duplicates w
+ test_grep "fetch> have $BETA_1" trace
+'
+
-+test_expect_success 'remote.<name>.negotiationRequire works with glob patterns' ' ++test_expect_success 'remote.<name>.negotiationInclude works with glob patterns' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ git -C client config --add remote.origin.negotiationRequire "refs/tags/beta_*" &&
++ git -C client config --add remote.origin.negotiationInclude "refs/tags/beta_*" &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
+ --negotiation-restrict=alpha_1 \
+ origin alpha_s beta_s &&
@@ t/t5510-fetch.sh: test_expect_success '--negotiation-require avoids duplicates w
+ test_grep "fetch> have $BETA_2" trace
+'
+
-+test_expect_success 'CLI --negotiation-require overrides remote.<name>.negotiationRequire' ' ++test_expect_success 'CLI --negotiation-include overrides remote.<name>.negotiationInclude' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
+
-+ git -C client config --add remote.origin.negotiationRequire refs/tags/beta_2 &&
++ git -C client config --add remote.origin.negotiationInclude refs/tags/beta_2 &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
+ --negotiation-restrict=alpha_1 \
-+ --negotiation-require=refs/tags/beta_1 \
++ --negotiation-include=refs/tags/beta_1 \
+ origin alpha_s beta_s &&
+
+ BETA_1=$(git -C client rev-parse beta_1) &&
7: 7cccf59beb ! 7: e6c79f0661 send-pack: pass negotiation config in push
@@ Commit message
When push.negotiate is enabled, 'git push' spawns a child 'git fetch
--negotiate-only' process to find common commits. Pass
- --negotiation-require and --negotiation-restrict options from the
- 'remote.<name>.negotiationRequire' and
+ --negotiation-include and --negotiation-restrict options from the
+ 'remote.<name>.negotiationInclude' and
'remote.<name>.negotiationRestrict' config keys to this child process.
When negotiationRestrict is configured, it replaces the default
behavior of using all remote refs as negotiation tips. This allows
the user to control which local refs are used for push negotiation.
- When negotiationRequire is configured, the specified ref patterns
- are passed as --negotiation-require to ensure their tips are always
+ When negotiationInclude is configured, the specified ref patterns
+ are passed as --negotiation-include to ensure their tips are always
sent as 'have' lines during push negotiation.
This change also updates the use of --negotiation-tip into
@@ send-pack.c: static void reject_invalid_nonce(const char *nonce, int len)
static void get_commons_through_negotiation(struct repository *r,
const char *url,
-+ const struct string_list *negotiation_require,
++ const struct string_list *negotiation_include,
+ const struct string_list *negotiation_restrict,
const struct ref *remote_refs,
struct oid_array *commons)
{
-@@ send-pack.c: static void get_commons_through_negotiation(struct repository *r,
+ struct child_process child = CHILD_PROCESS_INIT;
+ const struct ref *ref;
+ int len = r->hash_algo->hexsz + 1; /* hash + NL */
+- int nr_negotiation_tip = 0;
++ int nr_negotiation = 0;
+
+ child.git_cmd = 1;
child.no_stdin = 1;
child.out = -1;
strvec_pushl(&child.args, "fetch", "--negotiate-only", NULL);
@@ send-pack.c: static void get_commons_through_negotiation(struct repository *r,
+ for_each_string_list_item(item, negotiation_restrict)
+ strvec_pushf(&child.args, "--negotiation-restrict=%s",
+ item->string);
-+ nr_negotiation_tip = negotiation_restrict->nr;
++ nr_negotiation = negotiation_restrict->nr;
+ } else {
+ for (ref = remote_refs; ref; ref = ref->next) {
+ if (!is_null_oid(&ref->new_oid)) {
-+ strvec_pushf(&child.args, "--negotiation-tip=%s",
++ strvec_pushf(&child.args, "--negotiation-restrict=%s",
+ oid_to_hex(&ref->new_oid));
-+ nr_negotiation_tip++;
++ nr_negotiation++;
+ }
}
}
+
-+ if (negotiation_require && negotiation_require->nr) {
++ if (negotiation_include && negotiation_include->nr) {
+ struct string_list_item *item;
-+ for_each_string_list_item(item, negotiation_require)
-+ strvec_pushf(&child.args, "--negotiation-require=%s",
++ for_each_string_list_item(item, negotiation_include)
++ strvec_pushf(&child.args, "--negotiation-include=%s",
+ item->string);
++ nr_negotiation += negotiation_include->nr;
+ }
+
strvec_push(&child.args, url);
- if (!nr_negotiation_tip) {
+- if (!nr_negotiation_tip) {
++ if (!nr_negotiation) {
+ child_process_clear(&child);
+ return;
+ }
@@ send-pack.c: int send_pack(struct repository *r,
repo_config_get_bool(r, "push.negotiate", &push_negotiate);
if (push_negotiate) {
trace2_region_enter("send_pack", "push_negotiate", r);
- get_commons_through_negotiation(r, args->url, remote_refs, &commons);
+ get_commons_through_negotiation(r, args->url,
-+ args->negotiation_require,
++ args->negotiation_include,
+ args->negotiation_restrict,
+ remote_refs, &commons);
trace2_region_leave("send_pack", "push_negotiate", r);
@@ send-pack.h: struct repository;
struct send_pack_args {
const char *url;
-+ const struct string_list *negotiation_require;
++ const struct string_list *negotiation_include;
+ const struct string_list *negotiation_restrict;
unsigned verbose:1,
quiet:1,
@@ t/t5516-fetch-push.sh: test_expect_success 'push with negotiation does not attem
! grep "Fetching submodule" err
'
-+test_expect_success 'push with negotiation and remote.<name>.negotiationRequire' '
-+ test_when_finished rm -rf negotiation_require &&
-+ mk_empty negotiation_require &&
-+ git push negotiation_require $the_first_commit:refs/remotes/origin/first_commit &&
-+ test_commit -C negotiation_require unrelated_commit &&
-+ git -C negotiation_require config receive.hideRefs refs/remotes/origin/first_commit &&
++test_expect_success 'push with negotiation and remote.<name>.negotiationInclude' '
++ test_when_finished rm -rf negotiation_include &&
++ mk_empty negotiation_include &&
++ git push negotiation_include $the_first_commit:refs/remotes/origin/first_commit &&
++ test_commit -C negotiation_include unrelated_commit &&
++ git -C negotiation_include config receive.hideRefs refs/remotes/origin/first_commit &&
+ test_when_finished "rm event" &&
+ GIT_TRACE2_EVENT="$(pwd)/event" \
+ git -c protocol.version=2 -c push.negotiate=1 \
-+ -c remote.negotiation_require.negotiationRequire=refs/heads/main \
-+ push negotiation_require refs/heads/main:refs/remotes/origin/main &&
++ -c remote.negotiation_include.negotiationInclude=refs/heads/main \
++ push negotiation_include refs/heads/main:refs/remotes/origin/main &&
+ test_grep \"key\":\"total_rounds\" event &&
+ grep_wrote 2 event # 1 commit, 1 tree
+'
@@ transport.c: static int git_transport_push(struct transport *transport, struct r
args.atomic = !!(flags & TRANSPORT_PUSH_ATOMIC);
args.push_options = transport->push_options;
args.url = transport->url;
-+ args.negotiation_require = &transport->remote->negotiation_require;
++ args.negotiation_include = &transport->remote->negotiation_include;
+ args.negotiation_restrict = &transport->remote->negotiation_restrict;
if (flags & TRANSPORT_PUSH_CERT_ALWAYS)Signature · 1 line
gitgitgadget
PATCH v3 1/7 t5516: fix test order flakiness
From: Derrick Stolee <[email protected]>
The 'fetch follows tags by default' test sorts using 'sort -k 4', but for-each-ref output only has 3 columns. This relies on sort treating records with fewer fields as having an empty fourth field, which may produce unstable results depending on locale. Use 'sort -k 3' to match the actual number of columns in the output.
--- t/t5516-fetch-push.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5516-fetch-push.sh +1 -1
@@ -1349,7 +1349,7 @@ test_expect_success 'fetch follows tags by default' ' git for-each-ref >tmp1 && sed -n "p; s|refs/heads/main$|refs/remotes/origin/main|p" tmp1 | sed -n "p; s|refs/heads/main$|refs/remotes/origin/HEAD|p" | - sort -k 4 >../expect + sort -k 3 >../expect ) && test_when_finished "rm -rf dst" && git init dst &&
Signature · 1 line
gitgitgadget
PATCH v3 2/7 fetch: add --negotiation-restrict option
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
The --negotiation-tip option to 'git fetch' and 'git pull' allows users to specify that they want to focus negotiation on a small set of references. This is a _restriction_ on the negotiation set, helping to focus the negotiation when the ref count is high. However, it doesn't allow for the ability to opportunistically select references beyond that list.
This subtle detail that this is a 'maximum set' and not a 'minimum set' is not immediately clear from the option name. This makes it more complicated to add a new option that provides the complementary behavior of a minimum set.
For now, create a new synonym option, --negotiation-restrict, that behaves identically to --negotiation-tip. Update the documentation to make it clear that this new name is the preferred option, but we keep the old name for compatibility. Mark --negotiation-tip as an alias of the new, preferred option.
Update a few warning messages with the new option, but also make them translatable with the option name inserted by formatting. At least one of these messages will be reused later for a new option.
--- Documentation/fetch-options.adoc | 4 ++++ builtin/fetch.c | 13 ++++++++----- builtin/pull.c | 3 +++ t/t5510-fetch.sh | 25 +++++++++++++++++++++++++ t/t5702-protocol-v2.sh | 4 ++-- 5 files changed, 42 insertions(+), 7 deletions(-)
Documentation/fetch-options.adoc +4 -0
@@ -49,6 +49,7 @@ the current repository has the same history as the source repository. `.git/shallow`. This option updates `.git/shallow` and accepts such refs. +`--negotiation-restrict=(<commit>|<glob>)`:: `--negotiation-tip=(<commit>|<glob>)`:: By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt to
@@ -58,6 +59,9 @@ the current repository has the same history as the source repository. local ref is likely to have commits in common with the upstream ref being fetched. + +`--negotiation-restrict` is the preferred name for this option; +`--negotiation-tip` is accepted as a synonym. ++ This option may be specified more than once; if so, Git will report commits reachable from any of the given commits. +
builtin/fetch.c +8 -5
@@ -1558,8 +1558,8 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)refs_for_each_ref_ext (get_main_ref_store (the_repository), add_oid, oids, &opts);if (old_nr == oids->nr) -warning ("ignoring --negotiation-tip=%s because it does not match any refs" , - s); +warning (_ ("ignoring %s=%s because it does not match any refs" ), +"--negotiation-restrict" , s); } smart_options->negotiation_tips = oids; }
@@ -1599,7 +1599,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,if (transport->smart_options)add_negotiation_tips (transport->smart_options);else -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-restrict" ); }return transport; }
@@ -2565,8 +2566,9 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
@@ -2657,7 +2659,8 @@ int cmd_fetch(int argc, }if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-tip=*" )); +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" );if (deepen_relative) {if (deepen_relative <0 )
builtin/pull.c +3 -0
@@ -999,6 +999,9 @@ int cmd_pull(int argc,OPT_PASSTHRU_ARGV (0 ,"negotiation-tip" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ), +N_ ("report that we have only objects reachable from this object" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
t/t5510-fetch.sh +25 -0
@@ -1460,6 +1460,31 @@ EOF test_cmp fatal-expect fatal-actual ' +test_expect_success '--negotiation-restrict limits "have" lines sent' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 --negotiation-restrict=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict understands globs' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=*_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-tip=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
t/t5702-protocol-v2.sh +2 -2
@@ -869,14 +869,14 @@ setup_negotiate_only () { test_commit -C client three } -test_expect_success 'usage: --negotiate-only without --negotiation-tip' ' +test_expect_success 'usage: --negotiate-only without --negotiation-restrict' ' SERVER="server" && URI="file://$(pwd)/server" && setup_negotiate_only "$SERVER" "$URI" && cat >err.expect <<-\EOF && - fatal: --negotiate-only needs one or more --negotiation-tip=* + fatal: --negotiate-only needs one or more --negotiation-restrict=* EOF test_must_fail git -c protocol.version=2 -C client fetch \
Signature · 1 line
gitgitgadget
PATCH v3 3/7 transport: rename negotiation_tips
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
The previous change added the --negotiation-restrict synonym for the --negotiation-tips option for 'git fetch'. In anticipation of adding a new option that behaves similarly but with distinct changes to its behavior, rename the internal representation of this data from 'negotiation_tips' to 'negotiation_restrict_tips'.
The 'tips' part is kept because this is an oid_array in the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data.
Also update the string_list used to store the inputs from command-line options.
--- builtin/fetch.c | 18 +++++++++--------- fetch-pack.c | 18 +++++++++--------- fetch-pack.h | 4 ++-- transport-helper.c | 2 +- transport.c | 10 +++++----- transport.h | 4 ++-- 6 files changed, 28 insertions(+), 28 deletions(-)
builtin/fetch.c +9 -9
@@ -98,7 +98,7 @@ static struct transport *gtransport;static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ; -static struct string_list negotiation_tip =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;
@@ -1534,13 +1534,13 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_tips (struct git_transport_options *smart_options) +static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) {struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i; -for (i =0 ; i < negotiation_tip.nr; i++) { -const char *s = negotiation_tip.items[i].string; +for (i =0 ; i < negotiation_restrict.nr; i++) { +const char *s = negotiation_restrict.items[i].string;struct refs_for_each_ref_options opts = { .pattern = s, };
@@ -1561,7 +1561,7 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)warning (_ ("ignoring %s=%s because it does not match any refs" ),"--negotiation-restrict" , s); } - smart_options->negotiation_tips = oids; + smart_options->negotiation_restrict_tips = oids; } static struct transport *prepare_transport(struct remote *remote, int deepen,
@@ -1595,9 +1595,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,set_option (transport,TRANS_OPT_LIST_OBJECTS_FILTER , spec);set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" ); } -if (negotiation_tip.nr) { +if (negotiation_restrict.nr) {if (transport->smart_options) -add_negotiation_tips (transport->smart_options); +add_negotiation_restrict_tips (transport->smart_options);else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );
@@ -2566,7 +2566,7 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), OPT_BOOL(0, "negotiate-only", &negotiate_only,
@@ -2658,7 +2658,7 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_tip.nr) +if (negotiate_only && !negotiation_restrict.nr)die (_ ("%s needs one or more %s" ),"--negotiate-only" ,"--negotiation-restrict=*" );
fetch-pack.c +9 -9
@@ -291,21 +291,21 @@ static int next_flush(int stateless_rpc, int count) }static void mark_tips (struct fetch_negotiator *negotiator, -const struct oid_array *negotiation_tips) +const struct oid_array *negotiation_restrict_tips) {struct refs_for_each_ref_options opts = { .flags =REFS_FOR_EACH_INCLUDE_BROKEN , };int i; -if (!negotiation_tips) { +if (!negotiation_restrict_tips) {refs_for_each_ref_ext (get_main_ref_store (the_repository), rev_list_insert_ref_oid, negotiator, &opts);return ; } -for (i =0 ; i < negotiation_tips->nr; i++) -rev_list_insert_ref (negotiator, &negotiation_tips->oid[i]); +for (i =0 ; i < negotiation_restrict_tips->nr; i++) +rev_list_insert_ref (negotiator, &negotiation_restrict_tips->oid[i]);return ; }
@@ -355,7 +355,7 @@ static int find_common(struct fetch_negotiator *negotiator,PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object); fetching =0 ;
@@ -1728,7 +1728,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,else state =FETCH_SEND_REQUEST ; -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object);break ;
@@ -2177,7 +2177,7 @@ static void clear_common_flag(struct oidset *s) } } -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
@@ -2195,13 +2195,13 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,timestamp_t min_generation =GENERATION_NUMBER_INFINITY ;fetch_negotiator_init (the_repository, &negotiator); -mark_tips (&negotiator, negotiation_tips); +mark_tips (&negotiator, negotiation_restrict_tips);packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE |PACKET_READ_DIE_ON_ERR_PACKET ); -oid_array_for_each ((struct oid_array *) negotiation_tips, +oid_array_for_each ((struct oid_array *) negotiation_restrict_tips, add_to_object_array, &nt_object_array);
fetch-pack.h +2 -2
@@ -21,7 +21,7 @@ struct fetch_pack_args {* If not NULL, during packfile negotiation, fetch-pack will send "have" * lines only with these tips and their ancestors. */ -const struct oid_array *negotiation_tips; +const struct oid_array *negotiation_restrict_tips; unsigned deepen_relative:1; unsigned quiet:1;
@@ -89,7 +89,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,* In the capability advertisement that has happened prior to invoking this * function, the "wait-for-done" capability must be present. */ -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
transport-helper.c +1 -1
@@ -754,7 +754,7 @@ static int fetch_refs(struct transport *transport,set_helper_option (transport,"filter" , spec); } -if (data->transport_options.negotiation_tips) +if (data->transport_options.negotiation_restrict_tips)warning ("Ignoring --negotiation-tip because the protocol does not support it." );if (data->fetch)
transport.c +5 -5
@@ -463,7 +463,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.refetch = data->options.refetch; args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; - args.negotiation_tips = data->options.negotiation_tips; + args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -491,7 +491,7 @@ static int fetch_refs_via_pack(struct transport *transport,warning (_ ("server does not support wait-for-done" )); ret =-1 ; }else { - negotiate_using_fetch(data->options.negotiation_tips, + negotiate_using_fetch(data->options.negotiation_restrict_tips, transport->server_options, transport->stateless_rpc, data->fd,
@@ -979,9 +979,9 @@ static int disconnect_git(struct transport *transport)finish_connect (data->conn); } -if (data->options.negotiation_tips) { -oid_array_clear (data->options.negotiation_tips); -free (data->options.negotiation_tips); +if (data->options.negotiation_restrict_tips) { +oid_array_clear (data->options.negotiation_restrict_tips); +free (data->options.negotiation_restrict_tips); }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);
transport.h +2 -2
@@ -40,13 +40,13 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_tips in struct fetch_pack_args. +* negotiation_restrict_tips in struct fetch_pack_args. * * This field is only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ -struct oid_array *negotiation_tips; +struct oid_array *negotiation_restrict_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
Signature · 1 line
gitgitgadget
PATCH v3 4/7 remote: add remote.*.negotiationRestrict config
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tips. Both of these options restrict the set of 'haves' the client can send as part of negotiation.
This was previously not available via a configuration option. Add a new 'remote.<name>.negotiationRestrict' multi-valued config option that updates 'git fetch <name>' to use these restrictions by default.
If the user provides even one --negotiation-restrict argument, then the config is ignored.
An empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
--- Documentation/config/remote.adoc | 19 +++++++++++++++++++ builtin/fetch.c | 21 +++++++++++++++++---- remote.c | 8 ++++++++ remote.h | 1 + t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++ 5 files changed, 71 insertions(+), 4 deletions(-)
Documentation/config/remote.adoc +19 -0
@@ -107,6 +107,25 @@ priority configuration file (e.g. `.git/config` in a repository) to clear the values inherited from a lower priority configuration files (e.g. `$HOME/.gitconfig`). +remote.<name>.negotiationRestrict:: + When negotiating with this remote during `git fetch` and `git push`, + restrict the commits advertised as "have" lines to only those + reachable from refs matching the given patterns. This multi-valued + config option behaves like `--negotiation-restrict` on the command + line. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the +same as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-restrict` +command-line option. If `--negotiation-restrict` (or its synonym +`--negotiation-tip`) is specified on the command line, then the config +values are not used. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
builtin/fetch.c +17 -4
@@ -1601,6 +1601,19 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" ); + }else if (remote->negotiation_restrict.nr) { +struct string_list_item *item; +for_each_string_list_item (item, &remote->negotiation_restrict) +string_list_append (&negotiation_restrict, item->string); +if (transport->smart_options) +add_negotiation_restrict_tips (transport->smart_options); +else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
@@ -2658,10 +2671,6 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_restrict.nr) -die (_ ("%s needs one or more %s" ),"--negotiate-only" , -"--negotiation-restrict=*" ); -if (deepen_relative) {if (deepen_relative <0 )die (_ ("negative depth in --deepen is not supported" ));
@@ -2749,6 +2758,10 @@ int cmd_fetch(int argc,if (!remote)die (_ ("must supply remote when using --negotiate-only" )); gtransport =prepare_transport (remote,1 , &filter_options); +if (!gtransport->smart_options || + !gtransport->smart_options->negotiation_restrict_tips) +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" );if (gtransport->smart_options) { gtransport->smart_options->acked_commits = &acked_commits; }else {
remote.c +8 -0
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_push (&ret->push);refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options); +string_list_init_dup (&ret->negotiation_restrict);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -179,6 +180,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy);FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 ); +string_list_clear (&remote->negotiation_restrict,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -562,6 +564,12 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"serveroption" )) {return parse_transport_option (key, value, &remote->server_options); + }else if (!strcmp (subkey,"negotiationrestrict" )) { +/* reset list on empty value. */ +if (!value || !*value) +string_list_clear (&remote->negotiation_restrict,0 ); +else +string_list_append (&remote->negotiation_restrict, value); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -117,6 +117,7 @@ struct remote {char *http_proxy_authmethod;struct string_list server_options; +struct string_list negotiation_restrict;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +26 -0
@@ -1485,6 +1485,32 @@ test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' check_negotiation_tip ' +test_expect_success 'remote.<name>.negotiationRestrict used as default' ' + setup_negotiation_tip server server 0 && + + # test the reset of the list on an empty value + git -C client config --add remote.origin.negotiationRestrict alpha_2 && + git -C client config --add remote.origin.negotiationRestrict "" && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success 'CLI --negotiation-restrict overrides remote config' ' + setup_negotiation_tip server server 0 && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + ALPHA_1=$(git -C client rev-parse alpha_1) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + origin alpha_s beta_s && + test_grep "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep ! "fetch> have $BETA_1" trace +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
Signature · 1 line
gitgitgadget
PATCH v3 5/7 fetch: add --negotiation-include option for negotiation
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new --negotiation-include option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
This is useful when the repository has a large number of references, so the normal negotiation algorithm truncates the list. This is especially important in repositories with long parallel commit histories. For example, a repo could have a 'dev' branch for development and a 'release' branch for released versions. If the 'dev' branch isn't selected for negotiation, then it's not a big deal because there are many in-progress development branches with a shared history. However, if 'release' is not selected for negotiation, then the server may think that this is the first time the client has asked for that reference, causing a full download of its parallel commit history (and any extra data that may be unique to that branch). This is based on a real example where certain fetches would grow to 60+ GB when a release branch updated.
This option is a complement to --negotiation-restrict, which reduces the negotiation ref set to a specific list. In the earlier example, using --negotiation-restrict to focus the negotiation to 'dev' and 'release' would avoid those problematic downloads, but would still not allow advertising potentially-relevant user brances. In this way, the 'include' version solves the problem I mention while allowing negotiation to pick other references opportunistically. The two options can also be combined to allow the best of both worlds.
The argument may be an exact ref name or a glob pattern. Non-existent refs are silently ignored. This behavior is also updated in the ref matching logic for the related --negotiation-restrict option to match.
The implementation outputs the requested objects as haves before the negotiation algorithm kicks in and performs a priority-queue walk from the tip commits. In order to avoid duplicates, we mark the requested objects as COMMON so they (and their descendants) are not output by the negotiator. The negotiator still outputs at least one have before a round is flushed, when the server could ACK to stop the negotiation.
Also add --negotiation-include to 'git pull' passthrough options.
--- Documentation/fetch-options.adoc | 19 ++++++ builtin/fetch.c | 16 ++++- builtin/pull.c | 3 + fetch-pack.c | 112 +++++++++++++++++++++++++++++-- fetch-pack.h | 10 ++- t/t5510-fetch.sh | 66 ++++++++++++++++++ transport.c | 4 +- transport.h | 6 ++ 8 files changed, 227 insertions(+), 9 deletions(-)
Documentation/fetch-options.adoc +19 -0
@@ -73,6 +73,25 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate` configuration variables documented in linkgit:git-config[1], and the `--negotiate-only` option below. +`--negotiation-include=<revision>`:: + Ensure that the given ref tip is always sent as a "have" line + during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-restrict` restricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++ +This option may be specified more than once; if so, each ref is sent +unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax +is the same as for `--negotiation-restrict`. ++ +If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-include`. + `--negotiate-only`:: Do not fetch anything from the server, and instead print the ancestors of the provided `--negotiation-tip=` arguments,
builtin/fetch.c +15 -1
@@ -99,6 +99,7 @@ static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ;static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_include =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;
@@ -1547,10 +1548,14 @@ static void add_negotiation_restrict_tips(struct git_transport_options *smart_opint old_nr;if (!has_glob_specials (s)) {struct object_id oid; + +/* Ignore missing reference. */ if (repo_get_oid (the_repository, s, &oid)) -die (_ ("%s is not a valid object" ), s); +continue ; +/* Fail on missing object pointed by ref. */ if (!odb_has_object (the_repository->objects, &oid,0 ))die (_ ("the object %s does not exist" ), s); +oid_array_append (oids, &oid);continue ; }
@@ -1615,6 +1620,13 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,strbuf_release (&config_name); } } +if (negotiation_include.nr) { +if (transport->smart_options) + transport->smart_options->negotiation_include = &negotiation_include; +else +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-include" ); + }return transport; }
@@ -2582,6 +2594,8 @@ int cmd_fetch(int argc,OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), +OPT_STRING_LIST (0 ,"negotiation-include" , &negotiation_include,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" )),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
builtin/pull.c +3 -0
@@ -1002,6 +1002,9 @@ int cmd_pull(int argc,OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-include" , &opt_fetch,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
fetch-pack.c +106 -6
218 lines not on this page · raw message
fetch-pack.h +9 -1
@@ -23,6 +23,13 @@ struct fetch_pack_args {*/ const struct oid_array *negotiation_restrict_tips; +/* +* If non-empty, ref patterns whose tips should always be sent +* as "have" lines during negotiation, regardless of what the +* negotiation algorithm selects. +*/ +const struct string_list *negotiation_include; + unsigned deepen_relative:1; unsigned quiet:1; unsigned keep_pack:1;
@@ -93,7 +100,8 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,const struct string_list *server_options, int stateless_rpc, int fd[], - struct oidset *acked_commits); + struct oidset *acked_commits, + const struct string_list *negotiation_include);/* * Print an appropriate error message for each sought ref that wasn't
t/t5510-fetch.sh +66 -0
72 lines not on this page · raw message
transport.c +3 -1
@@ -464,6 +464,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; + args.negotiation_include = data->options.negotiation_include; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -495,7 +496,8 @@ static int fetch_refs_via_pack(struct transport *transport, transport->server_options, transport->stateless_rpc, data->fd, - data->options.acked_commits); + data->options.acked_commits, + data->options.negotiation_include); ret =0 ; } goto cleanup;
transport.h +6 -0
@@ -48,6 +48,12 @@ struct git_transport_options {*/ struct oid_array *negotiation_restrict_tips; +/* +* If non-empty, ref patterns whose tips should always be sent +* as "have" lines during negotiation. +*/ +const struct string_list *negotiation_include; +/* * If allocated, whenever transport_fetch_refs() is called, add known * common commits to this oidset instead of fetching any packfiles.
Signature · 1 line
gitgitgadget
PATCH v3 6/7 remote: add remote.*.negotiationInclude config
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new 'remote.<name>.negotiationInclude' multi-valued config option that provides default values for --negotiation-include when no --negotiation-include arguments are specified over the command line. This is a mirror of how 'remote.<name>.negotiationRestrict' specifies defaults for the --negotiation-restrict arguments.
Each value is either an exact ref name or a glob pattern whose tips should always be sent as 'have' lines during negotiation. The config values are resolved through the same resolve_negotiation_include() codepath as the CLI options.
This option is additive with the normal negotiation process: the negotiation algorithm still runs and advertises its own selected commits, but the refs matching the config are sent unconditionally on top of those heuristically selected commits.
Similar to the negotiationRestrict config, an empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
--- Documentation/config/remote.adoc | 27 ++++++++++++++++++ Documentation/fetch-options.adoc | 4 +++ builtin/fetch.c | 10 +++++++ remote.c | 8 ++++++ remote.h | 1 + t/t5510-fetch.sh | 49 ++++++++++++++++++++++++++++++++ 6 files changed, 99 insertions(+)
Documentation/config/remote.adoc +27 -0
@@ -126,6 +126,33 @@ values are not used. Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios. +remote.<name>.negotiationInclude:: + When negotiating with this remote during `git fetch` and `git push`, + the client advertises a list of commits that exist locally. In + repos with many references, this list of "haves" can be truncated. + Depending on data shape, dropping certain references may be + expensive. This multi-valued config option specifies ref patterns + whose tips should always be sent as "have" commits during fetch + negotiation with this remote. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the same +as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-include` +command-line option. If `--negotiation-include` is specified on the +command line, then the config values are not used. ++ +This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits, +but the refs matching `remote.<name>.negotiationInclude` are sent +unconditionally on top of those heuristically selected commits. This +option is also used during push negotiation when `push.negotiate` is +enabled. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
Documentation/fetch-options.adoc +4 -0
@@ -91,6 +91,10 @@ is the same as for `--negotiation-restrict`. If `--negotiation-restrict` is used, the have set is first restricted by that option and then increased to include the tips specified by `--negotiation-include`. ++ +If this option is not specified on the command line, then any +`remote.<name>.negotiationInclude` config values for the current remote +are used instead. `--negotiate-only`:: Do not fetch anything from the server, and instead print the
builtin/fetch.c +10 -0
@@ -1626,6 +1626,16 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-include" ); + }else if (remote->negotiation_include.nr) { +if (transport->smart_options) { + transport->smart_options->negotiation_include = &remote->negotiation_include; + }else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationInclude" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
remote.c +8 -0
@@ -153,6 +153,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options);string_list_init_dup (&ret->negotiation_restrict); +string_list_init_dup (&ret->negotiation_include);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -181,6 +182,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 );string_list_clear (&remote->negotiation_restrict,0 ); +string_list_clear (&remote->negotiation_include,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -570,6 +572,12 @@ static int handle_config(const char *key, const char *value,string_list_clear (&remote->negotiation_restrict,0 ); else string_list_append(&remote->negotiation_restrict, value); + }else if (!strcmp (subkey,"negotiationinclude" )) { +/* reset list on empty value. */ +if (!value || !*value) +string_list_clear (&remote->negotiation_include,0 ); +else +string_list_append (&remote->negotiation_include, value); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -118,6 +118,7 @@ struct remote {struct string_list server_options;struct string_list negotiation_restrict; +struct string_list negotiation_include;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +49 -0
55 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v3 7/7 send-pack: pass negotiation config in push
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
When push.negotiate is enabled, 'git push' spawns a child 'git fetch --negotiate-only' process to find common commits. Pass --negotiation-include and --negotiation-restrict options from the 'remote.<name>.negotiationInclude' and 'remote.<name>.negotiationRestrict' config keys to this child process.
When negotiationRestrict is configured, it replaces the default behavior of using all remote refs as negotiation tips. This allows the user to control which local refs are used for push negotiation.
When negotiationInclude is configured, the specified ref patterns are passed as --negotiation-include to ensure their tips are always sent as 'have' lines during push negotiation.
This change also updates the use of --negotiation-tip into --negotiation-restrict now that the new synonym exists.
--- send-pack.c | 39 +++++++++++++++++++++++++++++++-------- send-pack.h | 2 ++ t/t5516-fetch-push.sh | 30 ++++++++++++++++++++++++++++++ transport.c | 2 ++ 4 files changed, 65 insertions(+), 8 deletions(-)
send-pack.c +31 -8
@@ -433,28 +433,48 @@ static void reject_invalid_nonce(const char *nonce, int len)static void get_commons_through_negotiation (struct repository *r,const char *url, +const struct string_list *negotiation_include, +const struct string_list *negotiation_restrict,const struct ref *remote_refs,struct oid_array *commons) {struct child_process child =CHILD_PROCESS_INIT ;const struct ref *ref;int len = r->hash_algo->hexsz +1 ;/* hash + NL */ -int nr_negotiation_tip =0 ; +int nr_negotiation =0 ; child.git_cmd =1 ; child.no_stdin =1 ; child.out =-1 ;strvec_pushl (&child.args,"fetch" ,"--negotiate-only" ,NULL ); -for (ref = remote_refs; ref; ref = ref->next) { -if (!is_null_oid (&ref->new_oid)) { -strvec_pushf (&child.args,"--negotiation-tip=%s" , -oid_to_hex (&ref->new_oid)); - nr_negotiation_tip++; + +if (negotiation_restrict && negotiation_restrict->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_restrict) +strvec_pushf (&child.args,"--negotiation-restrict=%s" , + item->string); + nr_negotiation = negotiation_restrict->nr; + }else { +for (ref = remote_refs; ref; ref = ref->next) { +if (!is_null_oid (&ref->new_oid)) { +strvec_pushf (&child.args,"--negotiation-restrict=%s" , +oid_to_hex (&ref->new_oid)); + nr_negotiation++; + } } } + +if (negotiation_include && negotiation_include->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_include) +strvec_pushf (&child.args,"--negotiation-include=%s" , + item->string); + nr_negotiation += negotiation_include->nr; + } +strvec_push (&child.args, url); -if (!nr_negotiation_tip) { +if (!nr_negotiation) {child_process_clear (&child);return ; }
@@ -528,7 +548,10 @@ int send_pack(struct repository *r,repo_config_get_bool (r,"push.negotiate" , &push_negotiate);if (push_negotiate) {trace2_region_enter ("send_pack" ,"push_negotiate" , r); -get_commons_through_negotiation (r, args->url, remote_refs, &commons); +get_commons_through_negotiation (r, args->url, + args->negotiation_include, + args->negotiation_restrict, + remote_refs, &commons);trace2_region_leave ("send_pack" ,"push_negotiate" , r); }
send-pack.h +2 -0
@@ -18,6 +18,8 @@ struct repository;struct send_pack_args {const char *url; +const struct string_list *negotiation_include; +const struct string_list *negotiation_restrict;unsigned verbose:1 , quiet:1 , porcelain:1,
t/t5516-fetch-push.sh +30 -0
@@ -254,6 +254,36 @@ test_expect_success 'push with negotiation does not attempt to fetch submodules' ! grep "Fetching submodule" err ' +test_expect_success 'push with negotiation and remote.<name>.negotiationInclude' ' + test_when_finished rm -rf negotiation_include && + mk_empty negotiation_include && + git push negotiation_include $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_include unrelated_commit && + git -C negotiation_include config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_include.negotiationInclude=refs/heads/main \ + push negotiation_include refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + +test_expect_success 'push with negotiation and remote.<name>.negotiationRestrict' ' + test_when_finished rm -rf negotiation_restrict && + mk_empty negotiation_restrict && + git push negotiation_restrict $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_restrict unrelated_commit && + git -C negotiation_restrict config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_restrict.negotiationRestrict=refs/heads/main \ + push negotiation_restrict refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + test_expect_success 'push without wildcard' ' mk_empty testrepo &&
transport.c +2 -0
@@ -921,6 +921,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re args.atomic = !!(flags &TRANSPORT_PUSH_ATOMIC ); args.push_options = transport->push_options; args.url = transport->url; + args.negotiation_include = &transport->remote->negotiation_include; + args.negotiation_restrict = &transport->remote->negotiation_restrict;if (flags &TRANSPORT_PUSH_CERT_ALWAYS ) args.push_cert =SEND_PACK_PUSH_CERT_ALWAYS ;
Signature · 1 line
gitgitgadget
Re: [PATCH v3 1/7] t5516: fix test order flakiness
In reply to Derrick Stolee via GitGitGadgetQuoted 18 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
The 'fetch follows tags by default' test sorts using 'sort -k 4', but for-each-ref output only has 3 columns. This relies on sort treating records with fewer fields as having an empty fourth field, which may produce unstable results depending on locale. Use 'sort -k 3' to match the actual number of columns in the output.
Signed-off-by Derrick Stolee <[email protected]>--- t/t5516-fetch-push.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)t/t5516-fetch-push.sh +0
-0
t/t5516-fetch-push.sh test_expect_success 'fetch follows tags by default' ' · line 1355
Quoted 8 lines
t/t5516-fetch-push.sh +1
-1@@ -1349,7 +1349,7 @@ test_expect_success 'fetch follows tags by default' ' git for-each-ref >tmp1 && sed -n "p; s|refs/heads/main$|refs/remotes/origin/main|p" tmp1 | sed -n "p; s|refs/heads/main$|refs/remotes/origin/HEAD|p" | - sort -k 4 >../expect + sort -k 3 >../expect ) && test_when_finished "rm -rf dst" && git init dst &&
Makes sense. Looks like 3f763ddf28 ("fetch: set remote/HEAD if it does not exist") originally changed it from -k3 to -k4 by mistake.
Thanks,
Matthew
Re: [PATCH v3 2/7] fetch: add --negotiation-restrict option
In reply to Derrick Stolee via GitGitGadgetQuoted 14 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
The --negotiation-tip option to 'git fetch' and 'git pull' allows users to specify that they want to focus negotiation on a small set of references. This is a _restriction_ on the negotiation set, helping to focus the negotiation when the ref count is high. However, it doesn't allow for the ability to opportunistically select references beyond that list.
This subtle detail that this is a 'maximum set' and not a 'minimum set' is not immediately clear from the option name. This makes it more complicated to add a new option that provides the complementary behavior of a minimum set.
Quoted 5 lines
For now, create a new synonym option, --negotiation-restrict, that behaves identically to --negotiation-tip. Update the documentation to make it clear that this new name is the preferred option, but we keep the old name for compatibility. Mark --negotiation-tip as an alias of the new, preferred option.
This motivation reads well. Preparing the new naming convention before introducing the new, complementary option, is the right order to do this in, IMO.
Quoted 3 lines
Update a few warning messages with the new option, but also make them translatable with the option name inserted by formatting. At least one of these messages will be reused later for a new option.
Nice extra win!
Quoted 22 lines
Signed-off-by Derrick Stolee <[email protected]>--- Documentation/fetch-options.adoc | 4 ++++ builtin/fetch.c | 13 ++++++++----- builtin/pull.c | 3 +++ t/t5510-fetch.sh | 25 +++++++++++++++++++++++++ t/t5702-protocol-v2.sh | 4 ++-- 5 files changed, 42 insertions(+), 7 deletions(-)Documentation/fetch-options.adoc +1
-0@@ -49,6 +49,7 @@ the current repository has the same history as the source repository. `.git/shallow`. This option updates `.git/shallow` and accepts such refs. +`--negotiation-restrict=(<commit>|<glob>)`:: `--negotiation-tip=(<commit>|<glob>)`:: By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt to
Documentation/fetch-options.adoc the current repository has the same history as the source repository. · line 67
Quoted 9 lines
Documentation/fetch-options.adoc +3
-0@@ -58,6 +59,9 @@ the current repository has the same history as the source repository. local ref is likely to have commits in common with the upstream ref being fetched. + +`--negotiation-restrict` is the preferred name for this option; +`--negotiation-tip` is accepted as a synonym. ++ This option may be specified more than once; if so, Git will report commits reachable from any of the given commits. +
By my eyes it looks like two other references to the old name remain and could also be updated for consistency (since --negotiation-restrict is now the preferred name):
1. Documentation/fetch-options.adoc, under --negotiate-only:
"ancestors of the provided --negotiation-tip= arguments"
2. Documentation/config/fetch.adoc:
"See also the --negotiate-only and --negotiation-tip options"
Of course the old name will still work, so this is more a nit-pick :-)
Quoted 5 lines
builtin/fetch.c +0
-0
builtin/fetch.c static void add_negotiation_tips(struct git_transport_options *smart_options) · line 1565
Quoted 10 lines
builtin/fetch.c +2
-2@@ -1558,8 +1558,8 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)refs_for_each_ref_ext (get_main_ref_store (the_repository), add_oid, oids, &opts);if (old_nr == oids->nr) -warning ("ignoring --negotiation-tip=%s because it does not match any refs" , - s); +warning (_ ("ignoring %s=%s because it does not match any refs" ), +"--negotiation-restrict" , s); } smart_options->negotiation_tips = oids; }
Keeping the option name out of the translation string prevents accidental translation of a fixed symbol - good.
builtin/fetch.c static struct transport *prepare_transport(struct remote *remote, int deepen, · line 1606
Quoted 10 lines
builtin/fetch.c +2
-1@@ -1599,7 +1599,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,if (transport->smart_options)add_negotiation_tips (transport->smart_options);else -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-restrict" ); }return transport; }
Same as above - good.
builtin/fetch.c int cmd_fetch(int argc, · line 2574
Quoted 11 lines
builtin/fetch.c +2
-1@@ -2565,8 +2566,9 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
Good. Makes the --negotiate-restrict name the primary and the 'tip' the alias, matching the docs' preference.
Keeping the variable named negotiate_tip helps reduce churn in this patch (and has no outwardly visible impact anyway). I see a future patch renames the variable - nice choice for reviewability.
builtin/fetch.c int cmd_fetch(int argc, · line 2666
Quoted 10 lines
builtin/fetch.c +2
-1@@ -2657,7 +2659,8 @@ int cmd_fetch(int argc, }if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-tip=*" )); +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" );if (deepen_relative) {if (deepen_relative <0 )
Much love for i18n!
Quoted 5 lines
builtin/pull.c +0
-0
builtin/pull.c int cmd_pull(int argc, · line 1007
Quoted 9 lines
builtin/pull.c +3
-0@@ -999,6 +999,9 @@ int cmd_pull(int argc,OPT_PASSTHRU_ARGV (0 ,"negotiation-tip" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ), +N_ ("report that we have only objects reachable from this object" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
It's a shame we don't have a nice way to combine the OPT_ALIAS and OPT_PASSTHRU_ARGV functionality, but it's only a small duplication cost of the repeated definition.
Quoted 47 lines
t/t5510-fetch.sh +25
-0@@ -1460,6 +1460,31 @@ EOF test_cmp fatal-expect fatal-actual ' +test_expect_success '--negotiation-restrict limits "have" lines sent' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 --negotiation-restrict=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict understands globs' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=*_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-tip=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (t/t5702-protocol-v2.sh +1
-1@@ -869,14 +869,14 @@ setup_negotiate_only () { test_commit -C client three } -test_expect_success 'usage: --negotiate-only without --negotiation-tip' ' +test_expect_success 'usage: --negotiate-only without --negotiation-restrict' ' SERVER="server" &&
t/t5702-protocol-v2.sh setup_negotiate_only () { · line 882
Quoted 10 lines
t/t5702-protocol-v2.sh +1
-1URI="file://$(pwd)/server" && setup_negotiate_only "$SERVER" "$URI" && cat >err.expect <<-\EOF && - fatal: --negotiate-only needs one or more --negotiation-tip=* + fatal: --negotiate-only needs one or more --negotiation-restrict=* EOF test_must_fail git -c protocol.version=2 -C client fetch \
Looks like this test is the only place asserting the '--negotiate-tip' string literal in the tree - good, no others to update.
Except the two doc cross-references above (nits) this looks good to me.
Thanks,
Matthew
Re: [PATCH v3 3/7] transport: rename negotiation_tips
In reply to Derrick Stolee via GitGitGadgetQuoted 7 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
The previous change added the --negotiation-restrict synonym for the --negotiation-tips option for 'git fetch'. In anticipation of adding a new option that behaves similarly but with distinct changes to its behavior, rename the internal representation of this data from '
negotiation_tips' to 'negotiation_restrict_tips'.
Nitpick: s/tips/tip/ .. no trailing s for either the option name, nor the (old) variable name. The function names do use the plural however.
Quoted 22 lines
The 'tips' part is kept because this is an
oid_arrayin the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data.Also update the
string_listused to store the inputs from command-line options.Signed-off-by Derrick Stolee <[email protected]>--- builtin/fetch.c | 18 +++++++++--------- fetch-pack.c | 18 +++++++++--------- fetch-pack.h | 4 ++-- transport-helper.c | 2 +- transport.c | 10 +++++----- transport.h | 4 ++-- 6 files changed, 28 insertions(+), 28 deletions(-)builtin/fetch.c +0
-0
builtin/fetch.c static struct transport *gtransport; · line 101
Quoted 5 lines
builtin/fetch.c +1
-1@@ -98,7 +98,7 @@ static struct transport *gtransport;static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ; -static struct string_list negotiation_tip =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ;
Good - now mirrors the new, preferred, option name.
Quoted 9 lines
builtin/fetch.c +1
-1struct fetch_config {enum display_format display_format;@@ -1534,13 +1534,13 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_tips (struct git_transport_options *smart_options) +static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) {
builtin/fetch.c static int add_oid(const struct reference *ref, void *cb_data) · line 1546
Quoted 10 lines
builtin/fetch.c +2
-2struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i; -for (i =0 ; i < negotiation_tip.nr; i++) { -const char *s = negotiation_tip.items[i].string; +for (i =0 ; i < negotiation_restrict.nr; i++) { +const char *s = negotiation_restrict.items[i].string;struct refs_for_each_ref_options opts = { .pattern = s, };
All callers and references are renamed to match consistency. Good.
Quoted 126 lines
builtin/fetch.c +5
-5@@ -1561,7 +1561,7 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)warning (_ ("ignoring %s=%s because it does not match any refs" ),"--negotiation-restrict" , s); } - smart_options->negotiation_tips = oids; + smart_options->negotiation_restrict_tips = oids; } static struct transport *prepare_transport(struct remote *remote, int deepen,@@ -1595,9 +1595,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,set_option (transport,TRANS_OPT_LIST_OBJECTS_FILTER , spec);set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" ); } -if (negotiation_tip.nr) { +if (negotiation_restrict.nr) {if (transport->smart_options) -add_negotiation_tips (transport->smart_options); +add_negotiation_restrict_tips (transport->smart_options);else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );@@ -2566,7 +2566,7 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), OPT_BOOL(0, "negotiate-only", &negotiate_only,@@ -2658,7 +2658,7 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_tip.nr) +if (negotiate_only && !negotiation_restrict.nr)die (_ ("%s needs one or more %s" ),"--negotiate-only" ,"--negotiation-restrict=*" );fetch-pack.c +9
-9@@ -291,21 +291,21 @@ static int next_flush(int stateless_rpc, int count) }static void mark_tips (struct fetch_negotiator *negotiator, -const struct oid_array *negotiation_tips) +const struct oid_array *negotiation_restrict_tips) {struct refs_for_each_ref_options opts = { .flags =REFS_FOR_EACH_INCLUDE_BROKEN , };int i; -if (!negotiation_tips) { +if (!negotiation_restrict_tips) {refs_for_each_ref_ext (get_main_ref_store (the_repository), rev_list_insert_ref_oid, negotiator, &opts);return ; } -for (i =0 ; i < negotiation_tips->nr; i++) -rev_list_insert_ref (negotiator, &negotiation_tips->oid[i]); +for (i =0 ; i < negotiation_restrict_tips->nr; i++) +rev_list_insert_ref (negotiator, &negotiation_restrict_tips->oid[i]);return ; }@@ -355,7 +355,7 @@ static int find_common(struct fetch_negotiator *negotiator,PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object); fetching =0 ;@@ -1728,7 +1728,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,else state =FETCH_SEND_REQUEST ; -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object);break ;@@ -2177,7 +2177,7 @@ static void clear_common_flag(struct oidset *s) } } -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],@@ -2195,13 +2195,13 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,timestamp_t min_generation =GENERATION_NUMBER_INFINITY ;fetch_negotiator_init (the_repository, &negotiator); -mark_tips (&negotiator, negotiation_tips); +mark_tips (&negotiator, negotiation_restrict_tips);packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE |PACKET_READ_DIE_ON_ERR_PACKET ); -oid_array_for_each ((struct oid_array *) negotiation_tips, +oid_array_for_each ((struct oid_array *) negotiation_restrict_tips, add_to_object_array, &nt_object_array);fetch-pack.h +1
-1@@ -21,7 +21,7 @@ struct fetch_pack_args {* If not NULL, during packfile negotiation, fetch-pack will send "have" * lines only with these tips and their ancestors. */ -const struct oid_array *negotiation_tips; +const struct oid_array *negotiation_restrict_tips; unsigned deepen_relative:1; unsigned quiet:1;
fetch-pack.h struct ref *fetch_pack(struct fetch_pack_args *args, · line 95
Quoted 8 lines
fetch-pack.h +1
-1@@ -89,7 +89,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,* In the capability advertisement that has happened prior to invoking this * function, the "wait-for-done" capability must be present. */ -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
LGTM up to here.
transport-helper.c static int fetch_refs(struct transport *transport, · line 760
Quoted 13 lines
transport-helper.c +1
-1@@ -754,7 +754,7 @@ static int fetch_refs(struct transport *transport,set_helper_option (transport,"filter" , spec); } -if (data->transport_options.negotiation_tips) +if (data->transport_options.negotiation_restrict_tips)warning ("Ignoring --negotiation-tip because the protocol does not support it." );if (data->fetch)
Oh! Looks like a place was missed when renaming the preferred option name in strings. It probably makes sense to do this rename in this patch (rather than in patch 1) since we're already updating the struct field
name here anyway, but up to you.
Also do we also want to make it translatable like the others?
Quoted 45 lines
transport.c +5
-5@@ -463,7 +463,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.refetch = data->options.refetch; args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; - args.negotiation_tips = data->options.negotiation_tips; + args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {@@ -491,7 +491,7 @@ static int fetch_refs_via_pack(struct transport *transport,warning (_ ("server does not support wait-for-done" )); ret =-1 ; }else { - negotiate_using_fetch(data->options.negotiation_tips, + negotiate_using_fetch(data->options.negotiation_restrict_tips, transport->server_options, transport->stateless_rpc, data->fd,@@ -979,9 +979,9 @@ static int disconnect_git(struct transport *transport)finish_connect (data->conn); } -if (data->options.negotiation_tips) { -oid_array_clear (data->options.negotiation_tips); -free (data->options.negotiation_tips); +if (data->options.negotiation_restrict_tips) { +oid_array_clear (data->options.negotiation_restrict_tips); +free (data->options.negotiation_restrict_tips); }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);transport.h +1
-1@@ -40,13 +40,13 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_tips in struct fetch_pack_args. +* negotiation_restrict_tips in struct fetch_pack_args.
transport.h struct git_transport_options { · line 52
Quoted 10 lines
transport.h +1
-1* * This field is only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ -struct oid_array *negotiation_tips; +struct oid_array *negotiation_restrict_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
Just a missing string rename, and a nitpick typo in the commit message, but otherwise this patch looks functionally correct.
Aside: I just noticed another '--negotiation-tip' instance in the get_commons_through_negotiation function in send-pack.c. It still uses the 'tip' option name when forming the shell cmdline.
Thanks,
Matthew
Re: [PATCH v3 4/7] remote: add remote.*.negotiationRestrict config
In reply to Derrick Stolee via GitGitGadgetQuoted 6 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee<[email protected]>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tips. Both of these options restrict the set of 'haves' the client can send as part of negotiation.
s/tips/tip/ as per the previous patch comments. Not important either way.
Quoted 39 lines
This was previously not available via a configuration option. Add a new 'remote.<name>.negotiationRestrict' multi-valued config option that updates 'git fetch <name>' to use these restrictions by default.
If the user provides even one --negotiation-restrict argument, then the config is ignored.
An empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
Signed-off-by Derrick Stolee<[email protected]>--- Documentation/config/remote.adoc | 19 +++++++++++++++++++ builtin/fetch.c | 21 +++++++++++++++++---- remote.c | 8 ++++++++ remote.h | 1 + t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++ 5 files changed, 71 insertions(+), 4 deletions(-)Documentation/config/remote.adoc +12
-0@@ -107,6 +107,25 @@ priority configuration file (e.g. `.git/config` in a repository) to clear the values inherited from a lower priority configuration files (e.g. `$HOME/.gitconfig`). +remote.<name>.negotiationRestrict:: + When negotiating with this remote during `git fetch` and `git push`, + restrict the commits advertised as "have" lines to only those + reachable from refs matching the given patterns. This multi-valued + config option behaves like `--negotiation-restrict` on the command + line. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the +same as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-restrict`
Documentation/config/remote.adoc priority configuration file (e.g. `.git/config` in a repository) to clear · line 131
Quoted 10 lines
Documentation/config/remote.adoc +7
-0+command-line option. If `--negotiation-restrict` (or its synonym +`--negotiation-tip`) is specified on the command line, then the config +values are not used. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
You say "during git fetch and git push", but does push actually honour the new config?
When the push.negotiate config is on thenget_commons_through_negotiation() from send-pack.c shells out togit fetch --negotiate-only with one --negotiation-tip=<oid> arg per
ref being pushed, then the URL. This means the CLI restrict list is
always non-empty in the subprocess so in prepare_transport() (in the
below hunk) the if (negotiation_restrict.nr) arm is always taken and the new else if (remote->negotiation_restrict.nr) arm is never taken.
BUT.. reading ahead I see that patch 7 actually wires up negotiation config for push - so my commentary here will be moot! Do we want to drop the "and git push" part from this until patch 7, when it is wired up appropriately?
One other suggestion: perhaps we should clarify that push.negotiate needs to be set for remote.<name>.negotiationRestrict to be honoured during pushes?
Quoted 14 lines
builtin/fetch.c +6
-0@@ -1601,6 +1601,19 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" ); + }else if (remote->negotiation_restrict.nr) { +struct string_list_item *item; +for_each_string_list_item (item, &remote->negotiation_restrict) +string_list_append (&negotiation_restrict, item->string); +if (transport->smart_options) +add_negotiation_restrict_tips (transport->smart_options);
builtin/fetch.c static struct transport *prepare_transport(struct remote *remote, int deepen, · line 1619
Quoted 10 lines
builtin/fetch.c +7
-0+else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
See above - this new arm is not reachable on the push.negotiate=true path until patch 7 wires send-pack up.
Quoted 12 lines
builtin/fetch.c +0
-4@@ -2658,10 +2671,6 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_restrict.nr) -die (_ ("%s needs one or more %s" ),"--negotiate-only" , -"--negotiation-restrict=*" ); -if (deepen_relative) {if (deepen_relative <0 )die (_ ("negative depth in --deepen is not supported" ));
builtin/fetch.c int cmd_fetch(int argc, · line 2767
Quoted 10 lines
builtin/fetch.c +4
-0@@ -2749,6 +2758,10 @@ int cmd_fetch(int argc,if (!remote)die (_ ("must supply remote when using --negotiate-only" )); gtransport =prepare_transport (remote,1 , &filter_options); +if (!gtransport->smart_options || + !gtransport->smart_options->negotiation_restrict_tips) +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" );if (gtransport->smart_options) { gtransport->smart_options->acked_commits = &acked_commits; }else {
This new condition fires whenever gtransport->smart_options is NULL, i.e. the transport doesn't support smart options. Before this case was handled three lines after this hunk by:
} else {
warning(_("protocol does not support --negotiate-only, exiting"));
result = 1;
trace2_region_leave("fetch", "negotiate-only", the_repository);
goto cleanup;
}What happens now if a user runs --negotiate-only against a non-smart transport is they see an odd message:
fatal: --negotiate-only needs one or more --negotiation-restrict=*
..but they may have specified --negotiation-restrict options.
Do we instead want &&?
if (gtransport->smart_options &&
!gtransport->smart_options->negotiation_restrict_tips)
die(_("%s needs one or more %s"), "--negotiate-only",
"--negotiation-restrict=*");Quoted 23 lines
remote.c +2
-0@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_push (&ret->push);refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options); +string_list_init_dup (&ret->negotiation_restrict);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);@@ -179,6 +180,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy);FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 ); +string_list_clear (&remote->negotiation_restrict,0 ); }static void add_merge (struct branch *branch,const char *name)@@ -562,6 +564,12 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"serveroption" )) { return parse_transport_option(key, value,
remote.c static int handle_config(const char *key, const char *value, · line 575
Quoted 10 lines
remote.c +6
-0&remote->server_options); + }else if (!strcmp (subkey,"negotiationrestrict" )) { +/* reset list on empty value. */ +if (!value || !*value) +string_list_clear (&remote->negotiation_restrict,0 ); +else +string_list_append (&remote->negotiation_restrict, value); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
Here we use the 'empty value means reset the list' pattern, but I notice that the parse_transport_option() function already supports this reset pattern (and used by serveroption above), with a small difference:
if (!value)
return config_error_nonbool(var);
if (!*value)
string_list_clear(transport_options, 0);So NULL is an error, but empty string is 'reset'. Is it worth being consistent with other options that use parse_transport_options?
Quoted 41 lines
remote.h +1
-0@@ -117,6 +117,7 @@ struct remote {char *http_proxy_authmethod;struct string_list server_options; +struct string_list negotiation_restrict;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;t/t5510-fetch.sh +20
-0@@ -1485,6 +1485,32 @@ test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' check_negotiation_tip ' +test_expect_success 'remote.<name>.negotiationRestrict used as default' ' + setup_negotiation_tip server server 0 && + + # test the reset of the list on an empty value + git -C client config --add remote.origin.negotiationRestrict alpha_2 && + git -C client config --add remote.origin.negotiationRestrict "" && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success 'CLI --negotiation-restrict overrides remote config' ' + setup_negotiation_tip server server 0 && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + ALPHA_1=$(git -C client rev-parse alpha_1) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \
t/t5510-fetch.sh test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed'
Quoted 10 lines
t/t5510-fetch.sh +6
-1+ origin alpha_s beta_s && + test_grep "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep ! "fetch> have $BETA_1" trace +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && ( -- gitgitgadget
General shape of this patch is good. The main thing that tripped me up
when reading this patch is the doc claim about push, which only becomes true after patch 7 lands.
Thanks,
Matthew
Re: [PATCH v3 2/7] fetch: add --negotiation-restrict option
In reply to Matthew John CheethamQuoted 29 lines · Matthew John Cheetham
Quoted 15 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
Documentation/fetch-options.adoc +0
-0@@ -49,6 +49,7 @@ the current repository has the same history as the sourcerepository.
.git/shallow. This option updates.git/shallowand accepts such refs.
+--negotiation-restrict=(<commit>|<glob>)::
--negotiation-tip=(<commit>|<glob>)::
By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt toDocumentation/fetch-options.adoc +0
-0@@ -58,6 +59,9 @@ the current repository has the same history as the sourcerepository.
local ref is likely to have commits in common with theQuoted 8 lines
upstream ref being fetched.
+
+--negotiation-restrictis the preferred name for this option; +--negotiation-tipis accepted as a synonym.
++
This option may be specified more than once; if so, Git will report commits reachable from any of the given commits.
+By my eyes it looks like two other references to the old name remain and could also be updated for consistency (since --negotiation-restrict is now the preferred name):
Quoted 7 lines
1.
Documentation/fetch-options.adoc, under--negotiate-only:
"ancestors of the provided--negotiation-tip=arguments"2.
Documentation/config/fetch.adoc:
"See also the--negotiate-onlyand--negotiation-tipoptions"Of course the old name will still work, so this is more a nit-pick :-)
Thanks for catching these! I will make the correct updates in the next version.
Quoted 10 lines
Quoted 10 lines
builtin/pull.c +2
-0@@ -999,6 +999,9 @@ int cmd_pull(int argc, OPT_PASSTHRU_ARGV(0, "negotiation-tip", &opt_fetch, N_("revision"), N_("report that we have only objects reachable from this object"), 0), + OPT_PASSTHRU_ARGV(0, "negotiation-restrict", &opt_fetch, N_("revision"), + N_("report that we have only objects reachable from this object"),
Quoted 8 lines
builtin/pull.c int cmd_pull(int argc, · line 1007
Quoted 4 lines
builtin/pull.c +1
-0+ 0), OPT_BOOL(0, "show-forced-updates", &opt_show_forced_updates, N_("check for forced-updates on all updated branches")), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,It's a shame we don't have a nice way to combine the
OPT_ALIASandOPT_PASSTHRU_ARGVfunctionality, but it's only a small duplication cost of the repeated definition.
Actually, I just missed that I should use OPT_ALIAS in 'pull' as well as how it's used in 'fetch'. Will fix.
Thanks,
-Stolee
Re: [PATCH v3 3/7] transport: rename negotiation_tips
In reply to Matthew John CheethamQuoted 11 lines · Matthew John Cheetham
Quoted 7 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
The previous change added the --negotiation-restrict synonym for the --negotiation-tips option for 'git fetch'. In anticipation of adding a new option that behaves similarly but with distinct changes to its behavior, rename the internal representation of this data from '
negotiation_tips' to 'negotiation_restrict_tips'.Nitpick: s/tips/tip/ .. no trailing s for either the option name, nor the (old) variable name. The function names do use the plural however.
Thanks for the close eye!
Quoted 16 lines
transport-helper.c static int fetch_refs(struct transport *transport, · line 758
Quoted 11 lines
transport-helper.c +1
-1@@ -754,7 +754,7 @@ static int fetch_refs(struct transport *transport, set_helper_option(transport, "filter", spec); } - if (data->transport_options.negotiation_tips) + if (data->transport_options.negotiation_restrict_tips) warning("Ignoring --negotiation-tip because the protocol does not support it."); if (data->fetch)Oh! Looks like a place was missed when renaming the preferred option name in strings. It probably makes sense to do this rename in this patch (rather than in patch 1) since we're already updating the struct field
name here anyway, but up to you.
Quoted 1 line
Also do we also want to make it translatable like the others?
I will update this as part of the previous patch that handles all the strings, including making it translatable and dropping the capital "I" at the start. Good find.
Quoted 3 lines
Aside: I just noticed another '--negotiation-tip' instance in the
get_commons_through_negotiationfunction insend-pack.c. It still uses the 'tip' option name when forming the shell cmdline.
Thanks! I've done a more careful search and confirmed that you found everything I had missed.
-Stolee
Re: [PATCH v3 5/7] fetch: add --negotiation-include option for negotiation
In reply to Derrick Stolee via GitGitGadgetQuoted 20 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
Add a new --negotiation-include option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
This is useful when the repository has a large number of references, so the normal negotiation algorithm truncates the list. This is especially important in repositories with long parallel commit histories. For example, a repo could have a 'dev' branch for development and a 'release' branch for released versions. If the 'dev' branch isn't selected for negotiation, then it's not a big deal because there are many in-progress development branches with a shared history. However, if 'release' is not selected for negotiation, then the server may think that this is the first time the client has asked for that reference, causing a full download of its parallel commit history (and any extra data that may be unique to that branch). This is based on a real example where certain fetches would grow to 60+ GB when a release branch updated.
Quoted 8 lines
This option is a complement to --negotiation-restrict, which reduces the negotiation ref set to a specific list. In the earlier example, using --negotiation-restrict to focus the negotiation to 'dev' and 'release' would avoid those problematic downloads, but would still not allow advertising potentially-relevant user brances. In this way, the 'include' version solves the problem I mention while allowing negotiation to pick other references opportunistically. The two options can also be combined to allow the best of both worlds.
Nice explanation and motivation for the need of such as feature.
One small typo: s/brances/branches/
Quoted 3 lines
The argument may be an exact ref name or a glob pattern. Non-existent refs are silently ignored. This behavior is also updated in the ref matching logic for the related --negotiation-restrict option to match.
Calling out the intent for the behaviour change (non-existent refs are silently ignored). This is an important point.
Quoted 38 lines
The implementation outputs the requested objects as haves before the negotiation algorithm kicks in and performs a priority-queue walk from the tip commits. In order to avoid duplicates, we mark the requested objects as COMMON so they (and their descendants) are not output by the negotiator. The negotiator still outputs at least one have before a round is flushed, when the server could ACK to stop the negotiation.
Also add --negotiation-include to 'git pull' passthrough options.
Signed-off-by Derrick Stolee <[email protected]>--- Documentation/fetch-options.adoc | 19 ++++++ builtin/fetch.c | 16 ++++- builtin/pull.c | 3 + fetch-pack.c | 112 +++++++++++++++++++++++++++++-- fetch-pack.h | 10 ++- t/t5510-fetch.sh | 66 ++++++++++++++++++ transport.c | 4 +- transport.h | 6 ++ 8 files changed, 227 insertions(+), 9 deletions(-)Documentation/fetch-options.adoc +9
-0@@ -73,6 +73,25 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate` configuration variables documented in linkgit:git-config[1], and the `--negotiate-only` option below. +`--negotiation-include=<revision>`:: + Ensure that the given ref tip is always sent as a "have" line + during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-restrict` restricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++ +This option may be specified more than once; if so, each ref is sent
Documentation/fetch-options.adoc See also the `fetch.negotiationAlgorithm` and `push.negotiate` · line 94
Quoted 10 lines
Documentation/fetch-options.adoc +10
-0+unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax +is the same as for `--negotiation-restrict`. ++ +If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-include`. +
The placeholder <revision> and the description in the body of "ref
name or glob" slightly disagree with each other. The --negotiation-restrict docs use (<commit>|<glob>) in the syntax definition and "a glob on ref names, a ref, or .. SHA-1 of a commit".
resolve_negotiation_include() calls repo_get_oid() for non-globs
so bare OIDs and abbreviated SHAs work too. Perhaps consider aligning the syntaxes, and mention that OIDs work too.
Quoted 21 lines
Documentation/fetch-options.adoc +0
-0`--negotiate-only`:: Do not fetch anything from the server, and instead print the ancestors of the provided `--negotiation-tip=` arguments,builtin/fetch.c +3
-0@@ -99,6 +99,7 @@ static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ;static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_include =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;@@ -1547,10 +1548,14 @@ static void add_negotiation_restrict_tips(struct git_transport_options *smart_opint old_nr;if (!has_glob_specials (s)) {struct object_id oid; + +/* Ignore missing reference. */
builtin/fetch.c static void add_negotiation_restrict_tips(struct git_transport_options *smart_op · line 1561
Quoted 10 lines
builtin/fetch.c +3
-1if (repo_get_oid (the_repository, s, &oid)) -die (_ ("%s is not a valid object" ), s); +continue ; +/* Fail on missing object pointed by ref. */ if (!odb_has_object (the_repository->objects, &oid,0 ))die (_ ("the object %s does not exist" ), s); +oid_array_append (oids, &oid);continue ; }
This is the change in behaviour - unresolvable revs were a fatal error and are now silently ignored.
Note that t5510 '--negotiation-tip rejects missing OIDs' still passes because it uses an all-zero OID, which parses as a valid hex string, and dies on the second check "object does not exist". Using something like --negotiation-tip=notreal that previously would error will now silently be ignored.
Is it worth another test? (invalid object vs not exists)?
builtin/fetch.c static struct transport *prepare_transport(struct remote *remote, int deepen, · line 1631
Quoted 13 lines
builtin/fetch.c +7
-0@@ -1615,6 +1620,13 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,strbuf_release (&config_name); } } +if (negotiation_include.nr) { +if (transport->smart_options) + transport->smart_options->negotiation_include = &negotiation_include; +else +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-include" ); + }return transport; }
There is a difference between the existing --negotiation-restrict option and the new --negotiation-include option. Patch 3's commit message says:
"The 'tips' part is kept because this is an oid_array in the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data."
The new option passes the raw string_list to the transport layer and lets it resolve it instead. If the transport layer now learns how to resolve refs to oids, why not for tips/restrict?
Would it be easier for future readers for these complementary options to resolve their inputs at the same layer? Or at least call out why:
"would prefer raw tips but for back-compat we resolve in the built-in" for example.
Quoted 71 lines
builtin/fetch.c +2
-0@@ -2582,6 +2594,8 @@ int cmd_fetch(int argc,OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), +OPT_STRING_LIST (0 ,"negotiation-include" , &negotiation_include,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" )),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),builtin/pull.c +3
-0@@ -1002,6 +1002,9 @@ int cmd_pull(int argc,OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-include" , &opt_fetch,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,fetch-pack.c +33
-0@@ -25,6 +25,7 @@#include "oidset.h" #include "packfile.h" #include "odb.h" +#include "object-name.h" #include "path.h" #include "connected.h" #include "fetch-negotiator.h" @@ -332,6 +333,48 @@ static void send_filter(struct fetch_pack_args *args, } } +static int add_oid_to_oidset (const struct reference *ref,void *cb_data) +{ +struct oidset *set = cb_data; +if (!odb_has_object (the_repository->objects, ref->oid,0 )) +die (_ ("the object %s does not exist" ),oid_to_hex (ref->oid)); +oidset_insert (set, ref->oid); +return 0 ; +} + +static void resolve_negotiation_include (const struct string_list *negotiation_include, +struct oidset *result) +{ +struct string_list_item *item; + +if (!negotiation_include || !negotiation_include->nr) +return ; + +for_each_string_list_item (item, negotiation_include) { +if (!has_glob_specials (item->string)) { +struct object_id oid; + +/* Ignore missing reference. */ +if (repo_get_oid (the_repository, item->string, &oid)) +continue ; + +/* Fail on missing object pointed by ref. */ +if (!odb_has_object (the_repository->objects, &oid,0 )) +die (_ ("the object %s does not exist" ), + item->string); + +oidset_insert (result, &oid); + }else {
fetch-pack.c static void send_filter(struct fetch_pack_args *args, · line 377
Quoted 10 lines
fetch-pack.c +10
-0+struct refs_for_each_ref_options opts = { + .pattern = item->string, + }; +refs_for_each_ref_ext ( +get_main_ref_store (the_repository), + add_oid_to_oidset, result, &opts); + } + } +} +
resolve_negotiation_include() is basically doing the same as add_negotiation_restrict_tips() except outputting to an oidset vs oid_array. This is a result of the difference in ref resolution layer between --negotiation-restrict/tip and -include.
Quoted 32 lines
fetch-pack.c +18
-0static int find_common(struct fetch_negotiator *negotiator, struct fetch_pack_args *args, int fd[2], struct object_id *result_oid,@@ -347,6 +390,7 @@ static int find_common(struct fetch_negotiator *negotiator,struct strbuf req_buf =STRBUF_INIT ;size_t state_len =0 ;struct packet_reader reader; +struct oidset negotiation_include_oids =OIDSET_INIT ;if (args->stateless_rpc && multi_ack ==1 )die (_ ("the option '%s' requires '%s'" ),"--stateless-rpc" ,"multi_ack_detailed" );@@ -474,6 +518,33 @@ static int find_common(struct fetch_negotiator *negotiator,trace2_region_enter ("fetch-pack" ,"negotiation_v0_v1" , the_repository); flushes =0 ; retval =-1 ; + +/* Send unconditional haves from --negotiation-include */ +resolve_negotiation_include (args->negotiation_include, + &negotiation_include_oids); +if (oidset_size (&negotiation_include_oids)) { +struct oidset_iter iter; +oidset_iter_init (&negotiation_include_oids, &iter); + +while ((oid =oidset_iter_next (&iter))) { +struct commit *commit; +packet_buf_write (&req_buf,"have %s\n" , +oid_to_hex (oid)); +print_verbose (args,"have %s" ,oid_to_hex (oid)); + count++; + +/* +* If this is a commit, then mark as COMMON to
fetch-pack.c static int find_common(struct fetch_negotiator *negotiator, · line 547
Quoted 10 lines
fetch-pack.c +10
-0+* avoid the negotiator also outputting it as +* a have. +*/ + commit =lookup_commit (the_repository, oid); +if (commit && + !repo_parse_commit (the_repository, commit)) + commit->object.flags |=COMMON ; + } + } +
I want to make sure I understand the COMMON pre-marking before commenting further on this patch. My understanding is there are actually two different COMMON bits in the tree, one defined in fetch-pack.c (bit 6) and one in negotiator/default.c (bit 2):
- fetch-pack.c's COMMON (bit 6) is set after a server ACK confirms an OID is common with us and is read to decide when we've established enough common ground to terminate negotiation. This is not consulted in find_common().
- negotiator/default.c's COMMON (bit 2) is a book-keeping flag used by get_rev() to decide if we skip emitting a commit as a 'have'.
Since we're in fetch-pack.c here, the commit->object.flags |= COMMON line is setting bit 6. The get_rev() call in negotiator/default.c never checks bit 6, only bit 2. As far as I can tell, this mark won't suppress the negotiator from emitting another 'have' line in the protocol v0/v1 paths in find_common().
The v2 path doesn't touch the flags.. add_haves dedups via oidset_contains():
while ((oid = negotiator->next(negotiator))) {
if (negotiation_include_oids &&
oidset_contains(negotiation_include_oids, oid))
continue;
packet_buf_write(req_buf, "have %s\n", ...);
}This works, and is what the new 'avoids duplicates with negotiator' test runs against, on protocol v2. If we run on protocol v0/v1, and if my assessment is correct, then we'd see a duplicate I think?
Sorry if I've not understood correctly or am missing something, which is entirely possible :-)
Thanks,
Matthew
Re: [PATCH v3 4/7] remote: add remote.*.negotiationRestrict config
In reply to Matthew John CheethamQuoted 11 lines · Matthew John Cheetham
Quoted 6 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee<[email protected]>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tips. Both of these options restrict the set of 'haves' the client can send as part of negotiation.
s/tips/tip/ as per the previous patch comments. Not important either way.
Thanks.
Quoted 34 lines
Quoted 19 lines
Documentation/config/remote.adoc +19
-0+remote.<name>.negotiationRestrict:: + When negotiating with this remote during `git fetch` and `git push`, + restrict the commits advertised as "have" lines to only those + reachable from refs matching the given patterns. This multi-valued + config option behaves like `--negotiation-restrict` on the command + line. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the +same as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-restrict` +command-line option. If `--negotiation-restrict` (or its synonym +`--negotiation-tip`) is specified on the command line, then the config +values are not used. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. +Quoted 3 lines
remote.<name>.followRemoteHEAD::
How linkgit:git-fetch[1] should handle updates toremotes/<name>/HEADwhen fetching using the configured refspecs of a remote.You say "during
git fetchandgit push", but doespushactually honour the new config?When the
push.negotiateconfig is on thenget_commons_through_negotiation()fromsend-pack.cshells out togit fetch --negotiate-onlywith one--negotiation-tip=<oid>arg per
ref being pushed, then the URL. This means the CLI restrict list is
always non-empty in the subprocess so inprepare_transport()(in the
below hunk) theif (negotiation_restrict.nr)arm is always taken and the newelse if (remote->negotiation_restrict.nr)arm is never taken.
Quoted 4 lines
BUT.. reading ahead I see that patch 7 actually wires up negotiation config for push - so my commentary here will be moot! Do we want to drop the "and
git push" part from this until patch 7, when it is wired up appropriately?
You're right that this documentation is premature about 'git push'.
Quoted 3 lines
One other suggestion: perhaps we should clarify that
push.negotiateneeds to be set forremote.<name>.negotiationRestrictto be honoured during pushes?
Yes. I'll rewrite this to focus on 'git fetch'. Then in patch 7 I can add a new detail about how to make this behavior be respected in 'git push'.
Quoted 32 lines
Quoted 4 lines
if (
deepen_relative) {
if (deepen_relative< 0)
die(_("negative depth in --deepen is not supported"));builtin/fetch.c int cmd_fetch(int argc, · line 2767
Quoted 10 lines
builtin/fetch.c +4
-0@@ -2749,6 +2758,10 @@ int cmd_fetch(int argc, if (!remote) die(_("must supply remote when using --negotiate-only")); gtransport = prepare_transport(remote, 1, &filter_options); + if (!gtransport->smart_options || + !gtransport->smart_options->negotiation_restrict_tips) + die(_("%s needs one or more %s"), "--negotiate-only", + "--negotiation-restrict=*"); if (gtransport->smart_options) { gtransport->smart_options->acked_commits = &acked_commits; } else {This new condition fires whenever
gtransport->smart_optionsis NULL, i.e. the transport doesn't support smart options. Before this case was handled three lines after this hunk by:} else {
warning(_("protocol does not support --negotiate-only, exiting")); result = 1;
trace2_region_leave("fetch", "negotiate-only", the_repository); goto cleanup;
}What happens now if a user runs --negotiate-only against a non-smart transport is they see an odd message:
fatal: --negotiate-only needs one or more --negotiation-restrict=*
Quoted 8 lines
..but they may have specified --negotiation-restrict options.
Do we instead want &&?
if (
gtransport->smart_options&& !gtransport->smart_options->negotiation_restrict_tips)die(_("%s needs one or more %s"), "--negotiate-only", "--negotiation-restrict=*");
You are right that we want to say "we have smart options but haven't specified restrict arguments" so we can leave the later if/else to handle the null smart_options case. But actually, I think that it would be better to reorganize the conditions altogether:
if (!gtransport->smart_options) {
warning(_("protocol does not support --negotiate-only, "exiting"));
result = 1;
trace2_region_leave("fetch", "negotiate-only", the_repository);
goto cleanup;
}
if (!gtransport->smart_options->negotiation_restrict_tips)
die(_("%s needs one or more %s"), "--negotiate-only",
"--negotiation-restrict=*");
gtransport->smart_options->acked_commits = &acked_commits;This is easier to reason about:
* If we don't have smart options, then skip out of the negotiation logic.
* If we don't have restrict tips, then die().
* Do the negotiation logic only if the previous two conditions didn't hold.
Quoted 19 lines
remote.c static int handle_config(const char *key, const char *value, · line 575
Quoted 13 lines
remote.c +6
-0@@ -562,6 +564,12 @@ static int handle_config(const char *key, const char *value, } else if (!strcmp(subkey, "serveroption")) { return parse_transport_option(key, value, &remote->server_options); + } else if (!strcmp(subkey, "negotiationrestrict")) { + /* reset list on empty value. */ + if (!value || !*value) + string_list_clear(&remote->negotiation_restrict, 0); + else + string_list_append(&remote->negotiation_restrict, value); } else if (!strcmp(subkey, "followremotehead")) { const char *no_warn_branch; if (!strcmp(value, "never"))Here we use the 'empty value means reset the list' pattern, but I notice that the
parse_transport_option()function already supports this reset pattern (and used by serveroption above), with a small difference:
Quoted 7 lines
if (!value)
returnconfig_error_nonbool(var);
if (!*value)
string_list_clear(transport_options, 0);So NULL is an error, but empty string is 'reset'. Is it worth being consistent with other options that use
parse_transport_options?
Thanks for catching this! Let's be consistent. NULL is likely impossible in this case, but let's be consistent. It also needs to return.
Thanks,
-Stolee
Re: [PATCH v3 6/7] remote: add remote.*.negotiationInclude config
In reply to Derrick Stolee via GitGitGadgetQuoted 23 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
Add a new 'remote.<name>.negotiationInclude' multi-valued config option that provides default values for --negotiation-include when no --negotiation-include arguments are specified over the command line. This is a mirror of how 'remote.<name>.negotiationRestrict' specifies defaults for the --negotiation-restrict arguments.
Each value is either an exact ref name or a glob pattern whose tips should always be sent as 'have' lines during negotiation. The config values are resolved through the same
resolve_negotiation_include()codepath as the CLI options.This option is additive with the normal negotiation process: the negotiation algorithm still runs and advertises its own selected commits, but the refs matching the config are sent unconditionally on top of those heuristically selected commits.
Similar to the negotiationRestrict config, an empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
Signed-off-by Derrick Stolee <[email protected]>
Quoted 8 lines
--- Documentation/config/remote.adoc | 27 ++++++++++++++++++ Documentation/fetch-options.adoc | 4 +++ builtin/fetch.c | 10 +++++++ remote.c | 8 ++++++ remote.h | 1 + t/t5510-fetch.sh | 49 ++++++++++++++++++++++++++++++++ 6 files changed, 99 insertions(+)
This patch is a mirror of patch 4 that added the remote config for negotiateRestrict. Some of the same comments apply here too:
- reusing parse_transport_option() vs inline resetting the list
- values could be commit SHAs as well as refs/globs
Quoted 10 lines
Documentation/config/remote.adoc +2
-0@@ -126,6 +126,33 @@ values are not used. Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios. +remote.<name>.negotiationInclude:: + When negotiating with this remote during `git fetch` and `git push`,
Documentation/config/remote.adoc values are not used. · line 140
Quoted 10 lines
Documentation/config/remote.adoc +10
-0+ the client advertises a list of commits that exist locally. In + repos with many references, this list of "haves" can be truncated. + Depending on data shape, dropping certain references may be + expensive. This multi-valued config option specifies ref patterns + whose tips should always be sent as "have" commits during fetch + negotiation with this remote. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the same +as for `--negotiation-restrict`.
Should this say "..same as for --negotiation-include"?
This way each remote.<name>.negotiationX doc cross-references the corresponding --negotiation-X command line option.
Quoted 15 lines
t/t5510-fetch.sh +7
-0@@ -1577,6 +1577,55 @@ test_expect_success '--negotiation-include avoids duplicates with negotiator' ' test_line_count = 1 matches ' +test_expect_success 'remote.<name>.negotiationInclude used as default for --negotiation-include' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + # test the reset of the list on an empty value + git -C client config --add remote.origin.negotiationInclude refs/tags/alpha_1 && + git -C client config --add remote.origin.negotiationInclude "" &&
t/t5510-fetch.sh test_expect_success '--negotiation-include avoids duplicates with negotiator' ' · line 1596
Quoted 10 lines
t/t5510-fetch.sh +10
-0+ git -C client config --add remote.origin.negotiationInclude refs/tags/beta_1 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + origin alpha_s beta_s && + + ALPHA_1=$(git -C client rev-parse alpha_1) && + test_grep "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace +'
This test sets up the include list as [alpha_1, "", beta_1] which after the reset should become [beta_1], but the assertions in the test only check that alpha_1 (sent via the --negotiation-restrict option) and beta_1 (sent via the include) appear. If the reset of the list didn't work then the test still passes because alpha_1 is sent via the CLI option.
Quoted 24 lines
t/t5510-fetch.sh +24
-0+test_expect_success 'remote.<name>.negotiationInclude works with glob patterns' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + git -C client config --add remote.origin.negotiationInclude "refs/tags/beta_*" && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + origin alpha_s beta_s && + + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace && + BETA_2=$(git -C client rev-parse beta_2) && + test_grep "fetch> have $BETA_2" trace +' + +test_expect_success 'CLI --negotiation-include overrides remote.<name>.negotiationInclude' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + git -C client config --add remote.origin.negotiationInclude refs/tags/beta_2 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-include=refs/tags/beta_1 \ + origin alpha_s beta_s &&
t/t5510-fetch.sh test_expect_success '--negotiation-include avoids duplicates with negotiator' ' · line 1631
Quoted 10 lines
t/t5510-fetch.sh +7
-0+ + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace && + BETA_2=$(git -C client rev-parse beta_2) && + test_grep ! "fetch> have $BETA_2" trace +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
Thanks,
Matthew
Re: [PATCH v3 7/7] send-pack: pass negotiation config in push
In reply to Derrick Stolee via GitGitGadgetQuoted 19 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
When push.negotiate is enabled, 'git push' spawns a child 'git fetch --negotiate-only' process to find common commits. Pass --negotiation-include and --negotiation-restrict options from the 'remote.<name>.negotiationInclude' and 'remote.<name>.negotiationRestrict' config keys to this child process.
When negotiationRestrict is configured, it replaces the default behavior of using all remote refs as negotiation tips. This allows the user to control which local refs are used for push negotiation.
When negotiationInclude is configured, the specified ref patterns are passed as --negotiation-include to ensure their tips are always sent as 'have' lines during push negotiation.
This change also updates the use of --negotiation-tip into --negotiation-restrict now that the new synonym exists.
Quoted 7 lines
Signed-off-by Derrick Stolee <[email protected]>--- send-pack.c | 39 +++++++++++++++++++++++++++++++-------- send-pack.h | 2 ++ t/t5516-fetch-push.sh | 30 ++++++++++++++++++++++++++++++ transport.c | 2 ++ 4 files changed, 65 insertions(+), 8 deletions(-)
This patch wires up the negotiation behaviour with push, added in the previous patches.
Quoted 50 lines
send-pack.c +23
-6@@ -433,28 +433,48 @@ static void reject_invalid_nonce(const char *nonce, int len)static void get_commons_through_negotiation (struct repository *r,const char *url, +const struct string_list *negotiation_include, +const struct string_list *negotiation_restrict,const struct ref *remote_refs,struct oid_array *commons) {struct child_process child =CHILD_PROCESS_INIT ;const struct ref *ref;int len = r->hash_algo->hexsz +1 ;/* hash + NL */ -int nr_negotiation_tip =0 ; +int nr_negotiation =0 ; child.git_cmd =1 ; child.no_stdin =1 ; child.out =-1 ;strvec_pushl (&child.args,"fetch" ,"--negotiate-only" ,NULL ); -for (ref = remote_refs; ref; ref = ref->next) { -if (!is_null_oid (&ref->new_oid)) { -strvec_pushf (&child.args,"--negotiation-tip=%s" , -oid_to_hex (&ref->new_oid)); - nr_negotiation_tip++; + +if (negotiation_restrict && negotiation_restrict->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_restrict) +strvec_pushf (&child.args,"--negotiation-restrict=%s" , + item->string); + nr_negotiation = negotiation_restrict->nr; + }else { +for (ref = remote_refs; ref; ref = ref->next) { +if (!is_null_oid (&ref->new_oid)) { +strvec_pushf (&child.args,"--negotiation-restrict=%s" , +oid_to_hex (&ref->new_oid)); + nr_negotiation++; + } } } + +if (negotiation_include && negotiation_include->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_include) +strvec_pushf (&child.args,"--negotiation-include=%s" , + item->string);
send-pack.c static void reject_invalid_nonce(const char *nonce, int len) · line 480
Quoted 10 lines
send-pack.c +4
-1+ nr_negotiation += negotiation_include->nr; + } +strvec_push (&child.args, url); -if (!nr_negotiation_tip) { +if (!nr_negotiation) {child_process_clear (&child);return ; }
Reads cleanly, and also updates the calls to fetch to use the new preferred option name restrict vs the older tip. Nice!
Quoted 71 lines
send-pack.c +4
-1@@ -528,7 +548,10 @@ int send_pack(struct repository *r,repo_config_get_bool (r,"push.negotiate" , &push_negotiate);if (push_negotiate) {trace2_region_enter ("send_pack" ,"push_negotiate" , r); -get_commons_through_negotiation (r, args->url, remote_refs, &commons); +get_commons_through_negotiation (r, args->url, + args->negotiation_include, + args->negotiation_restrict, + remote_refs, &commons);trace2_region_leave ("send_pack" ,"push_negotiate" , r); }send-pack.h +2
-0@@ -18,6 +18,8 @@ struct repository;struct send_pack_args {const char *url; +const struct string_list *negotiation_include; +const struct string_list *negotiation_restrict;unsigned verbose:1 , quiet:1 , porcelain:1,t/t5516-fetch-push.sh +30
-0@@ -254,6 +254,36 @@ test_expect_success 'push with negotiation does not attempt to fetch submodules' ! grep "Fetching submodule" err ' +test_expect_success 'push with negotiation and remote.<name>.negotiationInclude' ' + test_when_finished rm -rf negotiation_include && + mk_empty negotiation_include && + git push negotiation_include $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_include unrelated_commit && + git -C negotiation_include config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_include.negotiationInclude=refs/heads/main \ + push negotiation_include refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + +test_expect_success 'push with negotiation and remote.<name>.negotiationRestrict' ' + test_when_finished rm -rf negotiation_restrict && + mk_empty negotiation_restrict && + git push negotiation_restrict $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_restrict unrelated_commit && + git -C negotiation_restrict config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_restrict.negotiationRestrict=refs/heads/main \ + push negotiation_restrict refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + test_expect_success 'push without wildcard' ' mk_empty testrepo &&transport.c +0
-0
transport.c static int git_transport_push(struct transport *transport, struct ref *remote_re · line 928
Quoted 8 lines
transport.c +2
-0@@ -921,6 +921,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re args.atomic = !!(flags &TRANSPORT_PUSH_ATOMIC ); args.push_options = transport->push_options; args.url = transport->url; + args.negotiation_include = &transport->remote->negotiation_include; + args.negotiation_restrict = &transport->remote->negotiation_restrict;if (flags &TRANSPORT_PUSH_CERT_ALWAYS ) args.push_cert =SEND_PACK_PUSH_CERT_ALWAYS ;
I've got no other comments on this patch :-)
Thanks,
Matthew
Re: [PATCH v3 5/7] fetch: add --negotiation-include option for negotiation
In reply to Matthew John CheethamQuoted 30 lines · Matthew John Cheetham
Quoted 20 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
Add a new --negotiation-include option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
This is useful when the repository has a large number of references, so the normal negotiation algorithm truncates the list. This is especially important in repositories with long parallel commit histories. For example, a repo could have a 'dev' branch for development and a 'release' branch for released versions. If the 'dev' branch isn't selected for negotiation, then it's not a big deal because there are many in-progress development branches with a shared history. However, if 'release' is not selected for negotiation, then the server may think that this is the first time the client has asked for that reference, causing a full download of its parallel commit history (and any extra data that may be unique to that branch). This is based on a real example where certain fetches would grow to 60+ GB when a release branch updated.
Quoted 8 lines
This option is a complement to --negotiation-restrict, which reduces the negotiation ref set to a specific list. In the earlier example, using --negotiation-restrict to focus the negotiation to 'dev' and 'release' would avoid those problematic downloads, but would still not allow advertising potentially-relevant user brances. In this way, the 'include' version solves the problem I mention while allowing negotiation to pick other references opportunistically. The two options can also be combined to allow the best of both worlds.
Quoted 3 lines
Nice explanation and motivation for the need of such as feature.
One small typo: s/brances/branches/
Thanks.
Quoted 20 lines
Quoted 20 lines
Documentation/fetch-options.adoc +0
-0@@ -73,6 +73,25 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate`configuration variables documented in linkgit:git-config[1], and the
--negotiate-onlyoption below. +--negotiation-include=<revision>::
+ Ensure that the given ref tip is always sent as a "have" line + during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when--negotiation-restrictrestricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++Documentation/fetch-options.adoc +7
-0+This option may be specified more than once; if so, each ref is sent +unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax +is the same as for `--negotiation-restrict`. ++
Quoted 8 lines
Documentation/fetch-options.adoc See also the `fetch.negotiationAlgorithm` and `push.negotiate` · line 94
Quoted 4 lines
Documentation/fetch-options.adoc +4
-0+If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-include`. +The placeholder
<revision>and the description in the body of "ref
name or glob" slightly disagree with each other. The--negotiation-restrictdocs use(<commit>|<glob>)in the syntax definition and "a glob on ref names, a ref, or .. SHA-1 of a commit".
Good eye.
Quoted 2 lines
resolve_negotiation_include()callsrepo_get_oid()for non-globs
so bare OIDs and abbreviated SHAs work too. Perhaps consider aligning the syntaxes, and mention that OIDs work too.
Will do.
Quoted 21 lines
Quoted 9 lines
Documentation/fetch-options.adoc +0
-0@@ -1547,10 +1548,14 @@ static void add_negotiation_restrict_tips(struct
git_transport_options*smart_op
intold_nr;
if (!has_glob_specials(s)) {
structobject_idoid;
+
+ /* Ignore missing reference. */
if (repo_get_oid(the_repository, s, &oid)) -die(_("%s is not a valid object"), s);Quoted 8 lines
+ continue;
+ /* Fail on missing object pointed by ref. */ if (!odb_has_object(the_repository->objects, &oid, 0))die(_("the object %s does not exist"), s); +
oid_array_append(oids, &oid);
continue;
}This is the change in behaviour - unresolvable revs were a fatal error and are now silently ignored.
Quoted 7 lines
Note that t5510 '--negotiation-tip rejects missing OIDs' still passes because it uses an all-zero OID, which parses as a valid hex string, and dies on the second check "object does not exist". Using something like
--negotiation-tip=notrealthat previously would error will now silently be ignored.Is it worth another test? (invalid object vs not exists)?
Yes, let's add a test to guarantee this behavior works.
Quoted 25 lines
Quoted 14 lines
Documentation/fetch-options.adoc +7
-0@@ -1615,6 +1620,13 @@ static struct transport *prepare_transport(struct remote *remote, int deepen, strbuf_release(&config_name); } } + if (negotiation_include.nr) { + if (transport->smart_options) + transport->smart_options->negotiation_include = &negotiation_include; + else + warning(_("ignoring %s because the protocol does not support it"), + "--negotiation-include"); + } return transport;Quoted 1 line
}
There is a difference between the existing
--negotiation-restrictoption and the new--negotiation-includeoption. Patch 3's commit message says:"The 'tips' part is kept because this is an
oid_arrayin the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data."
Quoted 8 lines
The new option passes the raw
string_listto the transport layer and lets it resolve it instead. If the transport layer now learns how to resolve refs to oids, why not for tips/restrict?Would it be easier for future readers for these complementary options to resolve their inputs at the same layer? Or at least call out why:
"would prefer raw tips but for back-compat we resolve in the built-in" for example.
This is a really key observation. It's a bit of work to unravel, but I think it's better for unifying these things. Look forward to a better organization in the next version.
Quoted 31 lines
Quoted 31 lines
+static void resolve_negotiation_include(const struct
string_list*negotiation_include,
+ struct oidset *result)Documentation/fetch-options.adoc +28
-0+{ + struct string_list_item *item; + + if (!negotiation_include || !negotiation_include->nr) + return; + + for_each_string_list_item(item, negotiation_include) { + if (!has_glob_specials(item->string)) { + struct object_id oid; + + /* Ignore missing reference. */ + if (repo_get_oid(the_repository, item->string, &oid)) + continue; + + /* Fail on missing object pointed by ref. */ + if (!odb_has_object(the_repository->objects, &oid, 0)) + die(_("the object %s does not exist"), + item->string); + + oidset_insert(result, &oid); + } else { + struct refs_for_each_ref_options opts = { + .pattern = item->string, + }; + refs_for_each_ref_ext( + get_main_ref_store(the_repository), + add_oid_to_oidset, result, &opts); + }
Quoted 8 lines
Documentation/fetch-options.adoc
Quoted 3 lines
+ }
Documentation/fetch-options.adoc +2
-0+} +
resolve_negotiation_include()is basically doing the same asadd_negotiation_restrict_tips()except outputting to anoidsetvsoid_array. This is a result of the difference in ref resolution layer between--negotiation-restrict/tipand-include.
Yes, this code will be replaced with a unified approach in the next version.
Quoted 70 lines
Quoted 31 lines
static int find_common(struct
fetch_negotiator*negotiator, structfetch_pack_args*args,
int fd[2], structobject_id*result_oid,Documentation/fetch-options.adoc +18
-0@@ -347,6 +390,7 @@ static int find_common(struct fetch_negotiator *negotiator, struct strbuf req_buf = STRBUF_INIT; size_t state_len = 0; struct packet_reader reader; + struct oidset negotiation_include_oids = OIDSET_INIT; if (args->stateless_rpc && multi_ack == 1) die(_("the option '%s' requires '%s'"), "--stateless-rpc", "multi_ack_detailed");@@ -474,6 +518,33 @@ static int find_common(struct fetch_negotiator *negotiator, trace2_region_enter("fetch-pack", "negotiation_v0_v1", the_repository); flushes = 0; retval = -1; + + /* Send unconditional haves from --negotiation-include */ + resolve_negotiation_include(args->negotiation_include, + &negotiation_include_oids); + if (oidset_size(&negotiation_include_oids)) { + struct oidset_iter iter; + oidset_iter_init(&negotiation_include_oids, &iter); + + while ((oid = oidset_iter_next(&iter))) { + struct commit *commit; + packet_buf_write(&req_buf, "have %s\n", + oid_to_hex(oid)); + print_verbose(args, "have %s", oid_to_hex(oid)); + count++; + + /* + * If this is a commit, then mark as COMMON toDocumentation/fetch-options.adoc static int find_common(struct fetch_negotiator *negotiator, · line 547
Quoted 10 lines
Documentation/fetch-options.adoc +10
-0+ * avoid the negotiator also outputting it as + * a have. + */ + commit = lookup_commit(the_repository, oid); + if (commit && + !repo_parse_commit(the_repository, commit)) + commit->object.flags |= COMMON; + } + } +I want to make sure I understand the COMMON pre-marking before commenting further on this patch. My understanding is there are actually two different COMMON bits in the tree, one defined in
fetch-pack.c(bit 6) and one innegotiator/default.c(bit 2):- fetch-pack.c's COMMON (bit 6) is set after a server ACK confirms an OID is common with us and is read to decide when we've established enough common ground to terminate negotiation. This is not consulted in
find_common().- negotiator/default.c's COMMON (bit 2) is a book-keeping flag used by
get_rev()to decide if we skip emitting a commit as a 'have'.Since we're in
fetch-pack.chere, thecommit->object.flags |= COMMONline is setting bit 6. Theget_rev()call innegotiator/default.cnever checks bit 6, only bit 2. As far as I can tell, this mark won't suppress the negotiator from emitting another 'have' line in the protocol v0/v1 paths infind_common().The v2 path doesn't touch the flags..
add_havesdedups viaoidset_contains():while ((oid =
negotiator->next(negotiator))) { if (negotiation_include_oids&&oidset_contains(negotiation_include_oids, oid)) continue;
packet_buf_write(req_buf, "have %s\n", ...); }
Quoted 6 lines
This works, and is what the new 'avoids duplicates with negotiator' test runs against, on protocol v2. If we run on protocol v0/v1, and if my assessment is correct, then we'd see a duplicate I think?
Sorry if I've not understood correctly or am missing something, which is entirely possible :-)
This is a great catch! It shows that I'm breaking some abstractions here, and thus it's easy to make such a mistake. It's worse that I don't catch this problem in the tests that I am adding. I'll add a test that demonstrates the difference.
But beyond that, I think the biggest issue is that the consumer of an abstract 'negotiator' is assuming something about its implementation. This means that I should update the negotiator struct to have a function pointer dedicated to "I chose to send this 'have'" and then the negotiator can control how to prevent sending more 'have's reachable from those tips.
Thanks,
-Stolee
Re: [PATCH v3 6/7] remote: add remote.*.negotiationInclude config
In reply to Matthew John CheethamQuoted 8 lines · Matthew John Cheetham
On 2026-04-22 16:25, Derrick Stolee via GitGitGadget wrote:
This patch is a mirror of patch 4 that added the remote config for negotiateRestrict. Some of the same comments apply here too:
- reusing
parse_transport_option()vs inline resetting the list- values could be commit SHAs as well as refs/globs
Will do. Thanks.
Quoted 22 lines
Quoted 9 lines
Documentation/config/remote.adoc +0
-0@@ -126,6 +126,33 @@ values are not used.Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios.
Documentation/config/remote.adoc +2
-0+remote.<name>.negotiationInclude:: + When negotiating with this remote during `git fetch` and `git push`,Documentation/config/remote.adoc values are not used. · line 140
Quoted 10 lines
Documentation/config/remote.adoc +10
-0+ the client advertises a list of commits that exist locally. In + repos with many references, this list of "haves" can be truncated. + Depending on data shape, dropping certain references may be + expensive. This multi-valued config option specifies ref patterns + whose tips should always be sent as "have" commits during fetch + negotiation with this remote. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the same +as for `--negotiation-restrict`.Should this say "..same as for
--negotiation-include"?
Quoted 2 lines
This way each
remote.<name>.negotiationXdoc cross-references the corresponding--negotiation-Xcommand line option.
Good find. The rest of the description uses *-include.
Quoted 27 lines
Quoted 27 lines
t/t5510-fetch.sh +0
-0@@ -1577,6 +1577,55 @@ test_expect_success '--negotiation-include avoidsduplicates with negotiator' '
test_line_count= 1 matches
'
+test_expect_success 'remote.<name>.negotiationInclude used as default for -- negotiation-include' '
+test_when_finishedrm -f trace &&
+setup_negotiation_tipserver server 0 &&
+
+ # test the reset of the list on an empty value
+ git -C client config --add remote.origin.negotiationInclude refs/tags/alpha_1&&
+ git -C client config --add remote.origin.negotiationInclude "" && + git -C client config --add remote.origin.negotiationInclude refs/tags/beta_1&&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
+ --negotiation-restrict=alpha_1 \
+ originalpha_sbeta_s&&
+
+ ALPHA_1=$(git -C client rev-parsealpha_1) &&
+test_grep"fetch> have $ALPHA_1" trace &&
+ BETA_1=$(git -C client rev-parsebeta_1) &&
+test_grep"fetch> have $BETA_1" trace
Quoted 8 lines
Quoted 1 line
+'
This test sets up the include list as [
alpha_1, "",beta_1] which after the reset should become [beta_1], but the assertions in the test only check thatalpha_1(sent via the --negotiation-restrict option) andbeta_1(sent via the include) appear. If the reset of the list didn't work then the test still passes becausealpha_1is sent via the CLI option.
Good point. the negotiation-restrict option is making this less clear. If I point the restrict option at alpha_2, then I think it exercises things correctly.
Thanks,
-Stolee
v4 cover fetch: rework negotiation tip options
In reply to Derrick Stolee via GitGitGadgetFetch negotiation aims to find enough information from haves and wants such that the server can be reasonably confident that it will send all necessary objects and not too many "extra" objects that the client already has. However, this can break down if there are too many references, since Git truncates the list of haves based on a few factors (a 256 count limit or the server sending an ACK at the right time).
We already have the --negotiation-tip feature to focus the set of references that are used in negotiation, but I feel like this is designed backwards. I'd rather that we have a way to say "this is an important set of refs, but feel free to add more refs if needed" than "only use these refs for negotiation".
Here's an example that demonstrates the problem. In an internal monorepo, developers work off of the 'main' branch so there are thousands of user branches that each add a few commits different from the 'main' branch. However, there is also a long-lived 'release' branch. This branch has a first-parent history that is parallel to 'main' and each of those commits is a merge whose second parent is a commit from 'main' that had a successful CI run. There are additional changes in the 'release' branch merge commits that add some changelog data, so there is a nontrivial set of novel blob content in that branch and not just a different set of commits.
The problem we had was that our georeplication system was regularly fetching from the origin and trying to get all data from all reachable branches. When the 'release' branch updated, the client would run out of haves before advertising its copy of the 'release' branch, but it would still list the new 'release' tip as a want. The server would then think that the client had never fetched that branch before and would send all of the changelog data from the whole history of the repo. (This led to a lot of downstream problems; we mitigated by setting a refspec that stopped fetching the 'release' branch, but this is not ideal.)
What I'd like is a mechanism to say "always advertise the client's version of 'main' and 'release' but also opportunistically include some user branches".
Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.
There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.
This patch series adds a new '--negotiation-include' option that does what I
want: it makes sure that these references are included as 'have's during negotiation. In order to help clarify the difference between this and '--negotiation-tip', I first create a synonym called '--negotiation-restrict'.
Both of these options get 'remote.*.negotiation(Include|Restrict)' config options that enable their behavior by default.
During development, I had briefly considered only using config values, but that required some strange changes to care about the remote name in the transport layer. This was most different in the 'git push' integration. When I discovered the '--negotiation-tip' feature during the process, that gave me a clear pattern to follow with the addition of a config on top.
Updates in v2
=============
This version is a near-complete rewrite based on feedback around the names of the previous option and config. The --negotiation-restrict option is new and the ability to set it via config is also new.
I did try to be more careful around translatable error messages, too.
Updates in v3
=============
* --negotiation-tip is now an alias of --negotiation-restrict.
* More translatable strings use %s to isolate non-translatable options from translatable words.
* The string_list named negotiation_tip is now renamed to negotiation_restrict.
* The config options now allow an empty value to reset the list.
* The --negotiation-require option is now called --negotiation-include.
* Similarly, the config option is renamed and all code references.
* The included haves now mark their commits as COMMON so commits that they can reach are not included in the negotiation walk if they are reached from the restricted commits.
* The ref iterators are more careful about failing on bad references (ref exists but object doesn't) and ignoring missing references (perhaps config is erroneous?).
* When sending tips during push negotiation, use the --negotiation-restrict option instead of -tip.
Updates in v4
=============
Thanks, Matthew, for the detailed review! There are some big changes in this version.
* Expanded commit message to cite the commit that introduced the bug (3f763ddf28).
* Renamed --negotiation-tip to --negotiation-restrict throughout docs/code (including send-pack.c, transport-helper.c, builtin/pull.c). Added OPT_ALIAS in git-pull.
* Switched config parsing to use parse_transport_option() helper. Removed git push from docs (not implemented yet). Restructured --negotiate-only validation flow.
* NEW Patch 5: Added have_sent() interface to negotiators, so included haves can be de-duplicated properly by the negotiation algorithm.
* Replaced COMMON flag hack with negotiator->have_sent() calls. Moved ref-pattern resolution into builtin/fetch.c (add_negotiation_tips()) so fetch-pack receives pre-resolved oid_array instead of string_list. Added test for --negotiation-tip ignoring missing refs. Added duplicate-avoidance test for v0. Accepts commit hashes in addition to ref names/globs.
* Use parse_transport_option() for config. Updated docs to mention commit hashes. Removed git push from config docs. Fixed test to use correct restrict/include combinations.
* In the last patch, add doc notes that remote config values also apply during git push with push.negotiate, now that they are integrated by that change.
Thanks, -Stolee
Derrick Stolee (8):
t5516: fix test order flakiness
fetch: add --negotiation-restrict option
transport: rename negotiation_tips
remote: add remote.*.negotiationRestrict config
negotiator: add have_sent() interface
fetch: add --negotiation-include option for negotiation
remote: add remote.*.negotiationInclude config
send-pack: pass negotiation config in push
Documentation/config/fetch.adoc | 2 +- Documentation/config/remote.adoc | 51 +++++++++ Documentation/fetch-options.adoc | 29 ++++- builtin/fetch.c | 82 ++++++++++--- builtin/pull.c | 6 +- fetch-negotiator.h | 9 ++ fetch-pack.c | 99 +++++++++++++--- fetch-pack.h | 10 +- negotiator/default.c | 8 ++ negotiator/noop.c | 7 ++ negotiator/skipping.c | 8 ++ remote.c | 10 ++ remote.h | 2 + send-pack.c | 39 +++++-- send-pack.h | 2 + t/t5510-fetch.sh | 191 +++++++++++++++++++++++++++++++ t/t5516-fetch-push.sh | 32 +++++- t/t5702-protocol-v2.sh | 4 +- transport-helper.c | 5 +- transport.c | 20 +++- transport.h | 7 +- 21 files changed, 561 insertions(+), 62 deletions(-)
base-commit: 6e8d538aab8fe4dd07ba9fb87b5c7edcfa5706ad
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2085%2Fderrickstolee%2Fmust-have-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2085/derrickstolee/must-have-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/2085
Range-diff vs v3:
1: 466c56abe0 ! 1: 7409a479d6 t5516: fix test order flakiness
@@ Commit message
t5516: fix test order flakiness
The 'fetch follows tags by default' test sorts using 'sort -k 4', but
- for-each-ref output only has 3 columns. This relies on sort treating
- records with fewer fields as having an empty fourth field, which may
- produce unstable results depending on locale. Use 'sort -k 3' to match
- the actual number of columns in the output.
+ for-each-ref output only has 3 columns. This relies on sort treating records
+ with fewer fields as having an empty fourth field, which may produce
+ unstable results depending on locale. This appears to be an accident added
+ in 3f763ddf28 (fetch: set remote/HEAD if it does not exist, 2024-11-22).
+
+ Use 'sort -k 3' to match the actual number of columns in the output.
2: fe875399a8 ! 2: 7836a2d6a5 fetch: add --negotiation-restrict option
@@ Commit message
+ ## Documentation/config/fetch.adoc ##
+@@
+ default is skipping. Unknown values will cause git fetch to
+ error out.
+ +
+-See also the --negotiate-only and --negotiation-tip options to
++See also the --negotiate-only and --negotiation-restrict options to
+ linkgit:git-fetch[1].
+
+ fetch.showForcedUpdates::
+
## Documentation/fetch-options.adoc ##
@@ Documentation/fetch-options.adoc: the current repository has the same history as the source repository.
.git/shallow. This option updates .git/shallow and accepts such
@@ Documentation/fetch-options.adoc: the current repository has the same history as
This option may be specified more than once; if so, Git will report
commits reachable from any of the given commits.
+
+@@ Documentation/fetch-options.adoc: configuration variables documented in linkgit:git-config[1], and the +
+ --negotiate-only::
+ Do not fetch anything from the server, and instead print the
+- ancestors of the provided --negotiation-tip= arguments,
++ ancestors of the provided --negotiation-restrict= arguments,
+ which we have in common with the server.
+ +
+ This is incompatible with --recurse-submodules=(yes|on-demand).
## builtin/fetch.c ##
@@ builtin/fetch.c: static void add_negotiation_tips(struct git_transport_options *smart_options)
@@ builtin/fetch.c: int cmd_fetch(int argc,
## builtin/pull.c ##
@@ builtin/pull.c: int cmd_pull(int argc,
- OPT_PASSTHRU_ARGV(0, "negotiation-tip", &opt_fetch, N_("revision"),
+ OPT_PASSTHRU('6', "ipv6", &opt_ipv6, NULL,
+ N_("use IPv6 addresses only"),
+ PARSE_OPT_NOARG),
+- OPT_PASSTHRU_ARGV(0, "negotiation-tip", &opt_fetch, N_("revision"),
++ OPT_PASSTHRU_ARGV(0, "negotiation-restrict", &opt_fetch, N_("revision"),
N_("report that we have only objects reachable from this object"), 0),
-+ OPT_PASSTHRU_ARGV(0, "negotiation-restrict", &opt_fetch, N_("revision"),
-+ N_("report that we have only objects reachable from this object"),
-+ 0),
++ OPT_ALIAS(0, "negotiation-tip", "negotiation-restrict"),
OPT_BOOL(0, "show-forced-updates", &opt_show_forced_updates,
N_("check for forced-updates on all updated branches")),
OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
+ ## send-pack.c ##
+@@ send-pack.c: static void get_commons_through_negotiation(struct repository *r,
+ strvec_pushl(&child.args, "fetch", "--negotiate-only", NULL);
+ for (ref = remote_refs; ref; ref = ref->next) {
+ if (!is_null_oid(&ref->new_oid)) {
+- strvec_pushf(&child.args, "--negotiation-tip=%s",
++ strvec_pushf(&child.args, "--negotiation-restrict=%s",
+ oid_to_hex(&ref->new_oid));
+ nr_negotiation_tip++;
+ }
+
## t/t5510-fetch.sh ##
@@ t/t5510-fetch.sh: EOF
test_cmp fatal-expect fatal-actual
@@ t/t5702-protocol-v2.sh: setup_negotiate_only () {
EOF
test_must_fail git -c protocol.version=2 -C client fetch \
+
+ ## transport-helper.c ##
+@@ transport-helper.c: static int fetch_refs(struct transport *transport,
+ }
+
+ if (data->transport_options.negotiation_tips)
+- warning("Ignoring --negotiation-tip because the protocol does not support it.");
++ warning(_("ignoring %s because the protocol does not support it."),
++ "--negotiation-restrict");
+
+ if (data->fetch)
+ return fetch_with_fetch(transport, nr_heads, to_fetch);
3: 4332cbf266 ! 3: 401bdaff7c transport: rename negotiation_tips
@@ Commit message
transport: rename negotiation_tips
The previous change added the --negotiation-restrict synonym for the
- --negotiation-tips option for 'git fetch'. In anticipation of adding a
- new option that behaves similarly but with distinct changes to its
- behavior, rename the internal representation of this data from
- 'negotiation_tips' to 'negotiation_restrict_tips'.
+ --negotiation-tip option for 'git fetch'. In anticipation of adding a new
+ option that behaves similarly but with distinct changes to its behavior,
+ rename the internal representation of this data from 'negotiation_tips' to
+ 'negotiation_restrict_tips'.
- The 'tips' part is kept because this is an oid_array in the transport
- layer. This requires the builtin to handle parsing refs into collections
- of oids so the transport layer can handle this cleaner form of the data.
+ The 'tips' part is kept because this is an oid_array in the transport layer.
+ This requires the builtin to handle parsing refs into collections of oids so
+ the transport layer can handle this cleaner form of the data.
Also update the string_list used to store the inputs from command-line options.
@@ transport-helper.c: static int fetch_refs(struct transport *transport,
- if (data->transport_options.negotiation_tips)
+ if (data->transport_options.negotiation_restrict_tips)
- warning("Ignoring --negotiation-tip because the protocol does not support it.");
+ warning(_("ignoring %s because the protocol does not support it."),
+ "--negotiation-restrict");
- if (data->fetch)
## transport.c ##
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
4: d2f48b78b5 ! 4: a14c568a1f remote: add remote.*.negotiationRestrict config
@@ Metadata
## Commit message ##
remote: add remote.*.negotiationRestrict config
- In a previous change, the --negotiation-restrict command-line option of
- 'git fetch' was added as a synonym of --negotiation-tips. Both of these
- options restrict the set of 'haves' the client can send as part of
- negotiation.
+ In a previous change, the --negotiation-restrict command-line option of 'git
+ fetch' was added as a synonym of --negotiation-tip. Both of these options
+ restrict the set of 'haves' the client can send as part of negotiation.
This was previously not available via a configuration option. Add a new
- 'remote.<name>.negotiationRestrict' multi-valued config option that
- updates 'git fetch <name>' to use these restrictions by default.
+ 'remote.<name>.negotiationRestrict' multi-valued config option that updates
+ 'git fetch <name>' to use these restrictions by default.
If the user provides even one --negotiation-restrict argument, then the config is ignored.
@@ Documentation/config/remote.adoc: priority configuration file (e.g. `.git/config
$HOME/.gitconfig).
+remote.<name>.negotiationRestrict::
-+ When negotiating with this remote during git fetch and git push,
-+ restrict the commits advertised as "have" lines to only those
-+ reachable from refs matching the given patterns. This multi-valued
-+ config option behaves like --negotiation-restrict on the command
-+ line.
++ When negotiating with this remote during git fetch, restrict the
++ commits advertised as "have" lines to only those reachable from refs
++ matching the given patterns. This multi-valued config option behaves
++ like --negotiation-restrict on the command line.
++
+Each value is either an exact ref name (e.g. refs/heads/release) or a +glob pattern (e.g. refs/heads/release/*). The pattern syntax is the
@@ builtin/fetch.c: int cmd_fetch(int argc,
if (!remote)
die(_("must supply remote when using --negotiate-only"));
gtransport = prepare_transport(remote, 1, &filter_options);
-+ if (!gtransport->smart_options ||
-+ !gtransport->smart_options->negotiation_restrict_tips)
+- if (gtransport->smart_options) {
+- gtransport->smart_options->acked_commits = &acked_commits;
+- } else {
++
++ if (!gtransport->smart_options) {
+ warning(_("protocol does not support --negotiate-only, exiting"));
+ result = 1;
+ trace2_region_leave("fetch", "negotiate-only", the_repository);
+ goto cleanup;
+ }
++ if (!gtransport->smart_options->negotiation_restrict_tips)
+ die(_("%s needs one or more %s"), "--negotiate-only",
+ "--negotiation-restrict=*");
- if (gtransport->smart_options) {
- gtransport->smart_options->acked_commits = &acked_commits;
- } else {
++
++ gtransport->smart_options->acked_commits = &acked_commits;
++
+ if (server_options.nr)
+ gtransport->server_options = &server_options;
+ result = transport_fetch_refs(gtransport, NULL);
## remote.c ##
@@ remote.c: static struct remote *make_remote(struct remote_state *remote_state,
@@ remote.c: static int handle_config(const char *key, const char *value,
return parse_transport_option(key, value,
&remote->server_options);
+ } else if (!strcmp(subkey, "negotiationrestrict")) {
-+ /* reset list on empty value. */
-+ if (!value || !*value)
-+ string_list_clear(&remote->negotiation_restrict, 0);
-+ else
-+ string_list_append(&remote->negotiation_restrict, value);
++ return parse_transport_option(key, value,
++ &remote->negotiation_restrict);
} else if (!strcmp(subkey, "followremotehead")) {
const char *no_warn_branch;
if (!strcmp(value, "never"))
-: ---------- > 5: 94b79784fe negotiator: add have_sent() interface
5: ae81ef36a1 ! 6: b4cd458fe0 fetch: add --negotiation-include option for negotiation
@@ Commit message
negotiation ref set to a specific list. In the earlier example, using
--negotiation-restrict to focus the negotiation to 'dev' and 'release'
would avoid those problematic downloads, but would still not allow
- advertising potentially-relevant user brances. In this way, the
+ advertising potentially-relevant user branches. In this way, the
'include' version solves the problem I mention while allowing
negotiation to pick other references opportunistically. The two options
can also be combined to allow the best of both worlds.
@@ Commit message
logic for the related --negotiation-restrict option to match.
The implementation outputs the requested objects as haves before the
- negotiation algorithm kicks in and performs a priority-queue walk from the
- tip commits. In order to avoid duplicates, we mark the requested objects as
- COMMON so they (and their descendants) are not output by the negotiator. The
- negotiator still outputs at least one have before a round is flushed, when
- the server could ACK to stop the negotiation.
+ negotiator performs its own algorithm to choose the next haves. Use the new
+ have_sent() interface to signal these have commits were sent before engaging
+ with the negotiator's next() iterator.
Also add --negotiation-include to 'git pull' passthrough options.
@@ Documentation/fetch-options.adoc: See also the `fetch.negotiationAlgorithm` and
configuration variables documented in linkgit:git-config[1], and the
`--negotiate-only` option below.
-+`--negotiation-include=<revision>`::
-+ Ensure that the given ref tip is always sent as a "have" line
-+ during fetch negotiation, regardless of what the negotiation
++`--negotiation-include=(<commit>|<glob>)`::
++ Ensure that the commits at the given tips are always sent as "have"
++ lines during fetch negotiation, regardless of what the negotiation
+ algorithm selects. This is useful to guarantee that common
+ history reachable from specific refs is always considered, even
+ when `--negotiation-restrict` restricts the set of tips or when
+ the negotiation algorithm would otherwise skip them.
++
-+This option may be specified more than once; if so, each ref is sent
++This option may be specified more than once; if so, each commit is sent
+unconditionally.
++
-+The argument may be an exact ref name (e.g. `refs/heads/release`) or a
-+glob pattern (e.g. `refs/heads/release/{asterisk}`). The pattern syntax
-+is the same as for `--negotiation-restrict`.
++The argument may be an exact ref name (e.g. `refs/heads/release`), an
++object hash, or a glob pattern (e.g. `refs/heads/release/{asterisk}`).
++The pattern syntax is the same as for `--negotiation-restrict`.
++
+If `--negotiation-restrict` is used, the have set is first restricted by
+that option and then increased to include the tips specified by
@@ Documentation/fetch-options.adoc: See also the `fetch.negotiationAlgorithm` and
+
`--negotiate-only`::
Do not fetch anything from the server, and instead print the
- ancestors of the provided `--negotiation-tip=` arguments,
+ ancestors of the provided `--negotiation-restrict=` arguments,
## builtin/fetch.c ##
@@ builtin/fetch.c: static struct transport *gsecondary;
@@ builtin/fetch.c: static struct transport *gsecondary;
struct fetch_config {
enum display_format display_format;
-@@ builtin/fetch.c: static void add_negotiation_restrict_tips(struct git_transport_options *smart_op
+@@ builtin/fetch.c: static int add_oid(const struct reference *ref, void *cb_data)
+ return 0;
+ }
+
+-static void add_negotiation_restrict_tips(struct git_transport_options *smart_options)
++static void add_negotiation_tips(struct string_list *input_list,
++ struct oid_array **output_list)
+ {
+ struct oid_array *oids = xcalloc(1, sizeof(*oids));
+ int i;
+
+- for (i = 0; i < negotiation_restrict.nr; i++) {
+- const char *s = negotiation_restrict.items[i].string;
++ for (i = 0; i < input_list->nr; i++) {
++ const char *s = input_list->items[i].string;
+ struct refs_for_each_ref_options opts = {
+ .pattern = s,
+ };
int old_nr;
if (!has_glob_specials(s)) {
struct object_id oid;
@@ builtin/fetch.c: static void add_negotiation_restrict_tips(struct git_transport_
oid_array_append(oids, &oid);
continue;
}
+@@ builtin/fetch.c: static void add_negotiation_restrict_tips(struct git_transport_options *smart_op
+ warning(_("ignoring %s=%s because it does not match any refs"),
+ "--negotiation-restrict", s);
+ }
+- smart_options->negotiation_restrict_tips = oids;
++ *output_list = oids;
+ }
+
+ static struct transport *prepare_transport(struct remote *remote, int deepen,
+@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remote, int deepen,
+ }
+ if (negotiation_restrict.nr) {
+ if (transport->smart_options)
+- add_negotiation_restrict_tips(transport->smart_options);
++ add_negotiation_tips(&negotiation_restrict,
++ &transport->smart_options->negotiation_restrict_tips);
+ else
+ warning(_("ignoring %s because the protocol does not support it"),
+ "--negotiation-restrict");
+@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remote, int deepen,
+ for_each_string_list_item(item, &remote->negotiation_restrict)
+ string_list_append(&negotiation_restrict, item->string);
+ if (transport->smart_options)
+- add_negotiation_restrict_tips(transport->smart_options);
++ add_negotiation_tips(&negotiation_restrict,
++ &transport->smart_options->negotiation_restrict_tips);
+ else {
+ struct strbuf config_name = STRBUF_INIT;
+ strbuf_addf(&config_name, "remote.%s.negotiationRestrict", remote->name);
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remote, int deepen,
strbuf_release(&config_name);
}
}
+ if (negotiation_include.nr) {
+ if (transport->smart_options)
-+ transport->smart_options->negotiation_include = &negotiation_include;
++ add_negotiation_tips(&negotiation_include,
++ &transport->smart_options->negotiation_include_tips);
+ else
+ warning(_("ignoring %s because the protocol does not support it"),
+ "--negotiation-include");
@@ builtin/fetch.c: int cmd_fetch(int argc,
## builtin/pull.c ##
@@ builtin/pull.c: int cmd_pull(int argc,
- OPT_PASSTHRU_ARGV(0, "negotiation-restrict", &opt_fetch, N_("revision"),
N_("report that we have only objects reachable from this object"),
0),
+ OPT_ALIAS(0, "negotiation-tip", "negotiation-restrict"),
+ OPT_PASSTHRU_ARGV(0, "negotiation-include", &opt_fetch, N_("revision"),
+ N_("ensure this ref is always sent as a negotiation have"),
+ 0),
@@ fetch-pack.c: static void send_filter(struct fetch_pack_args *args,
}
}
-+static int add_oid_to_oidset(const struct reference *ref, void *cb_data)
++static void add_oids_to_set(const struct oid_array *array,
++ struct oidset *set)
+{
-+ struct oidset *set = cb_data;
-+ if (!odb_has_object(the_repository->objects, ref->oid, 0))
-+ die(_("the object %s does not exist"), oid_to_hex(ref->oid));
-+ oidset_insert(set, ref->oid);
-+ return 0;
-+}
-+
-+static void resolve_negotiation_include(const struct string_list *negotiation_include,
-+ struct oidset *result)
-+{
-+ struct string_list_item *item;
-+
-+ if (!negotiation_include || !negotiation_include->nr)
++ if (!array)
+ return;
+
-+ for_each_string_list_item(item, negotiation_include) {
-+ if (!has_glob_specials(item->string)) {
-+ struct object_id oid;
++ for (size_t i = 0; i < array->nr; i++) {
++ struct object_id *oid = &array->oid[i];
++ if (!odb_has_object(the_repository->objects, oid, 0))
++ die(_("the object %s does not exist"), oid_to_hex(oid));
+
-+ /* Ignore missing reference. */
-+ if (repo_get_oid(the_repository, item->string, &oid))
-+ continue;
-+
-+ /* Fail on missing object pointed by ref. */
-+ if (!odb_has_object(the_repository->objects, &oid, 0))
-+ die(_("the object %s does not exist"),
-+ item->string);
-+
-+ oidset_insert(result, &oid);
-+ } else {
-+ struct refs_for_each_ref_options opts = {
-+ .pattern = item->string,
-+ };
-+ refs_for_each_ref_ext(
-+ get_main_ref_store(the_repository),
-+ add_oid_to_oidset, result, &opts);
-+ }
++ oidset_insert(set, oid);
+ }
+}
+
@@ fetch-pack.c: static int find_common(struct fetch_negotiator *negotiator,
retval = -1;
+
+ /* Send unconditional haves from --negotiation-include */
-+ resolve_negotiation_include(args->negotiation_include,
-+ &negotiation_include_oids);
++ add_oids_to_set(args->negotiation_include_tips,
++ &negotiation_include_oids);
+ if (oidset_size(&negotiation_include_oids)) {
+ struct oidset_iter iter;
+ oidset_iter_init(&negotiation_include_oids, &iter);
@@ fetch-pack.c: static int find_common(struct fetch_negotiator *negotiator,
+ print_verbose(args, "have %s", oid_to_hex(oid));
+ count++;
+
-+ /*
-+ * If this is a commit, then mark as COMMON to
-+ * avoid the negotiator also outputting it as
-+ * a have.
-+ */
+ commit = lookup_commit(the_repository, oid);
-+ if (commit &&
-+ !repo_parse_commit(the_repository, commit))
-+ commit->object.flags |= COMMON;
++ if (commit)
++ negotiator->have_sent(negotiator, commit);
+ }
+ }
+
@@ fetch-pack.c: static void add_common(struct strbuf *req_buf, struct oidset *comm
+ struct oidset_iter iter;
+ oidset_iter_init(negotiation_include_oids, &iter);
+
-+ while ((oid = oidset_iter_next(&iter)))
-+ packet_buf_write(req_buf, "have %s\n",
-+ oid_to_hex(oid));
++ while ((oid = oidset_iter_next(&iter))) {
++ struct commit *commit = lookup_commit(the_repository, oid);
++ if (commit) {
++ packet_buf_write(req_buf, "have %s\n",
++ oid_to_hex(oid));
++ negotiator->have_sent(negotiator, commit);
++ }
++ }
+ }
+
while ((oid = negotiator->next(negotiator))) {
-+ if (negotiation_include_oids &&
-+ oidset_contains(negotiation_include_oids, oid))
-+ continue;
packet_buf_write(req_buf, "have %s\n", oid_to_hex(oid));
if (++haves_added >= *haves_to_send)
- break;
@@ fetch-pack.c: static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,
struct fetch_pack_args *args,
const struct ref *wants, struct oidset *common,
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
state = FETCH_SEND_REQUEST;
mark_tips(negotiator, args->negotiation_restrict_tips);
-+ resolve_negotiation_include(args->negotiation_include,
-+ &negotiation_include_oids);
++ add_oids_to_set(args->negotiation_include_tips,
++ &negotiation_include_oids);
for_each_cached_alternate(negotiator,
insert_one_alternate_object);
break;
@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_res
int fd[],
- struct oidset *acked_commits)
+ struct oidset *acked_commits,
-+ const struct string_list *negotiation_include)
++ const struct oid_array *negotiation_include_tips)
{
struct fetch_negotiator negotiator;
struct packet_reader reader;
@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_res
fetch_negotiator_init(the_repository, &negotiator);
mark_tips(&negotiator, negotiation_restrict_tips);
-+ resolve_negotiation_include(negotiation_include,
-+ &negotiation_include_oids);
++ add_oids_to_set(negotiation_include_tips,
++ &negotiation_include_oids);
+
packet_reader_init(&reader, fd[0], NULL, 0,
PACKET_READ_CHOMP_NEWLINE |
@@ fetch-pack.c: void negotiate_using_fetch(const struct oid_array *negotiation_res
## fetch-pack.h ##
@@ fetch-pack.h: struct fetch_pack_args {
+
+ /*
+ * If not NULL, during packfile negotiation, fetch-pack will send "have"
+- * lines only with these tips and their ancestors.
++ * lines for all _include_ tips and then a subset of the _restrict_ tips.
*/
const struct oid_array *negotiation_restrict_tips;
++ const struct oid_array *negotiation_include_tips;
-+ /*
-+ * If non-empty, ref patterns whose tips should always be sent
-+ * as "have" lines during negotiation, regardless of what the
-+ * negotiation algorithm selects.
-+ */
-+ const struct string_list *negotiation_include;
-+
unsigned deepen_relative:1;
unsigned quiet:1;
- unsigned keep_pack:1;
@@ fetch-pack.h: void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,
const struct string_list *server_options,
int stateless_rpc,
int fd[],
- struct oidset *acked_commits);
+ struct oidset *acked_commits,
-+ const struct string_list *negotiation_include);
++ const struct oid_array *negotiation_include_tips);
/*
* Print an appropriate error message for each sought ref that wasn't
## t/t5510-fetch.sh ##
+@@ t/t5510-fetch.sh: EOF
+ test_cmp fatal-expect fatal-actual
+ '
+
++test_expect_success '--negotiation-tip ignores missing refs and invalid hashes' '
++ setup_negotiation_tip server server 0 &&
++ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
++ --negotiation-tip=alpha_1 --negotiation-tip=beta_1 \
++ --negotiation-tip=no-such-ref \
++ --negotiation-tip=invalid-hash \
++ origin alpha_s beta_s &&
++ check_negotiation_tip
++'
++
+ test_expect_success '--negotiation-restrict limits "have" lines sent' '
+ setup_negotiation_tip server server 0 &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
@@ t/t5510-fetch.sh: test_expect_success 'CLI --negotiation-restrict overrides remote config' '
test_grep ! "fetch> have $BETA_1" trace
'
@@ t/t5510-fetch.sh: test_expect_success 'CLI --negotiation-restrict overrides remo
+ test_grep "fetch> have $ALPHA_1" trace >matches &&
+ test_line_count = 1 matches
+'
++
++test_expect_success '--negotiation-include avoids duplicates with v0' '
++ test_when_finished rm -f trace &&
++ setup_negotiation_tip server server 0 &&
++
++ ALPHA_1=$(git -C client rev-parse alpha_1) &&
++ GIT_TRACE_PACKET="$(pwd)/trace" git -C client \
++ -c protocol.version=0 fetch \
++ --negotiation-restrict=alpha_1 \
++ --negotiation-include=refs/tags/alpha_1 \
++ origin alpha_s beta_s &&
++
++ test_grep "fetch> have $ALPHA_1" trace >matches &&
++ test_line_count = 1 matches
++'
+
test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' '
git init df-conflict &&
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
args.stateless_rpc = transport->stateless_rpc;
args.server_options = transport->server_options;
args.negotiation_restrict_tips = data->options.negotiation_restrict_tips;
-+ args.negotiation_include = data->options.negotiation_include;
++ args.negotiation_include_tips = data->options.negotiation_include_tips;
args.reject_shallow_remote = transport->smart_options->reject_shallow;
if (!data->finished_handshake) {
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
data->fd,
- data->options.acked_commits);
+ data->options.acked_commits,
-+ data->options.negotiation_include);
++ data->options.negotiation_include_tips);
ret = 0;
}
goto cleanup;
+@@ transport.c: static int disconnect_git(struct transport *transport)
+ oid_array_clear(data->options.negotiation_restrict_tips);
+ free(data->options.negotiation_restrict_tips);
+ }
++ if (data->options.negotiation_include_tips) {
++ oid_array_clear(data->options.negotiation_include_tips);
++ free(data->options.negotiation_include_tips);
++ }
+ list_objects_filter_release(&data->options.filter_options);
+ oid_array_clear(&data->extra_have);
+ oid_array_clear(&data->shallow);
## transport.h ##
@@ transport.h: struct git_transport_options {
+
+ /*
+ * This is only used during fetch. See the documentation of
+- * negotiation_restrict_tips in struct fetch_pack_args.
++ * these member names in struct fetch_pack_args.
+ *
+- * This field is only supported by transports that support connect or
++ * These fields are only supported by transports that support connect or
+ * stateless_connect. Set this field directly instead of using
+ * transport_set_option().
*/
struct oid_array *negotiation_restrict_tips;
++ struct oid_array *negotiation_include_tips;
-+ /*
-+ * If non-empty, ref patterns whose tips should always be sent
-+ * as "have" lines during negotiation.
-+ */
-+ const struct string_list *negotiation_include;
-+
/*
* If allocated, whenever transport_fetch_refs() is called, add known
- * common commits to this oidset instead of fetching any packfiles. 6: a2d15fa12a ! 7: 7bd70a970b remote: add remote.*.negotiationInclude config
@@ Documentation/config/remote.adoc: values are not used.
the list from broader config scenarios.
+remote.<name>.negotiationInclude::
-+ When negotiating with this remote during git fetch and git push,
-+ the client advertises a list of commits that exist locally. In
-+ repos with many references, this list of "haves" can be truncated.
-+ Depending on data shape, dropping certain references may be
-+ expensive. This multi-valued config option specifies ref patterns
-+ whose tips should always be sent as "have" commits during fetch
-+ negotiation with this remote.
++ When negotiating with this remote during git fetch, the client
++ advertises a list of commits that exist locally. In repos with
++ many references, this list of "haves" can be truncated. Depending
++ on data shape, dropping certain references may be expensive. This
++ multi-valued config option specifies references, commit hashes,
++ or ref pattern globs whose tips should always be sent as "have"
++ commits during fetch negotiation with this remote.
++
-+Each value is either an exact ref name (e.g. refs/heads/release) or a -+glob pattern (e.g. refs/heads/release/*). The pattern syntax is the same -+as for --negotiation-restrict.
++Each value is either an exact ref name (e.g. refs/heads/release), a ++commit hash, or a glob pattern (e.g. refs/heads/release/*). The ++pattern syntax is the same as for --negotiation-include.
++
+These config values are used as defaults for the --negotiation-include +command-line option. If --negotiation-include is specified on the
@@ Documentation/config/remote.adoc: values are not used.
when fetching using the configured refspecs of a remote.
## Documentation/fetch-options.adoc ##
-@@ Documentation/fetch-options.adoc: is the same as for --negotiation-restrict. +@@ Documentation/fetch-options.adoc: The pattern syntax is the same as for --negotiation-restrict.
If --negotiation-restrict is used, the have set is first restricted by
that option and then increased to include the tips specified by --negotiation-include.
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remot
"--negotiation-include");
+ } else if (remote->negotiation_include.nr) {
+ if (transport->smart_options) {
-+ transport->smart_options->negotiation_include = &remote->negotiation_include;
++ add_negotiation_tips(&remote->negotiation_include,
++ &transport->smart_options->negotiation_include_tips);
+ } else {
+ struct strbuf config_name = STRBUF_INIT;
+ strbuf_addf(&config_name, "remote.%s.negotiationInclude", remote->name);
@@ remote.c: static void remote_clear(struct remote *remote)
static void add_merge(struct branch *branch, const char *name)
@@ remote.c: static int handle_config(const char *key, const char *value,
- string_list_clear(&remote->negotiation_restrict, 0);
- else
- string_list_append(&remote->negotiation_restrict, value);
+ } else if (!strcmp(subkey, "negotiationrestrict")) {
+ return parse_transport_option(key, value,
+ &remote->negotiation_restrict);
+ } else if (!strcmp(subkey, "negotiationinclude")) {
-+ /* reset list on empty value. */
-+ if (!value || !*value)
-+ string_list_clear(&remote->negotiation_include, 0);
-+ else
-+ string_list_append(&remote->negotiation_include, value);
++ return parse_transport_option(key, value,
++ &remote->negotiation_include);
} else if (!strcmp(subkey, "followremotehead")) {
const char *no_warn_branch;
if (!strcmp(value, "never"))
@@ t/t5510-fetch.sh: test_expect_success '--negotiation-include avoids duplicates w
+ git -C client config --add remote.origin.negotiationInclude "" &&
+ git -C client config --add remote.origin.negotiationInclude refs/tags/beta_1 &&
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
-+ --negotiation-restrict=alpha_1 \
++ --negotiation-restrict=beta_2 \
+ origin alpha_s beta_s &&
+
+ ALPHA_1=$(git -C client rev-parse alpha_1) &&
-+ test_grep "fetch> have $ALPHA_1" trace &&
++ test_grep ! "fetch> have $ALPHA_1" trace &&
+ BETA_1=$(git -C client rev-parse beta_1) &&
+ test_grep "fetch> have $BETA_1" trace
+'
@@ t/t5510-fetch.sh: test_expect_success '--negotiation-include avoids duplicates w
+ test_grep ! "fetch> have $BETA_2" trace
+'
+
- test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' '
- git init df-conflict &&
- (
+ test_expect_success '--negotiation-include avoids duplicates with v0' '
+ test_when_finished rm -f trace &&
+ setup_negotiation_tip server server 0 &&
7: e6c79f0661 ! 8: 5b968245eb send-pack: pass negotiation config in push
@@ Commit message
+ ## Documentation/config/remote.adoc ##
+@@ Documentation/config/remote.adoc: command-line option. If --negotiation-restrict (or its synonym
+ --negotiation-tip) is specified on the command line, then the config
+ values are not used.
+ +
++These values also influence negotiation during git push if
++push.negotiate is enabled.
+++
+ Blank values signal to ignore all previous values, allowing a reset of
+ the list from broader config scenarios.
+
+@@ Documentation/config/remote.adoc: unconditionally on top of those heuristically selected commits. This
+ option is also used during push negotiation when push.negotiate is
+ enabled.
+ +
++These values also influence negotiation during git push if
++push.negotiate is enabled.
+++
+ Blank values signal to ignore all previous values, allowing a reset of
+ the list from broader config scenarios.
+
+
## send-pack.c ##
@@ send-pack.c: static void reject_invalid_nonce(const char *nonce, int len)
@@ send-pack.c: static void reject_invalid_nonce(const char *nonce, int len)
strvec_pushl(&child.args, "fetch", "--negotiate-only", NULL);
- for (ref = remote_refs; ref; ref = ref->next) {
- if (!is_null_oid(&ref->new_oid)) {
-- strvec_pushf(&child.args, "--negotiation-tip=%s",
-- oid_to_hex(&ref->new_oid));
-- nr_negotiation_tip++;
+
+ if (negotiation_restrict && negotiation_restrict->nr) {
+ struct string_list_item *item;
+ for_each_string_list_item(item, negotiation_restrict)
-+ strvec_pushf(&child.args, "--negotiation-restrict=%s",
+ strvec_pushf(&child.args, "--negotiation-restrict=%s",
+- oid_to_hex(&ref->new_oid));
+- nr_negotiation_tip++;
+ item->string);
+ nr_negotiation = negotiation_restrict->nr;
+ } else {
Signature · 1 line
gitgitgadget
PATCH v4 1/8 t5516: fix test order flakiness
From: Derrick Stolee <[email protected]>
The 'fetch follows tags by default' test sorts using 'sort -k 4', but for-each-ref output only has 3 columns. This relies on sort treating records with fewer fields as having an empty fourth field, which may produce unstable results depending on locale. This appears to be an accident added in 3f763ddf28 (fetch: set remote/HEAD if it does not exist, 2024-11-22).
Use 'sort -k 3' to match the actual number of columns in the output.
--- t/t5516-fetch-push.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5516-fetch-push.sh +1 -1
@@ -1349,7 +1349,7 @@ test_expect_success 'fetch follows tags by default' ' git for-each-ref >tmp1 && sed -n "p; s|refs/heads/main$|refs/remotes/origin/main|p" tmp1 | sed -n "p; s|refs/heads/main$|refs/remotes/origin/HEAD|p" | - sort -k 4 >../expect + sort -k 3 >../expect ) && test_when_finished "rm -rf dst" && git init dst &&
Signature · 1 line
gitgitgadget
PATCH v4 2/8 fetch: add --negotiation-restrict option
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
The --negotiation-tip option to 'git fetch' and 'git pull' allows users to specify that they want to focus negotiation on a small set of references. This is a _restriction_ on the negotiation set, helping to focus the negotiation when the ref count is high. However, it doesn't allow for the ability to opportunistically select references beyond that list.
This subtle detail that this is a 'maximum set' and not a 'minimum set' is not immediately clear from the option name. This makes it more complicated to add a new option that provides the complementary behavior of a minimum set.
For now, create a new synonym option, --negotiation-restrict, that behaves identically to --negotiation-tip. Update the documentation to make it clear that this new name is the preferred option, but we keep the old name for compatibility. Mark --negotiation-tip as an alias of the new, preferred option.
Update a few warning messages with the new option, but also make them translatable with the option name inserted by formatting. At least one of these messages will be reused later for a new option.
--- Documentation/config/fetch.adoc | 2 +- Documentation/fetch-options.adoc | 6 +++++- builtin/fetch.c | 13 ++++++++----- builtin/pull.c | 3 ++- send-pack.c | 2 +- t/t5510-fetch.sh | 25 +++++++++++++++++++++++++ t/t5702-protocol-v2.sh | 4 ++-- transport-helper.c | 3 ++- 8 files changed, 46 insertions(+), 12 deletions(-)
Documentation/config/fetch.adoc +1 -1
@@ -76,7 +76,7 @@ default is `skipping`. Unknown values will cause `git fetch` to error out. + -See also the `--negotiate-only` and `--negotiation-tip` options to +See also the `--negotiate-only` and `--negotiation-restrict` options to linkgit:git-fetch[1]. `fetch.showForcedUpdates`::
Documentation/fetch-options.adoc +5 -1
@@ -49,6 +49,7 @@ the current repository has the same history as the source repository. `.git/shallow`. This option updates `.git/shallow` and accepts such refs. +`--negotiation-restrict=(<commit>|<glob>)`:: `--negotiation-tip=(<commit>|<glob>)`:: By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt to
@@ -58,6 +59,9 @@ the current repository has the same history as the source repository. local ref is likely to have commits in common with the upstream ref being fetched. + +`--negotiation-restrict` is the preferred name for this option; +`--negotiation-tip` is accepted as a synonym. ++ This option may be specified more than once; if so, Git will report commits reachable from any of the given commits. +
@@ -71,7 +75,7 @@ configuration variables documented in linkgit:git-config[1], and the `--negotiate-only`:: Do not fetch anything from the server, and instead print the - ancestors of the provided `--negotiation-tip=` arguments, + ancestors of the provided `--negotiation-restrict=` arguments, which we have in common with the server. + This is incompatible with `--recurse-submodules=(yes|on-demand)`.
builtin/fetch.c +8 -5
@@ -1558,8 +1558,8 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)refs_for_each_ref_ext (get_main_ref_store (the_repository), add_oid, oids, &opts);if (old_nr == oids->nr) -warning ("ignoring --negotiation-tip=%s because it does not match any refs" , - s); +warning (_ ("ignoring %s=%s because it does not match any refs" ), +"--negotiation-restrict" , s); } smart_options->negotiation_tips = oids; }
@@ -1599,7 +1599,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,if (transport->smart_options)add_negotiation_tips (transport->smart_options);else -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-restrict" ); }return transport; }
@@ -2565,8 +2566,9 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
@@ -2657,7 +2659,8 @@ int cmd_fetch(int argc, }if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-tip=*" )); +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" );if (deepen_relative) {if (deepen_relative <0 )
builtin/pull.c +2 -1
@@ -996,9 +996,10 @@ int cmd_pull(int argc,OPT_PASSTHRU ('6' ,"ipv6" , &opt_ipv6,NULL ,N_ ("use IPv6 addresses only" ),PARSE_OPT_NOARG ), -OPT_PASSTHRU_ARGV (0 ,"negotiation-tip" , &opt_fetch,N_ ("revision" ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
send-pack.c +1 -1
@@ -447,7 +447,7 @@ static void get_commons_through_negotiation(struct repository *r,strvec_pushl (&child.args,"fetch" ,"--negotiate-only" ,NULL );for (ref = remote_refs; ref; ref = ref->next) {if (!is_null_oid (&ref->new_oid)) { -strvec_pushf (&child.args,"--negotiation-tip=%s" , +strvec_pushf (&child.args,"--negotiation-restrict=%s" ,oid_to_hex (&ref->new_oid)); nr_negotiation_tip++; }
t/t5510-fetch.sh +25 -0
@@ -1460,6 +1460,31 @@ EOF test_cmp fatal-expect fatal-actual ' +test_expect_success '--negotiation-restrict limits "have" lines sent' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 --negotiation-restrict=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict understands globs' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=*_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-tip=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
t/t5702-protocol-v2.sh +2 -2
@@ -869,14 +869,14 @@ setup_negotiate_only () { test_commit -C client three } -test_expect_success 'usage: --negotiate-only without --negotiation-tip' ' +test_expect_success 'usage: --negotiate-only without --negotiation-restrict' ' SERVER="server" && URI="file://$(pwd)/server" && setup_negotiate_only "$SERVER" "$URI" && cat >err.expect <<-\EOF && - fatal: --negotiate-only needs one or more --negotiation-tip=* + fatal: --negotiate-only needs one or more --negotiation-restrict=* EOF test_must_fail git -c protocol.version=2 -C client fetch \
transport-helper.c +2 -1
@@ -755,7 +755,8 @@ static int fetch_refs(struct transport *transport, }if (data->transport_options.negotiation_tips) -warning ("Ignoring --negotiation-tip because the protocol does not support it." ); +warning (_ ("ignoring %s because the protocol does not support it." ), +"--negotiation-restrict" );if (data->fetch)return fetch_with_fetch (transport, nr_heads, to_fetch);
Signature · 1 line
gitgitgadget
PATCH v4 3/8 transport: rename negotiation_tips
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
The previous change added the --negotiation-restrict synonym for the --negotiation-tip option for 'git fetch'. In anticipation of adding a new option that behaves similarly but with distinct changes to its behavior, rename the internal representation of this data from 'negotiation_tips' to 'negotiation_restrict_tips'.
The 'tips' part is kept because this is an oid_array in the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data.
Also update the string_list used to store the inputs from command-line options.
--- builtin/fetch.c | 18 +++++++++--------- fetch-pack.c | 18 +++++++++--------- fetch-pack.h | 4 ++-- transport-helper.c | 2 +- transport.c | 10 +++++----- transport.h | 4 ++-- 6 files changed, 28 insertions(+), 28 deletions(-)
builtin/fetch.c +9 -9
@@ -98,7 +98,7 @@ static struct transport *gtransport;static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ; -static struct string_list negotiation_tip =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;
@@ -1534,13 +1534,13 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_tips (struct git_transport_options *smart_options) +static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) {struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i; -for (i =0 ; i < negotiation_tip.nr; i++) { -const char *s = negotiation_tip.items[i].string; +for (i =0 ; i < negotiation_restrict.nr; i++) { +const char *s = negotiation_restrict.items[i].string;struct refs_for_each_ref_options opts = { .pattern = s, };
@@ -1561,7 +1561,7 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)warning (_ ("ignoring %s=%s because it does not match any refs" ),"--negotiation-restrict" , s); } - smart_options->negotiation_tips = oids; + smart_options->negotiation_restrict_tips = oids; } static struct transport *prepare_transport(struct remote *remote, int deepen,
@@ -1595,9 +1595,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,set_option (transport,TRANS_OPT_LIST_OBJECTS_FILTER , spec);set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" ); } -if (negotiation_tip.nr) { +if (negotiation_restrict.nr) {if (transport->smart_options) -add_negotiation_tips (transport->smart_options); +add_negotiation_restrict_tips (transport->smart_options);else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );
@@ -2566,7 +2566,7 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), OPT_BOOL(0, "negotiate-only", &negotiate_only,
@@ -2658,7 +2658,7 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_tip.nr) +if (negotiate_only && !negotiation_restrict.nr)die (_ ("%s needs one or more %s" ),"--negotiate-only" ,"--negotiation-restrict=*" );
fetch-pack.c +9 -9
@@ -291,21 +291,21 @@ static int next_flush(int stateless_rpc, int count) }static void mark_tips (struct fetch_negotiator *negotiator, -const struct oid_array *negotiation_tips) +const struct oid_array *negotiation_restrict_tips) {struct refs_for_each_ref_options opts = { .flags =REFS_FOR_EACH_INCLUDE_BROKEN , };int i; -if (!negotiation_tips) { +if (!negotiation_restrict_tips) {refs_for_each_ref_ext (get_main_ref_store (the_repository), rev_list_insert_ref_oid, negotiator, &opts);return ; } -for (i =0 ; i < negotiation_tips->nr; i++) -rev_list_insert_ref (negotiator, &negotiation_tips->oid[i]); +for (i =0 ; i < negotiation_restrict_tips->nr; i++) +rev_list_insert_ref (negotiator, &negotiation_restrict_tips->oid[i]);return ; }
@@ -355,7 +355,7 @@ static int find_common(struct fetch_negotiator *negotiator,PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object); fetching =0 ;
@@ -1728,7 +1728,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,else state =FETCH_SEND_REQUEST ; -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object);break ;
@@ -2177,7 +2177,7 @@ static void clear_common_flag(struct oidset *s) } } -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
@@ -2195,13 +2195,13 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,timestamp_t min_generation =GENERATION_NUMBER_INFINITY ;fetch_negotiator_init (the_repository, &negotiator); -mark_tips (&negotiator, negotiation_tips); +mark_tips (&negotiator, negotiation_restrict_tips);packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE |PACKET_READ_DIE_ON_ERR_PACKET ); -oid_array_for_each ((struct oid_array *) negotiation_tips, +oid_array_for_each ((struct oid_array *) negotiation_restrict_tips, add_to_object_array, &nt_object_array);
fetch-pack.h +2 -2
@@ -21,7 +21,7 @@ struct fetch_pack_args {* If not NULL, during packfile negotiation, fetch-pack will send "have" * lines only with these tips and their ancestors. */ -const struct oid_array *negotiation_tips; +const struct oid_array *negotiation_restrict_tips; unsigned deepen_relative:1; unsigned quiet:1;
@@ -89,7 +89,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,* In the capability advertisement that has happened prior to invoking this * function, the "wait-for-done" capability must be present. */ -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
transport-helper.c +1 -1
@@ -754,7 +754,7 @@ static int fetch_refs(struct transport *transport,set_helper_option (transport,"filter" , spec); } -if (data->transport_options.negotiation_tips) +if (data->transport_options.negotiation_restrict_tips)warning (_ ("ignoring %s because the protocol does not support it." ),"--negotiation-restrict" );
transport.c +5 -5
@@ -463,7 +463,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.refetch = data->options.refetch; args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; - args.negotiation_tips = data->options.negotiation_tips; + args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -491,7 +491,7 @@ static int fetch_refs_via_pack(struct transport *transport,warning (_ ("server does not support wait-for-done" )); ret =-1 ; }else { - negotiate_using_fetch(data->options.negotiation_tips, + negotiate_using_fetch(data->options.negotiation_restrict_tips, transport->server_options, transport->stateless_rpc, data->fd,
@@ -979,9 +979,9 @@ static int disconnect_git(struct transport *transport)finish_connect (data->conn); } -if (data->options.negotiation_tips) { -oid_array_clear (data->options.negotiation_tips); -free (data->options.negotiation_tips); +if (data->options.negotiation_restrict_tips) { +oid_array_clear (data->options.negotiation_restrict_tips); +free (data->options.negotiation_restrict_tips); }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);
transport.h +2 -2
@@ -40,13 +40,13 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_tips in struct fetch_pack_args. +* negotiation_restrict_tips in struct fetch_pack_args. * * This field is only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ -struct oid_array *negotiation_tips; +struct oid_array *negotiation_restrict_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
Signature · 1 line
gitgitgadget
PATCH v4 4/8 remote: add remote.*.negotiationRestrict config
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tip. Both of these options restrict the set of 'haves' the client can send as part of negotiation.
This was previously not available via a configuration option. Add a new 'remote.<name>.negotiationRestrict' multi-valued config option that updates 'git fetch <name>' to use these restrictions by default.
If the user provides even one --negotiation-restrict argument, then the config is ignored.
An empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
--- Documentation/config/remote.adoc | 18 ++++++++++++++++++ builtin/fetch.c | 28 +++++++++++++++++++++------- remote.c | 5 +++++ remote.h | 1 + t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++ 5 files changed, 71 insertions(+), 7 deletions(-)
Documentation/config/remote.adoc +18 -0
@@ -107,6 +107,24 @@ priority configuration file (e.g. `.git/config` in a repository) to clear the values inherited from a lower priority configuration files (e.g. `$HOME/.gitconfig`). +remote.<name>.negotiationRestrict:: + When negotiating with this remote during `git fetch`, restrict the + commits advertised as "have" lines to only those reachable from refs + matching the given patterns. This multi-valued config option behaves + like `--negotiation-restrict` on the command line. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the +same as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-restrict` +command-line option. If `--negotiation-restrict` (or its synonym +`--negotiation-tip`) is specified on the command line, then the config +values are not used. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
builtin/fetch.c +21 -7
@@ -1601,6 +1601,19 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" ); + }else if (remote->negotiation_restrict.nr) { +struct string_list_item *item; +for_each_string_list_item (item, &remote->negotiation_restrict) +string_list_append (&negotiation_restrict, item->string); +if (transport->smart_options) +add_negotiation_restrict_tips (transport->smart_options); +else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
@@ -2658,10 +2671,6 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_restrict.nr) -die (_ ("%s needs one or more %s" ),"--negotiate-only" , -"--negotiation-restrict=*" ); -if (deepen_relative) {if (deepen_relative <0 )die (_ ("negative depth in --deepen is not supported" ));
@@ -2749,14 +2758,19 @@ int cmd_fetch(int argc,if (!remote)die (_ ("must supply remote when using --negotiate-only" )); gtransport =prepare_transport (remote,1 , &filter_options); -if (gtransport->smart_options) { - gtransport->smart_options->acked_commits = &acked_commits; - }else { + +if (!gtransport->smart_options) {warning (_ ("protocol does not support --negotiate-only, exiting" )); result =1 ;trace2_region_leave ("fetch" ,"negotiate-only" , the_repository); goto cleanup; } +if (!gtransport->smart_options->negotiation_restrict_tips) +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" ); + + gtransport->smart_options->acked_commits = &acked_commits; +if (server_options.nr) gtransport->server_options = &server_options; result =transport_fetch_refs (gtransport,NULL );
remote.c +5 -0
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_push (&ret->push);refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options); +string_list_init_dup (&ret->negotiation_restrict);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -179,6 +180,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy);FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 ); +string_list_clear (&remote->negotiation_restrict,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -562,6 +564,9 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"serveroption" )) {return parse_transport_option (key, value, &remote->server_options); + }else if (!strcmp (subkey,"negotiationrestrict" )) { +return parse_transport_option (key, value, + &remote->negotiation_restrict); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -117,6 +117,7 @@ struct remote {char *http_proxy_authmethod;struct string_list server_options; +struct string_list negotiation_restrict;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +26 -0
@@ -1485,6 +1485,32 @@ test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' check_negotiation_tip ' +test_expect_success 'remote.<name>.negotiationRestrict used as default' ' + setup_negotiation_tip server server 0 && + + # test the reset of the list on an empty value + git -C client config --add remote.origin.negotiationRestrict alpha_2 && + git -C client config --add remote.origin.negotiationRestrict "" && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success 'CLI --negotiation-restrict overrides remote config' ' + setup_negotiation_tip server server 0 && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + ALPHA_1=$(git -C client rev-parse alpha_1) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + origin alpha_s beta_s && + test_grep "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep ! "fetch> have $BETA_1" trace +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
Signature · 1 line
gitgitgadget
PATCH v4 5/8 negotiator: add have_sent() interface
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
In a future change, we will introduce a capability to choose specific commit OIDs as 'have's in fetch negotiation, with the ability to have the negotiator choose more 'have's to increase coverage beyond that required core set. The negotiator works to avoid emitting 'have's that can reach each other, but that logic is hidden beneath the negotiator's iterator function pointer ('next'). We need a way to communicate to the negotiator that we have picked a 'have' so it could incorporate that into its logic.
Add a have_sent() method to the fetch_negotiator interface. This is the signal that allows the negotiator to track the commit as already shown and can perform the proper bookkeeping to avoid emitting those objects or anything they can reach.
For our non-trivial negotiators, it is sufficient to mark these commits as common, so the implementation is quite simple. This logic will be exercised in the next change.
--- fetch-negotiator.h | 9 +++++++++ negotiator/default.c | 8 ++++++++ negotiator/noop.c | 7 +++++++ negotiator/skipping.c | 8 ++++++++ 4 files changed, 32 insertions(+)
fetch-negotiator.h +9 -0
@@ -47,6 +47,15 @@ struct fetch_negotiator {*/ int (*ack)(struct fetch_negotiator *,struct commit *); +/* +* Inform the negotiator that this commit has already been sent as +* a "have" line outside of the negotiator's control. The negotiator +* should avoid outputting it from next() and may use it to optimize +* further negotiation (e.g., by treating it and its ancestors as +* common). +*/ +void (*have_sent)(struct fetch_negotiator *,struct commit *); +void (*release)(struct fetch_negotiator *);/* internal use */
negotiator/default.c +8 -0
@@ -175,6 +175,13 @@ static int ack(struct fetch_negotiator *n, struct commit *c)return known_to_be_common; } +static void have_sent (struct fetch_negotiator *n,struct commit *c) +{ +if (repo_parse_commit (the_repository, c)) +return ; +mark_common (n->data, c,0 ,0 ); +} +static void release (struct fetch_negotiator *n) {clear_prio_queue (&((struct negotiation_state *)n->data)->rev_list);
@@ -188,6 +195,7 @@ void default_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =CALLOC_ARRAY (ns,1 ); ns->rev_list.compare = compare_commits_by_commit_date;
negotiator/noop.c +7 -0
@@ -29,6 +29,12 @@ static int ack(struct fetch_negotiator *n UNUSED, struct commit *c UNUSED)return 0 ; } +static void have_sent(struct fetch_negotiator *n UNUSED, + struct commit *c UNUSED) +{ +/* nothing to do */ +} + static void release(struct fetch_negotiator *n UNUSED) {/* nothing to release */
@@ -40,6 +46,7 @@ void noop_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =NULL ; }
negotiator/skipping.c +8 -0
@@ -243,6 +243,13 @@ static int ack(struct fetch_negotiator *n, struct commit *c)return known_to_be_common; } +static void have_sent (struct fetch_negotiator *n,struct commit *c) +{ +if (repo_parse_commit (the_repository, c)) +return ; +mark_common (n->data, c); +} +static void release (struct fetch_negotiator *n) {struct data *data = n->data;
@@ -259,6 +266,7 @@ void skipping_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =CALLOC_ARRAY (data,1 ); data->rev_list.compare = compare;
Signature · 1 line
gitgitgadget
PATCH v4 6/8 fetch: add --negotiation-include option for negotiation
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new --negotiation-include option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
This is useful when the repository has a large number of references, so the normal negotiation algorithm truncates the list. This is especially important in repositories with long parallel commit histories. For example, a repo could have a 'dev' branch for development and a 'release' branch for released versions. If the 'dev' branch isn't selected for negotiation, then it's not a big deal because there are many in-progress development branches with a shared history. However, if 'release' is not selected for negotiation, then the server may think that this is the first time the client has asked for that reference, causing a full download of its parallel commit history (and any extra data that may be unique to that branch). This is based on a real example where certain fetches would grow to 60+ GB when a release branch updated.
This option is a complement to --negotiation-restrict, which reduces the negotiation ref set to a specific list. In the earlier example, using --negotiation-restrict to focus the negotiation to 'dev' and 'release' would avoid those problematic downloads, but would still not allow advertising potentially-relevant user branches. In this way, the 'include' version solves the problem I mention while allowing negotiation to pick other references opportunistically. The two options can also be combined to allow the best of both worlds.
The argument may be an exact ref name or a glob pattern. Non-existent refs are silently ignored. This behavior is also updated in the ref matching logic for the related --negotiation-restrict option to match.
The implementation outputs the requested objects as haves before the negotiator performs its own algorithm to choose the next haves. Use the new have_sent() interface to signal these have commits were sent before engaging with the negotiator's next() iterator.
Also add --negotiation-include to 'git pull' passthrough options.
--- Documentation/fetch-options.adoc | 19 +++++++ builtin/fetch.c | 32 ++++++++--- builtin/pull.c | 3 ++ fetch-pack.c | 81 +++++++++++++++++++++++++--- fetch-pack.h | 6 ++- t/t5510-fetch.sh | 91 ++++++++++++++++++++++++++++++++ transport.c | 8 ++- transport.h | 5 +- 8 files changed, 227 insertions(+), 18 deletions(-)
Documentation/fetch-options.adoc +19 -0
@@ -73,6 +73,25 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate` configuration variables documented in linkgit:git-config[1], and the `--negotiate-only` option below. +`--negotiation-include=(<commit>|<glob>)`:: + Ensure that the commits at the given tips are always sent as "have" + lines during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-restrict` restricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++ +This option may be specified more than once; if so, each commit is sent +unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`), an +object hash, or a glob pattern (e.g. `refs/heads/release/{asterisk}`). +The pattern syntax is the same as for `--negotiation-restrict`. ++ +If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-include`. + `--negotiate-only`:: Do not fetch anything from the server, and instead print the ancestors of the provided `--negotiation-restrict=` arguments,
builtin/fetch.c +25 -7
@@ -99,6 +99,7 @@ static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ;static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_include =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;
@@ -1534,23 +1535,28 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) +static void add_negotiation_tips (struct string_list *input_list, +struct oid_array **output_list) {struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i; -for (i =0 ; i < negotiation_restrict.nr; i++) { -const char *s = negotiation_restrict.items[i].string; +for (i =0 ; i < input_list->nr; i++) { +const char *s = input_list->items[i].string;struct refs_for_each_ref_options opts = { .pattern = s, };int old_nr;if (!has_glob_specials (s)) {struct object_id oid; + +/* Ignore missing reference. */ if (repo_get_oid (the_repository, s, &oid)) -die (_ ("%s is not a valid object" ), s); +continue ; +/* Fail on missing object pointed by ref. */ if (!odb_has_object (the_repository->objects, &oid,0 ))die (_ ("the object %s does not exist" ), s); +oid_array_append (oids, &oid);continue ; }
@@ -1561,7 +1567,7 @@ static void add_negotiation_restrict_tips(struct git_transport_options *smart_opwarning (_ ("ignoring %s=%s because it does not match any refs" ),"--negotiation-restrict" , s); } - smart_options->negotiation_restrict_tips = oids; + *output_list = oids; } static struct transport *prepare_transport(struct remote *remote, int deepen,
@@ -1597,7 +1603,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen, }if (negotiation_restrict.nr) {if (transport->smart_options) -add_negotiation_restrict_tips (transport->smart_options); +add_negotiation_tips (&negotiation_restrict, + &transport->smart_options->negotiation_restrict_tips);else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );
@@ -1606,7 +1613,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,for_each_string_list_item (item, &remote->negotiation_restrict)string_list_append (&negotiation_restrict, item->string);if (transport->smart_options) -add_negotiation_restrict_tips (transport->smart_options); +add_negotiation_tips (&negotiation_restrict, + &transport->smart_options->negotiation_restrict_tips);else {struct strbuf config_name =STRBUF_INIT ;strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name);
@@ -1615,6 +1623,14 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,strbuf_release (&config_name); } } +if (negotiation_include.nr) { +if (transport->smart_options) +add_negotiation_tips (&negotiation_include, + &transport->smart_options->negotiation_include_tips); +else +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-include" ); + }return transport; }
@@ -2582,6 +2598,8 @@ int cmd_fetch(int argc,OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), +OPT_STRING_LIST (0 ,"negotiation-include" , &negotiation_include,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" )),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
builtin/pull.c +3 -0
@@ -1000,6 +1000,9 @@ int cmd_pull(int argc, N_("report that we have only objects reachable from this object"), 0),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-include" , &opt_fetch,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
fetch-pack.c +75 -6
186 lines not on this page · raw message
fetch-pack.h +4 -2
@@ -19,9 +19,10 @@ struct fetch_pack_args {/* * If not NULL, during packfile negotiation, fetch-pack will send "have" -* lines only with these tips and their ancestors. +* lines for all _include_ tips and then a subset of the _restrict_ tips. */ const struct oid_array *negotiation_restrict_tips; +const struct oid_array *negotiation_include_tips; unsigned deepen_relative:1; unsigned quiet:1;
@@ -93,7 +94,8 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,const struct string_list *server_options, int stateless_rpc, int fd[], - struct oidset *acked_commits); + struct oidset *acked_commits, + const struct oid_array *negotiation_include_tips);/* * Print an appropriate error message for each sought ref that wasn't
t/t5510-fetch.sh +91 -0
103 lines not on this page · raw message
transport.c +7 -1
@@ -464,6 +464,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; + args.negotiation_include_tips = data->options.negotiation_include_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -495,7 +496,8 @@ static int fetch_refs_via_pack(struct transport *transport, transport->server_options, transport->stateless_rpc, data->fd, - data->options.acked_commits); + data->options.acked_commits, + data->options.negotiation_include_tips); ret =0 ; } goto cleanup;
@@ -983,6 +985,10 @@ static int disconnect_git(struct transport *transport)oid_array_clear (data->options.negotiation_restrict_tips);free (data->options.negotiation_restrict_tips); } +if (data->options.negotiation_include_tips) { +oid_array_clear (data->options.negotiation_include_tips); +free (data->options.negotiation_include_tips); + }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);oid_array_clear (&data->shallow);
transport.h +3 -2
@@ -40,13 +40,14 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_restrict_tips in struct fetch_pack_args. +* these member names in struct fetch_pack_args. * -* This field is only supported by transports that support connect or +* These fields are only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ struct oid_array *negotiation_restrict_tips; +struct oid_array *negotiation_include_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
Signature · 1 line
gitgitgadget
PATCH v4 7/8 remote: add remote.*.negotiationInclude config
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new 'remote.<name>.negotiationInclude' multi-valued config option that provides default values for --negotiation-include when no --negotiation-include arguments are specified over the command line. This is a mirror of how 'remote.<name>.negotiationRestrict' specifies defaults for the --negotiation-restrict arguments.
Each value is either an exact ref name or a glob pattern whose tips should always be sent as 'have' lines during negotiation. The config values are resolved through the same resolve_negotiation_include() codepath as the CLI options.
This option is additive with the normal negotiation process: the negotiation algorithm still runs and advertises its own selected commits, but the refs matching the config are sent unconditionally on top of those heuristically selected commits.
Similar to the negotiationRestrict config, an empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
--- Documentation/config/remote.adoc | 27 ++++++++++++++++++ Documentation/fetch-options.adoc | 4 +++ builtin/fetch.c | 11 +++++++ remote.c | 5 ++++ remote.h | 1 + t/t5510-fetch.sh | 49 ++++++++++++++++++++++++++++++++ 6 files changed, 97 insertions(+)
Documentation/config/remote.adoc +27 -0
@@ -125,6 +125,33 @@ values are not used. Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios. +remote.<name>.negotiationInclude:: + When negotiating with this remote during `git fetch`, the client + advertises a list of commits that exist locally. In repos with + many references, this list of "haves" can be truncated. Depending + on data shape, dropping certain references may be expensive. This + multi-valued config option specifies references, commit hashes, + or ref pattern globs whose tips should always be sent as "have" + commits during fetch negotiation with this remote. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`), a +commit hash, or a glob pattern (e.g. `refs/heads/release/*`). The +pattern syntax is the same as for `--negotiation-include`. ++ +These config values are used as defaults for the `--negotiation-include` +command-line option. If `--negotiation-include` is specified on the +command line, then the config values are not used. ++ +This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits, +but the refs matching `remote.<name>.negotiationInclude` are sent +unconditionally on top of those heuristically selected commits. This +option is also used during push negotiation when `push.negotiate` is +enabled. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
Documentation/fetch-options.adoc +4 -0
@@ -91,6 +91,10 @@ The pattern syntax is the same as for `--negotiation-restrict`. If `--negotiation-restrict` is used, the have set is first restricted by that option and then increased to include the tips specified by `--negotiation-include`. ++ +If this option is not specified on the command line, then any +`remote.<name>.negotiationInclude` config values for the current remote +are used instead. `--negotiate-only`:: Do not fetch anything from the server, and instead print the
builtin/fetch.c +11 -0
@@ -1630,6 +1630,17 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-include" ); + }else if (remote->negotiation_include.nr) { +if (transport->smart_options) { +add_negotiation_tips (&remote->negotiation_include, + &transport->smart_options->negotiation_include_tips); + }else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationInclude" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
remote.c +5 -0
@@ -153,6 +153,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options);string_list_init_dup (&ret->negotiation_restrict); +string_list_init_dup (&ret->negotiation_include);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -181,6 +182,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 );string_list_clear (&remote->negotiation_restrict,0 ); +string_list_clear (&remote->negotiation_include,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -567,6 +569,9 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"negotiationrestrict" )) {return parse_transport_option (key, value, &remote->negotiation_restrict); + }else if (!strcmp (subkey,"negotiationinclude" )) { +return parse_transport_option (key, value, + &remote->negotiation_include); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -118,6 +118,7 @@ struct remote {struct string_list server_options;struct string_list negotiation_restrict; +struct string_list negotiation_include;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +49 -0
55 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v4 8/8 send-pack: pass negotiation config in push
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
When push.negotiate is enabled, 'git push' spawns a child 'git fetch --negotiate-only' process to find common commits. Pass --negotiation-include and --negotiation-restrict options from the 'remote.<name>.negotiationInclude' and 'remote.<name>.negotiationRestrict' config keys to this child process.
When negotiationRestrict is configured, it replaces the default behavior of using all remote refs as negotiation tips. This allows the user to control which local refs are used for push negotiation.
When negotiationInclude is configured, the specified ref patterns are passed as --negotiation-include to ensure their tips are always sent as 'have' lines during push negotiation.
This change also updates the use of --negotiation-tip into --negotiation-restrict now that the new synonym exists.
--- Documentation/config/remote.adoc | 6 ++++++ send-pack.c | 37 ++++++++++++++++++++++++++------ send-pack.h | 2 ++ t/t5516-fetch-push.sh | 30 ++++++++++++++++++++++++++ transport.c | 2 ++ 5 files changed, 70 insertions(+), 7 deletions(-)
Documentation/config/remote.adoc +6 -0
@@ -122,6 +122,9 @@ command-line option. If `--negotiation-restrict` (or its synonym `--negotiation-tip`) is specified on the command line, then the config values are not used. + +These values also influence negotiation during `git push` if +`push.negotiate` is enabled. ++ Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios.
@@ -149,6 +152,9 @@ unconditionally on top of those heuristically selected commits. This option is also used during push negotiation when `push.negotiate` is enabled. + +These values also influence negotiation during `git push` if +`push.negotiate` is enabled. ++ Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios.
send-pack.c +30 -7
@@ -433,28 +433,48 @@ static void reject_invalid_nonce(const char *nonce, int len)static void get_commons_through_negotiation (struct repository *r,const char *url, +const struct string_list *negotiation_include, +const struct string_list *negotiation_restrict,const struct ref *remote_refs,struct oid_array *commons) {struct child_process child =CHILD_PROCESS_INIT ;const struct ref *ref;int len = r->hash_algo->hexsz +1 ;/* hash + NL */ -int nr_negotiation_tip =0 ; +int nr_negotiation =0 ; child.git_cmd =1 ; child.no_stdin =1 ; child.out =-1 ;strvec_pushl (&child.args,"fetch" ,"--negotiate-only" ,NULL ); -for (ref = remote_refs; ref; ref = ref->next) { -if (!is_null_oid (&ref->new_oid)) { + +if (negotiation_restrict && negotiation_restrict->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_restrict)strvec_pushf (&child.args,"--negotiation-restrict=%s" , -oid_to_hex (&ref->new_oid)); - nr_negotiation_tip++; + item->string); + nr_negotiation = negotiation_restrict->nr; + }else { +for (ref = remote_refs; ref; ref = ref->next) { +if (!is_null_oid (&ref->new_oid)) { +strvec_pushf (&child.args,"--negotiation-restrict=%s" , +oid_to_hex (&ref->new_oid)); + nr_negotiation++; + } } } + +if (negotiation_include && negotiation_include->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_include) +strvec_pushf (&child.args,"--negotiation-include=%s" , + item->string); + nr_negotiation += negotiation_include->nr; + } +strvec_push (&child.args, url); -if (!nr_negotiation_tip) { +if (!nr_negotiation) {child_process_clear (&child);return ; }
@@ -528,7 +548,10 @@ int send_pack(struct repository *r,repo_config_get_bool (r,"push.negotiate" , &push_negotiate);if (push_negotiate) {trace2_region_enter ("send_pack" ,"push_negotiate" , r); -get_commons_through_negotiation (r, args->url, remote_refs, &commons); +get_commons_through_negotiation (r, args->url, + args->negotiation_include, + args->negotiation_restrict, + remote_refs, &commons);trace2_region_leave ("send_pack" ,"push_negotiate" , r); }
send-pack.h +2 -0
@@ -18,6 +18,8 @@ struct repository;struct send_pack_args {const char *url; +const struct string_list *negotiation_include; +const struct string_list *negotiation_restrict;unsigned verbose:1 , quiet:1 , porcelain:1,
t/t5516-fetch-push.sh +30 -0
@@ -254,6 +254,36 @@ test_expect_success 'push with negotiation does not attempt to fetch submodules' ! grep "Fetching submodule" err ' +test_expect_success 'push with negotiation and remote.<name>.negotiationInclude' ' + test_when_finished rm -rf negotiation_include && + mk_empty negotiation_include && + git push negotiation_include $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_include unrelated_commit && + git -C negotiation_include config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_include.negotiationInclude=refs/heads/main \ + push negotiation_include refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + +test_expect_success 'push with negotiation and remote.<name>.negotiationRestrict' ' + test_when_finished rm -rf negotiation_restrict && + mk_empty negotiation_restrict && + git push negotiation_restrict $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_restrict unrelated_commit && + git -C negotiation_restrict config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_restrict.negotiationRestrict=refs/heads/main \ + push negotiation_restrict refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + test_expect_success 'push without wildcard' ' mk_empty testrepo &&
transport.c +2 -0
@@ -921,6 +921,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re args.atomic = !!(flags &TRANSPORT_PUSH_ATOMIC ); args.push_options = transport->push_options; args.url = transport->url; + args.negotiation_include = &transport->remote->negotiation_include; + args.negotiation_restrict = &transport->remote->negotiation_restrict;if (flags &TRANSPORT_PUSH_CERT_ALWAYS ) args.push_cert =SEND_PACK_PUSH_CERT_ALWAYS ;
Signature · 1 line
gitgitgadget
Re: [PATCH v4 1/8] t5516: fix test order flakiness
In reply to Derrick Stolee via GitGitGadgetQuoted 9 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
The 'fetch follows tags by default' test sorts using 'sort -k 4', but for-each-ref output only has 3 columns. This relies on sort treating records with fewer fields as having an empty fourth field, which may produce unstable results depending on locale. This appears to be an accident added in
3f763ddf28(fetch: set remote/HEAD if it does not exist, 2024-11-22).Use 'sort -k 3' to match the actual number of columns in the output.
Expanding the message to back reference 3f763ddf28 is a nice change. Makes it easier for future people reading the history to follow back.
Quoted 10 lines
Signed-off-by Derrick Stolee <[email protected]>--- t/t5516-fetch-push.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)t/t5516-fetch-push.sh +0
-0
t/t5516-fetch-push.sh test_expect_success 'fetch follows tags by default' ' · line 1355
Quoted 8 lines
t/t5516-fetch-push.sh +1
-1@@ -1349,7 +1349,7 @@ test_expect_success 'fetch follows tags by default' ' git for-each-ref >tmp1 && sed -n "p; s|refs/heads/main$|refs/remotes/origin/main|p" tmp1 | sed -n "p; s|refs/heads/main$|refs/remotes/origin/HEAD|p" | - sort -k 4 >../expect + sort -k 3 >../expect ) && test_when_finished "rm -rf dst" && git init dst &&
Unchanged from v3 and still LGTM!
Thanks,
Matthew
Re: [PATCH v4 2/8] fetch: add --negotiation-restrict option
In reply to Derrick Stolee via GitGitGadgetQuoted 41 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
The --negotiation-tip option to 'git fetch' and 'git pull' allows users to specify that they want to focus negotiation on a small set of references. This is a _restriction_ on the negotiation set, helping to focus the negotiation when the ref count is high. However, it doesn't allow for the ability to opportunistically select references beyond that list.
This subtle detail that this is a 'maximum set' and not a 'minimum set' is not immediately clear from the option name. This makes it more complicated to add a new option that provides the complementary behavior of a minimum set.
For now, create a new synonym option, --negotiation-restrict, that behaves identically to --negotiation-tip. Update the documentation to make it clear that this new name is the preferred option, but we keep the old name for compatibility. Mark --negotiation-tip as an alias of the new, preferred option.
Update a few warning messages with the new option, but also make them translatable with the option name inserted by formatting. At least one of these messages will be reused later for a new option.
Signed-off-by Derrick Stolee <[email protected]>--- Documentation/config/fetch.adoc | 2 +- Documentation/fetch-options.adoc | 6 +++++- builtin/fetch.c | 13 ++++++++----- builtin/pull.c | 3 ++- send-pack.c | 2 +- t/t5510-fetch.sh | 25 +++++++++++++++++++++++++ t/t5702-protocol-v2.sh | 4 ++-- transport-helper.c | 3 ++- 8 files changed, 46 insertions(+), 12 deletions(-)Documentation/config/fetch.adoc +0
-0
Documentation/config/fetch.adoc line 80
Quoted 6 lines
Documentation/config/fetch.adoc +1
-1@@ -76,7 +76,7 @@ default is `skipping`. Unknown values will cause `git fetch` to error out. + -See also the `--negotiate-only` and `--negotiation-tip` options to +See also the `--negotiate-only` and `--negotiation-restrict` options to linkgit:git-fetch[1].
Good - this addressed my nit in v4 about mentioning the old name here.
Quoted 24 lines
Documentation/config/fetch.adoc +0
-0`fetch.showForcedUpdates`::Documentation/fetch-options.adoc +4
-0@@ -49,6 +49,7 @@ the current repository has the same history as the source repository. `.git/shallow`. This option updates `.git/shallow` and accepts such refs. +`--negotiation-restrict=(<commit>|<glob>)`:: `--negotiation-tip=(<commit>|<glob>)`:: By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt to@@ -58,6 +59,9 @@ the current repository has the same history as the source repository. local ref is likely to have commits in common with the upstream ref being fetched. + +`--negotiation-restrict` is the preferred name for this option; +`--negotiation-tip` is accepted as a synonym. ++ This option may be specified more than once; if so, Git will report commits reachable from any of the given commits. +
Documentation/fetch-options.adoc configuration variables documented in linkgit:git-config[1], and the · line 81
Quoted 8 lines
Documentation/fetch-options.adoc +1
-1@@ -71,7 +75,7 @@ configuration variables documented in linkgit:git-config[1], and the `--negotiate-only`:: Do not fetch anything from the server, and instead print the - ancestors of the provided `--negotiation-tip=` arguments, + ancestors of the provided `--negotiation-restrict=` arguments, which we have in common with the server. + This is incompatible with `--recurse-submodules=(yes|on-demand)`.
Good! The --negotiate-only paragraph now uses --negotiate-restrict too.
Quoted 37 lines
builtin/fetch.c +6
-4@@ -1558,8 +1558,8 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)refs_for_each_ref_ext (get_main_ref_store (the_repository), add_oid, oids, &opts);if (old_nr == oids->nr) -warning ("ignoring --negotiation-tip=%s because it does not match any refs" , - s); +warning (_ ("ignoring %s=%s because it does not match any refs" ), +"--negotiation-restrict" , s); } smart_options->negotiation_tips = oids; }@@ -1599,7 +1599,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,if (transport->smart_options)add_negotiation_tips (transport->smart_options);else -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-restrict" ); }return transport; }@@ -2565,8 +2566,9 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
builtin/fetch.c int cmd_fetch(int argc, · line 2666
Quoted 9 lines
builtin/fetch.c +2
-1@@ -2657,7 +2659,8 @@ int cmd_fetch(int argc, }if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-tip=*" )); +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" );if (deepen_relative) {if (deepen_relative <0 )
Unchanged from v3: OPT_ALIAS keeps back-compat and messages are i18n friendly. Nice.
Quoted 6 lines
builtin/pull.c +0
-0@@ -996,9 +996,10 @@ int cmd_pull(int argc, OPT_PASSTHRU('6', "ipv6", &opt_ipv6, NULL,
builtin/pull.c int cmd_pull(int argc, · line 1005
Quoted 10 lines
builtin/pull.c +2
-1N_("use IPv6 addresses only"), PARSE_OPT_NOARG), -OPT_PASSTHRU_ARGV (0 ,"negotiation-tip" , &opt_fetch,N_ ("revision" ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
Oh nice! v3 had two OPT_PASSTHRU_ARGV (one for each name). Now we're using OPT_ALIAS here too for the old name which avoids duplicated description strings - yay!
I had to convince myself this works (does OPT_ALIAS forward to the passthru option?). Looking at parse-options.c, preprocess_options() substitutes the alias with a copy of the source option keeping only the alias's long_name. So really the child proc will get the old --negotiate-tip name but the child has an alias anyway.. works fine.
send-pack.c static void get_commons_through_negotiation(struct repository *r, · line 453
Quoted 13 lines
send-pack.c +1
-1@@ -447,7 +447,7 @@ static void get_commons_through_negotiation(struct repository *r,strvec_pushl (&child.args,"fetch" ,"--negotiate-only" ,NULL );for (ref = remote_refs; ref; ref = ref->next) {if (!is_null_oid (&ref->new_oid)) { -strvec_pushf (&child.args,"--negotiation-tip=%s" , +strvec_pushf (&child.args,"--negotiation-restrict=%s" ,oid_to_hex (&ref->new_oid)); nr_negotiation_tip++; }
Good. v3 was using the old name when shelling out (just a nit), but nicer to have the rename fully consistent.
Quoted 47 lines
t/t5510-fetch.sh +25
-0@@ -1460,6 +1460,31 @@ EOF test_cmp fatal-expect fatal-actual ' +test_expect_success '--negotiation-restrict limits "have" lines sent' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 --negotiation-restrict=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict understands globs' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=*_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-tip=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (t/t5702-protocol-v2.sh +1
-1@@ -869,14 +869,14 @@ setup_negotiate_only () { test_commit -C client three } -test_expect_success 'usage: --negotiate-only without --negotiation-tip' ' +test_expect_success 'usage: --negotiate-only without --negotiation-restrict' ' SERVER="server" &&
t/t5702-protocol-v2.sh setup_negotiate_only () { · line 882
Quoted 10 lines
t/t5702-protocol-v2.sh +1
-1URI="file://$(pwd)/server" && setup_negotiate_only "$SERVER" "$URI" && cat >err.expect <<-\EOF && - fatal: --negotiate-only needs one or more --negotiation-tip=* + fatal: --negotiate-only needs one or more --negotiation-restrict=* EOF test_must_fail git -c protocol.version=2 -C client fetch \
Unchanged from v3 - still good!
transport-helper.c static int fetch_refs(struct transport *transport, · line 759
Quoted 11 lines
transport-helper.c +2
-1@@ -755,7 +755,8 @@ static int fetch_refs(struct transport *transport, }if (data->transport_options.negotiation_tips) -warning ("Ignoring --negotiation-tip because the protocol does not support it." ); +warning (_ ("ignoring %s because the protocol does not support it." ), +"--negotiation-restrict" );
Good - picks up a missed rename from v3.
transport-helper.c static int fetch_refs(struct transport *transport, · line 762
Quoted 2 lines
transport-helper.c +0
-0if (data->fetch)return fetch_with_fetch (transport, nr_heads, to_fetch);
Overall this patch LGTM! All my v3 issues were addressed.
Thanks,
Matthew
Re: [PATCH v4 3/8] transport: rename negotiation_tips
In reply to Derrick Stolee via GitGitGadgetQuoted 7 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
The previous change added the --negotiation-restrict synonym for the --negotiation-tip option for 'git fetch'. In anticipation of adding a new option that behaves similarly but with distinct changes to its behavior, rename the internal representation of this data from '
negotiation_tips' to 'negotiation_restrict_tips'.
The typo (plural vs singular 'tips') is fixed - thanks!
Quoted 48 lines
The 'tips' part is kept because this is an
oid_arrayin the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data.Also update the
string_listused to store the inputs from command-line options.Signed-off-by Derrick Stolee <[email protected]>--- builtin/fetch.c | 18 +++++++++--------- fetch-pack.c | 18 +++++++++--------- fetch-pack.h | 4 ++-- transport-helper.c | 2 +- transport.c | 10 +++++----- transport.h | 4 ++-- 6 files changed, 28 insertions(+), 28 deletions(-)builtin/fetch.c +4
-4@@ -98,7 +98,7 @@ static struct transport *gtransport;static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ; -static struct string_list negotiation_tip =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;@@ -1534,13 +1534,13 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_tips (struct git_transport_options *smart_options) +static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) {struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i; -for (i =0 ; i < negotiation_tip.nr; i++) { -const char *s = negotiation_tip.items[i].string; +for (i =0 ; i < negotiation_restrict.nr; i++) { +const char *s = negotiation_restrict.items[i].string;struct refs_for_each_ref_options opts = { .pattern = s, };
builtin/fetch.c static void add_negotiation_tips(struct git_transport_options *smart_options) · line 1565
Quoted 6 lines
builtin/fetch.c +1
-1@@ -1561,7 +1561,7 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)warning (_ ("ignoring %s=%s because it does not match any refs" ),"--negotiation-restrict" , s); } - smart_options->negotiation_tips = oids; + smart_options->negotiation_restrict_tips = oids; }
Same as in v3, this function will be refactored in a later patch, so just doing the rename now keeps things reviewable - thanks!
Quoted 23 lines
builtin/fetch.c +3
-3static struct transport *prepare_transport(struct remote *remote, int deepen,@@ -1595,9 +1595,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,set_option (transport,TRANS_OPT_LIST_OBJECTS_FILTER , spec);set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" ); } -if (negotiation_tip.nr) { +if (negotiation_restrict.nr) {if (transport->smart_options) -add_negotiation_tips (transport->smart_options); +add_negotiation_restrict_tips (transport->smart_options);else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );@@ -2566,7 +2566,7 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), OPT_BOOL(0, "negotiate-only", &negotiate_only,
builtin/fetch.c int cmd_fetch(int argc, · line 2663
Quoted 7 lines
builtin/fetch.c +1
-1@@ -2658,7 +2658,7 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_tip.nr) +if (negotiate_only && !negotiation_restrict.nr)die (_ ("%s needs one or more %s" ),"--negotiate-only" ,"--negotiation-restrict=*" );
Simple rename - looks good!
Quoted 153 lines
fetch-pack.c +9
-9@@ -291,21 +291,21 @@ static int next_flush(int stateless_rpc, int count) }static void mark_tips (struct fetch_negotiator *negotiator, -const struct oid_array *negotiation_tips) +const struct oid_array *negotiation_restrict_tips) {struct refs_for_each_ref_options opts = { .flags =REFS_FOR_EACH_INCLUDE_BROKEN , };int i; -if (!negotiation_tips) { +if (!negotiation_restrict_tips) {refs_for_each_ref_ext (get_main_ref_store (the_repository), rev_list_insert_ref_oid, negotiator, &opts);return ; } -for (i =0 ; i < negotiation_tips->nr; i++) -rev_list_insert_ref (negotiator, &negotiation_tips->oid[i]); +for (i =0 ; i < negotiation_restrict_tips->nr; i++) +rev_list_insert_ref (negotiator, &negotiation_restrict_tips->oid[i]);return ; }@@ -355,7 +355,7 @@ static int find_common(struct fetch_negotiator *negotiator,PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object); fetching =0 ;@@ -1728,7 +1728,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,else state =FETCH_SEND_REQUEST ; -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object);break ;@@ -2177,7 +2177,7 @@ static void clear_common_flag(struct oidset *s) } } -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],@@ -2195,13 +2195,13 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,timestamp_t min_generation =GENERATION_NUMBER_INFINITY ;fetch_negotiator_init (the_repository, &negotiator); -mark_tips (&negotiator, negotiation_tips); +mark_tips (&negotiator, negotiation_restrict_tips);packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE |PACKET_READ_DIE_ON_ERR_PACKET ); -oid_array_for_each ((struct oid_array *) negotiation_tips, +oid_array_for_each ((struct oid_array *) negotiation_restrict_tips, add_to_object_array, &nt_object_array);fetch-pack.h +2
-2@@ -21,7 +21,7 @@ struct fetch_pack_args {* If not NULL, during packfile negotiation, fetch-pack will send "have" * lines only with these tips and their ancestors. */ -const struct oid_array *negotiation_tips; +const struct oid_array *negotiation_restrict_tips; unsigned deepen_relative:1; unsigned quiet:1;@@ -89,7 +89,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,* In the capability advertisement that has happened prior to invoking this * function, the "wait-for-done" capability must be present. */ -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],transport-helper.c +1
-1@@ -754,7 +754,7 @@ static int fetch_refs(struct transport *transport,set_helper_option (transport,"filter" , spec); } -if (data->transport_options.negotiation_tips) +if (data->transport_options.negotiation_restrict_tips)warning (_ ("ignoring %s because the protocol does not support it." ),"--negotiation-restrict" );transport.c +5
-5@@ -463,7 +463,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.refetch = data->options.refetch; args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; - args.negotiation_tips = data->options.negotiation_tips; + args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {@@ -491,7 +491,7 @@ static int fetch_refs_via_pack(struct transport *transport,warning (_ ("server does not support wait-for-done" )); ret =-1 ; }else { - negotiate_using_fetch(data->options.negotiation_tips, + negotiate_using_fetch(data->options.negotiation_restrict_tips, transport->server_options, transport->stateless_rpc, data->fd,@@ -979,9 +979,9 @@ static int disconnect_git(struct transport *transport)finish_connect (data->conn); } -if (data->options.negotiation_tips) { -oid_array_clear (data->options.negotiation_tips); -free (data->options.negotiation_tips); +if (data->options.negotiation_restrict_tips) { +oid_array_clear (data->options.negotiation_restrict_tips); +free (data->options.negotiation_restrict_tips); }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);transport.h +1
-1@@ -40,13 +40,13 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_tips in struct fetch_pack_args. +* negotiation_restrict_tips in struct fetch_pack_args.
transport.h struct git_transport_options { · line 52
Quoted 10 lines
transport.h +1
-1* * This field is only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ -struct oid_array *negotiation_tips; +struct oid_array *negotiation_restrict_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
All just simple renames - looks good to me! This patch is good.
Thanks,
Matthew
Re: [PATCH v4 4/8] remote: add remote.*.negotiationRestrict config
In reply to Derrick Stolee via GitGitGadgetQuoted 44 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tip. Both of these options restrict the set of 'haves' the client can send as part of negotiation.
This was previously not available via a configuration option. Add a new 'remote.<name>.negotiationRestrict' multi-valued config option that updates 'git fetch <name>' to use these restrictions by default.
If the user provides even one --negotiation-restrict argument, then the config is ignored.
An empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
Signed-off-by Derrick Stolee <[email protected]>--- Documentation/config/remote.adoc | 18 ++++++++++++++++++ builtin/fetch.c | 28 +++++++++++++++++++++------- remote.c | 5 +++++ remote.h | 1 + t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++ 5 files changed, 71 insertions(+), 7 deletions(-)Documentation/config/remote.adoc +11
-0@@ -107,6 +107,24 @@ priority configuration file (e.g. `.git/config` in a repository) to clear the values inherited from a lower priority configuration files (e.g. `$HOME/.gitconfig`). +remote.<name>.negotiationRestrict:: + When negotiating with this remote during `git fetch`, restrict the + commits advertised as "have" lines to only those reachable from refs + matching the given patterns. This multi-valued config option behaves + like `--negotiation-restrict` on the command line. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the +same as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-restrict`
Documentation/config/remote.adoc priority configuration file (e.g. `.git/config` in a repository) to clear · line 130
Quoted 10 lines
Documentation/config/remote.adoc +7
-0+command-line option. If `--negotiation-restrict` (or its synonym +`--negotiation-tip`) is specified on the command line, then the config +values are not used. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
Good - now we don't talk about 'git push' before we have wired that functionality up.
Quoted 14 lines
builtin/fetch.c +6
-0@@ -1601,6 +1601,19 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" ); + }else if (remote->negotiation_restrict.nr) { +struct string_list_item *item; +for_each_string_list_item (item, &remote->negotiation_restrict) +string_list_append (&negotiation_restrict, item->string); +if (transport->smart_options) +add_negotiation_restrict_tips (transport->smart_options);
builtin/fetch.c static struct transport *prepare_transport(struct remote *remote, int deepen, · line 1619
Quoted 10 lines
builtin/fetch.c +7
-0+else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
Unchanged from v3 and still reads cleanly. Good.
Quoted 24 lines
builtin/fetch.c +2
-7@@ -2658,10 +2671,6 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_restrict.nr) -die (_ ("%s needs one or more %s" ),"--negotiate-only" , -"--negotiation-restrict=*" ); -if (deepen_relative) {if (deepen_relative <0 )die (_ ("negative depth in --deepen is not supported" ));@@ -2749,14 +2758,19 @@ int cmd_fetch(int argc,if (!remote)die (_ ("must supply remote when using --negotiate-only" )); gtransport =prepare_transport (remote,1 , &filter_options); -if (gtransport->smart_options) { - gtransport->smart_options->acked_commits = &acked_commits; - }else { + +if (!gtransport->smart_options) {warning (_ ("protocol does not support --negotiate-only, exiting" )); result =1 ;trace2_region_leave ("fetch" ,"negotiate-only" , the_repository); goto cleanup;
builtin/fetch.c int cmd_fetch(int argc, · line 2776
Quoted 10 lines
builtin/fetch.c +6
-0} +if (!gtransport->smart_options->negotiation_restrict_tips) +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" ); + + gtransport->smart_options->acked_commits = &acked_commits; +if (server_options.nr) gtransport->server_options = &server_options; result =transport_fetch_refs (gtransport,NULL );
This is nice! Addressed my concerns in v3 about the wrong message being shown, and makes it clearer with an up-front check for the protocol support.
Quoted 21 lines
remote.c +2
-0@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_push (&ret->push);refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options); +string_list_init_dup (&ret->negotiation_restrict);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);@@ -179,6 +180,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy);FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 ); +string_list_clear (&remote->negotiation_restrict,0 ); }static void add_merge (struct branch *branch,const char *name)
remote.c static int handle_config(const char *key, const char *value, · line 572
Quoted 9 lines
remote.c +3
-0@@ -562,6 +564,9 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"serveroption" )) {return parse_transport_option (key, value, &remote->server_options); + }else if (!strcmp (subkey,"negotiationrestrict" )) { +return parse_transport_option (key, value, + &remote->negotiation_restrict); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
Good - reusing the parse_transport_option() as suggested makes this a smaller diff and consistent with 'serveroption' above.
Quoted 39 lines
remote.h +1
-0@@ -117,6 +117,7 @@ struct remote {char *http_proxy_authmethod;struct string_list server_options; +struct string_list negotiation_restrict;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;t/t5510-fetch.sh +19
-0@@ -1485,6 +1485,32 @@ test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' check_negotiation_tip ' +test_expect_success 'remote.<name>.negotiationRestrict used as default' ' + setup_negotiation_tip server server 0 && + + # test the reset of the list on an empty value + git -C client config --add remote.origin.negotiationRestrict alpha_2 && + git -C client config --add remote.origin.negotiationRestrict "" && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success 'CLI --negotiation-restrict overrides remote config' ' + setup_negotiation_tip server server 0 && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + ALPHA_1=$(git -C client rev-parse alpha_1) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
t/t5510-fetch.sh test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' · line 1516
Quoted 10 lines
t/t5510-fetch.sh +7
-0+ --negotiation-restrict=alpha_1 \ + origin alpha_s beta_s && + test_grep "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep ! "fetch> have $BETA_1" trace +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
Again, this patch looks good to me and addresses my previous comments.
Thanks,
Matthew
Re: [PATCH v4 5/8] negotiator: add have_sent() interface
In reply to Derrick Stolee via GitGitGadgetQuoted 15 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
In a future change, we will introduce a capability to choose specific commit OIDs as 'have's in fetch negotiation, with the ability to have the negotiator choose more 'have's to increase coverage beyond that required core set. The negotiator works to avoid emitting 'have's that can reach each other, but that logic is hidden beneath the negotiator's iterator function pointer ('next'). We need a way to communicate to the negotiator that we have picked a 'have' so it could incorporate that into its logic.
Add a
have_sent()method to thefetch_negotiatorinterface. This is the signal that allows the negotiator to track the commit as already shown and can perform the proper bookkeeping to avoid emitting those objects or anything they can reach.
Quoted 5 lines
For our non-trivial negotiators, it is sufficient to mark these commits as common, so the implementation is quite simple. This logic will be exercised in the next change.
Signed-off-by Derrick Stolee <[email protected]>
This is a new patch in v4, and is in response to the COMMON (bit 2) vs COMMON (bit 6) issue of v3.
Having a defined have_sent() API on the negotiator, and letting the negotiator handle its own book-keeping is much nicer and safer.
Quoted 14 lines
--- fetch-negotiator.h | 9 +++++++++ negotiator/default.c | 8 ++++++++ negotiator/noop.c | 7 +++++++ negotiator/skipping.c | 8 ++++++++ 4 files changed, 32 insertions(+)fetch-negotiator.h +0
-0@@ -47,6 +47,15 @@ struct fetch_negotiator {*/ int (*ack)(struct fetch_negotiator *,struct commit *);
fetch-negotiator.h struct fetch_negotiator { · line 58
Quoted 10 lines
fetch-negotiator.h +9
-0+/* +* Inform the negotiator that this commit has already been sent as +* a "have" line outside of the negotiator's control. The negotiator +* should avoid outputting it from next() and may use it to optimize +* further negotiation (e.g., by treating it and its ancestors as +* common). +*/ +void (*have_sent)(struct fetch_negotiator *,struct commit *); +
Doc comment captures the contract well.
Quoted 22 lines
fetch-negotiator.h +0
-0void (*release)(struct fetch_negotiator *);/* internal use */ negotiator/default.c +7
-0@@ -175,6 +175,13 @@ static int ack(struct fetch_negotiator *n, struct commit *c)return known_to_be_common; } +static void have_sent (struct fetch_negotiator *n,struct commit *c) +{ +if (repo_parse_commit (the_repository, c)) +return ; +mark_common (n->data, c,0 ,0 ); +} +static void release (struct fetch_negotiator *n) {clear_prio_queue (&((struct negotiation_state *)n->data)->rev_list);
negotiator/default.c void default_negotiator_init(struct fetch_negotiator *negotiator) · line 201
Quoted 7 lines
negotiator/default.c +1
-0@@ -188,6 +195,7 @@ void default_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =CALLOC_ARRAY (ns,1 ); ns->rev_list.compare = compare_commits_by_commit_date;
I traced this through default.c's mark_common() to confirm this does what we want.. sets the COMMON bit (bit 2) and propogates COMMON to the ancestors correctly. Good!
Quoted 18 lines
negotiator/noop.c +6
-0@@ -29,6 +29,12 @@ static int ack(struct fetch_negotiator *n UNUSED, struct commit *c UNUSED)return 0 ; } +static void have_sent(struct fetch_negotiator *n UNUSED, + struct commit *c UNUSED) +{ +/* nothing to do */ +} + static void release(struct fetch_negotiator *n UNUSED) {/* nothing to release */
negotiator/noop.c void noop_negotiator_init(struct fetch_negotiator *negotiator) · line 52
Quoted 7 lines
negotiator/noop.c +1
-0@@ -40,6 +46,7 @@ void noop_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =NULL ; }
Trivial implementation of the noop negotiator. Good.
Quoted 19 lines
negotiator/skipping.c +7
-0@@ -243,6 +243,13 @@ static int ack(struct fetch_negotiator *n, struct commit *c)return known_to_be_common; } +static void have_sent (struct fetch_negotiator *n,struct commit *c) +{ +if (repo_parse_commit (the_repository, c)) +return ; +mark_common (n->data, c); +} +static void release (struct fetch_negotiator *n) {struct data *data = n->data;
negotiator/skipping.c void skipping_negotiator_init(struct fetch_negotiator *negotiator) · line 272
Quoted 7 lines
negotiator/skipping.c +1
-0@@ -259,6 +266,7 @@ void skipping_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =CALLOC_ARRAY (data,1 ); data->rev_list.compare = compare;
I traced this too, to skipping.c's mark_common(), and it sets the COMMON bit correctly.
This patch is a welcome addition and LGTM!
Thanks,
Matthew
Re: [PATCH v4 6/8] fetch: add --negotiation-include option for negotiation
In reply to Derrick Stolee via GitGitGadgetQuoted 20 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
Add a new --negotiation-include option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
This is useful when the repository has a large number of references, so the normal negotiation algorithm truncates the list. This is especially important in repositories with long parallel commit histories. For example, a repo could have a 'dev' branch for development and a 'release' branch for released versions. If the 'dev' branch isn't selected for negotiation, then it's not a big deal because there are many in-progress development branches with a shared history. However, if 'release' is not selected for negotiation, then the server may think that this is the first time the client has asked for that reference, causing a full download of its parallel commit history (and any extra data that may be unique to that branch). This is based on a real example where certain fetches would grow to 60+ GB when a release branch updated.
Quoted 8 lines
This option is a complement to --negotiation-restrict, which reduces the negotiation ref set to a specific list. In the earlier example, using --negotiation-restrict to focus the negotiation to 'dev' and 'release' would avoid those problematic downloads, but would still not allow advertising potentially-relevant user branches. In this way, the 'include' version solves the problem I mention while allowing negotiation to pick other references opportunistically. The two options can also be combined to allow the best of both worlds.
brances/branches from v3 fixed - thanks!
Quoted 8 lines
The argument may be an exact ref name or a glob pattern. Non-existent refs are silently ignored. This behavior is also updated in the ref matching logic for the related --negotiation-restrict option to match.
The implementation outputs the requested objects as haves before the negotiator performs its own algorithm to choose the next haves. Use the new
have_sent()interface to signal these have commits were sent before engaging with the negotiator'snext()iterator.
Now references the new have_sent() API - good!
Quoted 31 lines
Also add --negotiation-include to 'git pull' passthrough options.
Signed-off-by Derrick Stolee <[email protected]>--- Documentation/fetch-options.adoc | 19 +++++++ builtin/fetch.c | 32 ++++++++--- builtin/pull.c | 3 ++ fetch-pack.c | 81 +++++++++++++++++++++++++--- fetch-pack.h | 6 ++- t/t5510-fetch.sh | 91 ++++++++++++++++++++++++++++++++ transport.c | 8 ++- transport.h | 5 +- 8 files changed, 227 insertions(+), 18 deletions(-)Documentation/fetch-options.adoc +9
-0@@ -73,6 +73,25 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate` configuration variables documented in linkgit:git-config[1], and the `--negotiate-only` option below. +`--negotiation-include=(<commit>|<glob>)`:: + Ensure that the commits at the given tips are always sent as "have" + lines during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-restrict` restricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++ +This option may be specified more than once; if so, each commit is sent
Documentation/fetch-options.adoc See also the `fetch.negotiationAlgorithm` and `push.negotiate` · line 94
Quoted 10 lines
Documentation/fetch-options.adoc +10
-0+unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`), an +object hash, or a glob pattern (e.g. `refs/heads/release/{asterisk}`). +The pattern syntax is the same as for `--negotiation-restrict`. ++ +If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-include`. +
Good - the syntax placeholder matches --negotiate-restrict, and we mention object hashes being valid input.
Quoted 49 lines
Documentation/fetch-options.adoc +0
-0`--negotiate-only`:: Do not fetch anything from the server, and instead print the ancestors of the provided `--negotiation-restrict=` arguments,builtin/fetch.c +10
-4@@ -99,6 +99,7 @@ static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ;static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_include =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;@@ -1534,23 +1535,28 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) +static void add_negotiation_tips (struct string_list *input_list, +struct oid_array **output_list) {struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i; -for (i =0 ; i < negotiation_restrict.nr; i++) { -const char *s = negotiation_restrict.items[i].string; +for (i =0 ; i < input_list->nr; i++) { +const char *s = input_list->items[i].string;struct refs_for_each_ref_options opts = { .pattern = s, };int old_nr;if (!has_glob_specials (s)) {struct object_id oid; + +/* Ignore missing reference. */ if (repo_get_oid (the_repository, s, &oid)) -die (_ ("%s is not a valid object" ), s); +continue ; +/* Fail on missing object pointed by ref. */ if (!odb_has_object (the_repository->objects, &oid,0 ))die (_ ("the object %s does not exist" ), s); +oid_array_append (oids, &oid);continue ; }
builtin/fetch.c static void add_negotiation_restrict_tips(struct git_transport_options *smart_op · line 1573
Quoted 8 lines
builtin/fetch.c +1
-1@@ -1561,7 +1567,7 @@ static void add_negotiation_restrict_tips(struct git_transport_options *smart_opwarning (_ ("ignoring %s=%s because it does not match any refs" ),"--negotiation-restrict" , s); } - smart_options->negotiation_restrict_tips = oids; + *output_list = oids; } static struct transport *prepare_transport(struct remote *remote, int deepen,
Great! Now we have a unified implementation that takes a string_list and ouputs an oid_array, used for both restrict and include lists. We pre-resolve so that fetch-pack gets the oid_array so the ref resolution happens in the same layer for both.
Just one small issue I see - the function emits a warning for "--negotiate-restrict" when we don't match a ref, even if this is handling a --negotiate-include value. Perhaps we should pass the option name in as a parameter?
add_negotiation_tips(&negotiation_include,
"--negotiation-include",
&transport->smart_opt->negotiation_include_tips);Quoted 24 lines
builtin/fetch.c +4
-2@@ -1597,7 +1603,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen, }if (negotiation_restrict.nr) {if (transport->smart_options) -add_negotiation_restrict_tips (transport->smart_options); +add_negotiation_tips (&negotiation_restrict, + &transport->smart_options->negotiation_restrict_tips);else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );@@ -1606,7 +1613,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,for_each_string_list_item (item, &remote->negotiation_restrict)string_list_append (&negotiation_restrict, item->string);if (transport->smart_options) -add_negotiation_restrict_tips (transport->smart_options); +add_negotiation_tips (&negotiation_restrict, + &transport->smart_options->negotiation_restrict_tips);else {struct strbuf config_name =STRBUF_INIT ;strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name);@@ -1615,6 +1623,14 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,strbuf_release (&config_name); } }
builtin/fetch.c static struct transport *prepare_transport(struct remote *remote, int deepen, · line 1635
Quoted 10 lines
builtin/fetch.c +8
-0+if (negotiation_include.nr) { +if (transport->smart_options) +add_negotiation_tips (&negotiation_include, + &transport->smart_options->negotiation_include_tips); +else +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-include" ); + }return transport; }
Having the shared helper makes this much nicer. Good!
Quoted 14 lines
builtin/fetch.c +2
-0@@ -2582,6 +2598,8 @@ int cmd_fetch(int argc,OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), +OPT_STRING_LIST (0 ,"negotiation-include" , &negotiation_include,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" )),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),builtin/pull.c +0
-0
builtin/pull.c int cmd_pull(int argc, · line 1008
Quoted 9 lines
builtin/pull.c +3
-0@@ -1000,6 +1000,9 @@ int cmd_pull(int argc, N_("report that we have only objects reachable from this object"), 0),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-include" , &opt_fetch,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
LGTM
Quoted 21 lines
fetch-pack.c +6
-0@@ -25,6 +25,7 @@#include "oidset.h" #include "packfile.h" #include "odb.h" +#include "object-name.h" #include "path.h" #include "connected.h" #include "fetch-negotiator.h" @@ -332,6 +333,21 @@ static void send_filter(struct fetch_pack_args *args, } } +static void add_oids_to_set (const struct oid_array *array, +struct oidset *set) +{ +if (!array) +return ;
fetch-pack.c static void send_filter(struct fetch_pack_args *args, · line 350
Quoted 10 lines
fetch-pack.c +10
-0+ +for (size_t i =0 ; i < array->nr; i++) { +struct object_id *oid = &array->oid[i]; +if (!odb_has_object (the_repository->objects, oid,0 )) +die (_ ("the object %s does not exist" ),oid_to_hex (oid)); + +oidset_insert (set, oid); + } +} +
Nice - this is much simpler since ref resolution has been hoisted up and we only deal with pre-resolved OIDs.
Quoted 66 lines
fetch-pack.c +33
-1static int find_common(struct fetch_negotiator *negotiator, struct fetch_pack_args *args, int fd[2], struct object_id *result_oid,@@ -347,6 +363,7 @@ static int find_common(struct fetch_negotiator *negotiator,struct strbuf req_buf =STRBUF_INIT ;size_t state_len =0 ;struct packet_reader reader; +struct oidset negotiation_include_oids =OIDSET_INIT ;if (args->stateless_rpc && multi_ack ==1 )die (_ ("the option '%s' requires '%s'" ),"--stateless-rpc" ,"multi_ack_detailed" );@@ -474,6 +491,27 @@ static int find_common(struct fetch_negotiator *negotiator,trace2_region_enter ("fetch-pack" ,"negotiation_v0_v1" , the_repository); flushes =0 ; retval =-1 ; + +/* Send unconditional haves from --negotiation-include */ +add_oids_to_set (args->negotiation_include_tips, + &negotiation_include_oids); +if (oidset_size (&negotiation_include_oids)) { +struct oidset_iter iter; +oidset_iter_init (&negotiation_include_oids, &iter); + +while ((oid =oidset_iter_next (&iter))) { +struct commit *commit; +packet_buf_write (&req_buf,"have %s\n" , +oid_to_hex (oid)); +print_verbose (args,"have %s" ,oid_to_hex (oid)); + count++; + + commit =lookup_commit (the_repository, oid); +if (commit) + negotiator->have_sent (negotiator, commit); + } + } +while ((oid = negotiator->next (negotiator))) {packet_buf_write (&req_buf,"have %s\n" ,oid_to_hex (oid));print_verbose (args,"have %s" ,oid_to_hex (oid));@@ -584,6 +622,7 @@ done: flushes++; }strbuf_release (&req_buf); +oidset_clear (&negotiation_include_oids);if (!got_ready || !no_done)consume_shallow_list (args, &reader);@@ -1305,11 +1344,27 @@ static void add_common(struct strbuf *req_buf, struct oidset *common)static int add_haves (struct fetch_negotiator *negotiator,struct strbuf *req_buf, -int *haves_to_send) +int *haves_to_send, +struct oidset *negotiation_include_oids) {int haves_added =0 ;const struct object_id *oid; +/* Send unconditional haves from --negotiation-include */ +if (negotiation_include_oids) { +struct oidset_iter iter; +oidset_iter_init (negotiation_include_oids, &iter); + +while ((oid =oidset_iter_next (&iter))) { +struct commit *commit =lookup_commit (the_repository, oid); +if (commit) {
fetch-pack.c static void add_common(struct strbuf *req_buf, struct oidset *common) · line 1370
Quoted 10 lines
fetch-pack.c +7
-0+packet_buf_write (req_buf,"have %s\n" , +oid_to_hex (oid)); + negotiator->have_sent (negotiator, commit); + } + } + } +while ((oid = negotiator->next (negotiator))) {packet_buf_write (req_buf,"have %s\n" ,oid_to_hex (oid));if (++haves_added >= *haves_to_send)
Here we're using the new have_sent() API and replaces the manual COMMON bit flipping.
Quoted 48 lines
fetch-pack.c +9
-3@@ -1358,7 +1413,8 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,struct fetch_pack_args *args, const struct ref *wants, struct oidset *common, int *haves_to_send, int *in_vain, - int sideband_all, int seen_ack) + int sideband_all, int seen_ack, + struct oidset *negotiation_include_oids) { int haves_added;int done_sent =0 ;@@ -1413,7 +1469,8 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,/* Add all of the common commits we've found in previous rounds */ add_common (&req_buf, common); - haves_added =add_haves (negotiator, &req_buf, haves_to_send); + haves_added =add_haves (negotiator, &req_buf, haves_to_send, + negotiation_include_oids); *in_vain += haves_added;trace2_data_intmax ("negotiation_v2" , the_repository,"haves_added" , haves_added);trace2_data_intmax ("negotiation_v2" , the_repository,"in_vain" , *in_vain);@@ -1657,6 +1714,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,struct ref *ref =copy_ref_list (orig_ref);enum fetch_state state =FETCH_CHECK_LOCAL ;struct oidset common =OIDSET_INIT ; +struct oidset negotiation_include_oids =OIDSET_INIT ;struct packet_reader reader;int in_vain =0 , negotiation_started =0 ;int negotiation_round =0 ;@@ -1729,6 +1787,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, state =FETCH_SEND_REQUEST ;mark_tips (negotiator, args->negotiation_restrict_tips); +add_oids_to_set (args->negotiation_include_tips, + &negotiation_include_oids);for_each_cached_alternate (negotiator, insert_one_alternate_object);break ;@@ -1747,7 +1807,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, &common, &haves_to_send, &in_vain, reader.use_sideband, - seen_ack)) { + seen_ack, + &negotiation_include_oids)) {trace2_region_leave_printf ("negotiation_v2" ,"round" , the_repository,"%d" , negotiation_round);
fetch-pack.c static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, · line 1949
Quoted 6 lines
fetch-pack.c +1
-0@@ -1883,6 +1944,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, negotiator->release (negotiator);oidset_clear (&common); +oidset_clear (&negotiation_include_oids);return ref; }
The v2 wiring looks correct.
Quoted 78 lines
fetch-pack.c +9
-2@@ -2181,12 +2243,14 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,const struct string_list *server_options, int stateless_rpc, int fd[], - struct oidset *acked_commits) + struct oidset *acked_commits, + const struct oid_array *negotiation_include_tips) { struct fetch_negotiator negotiator;struct packet_reader reader;struct object_array nt_object_array =OBJECT_ARRAY_INIT ;struct strbuf req_buf =STRBUF_INIT ; +struct oidset negotiation_include_oids =OIDSET_INIT ;int haves_to_send =INITIAL_FLUSH ;int in_vain =0 ;int seen_ack =0 ;@@ -2197,6 +2261,9 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,fetch_negotiator_init (the_repository, &negotiator);mark_tips (&negotiator, negotiation_restrict_tips); +add_oids_to_set (negotiation_include_tips, + &negotiation_include_oids); +packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE |PACKET_READ_DIE_ON_ERR_PACKET );@@ -2221,7 +2288,8 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,packet_buf_write (&req_buf,"wait-for-done" ); - haves_added =add_haves (&negotiator, &req_buf, &haves_to_send); + haves_added =add_haves (&negotiator, &req_buf, &haves_to_send, + &negotiation_include_oids); in_vain += haves_added;if (!haves_added || (seen_ack && in_vain >=MAX_IN_VAIN )) last_iteration =1 ;@@ -2273,6 +2341,7 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,clear_common_flag (acked_commits);object_array_clear (&nt_object_array); +oidset_clear (&negotiation_include_oids); negotiator.release (&negotiator);strbuf_release (&req_buf); }fetch-pack.h +4
-2@@ -19,9 +19,10 @@ struct fetch_pack_args {/* * If not NULL, during packfile negotiation, fetch-pack will send "have" -* lines only with these tips and their ancestors. +* lines for all _include_ tips and then a subset of the _restrict_ tips. */ const struct oid_array *negotiation_restrict_tips; +const struct oid_array *negotiation_include_tips; unsigned deepen_relative:1; unsigned quiet:1;@@ -93,7 +94,8 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,const struct string_list *server_options, int stateless_rpc, int fd[], - struct oidset *acked_commits); + struct oidset *acked_commits, + const struct oid_array *negotiation_include_tips);/* * Print an appropriate error message for each sought ref that wasn't t/t5510-fetch.sh +0
-0@@ -1460,6 +1460,16 @@ EOF test_cmp fatal-expect fatal-actual '
t/t5510-fetch.sh EOF · line 1472
Quoted 10 lines
t/t5510-fetch.sh +10
-0+test_expect_success '--negotiation-tip ignores missing refs and invalid hashes' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-tip=alpha_1 --negotiation-tip=beta_1 \ + --negotiation-tip=no-such-ref \ + --negotiation-tip=invalid-hash \ + origin alpha_s beta_s && + check_negotiation_tip +' +
Good! This checks for missing refs (not just invalid ones with all zeros).
Quoted 135 lines
t/t5510-fetch.sh +81
-0test_expect_success '--negotiation-restrict limits "have" lines sent' ' setup_negotiation_tip server server 0 && GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \@@ -1511,6 +1521,87 @@ test_expect_success 'CLI --negotiation-restrict overrides remote config' ' test_grep ! "fetch> have $BETA_1" trace ' +test_expect_success '--negotiation-include includes configured refs as haves' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-include=refs/tags/beta_1 \ + origin alpha_s beta_s && + + ALPHA_1=$(git -C client rev-parse alpha_1) && + test_grep "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace +' + +test_expect_success '--negotiation-include works with glob patterns' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-include="refs/tags/beta_*" \ + origin alpha_s beta_s && + + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace && + BETA_2=$(git -C client rev-parse beta_2) && + test_grep "fetch> have $BETA_2" trace +' + +test_expect_success '--negotiation-include is additive with negotiation' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-include=refs/tags/beta_1 \ + origin alpha_s beta_s && + + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace +' + +test_expect_success '--negotiation-include ignores non-existent refs silently' ' + setup_negotiation_tip server server 0 && + + git -C client fetch --quiet \ + --negotiation-restrict=alpha_1 \ + --negotiation-include=refs/tags/nonexistent \ + origin alpha_s beta_s 2>err && + test_must_be_empty err +' + +test_expect_success '--negotiation-include avoids duplicates with negotiator' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + ALPHA_1=$(git -C client rev-parse alpha_1) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-include=refs/tags/alpha_1 \ + origin alpha_s beta_s && + + test_grep "fetch> have $ALPHA_1" trace >matches && + test_line_count = 1 matches +' + +test_expect_success '--negotiation-include avoids duplicates with v0' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + ALPHA_1=$(git -C client rev-parse alpha_1) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client \ + -c protocol.version=0 fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-include=refs/tags/alpha_1 \ + origin alpha_s beta_s && + + test_grep "fetch> have $ALPHA_1" trace >matches && + test_line_count = 1 matches +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (transport.c +7
-1@@ -464,6 +464,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; + args.negotiation_include_tips = data->options.negotiation_include_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {@@ -495,7 +496,8 @@ static int fetch_refs_via_pack(struct transport *transport, transport->server_options, transport->stateless_rpc, data->fd, - data->options.acked_commits); + data->options.acked_commits, + data->options.negotiation_include_tips); ret =0 ; } goto cleanup;@@ -983,6 +985,10 @@ static int disconnect_git(struct transport *transport)oid_array_clear (data->options.negotiation_restrict_tips);free (data->options.negotiation_restrict_tips); } +if (data->options.negotiation_include_tips) { +oid_array_clear (data->options.negotiation_include_tips); +free (data->options.negotiation_include_tips); + }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);oid_array_clear (&data->shallow);transport.h +1
-1@@ -40,13 +40,14 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_restrict_tips in struct fetch_pack_args. +* these member names in struct fetch_pack_args. *
transport.h struct git_transport_options { · line 53
Quoted 10 lines
transport.h +2
-1-* This field is only supported by transports that support connect or +* These fields are only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ struct oid_array *negotiation_restrict_tips; +struct oid_array *negotiation_include_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
Overall this version of the patch is much better IMO! Just that one comment about the incorrect option name in the warning message.
Thanks,
Matthew
Re: [PATCH v4 7/8] remote: add remote.*.negotiationInclude config
In reply to Derrick Stolee via GitGitGadgetQuoted 42 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
Add a new 'remote.<name>.negotiationInclude' multi-valued config option that provides default values for --negotiation-include when no --negotiation-include arguments are specified over the command line. This is a mirror of how 'remote.<name>.negotiationRestrict' specifies defaults for the --negotiation-restrict arguments.
Each value is either an exact ref name or a glob pattern whose tips should always be sent as 'have' lines during negotiation. The config values are resolved through the same
resolve_negotiation_include()codepath as the CLI options.This option is additive with the normal negotiation process: the negotiation algorithm still runs and advertises its own selected commits, but the refs matching the config are sent unconditionally on top of those heuristically selected commits.
Similar to the negotiationRestrict config, an empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
Signed-off-by Derrick Stolee <[email protected]>--- Documentation/config/remote.adoc | 27 ++++++++++++++++++ Documentation/fetch-options.adoc | 4 +++ builtin/fetch.c | 11 +++++++ remote.c | 5 ++++ remote.h | 1 + t/t5510-fetch.sh | 49 ++++++++++++++++++++++++++++++++ 6 files changed, 97 insertions(+)Documentation/config/remote.adoc +2
-0@@ -125,6 +125,33 @@ values are not used. Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios. +remote.<name>.negotiationInclude:: + When negotiating with this remote during `git fetch`, the client
Documentation/config/remote.adoc values are not used. · line 139
Quoted 10 lines
Documentation/config/remote.adoc +10
-0+ advertises a list of commits that exist locally. In repos with + many references, this list of "haves" can be truncated. Depending + on data shape, dropping certain references may be expensive. This + multi-valued config option specifies references, commit hashes, + or ref pattern globs whose tips should always be sent as "have" + commits during fetch negotiation with this remote. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`), a +commit hash, or a glob pattern (e.g. `refs/heads/release/*`). The +pattern syntax is the same as for `--negotiation-include`.
Thanks - references the correct cross-referenced option I raised in v3. Commit hashes are also explicitly mentioned - good.
Documentation/config/remote.adoc values are not used. · line 150
Quoted 10 lines
Documentation/config/remote.adoc +10
-0+These config values are used as defaults for the `--negotiation-include` +command-line option. If `--negotiation-include` is specified on the +command line, then the config values are not used. ++ +This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits, +but the refs matching `remote.<name>.negotiationInclude` are sent +unconditionally on top of those heuristically selected commits. This +option is also used during push negotiation when `push.negotiate` is +enabled.
One thing to mention: the "also used during push negotiation" sentence is added here, and then I see in the next patch (patch 8) we're adding another sentence "these values also influence negotiation during git push" to this same block.
Perhaps consider dropping the additional 'push' sentence here (patch 7) and let patch 8 add just its own sentence about push? Not too pressing.
Quoted 65 lines
Documentation/config/remote.adoc +4
-0++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.Documentation/fetch-options.adoc +4
-0@@ -91,6 +91,10 @@ The pattern syntax is the same as for `--negotiation-restrict`. If `--negotiation-restrict` is used, the have set is first restricted by that option and then increased to include the tips specified by `--negotiation-include`. ++ +If this option is not specified on the command line, then any +`remote.<name>.negotiationInclude` config values for the current remote +are used instead. `--negotiate-only`:: Do not fetch anything from the server, and instead print thebuiltin/fetch.c +11
-0@@ -1630,6 +1630,17 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-include" ); + }else if (remote->negotiation_include.nr) { +if (transport->smart_options) { +add_negotiation_tips (&remote->negotiation_include, + &transport->smart_options->negotiation_include_tips); + }else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationInclude" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }remote.c +2
-0@@ -153,6 +153,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options);string_list_init_dup (&ret->negotiation_restrict); +string_list_init_dup (&ret->negotiation_include);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);@@ -181,6 +182,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 );string_list_clear (&remote->negotiation_restrict,0 ); +string_list_clear (&remote->negotiation_include,0 ); }static void add_merge (struct branch *branch,const char *name)
remote.c static int handle_config(const char *key, const char *value, · line 577
Quoted 9 lines
remote.c +3
-0@@ -567,6 +569,9 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"negotiationrestrict" )) {return parse_transport_option (key, value, &remote->negotiation_restrict); + }else if (!strcmp (subkey,"negotiationinclude" )) { +return parse_transport_option (key, value, + &remote->negotiation_include); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
Good - uses the parse_transport_options() like with the earlier patch.
Quoted 27 lines
remote.h +1
-0@@ -118,6 +118,7 @@ struct remote {struct string_list server_options;struct string_list negotiation_restrict; +struct string_list negotiation_include;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;t/t5510-fetch.sh +7
-0@@ -1587,6 +1587,55 @@ test_expect_success '--negotiation-include avoids duplicates with negotiator' ' test_line_count = 1 matches ' +test_expect_success 'remote.<name>.negotiationInclude used as default for --negotiation-include' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + # test the reset of the list on an empty value + git -C client config --add remote.origin.negotiationInclude refs/tags/alpha_1 && + git -C client config --add remote.origin.negotiationInclude "" &&
t/t5510-fetch.sh test_expect_success '--negotiation-include avoids duplicates with negotiator' ' · line 1606
Quoted 10 lines
t/t5510-fetch.sh +10
-0+ git -C client config --add remote.origin.negotiationInclude refs/tags/beta_1 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=beta_2 \ + origin alpha_s beta_s && + + ALPHA_1=$(git -C client rev-parse alpha_1) && + test_grep ! "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace +'
Great! Now this test will catch failures to the reset-list behaviour by using a different CLI option ref name than is in the config list.
Quoted 25 lines
t/t5510-fetch.sh +25
-0+ +test_expect_success 'remote.<name>.negotiationInclude works with glob patterns' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + git -C client config --add remote.origin.negotiationInclude "refs/tags/beta_*" && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + origin alpha_s beta_s && + + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace && + BETA_2=$(git -C client rev-parse beta_2) && + test_grep "fetch> have $BETA_2" trace +' + +test_expect_success 'CLI --negotiation-include overrides remote.<name>.negotiationInclude' ' + test_when_finished rm -f trace && + setup_negotiation_tip server server 0 && + + git -C client config --add remote.origin.negotiationInclude refs/tags/beta_2 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-include=refs/tags/beta_1 \ + origin alpha_s beta_s &&
t/t5510-fetch.sh test_expect_success '--negotiation-include avoids duplicates with negotiator' ' · line 1641
Quoted 10 lines
t/t5510-fetch.sh +7
-0+ + BETA_1=$(git -C client rev-parse beta_1) && + test_grep "fetch> have $BETA_1" trace && + BETA_2=$(git -C client rev-parse beta_2) && + test_grep ! "fetch> have $BETA_2" trace +' + test_expect_success '--negotiation-include avoids duplicates with v0' ' test_when_finished rm -f trace && setup_negotiation_tip server server 0 &&
Overall this patch addresses my comments from v3's review. Just the minor issue with forward reference to 'git push' behaviour in the docs.
Thanks,
Matthew
Re: [PATCH v4 8/8] send-pack: pass negotiation config in push
In reply to Derrick Stolee via GitGitGadgetQuoted 12 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
When push.negotiate is enabled, 'git push' spawns a child 'git fetch --negotiate-only' process to find common commits. Pass --negotiation-include and --negotiation-restrict options from the 'remote.<name>.negotiationInclude' and 'remote.<name>.negotiationRestrict' config keys to this child process.
When negotiationRestrict is configured, it replaces the default behavior of using all remote refs as negotiation tips. This allows the user to control which local refs are used for push negotiation.
Quoted 6 lines
When negotiationInclude is configured, the specified ref patterns are passed as --negotiation-include to ensure their tips are always sent as 'have' lines during push negotiation.
This change also updates the use of --negotiation-tip into --negotiation-restrict now that the new synonym exists.
Is this paragraph stale? Didn't we rename the option in patch 2?
Quoted 14 lines
Signed-off-by Derrick Stolee <[email protected]>--- Documentation/config/remote.adoc | 6 ++++++ send-pack.c | 37 ++++++++++++++++++++++++++------ send-pack.h | 2 ++ t/t5516-fetch-push.sh | 30 ++++++++++++++++++++++++++ transport.c | 2 ++ 5 files changed, 70 insertions(+), 7 deletions(-)Documentation/config/remote.adoc +0
-0
Documentation/config/remote.adoc command-line option. If `--negotiation-restrict` (or its synonym · line 130
Quoted 9 lines
Documentation/config/remote.adoc +3
-0@@ -122,6 +122,9 @@ command-line option. If `--negotiation-restrict` (or its synonym `--negotiation-tip`) is specified on the command line, then the config values are not used. + +These values also influence negotiation during `git push` if +`push.negotiate` is enabled. ++ Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios.
Nice. We're now only mentioning git push behaviour once we've wired it up in this patch.
Quoted 140 lines
Documentation/config/remote.adoc +3
-0@@ -149,6 +152,9 @@ unconditionally on top of those heuristically selected commits. This option is also used during push negotiation when `push.negotiate` is enabled. + +These values also influence negotiation during `git push` if +`push.negotiate` is enabled. ++ Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios.send-pack.c +30
-7@@ -433,28 +433,48 @@ static void reject_invalid_nonce(const char *nonce, int len)static void get_commons_through_negotiation (struct repository *r,const char *url, +const struct string_list *negotiation_include, +const struct string_list *negotiation_restrict,const struct ref *remote_refs,struct oid_array *commons) {struct child_process child =CHILD_PROCESS_INIT ;const struct ref *ref;int len = r->hash_algo->hexsz +1 ;/* hash + NL */ -int nr_negotiation_tip =0 ; +int nr_negotiation =0 ; child.git_cmd =1 ; child.no_stdin =1 ; child.out =-1 ;strvec_pushl (&child.args,"fetch" ,"--negotiate-only" ,NULL ); -for (ref = remote_refs; ref; ref = ref->next) { -if (!is_null_oid (&ref->new_oid)) { + +if (negotiation_restrict && negotiation_restrict->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_restrict)strvec_pushf (&child.args,"--negotiation-restrict=%s" , -oid_to_hex (&ref->new_oid)); - nr_negotiation_tip++; + item->string); + nr_negotiation = negotiation_restrict->nr; + }else { +for (ref = remote_refs; ref; ref = ref->next) { +if (!is_null_oid (&ref->new_oid)) { +strvec_pushf (&child.args,"--negotiation-restrict=%s" , +oid_to_hex (&ref->new_oid)); + nr_negotiation++; + } } } + +if (negotiation_include && negotiation_include->nr) { +struct string_list_item *item; +for_each_string_list_item (item, negotiation_include) +strvec_pushf (&child.args,"--negotiation-include=%s" , + item->string); + nr_negotiation += negotiation_include->nr; + } +strvec_push (&child.args, url); -if (!nr_negotiation_tip) { +if (!nr_negotiation) {child_process_clear (&child);return ; }@@ -528,7 +548,10 @@ int send_pack(struct repository *r,repo_config_get_bool (r,"push.negotiate" , &push_negotiate);if (push_negotiate) {trace2_region_enter ("send_pack" ,"push_negotiate" , r); -get_commons_through_negotiation (r, args->url, remote_refs, &commons); +get_commons_through_negotiation (r, args->url, + args->negotiation_include, + args->negotiation_restrict, + remote_refs, &commons);trace2_region_leave ("send_pack" ,"push_negotiate" , r); }send-pack.h +2
-0@@ -18,6 +18,8 @@ struct repository;struct send_pack_args {const char *url; +const struct string_list *negotiation_include; +const struct string_list *negotiation_restrict;unsigned verbose:1 , quiet:1 , porcelain:1,t/t5516-fetch-push.sh +30
-0@@ -254,6 +254,36 @@ test_expect_success 'push with negotiation does not attempt to fetch submodules' ! grep "Fetching submodule" err ' +test_expect_success 'push with negotiation and remote.<name>.negotiationInclude' ' + test_when_finished rm -rf negotiation_include && + mk_empty negotiation_include && + git push negotiation_include $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_include unrelated_commit && + git -C negotiation_include config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_include.negotiationInclude=refs/heads/main \ + push negotiation_include refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + +test_expect_success 'push with negotiation and remote.<name>.negotiationRestrict' ' + test_when_finished rm -rf negotiation_restrict && + mk_empty negotiation_restrict && + git push negotiation_restrict $the_first_commit:refs/remotes/origin/first_commit && + test_commit -C negotiation_restrict unrelated_commit && + git -C negotiation_restrict config receive.hideRefs refs/remotes/origin/first_commit && + test_when_finished "rm event" && + GIT_TRACE2_EVENT="$(pwd)/event" \ + git -c protocol.version=2 -c push.negotiate=1 \ + -c remote.negotiation_restrict.negotiationRestrict=refs/heads/main \ + push negotiation_restrict refs/heads/main:refs/remotes/origin/main && + test_grep \"key\":\"total_rounds\" event && + grep_wrote 2 event # 1 commit, 1 tree +' + test_expect_success 'push without wildcard' ' mk_empty testrepo &&transport.c +0
-0
transport.c static int git_transport_push(struct transport *transport, struct ref *remote_re · line 928
Quoted 8 lines
transport.c +2
-0@@ -921,6 +921,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re args.atomic = !!(flags &TRANSPORT_PUSH_ATOMIC ); args.push_options = transport->push_options; args.url = transport->url; + args.negotiation_include = &transport->remote->negotiation_include; + args.negotiation_restrict = &transport->remote->negotiation_restrict;if (flags &TRANSPORT_PUSH_CERT_ALWAYS ) args.push_cert =SEND_PACK_PUSH_CERT_ALWAYS ;
Again v4 of this patch LGTM.
Thanks,
Matthew
Re: [PATCH v4 0/8] fetch: rework negotiation tip options
In reply to Derrick Stolee via GitGitGadgetQuoted 20 lines · Derrick Stolee via GitGitGadget
Updates in v4
=============Thanks, Matthew, for the detailed review! There are some big changes in this version.
* Expanded commit message to cite the commit that introduced the bug
(3f763ddf28).* Renamed --negotiation-tip to --negotiation-restrict throughout docs/code (including
send-pack.c,transport-helper.c,builtin/pull.c). AddedOPT_ALIASin git-pull.
* Switched config parsing to useparse_transport_option()helper. Removed git push from docs (not implemented yet). Restructured --negotiate-only validation flow.
* NEW Patch 5: Addedhave_sent()interface to negotiators, so included haves can be de-duplicated properly by the negotiation algorithm.
* Replaced COMMON flag hack withnegotiator->have_sent()calls. Moved ref-pattern resolution intobuiltin/fetch.c(add_negotiation_tips()) so fetch-pack receives pre-resolvedoid_arrayinstead ofstring_list. Added test for --negotiation-tip ignoring missing refs. Added
Quoted 9 lines
duplicate-avoidance test for v0. Accepts commit hashes in addition to ref names/globs.
* Useparse_transport_option()for config. Updated docs to mention commit hashes. Removed git push from config docs. Fixed test to use correct restrict/include combinations.
* In the last patch, add doc notes that remote config values also apply during git push with push.negotiate, now that they are integrated by that change.
Thank you for going through the comments on v3 in detail. This is a nice improvement overall.
The main thing flagged (the COMMON bit confusion) is resolved by adding the new have_sent() API on the negotiator interface, which is much clearer and cleaner. The hoisting of the ref resolution to the same layer and reuse of add_negotiate_tips() is also done and appreciated!
I've left replies on each patch, with only a small number of easily addressed comments.
Thanks,
Matthew
Re: [PATCH v4 0/8] fetch: rework negotiation tip options
Quoted 35 lines · Matthew John Cheetham
Quoted 20 lines · Derrick Stolee via GitGitGadget
Updates in v4
=============Thanks, Matthew, for the detailed review! There are some big changes in this version.
* Expanded commit message to cite the commit that introduced the bug (
3f763ddf28).
* Renamed --negotiation-tip to --negotiation-restrict throughout docs/code (includingsend-pack.c,transport-helper.c,builtin/pull.c). AddedOPT_ALIASin git-pull.
* Switched config parsing to useparse_transport_option()helper. Removed git push from docs (not implemented yet). Restructured --negotiate-only validation flow.
* NEW Patch 5: Addedhave_sent()interface to negotiators, so included haves can be de-duplicated properly by the negotiation algorithm.
* Replaced COMMON flag hack withnegotiator->have_sent()calls. Moved ref-pattern resolution intobuiltin/fetch.c(add_negotiation_tips()) so fetch-pack receives pre-resolvedoid_arrayinstead ofstring_list. Added test for --negotiation-tip ignoring missing refs. AddedQuoted 9 lines
duplicate-avoidance test for v0. Accepts commit hashes in addition to ref names/globs.
* Useparse_transport_option()for config. Updated docs to mention commit hashes. Removed git push from config docs. Fixed test to use correct restrict/include combinations.
* In the last patch, add doc notes that remote config values also apply during git push with push.negotiate, now that they are integrated by that change.Thank you for going through the comments on v3 in detail. This is a nice improvement overall.
Quoted 7 lines
The main thing flagged (the COMMON bit confusion) is resolved by adding the new
have_sent()API on the negotiator interface, which is much clearer and cleaner. The hoisting of the ref resolution to the same layer and reuse ofadd_negotiate_tips()is also done and appreciated!I've left replies on each patch, with only a small number of easily addressed comments.
Thanks for your review, including a confirmation that I properly responded to your earlier review. Soon, I'll send a new version with the few minor edits included.
Thanks,
-Stolee
v5 cover fetch: rework negotiation tip options
In reply to Derrick Stolee via GitGitGadgetFetch negotiation aims to find enough information from haves and wants such that the server can be reasonably confident that it will send all necessary objects and not too many "extra" objects that the client already has. However, this can break down if there are too many references, since Git truncates the list of haves based on a few factors (a 256 count limit or the server sending an ACK at the right time).
We already have the --negotiation-tip feature to focus the set of references that are used in negotiation, but I feel like this is designed backwards. I'd rather that we have a way to say "this is an important set of refs, but feel free to add more refs if needed" than "only use these refs for negotiation".
Here's an example that demonstrates the problem. In an internal monorepo, developers work off of the 'main' branch so there are thousands of user branches that each add a few commits different from the 'main' branch. However, there is also a long-lived 'release' branch. This branch has a first-parent history that is parallel to 'main' and each of those commits is a merge whose second parent is a commit from 'main' that had a successful CI run. There are additional changes in the 'release' branch merge commits that add some changelog data, so there is a nontrivial set of novel blob content in that branch and not just a different set of commits.
The problem we had was that our georeplication system was regularly fetching from the origin and trying to get all data from all reachable branches. When the 'release' branch updated, the client would run out of haves before advertising its copy of the 'release' branch, but it would still list the new 'release' tip as a want. The server would then think that the client had never fetched that branch before and would send all of the changelog data from the whole history of the repo. (This led to a lot of downstream problems; we mitigated by setting a refspec that stopped fetching the 'release' branch, but this is not ideal.)
What I'd like is a mechanism to say "always advertise the client's version of 'main' and 'release' but also opportunistically include some user branches".
Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.
There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.
This patch series adds a new '--negotiation-include' option that does what I
want: it makes sure that these references are included as 'have's during negotiation. In order to help clarify the difference between this and '--negotiation-tip', I first create a synonym called '--negotiation-restrict'.
Both of these options get 'remote.*.negotiation(Include|Restrict)' config options that enable their behavior by default.
During development, I had briefly considered only using config values, but that required some strange changes to care about the remote name in the transport layer. This was most different in the 'git push' integration. When I discovered the '--negotiation-tip' feature during the process, that gave me a clear pattern to follow with the addition of a config on top.
Updates in v2
=============
This version is a near-complete rewrite based on feedback around the names of the previous option and config. The --negotiation-restrict option is new and the ability to set it via config is also new.
I did try to be more careful around translatable error messages, too.
Updates in v3
=============
* --negotiation-tip is now an alias of --negotiation-restrict.
* More translatable strings use %s to isolate non-translatable options from translatable words.
* The string_list named negotiation_tip is now renamed to negotiation_restrict.
* The config options now allow an empty value to reset the list.
* The --negotiation-require option is now called --negotiation-include.
* Similarly, the config option is renamed and all code references.
* The included haves now mark their commits as COMMON so commits that they can reach are not included in the negotiation walk if they are reached from the restricted commits.
* The ref iterators are more careful about failing on bad references (ref exists but object doesn't) and ignoring missing references (perhaps config is erroneous?).
* When sending tips during push negotiation, use the --negotiation-restrict option instead of -tip.
Updates in v4
=============
Thanks, Matthew, for the detailed review! There are some big changes in this version.
* Expanded commit message to cite the commit that introduced the bug (3f763ddf28).
* Renamed --negotiation-tip to --negotiation-restrict throughout docs/code (including send-pack.c, transport-helper.c, builtin/pull.c). Added OPT_ALIAS in git-pull.
* Switched config parsing to use parse_transport_option() helper. Removed git push from docs (not implemented yet). Restructured --negotiate-only validation flow.
* NEW Patch 5: Added have_sent() interface to negotiators, so included haves can be de-duplicated properly by the negotiation algorithm.
* Replaced COMMON flag hack with negotiator->have_sent() calls. Moved ref-pattern resolution into builtin/fetch.c (add_negotiation_tips()) so fetch-pack receives pre-resolved oid_array instead of string_list. Added test for --negotiation-tip ignoring missing refs. Added duplicate-avoidance test for v0. Accepts commit hashes in addition to ref names/globs.
* Use parse_transport_option() for config. Updated docs to mention commit hashes. Removed git push from config docs. Fixed test to use correct restrict/include combinations.
* In the last patch, add doc notes that remote config values also apply during git push with push.negotiate, now that they are integrated by that change.
Thanks, -Stolee
Derrick Stolee (8):
t5516: fix test order flakiness
fetch: add --negotiation-restrict option
transport: rename negotiation_tips
remote: add remote.*.negotiationRestrict config
negotiator: add have_sent() interface
fetch: add --negotiation-include option for negotiation
remote: add remote.*.negotiationInclude config
send-pack: pass negotiation config in push
Documentation/config/fetch.adoc | 2 +- Documentation/config/remote.adoc | 49 ++++++++ Documentation/fetch-options.adoc | 29 ++++- builtin/fetch.c | 87 +++++++++++--- builtin/pull.c | 6 +- fetch-negotiator.h | 9 ++ fetch-pack.c | 99 +++++++++++++--- fetch-pack.h | 10 +- negotiator/default.c | 8 ++ negotiator/noop.c | 7 ++ negotiator/skipping.c | 8 ++ remote.c | 10 ++ remote.h | 2 + send-pack.c | 39 +++++-- send-pack.h | 2 + t/t5510-fetch.sh | 191 +++++++++++++++++++++++++++++++ t/t5516-fetch-push.sh | 32 +++++- t/t5702-protocol-v2.sh | 4 +- transport-helper.c | 5 +- transport.c | 20 +++- transport.h | 7 +- 21 files changed, 564 insertions(+), 62 deletions(-)
base-commit: 6e8d538aab8fe4dd07ba9fb87b5c7edcfa5706ad
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2085%2Fderrickstolee%2Fmust-have-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2085/derrickstolee/must-have-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/2085
Range-diff vs v4:
1: 7409a479d6 ! 1: 538913a327 t5516: fix test order flakiness
@@ Commit message
Use 'sort -k 3' to match the actual number of columns in the output.
+ Reviewed-by: Matthew John Cheetham <[email protected]>
## t/t5516-fetch-push.sh ##
2: 7836a2d6a5 ! 2: 580aa58943 fetch: add --negotiation-restrict option
@@ Commit message
translatable with the option name inserted by formatting. At least one of these messages will be reused later for a new option.
+ Reviewed-by: Matthew John Cheetham <[email protected]>
## Documentation/config/fetch.adoc ##
3: 401bdaff7c ! 3: eee0543647 transport: rename negotiation_tips
@@ Commit message
Also update the string_list used to store the inputs from command-line options.
+ Reviewed-by: Matthew John Cheetham <[email protected]>
## builtin/fetch.c ##
4: a14c568a1f ! 4: 63c675e93e remote: add remote.*.negotiationRestrict config
@@ Commit message
An empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
+ Reviewed-by: Matthew John Cheetham <[email protected]>
## Documentation/config/remote.adoc ##
5: 94b79784fe ! 5: d423c56283 negotiator: add have_sent() interface
@@ Commit message
common, so the implementation is quite simple. This logic will be exercised in the next change.
+ Reviewed-by: Matthew John Cheetham <[email protected]>
## fetch-negotiator.h ##
6: b4cd458fe0 ! 6: e86c9791e2 fetch: add --negotiation-include option for negotiation
@@ Commit message
Also add --negotiation-include to 'git pull' passthrough options.
+ Reviewed-by: Matthew John Cheetham <[email protected]>
Signed-off-by: Derrick Stolee <[email protected]>
## Documentation/fetch-options.adoc ##
@@ builtin/fetch.c: static int add_oid(const struct reference *ref, void *cb_data)
-static void add_negotiation_restrict_tips(struct git_transport_options *smart_options)
+static void add_negotiation_tips(struct string_list *input_list,
-+ struct oid_array **output_list)
++ struct oid_array **output_list,
++ const char *argname)
{
struct oid_array *oids = xcalloc(1, sizeof(*oids));
int i;
@@ builtin/fetch.c: static int add_oid(const struct reference *ref, void *cb_data)
continue;
}
@@ builtin/fetch.c: static void add_negotiation_restrict_tips(struct git_transport_options *smart_op
+ add_oid, oids, &opts);
+ if (old_nr == oids->nr)
warning(_("ignoring %s=%s because it does not match any refs"),
- "--negotiation-restrict", s);
+- "--negotiation-restrict", s);
++ argname, s);
}
- smart_options->negotiation_restrict_tips = oids;
+ *output_list = oids;
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remot
if (transport->smart_options)
- add_negotiation_restrict_tips(transport->smart_options);
+ add_negotiation_tips(&negotiation_restrict,
-+ &transport->smart_options->negotiation_restrict_tips);
++ &transport->smart_options->negotiation_restrict_tips,
++ "--negotiation-restrict");
else
warning(_("ignoring %s because the protocol does not support it"),
"--negotiation-restrict");
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remot
if (transport->smart_options)
- add_negotiation_restrict_tips(transport->smart_options);
+ add_negotiation_tips(&negotiation_restrict,
-+ &transport->smart_options->negotiation_restrict_tips);
++ &transport->smart_options->negotiation_restrict_tips,
++ "--negotiation-restrict");
else {
struct strbuf config_name = STRBUF_INIT;
strbuf_addf(&config_name, "remote.%s.negotiationRestrict", remote->name);
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remot
+ if (negotiation_include.nr) {
+ if (transport->smart_options)
+ add_negotiation_tips(&negotiation_include,
-+ &transport->smart_options->negotiation_include_tips);
++ &transport->smart_options->negotiation_include_tips,
++ "--negotiation-include");
+ else
+ warning(_("ignoring %s because the protocol does not support it"),
+ "--negotiation-include"); 7: 7bd70a970b ! 7: e5714115b5 remote: add remote.*.negotiationInclude config
@@ Commit message
list to allow ignoring earlier config values, such as those that might be set in system or global config.
+ Reviewed-by: Matthew John Cheetham <[email protected]>
## Documentation/config/remote.adoc ##
@@ Documentation/config/remote.adoc: values are not used.
+This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits, +but the refs matching remote.<name>.negotiationInclude are sent
-+unconditionally on top of those heuristically selected commits. This -+option is also used during push negotiation when push.negotiate is -+enabled.
++unconditionally on top of those heuristically selected commits.
++
+Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios.
@@ builtin/fetch.c: static struct transport *prepare_transport(struct remote *remot
+ } else if (remote->negotiation_include.nr) {
+ if (transport->smart_options) {
+ add_negotiation_tips(&remote->negotiation_include,
-+ &transport->smart_options->negotiation_include_tips);
++ &transport->smart_options->negotiation_include_tips,
++ "--negotiation-include");
+ } else {
+ struct strbuf config_name = STRBUF_INIT;
+ strbuf_addf(&config_name, "remote.%s.negotiationInclude", remote->name);
8: 5b968245eb ! 8: ed0be32e2c send-pack: pass negotiation config in push
@@ Commit message
are passed as --negotiation-include to ensure their tips are always sent as 'have' lines during push negotiation.
- This change also updates the use of --negotiation-tip into
- --negotiation-restrict now that the new synonym exists.
-
+ Reviewed-by: Matthew John Cheetham <[email protected]>
## Documentation/config/remote.adoc ##
@@ Documentation/config/remote.adoc: command-line option. If `--negotiation-restri
Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios.
-@@ Documentation/config/remote.adoc: unconditionally on top of those heuristically selected commits. This
- option is also used during push negotiation when push.negotiate is
- enabled.
+@@ Documentation/config/remote.adoc: negotiation algorithm still runs and advertises its own selected commits,
+ but the refs matching remote.<name>.negotiationInclude are sent
+ unconditionally on top of those heuristically selected commits.
+
+These values also influence negotiation during git push if
+push.negotiate is enabled.
Signature · 1 line
gitgitgadget
PATCH v5 2/8 fetch: add --negotiation-restrict option
From: Derrick Stolee <[email protected]>
The --negotiation-tip option to 'git fetch' and 'git pull' allows users to specify that they want to focus negotiation on a small set of references. This is a _restriction_ on the negotiation set, helping to focus the negotiation when the ref count is high. However, it doesn't allow for the ability to opportunistically select references beyond that list.
This subtle detail that this is a 'maximum set' and not a 'minimum set' is not immediately clear from the option name. This makes it more complicated to add a new option that provides the complementary behavior of a minimum set.
For now, create a new synonym option, --negotiation-restrict, that behaves identically to --negotiation-tip. Update the documentation to make it clear that this new name is the preferred option, but we keep the old name for compatibility. Mark --negotiation-tip as an alias of the new, preferred option.
Update a few warning messages with the new option, but also make them translatable with the option name inserted by formatting. At least one of these messages will be reused later for a new option.
--- Documentation/config/fetch.adoc | 2 +- Documentation/fetch-options.adoc | 6 +++++- builtin/fetch.c | 13 ++++++++----- builtin/pull.c | 3 ++- send-pack.c | 2 +- t/t5510-fetch.sh | 25 +++++++++++++++++++++++++ t/t5702-protocol-v2.sh | 4 ++-- transport-helper.c | 3 ++- 8 files changed, 46 insertions(+), 12 deletions(-)
Documentation/config/fetch.adoc +1 -1
@@ -76,7 +76,7 @@ default is `skipping`. Unknown values will cause `git fetch` to error out. + -See also the `--negotiate-only` and `--negotiation-tip` options to +See also the `--negotiate-only` and `--negotiation-restrict` options to linkgit:git-fetch[1]. `fetch.showForcedUpdates`::
Documentation/fetch-options.adoc +5 -1
@@ -49,6 +49,7 @@ the current repository has the same history as the source repository. `.git/shallow`. This option updates `.git/shallow` and accepts such refs. +`--negotiation-restrict=(<commit>|<glob>)`:: `--negotiation-tip=(<commit>|<glob>)`:: By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt to
@@ -58,6 +59,9 @@ the current repository has the same history as the source repository. local ref is likely to have commits in common with the upstream ref being fetched. + +`--negotiation-restrict` is the preferred name for this option; +`--negotiation-tip` is accepted as a synonym. ++ This option may be specified more than once; if so, Git will report commits reachable from any of the given commits. +
@@ -71,7 +75,7 @@ configuration variables documented in linkgit:git-config[1], and the `--negotiate-only`:: Do not fetch anything from the server, and instead print the - ancestors of the provided `--negotiation-tip=` arguments, + ancestors of the provided `--negotiation-restrict=` arguments, which we have in common with the server. + This is incompatible with `--recurse-submodules=(yes|on-demand)`.
builtin/fetch.c +8 -5
@@ -1558,8 +1558,8 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)refs_for_each_ref_ext (get_main_ref_store (the_repository), add_oid, oids, &opts);if (old_nr == oids->nr) -warning ("ignoring --negotiation-tip=%s because it does not match any refs" , - s); +warning (_ ("ignoring %s=%s because it does not match any refs" ), +"--negotiation-restrict" , s); } smart_options->negotiation_tips = oids; }
@@ -1599,7 +1599,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,if (transport->smart_options)add_negotiation_tips (transport->smart_options);else -warning ("ignoring --negotiation-tip because the protocol does not support it" ); +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-restrict" ); }return transport; }
@@ -2565,8 +2566,9 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-tip" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )), +OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
@@ -2657,7 +2659,8 @@ int cmd_fetch(int argc, }if (negotiate_only && !negotiation_tip.nr) -die (_ ("--negotiate-only needs one or more --negotiation-tip=*" )); +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" );if (deepen_relative) {if (deepen_relative <0 )
builtin/pull.c +2 -1
@@ -996,9 +996,10 @@ int cmd_pull(int argc,OPT_PASSTHRU ('6' ,"ipv6" , &opt_ipv6,NULL ,N_ ("use IPv6 addresses only" ),PARSE_OPT_NOARG ), -OPT_PASSTHRU_ARGV (0 ,"negotiation-tip" , &opt_fetch,N_ ("revision" ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-restrict" , &opt_fetch,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" ),0 ), +OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
send-pack.c +1 -1
@@ -447,7 +447,7 @@ static void get_commons_through_negotiation(struct repository *r,strvec_pushl (&child.args,"fetch" ,"--negotiate-only" ,NULL );for (ref = remote_refs; ref; ref = ref->next) {if (!is_null_oid (&ref->new_oid)) { -strvec_pushf (&child.args,"--negotiation-tip=%s" , +strvec_pushf (&child.args,"--negotiation-restrict=%s" ,oid_to_hex (&ref->new_oid)); nr_negotiation_tip++; }
t/t5510-fetch.sh +25 -0
@@ -1460,6 +1460,31 @@ EOF test_cmp fatal-expect fatal-actual ' +test_expect_success '--negotiation-restrict limits "have" lines sent' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 --negotiation-restrict=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict understands globs' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=*_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' ' + setup_negotiation_tip server server 0 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + --negotiation-tip=beta_1 \ + origin alpha_s beta_s && + check_negotiation_tip +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
t/t5702-protocol-v2.sh +2 -2
@@ -869,14 +869,14 @@ setup_negotiate_only () { test_commit -C client three } -test_expect_success 'usage: --negotiate-only without --negotiation-tip' ' +test_expect_success 'usage: --negotiate-only without --negotiation-restrict' ' SERVER="server" && URI="file://$(pwd)/server" && setup_negotiate_only "$SERVER" "$URI" && cat >err.expect <<-\EOF && - fatal: --negotiate-only needs one or more --negotiation-tip=* + fatal: --negotiate-only needs one or more --negotiation-restrict=* EOF test_must_fail git -c protocol.version=2 -C client fetch \
transport-helper.c +2 -1
@@ -755,7 +755,8 @@ static int fetch_refs(struct transport *transport, }if (data->transport_options.negotiation_tips) -warning ("Ignoring --negotiation-tip because the protocol does not support it." ); +warning (_ ("ignoring %s because the protocol does not support it." ), +"--negotiation-restrict" );if (data->fetch)return fetch_with_fetch (transport, nr_heads, to_fetch);
Signature · 1 line
gitgitgadget
PATCH v5 3/8 transport: rename negotiation_tips
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
The previous change added the --negotiation-restrict synonym for the --negotiation-tip option for 'git fetch'. In anticipation of adding a new option that behaves similarly but with distinct changes to its behavior, rename the internal representation of this data from 'negotiation_tips' to 'negotiation_restrict_tips'.
The 'tips' part is kept because this is an oid_array in the transport layer. This requires the builtin to handle parsing refs into collections of oids so the transport layer can handle this cleaner form of the data.
Also update the string_list used to store the inputs from command-line options.
--- builtin/fetch.c | 18 +++++++++--------- fetch-pack.c | 18 +++++++++--------- fetch-pack.h | 4 ++-- transport-helper.c | 2 +- transport.c | 10 +++++----- transport.h | 4 ++-- 6 files changed, 28 insertions(+), 28 deletions(-)
builtin/fetch.c +9 -9
@@ -98,7 +98,7 @@ static struct transport *gtransport;static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ; -static struct string_list negotiation_tip =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;
@@ -1534,13 +1534,13 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_tips (struct git_transport_options *smart_options) +static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) {struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i; -for (i =0 ; i < negotiation_tip.nr; i++) { -const char *s = negotiation_tip.items[i].string; +for (i =0 ; i < negotiation_restrict.nr; i++) { +const char *s = negotiation_restrict.items[i].string;struct refs_for_each_ref_options opts = { .pattern = s, };
@@ -1561,7 +1561,7 @@ static void add_negotiation_tips(struct git_transport_options *smart_options)warning (_ ("ignoring %s=%s because it does not match any refs" ),"--negotiation-restrict" , s); } - smart_options->negotiation_tips = oids; + smart_options->negotiation_restrict_tips = oids; } static struct transport *prepare_transport(struct remote *remote, int deepen,
@@ -1595,9 +1595,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,set_option (transport,TRANS_OPT_LIST_OBJECTS_FILTER , spec);set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" ); } -if (negotiation_tip.nr) { +if (negotiation_restrict.nr) {if (transport->smart_options) -add_negotiation_tips (transport->smart_options); +add_negotiation_restrict_tips (transport->smart_options);else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );
@@ -2566,7 +2566,7 @@ int cmd_fetch(int argc, N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),OPT_STRING_LIST ('o' ,"server-option" , &server_options,N_ ("server-specific" ),N_ ("option to transmit" )),OPT_IPVERSION (&family), -OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_tip,N_ ("revision" ), +OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), OPT_BOOL(0, "negotiate-only", &negotiate_only,
@@ -2658,7 +2658,7 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_tip.nr) +if (negotiate_only && !negotiation_restrict.nr)die (_ ("%s needs one or more %s" ),"--negotiate-only" ,"--negotiation-restrict=*" );
fetch-pack.c +9 -9
@@ -291,21 +291,21 @@ static int next_flush(int stateless_rpc, int count) }static void mark_tips (struct fetch_negotiator *negotiator, -const struct oid_array *negotiation_tips) +const struct oid_array *negotiation_restrict_tips) {struct refs_for_each_ref_options opts = { .flags =REFS_FOR_EACH_INCLUDE_BROKEN , };int i; -if (!negotiation_tips) { +if (!negotiation_restrict_tips) {refs_for_each_ref_ext (get_main_ref_store (the_repository), rev_list_insert_ref_oid, negotiator, &opts);return ; } -for (i =0 ; i < negotiation_tips->nr; i++) -rev_list_insert_ref (negotiator, &negotiation_tips->oid[i]); +for (i =0 ; i < negotiation_restrict_tips->nr; i++) +rev_list_insert_ref (negotiator, &negotiation_restrict_tips->oid[i]);return ; }
@@ -355,7 +355,7 @@ static int find_common(struct fetch_negotiator *negotiator,PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object); fetching =0 ;
@@ -1728,7 +1728,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,else state =FETCH_SEND_REQUEST ; -mark_tips (negotiator, args->negotiation_tips); +mark_tips (negotiator, args->negotiation_restrict_tips);for_each_cached_alternate (negotiator, insert_one_alternate_object);break ;
@@ -2177,7 +2177,7 @@ static void clear_common_flag(struct oidset *s) } } -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
@@ -2195,13 +2195,13 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,timestamp_t min_generation =GENERATION_NUMBER_INFINITY ;fetch_negotiator_init (the_repository, &negotiator); -mark_tips (&negotiator, negotiation_tips); +mark_tips (&negotiator, negotiation_restrict_tips);packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE |PACKET_READ_DIE_ON_ERR_PACKET ); -oid_array_for_each ((struct oid_array *) negotiation_tips, +oid_array_for_each ((struct oid_array *) negotiation_restrict_tips, add_to_object_array, &nt_object_array);
fetch-pack.h +2 -2
@@ -21,7 +21,7 @@ struct fetch_pack_args {* If not NULL, during packfile negotiation, fetch-pack will send "have" * lines only with these tips and their ancestors. */ -const struct oid_array *negotiation_tips; +const struct oid_array *negotiation_restrict_tips; unsigned deepen_relative:1; unsigned quiet:1;
@@ -89,7 +89,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,* In the capability advertisement that has happened prior to invoking this * function, the "wait-for-done" capability must be present. */ -void negotiate_using_fetch(const struct oid_array *negotiation_tips, +void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips, const struct string_list *server_options, int stateless_rpc, int fd[],
transport-helper.c +1 -1
@@ -754,7 +754,7 @@ static int fetch_refs(struct transport *transport,set_helper_option (transport,"filter" , spec); } -if (data->transport_options.negotiation_tips) +if (data->transport_options.negotiation_restrict_tips)warning (_ ("ignoring %s because the protocol does not support it." ),"--negotiation-restrict" );
transport.c +5 -5
@@ -463,7 +463,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.refetch = data->options.refetch; args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; - args.negotiation_tips = data->options.negotiation_tips; + args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -491,7 +491,7 @@ static int fetch_refs_via_pack(struct transport *transport,warning (_ ("server does not support wait-for-done" )); ret =-1 ; }else { - negotiate_using_fetch(data->options.negotiation_tips, + negotiate_using_fetch(data->options.negotiation_restrict_tips, transport->server_options, transport->stateless_rpc, data->fd,
@@ -979,9 +979,9 @@ static int disconnect_git(struct transport *transport)finish_connect (data->conn); } -if (data->options.negotiation_tips) { -oid_array_clear (data->options.negotiation_tips); -free (data->options.negotiation_tips); +if (data->options.negotiation_restrict_tips) { +oid_array_clear (data->options.negotiation_restrict_tips); +free (data->options.negotiation_restrict_tips); }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);
transport.h +2 -2
@@ -40,13 +40,13 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_tips in struct fetch_pack_args. +* negotiation_restrict_tips in struct fetch_pack_args. * * This field is only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ -struct oid_array *negotiation_tips; +struct oid_array *negotiation_restrict_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
Signature · 1 line
gitgitgadget
PATCH v5 4/8 remote: add remote.*.negotiationRestrict config
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
In a previous change, the --negotiation-restrict command-line option of 'git fetch' was added as a synonym of --negotiation-tip. Both of these options restrict the set of 'haves' the client can send as part of negotiation.
This was previously not available via a configuration option. Add a new 'remote.<name>.negotiationRestrict' multi-valued config option that updates 'git fetch <name>' to use these restrictions by default.
If the user provides even one --negotiation-restrict argument, then the config is ignored.
An empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
--- Documentation/config/remote.adoc | 18 ++++++++++++++++++ builtin/fetch.c | 28 +++++++++++++++++++++------- remote.c | 5 +++++ remote.h | 1 + t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++ 5 files changed, 71 insertions(+), 7 deletions(-)
Documentation/config/remote.adoc +18 -0
@@ -107,6 +107,24 @@ priority configuration file (e.g. `.git/config` in a repository) to clear the values inherited from a lower priority configuration files (e.g. `$HOME/.gitconfig`). +remote.<name>.negotiationRestrict:: + When negotiating with this remote during `git fetch`, restrict the + commits advertised as "have" lines to only those reachable from refs + matching the given patterns. This multi-valued config option behaves + like `--negotiation-restrict` on the command line. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`) or a +glob pattern (e.g. `refs/heads/release/*`). The pattern syntax is the +same as for `--negotiation-restrict`. ++ +These config values are used as defaults for the `--negotiation-restrict` +command-line option. If `--negotiation-restrict` (or its synonym +`--negotiation-tip`) is specified on the command line, then the config +values are not used. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
builtin/fetch.c +21 -7
@@ -1601,6 +1601,19 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" ); + }else if (remote->negotiation_restrict.nr) { +struct string_list_item *item; +for_each_string_list_item (item, &remote->negotiation_restrict) +string_list_append (&negotiation_restrict, item->string); +if (transport->smart_options) +add_negotiation_restrict_tips (transport->smart_options); +else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
@@ -2658,10 +2671,6 @@ int cmd_fetch(int argc, config.display_format =DISPLAY_FORMAT_PORCELAIN ; } -if (negotiate_only && !negotiation_restrict.nr) -die (_ ("%s needs one or more %s" ),"--negotiate-only" , -"--negotiation-restrict=*" ); -if (deepen_relative) {if (deepen_relative <0 )die (_ ("negative depth in --deepen is not supported" ));
@@ -2749,14 +2758,19 @@ int cmd_fetch(int argc,if (!remote)die (_ ("must supply remote when using --negotiate-only" )); gtransport =prepare_transport (remote,1 , &filter_options); -if (gtransport->smart_options) { - gtransport->smart_options->acked_commits = &acked_commits; - }else { + +if (!gtransport->smart_options) {warning (_ ("protocol does not support --negotiate-only, exiting" )); result =1 ;trace2_region_leave ("fetch" ,"negotiate-only" , the_repository); goto cleanup; } +if (!gtransport->smart_options->negotiation_restrict_tips) +die (_ ("%s needs one or more %s" ),"--negotiate-only" , +"--negotiation-restrict=*" ); + + gtransport->smart_options->acked_commits = &acked_commits; +if (server_options.nr) gtransport->server_options = &server_options; result =transport_fetch_refs (gtransport,NULL );
remote.c +5 -0
@@ -152,6 +152,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_push (&ret->push);refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options); +string_list_init_dup (&ret->negotiation_restrict);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -179,6 +180,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy);FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 ); +string_list_clear (&remote->negotiation_restrict,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -562,6 +564,9 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"serveroption" )) {return parse_transport_option (key, value, &remote->server_options); + }else if (!strcmp (subkey,"negotiationrestrict" )) { +return parse_transport_option (key, value, + &remote->negotiation_restrict); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -117,6 +117,7 @@ struct remote {char *http_proxy_authmethod;struct string_list server_options; +struct string_list negotiation_restrict;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +26 -0
@@ -1485,6 +1485,32 @@ test_expect_success '--negotiation-restrict and --negotiation-tip can be mixed' check_negotiation_tip ' +test_expect_success 'remote.<name>.negotiationRestrict used as default' ' + setup_negotiation_tip server server 0 && + + # test the reset of the list on an empty value + git -C client config --add remote.origin.negotiationRestrict alpha_2 && + git -C client config --add remote.origin.negotiationRestrict "" && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + origin alpha_s beta_s && + check_negotiation_tip +' + +test_expect_success 'CLI --negotiation-restrict overrides remote config' ' + setup_negotiation_tip server server 0 && + git -C client config --add remote.origin.negotiationRestrict alpha_1 && + git -C client config --add remote.origin.negotiationRestrict beta_1 && + ALPHA_1=$(git -C client rev-parse alpha_1) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --negotiation-restrict=alpha_1 \ + origin alpha_s beta_s && + test_grep "fetch> have $ALPHA_1" trace && + BETA_1=$(git -C client rev-parse beta_1) && + test_grep ! "fetch> have $BETA_1" trace +' + test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' ' git init df-conflict && (
Signature · 1 line
gitgitgadget
PATCH v5 5/8 negotiator: add have_sent() interface
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
In a future change, we will introduce a capability to choose specific commit OIDs as 'have's in fetch negotiation, with the ability to have the negotiator choose more 'have's to increase coverage beyond that required core set. The negotiator works to avoid emitting 'have's that can reach each other, but that logic is hidden beneath the negotiator's iterator function pointer ('next'). We need a way to communicate to the negotiator that we have picked a 'have' so it could incorporate that into its logic.
Add a have_sent() method to the fetch_negotiator interface. This is the signal that allows the negotiator to track the commit as already shown and can perform the proper bookkeeping to avoid emitting those objects or anything they can reach.
For our non-trivial negotiators, it is sufficient to mark these commits as common, so the implementation is quite simple. This logic will be exercised in the next change.
--- fetch-negotiator.h | 9 +++++++++ negotiator/default.c | 8 ++++++++ negotiator/noop.c | 7 +++++++ negotiator/skipping.c | 8 ++++++++ 4 files changed, 32 insertions(+)
fetch-negotiator.h +9 -0
@@ -47,6 +47,15 @@ struct fetch_negotiator {*/ int (*ack)(struct fetch_negotiator *,struct commit *); +/* +* Inform the negotiator that this commit has already been sent as +* a "have" line outside of the negotiator's control. The negotiator +* should avoid outputting it from next() and may use it to optimize +* further negotiation (e.g., by treating it and its ancestors as +* common). +*/ +void (*have_sent)(struct fetch_negotiator *,struct commit *); +void (*release)(struct fetch_negotiator *);/* internal use */
negotiator/default.c +8 -0
@@ -175,6 +175,13 @@ static int ack(struct fetch_negotiator *n, struct commit *c)return known_to_be_common; } +static void have_sent (struct fetch_negotiator *n,struct commit *c) +{ +if (repo_parse_commit (the_repository, c)) +return ; +mark_common (n->data, c,0 ,0 ); +} +static void release (struct fetch_negotiator *n) {clear_prio_queue (&((struct negotiation_state *)n->data)->rev_list);
@@ -188,6 +195,7 @@ void default_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =CALLOC_ARRAY (ns,1 ); ns->rev_list.compare = compare_commits_by_commit_date;
negotiator/noop.c +7 -0
@@ -29,6 +29,12 @@ static int ack(struct fetch_negotiator *n UNUSED, struct commit *c UNUSED)return 0 ; } +static void have_sent(struct fetch_negotiator *n UNUSED, + struct commit *c UNUSED) +{ +/* nothing to do */ +} + static void release(struct fetch_negotiator *n UNUSED) {/* nothing to release */
@@ -40,6 +46,7 @@ void noop_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =NULL ; }
negotiator/skipping.c +8 -0
@@ -243,6 +243,13 @@ static int ack(struct fetch_negotiator *n, struct commit *c)return known_to_be_common; } +static void have_sent (struct fetch_negotiator *n,struct commit *c) +{ +if (repo_parse_commit (the_repository, c)) +return ; +mark_common (n->data, c); +} +static void release (struct fetch_negotiator *n) {struct data *data = n->data;
@@ -259,6 +266,7 @@ void skipping_negotiator_init(struct fetch_negotiator *negotiator) negotiator->add_tip = add_tip; negotiator->next = next; negotiator->ack = ack; + negotiator->have_sent = have_sent; negotiator->release = release; negotiator->data =CALLOC_ARRAY (data,1 ); data->rev_list.compare = compare;
Signature · 1 line
gitgitgadget
PATCH v5 6/8 fetch: add --negotiation-include option for negotiation
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new --negotiation-include option to 'git fetch', which ensures that certain ref tips are always sent as 'have' lines during fetch negotiation, regardless of what the negotiation algorithm selects.
This is useful when the repository has a large number of references, so the normal negotiation algorithm truncates the list. This is especially important in repositories with long parallel commit histories. For example, a repo could have a 'dev' branch for development and a 'release' branch for released versions. If the 'dev' branch isn't selected for negotiation, then it's not a big deal because there are many in-progress development branches with a shared history. However, if 'release' is not selected for negotiation, then the server may think that this is the first time the client has asked for that reference, causing a full download of its parallel commit history (and any extra data that may be unique to that branch). This is based on a real example where certain fetches would grow to 60+ GB when a release branch updated.
This option is a complement to --negotiation-restrict, which reduces the negotiation ref set to a specific list. In the earlier example, using --negotiation-restrict to focus the negotiation to 'dev' and 'release' would avoid those problematic downloads, but would still not allow advertising potentially-relevant user branches. In this way, the 'include' version solves the problem I mention while allowing negotiation to pick other references opportunistically. The two options can also be combined to allow the best of both worlds.
The argument may be an exact ref name or a glob pattern. Non-existent refs are silently ignored. This behavior is also updated in the ref matching logic for the related --negotiation-restrict option to match.
The implementation outputs the requested objects as haves before the negotiator performs its own algorithm to choose the next haves. Use the new have_sent() interface to signal these have commits were sent before engaging with the negotiator's next() iterator.
Also add --negotiation-include to 'git pull' passthrough options.
--- Documentation/fetch-options.adoc | 19 +++++++ builtin/fetch.c | 38 ++++++++++--- builtin/pull.c | 3 ++ fetch-pack.c | 81 +++++++++++++++++++++++++--- fetch-pack.h | 6 ++- t/t5510-fetch.sh | 91 ++++++++++++++++++++++++++++++++ transport.c | 8 ++- transport.h | 5 +- 8 files changed, 232 insertions(+), 19 deletions(-)
Documentation/fetch-options.adoc +19 -0
@@ -73,6 +73,25 @@ See also the `fetch.negotiationAlgorithm` and `push.negotiate` configuration variables documented in linkgit:git-config[1], and the `--negotiate-only` option below. +`--negotiation-include=(<commit>|<glob>)`:: + Ensure that the commits at the given tips are always sent as "have" + lines during fetch negotiation, regardless of what the negotiation + algorithm selects. This is useful to guarantee that common + history reachable from specific refs is always considered, even + when `--negotiation-restrict` restricts the set of tips or when + the negotiation algorithm would otherwise skip them. ++ +This option may be specified more than once; if so, each commit is sent +unconditionally. ++ +The argument may be an exact ref name (e.g. `refs/heads/release`), an +object hash, or a glob pattern (e.g. `refs/heads/release/{asterisk}`). +The pattern syntax is the same as for `--negotiation-restrict`. ++ +If `--negotiation-restrict` is used, the have set is first restricted by +that option and then increased to include the tips specified by +`--negotiation-include`. + `--negotiate-only`:: Do not fetch anything from the server, and instead print the ancestors of the provided `--negotiation-restrict=` arguments,
builtin/fetch.c +30 -8
@@ -99,6 +99,7 @@ static struct transport *gsecondary;static struct refspec refmap =REFSPEC_INIT_FETCH ;static struct string_list server_options =STRING_LIST_INIT_DUP ;static struct string_list negotiation_restrict =STRING_LIST_INIT_NODUP ; +static struct string_list negotiation_include =STRING_LIST_INIT_NODUP ;struct fetch_config {enum display_format display_format;
@@ -1534,23 +1535,29 @@ static int add_oid(const struct reference *ref, void *cb_data)return 0 ; } -static void add_negotiation_restrict_tips (struct git_transport_options *smart_options) +static void add_negotiation_tips (struct string_list *input_list, +struct oid_array **output_list, +const char *argname) {struct oid_array *oids =xcalloc (1 ,sizeof (*oids));int i; -for (i =0 ; i < negotiation_restrict.nr; i++) { -const char *s = negotiation_restrict.items[i].string; +for (i =0 ; i < input_list->nr; i++) { +const char *s = input_list->items[i].string;struct refs_for_each_ref_options opts = { .pattern = s, };int old_nr;if (!has_glob_specials (s)) {struct object_id oid; + +/* Ignore missing reference. */ if (repo_get_oid (the_repository, s, &oid)) -die (_ ("%s is not a valid object" ), s); +continue ; +/* Fail on missing object pointed by ref. */ if (!odb_has_object (the_repository->objects, &oid,0 ))die (_ ("the object %s does not exist" ), s); +oid_array_append (oids, &oid);continue ; }
@@ -1559,9 +1566,9 @@ static void add_negotiation_restrict_tips(struct git_transport_options *smart_op add_oid, oids, &opts);if (old_nr == oids->nr)warning (_ ("ignoring %s=%s because it does not match any refs" ), -"--negotiation-restrict" , s); + argname, s); } - smart_options->negotiation_restrict_tips = oids; + *output_list = oids; } static struct transport *prepare_transport(struct remote *remote, int deepen,
@@ -1597,7 +1604,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen, }if (negotiation_restrict.nr) {if (transport->smart_options) -add_negotiation_restrict_tips (transport->smart_options); +add_negotiation_tips (&negotiation_restrict, + &transport->smart_options->negotiation_restrict_tips, +"--negotiation-restrict" );else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-restrict" );
@@ -1606,7 +1615,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,for_each_string_list_item (item, &remote->negotiation_restrict)string_list_append (&negotiation_restrict, item->string);if (transport->smart_options) -add_negotiation_restrict_tips (transport->smart_options); +add_negotiation_tips (&negotiation_restrict, + &transport->smart_options->negotiation_restrict_tips, +"--negotiation-restrict" );else {struct strbuf config_name =STRBUF_INIT ;strbuf_addf (&config_name,"remote.%s.negotiationRestrict" , remote->name);
@@ -1615,6 +1626,15 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,strbuf_release (&config_name); } } +if (negotiation_include.nr) { +if (transport->smart_options) +add_negotiation_tips (&negotiation_include, + &transport->smart_options->negotiation_include_tips, +"--negotiation-include" ); +else +warning (_ ("ignoring %s because the protocol does not support it" ), +"--negotiation-include" ); + }return transport; }
@@ -2582,6 +2602,8 @@ int cmd_fetch(int argc,OPT_STRING_LIST (0 ,"negotiation-restrict" , &negotiation_restrict,N_ ("revision" ),N_ ("report that we have only objects reachable from this object" )),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), +OPT_STRING_LIST (0 ,"negotiation-include" , &negotiation_include,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" )),OPT_BOOL (0 ,"negotiate-only" , &negotiate_only,N_ ("do not fetch a packfile; instead, print ancestors of negotiation tips" )), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
builtin/pull.c +3 -0
@@ -1000,6 +1000,9 @@ int cmd_pull(int argc, N_("report that we have only objects reachable from this object"), 0),OPT_ALIAS (0 ,"negotiation-tip" ,"negotiation-restrict" ), +OPT_PASSTHRU_ARGV (0 ,"negotiation-include" , &opt_fetch,N_ ("revision" ), +N_ ("ensure this ref is always sent as a negotiation have" ), +0 ),OPT_BOOL (0 ,"show-forced-updates" , &opt_show_forced_updates,N_ ("check for forced-updates on all updated branches" )), OPT_PASSTHRU(0, "set-upstream", &set_upstream, NULL,
fetch-pack.c +75 -6
186 lines not on this page · raw message
fetch-pack.h +4 -2
@@ -19,9 +19,10 @@ struct fetch_pack_args {/* * If not NULL, during packfile negotiation, fetch-pack will send "have" -* lines only with these tips and their ancestors. +* lines for all _include_ tips and then a subset of the _restrict_ tips. */ const struct oid_array *negotiation_restrict_tips; +const struct oid_array *negotiation_include_tips; unsigned deepen_relative:1; unsigned quiet:1;
@@ -93,7 +94,8 @@ void negotiate_using_fetch(const struct oid_array *negotiation_restrict_tips,const struct string_list *server_options, int stateless_rpc, int fd[], - struct oidset *acked_commits); + struct oidset *acked_commits, + const struct oid_array *negotiation_include_tips);/* * Print an appropriate error message for each sought ref that wasn't
t/t5510-fetch.sh +91 -0
103 lines not on this page · raw message
transport.c +7 -1
@@ -464,6 +464,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; args.negotiation_restrict_tips = data->options.negotiation_restrict_tips; + args.negotiation_include_tips = data->options.negotiation_include_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow;if (!data->finished_handshake) {
@@ -495,7 +496,8 @@ static int fetch_refs_via_pack(struct transport *transport, transport->server_options, transport->stateless_rpc, data->fd, - data->options.acked_commits); + data->options.acked_commits, + data->options.negotiation_include_tips); ret =0 ; } goto cleanup;
@@ -983,6 +985,10 @@ static int disconnect_git(struct transport *transport)oid_array_clear (data->options.negotiation_restrict_tips);free (data->options.negotiation_restrict_tips); } +if (data->options.negotiation_include_tips) { +oid_array_clear (data->options.negotiation_include_tips); +free (data->options.negotiation_include_tips); + }list_objects_filter_release (&data->options.filter_options);oid_array_clear (&data->extra_have);oid_array_clear (&data->shallow);
transport.h +3 -2
@@ -40,13 +40,14 @@ struct git_transport_options {/* * This is only used during fetch. See the documentation of -* negotiation_restrict_tips in struct fetch_pack_args. +* these member names in struct fetch_pack_args. * -* This field is only supported by transports that support connect or +* These fields are only supported by transports that support connect or * stateless_connect. Set this field directly instead of using * transport_set_option(). */ struct oid_array *negotiation_restrict_tips; +struct oid_array *negotiation_include_tips;/* * If allocated, whenever transport_fetch_refs() is called, add known
Signature · 1 line
gitgitgadget
PATCH v5 7/8 remote: add remote.*.negotiationInclude config
In reply to Derrick Stolee via GitGitGadgetFrom: Derrick Stolee <[email protected]>
Add a new 'remote.<name>.negotiationInclude' multi-valued config option that provides default values for --negotiation-include when no --negotiation-include arguments are specified over the command line. This is a mirror of how 'remote.<name>.negotiationRestrict' specifies defaults for the --negotiation-restrict arguments.
Each value is either an exact ref name or a glob pattern whose tips should always be sent as 'have' lines during negotiation. The config values are resolved through the same resolve_negotiation_include() codepath as the CLI options.
This option is additive with the normal negotiation process: the negotiation algorithm still runs and advertises its own selected commits, but the refs matching the config are sent unconditionally on top of those heuristically selected commits.
Similar to the negotiationRestrict config, an empty value resets the value list to allow ignoring earlier config values, such as those that might be set in system or global config.
--- Documentation/config/remote.adoc | 25 ++++++++++++++++ Documentation/fetch-options.adoc | 4 +++ builtin/fetch.c | 12 ++++++++ remote.c | 5 ++++ remote.h | 1 + t/t5510-fetch.sh | 49 ++++++++++++++++++++++++++++++++ 6 files changed, 96 insertions(+)
Documentation/config/remote.adoc +25 -0
@@ -125,6 +125,31 @@ values are not used. Blank values signal to ignore all previous values, allowing a reset of the list from broader config scenarios. +remote.<name>.negotiationInclude:: + When negotiating with this remote during `git fetch`, the client + advertises a list of commits that exist locally. In repos with + many references, this list of "haves" can be truncated. Depending + on data shape, dropping certain references may be expensive. This + multi-valued config option specifies references, commit hashes, + or ref pattern globs whose tips should always be sent as "have" + commits during fetch negotiation with this remote. ++ +Each value is either an exact ref name (e.g. `refs/heads/release`), a +commit hash, or a glob pattern (e.g. `refs/heads/release/*`). The +pattern syntax is the same as for `--negotiation-include`. ++ +These config values are used as defaults for the `--negotiation-include` +command-line option. If `--negotiation-include` is specified on the +command line, then the config values are not used. ++ +This option is additive with the normal negotiation process: the +negotiation algorithm still runs and advertises its own selected commits, +but the refs matching `remote.<name>.negotiationInclude` are sent +unconditionally on top of those heuristically selected commits. ++ +Blank values signal to ignore all previous values, allowing a reset of +the list from broader config scenarios. + remote.<name>.followRemoteHEAD:: How linkgit:git-fetch[1] should handle updates to `remotes/<name>/HEAD` when fetching using the configured refspecs of a remote.
Documentation/fetch-options.adoc +4 -0
@@ -91,6 +91,10 @@ The pattern syntax is the same as for `--negotiation-restrict`. If `--negotiation-restrict` is used, the have set is first restricted by that option and then increased to include the tips specified by `--negotiation-include`. ++ +If this option is not specified on the command line, then any +`remote.<name>.negotiationInclude` config values for the current remote +are used instead. `--negotiate-only`:: Do not fetch anything from the server, and instead print the
builtin/fetch.c +12 -0
@@ -1634,6 +1634,18 @@ static struct transport *prepare_transport(struct remote *remote, int deepen,else warning (_ ("ignoring %s because the protocol does not support it" ),"--negotiation-include" ); + }else if (remote->negotiation_include.nr) { +if (transport->smart_options) { +add_negotiation_tips (&remote->negotiation_include, + &transport->smart_options->negotiation_include_tips, +"--negotiation-include" ); + }else { +struct strbuf config_name =STRBUF_INIT ; +strbuf_addf (&config_name,"remote.%s.negotiationInclude" , remote->name); +warning (_ ("ignoring %s because the protocol does not support it" ), + config_name.buf); +strbuf_release (&config_name); + } }return transport; }
remote.c +5 -0
@@ -153,6 +153,7 @@ static struct remote *make_remote(struct remote_state *remote_state,refspec_init_fetch (&ret->fetch);string_list_init_dup (&ret->server_options);string_list_init_dup (&ret->negotiation_restrict); +string_list_init_dup (&ret->negotiation_include);ALLOC_GROW (remote_state->remotes, remote_state->remotes_nr +1 , remote_state->remotes_alloc);
@@ -181,6 +182,7 @@ static void remote_clear(struct remote *remote)FREE_AND_NULL (remote->http_proxy_authmethod);string_list_clear (&remote->server_options,0 );string_list_clear (&remote->negotiation_restrict,0 ); +string_list_clear (&remote->negotiation_include,0 ); }static void add_merge (struct branch *branch,const char *name)
@@ -567,6 +569,9 @@ static int handle_config(const char *key, const char *value, }else if (!strcmp (subkey,"negotiationrestrict" )) {return parse_transport_option (key, value, &remote->negotiation_restrict); + }else if (!strcmp (subkey,"negotiationinclude" )) { +return parse_transport_option (key, value, + &remote->negotiation_include); }else if (!strcmp (subkey,"followremotehead" )) {const char *no_warn_branch;if (!strcmp (value,"never" ))
remote.h +1 -0
@@ -118,6 +118,7 @@ struct remote {struct string_list server_options;struct string_list negotiation_restrict; +struct string_list negotiation_include;enum follow_remote_head_settings follow_remote_head;const char *no_warn_branch;
t/t5510-fetch.sh +49 -0
55 lines not on this page · raw message
Signature · 1 line
gitgitgadget