Fix overriding of partial clone filter when lazy-fetching
cover Fix overriding of partial clone filter when lazy-fetching
This was noticed at $DAYJOB when in a partial clone with a filter other than blob:none. The 1st patch is a code cleanup (so that the 2nd patch does not need to take into account the unused variable), and the 2nd patch contains the bug fix.
Jonathan Tan (2):
promisor-remote: remove unused variable
fetch: do not override partial clone filter
builtin/fetch.c | 2 +- list-objects-filter-options.c | 10 +++++++++- promisor-remote.c | 5 ----- promisor-remote.h | 2 +- t/t5601-clone.sh | 5 +++++ 5 files changed, 16 insertions(+), 8 deletions(-)
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
PATCH 2/2 fetch: do not override partial clone filter
When a fetch with the --filter argument is made, the configured default filter is set even if one already exists. This change was made in 5e46139376 ("builtin/fetch: remove unique promisor remote limitation", 2019-06-25) - in particular, changing from:
* If this is the FIRST partial-fetch request, we enable partial
* on this repo and remember the given filter-spec as the default
* for subsequent fetches to this remote.
to:
* If this is a partial-fetch request, we enable partial on
* this repo if not already enabled and remember the given
* filter-spec as the default for subsequent fetches to this
* remote.
(The given filter-spec is "remembered" even if there is already an existing one.)
This is problematic whenever a lazy fetch is made, because lazy fetches are made using "git fetch --filter=blob:none", but this will also happen if the user invokes "git fetch --filter=<filter>" manually. Therefore, restore the behavior prior to 5e46139376, which writes a filter-spec only if the current fetch request is the first partial-fetch one (for that remote).
--- builtin/fetch.c | 2 +- list-objects-filter-options.c | 10 +++++++++- t/t5601-clone.sh | 5 +++++ 3 files changed, 15 insertions(+), 2 deletions(-)
builtin/fetch.c +1 -1
@@ -1677,7 +1677,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)* If this is a partial-fetch request, we enable partial on * this repo if not already enabled and remember the given * filter-spec as the default for subsequent fetches to this -* remote. +* remote if there is currently no default filter-spec. */ if (filter_options.choice) {partial_clone_register (remote->name, &filter_options);
list-objects-filter-options.c +9 -1
@@ -344,11 +344,19 @@ void partial_clone_register(const char *remote, struct list_objects_filter_options *filter_options) { + struct promisor_remote *promisor_remote;char *cfg_name;char *filter_name;/* Check if it is already registered */ -if (!promisor_remote_find (remote)) { +if ((promisor_remote =promisor_remote_find (remote))) { +if (promisor_remote->partial_clone_filter) +/* +* Remote is already registered and a filter is already +* set, so we don't need to do anything here. +*/ +return ; + }else {if (upgrade_repository_format (1 ) <0 )die (_ ("unable to upgrade repository format to support partial clone" ));
t/t5601-clone.sh +5 -0
@@ -672,6 +672,11 @@ test_expect_success 'partial clone with -o' ' git clone -o blah --filter=blob:limit=0 "file://$(pwd)/server" client ' +test_expect_success 'ensure that filter is written to config' ' + FILTER=$(git -C client config --get remote.blah.partialclonefilter) && + test "$FILTER" == "blob:limit=0" +' + test_expect_success 'partial clone: warn if server does not support object filtering' ' rm -rf server client && test_create_repo server &&
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
PATCH 1/2 promisor-remote: remove unused variable
In reply to Jonathan TanThe variable core_partial_clone_filter_default has been unused since fa3d1b63e8 ("promisor-remote: parse remote.*.partialclonefilter", 2019-06-25), when Git was changed to refer to remote.*.partialclonefilter as the default filter when fetching in a partial clone, but (perhaps inadvertently) there was no fallback to core.partialclonefilter.
One alternative is to add the fallback, but the aforementioned change was made more than a year ago and I have not heard of any complaints regarding this matter. In addition, there is currently no mention of core.partialclonefilter in the user documentation. So it seems best to reaffirm that Git will only support remote.*.partialclonefilter.
--- promisor-remote.c | 5 ----- promisor-remote.h | 2 +- 2 files changed, 1 insertion(+), 6 deletions(-)
promisor-remote.c +0 -5
@@ -6,7 +6,6 @@#include "strvec.h" static char *repository_format_partial_clone; -static const char *core_partial_clone_filter_default;void set_repository_format_partial_clone (char *partial_clone) {
@@ -100,10 +99,6 @@ static int promisor_remote_config(const char *var, const char *value, void *datasize_t namelen;const char *subkey; -if (!strcmp (var,"core.partialclonefilter" )) -return git_config_string (&core_partial_clone_filter_default, - var, value); -if (parse_config_key (var,"remote" , &name, &namelen, &subkey) <0 )return 0 ;
promisor-remote.h +1 -1
@@ -9,7 +9,7 @@ struct object_id;* Promisor remote linked list * * Information in its fields come from remote.XXX config entries or -* from extensions.partialclone or core.partialclonefilter. +* from extensions.partialclone. */ struct promisor_remote {struct promisor_remote *next;
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Quoted 4 lines · Jonathan Tan
This was noticed at $DAYJOB when in a partial clone with a filter other than blob:none. The 1st patch is a code cleanup (so that the 2nd patch does not need to take into account the unused variable), and the 2nd patch contains the bug fix.
What is "this" is not clear at all. This is one of those cover letters unfriendly to readers who do not know what the author did before being told X-<. We got a vague "we found some unspecified bug and fixed it, and we need to read them through if we want to find out what it is".
Quoted 11 lines
Jonathan Tan (2):
promisor-remote: remove unused variable
fetch: do not override partial clone filterbuiltin/fetch.c | 2 +- list-objects-filter-options.c | 10 +++++++++- promisor-remote.c | 5 ----- promisor-remote.h | 2 +- t/t5601-clone.sh | 5 +++++ 5 files changed, 16 insertions(+), 8 deletions(-)
Re: [PATCH 2/2] fetch: do not override partial clone filter
In reply to Jonathan Tant/t5601-clone.sh test_expect_success 'partial clone with -o' ' · line 677
Quoted 3 lines · Jonathan Tan
t/t5601-clone.sh +3
-0+test_expect_success 'ensure that filter is written to config' ' + FILTER=$(git -C client config --get remote.blah.partialclonefilter) && + test "$FILTER" == "blob:limit=0"
I'll do s/==/=/ locally because otherwise the test-lint-shell-syntax will choke on this line. No need to resend only this line.
Thanks.
Quoted 5 lines
+'
t/t5601-clone.sh +1
-0+ test_expect_success 'partial clone: warn if server does not support object filtering' ' rm -rf server client && test_create_repo server &&
Re: [PATCH 2/2] fetch: do not override partial clone filter
Quoted 8 lines · Junio C Hamano
t/t5601-clone.sh test_expect_success 'partial clone with -o' ' · line 677
Quoted 3 lines · Jonathan Tan
t/t5601-clone.sh +3
-0+test_expect_success 'ensure that filter is written to config' ' + FILTER=$(git -C client config --get remote.blah.partialclonefilter) && + test "$FILTER" == "blob:limit=0"I'll do s/==/=/ locally because otherwise the test-lint-shell-syntax will choke on this line. No need to resend only this line.
Wouldn't "test_cmp_config" be a better here? It seems like this is testing the "git clone" behavior of the previous test, so it could be added to that test instead.
Thanks,
-Stolee
Re: [PATCH 2/2] fetch: do not override partial clone filter
Quoted 10 lines · Derrick Stolee
Quoted 8 lines · Junio C Hamano
t/t5601-clone.sh test_expect_success 'partial clone with -o' ' · line 677
Quoted 3 lines · Jonathan Tan
t/t5601-clone.sh +3
-0+test_expect_success 'ensure that filter is written to config' ' + FILTER=$(git -C client config --get remote.blah.partialclonefilter) && + test "$FILTER" == "blob:limit=0"I'll do s/==/=/ locally because otherwise the test-lint-shell-syntax will choke on this line. No need to resend only this line.
Quoted 3 lines
Wouldn't "
test_cmp_config" be a better here? It seems like this is testing the "git clone" behavior of the previous test, so it could be added to that test instead.
TIL that one ;-) "git config --get" gives only a single instance, but in this particular case, shouldn't we be verifying not just that the expected one is found, but it is the only one, with "--get-all"?
FWIW, many uses of that test helper may want to be tightened the same way. In short, unless we are testing the last-one-wins behaviour, and especially when we are interested in what changes our tested operation makes to the config file's contents, we should be using "--get-all" instead of the default "--get", I suspect.
Thanks.
Re: [PATCH v2 2/2] fetch: do not override partial clone filter
In reply to Jonathan Tant/t5601-clone.sh test_expect_success 'partial clone' ' · line 674
Quoted 6 lines · Jonathan Tan
test_expect_success'partial clone with -o' 't/t5601-clone.sh +2
-1partial_clone_server server && - git clone -o blah --filter=blob:limit=0 "file://$(pwd)/server" client + git clone -o blah --filter=blob:limit=0 "file://$(pwd)/server" client && + test_cmp_config -C client "blob:limit=0" --get-all remote.blah.partialclonefilter '
Makes sense to piggyback on this existing test. It also makes sense to ensure that the original one is all and the only filter that is being configured.
Will queue. Let's merge them to 'next'.
PATCH v2 2/2 fetch: do not override partial clone filter
In reply to Jonathan TanWhen a fetch with the --filter argument is made, the configured default filter is set even if one already exists. This change was made in 5e46139376 ("builtin/fetch: remove unique promisor remote limitation", 2019-06-25) - in particular, changing from:
* If this is the FIRST partial-fetch request, we enable partial
* on this repo and remember the given filter-spec as the default
* for subsequent fetches to this remote.
to:
* If this is a partial-fetch request, we enable partial on
* this repo if not already enabled and remember the given
* filter-spec as the default for subsequent fetches to this
* remote.
(The given filter-spec is "remembered" even if there is already an existing one.)
This is problematic whenever a lazy fetch is made, because lazy fetches are made using "git fetch --filter=blob:none", but this will also happen if the user invokes "git fetch --filter=<filter>" manually. Therefore, restore the behavior prior to 5e46139376, which writes a filter-spec only if the current fetch request is the first partial-fetch one (for that remote).
--- builtin/fetch.c | 2 +- list-objects-filter-options.c | 10 +++++++++- t/t5601-clone.sh | 3 ++- 3 files changed, 12 insertions(+), 3 deletions(-)
builtin/fetch.c +1 -1
@@ -1677,7 +1677,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)* If this is a partial-fetch request, we enable partial on * this repo if not already enabled and remember the given * filter-spec as the default for subsequent fetches to this -* remote. +* remote if there is currently no default filter-spec. */ if (filter_options.choice) {partial_clone_register (remote->name, &filter_options);
list-objects-filter-options.c +9 -1
@@ -344,11 +344,19 @@ void partial_clone_register(const char *remote, struct list_objects_filter_options *filter_options) { + struct promisor_remote *promisor_remote;char *cfg_name;char *filter_name;/* Check if it is already registered */ -if (!promisor_remote_find (remote)) { +if ((promisor_remote =promisor_remote_find (remote))) { +if (promisor_remote->partial_clone_filter) +/* +* Remote is already registered and a filter is already +* set, so we don't need to do anything here. +*/ +return ; + }else {if (upgrade_repository_format (1 ) <0 )die (_ ("unable to upgrade repository format to support partial clone" ));
t/t5601-clone.sh +2 -1
@@ -669,7 +669,8 @@ test_expect_success 'partial clone' ' test_expect_success 'partial clone with -o' ' partial_clone_server server && - git clone -o blah --filter=blob:limit=0 "file://$(pwd)/server" client + git clone -o blah --filter=blob:limit=0 "file://$(pwd)/server" client && + test_cmp_config -C client "blob:limit=0" --get-all remote.blah.partialclonefilter ' test_expect_success 'partial clone: warn if server does not support object filtering' '
Signature · 1 line
2.28.0.709.gb0816b6eb0-goog
PATCH v2 1/2 promisor-remote: remove unused variable
In reply to Jonathan TanThe variable core_partial_clone_filter_default has been unused since fa3d1b63e8 ("promisor-remote: parse remote.*.partialclonefilter", 2019-06-25), when Git was changed to refer to remote.*.partialclonefilter as the default filter when fetching in a partial clone, but (perhaps inadvertently) there was no fallback to core.partialclonefilter.
One alternative is to add the fallback, but the aforementioned change was made more than a year ago and I have not heard of any complaints regarding this matter. In addition, there is currently no mention of core.partialclonefilter in the user documentation. So it seems best to reaffirm that Git will only support remote.*.partialclonefilter.
--- promisor-remote.c | 5 ----- promisor-remote.h | 2 +- 2 files changed, 1 insertion(+), 6 deletions(-)
promisor-remote.c +0 -5
@@ -6,7 +6,6 @@#include "strvec.h" static char *repository_format_partial_clone; -static const char *core_partial_clone_filter_default;void set_repository_format_partial_clone (char *partial_clone) {
@@ -100,10 +99,6 @@ static int promisor_remote_config(const char *var, const char *value, void *datasize_t namelen;const char *subkey; -if (!strcmp (var,"core.partialclonefilter" )) -return git_config_string (&core_partial_clone_filter_default, - var, value); -if (parse_config_key (var,"remote" , &name, &namelen, &subkey) <0 )return 0 ;
promisor-remote.h +1 -1
@@ -9,7 +9,7 @@ struct object_id;* Promisor remote linked list * * Information in its fields come from remote.XXX config entries or -* from extensions.partialclone or core.partialclonefilter. +* from extensions.partialclone. */ struct promisor_remote {struct promisor_remote *next;
Signature · 1 line
2.28.0.709.gb0816b6eb0-goog
v2 cover Fix overriding of partial clone filter when lazy-fetching
In reply to Jonathan TanThanks everyone for your comments. Patch 1 is exactly the same, and in patch 2, I've changed the config check to use test_cmp_config with "--get-all", as requested.
Comment noted about saying "this" without informing the reader what it's referring to - in this case, it was about the issue in patch 2.
Jonathan Tan (2):
promisor-remote: remove unused variable
fetch: do not override partial clone filter
builtin/fetch.c | 2 +- list-objects-filter-options.c | 10 +++++++++- promisor-remote.c | 5 ----- promisor-remote.h | 2 +- t/t5601-clone.sh | 3 ++- 5 files changed, 13 insertions(+), 9 deletions(-)
Range-diff against v1:
-: ---------- > 1: 625e7f148e promisor-remote: remove unused variable
1: af960f33b6 ! 2: c3ced59875 fetch: do not override partial clone filter
@@ list-objects-filter-options.c: void partial_clone_register(
## t/t5601-clone.sh ##
-@@ t/t5601-clone.sh: test_expect_success 'partial clone with -o' '
- git clone -o blah --filter=blob:limit=0 "file://$(pwd)/server" client
+@@ t/t5601-clone.sh: test_expect_success 'partial clone' '
+
+ test_expect_success 'partial clone with -o' '
+ partial_clone_server server &&
+- git clone -o blah --filter=blob:limit=0 "file://$(pwd)/server" client
++ git clone -o blah --filter=blob:limit=0 "file://$(pwd)/server" client &&
++ test_cmp_config -C client "blob:limit=0" --get-all remote.blah.partialclonefilter
'
-+test_expect_success 'ensure that filter is written to config' '
-+ FILTER=$(git -C client config --get remote.blah.partialclonefilter) &&
-+ test "$FILTER" = "blob:limit=0"
-+'
-+
test_expect_success 'partial clone: warn if server does not support object filtering' '
- rm -rf server client &&
- test_create_repo server &&Signature · 1 line
2.28.0.709.gb0816b6eb0-goog