removed fetch_if_missing global
cover removed fetch_if_missing global
We are not much happy with global variable fetch_if_missing. So, in commit 6462d5eb9a ("fetch: remove fetch_if_missing=0", 2019-11-08) Jonathan Tan [email protected] [[email protected]] attempted to remove the need for fetch_if_missing=0 from the fetching mechanism. After that, fetch_if_missing is removed from clone and promisor-remote too.
I imitated the same logic to remove fetch_if_missing from fetch-pack & index-pack.
I'm looking forward to remove fetch_if_missing from other places too, but I not sure about how to handle it.
In fsck, fetch_if_missing is set to 0 in the beginning of cmd_fsck().
In rev-list, fetch_if_missing is set to 0 in parse_missing_action_value(), and in cmd_rev_list() while parsing the command-line parameters.(almost similar case in pack-objects)
fixes #251
Hariom Verma (2):
fetch-pack: remove fetch_if_missing=0
index-pack: remove fetch_if_missing=0
builtin/fetch-pack.c | 2 -- builtin/index-pack.c | 11 ++--------- fetch-pack.c | 2 +- 3 files changed, 3 insertions(+), 12 deletions(-)
base-commit: be8661a3286c67a5d4088f4226cbd7f8b76544b0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-606%2Fharry-hov%2Ffetch-if-missing-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-606/harry-hov/fetch-if-missing-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/606
Signature · 1 line
gitgitgadget
PATCH 1/2 fetch-pack: remove fetch_if_missing=0
From: Hariom Verma <[email protected]>
Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from fetch-pack as well.
--- builtin/fetch-pack.c | 2 -- fetch-pack.c | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-)
builtin/fetch-pack.c +0 -2
@@ -57,8 +57,6 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)struct packet_reader reader;enum protocol_version version; - fetch_if_missing =0 ; -packet_trace_identity ("fetch-pack" );memset (&args,0 ,sizeof (args));
fetch-pack.c +1 -1
@@ -1649,7 +1649,7 @@ static void update_shallow(struct fetch_pack_args *args,struct oid_array extra =OID_ARRAY_INIT ;struct object_id *oid = si->shallow->oid;for (i =0 ; i < si->shallow->nr; i++) -if (has_object_file (&oid[i])) +if (has_object_file_with_flags (&oid[i],OBJECT_INFO_SKIP_FETCH_OBJECT ))oid_array_append (&extra, &oid[i]);if (extra.nr) { setup_alternate_shallow(&shallow_lock,
Signature · 1 line
gitgitgadget
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
Quoted 7 lines · Hariom Verma via GitGitGadget
From: Hariom Verma <[email protected]>
Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from fetch-pack as well.
It's ok to refer to a previous commit, but I think it would be better if you could repeat a bit the reasons why removing the fetch_if_missing global is a good idea, and not just rely on the previous commit.
"it is plausible" also doesn't make it very clear that it's what the patch is actually doing.
Thanks,
Christian.
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
Quoted 10 lines · Christian Couder
Quoted 7 lines · Hariom Verma via GitGitGadget
From: Hariom Verma <[email protected]>
Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from fetch-pack as well.
Quoted 7 lines
It's ok to refer to a previous commit, but I think it would be better if you could repeat a bit the reasons why removing the
fetch_if_missingglobal is a good idea, and not just rely on the previous commit."it is plausible" also doesn't make it very clear that it's what the patch is actually doing.
I had the same reaction. You could even write a random gibberish in your patch and write "it's plausible this set of random changes made without understanding what is going on in the current code might have some chance to work" in your log message, and we would not even want to touch such a patch with 10-foot pole.
The proposed log message above unfortunately makes this patch indistinguishable from such a trash, unless we follow the codepaths that are *not* touched by this patch and think about ramifications of the removal *ourselves*. In other words, it does nothing to help the readers to support the change.
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
Quoted 17 lines · Junio C Hamano
Quoted 7 lines · Christian Couder
It's ok to refer to a previous commit, but I think it would be better if you could repeat a bit the reasons why removing the
fetch_if_missingglobal is a good idea, and not just rely on the previous commit."it is plausible" also doesn't make it very clear that it's what the patch is actually doing.
I had the same reaction. You could even write a random gibberish in your patch and write "it's plausible this set of random changes made without understanding what is going on in the current code might have some chance to work" in your log message, and we would not even want to touch such a patch with 10-foot pole.
Quoted 6 lines
The proposed log message above unfortunately makes this patch indistinguishable from such a trash, unless we follow the codepaths that are *not* touched by this patch and think about ramifications of the removal *ourselves*. In other words, it does nothing to help the readers to support the change.
I understand it must be too hard for you to deal with such [trash]patches. My apologies. Will improve in next revision
Thanks,
Hariom
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
In reply to Christian CouderQuoted 5 lines · Christian Couder
It's ok to refer to a previous commit, but I think it would be better if you could repeat a bit the reasons why removing the
fetch_if_missingglobal is a good idea, and not just rely on the previous commit.
I agree with that.
Quoted 2 lines
"it is plausible" also doesn't make it very clear that it's what the patch is actually doing.
Thanks for pointing it out. Will improve.
Regards,
Hariom
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
In reply to Hariom Verma via GitGitGadgetQuoted 8 lines
From: Hariom Verma <[email protected]>
Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from fetch-pack as well.
Signed-off-by Hariom Verma <[email protected]>
As Christian said [1], please include tests like in the commit you mentioned. For a change like this, I think that the test is the most important part.
Also include a justification for why it's safe to remove fetch_if_missing=0. You can probably cite the aforementioned commit to say that it covers the fetch_pack() method, and then go through the rest of the code to see if any may inadvertently fetch an object.
Also, the fetch-pack and index-pack parts can be sent in separate patch sets, so you might want to concentrate on one command first.
[1] https://lore.kernel.org/git/CAP8UFD2SNnpKWtYUztZ76OU7zBsrXyYhG_Zds1wi+NqBKCv+Qw@mail.gmail.com/
fetch-pack.c static void update_shallow(struct fetch_pack_args *args, · line 1655
Quoted 13 lines
fetch-pack.c +1
-1@@ -1649,7 +1649,7 @@ static void update_shallow(struct fetch_pack_args *args,struct oid_array extra =OID_ARRAY_INIT ;struct object_id *oid = si->shallow->oid;for (i =0 ; i < si->shallow->nr; i++) -if (has_object_file (&oid[i])) +if (has_object_file_with_flags (&oid[i],OBJECT_INFO_SKIP_FETCH_OBJECT ))oid_array_append (&extra, &oid[i]);if (extra.nr) { setup_alternate_shallow(&shallow_lock,
Hmm...this triggers when the user requests a clone that is both partial and shallow, and the server reports a shallow object that it didn't send back as a packfile; and it causes another fetch to be sent. This is a separate issue, but Hariom, if you'd like to take a look at this, that would work out too. You'll need to figure out how to make the server send back shallow lines referencing objects that are not in the packfile
- one way to do it is to use one-time-perl. (Search the codebase to see how it is used.) This is probably more complex, though.
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
Quoted 5 lines · Jonathan Tan
As Christian said [1], please include tests like in the commit you mentioned. For a change like this, I think that the test is the most important part.
I will definitely add tests.
Quoted 8 lines
Also include a justification for why it's safe to remove fetch_if_missing=0. You can probably cite the aforementioned commit to say that it covers the
fetch_pack()method, and then go through the rest of the code to see if any may inadvertently fetch an object.Also, the fetch-pack and index-pack parts can be sent in separate patch sets, so you might want to concentrate on one command first.
Thanks, Will split and concentrate on one at a time.
Quoted 15 lines
Quoted 6 lines
fetch-pack.c +0
-0fetch-pack.c static void update_shallow(struct fetch_pack_args *args, · line 1655
Quoted 8 lines
fetch-pack.c +1
-1@@ -1649,7 +1649,7 @@ static void update_shallow(struct fetch_pack_args *args,struct oid_array extra =OID_ARRAY_INIT ;struct object_id *oid = si->shallow->oid;for (i =0 ; i < si->shallow->nr; i++) -if (has_object_file (&oid[i])) +if (has_object_file_with_flags (&oid[i],OBJECT_INFO_SKIP_FETCH_OBJECT ))oid_array_append (&extra, &oid[i]);if (extra.nr) { setup_alternate_shallow(&shallow_lock,
Quoted 8 lines
Hmm...this triggers when the user requests a clone that is both partial and shallow, and the server reports a shallow object that it didn't send back as a packfile; and it causes another fetch to be sent. This is a separate issue, but Hariom, if you'd like to take a look at this, that would work out too. You'll need to figure out how to make the server send back shallow lines referencing objects that are not in the packfile
- one way to do it is to use one-time-perl. (Search the codebase to see how it is used.) This is probably more complex, though.
I'm clueless about "one-time-perl" thing(till now!). Will surely going to learn about that.
Thanks for the nice explanation.
Regards,
Hariom
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
In reply to Jonathan TanSo here, we are partial-cloning from a shallow remote and some objects are not sent due to our clone filters. Let's say that the shallow remote has a 5-commit history and we are cloning it into another repository with a blob:none filter. The expected behavior is cloning the 5 commits, with no blobs, except for the HEAD.
When executing the above process, it leads to errors:
fatal: the remote end hung up unexpectedly
fatal: protocol error: bad pack header
warning: Clone succeeded, but checkout failed You can inspect what was checked out with 'git status' and retry with 'git restore --source=HEAD :/'
I looked into it a bit and it seems that packet_read() is not successful. I'm not really sure how packet reading fits into the big picture but it looks like the buffer is not read completely.
It is a similar case with "bad pack header" too. The function read_pack_header() fails because the pack header was not fully read.
Also, is the shallow object not sent when cloning due to the partial clone filter and hence a subsequent fetching is done to ask for this object? If so, then will such a fetch counted as an args->update_shallow?
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
Quoted 3 lines · Kousik Sanagavarapu
Also, is the shallow object not sent when cloning due to the partial clone filter and hence a subsequent fetching is done to ask for this object? If so, then will such a fetch counted as an
args->update_shallow?
What do you mean by the shallow object? If you mean the last commit that is sent (that is, without its parents), then that is a commit and is not excluded by the filter.
As for args->update_shallow, that's a good question. Just glancing at the code, even if it is set, I don't think there would be any difference in operation since the lazy fetch does not fetch any refs (and in fact, in protocol v2, we skip the ref advertisement in this case, as far as I can remember).
Re: [PATCH 1/2] fetch-pack: remove fetch_if_missing=0
In reply to Jonathan TanSorry, it seems that I misunderstood the whole situation here. I didn't realize that the problem was on my end and not something to do with the code. Please ignore the above email except for the last question, which I still don't seem to understand. So, I would be grateful if you could clarify.
Thanks,
Kousik
PATCH 2/2 index-pack: remove fetch_if_missing=0
In reply to Hariom Verma via GitGitGadgetFrom: Hariom Verma <[email protected]>
Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from index-pack as well.
--- builtin/index-pack.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-)
builtin/index-pack.c +2 -9
@@ -782,7 +782,8 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,if (startup_info->have_repository) {read_lock (); collision_test_needed = -has_object_file_with_flags (oid,OBJECT_INFO_QUICK ); +has_object_file_with_flags (oid,OBJECT_INFO_QUICK | +OBJECT_INFO_SKIP_FETCH_OBJECT );read_unlock (); }
@@ -1673,14 +1674,6 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)unsigned foreign_nr =1 ;/* zero is a "good" value, assume bad */ int report_end_of_input =0 ; -/* -* index-pack never needs to fetch missing objects except when -* REF_DELTA bases are missing (which are explicitly handled). It only -* accesses the repo to do hash collision checks and to check which -* REF_DELTA bases need to be fetched. -*/ - fetch_if_missing =0 ; -if (argc ==2 && !strcmp (argv[1 ],"-h" ))usage (index_pack_usage);
Signature · 1 line
gitgitgadget
Quoted 6 lines · Hariom Verma via GitGitGadget
We are not much happy with global variable
fetch_if_missing. So, in commit6462d5eb9a("fetch: remove fetch_if_missing=0", 2019-11-08) Jonathan Tan [email protected] [[email protected]] attempted to remove the need for fetch_if_missing=0 from the fetching mechanism. After that,fetch_if_missingis removed from clone and promisor-remote too.
You might want to add Jonathan in Cc next time, as it could help your patches move forward. I have added him to this email.
Quoted 2 lines
I imitated the same logic to remove
fetch_if_missingfrom fetch-pack & index-pack.
Maybe you could add a few tests as in 6462d5eb9a.
Quoted 2 lines
I'm looking forward to remove
fetch_if_missingfrom other places too, but I not sure about how to handle it.
It is ok to not take care of the other places for now. If that was the only reason why this patch series is marked as WIP, then you might want to remove WIP, especially if you add tests.
Quoted 7 lines
In fsck,
fetch_if_missingis set to 0 in the beginning ofcmd_fsck().In rev-list,
fetch_if_missingis set to 0 inparse_missing_action_value(), and incmd_rev_list()while parsing the command-line parameters.(almost similar case in pack-objects)fixes #251
It would be nice if you could give the full URL of the bug, as there have been different bug trackers used by different people.
Thanks,
Christian.
Quoted 3 lines · Christian Couder
You might want to add Jonathan in Cc next time, as it could help your patches move forward. I have added him to this email.
Thanks, I'll remember next time.
Quoted 1 line
Maybe you could add a few tests as in
6462d5eb9a.
Sounds like a plan.
Quoted 3 lines
It is ok to not take care of the other places for now. If that was the only reason why this patch series is marked as WIP, then you might want to remove WIP, especially if you add tests.
I'll remove it, after writing tests.
Quoted 2 lines
It would be nice if you could give the full URL of the bug, as there have been different bug trackers used by different people.
I'll do this in future versions.
Thanks,
Hariom
Quoted 38 lines · Christian Couder
Quoted 6 lines · Hariom Verma via GitGitGadget
We are not much happy with global variable
fetch_if_missing. So, in commit6462d5eb9a("fetch: remove fetch_if_missing=0", 2019-11-08) Jonathan Tan [email protected] [[email protected]] attempted to remove the need for fetch_if_missing=0 from the fetching mechanism. After that,fetch_if_missingis removed from clone and promisor-remote too.You might want to add Jonathan in Cc next time, as it could help your patches move forward. I have added him to this email.
Quoted 2 lines
I imitated the same logic to remove
fetch_if_missingfrom fetch-pack & index-pack.Maybe you could add a few tests as in
6462d5eb9a.Quoted 2 lines
I'm looking forward to remove
fetch_if_missingfrom other places too, but I not sure about how to handle it.It is ok to not take care of the other places for now. If that was the only reason why this patch series is marked as WIP, then you might want to remove WIP, especially if you add tests.
Quoted 7 lines
In fsck,
fetch_if_missingis set to 0 in the beginning ofcmd_fsck().In rev-list,
fetch_if_missingis set to 0 inparse_missing_action_value(), and incmd_rev_list()while parsing the command-line parameters.(almost similar case in pack-objects)fixes #251
It would be nice if you could give the full URL of the bug, as there have been different bug trackers used by different people.
Thanks,
Christian.
Hi,
Are you still working on this? If not, then I would like to take this up and write the tests, if it is worth doing. I think it would be a better exposure of the codebase and would be helpful for GSoC.
Hi Kousik,
Quoted 3 lines · Kousik Sanagavarapu
Are you still working on this? If not, then I would like to take this up and write the tests, if it is worth doing. I think it would be a better exposure of the codebase and would be helpful for GSoC.
I don't know what's the state of this. I think only Hariom could answer.
I am not so sure it will be helpful for any of the GSoC project ideas we propose, but feel free to work on it if you want.
Thanks,
Christian.
Quoted 3 lines · Christian Couder
I am not so sure it will be helpful for any of the GSoC project ideas we propose, but feel free to work on it if you want.
Well, I wanted to work on something before I started working on my application and found this to be fun. So even if it would not really be helpful for the project ideas proposed, I would still like to work on it as something that could go into my application.
Thanks,
Kousik
Quoted 11 lines · Kousik Sanagavarapu
Quoted 2 lines · Christian Couder
I am not so sure it will be helpful for any of the GSoC project ideas we propose, but feel free to work on it if you want.
Well, I wanted to work on something before I started working on my application and found this to be fun. So even if it would not really be helpful for the project ideas proposed, I would still like to work on it as something that could go into my application.
Sure, you can mention anything you did related to the project in your application and we will take it into account.
Thanks,
Christian.
HI,
Quoted 4 lines · Kousik Sanagavarapu
Are you still working on this? If not, then I would like to take this up and write the tests, if it is worth doing. I think it would be a better exposure of the codebase and would be helpful for GSoC.
I'm not working on it. Feel free to take it forward.
Thanks,
Hariom