midx: disable replace objects
PATCH 1/1 midx: disable replace objects
From: Xing Xin <[email protected]>
We observed a series of clone failures arose in a specific set of repositories after we fully enabled the MIDX bitmap feature within our Codebase service. These failures were accompanied with error messages such as:
fatal: did not receive expected object ...
fatal: fetch-pack: invalid index-pack output
Temporarily disabling the MIDX feature eliminated the reported issues. After some investigation we found that all repositories experiencing failures contain replace references, which seem to be improperly acknowledged by the MIDX bitmap generation logic. During cloning or fetching, git-pack-objects, which may make use of MIDX bitmap to find objects to pack, would give wrong objects even if we explicitly specified not to enable replace refs by GIT_NO_REPLACE_OBJECTS=1. Indeed, this issue appears to have persisted since the introduction of MIDX.
This patch updates the MIDX logic to disable replace objects during operations, mirroring the handling seen in single pack index scenarios, i.e. git-index-pack and git-pack-objects. The added test uses git-rev-list to give a more intuitive check.
---
midx: disable replace objects
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1711%2Fblanet%2Fxx%2Fmidx-ignore-replace-objects-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1711/blanet/xx/midx-ignore-replace-objects-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1711
builtin/multi-pack-index.c | 3 +++ t/t5326-multi-pack-bitmaps.sh | 24 ++++++++++++++++++++++++ 2 files changed, 27 insertions(+)
builtin/multi-pack-index.c +3 -0
@@ -8,6 +8,7 @@#include "strbuf.h" #include "trace2.h" #include "object-store-ll.h" +#include "replace-object.h" #define BUILTIN_MIDX_WRITE_USAGE \ N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]" \
@@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv, };struct option *options =parse_options_concat (builtin_multi_pack_index_options, common_opts); +disable_replace_refs (); +git_config (git_default_config,NULL ); if (the_repository &&
t/t5326-multi-pack-bitmaps.sh +24 -0
@@ -434,6 +434,30 @@ test_expect_success 'tagged commits are selected for bitmapping' ' ) ' +test_expect_success 'do not follow replace objects for MIDX bitmap' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + + test_commit A && + A=$(git rev-parse HEAD) && + test_commit B && + B=$(git rev-parse HEAD) && + git checkout --orphan=orphan $A && + test_commit orphan && + C=$(git rev-parse HEAD) && + git rev-list --objects --no-object-names $B |sort >expected && + + git replace $A $C && + git repack -ad && + git multi-pack-index write --bitmap && + git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual && + test_cmp expected actual + ) +' + corrupt_file () { chmod a+w "$1" && printf "bogus" | dd of="$1" bs=1 seek="12" conv=notrunc
base-commit: 3c2a3fdc388747b9eaf4a4a4f2035c1c9ddb26d0
Signature · 1 line
gitgitgadget
Quoted 7 lines · blanet via GitGitGadget
From: Xing Xin <[email protected]>
We observed a series of clone failures arose in a specific set of repositories after we fully enabled the MIDX bitmap feature within our Codebase service. These failures were accompanied with error messages such as:
Quoted 7 lines
fatal: did not receive expected object ...
fatal: fetch-pack: invalid index-pack outputTemporarily disabling the MIDX feature eliminated the reported issues. After some investigation we found that all repositories experiencing failures contain replace references, which seem to be improperly acknowledged by the MIDX bitmap generation logic.
I was suspicious that this might be related to the MIDX or MIDX bitmap, but noticed something curious upon digging in. Applying the following on top of your patch:
--- 8< ---
t/t5326-multi-pack-bitmaps.sh +1 -3
@@ -451,9 +451,7 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' ' git rev-list --objects --no-object-names $B |sort >expected && git replace $A $C && - git repack -ad && - git multi-pack-index write --bitmap && - git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual && + git rev-list --objects --no-object-names $B |sort >actual && test_cmp expected actual ) '
--- >8 ---
, I can still produce the failure that you are seeing here. So I suspect that while it's entirely possible that there is a bug in the MIDX/bitmap code, that this test is not exercising it.
I think the first step to demonstrate a bug in the MIDX/bitmap machinery would be to provide a reproducer that fails only when using a MIDX and/or bitmap.
Review int cmd_multi_pack_index(int argc, const char **argv, · line 278
Quoted 6 lines
(quoted diff) +2
-0@@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv, }; struct option *options = parse_options_concat(builtin_multi_pack_index_options, common_opts); + disable_replace_refs(); +
Supposing for a moment that this issue is in the MIDX, we know that regardless of what replace refs might be in place, the MIDX should only be storing the objects that are in the packs being indexed, not the objects which are their replacements.
Are we storing objects in the MIDX that are replacements? Looking at midx.c::fill_pack_entry(), I think the answer is "no", since we're looking up packed objects by calling nth_packed_object_id(), which is just a table read into the .idx, all of which is beneath the level of replace refs.
Review test_expect_success 'tagged commits are selected for bitmapping' ' · line 445
Quoted 13 lines
(quoted diff) +9
-0@@ -434,6 +434,30 @@ test_expect_success 'tagged commits are selected for bitmapping' ' ) ' +test_expect_success 'do not follow replace objects for MIDX bitmap' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + + test_commit A && + A=$(git rev-parse HEAD) &&
It's possible that much of this will be moot if the current test gets rewritten, but here are a couple of suggestions for writing tests in Git's suite:
- test_commit will create a tag for you, so there is no need to store "$A", "$B", and "$C".
Review test_expect_success 'tagged commits are selected for bitmapping' ' · line 451
Quoted 6 lines
(quoted diff) +6
-0+ test_commit B && + B=$(git rev-parse HEAD) && + git checkout --orphan=orphan $A && + test_commit orphan && + C=$(git rev-parse HEAD) && + git rev-list --objects --no-object-names $B |sort >expected &&
- We do not allow Git invocations on the left-hand side of a pipe, since doing so will squelch its exit code. Instead, try:
git rev-list --objects --no-object-names B >expect.raw &&
sort expect.raw >expect &&Thanks,
Taylor
Quoted 7 lines · Taylor Blau
, I can still produce the failure that you are seeing here. So I suspect that while it's entirely possible that there is a bug in the MIDX/bitmap code, that this test is not exercising it.
I think the first step to demonstrate a bug in the MIDX/bitmap machinery would be to provide a reproducer that fails only when using a MIDX and/or bitmap.
I had a some more time to look into this, and I think that your original fix is correct.
The issue is, as you suggest, due to the following (from your original patch):
Quoted 3 lines
After some investigation we found that all repositories experiencing failures contain replace references, which seem to be improperly acknowledged by the MIDX bitmap generation logic.
Indeed, the pack-bitmap-write machinery does not itself call disable_replace_refs(). So when it generates a reachability bitmap, it is doing so with the replace refs in mind. You can see that this is indeed the cause of the problem by looking at the output of an instrumented version of Git that indicates what bits are being set during the bitmap generation phase.
With replace refs (incorrectly) enabled, we get:
[2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]
and doing the same after calling disable_replace_refs(), we instead get:
[2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]
Single pack bitmaps are unaffected by this issue because we generate them from within pack-objects, which does call disable_replace_refs().
It is tempting to instead do something like:
--- 8< ---
pack-bitmap-write.c +3 -0
@@ -17,6 +17,7 @@#include "trace2.h" #include "tree.h" #include "tree-walk.h" +#include "replace-object.h" struct bitmapped_commit {struct commit *commit;
@@ -223,6 +224,8 @@ static void bitmap_builder_init(struct bitmap_builder *bb,memset (bb,0 ,sizeof (*bb));init_bb_data (&bb->data); +parsed_object_pool_clear (the_repository->parsed_objects); +reset_revision_walk ();repo_init_revisions (writer->to_pack->repo, &revs,NULL ); revs.topo_order =1 ;
--- >8 ---
But by then it is too late, because the replace refs have already been taken into account for parsed objects.
An alternative is to clear the parsed_object_pool before (or after) calling disable_replace_refs(), but I think that approach that feels sub-optimal for a couple of reasons:
- We're wasting time re-parsing objects that we've already seen
- We're banking on the fact that the MIDX generation does not lookup objects with the OBJECT_INFO_LOOKUP_REPLACE flag set, which would cause the MIDX to be broken in the same way.
So I think that disabling replace refs at the outset within the multi-pack-index builtin is the right way to go. In addition to the test fixes I suggested earlier, I would instead demonstrate the bug by showing a clone (which fails with MIDXs, but doesn't without MIDXs) like so:
--- 8< ---
t/t5326-multi-pack-bitmaps.sh +7 -11
@@ -442,19 +442,16 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' ' cd repo && test_commit A && - A=$(git rev-parse HEAD) && test_commit B && - B=$(git rev-parse HEAD) && - git checkout --orphan=orphan $A && + git checkout --orphan=orphan A && test_commit orphan && - C=$(git rev-parse HEAD) && - git rev-list --objects --no-object-names $B |sort >expected && - git replace $A $C && - git repack -ad && - git multi-pack-index write --bitmap && - git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual && - test_cmp expected actual + git replace A HEAD && + git repack -ad --write-midx --write-bitmap-index && + + # generating reachability bitmaps with replace refs + # enabled will result in broken clones + git clone --no-local --bare . clone.git ) ' --- >8 ---
With the change in your patch to call disable_replace_refs() in builtin/multi-pack-index.c, this test passes as expected. With that change compiled out, we instead get:
[...]
+ git clone --no-local --bare . clone.git
Cloning into bare repository 'clone.git'...
remote: Enumerating objects: 8, done.
remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1) Receiving objects: 100% (8/8), done.
fatal: did not receive expected object da5497437fd67ca928333aab79c4b4b55036ea66
fatal: fetch-pack: invalid index-pack output
error: last command exited with $?=128
not ok 352 - do not follow replace objects for MIDX bitmap
as expected.
Thanks,
Taylor
Quoted 16 lines · Taylor Blau
It is tempting to instead do something like:
--- 8< ---
pack-bitmap-write.c +1
-0@@ -17,6 +17,7 @@#include "trace2.h" #include "tree.h" #include "tree-walk.h" +#include "replace-object.h" struct bitmapped_commit {struct commit *commit;
pack-bitmap-write.c static void bitmap_builder_init(struct bitmap_builder *bb,
Quoted 9 lines
pack-bitmap-write.c +2
-1@@ -223,6 +224,8 @@ static void bitmap_builder_init(struct bitmap_builder *bb,memset (bb,0 ,sizeof (*bb));init_bb_data (&bb->data); +parsed_object_pool_clear (the_repository->parsed_objects); +reset_revision_walk ();repo_init_revisions (writer->to_pack->repo, &revs,NULL ); revs.topo_order =1 ; --- >8 ---
Oops. I meant to attach a diff here that calls disable_replace_refs(), not parsed_object_pool_clear().
Quoted 2 lines
But by then it is too late, because the replace refs have already been taken into account for parsed objects.
Thanks,
Taylor
Quoted 19 lines
I had a some more time to look into this, and I think that your original fix is correct.
The issue is, as you suggest, due to the following (from your original patch):
Quoted 3 lines
After some investigation we found that all repositories experiencing failures contain replace references, which seem to be improperly acknowledged by the MIDX bitmap generation logic.
Indeed, the pack-bitmap-write machinery does not itself call
disable_replace_refs(). So when it generates a reachability bitmap, it is doing so with the replace refs in mind. You can see that this is indeed the cause of the problem by looking at the output of an instrumented version of Git that indicates what bits are being set during the bitmap generation phase.With replace refs (incorrectly) enabled, we get:
Quoted 8 lines
[2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]and doing the same after calling
disable_replace_refs(), we instead get:[2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]Single pack bitmaps are unaffected by this issue because we generate them from within pack-objects, which does call
disable_replace_refs().
Thank you for the comprehensive investigation. I have quoted them in the commit message to provide a clearer explanation of the patch.
Quoted 52 lines
In addition to the test fixes I suggested earlier, I would instead demonstrate the bug by showing a clone (which fails with MIDXs, but doesn't without MIDXs) like so:
--- 8< ---
t/t5326-multi-pack-bitmaps.sh +7
-11@@ -442,19 +442,16 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' ' cd repo && test_commit A && - A=$(git rev-parse HEAD) && test_commit B && - B=$(git rev-parse HEAD) && - git checkout --orphan=orphan $A && + git checkout --orphan=orphan A && test_commit orphan && - C=$(git rev-parse HEAD) && - git rev-list --objects --no-object-names $B |sort >expected && - git replace $A $C && - git repack -ad && - git multi-pack-index write --bitmap && - git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual && - test_cmp expected actual + git replace A HEAD && + git repack -ad --write-midx --write-bitmap-index && + + # generating reachability bitmaps with replace refs + # enabled will result in broken clones + git clone --no-local --bare . clone.git ) ' --- >8 ---With the change in your patch to call
disable_replace_refs()inbuiltin/multi-pack-index.c, this test passes as expected. With that change compiled out, we instead get:[...]
+ git clone --no-local --bare . clone.git
Cloning into bare repository 'clone.git'...
remote: Enumerating objects: 8, done.
remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1) Receiving objects: 100% (8/8), done.
fatal: did not receive expected objectda5497437fd67ca928333aab79c4b4b55036ea66
fatal: fetch-pack: invalid index-pack output
error: last command exited with $?=128
not ok 352 - do not follow replace objects for MIDX bitmap
Quoted 2 lines
as expected.
Applied! The test looks much clearer now, thanks!
Xing Xin
PATCH v2 1/1 midx: disable replace objects
In reply to blanet via GitGitGadgetFrom: Xing Xin <[email protected]>
We observed a series of clone failures arose in a specific set of repositories after we fully enabled the MIDX bitmap feature within our Codebase service. These failures were accompanied with error messages such as:
Cloning into bare repository 'clone.git'...
remote: Enumerating objects: 8, done.
remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1) Receiving objects: 100% (8/8), done.
fatal: did not receive expected object ...
fatal: fetch-pack: invalid index-pack output
Temporarily disabling the MIDX feature eliminated the reported issues. After some investigation we found that all repositories experiencing failures contain replace references, which seem to be improperly acknowledged by the MIDX bitmap generation logic.
A more thorough explanation about the root cause from Taylor Blau says:
Indeed, the pack-bitmap-write machinery does not itself call disable_replace_refs(). So when it generates a reachability bitmap, it is doing so with the replace refs in mind. You can see that this is indeed the cause of the problem by looking at the output of an instrumented version of Git that indicates what bits are being set during the bitmap generation phase.
With replace refs (incorrectly) enabled, we get:
[2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]
and doing the same after calling disable_replace_refs(), we instead get:
[2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]
Single pack bitmaps are unaffected by this issue because we generate them from within pack-objects, which does call disable_replace_refs().
This patch updates the MIDX logic to disable replace objects within the multi-pack-index builtin, and a test showing a clone (which would fail with MIDX bitmap) is added to demonstrate the bug.
---
midx: disable replace objects
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1711%2Fblanet%2Fxx%2Fmidx-ignore-replace-objects-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1711/blanet/xx/midx-ignore-replace-objects-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1711
Range-diff vs v1:
1: b1c838965ab ! 1: 1be25b55c5a midx: disable replace objects
@@ Commit message
Codebase service. These failures were accompanied with error messages such as:
- fatal: did not receive expected object ...
- fatal: fetch-pack: invalid index-pack output
+ Cloning into bare repository 'clone.git'...
+ remote: Enumerating objects: 8, done.
+ remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)
+ Receiving objects: 100% (8/8), done.
+ fatal: did not receive expected object ...
+ fatal: fetch-pack: invalid index-pack output
Temporarily disabling the MIDX feature eliminated the reported issues. After some investigation we found that all repositories experiencing failures contain replace references, which seem to be improperly
- acknowledged by the MIDX bitmap generation logic. During cloning or
- fetching, git-pack-objects, which may make use of MIDX bitmap to find
- objects to pack, would give wrong objects even if we explicitly
- specified not to enable replace refs by GIT_NO_REPLACE_OBJECTS=1.
- Indeed, this issue appears to have persisted since the introduction of
- MIDX.
+ acknowledged by the MIDX bitmap generation logic.
- This patch updates the MIDX logic to disable replace objects during
- operations, mirroring the handling seen in single pack index scenarios,
- i.e. git-index-pack and git-pack-objects. The added test uses
- git-rev-list to give a more intuitive check.
+ A more thorough explanation about the root cause from Taylor Blau says:
+ Indeed, the pack-bitmap-write machinery does not itself call
+ disable_replace_refs(). So when it generates a reachability bitmap, it
+ is doing so with the replace refs in mind. You can see that this is
+ indeed the cause of the problem by looking at the output of an
+ instrumented version of Git that indicates what bits are being set
+ during the bitmap generation phase.
+
+ With replace refs (incorrectly) enabled, we get:
+
+ [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]
+
+ and doing the same after calling disable_replace_refs(), we instead get:
+
+ [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]
+
+ Single pack bitmaps are unaffected by this issue because we generate
+ them from within pack-objects, which does call disable_replace_refs().
+
+ This patch updates the MIDX logic to disable replace objects within the
+ multi-pack-index builtin, and a test showing a clone (which would fail
+ with MIDX bitmap) is added to demonstrate the bug.
+
+ Helped-by: Taylor Blau <[email protected]>
## builtin/multi-pack-index.c ##
@@ t/t5326-multi-pack-bitmaps.sh: test_expect_success 'tagged commits are selected
+ cd repo &&
+
+ test_commit A &&
-+ A=$(git rev-parse HEAD) &&
+ test_commit B &&
-+ B=$(git rev-parse HEAD) &&
-+ git checkout --orphan=orphan $A &&
++ git checkout --orphan=orphan A &&
+ test_commit orphan &&
-+ C=$(git rev-parse HEAD) &&
-+ git rev-list --objects --no-object-names $B |sort >expected &&
+
-+ git replace $A $C &&
-+ git repack -ad &&
-+ git multi-pack-index write --bitmap &&
-+ git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&
-+ test_cmp expected actual
++ git replace A HEAD &&
++ git repack -ad --write-midx --write-bitmap-index &&
++
++ # generating reachability bitmaps with replace refs
++ # enabled will result in broken clones
++ git clone --no-local --bare . clone.git
+ )
+'
+
builtin/multi-pack-index.c | 3 +++ t/t5326-multi-pack-bitmaps.sh | 21 +++++++++++++++++++++ 2 files changed, 24 insertions(+)
builtin/multi-pack-index.c +3 -0
@@ -8,6 +8,7 @@#include "strbuf.h" #include "trace2.h" #include "object-store-ll.h" +#include "replace-object.h" #define BUILTIN_MIDX_WRITE_USAGE \ N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]" \
@@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv, };struct option *options =parse_options_concat (builtin_multi_pack_index_options, common_opts); +disable_replace_refs (); +git_config (git_default_config,NULL ); if (the_repository &&
t/t5326-multi-pack-bitmaps.sh +21 -0
@@ -434,6 +434,27 @@ test_expect_success 'tagged commits are selected for bitmapping' ' ) ' +test_expect_success 'do not follow replace objects for MIDX bitmap' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + + test_commit A && + test_commit B && + git checkout --orphan=orphan A && + test_commit orphan && + + git replace A HEAD && + git repack -ad --write-midx --write-bitmap-index && + + # generating reachability bitmaps with replace refs + # enabled will result in broken clones + git clone --no-local --bare . clone.git + ) +' + corrupt_file () { chmod a+w "$1" && printf "bogus" | dd of="$1" bs=1 seek="12" conv=notrunc
base-commit: 3c2a3fdc388747b9eaf4a4a4f2035c1c9ddb26d0
Signature · 1 line
gitgitgadget
Quoted 5 lines · blanet via GitGitGadget
From: Xing Xin <[email protected]>
...Helped-by Taylor Blau <[email protected]>Signed-off-by Xing Xin <[email protected]>---
I think this took the review in
https://lore.kernel.org/git/[email protected]/
into account and is in good shape?
Thanks, both.
Quoted 8 lines · Junio C Hamano
Quoted 5 lines · blanet via GitGitGadget
From: Xing Xin <[email protected]>
...Helped-by Taylor Blau <[email protected]>Signed-off-by Xing Xin <[email protected]>---
Quoted 5 lines
I think this took the review in
https://lore.kernel.org/git/[email protected]/
into account and is in good shape?
Yes, sorry for not explicitly ack-ing, this version looks good to me.
Thanks,
Taylor
Quoted 9 lines · Taylor Blau
Quoted 8 lines · Junio C Hamano
Quoted 5 lines · blanet via GitGitGadget
From: Xing Xin <[email protected]>
...Helped-by Taylor Blau <[email protected]>Signed-off-by Xing Xin <[email protected]>---
Quoted 7 lines
Quoted 5 lines
I think this took the review in
https://lore.kernel.org/git/[email protected]/
into account and is in good shape?
Yes, sorry for not explicitly ack-ing, this version looks good to me.
Thanks.