pack-objects: fix a pair of MIDX bitmap-related races
cover pack-objects: fix a pair of MIDX bitmap-related races
This series fixes two races related to using multi-pack bitmaps in pack-objects.
The crux of both is that we call is_pack_valid() far too late, leaving us in a situation where pack-objects committed to using objects from a specific pack in the MIDX bitmap, without having actually opened those packs. So if those packs are removed in the background (e.g., due to a simultaneous repack), any ongoing clones or fetches will see this error message:
remote: Enumerating objects: 1498802, done.
remote: fatal: packfile ./objects/pack/pack-$HASH.pack cannot be accessed
remote: aborting due to possible repository corruption on the remote side.
See 4c08018204 (pack-objects: protect against disappearing packs, 2011-10-14) and bab919bd44 (pack-bitmap: check pack validity when opening bitmap, 2015-03-26) for previous treatment of similar issues.
Taylor Blau (2):
pack-bitmap: check preferred pack validity when opening MIDX bitmap builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
builtin/pack-objects.c | 8 ++++++-- pack-bitmap.c | 19 +++++++++++++++++-- 2 files changed, 23 insertions(+), 4 deletions(-)
Signature · 1 line
2.36.1.76.g80c0bcd80e
PATCH 1/2 pack-bitmap: check preferred pack validity when opening MIDX bitmap
When pack-objects adds an entry to its packing list, it marks the packfile and offset containing the object, which we may later use during verbatim reuse (c.f., write_reused_pack_verbatim()).
If the packfile in question is deleted in the background (e.g., due to a concurrent git repack), we'll die() as a result of calling use_pack(). 4c08018204 (pack-objects: protect against disappearing packs, 2011-10-14) worked around this by opening the pack ahead of time before recording it as a valid source for reuse.
4c08018204's treatment meant that we could tolerate disappearing packs, since it ensures we always have an open file descriptor any pack that we mark as a valid source for reuse. This tightens the race to only happen when we need to close an open pack's file descriptor (c.f., the caller of packfile.c::get_max_fd_limit()) _and_ that pack was deleted, in which case we'll complain that a pack could not be accessed and die().
The pack bitmap code does this, too, since prior to bab919bd44 (pack-bitmap: check pack validity when opening bitmap, 2015-03-26) it was vulnerable to the same race.
The MIDX bitmap code does not do this, and is vulnerable to the same race. Apply the same treatment as bab919bd44 to the routine responsible for opening multi-pack bitmaps to close this race.
Similar to bab919bd44, we could technically just add this check in reuse_partial_packfile_from_bitmap(), since it's technically possible to use a MIDX .bitmap without needing to open any of its packs. But it's simpler to do the check as early as possible, covering all direct uses of the preferred pack. Note that doing this check early requires us to call prepare_midx_pack() early, too, so move the relevant part of that loop from load_reverse_index() into open_midx_bitmap_1().
--- pack-bitmap.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
pack-bitmap.c +17 -2
@@ -315,6 +315,8 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,struct stat st;char *idx_name =midx_bitmap_filename (midx);int fd =git_open (idx_name); +uint32_t i; +struct packed_git *preferred;free (idx_name);
@@ -353,6 +355,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,warning (_ ("multi-pack bitmap is missing required reverse index" )); goto cleanup; } + +for (i =0 ; i < bitmap_git->midx->num_packs; i++) { +if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) +die (_ ("could not open pack %s" ), + bitmap_git->midx->pack_names[i]); + } + + preferred = bitmap_git->midx->packs[midx_preferred_pack (bitmap_git)]; +if (!is_pack_valid (preferred)) { +close (fd); +warning (_ ("preferred pack (%s) is invalid" ), + preferred->pack_name); + goto cleanup; + } +return 0 ; cleanup:
@@ -429,8 +446,6 @@ static int load_reverse_index(struct bitmap_index *bitmap_git)* since we will need to make use of them in pack-objects. */ for (i =0 ; i < bitmap_git->midx->num_packs; i++) { -if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) -die (_ ("load_reverse_index: could not open pack" )); ret =load_pack_revindex (bitmap_git->midx->packs[i]);if (ret)return ret;
Signature · 1 line
2.36.1.76.g80c0bcd80e
Re: [PATCH 1/2] pack-bitmap: check preferred pack validity when opening MIDX bitmap
Quoted 3 lines · Taylor Blau
The pack bitmap code does this, too, since prior to
bab919bd44(pack-bitmap: check pack validity when opening bitmap, 2015-03-26) it was vulnerable to the same race.
That might be a GitHub internal reference to some other commit? dc1daacd (pack-bitmap: check pack validity when opening bitmap, 2021-07-23) is what I found.
Quoted 3 lines
The MIDX bitmap code does not do this, and is vulnerable to the same race. Apply the same treatment as
bab919bd44to the routine responsible for opening multi-pack bitmaps to close this race.
Same reference here and ...
Quoted 1 line
Similar to
bab919bd44, we could technically just add this check in
... here. But the solution in dc1daacd is quite different from what we see here in the posted patch, so perhaps you are referring to something different. I dunno.
The call graph around the functions involved is
prepare_midx_bitmap_git()
-> open_midx_bitmap_1()
* opens, mmaps and closes bitmap file
-> load_midx_revindex()
-> load_bitmap()
-> load_reverse_index()
-> prepare_midx_pack()
-> load_pack_revindex()And prepare_midx_pack() for these packs is moved from load_reverse_index() to open_midx_bitmap_1() in this patch.
In addition, after doing so, we call is_pack_valid() on the single preferred pack and return failure.
Because load_bitmap() or load_reverse_index() cannot be done before you do open_midx_bitmap_1(), calling prepare_midx_pack() early will end up calling add_packed_git() on underlying packs, allowing us to access them even when somebody else removed them from the disk? Is that the idea?
Quoted 6 lines
reuse_partial_packfile_from_bitmap(), since it's technically possible to use a MIDX .bitmap without needing to open any of its packs. But it's simpler to do the check as early as possible, covering all direct uses of the preferred pack. Note that doing this check early requires us to callprepare_midx_pack()early, too, so move the relevant part of that loop fromload_reverse_index()intoopen_midx_bitmap_1().
OK. That matches my observation above, I guess. I do not quite get why it is sufficient to check only the preferred one, though.
Quoted 22 lines
Signed-off-by Taylor Blau <[email protected]>--- pack-bitmap.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)pack-bitmap.c +2
-0@@ -315,6 +315,8 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,struct stat st;char *idx_name =midx_bitmap_filename (midx);int fd =git_open (idx_name); +uint32_t i; +struct packed_git *preferred;free (idx_name);@@ -353,6 +355,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,warning (_ ("multi-pack bitmap is missing required reverse index" )); goto cleanup; }
pack-bitmap.c static int open_midx_bitmap_1(struct bitmap_index *bitmap_git, · line 367
Quoted 10 lines
pack-bitmap.c +10
-0+ +for (i =0 ; i < bitmap_git->midx->num_packs; i++) { +if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) +die (_ ("could not open pack %s" ), + bitmap_git->midx->pack_names[i]); + } + + preferred = bitmap_git->midx->packs[midx_preferred_pack (bitmap_git)]; +if (!is_pack_valid (preferred)) { +close (fd);
This close() does not look correct. After calling xmmap() to map the bitmap file to bitmap_git->map, we do not need the underlying file descriptor in order to use the contents of the file. We have closed it already at this point.
Quoted 17 lines
pack-bitmap.c +5
-2+warning (_ ("preferred pack (%s) is invalid" ), + preferred->pack_name); + goto cleanup; + } +return 0 ; cleanup:@@ -429,8 +446,6 @@ static int load_reverse_index(struct bitmap_index *bitmap_git)* since we will need to make use of them in pack-objects. */ for (i =0 ; i < bitmap_git->midx->num_packs; i++) { -if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) -die (_ ("load_reverse_index: could not open pack" )); ret =load_pack_revindex (bitmap_git->midx->packs[i]);if (ret)return ret;
Re: [PATCH 1/2] pack-bitmap: check preferred pack validity when opening MIDX bitmap
Quoted 9 lines · Junio C Hamano
Quoted 3 lines · Taylor Blau
The pack bitmap code does this, too, since prior to
bab919bd44(pack-bitmap: check pack validity when opening bitmap, 2015-03-26) it was vulnerable to the same race.That might be a GitHub internal reference to some other commit?
dc1daacd(pack-bitmap: check pack validity when opening bitmap, 2021-07-23) is what I found.
Oops. dc1daacdcc is the right reference (it's the version of our bab919bd44 that got submitted upstream).
Quoted 5 lines
Quoted 1 line
Similar to
bab919bd44, we could technically just add this check in... here. But the solution in
dc1daacdis quite different from what we see here in the posted patch, so perhaps you are referring to something different. I dunno.
They are similar. Both dc1daacdcc and this patch ensure that the pack we're going to do verbatim reuse from (i.e., the one that gets passed to reuse_partial_pack_from_bitmap()) has an open handle.
In the case of a pack bitmap, there is only one pack to choose from (the pack corresponding to the bitmap itself). In the case of a multi-pack bitmap, the preferred pack is the one we choose, since it is the only pack among those in the MIDX that we attempt verbatim reuse out of.
Quoted 14 lines
The call graph around the functions involved is
prepare_midx_bitmap_git()
->open_midx_bitmap_1()
* opens, mmaps and closes bitmap file
->load_midx_revindex()
->load_bitmap()-> load_reverse_index() -> prepare_midx_pack() -> load_pack_revindex()And
prepare_midx_pack()for these packs is moved fromload_reverse_index()toopen_midx_bitmap_1()in this patch.
Quoted 8 lines
In addition, after doing so, we call
is_pack_valid()on the single preferred pack and return failure.Because
load_bitmap()orload_reverse_index()cannot be done before you doopen_midx_bitmap_1(), callingprepare_midx_pack()early will end up callingadd_packed_git()on underlying packs, allowing us to access them even when somebody else removed them from the disk? Is that the idea?
Yes, exactly. It's similar to dc1daacdcc in that we need to have an open handle on the packfile itself in order to reuse chunks of it verbatim. Having the bitmap open signals pack-objects to say "it is OK to call reuse_partial_packfile_from_bitmap()", but if that function tries to open the packfile because it wasn't already opened like above, and in the meantime it went away, we'll end up in the "cannot be accessed" scenario.
Quoted 9 lines
Quoted 6 lines
reuse_partial_packfile_from_bitmap(), since it's technically possible to use a MIDX .bitmap without needing to open any of its packs. But it's simpler to do the check as early as possible, covering all direct uses of the preferred pack. Note that doing this check early requires us to callprepare_midx_pack()early, too, so move the relevant part of that loop fromload_reverse_index()intoopen_midx_bitmap_1().OK. That matches my observation above, I guess. I do not quite get why it is sufficient to check only the preferred one, though.
Only the preferred pack is the subject of verbatim reuse. We could open all packs, but see my note in the patch message for why I don't think that's a great idea.
The subsequent patch more aggressively opens packs handed to us via traverse_bitmap_commit_list().
Quoted 11 lines
Quoted 11 lines
pack-bitmap.c +7
-0@@ -353,6 +355,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,warning (_ ("multi-pack bitmap is missing required reverse index" )); goto cleanup; } + +for (i =0 ; i < bitmap_git->midx->num_packs; i++) { +if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) +die (_ ("could not open pack %s" ), + bitmap_git->midx->pack_names[i]); + } +
Quoted 8 lines
pack-bitmap.c static int open_midx_bitmap_1(struct bitmap_index *bitmap_git, · line 367
Quoted 3 lines
pack-bitmap.c +3
-0+ preferred = bitmap_git->midx->packs[midx_preferred_pack (bitmap_git)]; +if (!is_pack_valid (preferred)) { +close (fd);This
close()does not look correct. After callingxmmap()to map the bitmap file tobitmap_git->map, we do not need the underlying file descriptor in order to use the contents of the file. We have closed it already at this point.
Definitely a mistake, thanks for catching. Will remove it and send another version after there is some review on the second patch.
Thanks in the meantime for giving it a look over!
Thanks,
Taylor
PATCH 2/2 builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
In reply to Taylor BlauWhen using a multi-pack bitmap, pack-objects will try to perform its traversal using a call to traverse_bitmap_commit_list(), which calls add_object_entry_from_bitmap() to add each object it finds to its packing list.
This path can cause pack-objects to add objects from packs that don't have open pack_fds on them, by avoiding a call to is_pack_valid(). This is because we only call is_pack_valid() on the preferred pack (in order to do verbatim reuse via reuse_partial_packfile_from_bitmap()) and not others when loading a MIDX bitmap.
In this case, add_object_entry_from_bitmap() will check whether it wants each object entry by calling want_object_in_pack(), which will call want_found_object (since its caller already supplied a found_pack). In most cases (particularly without --local, and when ignored_packed_keep_on_disk and ignored_packed_keep_in_core are both "0"), we'll take the entry from the pack contained in the MIDX bitmap, all without an open pack_fd.
When we then try to use that entry later to assemble the actual pack, we'll be susceptible to any simultaneous writers moving that pack out of the way (e.g., due to a concurrent repack) without having an open file descriptor, causing races that result in errors like:
remote: Enumerating objects: 1498802, done.
remote: fatal: packfile ./objects/pack/pack-e57d433b5a588daa37fbe946e2b28dfaec03a93e.pack cannot be accessed
remote: aborting due to possible repository corruption on the remote side.This race can happen even with multi-pack bitmaps, since we may open a MIDX bitmap that is being rewritten long before its packs are actually unlinked.
Work around this by calling is_pack_valid() from within want_found_object(), matching the behavior in want_object_in_pack_one() (which has an analogous call). Most calls to is_pack_valid() should be basically no-ops, since only the first call requires us to open a file (subsequent calls realize the file is already open, and return immediately).
This does require us to make a small change in want_object_in_pack_one(), since want_found_object() may return -1 (indicating "keep searching for other packs containing this object") when *found_pack is non-NULL. Force want_object_in_pack_one() to call is_pack_valid() when p != *found_pack, not just when *found_pack is non-NULL.
An alternative approach to closing this race would have been to call is_pack_valid() on _all_ packs in a multi-pack bitmap on load. This has a couple of problems:
- it is unnecessarily expensive in the cases where we don't actually need to open any packs (e.g., in git rev-list --use-bitmap-index --count)
- more importantly, it means any time we would have hit this race, we'll avoid using bitmaps altogether, leading to significant slowdowns by forcing a full object traversal
--- builtin/pack-objects.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
builtin/pack-objects.c +6 -2
@@ -1357,6 +1357,9 @@ static int want_found_object(const struct object_id *oid, int exclude,if (incremental)return 0 ; +if (!is_pack_valid (p)) +return -1 ; +/* * When asked to do --local (do not include an object that appears in a * pack we borrow from elsewhere) or --honor-pack-keep (do not include
@@ -1424,14 +1427,15 @@ static int want_object_in_pack_one(struct packed_git *p, off_t *found_offset) {off_t offset; +int use_found = p == *found_pack; -if (p == *found_pack) +if (use_found) offset = *found_offset;else offset =find_pack_entry_one (oid->hash, p);if (offset) { -if (!*found_pack) { +if (!use_found) {if (!is_pack_valid (p))return -1 ; *found_offset = offset;
Signature · 1 line
2.36.1.76.g80c0bcd80e
Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
Quoted 11 lines · Taylor Blau
An alternative approach to closing this race would have been to call
is_pack_valid()on _all_ packs in a multi-pack bitmap on load. This has a couple of problems:- it is unnecessarily expensive in the cases where we don't actually need to open any packs (e.g., in
git rev-list --use-bitmap-index --count)- more importantly, it means any time we would have hit this race, we'll avoid using bitmaps altogether, leading to significant slowdowns by forcing a full object traversal
This answers a question I had about why we're only opening the preferred pack instead of all packs. (You mention in [1] that it's also answered in that patch message, but I didn't see it.) In any case, it might be clearer to move this part to the 1st commit.
[1] https://lore.kernel.org/git/Yn63nDhSBIEa3%2F+%[email protected]/
Quoted 7 lines
Work around this by calling
is_pack_valid()from withinwant_found_object(), matching the behavior inwant_object_in_pack_one()(which has an analogous call). Most calls tois_pack_valid()should be basically no-ops, since only the first call requires us to open a file (subsequent calls realize the file is already open, and return immediately).
Quoted 6 lines
This does require us to make a small change in
want_object_in_pack_one(), sincewant_found_object()may return-1(indicating "keep searching for other packs containing this object") when*found_packis non-NULL. Forcewant_object_in_pack_one()to callis_pack_valid()whenp != *found_pack, not just when*found_packis non-NULL.
It took me a while to realize that the relevant want_found_object() invocation that may return -1 is not the one in want_object_in_pack_one(), but in the latter's caller want_object_in_pack(). But even in this case, couldn't want_found_object() return -1 (see the very end of the function) even before this patch?
Quoted 8 lines
builtin/pack-objects.c +2
-1@@ -1424,14 +1427,15 @@ static int want_object_in_pack_one(struct packed_git *p, off_t *found_offset) {off_t offset; +int use_found = p == *found_pack; -if (p == *found_pack) +if (use_found)
builtin/pack-objects.c static int want_object_in_pack_one(struct packed_git *p, · line 1441
Quoted 10 lines
builtin/pack-objects.c +1
-1offset = *found_offset; else offset =find_pack_entry_one (oid->hash, p);if (offset) { -if (!*found_pack) { +if (!use_found) {if (!is_pack_valid (p))return -1 ; *found_offset = offset;
My understanding of the purpose of this code change is that if we reach here with a non-NULL *found_pack, it is likely that *found_pack contains an invalid pack, and this part overwrites *found_pack (and *found_offset) if it finds a valid pack. This seems like a good change, but I don't see how this is a result of something that "does require us" (as far as I can tell, *found_pack could have already been invalid before this patch, so the downstream code should already be able to handle it). Maybe it's just that we couldn't tell if the pack is invalid previously, but now we can; but if so, it would be better to say "use this added information to overwrite *found_pack when it makes sense" or something like that.
(An alternative to the change in this patch may be to reset *found_pack to NULL when it is found that the pack is invalid, but I haven't investigated all the callers to see if they can tolerate *found_pack moving changing non-NULL to NULL, so the change in this patch is probably more practical.)
Other than that (and the "close(fd)" thing in patch 1 that Junio mentioned [2]), this series looks good to me. Thanks for noticing and fixing the bug.
Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
Quoted 13 lines · Jonathan Tan
Quoted 11 lines · Taylor Blau
An alternative approach to closing this race would have been to call
is_pack_valid()on _all_ packs in a multi-pack bitmap on load. This has a couple of problems:- it is unnecessarily expensive in the cases where we don't actually need to open any packs (e.g., in
git rev-list --use-bitmap-index --count)- more importantly, it means any time we would have hit this race, we'll avoid using bitmaps altogether, leading to significant slowdowns by forcing a full object traversal
Quoted 6 lines
This answers a question I had about why we're only opening the preferred pack instead of all packs. (You mention in [1] that it's also answered in that patch message, but I didn't see it.) In any case, it might be clearer to move this part to the 1st commit.
[1] https://lore.kernel.org/git/Yn63nDhSBIEa3%2F+%[email protected]/
Makes sense, will do. In [1] I was referring to why we wanted to call is_pack_valid() as early as we did, and not in reuse_partial_packfile_from_bitmap().
But the quoted part here is useful context for the first patch, too, so I moved it up.
Quoted 14 lines
Quoted 7 lines
Work around this by calling
is_pack_valid()from withinwant_found_object(), matching the behavior inwant_object_in_pack_one()(which has an analogous call). Most calls tois_pack_valid()should be basically no-ops, since only the first call requires us to open a file (subsequent calls realize the file is already open, and return immediately).Quoted 6 lines
This does require us to make a small change in
want_object_in_pack_one(), sincewant_found_object()may return-1(indicating "keep searching for other packs containing this object") when*found_packis non-NULL. Forcewant_object_in_pack_one()to callis_pack_valid()whenp != *found_pack, not just when*found_packis non-NULL.
Quoted 6 lines
It took me a while to realize that the relevant
want_found_object()invocation that may return -1 is not the one inwant_object_in_pack_one(), but in the latter's callerwant_object_in_pack(). But even in this case, couldn'twant_found_object()return -1 (see the very end of the function) even before this patch?
Perhaps changing the parenthetical to be:
(indicating that want_object_in_pack() should continue searching for other packs containing this object)
Yes, want_found_object() could have returned -1 before, but the only time when *found_pack != NULL and want_found_object() would have returned -1 is when given --local with at least one non-local pack.
I actually think it's possible we have a bug there, since AFAICT the pre-image of this patch would have left the non-NULL *found_pack alone but picked a copy of the given object from a _different_ pack. So I think this change inadvertently resolves that bug.
Quoted 22 lines
Quoted 8 lines
builtin/pack-objects.c +2
-1@@ -1424,14 +1427,15 @@ static int want_object_in_pack_one(struct packed_git *p, off_t *found_offset) {off_t offset; +int use_found = p == *found_pack; -if (p == *found_pack) +if (use_found)builtin/pack-objects.c static int want_object_in_pack_one(struct packed_git *p, · line 1441
Quoted 10 lines
builtin/pack-objects.c +1
-1offset = *found_offset; else offset =find_pack_entry_one (oid->hash, p);if (offset) { -if (!*found_pack) { +if (!use_found) {if (!is_pack_valid (p))return -1 ; *found_offset = offset;My understanding of the purpose of this code change is that if we reach here with a non-NULL
*found_pack, it is likely that*found_packcontains an invalid pack, and this part overwrites*found_pack(and
Quoted 8 lines
*found_offset) if it finds a valid pack. This seems like a good change, but I don't see how this is a result of something that "does require us" (as far as I can tell,*found_packcould have already been invalid before this patch, so the downstream code should already be able to handle it). Maybe it's just that we couldn't tell if the pack is invalid previously, but now we can; but if so, it would be better to say "use this added information to overwrite*found_packwhen it makes sense" or something like that.
I think my reply above indicates why this change is necessary, but if we're talking about separate issues, let me know.
Thanks,
Taylor
Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
Quoted 4 lines · Taylor Blau
Perhaps changing the parenthetical to be:
(indicating that
want_object_in_pack()should continue searching for other packs containing this object)
This works, thanks.
Quoted 8 lines
Yes,
want_found_object()could have returned -1 before, but the only time when*found_pack != NULLandwant_found_object()would have returned -1 is when given--localwith at least one non-local pack.I actually think it's possible we have a bug there, since AFAICT the pre-image of this patch would have left the non-NULL
*found_packalone but picked a copy of the given object from a _different_ pack. So I think this change inadvertently resolves that bug.
[snip]
Quoted 2 lines
I think my reply above indicates why this change is necessary, but if we're talking about separate issues, let me know.
Ah, I was presupposing that the existing code works correctly. But yes, this makes sense.
Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
In reply to Jonathan TanQuoted 5 lines · Jonathan Tan
(An alternative to the change in this patch may be to reset
*found_packto NULL when it is found that the pack is invalid, but I haven't investigated all the callers to see if they can tolerate*found_packmoving changing non-NULL to NULL, so the change in this patch is probably more practical.)
I haven't either, but I think that this points out a flaw in the patch I originally posted.
Consider this:
- want_object_in_pack() calls want_found_object() with a pack that has gone away and has zero open fds, and want_found_object() returns -1
- want_object_in_pack() continues and calls want_object_in_pack_one() later on, with some pack that is the same as *found_pack
- want_object_in_pack_one() then _doesn't_ call is_pack_valid()
(since `p == *found_pack`), leaving us in the same situation as
before.I think that would be sufficient to hit this race even after this patch.
I'll take a look to see if want_object_in_pack() callers can handle *found_pack being set back to NULL. They should be able to, but I want to do a little more careful analysis to be sure.
Thanks for pointing this out, I am so glad for your review! :-)
Thanks,
Taylor
Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
Quoted 19 lines · Taylor Blau
Quoted 5 lines · Jonathan Tan
(An alternative to the change in this patch may be to reset
*found_packto NULL when it is found that the pack is invalid, but I haven't investigated all the callers to see if they can tolerate*found_packmoving changing non-NULL to NULL, so the change in this patch is probably more practical.)I haven't either, but I think that this points out a flaw in the patch I originally posted.
Consider this:
-
want_object_in_pack()callswant_found_object()with a pack that has gone away and has zero open fds, andwant_found_object()returns -1
-want_object_in_pack()continues and callswant_object_in_pack_one()later on, with some pack that is the same as*found_pack
-want_object_in_pack_one()then _doesn't_ callis_pack_valid()
Quoted 4 lines
(since `p == *found_pack`), leaving us in the same situation as before.I think that would be sufficient to hit this race even after this patch.
Ah, yes, indeed this would be a problem.
Quoted 3 lines
I'll take a look to see if
want_object_in_pack()callers can handle*found_packbeing set back to NULL. They should be able to, but I want to do a little more careful analysis to be sure.
Sounds good.
Quoted 1 line
Thanks for pointing this out, I am so glad for your review! :-)
Thanks for your kind words! Thanks for your explanations too.
v2 cover pack-objects: fix a pair of MIDX bitmap-related races
In reply to Taylor BlauThis is a small-ish reroll of mine and Victoria's series to fix a couple of races related to using multi-pack bitmaps in pack-objects.
The crux of both is that we call is_pack_valid() far too late, leaving us in a situation where pack-objects committed to using objects from a specific pack in the MIDX bitmap, without having actually opened those packs. So if those packs are removed in the background (e.g., due to a simultaneous repack), any ongoing clones or fetches will see this error message:
remote: Enumerating objects: 1498802, done.
remote: fatal: packfile ./objects/pack/pack-$HASH.pack cannot be accessed
remote: aborting due to possible repository corruption on the remote side.
The first patch is mostly unchanged (except for removing an accidental double-close()), but the second patch has itself turned into three new patches in order to resolve the issue described in [1].
Thanks in advance for your review!
[1]: https://lore.kernel.org/git/[email protected]/
Taylor Blau (4):
pack-bitmap.c: check preferred pack validity when opening MIDX bitmap builtin/pack-objects.c: avoid redundant NULL check builtin/pack-objects.c: ensure included --stdin-packs exist builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
builtin/pack-objects.c | 43 +++++++++++++++++++++++++----------------- pack-bitmap.c | 18 ++++++++++++++++-- 2 files changed, 42 insertions(+), 19 deletions(-)
Range-diff against v1:
1: 83e2ad3962 ! 1: 618e8a6166 pack-bitmap: check preferred pack validity when opening MIDX bitmap
@@ Metadata
Author: Taylor Blau <[email protected]>
## Commit message ##
- pack-bitmap: check preferred pack validity when opening MIDX bitmap
+ pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
When pack-objects adds an entry to its packing list, it marks the packfile and offset containing the object, which we may later use during verbatim reuse (c.f., write_reused_pack_verbatim()).
If the packfile in question is deleted in the background (e.g., due to a
- concurrent git repack), we'll die() as a result of calling use_pack().
- 4c08018204 (pack-objects: protect against disappearing packs,
- 2011-10-14) worked around this by opening the pack ahead of time before
- recording it as a valid source for reuse.
+ concurrent git repack), we'll die() as a result of calling use_pack(),
+ unless we have an open file descriptor on the pack itself. 4c08018204
+ (pack-objects: protect against disappearing packs, 2011-10-14) worked
+ around this by opening the pack ahead of time before recording it as a
+ valid source for reuse.
4c08018204's treatment meant that we could tolerate disappearing packs,
- since it ensures we always have an open file descriptor any pack that we
- mark as a valid source for reuse. This tightens the race to only happen
- when we need to close an open pack's file descriptor (c.f., the caller
- of packfile.c::get_max_fd_limit()) _and_ that pack was deleted, in
- which case we'll complain that a pack could not be accessed and die().
+ since it ensures we always have an open file descriptor on any pack that
+ we mark as a valid source for reuse. This tightens the race to only
+ happen when we need to close an open pack's file descriptor (c.f., the
+ caller of packfile.c::get_max_fd_limit()) _and_ that pack was deleted,
+ in which case we'll complain that a pack could not be accessed and
+ die().
- The pack bitmap code does this, too, since prior to bab919bd44
- (pack-bitmap: check pack validity when opening bitmap, 2015-03-26) it
+ The pack bitmap code does this, too, since prior to dc1daacdcc
+ (pack-bitmap: check pack validity when opening bitmap, 2021-07-23) it
was vulnerable to the same race.
The MIDX bitmap code does not do this, and is vulnerable to the same
- race. Apply the same treatment as bab919bd44 to the routine responsible
- for opening multi-pack bitmaps to close this race.
+ race. Apply the same treatment as dc1daacdcc to the routine responsible
+ for opening the multi-pack bitmap's preferred pack to close this race.
- Similar to bab919bd44, we could technically just add this check in
- reuse_partial_packfile_from_bitmap(), since it's technically possible to
- use a MIDX .bitmap without needing to open any of its packs. But it's
- simpler to do the check as early as possible, covering all direct uses
- of the preferred pack. Note that doing this check early requires us to
- call prepare_midx_pack() early, too, so move the relevant part of that
- loop from load_reverse_index() into open_midx_bitmap_1().
+ This patch handles the "preferred" pack (c.f., the section
+ "multi-pack-index reverse indexes" in
+ Documentation/technical/pack-format.txt) specially, since pack-objects
+ depends on reusing exact chunks of that pack verbatim in
+ reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,
+ the utility of a bitmap is significantly diminished.
+
+ Similar to dc1daacdcc, we could technically just add this check in
+ reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX
+ .bitmap without needing to open any of its packs. But it's simpler to do
+ the check as early as possible, covering all direct uses of the
+ preferred pack. Note that doing this check early requires us to call
+ prepare_midx_pack() early, too, so move the relevant part of that loop
+ from load_reverse_index() into open_midx_bitmap_1().
+
+ Subsequent patches handle the non-preferred packs in a slightly
+ different fashion.
@@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
+
+ preferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];
+ if (!is_pack_valid(preferred)) {
-+ close(fd);
+ warning(_("preferred pack (%s) is invalid"),
+ preferred->pack_name);
+ goto cleanup;
2: 9adf6e1341 < -: ---------- builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects -: ---------- > 2: 2719d33f32 builtin/pack-objects.c: avoid redundant NULL check -: ---------- > 3: cdc3265ec2 builtin/pack-objects.c: ensure included `--stdin-packs` exist -: ---------- > 4: 3fc3a95517 builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
Signature · 1 line
2.36.1.94.gb0d54bedca
PATCH v2 1/4 pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
When pack-objects adds an entry to its packing list, it marks the packfile and offset containing the object, which we may later use during verbatim reuse (c.f., write_reused_pack_verbatim()).
If the packfile in question is deleted in the background (e.g., due to a concurrent git repack), we'll die() as a result of calling use_pack(), unless we have an open file descriptor on the pack itself. 4c08018204 (pack-objects: protect against disappearing packs, 2011-10-14) worked around this by opening the pack ahead of time before recording it as a valid source for reuse.
4c08018204's treatment meant that we could tolerate disappearing packs, since it ensures we always have an open file descriptor on any pack that we mark as a valid source for reuse. This tightens the race to only happen when we need to close an open pack's file descriptor (c.f., the caller of packfile.c::get_max_fd_limit()) _and_ that pack was deleted, in which case we'll complain that a pack could not be accessed and die().
The pack bitmap code does this, too, since prior to dc1daacdcc (pack-bitmap: check pack validity when opening bitmap, 2021-07-23) it was vulnerable to the same race.
The MIDX bitmap code does not do this, and is vulnerable to the same race. Apply the same treatment as dc1daacdcc to the routine responsible for opening the multi-pack bitmap's preferred pack to close this race.
This patch handles the "preferred" pack (c.f., the section "multi-pack-index reverse indexes" in Documentation/technical/pack-format.txt) specially, since pack-objects depends on reusing exact chunks of that pack verbatim in reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded, the utility of a bitmap is significantly diminished.
Similar to dc1daacdcc, we could technically just add this check in reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX .bitmap without needing to open any of its packs. But it's simpler to do the check as early as possible, covering all direct uses of the preferred pack. Note that doing this check early requires us to call prepare_midx_pack() early, too, so move the relevant part of that loop from load_reverse_index() into open_midx_bitmap_1().
Subsequent patches handle the non-preferred packs in a slightly different fashion.
--- pack-bitmap.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
pack-bitmap.c +16 -2
@@ -315,6 +315,8 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,struct stat st;char *idx_name =midx_bitmap_filename (midx);int fd =git_open (idx_name); +uint32_t i; +struct packed_git *preferred;free (idx_name);
@@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,warning (_ ("multi-pack bitmap is missing required reverse index" )); goto cleanup; } + +for (i =0 ; i < bitmap_git->midx->num_packs; i++) { +if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) +die (_ ("could not open pack %s" ), + bitmap_git->midx->pack_names[i]); + } + + preferred = bitmap_git->midx->packs[midx_preferred_pack (bitmap_git)]; +if (!is_pack_valid (preferred)) { +warning (_ ("preferred pack (%s) is invalid" ), + preferred->pack_name); + goto cleanup; + } +return 0 ; cleanup:
@@ -429,8 +445,6 @@ static int load_reverse_index(struct bitmap_index *bitmap_git)* since we will need to make use of them in pack-objects. */ for (i =0 ; i < bitmap_git->midx->num_packs; i++) { -if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) -die (_ ("load_reverse_index: could not open pack" )); ret =load_pack_revindex (bitmap_git->midx->packs[i]);if (ret)return ret;
Signature · 1 line
2.36.1.94.gb0d54bedca
Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
On Tue, May 24 2022, Taylor Blau wrote:
Just nits on the error reporting:
pack-bitmap.c static int open_midx_bitmap_1(struct bitmap_index *bitmap_git, · line 362
Quoted 9 lines
pack-bitmap.c +5
-0@@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,warning (_ ("multi-pack bitmap is missing required reverse index" )); goto cleanup; } + +for (i =0 ; i < bitmap_git->midx->num_packs; i++) { +if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) +die (_ ("could not open pack %s" ), + bitmap_git->midx->pack_names[i]);
Some existing API users of this & their error handling suggest that this message is wrong. I.e. it's not that we couldn't open it, but that we could open it and there's something wrong with it. Or perhaps their messages are misleading?
pack-bitmap.c static int open_midx_bitmap_1(struct bitmap_index *bitmap_git, · line 368
Quoted 6 lines
pack-bitmap.c +6
-0+ } + + preferred = bitmap_git->midx->packs[midx_preferred_pack (bitmap_git)]; +if (!is_pack_valid (preferred)) { +warning (_ ("preferred pack (%s) is invalid" ), + preferred->pack_name);
Likewise this? E.g. perhaps the permissions are just wrong or whatever, per open_packed_git_1().
Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
Quoted 5 lines · Ævar Arnfjörð Bjarmason
On Tue, May 24 2022, Taylor Blau wrote:
Just nits on the error reporting:
Quoted 14 lines
pack-bitmap.c static int open_midx_bitmap_1(struct bitmap_index *bitmap_git, · line 362
Quoted 9 lines
pack-bitmap.c +5
-0@@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,warning (_ ("multi-pack bitmap is missing required reverse index" )); goto cleanup; } + +for (i =0 ; i < bitmap_git->midx->num_packs; i++) { +if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) +die (_ ("could not open pack %s" ), + bitmap_git->midx->pack_names[i]);Some existing API users of this & their error handling suggest that this message is wrong. I.e. it's not that we couldn't open it, but that we could open it and there's something wrong with it. Or perhaps their messages are misleading?
I tried to reuse some similar message based on "git grep 'if (.*prepare_midx_pack'", so this was inspired by:
- the caller in midx.c::write_midx_internal(), whose error is "could not load pack", and
- the caller in midx.c::verify_midx_file(), whose error is "failed to load pack"
Are you suggesting we should s/open/load here and use the above error message? My feeling at the time was that "load" was basically synonymous with "open" given the context, but if you think they're different enough, or have a different suggestion LMK.
Thanks,
Taylor
Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
Quoted 29 lines · Taylor Blau
Quoted 5 lines · Ævar Arnfjörð Bjarmason
On Tue, May 24 2022, Taylor Blau wrote:
Just nits on the error reporting:
Quoted 14 lines
pack-bitmap.c static int open_midx_bitmap_1(struct bitmap_index *bitmap_git, · line 362
Quoted 9 lines
pack-bitmap.c +5
-0@@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,warning (_ ("multi-pack bitmap is missing required reverse index" )); goto cleanup; } + +for (i =0 ; i < bitmap_git->midx->num_packs; i++) { +if (prepare_midx_pack (the_repository, bitmap_git->midx, i)) +die (_ ("could not open pack %s" ), + bitmap_git->midx->pack_names[i]);Some existing API users of this & their error handling suggest that this message is wrong. I.e. it's not that we couldn't open it, but that we could open it and there's something wrong with it. Or perhaps their messages are misleading?
I tried to reuse some similar message based on "git grep 'if (.*prepare_midx_pack'", so this was inspired by:
- the caller in
midx.c::write_midx_internal(), whose error is "could not load pack", and
- the caller inmidx.c::verify_midx_file(), whose error is "failed to load pack"
Quoted 4 lines
Are you suggesting we should s/open/load here and use the above error message? My feeling at the time was that "load" was basically synonymous with "open" given the context, but if you think they're different enough, or have a different suggestion LMK.
Perhaps "parse" or something? Anyway with "could not open" I'd assume open() failed, but in this case it looks like we could open it, but (mostly?) failed later.
Maybe "could not load midx"? I don't know...
PATCH v2 2/4 builtin/pack-objects.c: avoid redundant NULL check
In reply to Taylor BlauBefore calling for_each_object_in_pack(), the caller read_packs_list_from_stdin() loops through each of the include_packs and checks that its ->util pointer (which is used to store the struct packed_git * itself) is non-NULL.
This check is redundant, because read_packs_list_from_stdin() already checks that the included packs are non-NULL earlier on in the same function (and it does not add any new entries in between).
Remove this check, since it is not doing anything in the meantime.
--- builtin/pack-objects.c | 2 -- 1 file changed, 2 deletions(-)
builtin/pack-objects.c +0 -2
@@ -3369,8 +3369,6 @@ static void read_packs_list_from_stdin(void)for_each_string_list_item (item, &include_packs) {struct packed_git *p = item->util; -if (!p) -die (_ ("could not find pack '%s'" ), item->string); for_each_object_in_pack(p, add_object_entry_from_pack, &revs,
Signature · 1 line
2.36.1.94.gb0d54bedca
Re: [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check
Quoted 10 lines · Taylor Blau
Before calling
for_each_object_in_pack(), the callerread_packs_list_from_stdin()loops through each of theinclude_packsand checks that its->utilpointer (which is used to store thestruct packed_git *itself) is non-NULL.This check is redundant, because
read_packs_list_from_stdin()already checks that the included packs are non-NULL earlier on in the same function (and it does not add any new entries in between).Remove this check, since it is not doing anything in the meantime.
Will it start doing something soon in a later step?
Oh, did you mean that after the earlier for_each_string_list_item() iteration over include_packs that died on an item with a NULL .util member, the code did not do anything to cause this second iteration over the same list to suddenly start seeing an item with NULL .util?
I am puzzled by the mention of "in the meantime".
The patch itself looks correct, of course.
Thanks.
Quoted 19 lines
Co-authored-by Victoria Dye <[email protected]>Signed-off-by Taylor Blau <[email protected]>--- builtin/pack-objects.c | 2 -- 1 file changed, 2 deletions(-)builtin/pack-objects.c +0
-2@@ -3369,8 +3369,6 @@ static void read_packs_list_from_stdin(void)for_each_string_list_item (item, &include_packs) {struct packed_git *p = item->util; -if (!p) -die (_ ("could not find pack '%s'" ), item->string); for_each_object_in_pack(p, add_object_entry_from_pack, &revs,
Re: [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check
Quoted 13 lines · Junio C Hamano
Quoted 10 lines · Taylor Blau
Before calling
for_each_object_in_pack(), the callerread_packs_list_from_stdin()loops through each of theinclude_packsand checks that its->utilpointer (which is used to store thestruct packed_git *itself) is non-NULL.This check is redundant, because
read_packs_list_from_stdin()already checks that the included packs are non-NULL earlier on in the same function (and it does not add any new entries in between).Remove this check, since it is not doing anything in the meantime.
Quoted 8 lines
Will it start doing something soon in a later step?
Oh, did you mean that after the earlier
for_each_string_list_item()iteration overinclude_packsthat died on an item with a NULL .util member, the code did not do anything to cause this second iteration over the same list to suddenly start seeing an item with NULL .util?I am puzzled by the mention of "in the meantime".
The latter, sorry for the confusion.
Quoted 1 line
The patch itself looks correct, of course.
Thanks.
Taylor
PATCH v2 3/4 builtin/pack-objects.c: ensure included `--stdin-packs` exist
In reply to Taylor BlauA subsequent patch will teach want_object_in_pack() to set its *found_pack and *found_offset poitners to NULL when the provided pack does not pass the is_pack_valid() check.
The --stdin-packs mode of pack-objects is not quite prepared to handle this. To prepare it for this change, do the following two things:
- Ensure provided packs pass the is_pack_valid() check when collecting the caller-provided packs into the "included" and "excluded" lists.
- Gracefully handle any _invalid_ packs being passed to
`want_object_in_pack()`.
Calling is_pack_valid() early on makes it substantially less likely that we will have to deal with a pack going away, since we'll have an open file descriptor on its contents much earlier.
But even packs with open descriptors can become invalid in the future if we (a) hit our open descriptor limit, forcing us to close some open packs, and (b) one of those just-closed packs has gone away in the meantime.
add_object_entry_from_pack() depends on having a non-NULL *found_pack, since it passes that pointer to packed_object_info(), meaning that we would SEGV if the pointer became NULL (like we propose to do in want_object_in_pack() in the following patch).
But avoiding calling packed_object_info() entirely is OK, too, since its only purpose is to identify which objects in the included packs are commits, so that they can form the tips of the advisory traversal used to discover the object namehashes.
Failing to do this means that at worst we will produce lower-quality deltas, but it does not prevent us from generating the pack as long as we can find a copy of each object from the disappearing pack in some other part of the repository.
--- builtin/pack-objects.c | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-)
builtin/pack-objects.c +20 -15
@@ -3201,10 +3201,8 @@ static int add_object_entry_from_pack(const struct object_id *oid,uint32_t pos, void *_data) { - struct rev_info *revs = _data; -struct object_info oi =OBJECT_INFO_INIT ; off_t ofs; -enum object_type type; +enum object_type type =OBJ_NONE ;display_progress (progress_state, ++nr_seen);
@@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,if (!want_object_in_pack (oid,0 , &p, &ofs))return 0 ; - oi.typep = &type; -if (packed_object_info (the_repository, p, ofs, &oi) <0 ) -die (_ ("could not get type of object %s in pack %s" ), -oid_to_hex (oid), p->pack_name); -else if (type ==OBJ_COMMIT ) { -/* -* commits in included packs are used as starting points for the -* subsequent revision walk -*/ -add_pending_oid (revs,NULL , oid,0 ); +if (p) { +struct rev_info *revs = _data; +struct object_info oi =OBJECT_INFO_INIT ; + + oi.typep = &type; +if (packed_object_info (the_repository, p, ofs, &oi) <0 ) { +die (_ ("could not get type of object %s in pack %s" ), +oid_to_hex (oid), p->pack_name); + }else if (type ==OBJ_COMMIT ) { +/* +* commits in included packs are used as starting points for the +* subsequent revision walk +*/ +add_pending_oid (revs,NULL , oid,0 ); + } + + stdin_packs_found_nr++; } - stdin_packs_found_nr++; -create_object_entry (oid, type,0 ,0 ,0 , p, ofs);return 0 ;
@@ -3346,6 +3349,8 @@ static void read_packs_list_from_stdin(void)struct packed_git *p = item->util;if (!p)die (_ ("could not find pack '%s'" ), item->string); +if (!is_pack_valid (p)) +die (_ ("packfile %s cannot be accessed" ), p->pack_name); }/*
Signature · 1 line
2.36.1.94.gb0d54bedca
Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
Quoted 34 lines · Taylor Blau
builtin/pack-objects.c +13
-13-struct rev_info *revs = _data; -struct object_info oi =OBJECT_INFO_INIT ;off_t ofs; -enum object_type type; +enum object_type type =OBJ_NONE ;display_progress (progress_state, ++nr_seen);@@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,if (!want_object_in_pack (oid,0 , &p, &ofs))return 0 ; - oi.typep = &type; -if (packed_object_info (the_repository, p, ofs, &oi) <0 ) -die (_ ("could not get type of object %s in pack %s" ), -oid_to_hex (oid), p->pack_name); -else if (type ==OBJ_COMMIT ) { -/* -* commits in included packs are used as starting points for the -* subsequent revision walk -*/ -add_pending_oid (revs,NULL , oid,0 ); +if (p) { +struct rev_info *revs = _data; +struct object_info oi =OBJECT_INFO_INIT ; + + oi.typep = &type; +if (packed_object_info (the_repository, p, ofs, &oi) <0 ) { +die (_ ("could not get type of object %s in pack %s" ), +oid_to_hex (oid), p->pack_name); + }else if (type ==OBJ_COMMIT ) { +/* +* commits in included packs are used as starting points for the +* subsequent revision walk
builtin/pack-objects.c static int add_object_entry_from_pack(const struct object_id *oid, · line 3235
Quoted 10 lines
builtin/pack-objects.c +5
-2+*/ +add_pending_oid (revs,NULL , oid,0 ); + } + + stdin_packs_found_nr++; } - stdin_packs_found_nr++; -create_object_entry (oid, type,0 ,0 ,0 , p, ofs);
Not rhetorical, since I have no idea: Is the behavior change here to make create_object_entry with type=OBJ_NONE desired? I.e. do we actually want to create object entries for OBJ_NONE?
If that is the case I for one would find this a bit easier to follow like this, even if it has some minor duplication, i.e. the intent is clearer:
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index ffeaecd1d84..a447f6d5164 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,
void *_data)
{
off_t ofs;
- enum object_type type = OBJ_NONE;
display_progress(progress_state, ++nr_seen);
@@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,
if (p) {
struct rev_info *revs = _data;
struct object_info oi = OBJECT_INFO_INIT;
+ enum object_type type;
oi.typep = &type;
if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
@@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid,
}
stdin_packs_found_nr++;
- }
- create_object_entry(oid, type, 0, 0, 0, p, ofs);
+ create_object_entry(oid, type, 0, 0, 0, p, ofs);
+ } else {
+ create_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs);
+ }
return 0;
}Or the same with adding "type = OBJ_NONE" to the "else" branch, leaving the initial "type" uninitialized"?
Or perhaps this is a bug? I see some OBJ_NONE mentions in the code, but do packfiles really have "none" objects in some fashion as far as add_object_entry_from_pack() is concerned? (I'm not familiar enough with this part of the codebase to know).
Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
Quoted 44 lines · Ævar Arnfjörð Bjarmason
Quoted 41 lines · Taylor Blau
builtin/pack-objects.c +18
-13-struct rev_info *revs = _data; -struct object_info oi =OBJECT_INFO_INIT ;off_t ofs; -enum object_type type; +enum object_type type =OBJ_NONE ;display_progress (progress_state, ++nr_seen);@@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,if (!want_object_in_pack (oid,0 , &p, &ofs))return 0 ; - oi.typep = &type; -if (packed_object_info (the_repository, p, ofs, &oi) <0 ) -die (_ ("could not get type of object %s in pack %s" ), -oid_to_hex (oid), p->pack_name); -else if (type ==OBJ_COMMIT ) { -/* -* commits in included packs are used as starting points for the -* subsequent revision walk -*/ -add_pending_oid (revs,NULL , oid,0 ); +if (p) { +struct rev_info *revs = _data; +struct object_info oi =OBJECT_INFO_INIT ; + + oi.typep = &type; +if (packed_object_info (the_repository, p, ofs, &oi) <0 ) { +die (_ ("could not get type of object %s in pack %s" ), +oid_to_hex (oid), p->pack_name); + }else if (type ==OBJ_COMMIT ) { +/* +* commits in included packs are used as starting points for the +* subsequent revision walk +*/ +add_pending_oid (revs,NULL , oid,0 ); + } + + stdin_packs_found_nr++; }
Quoted 7 lines
builtin/pack-objects.c static int add_object_entry_from_pack(const struct object_id *oid, · line 3235
Quoted 3 lines
builtin/pack-objects.c +0
-2- stdin_packs_found_nr++; -create_object_entry (oid, type,0 ,0 ,0 , p, ofs);Not rhetorical, since I have no idea: Is the behavior change here to make
create_object_entrywith type=OBJ_NONE desired? I.e. do we actually want to create object entries forOBJ_NONE?
This is intentional. OBJ_NONE tells create_object_entry() "we don't know the type of this object yet", and then check_object() (which does the bulk of the work in the "Counting objects" phase) goes through and fills in any missing type information.
The caller in builtin/pack-objects.c::read_object_list_from_stdin() is a good example of this (all of the objects created this way start out with OBJ_NONE).
Quoted 39 lines
If that is the case I for one would find this a bit easier to follow like this, even if it has some minor duplication, i.e. the intent is clearer:
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index ffeaecd1d84..a447f6d5164 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid, void *_data) { off_t ofs; - enum object_type type = OBJ_NONE; display_progress(progress_state, ++nr_seen); @@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid, if (p) { struct rev_info *revs = _data; struct object_info oi = OBJECT_INFO_INIT; + enum object_type type; oi.typep = &type; if (packed_object_info(the_repository, p, ofs, &oi) < 0) { @@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid, } stdin_packs_found_nr++; - } - create_object_entry(oid, type, 0, 0, 0, p, ofs); + create_object_entry(oid, type, 0, 0, 0, p, ofs); + } else { + create_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs); + } return 0; }
Quoted 2 lines
Or the same with adding "type =
OBJ_NONE" to the "else" branch, leaving the initial "type" uninitialized"?
I'd be fine with that (and don't really have a very strong opinion either way). Let's see if anybody else has thoughts about it, and then I'm happy to change it in a subsequent version.
Thanks,
Taylor
Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
Quoted 53 lines · Taylor Blau
Quoted 44 lines · Ævar Arnfjörð Bjarmason
Quoted 41 lines · Taylor Blau
builtin/pack-objects.c +18
-13-struct rev_info *revs = _data; -struct object_info oi =OBJECT_INFO_INIT ;off_t ofs; -enum object_type type; +enum object_type type =OBJ_NONE ;display_progress (progress_state, ++nr_seen);@@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,if (!want_object_in_pack (oid,0 , &p, &ofs))return 0 ; - oi.typep = &type; -if (packed_object_info (the_repository, p, ofs, &oi) <0 ) -die (_ ("could not get type of object %s in pack %s" ), -oid_to_hex (oid), p->pack_name); -else if (type ==OBJ_COMMIT ) { -/* -* commits in included packs are used as starting points for the -* subsequent revision walk -*/ -add_pending_oid (revs,NULL , oid,0 ); +if (p) { +struct rev_info *revs = _data; +struct object_info oi =OBJECT_INFO_INIT ; + + oi.typep = &type; +if (packed_object_info (the_repository, p, ofs, &oi) <0 ) { +die (_ ("could not get type of object %s in pack %s" ), +oid_to_hex (oid), p->pack_name); + }else if (type ==OBJ_COMMIT ) { +/* +* commits in included packs are used as starting points for the +* subsequent revision walk +*/ +add_pending_oid (revs,NULL , oid,0 ); + } + + stdin_packs_found_nr++; }Quoted 7 lines
builtin/pack-objects.c static int add_object_entry_from_pack(const struct object_id *oid, · line 3235
Quoted 3 lines
builtin/pack-objects.c +0
-2- stdin_packs_found_nr++; -create_object_entry (oid, type,0 ,0 ,0 , p, ofs);Not rhetorical, since I have no idea: Is the behavior change here to make
create_object_entrywith type=OBJ_NONE desired? I.e. do we actually want to create object entries forOBJ_NONE?
Quoted 4 lines
This is intentional.
OBJ_NONEtellscreate_object_entry()"we don't know the type of this object yet", and thencheck_object()(which does the bulk of the work in the "Counting objects" phase) goes through and fills in any missing type information.
Ah, I didn't know that.
Quoted 43 lines
The caller in
builtin/pack-objects.c::read_object_list_from_stdin()is a good example of this (all of the objects created this way start out withOBJ_NONE).Quoted 39 lines
If that is the case I for one would find this a bit easier to follow like this, even if it has some minor duplication, i.e. the intent is clearer:
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index ffeaecd1d84..a447f6d5164 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid, void *_data) { off_t ofs; - enum object_type type = OBJ_NONE; display_progress(progress_state, ++nr_seen); @@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid, if (p) { struct rev_info *revs = _data; struct object_info oi = OBJECT_INFO_INIT; + enum object_type type; oi.typep = &type; if (packed_object_info(the_repository, p, ofs, &oi) < 0) { @@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid, } stdin_packs_found_nr++; - } - create_object_entry(oid, type, 0, 0, 0, p, ofs); + create_object_entry(oid, type, 0, 0, 0, p, ofs); + } else { + create_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs); + } return 0; }
Quoted 6 lines
Quoted 2 lines
Or the same with adding "type =
OBJ_NONE" to the "else" branch, leaving the initial "type" uninitialized"?I'd be fine with that (and don't really have a very strong opinion either way). Let's see if anybody else has thoughts about it, and then I'm happy to change it in a subsequent version.
FWIW I think you should place a particularly low value on my suggestion of this.
I.e. the last thing we should do is probably to optimize the code to be read by someone who hadn't spent even 10 minutes finding out such obvious major code-flow details, i.e. me not knowing about how OBJ_NONE was used in this case....
So it's probably all fine as-is, but p erhaps others will think it's good or whatever...
Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
In reply to Taylor BlauQuoted 3 lines · Taylor Blau
Calling
is_pack_valid()early on makes it substantially less likely that we will have to deal with a pack going away, since we'll have an open file descriptor on its contents much earlier.
Sorry for asking a stupid question (or two), but I am confused.
This does make sure that we can read and use the contents of the packfile even when somebody else removes it from the disk by ensuring that
(1) we have an open file descriptor to it, so that we could open mmap window into it at will; or
(2) we have a mmap window that covers all of it (this should be the norm on platforms with vast address space); or
(3) we are in the same state as (1) by opening the packfile to validate the pack right now.
and during the pack-object we are running (aka "repack"), we can continue to read from that pack that may have already disappeared from the disk.
But is that sufficient? Are we writing the resulting new pack(s) out in such a way that the repository is healthy without the pack we noticed is disappearing? How do we ensure that?
Thanks.
Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
Quoted 7 lines · Junio C Hamano
Quoted 3 lines · Taylor Blau
Calling
is_pack_valid()early on makes it substantially less likely that we will have to deal with a pack going away, since we'll have an open file descriptor on its contents much earlier.Sorry for asking a stupid question (or two), but I am confused.
No such thing as a stupid question, so your apology is not necessary in the slightest :).
Quoted 13 lines
This does make sure that we can read and use the contents of the packfile even when somebody else removes it from the disk by ensuring that
(1) we have an open file descriptor to it, so that we could open mmap window into it at will; or
(2) we have a mmap window that covers all of it (this should be the norm on platforms with vast address space); or
(3) we are in the same state as (1) by opening the packfile to validate the pack right now.
Quoted 7 lines
and during the pack-object we are running (aka "repack"), we can continue to read from that pack that may have already disappeared from the disk.
But is that sufficient? Are we writing the resulting new pack(s) out in such a way that the repository is healthy without the pack we noticed is disappearing? How do we ensure that?
It's sufficient in the sense that we're writing out all of the objects we were asked to (from pack-objects's perspective). Of course, if the "simultaneous writer" is just removing packs right after they are opened, that will produce an obviously-broken state. But assuming that repack isn't removing objects it shouldn't (which I think is a safe assumption from pack-objects' perspective, since all it cares about is writing packs that contain the desired set of objects), then we are OK.
Thanks,
Taylor
Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
In reply to Taylor BlauQuoted 2 lines · Taylor Blau
A subsequent patch will teach
want_object_in_pack()to set its*found_packand*found_offsetpoitners to NULL when the provided
s/poitners/pointers
Quoted 9 lines
pack does not pass the
is_pack_valid()check.The
--stdin-packsmode ofpack-objectsis not quite prepared to handle this. To prepare it for this change, do the following two things:- Ensure provided packs pass the
is_pack_valid()check when collecting the caller-provided packs into the "included" and "excluded" lists.
Is the 'is_pack_valid()' check happening for the "excluded" packs? It looks like you only added it for the packs in the "included" list in this patch.
Quoted 22 lines
- Gracefully handle any _invalid_ packs being passed to
`want_object_in_pack()`.Calling
is_pack_valid()early on makes it substantially less likely that we will have to deal with a pack going away, since we'll have an open file descriptor on its contents much earlier.But even packs with open descriptors can become invalid in the future if we (a) hit our open descriptor limit, forcing us to close some open packs, and (b) one of those just-closed packs has gone away in the meantime.
add_object_entry_from_pack()depends on having a non-NULL*found_pack, since it passes that pointer topacked_object_info(), meaning that we would SEGV if the pointer became NULL (like we propose to do inwant_object_in_pack()in the following patch).But avoiding calling
packed_object_info()entirely is OK, too, since its only purpose is to identify which objects in the included packs are commits, so that they can form the tips of the advisory traversal used to discover the object namehashes.
Quoted 5 lines
Failing to do this means that at worst we will produce lower-quality deltas, but it does not prevent us from generating the pack as long as we can find a copy of each object from the disappearing pack in some other part of the repository.
The rest of this makes sense and (as far as I can tell) lines up with the implementation below.
Quoted 69 lines
Co-authored-by Victoria Dye <[email protected]>Signed-off-by Taylor Blau <[email protected]>--- builtin/pack-objects.c | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-)builtin/pack-objects.c +20
-15@@ -3201,10 +3201,8 @@ static int add_object_entry_from_pack(const struct object_id *oid,uint32_t pos, void *_data) { - struct rev_info *revs = _data; -struct object_info oi =OBJECT_INFO_INIT ; off_t ofs; -enum object_type type; +enum object_type type =OBJ_NONE ;display_progress (progress_state, ++nr_seen);@@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,if (!want_object_in_pack (oid,0 , &p, &ofs))return 0 ; - oi.typep = &type; -if (packed_object_info (the_repository, p, ofs, &oi) <0 ) -die (_ ("could not get type of object %s in pack %s" ), -oid_to_hex (oid), p->pack_name); -else if (type ==OBJ_COMMIT ) { -/* -* commits in included packs are used as starting points for the -* subsequent revision walk -*/ -add_pending_oid (revs,NULL , oid,0 ); +if (p) { +struct rev_info *revs = _data; +struct object_info oi =OBJECT_INFO_INIT ; + + oi.typep = &type; +if (packed_object_info (the_repository, p, ofs, &oi) <0 ) { +die (_ ("could not get type of object %s in pack %s" ), +oid_to_hex (oid), p->pack_name); + }else if (type ==OBJ_COMMIT ) { +/* +* commits in included packs are used as starting points for the +* subsequent revision walk +*/ +add_pending_oid (revs,NULL , oid,0 ); + } + + stdin_packs_found_nr++; } - stdin_packs_found_nr++; -create_object_entry (oid, type,0 ,0 ,0 , p, ofs);return 0 ;@@ -3346,6 +3349,8 @@ static void read_packs_list_from_stdin(void)struct packed_git *p = item->util;if (!p)die (_ ("could not find pack '%s'" ), item->string); +if (!is_pack_valid (p)) +die (_ ("packfile %s cannot be accessed" ), p->pack_name); }/*
Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
Quoted 5 lines · Victoria Dye
Quoted 5 lines
pack does not pass the
is_pack_valid()check.The
--stdin-packsmode ofpack-objectsis not quite prepared to handle this. To prepare it for this change, do the following two things:
Quoted 7 lines
Quoted 4 lines
- Ensure provided packs pass the
is_pack_valid()check when collecting the caller-provided packs into the "included" and "excluded" lists.Is the '
is_pack_valid()' check happening for the "excluded" packs? It looks like you only added it for the packs in the "included" list in this patch.
You're right that we don't do it explicitly. That's OK, since we won't use any objects in excluded packs, and thus don't need to eagerly grab an descriptor on it to prevent against the race we're handling here.
(In practice, we do end up calling is_pack_valid() on excluded packs later on, via
- want_found_object() (or one of its many callers), which itself calls
- has_object_kept_pack(), which calls
- find_kept_pack_entry(), which calls
- fill_pack_entry(), which calls
- is_pack_valid(), which calls
but that's a side-effect that doesn't help or hurt us.)
We _do_ need to be able to open the .idx (which happens in fill_pack_entry() -> find_pack_entry_one() -> open_pack_index() , but we'll fail appropriately when the index cannot be located.
Quoted 2 lines
The rest of this makes sense and (as far as I can tell) lines up with the implementation below.
Thanks for taking a look!
Thanks,
Taylor
PATCH v2 4/4 builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
In reply to Taylor BlauWhen using a multi-pack bitmap, pack-objects will try to perform its traversal using a call to traverse_bitmap_commit_list(), which calls add_object_entry_from_bitmap() to add each object it finds to its packing list.
This path can cause pack-objects to add objects from packs that don't have open pack_fds on them, by avoiding a call to is_pack_valid(). This is because we only call is_pack_valid() on the preferred pack (in order to do verbatim reuse via reuse_partial_packfile_from_bitmap()) and not others when loading a MIDX bitmap.
In this case, add_object_entry_from_bitmap() will check whether it wants each object entry by calling want_object_in_pack(), which will call want_found_object (since its caller already supplied a found_pack). In most cases (particularly without --local, and when ignored_packed_keep_on_disk and ignored_packed_keep_in_core are both "0"), we'll take the entry from the pack contained in the MIDX bitmap, all without an open pack_fd.
When we then try to use that entry later to assemble the actual pack, we'll be susceptible to any simultaneous writers moving that pack out of the way (e.g., due to a concurrent repack) without having an open file descriptor, causing races that result in errors like:
remote: Enumerating objects: 1498802, done.
remote: fatal: packfile ./objects/pack/pack-e57d433b5a588daa37fbe946e2b28dfaec03a93e.pack cannot be accessed
remote: aborting due to possible repository corruption on the remote side.This race can happen even with multi-pack bitmaps, since we may open a MIDX bitmap that is being rewritten long before its packs are actually unlinked.
Work around this by calling is_pack_valid() from within want_found_object(), matching the behavior in want_object_in_pack_one() (which has an analogous call). Most calls to is_pack_valid() should be basically no-ops, since only the first call requires us to open a file (subsequent calls realize the file is already open, and return immediately).
Importantly, when want_object_in_pack() is given a non-NULL *found_pack, but want_found_object() rejects the copy of the object in that pack, we must reset *found_pack and *found_offset to NULL and 0, respectively. Failing to do so could lead to other checks in want_object_in_pack() (such as want_object_in_pack_one()) using the same (invalid) pack as *found_pack, meaning that we don't call is_pack_valid() because p == *found_pack. This can lead the caller to believe it can use a copy of an object from an invalid pack.
An alternative approach to closing this race would have been to call is_pack_valid() on _all_ packs in a multi-pack bitmap on load. This has a couple of problems:
- it is unnecessarily expensive in the cases where we don't actually need to open any packs (e.g., in git rev-list --use-bitmap-index --count)
- more importantly, it means any time we would have hit this race, we'll avoid using bitmaps altogether, leading to significant slowdowns by forcing a full object traversal
--- builtin/pack-objects.c | 6 ++++++ 1 file changed, 6 insertions(+)
builtin/pack-objects.c +6 -0
@@ -1357,6 +1357,9 @@ static int want_found_object(const struct object_id *oid, int exclude,if (incremental)return 0 ; +if (!is_pack_valid (p)) +return -1 ; +/* * When asked to do --local (do not include an object that appears in a * pack we borrow from elsewhere) or --honor-pack-keep (do not include
@@ -1472,6 +1475,9 @@ static int want_object_in_pack(const struct object_id *oid, want =want_found_object (oid, exclude, *found_pack);if (want !=-1 )return want; + + *found_pack =NULL ; + *found_offset =0 ; }for (m =get_multi_pack_index (the_repository); m; m = m->next) {
Signature · 1 line
2.36.1.94.gb0d54bedca
Quoted 1 line · Taylor Blau
verbatim reuse (c.f.,
write_reused_pack_verbatim()).
Unlike "e.g." and "i.e.", I think these should all be "cf." (there are many others).
Quoted 6 lines
+ This patch handles the "preferred" pack (c.f., the section
+ "multi-pack-index reverse indexes" in
+ Documentation/technical/pack-format.txt) specially, since pack-objects
+ depends on reusing exact chunks of that pack verbatim in
+ reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,
+ the utility of a bitmap is significantly diminished.It explains why we care about the "preferred" pack, which is a nice clarification. It hints that the other packs do not matter as much, and it is clearly stated that how they are handled is ...
Quoted 10 lines
+ Similar to dc1daacdcc, we could technically just add this check in + reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX + .bitmap without needing to open any of its packs. But it's simpler to do + the check as early as possible, covering all direct uses of the + preferred pack. Note that doing this check early requires us to call + prepare_midx_pack() early, too, so move the relevant part of that loop + from load_reverse_index() into open_midx_bitmap_1().+
+ Subsequent patches handle the non-preferred packs in a slightly
+ different fashion.
... left for later steps.
Excellent write-up.
Quoted 14 lines
Signed-off-by: Taylor Blau <[email protected]> @@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git, + + preferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)]; + if (!is_pack_valid(preferred)) { -+ close(fd); + warning(_("preferred pack (%s) is invalid"), + preferred->pack_name); + goto cleanup; 2: 9adf6e1341 < -: ---------- builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects -: ---------- > 2: 2719d33f32 builtin/pack-objects.c: avoid redundant NULL check -: ---------- > 3: cdc3265ec2 builtin/pack-objects.c: ensure included `--stdin-packs` exist -: ---------- > 4: 3fc3a95517 builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
Quoted 6 lines · Junio C Hamano
Quoted 1 line · Taylor Blau
verbatim reuse (c.f.,
write_reused_pack_verbatim()).Unlike "e.g." and "i.e.", I think these should all be "cf." (there are many others).
Oops. You learn something new everyday! ;-)
Quoted 22 lines
Quoted 6 lines
+ This patch handles the "preferred" pack (c.f., the section + "multi-pack-index reverse indexes" in + Documentation/technical/pack-format.txt) specially, since pack-objects + depends on reusing exact chunks of that pack verbatim in + reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded, + the utility of a bitmap is significantly diminished.It explains why we care about the "preferred" pack, which is a nice clarification. It hints that the other packs do not matter as much, and it is clearly stated that how they are handled is ...
Quoted 10 lines
+ Similar to dc1daacdcc, we could technically just add this check in + reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX + .bitmap without needing to open any of its packs. But it's simpler to do + the check as early as possible, covering all direct uses of the + preferred pack. Note that doing this check early requires us to call + prepare_midx_pack() early, too, so move the relevant part of that loop + from load_reverse_index() into open_midx_bitmap_1().+
+ Subsequent patches handle the non-preferred packs in a slightly
+ different fashion.
Quoted 3 lines
... left for later steps.
Excellent write-up.
Thanks very much!
Taylor