Skip to contentSkip to conversation

midx-write.c: do not optimize out writes with corrupt MIDXs

PATCH In review Taylor Blau 2 participants 5 messages Jan 12 – Jan 13

lore
Share & tools
lore
#0 lore raw

cover midx-write.c: do not optimize out writes with corrupt MIDXs

These two patches came from my work on implementing incremental MIDX layer compaction.

When rebasing on top of current 'master' (at the
time of writing, 8745eae506f (The 17th batch, 2026-01-11)), I noticed an early 'test_done' in t5319 added by 6ce9d558ced (midx-write: skip rewriting MIDX with --stdin-packs unless needed, 2025-12-10). The series is structured as follows:

- The first patch removes the extraneous 'test_done', which exposes a failing test which is marked as such.

- The second patch explains and fixes the bug, un-marking the test as test_expect_failure.

I was originally planning on adding these onto my series under tb/incremental-midx-part-3.2. But I opted to split these patches out into their own topic to ensure they are picked up before v2.53.0 is tagged, should the larger series not be ready by then.

Thanks in advance for your review!

Taylor Blau (2):
t/t5319-multi-pack-index.sh: drop early 'test_done' midx-write.c: assume checksum-invalid MIDXs require an update

 midx-write.c                | 14 ++++++++++++++
 t/t5319-multi-pack-index.sh |  2 --
 2 files changed, 14 insertions(+), 2 deletions(-)

base-commit: 8745eae506f700657882b9e32b2aa00f234a6fb6

Signature · 1 line
2.52.0.437.gcc6f76a88cd
#1 lore raw

PATCH 1/2 t/t5319-multi-pack-index.sh: drop early 'test_done'

In 6ce9d558ced (midx-write: skip rewriting MIDX with --stdin-packs unless needed, 2025-12-10), an extra 'test_done' was added, causing the test script to finish before having run all of its tests.

Dropping this extraneous 'test_done' exposes a bug from commit 6ce9d558ced that causes a subsequent test to fail. Mark that test with a 'test_expect_failure' for now, and the subsequent commit will explain and fix the bug.

---
 t/t5319-multi-pack-index.sh | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
t/t5319-multi-pack-index.sh +1 -3
index 794f8b5ab4e..b6622849db7 100755
@@ -415,8 +415,6 @@ test_expect_success 'up-to-date multi-pack-index is retained' '
 	)
 '
 
-test_done
-
 test_expect_success 'verify multi-pack-index success' '
 	git multi-pack-index verify --object-dir=$objdir
 '
@@ -565,7 +563,7 @@ test_expect_success 'git fsck suppresses MIDX output with --no-progress' '
 	! grep "Verifying object offsets" err
 '
 
-test_expect_success 'corrupt MIDX is not reused' '
+test_expect_failure 'corrupt MIDX is not reused' '
 	corrupt_midx_and_verify $MIDX_BYTE_OFFSET "\377" $objdir \
 		"incorrect object offset" &&
 	git multi-pack-index write 2>err &&
Signature · 1 line
2.52.0.437.gcc6f76a88cd
#3 lore raw

PATCH 2/2 midx-write.c: assume checksum-invalid MIDXs require an update

In reply to Taylor Blau

In 6ce9d558ced (midx-write: skip rewriting MIDX with --stdin-packs unless needed, 2025-12-10), the MIDX machinery learned how to optimize out unnecessary writes with "--stdin-packs".

In order to do this, it compares the contents of the in-progress write against a MIDX loaded directly from the object store. We load a separate MIDX (as opposed to checking our update relative to "ctx.m") because the MIDX code does not reuse an existing MIDX with --stdin-packs, and always leaves "ctx.m" as NULL. See commit 0c5a62f14bc (midx-write.c: do not read existing MIDX with packs_to_include, 2024-06-11) for details on why.

If "ctx.m" is non-NULL, however, it is guaranteed to be checksum-valid, since we only assign "ctx.m" when "midx_checksum_valid()" returns true. Since the same guard does not exist for the MIDX we pass to "midx_needs_update()", we may ignore on-disk corruption when determining whether or not we can optimize out the write.

Add a similar guard within "midx_needs_update()" to prevent such an issue.

A more robust fix would involve revising 0c5a62f14bc and teaching the MIDX generation code how to reuse an existing MIDX even when invoked with "--stdin-packs", such that we could avoid side-loading the MIDX directly from the object store in order to call "midx_needs_update()". For now, pursue the minimal fix.

---
 midx-write.c                | 14 ++++++++++++++
 t/t5319-multi-pack-index.sh |  2 +-
 2 files changed, 15 insertions(+), 1 deletion(-)
midx-write.c +14 -0
index 87b97c70872..6485cb67068 100644
@@ -1011,6 +1011,20 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c
 	struct strbuf buf = STRBUF_INIT;
 	bool needed = true;
 
+	/*
+	 * Ensure that we have a valid checksum before consulting the
+	 * exisiting MIDX in order to determine if we can avoid an
+	 * update.
+	 *
+	 * This is necessary because the given MIDX is loaded directly
+	 * from the object store (because we still compare our proposed
+	 * update to any on-disk MIDX regardless of whether or not we
+	 * have assigned "ctx.m") and is thus not guaranteed to have a
+	 * valid checksum.
+	 */
+	if (!midx_checksum_valid(midx))
+		goto out;
+
 	/*
 	 * Ignore incremental updates for now. The assumption is that any
 	 * incremental update would be either empty (in which case we will bail
t/t5319-multi-pack-index.sh +1 -1
index b6622849db7..faae98c7e76 100755
@@ -563,7 +563,7 @@ test_expect_success 'git fsck suppresses MIDX output with --no-progress' '
 	! grep "Verifying object offsets" err
 '
 
-test_expect_failure 'corrupt MIDX is not reused' '
+test_expect_success 'corrupt MIDX is not reused' '
 	corrupt_midx_and_verify $MIDX_BYTE_OFFSET "\377" $objdir \
 		"incorrect object offset" &&
 	git multi-pack-index write 2>err &&
Signature · 1 line
2.52.0.437.gcc6f76a88cd
#2 lore raw

Re: [PATCH 1/2] t/t5319-multi-pack-index.sh: drop early 'test_done'

In reply to Taylor Blau

t/t5319-multi-pack-index.sh test_expect_success 'up-to-date multi-pack-index is retained' ' · line 420

Quoted 13 lines · Taylor Blau
t/t5319-multi-pack-index.sh +0 -2
index 794f8b5ab4e..b6622849db7 100755
@@ -415,8 +415,6 @@ test_expect_success 'up-to-date multi-pack-index is retained' '
 	)
 '
 
-test_done
-
 test_expect_success 'verify multi-pack-index success' '
 	git multi-pack-index verify --object-dir=$objdir
 '

Oh dear, this is embarassing.

Patrick

#4 lore raw

Re: [PATCH 2/2] midx-write.c: assume checksum-invalid MIDXs require an update

In reply to Taylor Blau
Quoted 15 lines · Taylor Blau
midx-write.c +7 -0
index 87b97c70872..6485cb67068 100644
@@ -1011,6 +1011,20 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c
 	struct strbuf buf = STRBUF_INIT;
 	bool needed = true;
 
+	/*
+	 * Ensure that we have a valid checksum before consulting the
+	 * exisiting MIDX in order to determine if we can avoid an
+	 * update.
+	 *
+	 * This is necessary because the given MIDX is loaded directly
+	 * from the object store (because we still compare our proposed

midx-write.c static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c · line 1030

Quoted 10 lines
midx-write.c +7 -0
+	 * update to any on-disk MIDX regardless of whether or not we
+	 * have assigned "ctx.m") and is thus not guaranteed to have a
+	 * valid checksum.
+	 */
+	if (!midx_checksum_valid(midx))
+		goto out;
+
 	/*
 	 * Ignore incremental updates for now. The assumption is that any
 	 * incremental update would be either empty (in which case we will bail

This looks sensible to me, thanks for the fix!

Patrick