Skip to content

pack-bitmap: convert offset to ref deltas where possible

Taylor Blau git 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest11 patches 19 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11pack-bitmap.c: do not pass `pack_pos` to `try_partial_reuse()`pack-bitmap.c+9 -120 replies
2/11pack-bitmap.c: avoid unnecessary `offset_to_pack_pos()`pack-bitmap.c+2 -30 replies
3/11pack-bitmap.c: delay calling 'offset_to_pack_pos()'pack-bitmap.c+3 -60 replies
4/11pack-bitmap.c: compare `base_offset` to `delta_obj_offset`pack-bitmap.c+1 -10 replies
5/11pack-bitmap.c: extract `find_base_bitmap_pos()`pack-bitmap.c+49 -410 replies
6/11pack-bitmap: drop `from_midx` field from `bitmapped_pack`builtin/pack-objects.c midx.c pack-bitmap.c +1 more+8 -50 replies
7/11write_reused_pack_one(): translate bit positions directlybuiltin/pack-objects.c+41 -372 replies
8/11t5332: enable OFS_DELTAs via test_pack_objects_reusedbuiltin/pack-objects.c t/t5332-multi-pack-reuse.sh+3 -12 replies
9/11pack-bitmap: enable cross-pack delta reusebuiltin/pack-objects.c pack-bitmap.c pack-bitmap.h +1 more+75 -222 replies
10/11pack-bitmap.c: record whether the result was filteredpack-bitmap.c+12 -02 replies
11/11pack-bitmap: enable reusing deltas with base objects in 'haves' bitmapbuiltin/pack-objects.c pack-bitmap.c pack-bitmap.h +1 more+107 -170 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 00/11] pack-bitmap: convert offset to ref deltas where possible

This patch series enables more objects to be candidates for verbatim reuse when generating a pack with the aide of reachability bitmaps.

By the end of this series, two new classes of objects are now reuse candidates which were not before. They are:

- Cross-pack deltas. In multi-pack bitmaps, if the delta and base were selected from different packs, the delta was not reusable.

- Thin deltas. In both single- and multi-pack bitmaps, we did not consider reusing deltas whose base object appears in the 'haves' bitmap.

This series teaches the pack-objects machinery to convert cross-pack and thin deltas to reference deltas so that they may be reused.

The series started off as a couple of patches, each addressing one of the two classes from the above. But as I started refining these patches, the series grew longer, as there were a handful of opportunities to clean up some of the existing pack-reuse bitmap code.

The series is organized as follows:

- The first seven patches are various cleanups that make the pack-reuse code more readable and prepare us to enable delta reuse in more situation as above.

pack-bitmap.c: do not pass pack_pos to try_partial_reuse() pack-bitmap.c: avoid unnecessary offset_to_pack_pos() pack-bitmap.c: delay calling 'offset_to_pack_pos()' pack-bitmap.c: compare base_offset to delta_obj_offset pack-bitmap.c: extract find_base_bitmap_pos()
pack-bitmap: drop from_midx field from bitmapped_pack write_reused_pack_one(): translate bit positions directly

- The next patch is a test fixup:

t5332: enable OFS_DELTAs via test_pack_objects_reused

- The next patch after that enables us to reuse deltas whose base appears in another pack:

pack-bitmap: enable cross-pack delta reuse

- The final two patches enable the "thin deltas" optimization above:

pack-bitmap.c: record whether the result was filtered
pack-bitmap: enable reusing deltas with base objects in 'haves' bitmap

I think that the first seven patches are worthwhile on their own. I considered splitting those out into their own series, but decided against it to so that we could realize their benefit more readily.

The first optimization (cross-pack deltas) should help clones and fetches, but the second optimization (thin deltas) will only help fetches, since the 'haves' bitmap will necessarily be empty for clones.

Of course, REF_DELTAs have a less compact representation than OFS_DELTAs, so the resulting packs will trade off some CPU time for a slightly larger pack. But we're only talking about a handful of extra bytes, and network bandwidth is pretty cheap, so I think the optimization is worthwhile here too.

Thanks in advance for your review!

Taylor Blau (11):
pack-bitmap.c: do not pass pack_pos to try_partial_reuse() pack-bitmap.c: avoid unnecessary offset_to_pack_pos() pack-bitmap.c: delay calling 'offset_to_pack_pos()' pack-bitmap.c: compare base_offset to delta_obj_offset pack-bitmap.c: extract find_base_bitmap_pos()
pack-bitmap: drop from_midx field from bitmapped_pack write_reused_pack_one(): translate bit positions directly
t5332: enable OFS_DELTAs via test_pack_objects_reused
pack-bitmap: enable cross-pack delta reuse
pack-bitmap.c: record whether the result was filtered
pack-bitmap: enable reusing deltas with base objects in 'haves' bitmap

 builtin/pack-objects.c      | 110 ++++++++++++--------
 midx.c                      |   1 -
 pack-bitmap.c               | 198 ++++++++++++++++++++++++------------
 pack-bitmap.h               |   6 +-
 t/t5332-multi-pack-reuse.sh |  72 ++++++++++++-
 5 files changed, 275 insertions(+), 112 deletions(-)

base-commit: 777489f9e09c8d0dd6b12f9d90de6376330577a2

Signature · 1 line
2.47.0.11.g487258bca34