Skip to content

pack-bitmap-write: speed up bitmap generation

Taylor Blau git 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest8 patches 1 replyIn review
  2. v18 patches 19 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8pack-bitmap: pass object position to `fill_bitmap_tree()`pack-bitmap-write.c+15 -80 replies
2/8pack-bitmap: check subtree bits before recursingpack-bitmap-write.c+17 -60 replies
3/8pack-bitmap: reuse stored selected bitmapspack-bitmap-write.c+34 -00 replies
4/8pack-bitmap: consolidate `find_object_pos()` success pathpack-bitmap-write.c+9 -90 replies
5/8pack-bitmap: cache object positions during fillpack-bitmap-write.c pack-bitmap.h+94 -10 replies
6/8pack-bitmap: sort bitmaps before XORingpack-bitmap-write.c+29 -00 replies
7/8pack-bitmap: remember pseudo-merge parentspack-bitmap-write.c+17 -130 replies
8/8pack-bitmap: build pseudo-merge bitmaps after regular bitmapspack-bitmap-write.c+174 -360 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 0/8] pack-bitmap-write: speed up bitmap generation

Here is a reroll of my series to improve the performance of reachability bitmap generation, focusing on very large repositories and the penalty to generate pseudo-merge reachability bitmaps.

The series is largely unchanged since last time. Notable changes in this round include:

- minor refactoring in the pair of patches which consolidate the find_object_pos() success path and introduce the object position cache during bitmap fills, and

- dropping a stale paragraph from the final patch's message, which described follow-up commits that are no longer part of this series.

As usual, a range-diff against v1 is included below for convenience.

Thanks in advance for your review!

Taylor Blau (8):
pack-bitmap: pass object position to fill_bitmap_tree()
pack-bitmap: check subtree bits before recursing
pack-bitmap: reuse stored selected bitmaps
pack-bitmap: consolidate find_object_pos() success path
pack-bitmap: cache object positions during fill
pack-bitmap: sort bitmaps before XORing
pack-bitmap: remember pseudo-merge parents
pack-bitmap: build pseudo-merge bitmaps after regular bitmaps

 pack-bitmap-write.c | 431 +++++++++++++++++++++++++++++++++++++-------
 pack-bitmap.h       |   7 +
 2 files changed, 377 insertions(+), 61 deletions(-)

Range-diff against v1:

1:  13191c19b91 = 1:  ad025810ab3 pack-bitmap: pass object position to `fill_bitmap_tree()`
2:  7d6d1cec0dd = 2:  59da63d0330 pack-bitmap: check subtree bits before recursing
3:  6e1f6bef5f6 = 3:  f13d65c0ad9 pack-bitmap: reuse stored selected bitmaps
4:  c9a56066094 ! 4:  856aa3a6ab7 pack-bitmap: consolidate `find_object_pos()` success path

@@ Commit message

## pack-bitmap-write.c ##
+@@ pack-bitmap-write.c: static uint32_t find_object_pos(struct bitmap_writer *writer,
+ const struct object_id *oid, int *found)
+ {
+ struct object_entry *entry;
++ uint32_t pos;
+
+ entry = packlist_find(writer->to_pack, oid);
+ if (entry) {
@@ pack-bitmap-write.c: static uint32_t find_object_pos(struct bitmap_writer *writer,
if (writer->midx)
base_objects = writer->midx->num_objects +
5: e43ef6a42d1 ! 5: 70dfa80d543 pack-bitmap: cache object positions during fill

    @@ pack-bitmap-write.c: void bitmap_writer_push_commit(struct bitmap_writer *writer
      				const struct object_id *oid, int *found)
      {
      	struct object_entry *entry;
    -+	uint32_t pos;
    -+
    + 	uint32_t pos;
    + 
     +	bitmap_writer_init_pos_cache(writer);
     +
     +	if (find_cached_object_pos(writer, oid, &pos)) {
    @@ pack-bitmap-write.c: void bitmap_writer_push_commit(struct bitmap_writer *writer
     +			*found = 1;
     +		return pos;
     +	}
    - 
    ++
      	entry = packlist_find(writer->to_pack, oid);
      	if (entry) {
      		uint32_t base_objects = 0;
6:  b0a4f31353a = 6:  b1184792d23 pack-bitmap: sort bitmaps before XORing
7:  0bd88e6a096 = 7:  673b6262911 pack-bitmap: remember pseudo-merge parents
8:  30ce254312c ! 8:  8722242f1bb pack-bitmap: build pseudo-merge bitmaps after regular bitmaps

@@ Commit message
portion of history reachable by one or more pseudo-merge(s), but not by any non-pseudo-merge commit selected for bitmapping.

- Now that we have decoupled how we generate pseudo-merges from their
- representation, the following commits will improve the API around
- specifying pseudo-merge groupings during bitmap generation.
-

## pack-bitmap-write.c ##

base-commit: c3d7ca7d982efc3a848fd85f34e867cfc0a99479

Signature · 1 line
2.54.0.rc1.84.g1cf18622df7