Skip to content

multi-pack reachability bitmaps

Taylor Blau git 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest27 patches 9 repliesIn review
  2. v425 patches 37 repliesSuperseded
  3. v325 patches 15 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v224 patches75 repliesSuperseded
v122 patches9 repliesSuperseded

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/27pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmapspack-bitmap.c+48 -00 replies
2/27pack-bitmap-write.c: gracefully fail to write non-closed bitmapsbuiltin/pack-objects.c pack-bitmap-write.c pack-bitmap.h +1 more+64 -260 replies
3/27pack-bitmap-write.c: free existing bitmapspack-bitmap-write.c+1 -00 replies
4/27Documentation: describe MIDX-based bitmapsDocumentation/technical/bitmap-format.txt Documentation/technical/multi-pack-index.txt+60 -210 replies
5/27midx: disallow running outside of a repositorygit.c t/t5319-multi-pack-index.sh+6 -10 replies
6/27midx: fix `*.rev` cleanups with `--object-dir`midx.c t/t5319-multi-pack-index.sh+33 -50 replies
7/27midx: clear auxiliary .rev after replacing the MIDXmidx.c+2 -10 replies
8/27midx: reject empty `--preferred-pack`'sDocumentation/git-multi-pack-index.txt midx.c t/t5319-multi-pack-index.sh+49 -30 replies
9/27midx: infer preferred pack when not given onemidx.c+44 -60 replies
10/27midx: close linked MIDXs, avoid leaking memorymidx.c+3 -00 replies
11/27midx: avoid opening multiple MIDXs when writingDocumentation/git-multi-pack-index.txt midx.c+17 -110 replies
12/27pack-bitmap.c: introduce 'bitmap_num_objects()'pack-bitmap.c+21 -160 replies
13/27pack-bitmap.c: introduce 'nth_bitmap_object_oid()'pack-bitmap.c+10 -30 replies
14/27pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'pack-bitmap.c pack-bitmap.h+17 -00 replies
15/27pack-bitmap.c: avoid redundant calls to try_partial_reusepack-bitmap.c+29 -110 replies
16/27pack-bitmap: read multi-pack bitmapsbuiltin/pack-objects.c midx.c midx.h +4 more+336 -420 replies
17/27pack-bitmap: write multi-pack bitmapsDocumentation/git-multi-pack-index.txt builtin/multi-pack-index.c midx.c +1 more+215 -90 replies
18/27t5310: move some tests to lib-bitmap.sht/lib-bitmap.sh t/t5310-pack-bitmaps.sh+240 -2230 replies
19/27t/helper/test-read-midx.c: add --checksum modet/helper/test-read-midx.c t/lib-bitmap.sh+19 -10 replies
20/27t5326: test multi-pack bitmap behaviort/t5326-multi-pack-bitmaps.sh+286 -00 replies
21/27t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAPt/t0410-partial-clone.sh+3 -00 replies
22/27t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAPt/t5310-pack-bitmaps.sh+4 -00 replies
23/27t5319: don't write MIDX bitmaps in t5319t/t5319-multi-pack-index.sh+2 -10 replies
24/27t7700: update to work with MIDX bitmap test knobt/t7700-repack.sh+12 -60 replies
25/27midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'builtin/repack.c ci/run-build-and-tests.sh midx.h +1 more+17 -20 replies
26/27p5310: extract full and partial bitmap testst/perf/lib-bitmap.sh t/perf/p5310-pack-bitmaps.sh+72 -620 replies
27/27p5326: perf tests for MIDX bitmapst/perf/p5326-multi-pack-bitmaps.sh+43 -00 replies

Reviews trailers per patch

None of the 27 patches has a Reviewed-by or Acked-by.

PatchJeff KingJohannes Berg
1/27Co-authored-by
2/27
3/27
4/27
5/27
6/27Original-patch-by
7/27
8/27
9/27
10/27
11/27
12/27
13/27
14/27
15/27
16/27
17/27
18/27
19/27
20/27
21/27
22/27
23/27
24/27
25/27
26/27
27/27

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v5 thread

Cover letter [PATCH v5 00/27] multi-pack reachability bitmaps

Here is another version of the multi-pack reachability bitmaps series. It is virtually unchanged since last time.

The changes that did occur is that I integrated Johannes' patch from [1] to fix cleaning up MIDX .rev and .bitmap files when using --object-dir. That inspired a lengthy discussion [2] about --object-dir, alternates, object-format and running the MIDX builtin outside of a Git repository.

This series resolves that discussion by leaving everything as-is, and only changing the following:

- git multi-pack-index will not run when outside of a Git repository.

- The --object-dir argument will only recognize object directories belonging to an alternate of the current repository.

- Using --object-dir to point to a repository which uses a different hash than the repository in the current working directory will continue to not work (as was the case before this series).

And because this incorporates [1], we will also not accidentally clean .rev files from the wrong object directory.

I think that this version is ready-to-go, and that we can turn our attention to squashing some of these cross-alternate buglets, and integrating MIDX bitmaps with git repack.

[1]: https://lore.kernel.org/git/[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/

Jeff King (2):
t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

Taylor Blau (25):
pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps pack-bitmap-write.c: gracefully fail to write non-closed bitmaps pack-bitmap-write.c: free existing bitmaps
Documentation: describe MIDX-based bitmaps
midx: disallow running outside of a repository
midx: fix *.rev cleanups with --object-dir
midx: clear auxiliary .rev after replacing the MIDX
midx: reject empty --preferred-pack's
midx: infer preferred pack when not given one
midx: close linked MIDXs, avoid leaking memory
midx: avoid opening multiple MIDXs when writing pack-bitmap.c: introduce 'bitmap_num_objects()' pack-bitmap.c: introduce 'nth_bitmap_object_oid()' pack-bitmap.c: introduce 'bitmap_is_preferred_refname()' pack-bitmap.c: avoid redundant calls to try_partial_reuse
pack-bitmap: read multi-pack bitmaps
pack-bitmap: write multi-pack bitmaps
t5310: move some tests to lib-bitmap.sh t/helper/test-read-midx.c: add --checksum mode
t5326: test multi-pack bitmap behavior
t5319: don't write MIDX bitmaps in t5319
t7700: update to work with MIDX bitmap test knob
midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'
p5310: extract full and partial bitmap tests
p5326: perf tests for MIDX bitmaps

 Documentation/git-multi-pack-index.txt       |  20 +-
 Documentation/technical/bitmap-format.txt    |  71 ++-
 Documentation/technical/multi-pack-index.txt |  10 +-
 builtin/multi-pack-index.c                   |   2 +
 builtin/pack-objects.c                       |   8 +-
 builtin/repack.c                             |  12 +-
 ci/run-build-and-tests.sh                    |   1 +
 git.c                                        |   2 +-
 midx.c                                       | 328 ++++++++++--
 midx.h                                       |   5 +
 pack-bitmap-write.c                          |  79 ++-
 pack-bitmap.c                                | 499 ++++++++++++++++---
 pack-bitmap.h                                |   9 +-
 packfile.c                                   |   2 +-
 t/README                                     |   4 +
 t/helper/test-read-midx.c                    |  16 +-
 t/lib-bitmap.sh                              | 240 +++++++++
 t/perf/lib-bitmap.sh                         |  69 +++
 t/perf/p5310-pack-bitmaps.sh                 |  65 +--
 t/perf/p5326-multi-pack-bitmaps.sh           |  43 ++
 t/t0410-partial-clone.sh                     |  12 +-
 t/t5310-pack-bitmaps.sh                      | 231 +--------
 t/t5319-multi-pack-index.sh                  |  53 +-
 t/t5326-multi-pack-bitmaps.sh                | 286 +++++++++++
 t/t7700-repack.sh                            |  18 +-
 25 files changed, 1644 insertions(+), 441 deletions(-)
 create mode 100644 t/perf/lib-bitmap.sh
 create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh
 create mode 100755 t/t5326-multi-pack-bitmaps.sh

Range-diff against v4:

 1:  92dc0bbc0d =  1:  7815d9929d pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps
 2:  979276bc74 =  2:  629171115a pack-bitmap-write.c: gracefully fail to write non-closed bitmaps
 3:  8f00493955 =  3:  d469c1d8f6 pack-bitmap-write.c: free existing bitmaps
 4:  bc7db926d8 =  4:  158ff797c4 Documentation: describe MIDX-based bitmaps
 -:  ---------- >  5:  5f24be8985 midx: disallow running outside of a repository
 -:  ---------- >  6:  0aacaa9283 midx: fix `*.rev` cleanups with `--object-dir`
 5:  771741844b !  7:  d30e6fe9a5 midx: clear auxiliary .rev after replacing the MIDX
    @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
      
      	if (flags & MIDX_WRITE_REV_INDEX)
      		write_midx_reverse_index(midx_name, midx_hash, &ctx);
    --	clear_midx_files_ext(the_repository, ".rev", midx_hash);
    +-	clear_midx_files_ext(object_dir, ".rev", midx_hash);
      
      	commit_lock_file(&lk);
      
    -+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
    ++	clear_midx_files_ext(object_dir, ".rev", midx_hash);
     +
      cleanup:
      	for (i = 0; i < ctx.nr; i++) {
 6:  dab5dbf228 =  8:  db2a24a8ae midx: reject empty `--preferred-pack`'s
 7:  31f4517de0 =  9:  059c583e34 midx: infer preferred pack when not given one
 8:  aa3bd96d9b = 10:  6f5ca446f3 midx: close linked MIDXs, avoid leaking memory
 9:  c9fea31fa8 ! 11:  4656608f73 midx: avoid opening multiple MIDXs when writing

@@ Commit message

+ ## Documentation/git-multi-pack-index.txt ##
+@@ Documentation/git-multi-pack-index.txt: OPTIONS
+ Use given directory for the location of Git objects. We check
+ <dir>/packs/multi-pack-index for the current MIDX file, and
+ <dir>/packs for the pack-files to index.
+++
++<dir> must be an alternate of the current repository. +
+ --[no-]progress::
+ Turn progress on/off explicitly. If neither is specified, progress is
+
## midx.c ##
@@ midx.c: static int midx_checksum_valid(struct multi_pack_index *m)
return hashfile_checksum_valid(m->data, m->data_len);

10:  ee72fb7e38 = 12:  4c793df9d1 pack-bitmap.c: introduce 'bitmap_num_objects()'
11:  ede0bf1ce1 = 13:  9f165037ce pack-bitmap.c: introduce 'nth_bitmap_object_oid()'
12:  df6844def0 = 14:  ba5fd71fb3 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'
13:  4e06f051a7 = 15:  06db8dbbc1 pack-bitmap.c: avoid redundant calls to try_partial_reuse
14:  a0d73eb3d3 = 16:  61798853b6 pack-bitmap: read multi-pack bitmaps
15:  9d83ad77ab ! 17:  4968229663 pack-bitmap: write multi-pack bitmaps
    @@ midx.c: static int write_midx_internal(const char *object_dir,
     +			 * corresponding bitmap (or one wasn't requested).
     +			 */
     +			if (!want_bitmap)
    -+				clear_midx_files_ext(the_repository, ".bitmap",
    ++				clear_midx_files_ext(object_dir, ".bitmap",
     +						     NULL);
     +			goto cleanup;
     +		}
    @@ midx.c: static int write_midx_internal(const char *object_dir,
     +		}
     +	}
     +
    -+	close_object_store(the_repository->objects);
    ++	if (ctx.m)
    ++		close_object_store(the_repository->objects);
      
      	commit_lock_file(&lk);
      
    -+	clear_midx_files_ext(the_repository, ".bitmap", midx_hash);
    - 	clear_midx_files_ext(the_repository, ".rev", midx_hash);
    ++	clear_midx_files_ext(object_dir, ".bitmap", midx_hash);
    + 	clear_midx_files_ext(object_dir, ".rev", midx_hash);
      
      cleanup:
     @@ midx.c: static int write_midx_internal(const char *object_dir,
    @@ midx.c: void clear_midx_file(struct repository *r)
      	if (remove_path(midx))
      		die(_("failed to clear multi-pack-index at %s"), midx);
      
    -+	clear_midx_files_ext(r, ".bitmap", NULL);
    - 	clear_midx_files_ext(r, ".rev", NULL);
    ++	clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL);
    + 	clear_midx_files_ext(r->objects->odb->path, ".rev", NULL);
      
      	free(midx);
16:  a92af89884 = 18:  5d60b07e2e t5310: move some tests to lib-bitmap.sh
17:  d47aa4a919 = 19:  1a9c3538db t/helper/test-read-midx.c: add --checksum mode
18:  9d9d9f28a6 = 20:  8895114ace t5326: test multi-pack bitmap behavior
19:  3e0da7e5ed = 21:  94b1317e0c t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
20:  4e0d49a2dd = 22:  a4f4d90bba t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
21:  47eba8ecf9 = 23:  92a6370e77 t5319: don't write MIDX bitmaps in t5319
22:  3d78afa2ad = 24:  c49dc46fb2 t7700: update to work with MIDX bitmap test knob
23:  c2f94e033d = 25:  44a4800756 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'
24:  6b03016c99 = 26:  bf0981b606 p5310: extract full and partial bitmap tests
25:  d98faa4c2c = 27:  6888fe01aa p5326: perf tests for MIDX bitmaps
Signature · 1 line
2.33.0.96.g73915697e6