multi-pack reachability bitmaps
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v5 Latest27 patches 9 repliesIn review
- v425 patches 37 repliesSuperseded
- v325 patches 15 repliesSuperseded
Earlier versions · 2
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v2 | 24 patches | 75 replies | Superseded | |
| v1 | 22 patches | 9 replies | Superseded |
Patches in v5 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/27 | pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmapspack-bitmap.c | +48 | 0 replies | |
| 2/27 | pack-bitmap-write.c: gracefully fail to write non-closed bitmapsbuiltin/ pack-bitmap-write.c pack-bitmap.h +1 more | +64 | 0 replies | |
| 3/27 | pack-bitmap-write.c: free existing bitmapspack-bitmap-write.c | +1 | 0 replies | |
| 4/27 | Documentation: describe MIDX-based bitmapsDocumentation/ Documentation/ | +60 | 0 replies | |
| 5/27 | midx: disallow running outside of a repositorygit.c t/ | +6 | 0 replies | |
| 6/27 | midx: fix `*.rev` cleanups with `--object-dir`midx.c t/ | +33 | 0 replies | |
| 7/27 | midx: clear auxiliary .rev after replacing the MIDXmidx.c | +2 | 0 replies | |
| 8/27 | midx: reject empty `--preferred-pack`'sDocumentation/ midx.c t/ | +49 | 0 replies | |
| 9/27 | midx: infer preferred pack when not given onemidx.c | +44 | 0 replies | |
| 10/27 | midx: close linked MIDXs, avoid leaking memorymidx.c | +3 | 0 replies | |
| 11/27 | midx: avoid opening multiple MIDXs when writingDocumentation/ midx.c | +17 | 0 replies | |
| 12/27 | pack-bitmap.c: introduce 'bitmap_num_objects()'pack-bitmap.c | +21 | 0 replies | |
| 13/27 | pack-bitmap.c: introduce 'nth_bitmap_object_oid()'pack-bitmap.c | +10 | 0 replies | |
| 14/27 | pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'pack-bitmap.c pack-bitmap.h | +17 | 0 replies | |
| 15/27 | pack-bitmap.c: avoid redundant calls to try_partial_reusepack-bitmap.c | +29 | 0 replies | |
| 16/27 | pack-bitmap: read multi-pack bitmapsbuiltin/ midx.c midx.h +4 more | +336 | 0 replies | |
| 17/27 | pack-bitmap: write multi-pack bitmapsDocumentation/ builtin/ midx.c +1 more | +215 | 0 replies | |
| 18/27 | t5310: move some tests to lib-bitmap.sht/ t/ | +240 | 0 replies | |
| 19/27 | t/helper/test-read-midx.c: add --checksum modet/ t/ | +19 | 0 replies | |
| 20/27 | t5326: test multi-pack bitmap behaviort/ | +286 | 0 replies | |
| 21/27 | t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAPt/ | +3 | 0 replies | |
| 22/27 | t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAPt/ | +4 | 0 replies | |
| 23/27 | t5319: don't write MIDX bitmaps in t5319t/ | +2 | 0 replies | |
| 24/27 | t7700: update to work with MIDX bitmap test knobt/ | +12 | 0 replies | |
| 25/27 | midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'builtin/ ci/ midx.h +1 more | +17 | 0 replies | |
| 26/27 | p5310: extract full and partial bitmap testst/ t/ | +72 | 0 replies | |
| 27/27 | p5326: perf tests for MIDX bitmapst/ | +43 | 0 replies |
Reviews trailers per patch
None of the 27 patches has a Reviewed-by or Acked-by.
| Patch | Jeff King | Johannes Berg |
|---|---|---|
| 1/27 | Co-authored-by | |
| 2/27 | ||
| 3/27 | ||
| 4/27 | ||
| 5/27 | ||
| 6/27 | Original-patch-by | |
| 7/27 | ||
| 8/27 | ||
| 9/27 | ||
| 10/27 | ||
| 11/27 | ||
| 12/27 | ||
| 13/27 | ||
| 14/27 | ||
| 15/27 | ||
| 16/27 | ||
| 17/27 | ||
| 18/27 | ||
| 19/27 | ||
| 20/27 | ||
| 21/27 | ||
| 22/27 | ||
| 23/27 | ||
| 24/27 | ||
| 25/27 | ||
| 26/27 | ||
| 27/27 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH v5 00/27] multi-pack reachability bitmaps
Here is another version of the multi-pack reachability bitmaps series. It is virtually unchanged since last time.
The changes that did occur is that I integrated Johannes' patch from [1] to fix cleaning up MIDX .rev and .bitmap files when using --object-dir. That inspired a lengthy discussion [2] about --object-dir, alternates, object-format and running the MIDX builtin outside of a Git repository.
This series resolves that discussion by leaving everything as-is, and only changing the following:
- git multi-pack-index will not run when outside of a Git repository.
- The --object-dir argument will only recognize object directories belonging to an alternate of the current repository.
- Using --object-dir to point to a repository which uses a different hash than the repository in the current working directory will continue to not work (as was the case before this series).
And because this incorporates [1], we will also not accidentally clean .rev files from the wrong object directory.
I think that this version is ready-to-go, and that we can turn our attention to squashing some of these cross-alternate buglets, and integrating MIDX bitmaps with git repack.
[1]: https://lore.kernel.org/git/[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/
Jeff King (2):
t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
Taylor Blau (25):
pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps pack-bitmap-write.c: gracefully fail to write non-closed bitmaps pack-bitmap-write.c: free existing bitmaps
Documentation: describe MIDX-based bitmaps
midx: disallow running outside of a repository
midx: fix *.rev cleanups with --object-dir
midx: clear auxiliary .rev after replacing the MIDX
midx: reject empty --preferred-pack's
midx: infer preferred pack when not given one
midx: close linked MIDXs, avoid leaking memory
midx: avoid opening multiple MIDXs when writing pack-bitmap.c: introduce 'bitmap_num_objects()' pack-bitmap.c: introduce 'nth_bitmap_object_oid()' pack-bitmap.c: introduce 'bitmap_is_preferred_refname()' pack-bitmap.c: avoid redundant calls to try_partial_reuse
pack-bitmap: read multi-pack bitmaps
pack-bitmap: write multi-pack bitmaps
t5310: move some tests to lib-bitmap.sh t/helper/test-read-midx.c: add --checksum mode
t5326: test multi-pack bitmap behavior
t5319: don't write MIDX bitmaps in t5319
t7700: update to work with MIDX bitmap test knob
midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'
p5310: extract full and partial bitmap tests
p5326: perf tests for MIDX bitmaps
Documentation/git-multi-pack-index.txt | 20 +- Documentation/technical/bitmap-format.txt | 71 ++- Documentation/technical/multi-pack-index.txt | 10 +- builtin/multi-pack-index.c | 2 + builtin/pack-objects.c | 8 +- builtin/repack.c | 12 +- ci/run-build-and-tests.sh | 1 + git.c | 2 +- midx.c | 328 ++++++++++-- midx.h | 5 + pack-bitmap-write.c | 79 ++- pack-bitmap.c | 499 ++++++++++++++++--- pack-bitmap.h | 9 +- packfile.c | 2 +- t/README | 4 + t/helper/test-read-midx.c | 16 +- t/lib-bitmap.sh | 240 +++++++++ t/perf/lib-bitmap.sh | 69 +++ t/perf/p5310-pack-bitmaps.sh | 65 +-- t/perf/p5326-multi-pack-bitmaps.sh | 43 ++ t/t0410-partial-clone.sh | 12 +- t/t5310-pack-bitmaps.sh | 231 +-------- t/t5319-multi-pack-index.sh | 53 +- t/t5326-multi-pack-bitmaps.sh | 286 +++++++++++ t/t7700-repack.sh | 18 +- 25 files changed, 1644 insertions(+), 441 deletions(-) create mode 100644 t/perf/lib-bitmap.sh create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh create mode 100755 t/t5326-multi-pack-bitmaps.sh
Range-diff against v4:
1: 92dc0bbc0d = 1: 7815d9929d pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps
2: 979276bc74 = 2: 629171115a pack-bitmap-write.c: gracefully fail to write non-closed bitmaps
3: 8f00493955 = 3: d469c1d8f6 pack-bitmap-write.c: free existing bitmaps
4: bc7db926d8 = 4: 158ff797c4 Documentation: describe MIDX-based bitmaps
-: ---------- > 5: 5f24be8985 midx: disallow running outside of a repository
-: ---------- > 6: 0aacaa9283 midx: fix `*.rev` cleanups with `--object-dir`
5: 771741844b ! 7: d30e6fe9a5 midx: clear auxiliary .rev after replacing the MIDX
@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
if (flags & MIDX_WRITE_REV_INDEX)
write_midx_reverse_index(midx_name, midx_hash, &ctx);
-- clear_midx_files_ext(the_repository, ".rev", midx_hash);
+- clear_midx_files_ext(object_dir, ".rev", midx_hash);
commit_lock_file(&lk);
-+ clear_midx_files_ext(the_repository, ".rev", midx_hash);
++ clear_midx_files_ext(object_dir, ".rev", midx_hash);
+
cleanup:
for (i = 0; i < ctx.nr; i++) {
6: dab5dbf228 = 8: db2a24a8ae midx: reject empty `--preferred-pack`'s
7: 31f4517de0 = 9: 059c583e34 midx: infer preferred pack when not given one
8: aa3bd96d9b = 10: 6f5ca446f3 midx: close linked MIDXs, avoid leaking memory
9: c9fea31fa8 ! 11: 4656608f73 midx: avoid opening multiple MIDXs when writing@@ Commit message
+ ## Documentation/git-multi-pack-index.txt ##
+@@ Documentation/git-multi-pack-index.txt: OPTIONS
+ Use given directory for the location of Git objects. We check
+ <dir>/packs/multi-pack-index for the current MIDX file, and
+ <dir>/packs for the pack-files to index.
+++
++<dir> must be an alternate of the current repository. +
+ --[no-]progress::
+ Turn progress on/off explicitly. If neither is specified, progress is
+
## midx.c ##
@@ midx.c: static int midx_checksum_valid(struct multi_pack_index *m)
return hashfile_checksum_valid(m->data, m->data_len);
10: ee72fb7e38 = 12: 4c793df9d1 pack-bitmap.c: introduce 'bitmap_num_objects()'
11: ede0bf1ce1 = 13: 9f165037ce pack-bitmap.c: introduce 'nth_bitmap_object_oid()'
12: df6844def0 = 14: ba5fd71fb3 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'
13: 4e06f051a7 = 15: 06db8dbbc1 pack-bitmap.c: avoid redundant calls to try_partial_reuse
14: a0d73eb3d3 = 16: 61798853b6 pack-bitmap: read multi-pack bitmaps
15: 9d83ad77ab ! 17: 4968229663 pack-bitmap: write multi-pack bitmaps
@@ midx.c: static int write_midx_internal(const char *object_dir,
+ * corresponding bitmap (or one wasn't requested).
+ */
+ if (!want_bitmap)
-+ clear_midx_files_ext(the_repository, ".bitmap",
++ clear_midx_files_ext(object_dir, ".bitmap",
+ NULL);
+ goto cleanup;
+ }
@@ midx.c: static int write_midx_internal(const char *object_dir,
+ }
+ }
+
-+ close_object_store(the_repository->objects);
++ if (ctx.m)
++ close_object_store(the_repository->objects);
commit_lock_file(&lk);
-+ clear_midx_files_ext(the_repository, ".bitmap", midx_hash);
- clear_midx_files_ext(the_repository, ".rev", midx_hash);
++ clear_midx_files_ext(object_dir, ".bitmap", midx_hash);
+ clear_midx_files_ext(object_dir, ".rev", midx_hash);
cleanup:
@@ midx.c: static int write_midx_internal(const char *object_dir,
@@ midx.c: void clear_midx_file(struct repository *r)
if (remove_path(midx))
die(_("failed to clear multi-pack-index at %s"), midx);
-+ clear_midx_files_ext(r, ".bitmap", NULL);
- clear_midx_files_ext(r, ".rev", NULL);
++ clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL);
+ clear_midx_files_ext(r->objects->odb->path, ".rev", NULL);
free(midx);16: a92af89884 = 18: 5d60b07e2e t5310: move some tests to lib-bitmap.sh 17: d47aa4a919 = 19: 1a9c3538db t/helper/test-read-midx.c: add --checksum mode 18: 9d9d9f28a6 = 20: 8895114ace t5326: test multi-pack bitmap behavior 19: 3e0da7e5ed = 21: 94b1317e0c t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 20: 4e0d49a2dd = 22: a4f4d90bba t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 21: 47eba8ecf9 = 23: 92a6370e77 t5319: don't write MIDX bitmaps in t5319 22: 3d78afa2ad = 24: c49dc46fb2 t7700: update to work with MIDX bitmap test knob 23: c2f94e033d = 25: 44a4800756 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' 24: 6b03016c99 = 26: bf0981b606 p5310: extract full and partial bitmap tests 25: d98faa4c2c = 27: 6888fe01aa p5326: perf tests for MIDX bitmaps
Signature · 1 line
2.33.0.96.g73915697e6