repack: incremental MIDX/bitmap-based repacking

Here is another small reroll of my series to implement the last remaining component of the incremental MIDX/bitmap-based repacking strategy [1].

The changes since v3 are fairly small:

- repack_prepare_midx_command() now calls its third parameter "subcommand" instead of "verb".

- clear_incremental_midx_files() now avoids a redundant conditional around r->objects and reuses a local struct odb_source * when closing any open incremental MIDX handles.

As usual, a range-diff is included below for convenience. Thanks in advance for reviewing!

[1]: https://lore.kernel.org/git/[email protected]/

Taylor Blau (16):
midx-write: handle noop writes when converting incremental chains
midx: use strset for retained MIDX files
midx: build keep_hashes array in order
midx: use strvec for keep_hashes
midx: introduce --no-write-chain-file for incremental MIDX writes
midx: support custom --base for incremental MIDX writes
repack: track the ODB source via existing_packs
midx: expose midx_layer_contains_pack()
repack-midx: factor out repack_prepare_midx_command()
repack-midx: extract repack_fill_midx_stdin_packs()
repack-geometry: prepare for incremental MIDX repacking builtin/repack.c: convert --write-midx to an OPT_CALLBACK
packfile: ensure close_pack_revindex() frees in-memory revindex
repack: implement incremental MIDX repacking
repack: introduce --write-midx=incremental
repack: allow --write-midx=incremental without --geometric

 Documentation/config/repack.adoc        |  18 +
 Documentation/git-multi-pack-index.adoc |  32 +-
 Documentation/git-repack.adoc           |  44 +-
 builtin/multi-pack-index.c              |  48 +-
 builtin/repack.c                        | 102 +++-
 midx-write.c                            | 206 ++++---
 midx.c                                  | 102 ++--
 midx.h                                  |  11 +-
 packfile.c                              |   2 +
 repack-geometry.c                       |  48 +-
 repack-midx.c                           | 710 +++++++++++++++++++++++-
 repack.c                                |  58 +-
 repack.h                                |  26 +-
 t/meson.build                           |   1 +
 t/t5334-incremental-multi-pack-index.sh |  63 +++
 t/t5335-compact-multi-pack-index.sh     | 113 ++++
 t/t7705-repack-incremental-midx.sh      | 525 ++++++++++++++++++
 17 files changed, 1909 insertions(+), 200 deletions(-)
 create mode 100755 t/t7705-repack-incremental-midx.sh

Range-diff against v3:

 1:  d6c27317c25 =  1:  ead11e610c8 midx-write: handle noop writes when converting incremental chains
 2:  629c8d23116 =  2:  ece55bf2957 midx: use `strset` for retained MIDX files
 3:  e303bf6a4ac =  3:  5609d1941e6 midx: build `keep_hashes` array in order
 4:  42d76c70060 =  4:  13b7c808860 midx: use `strvec` for `keep_hashes`
 5:  2c80aa34fac =  5:  cac3fd54bf0 midx: introduce `--no-write-chain-file` for incremental MIDX writes
 6:  2a05f4b86f3 =  6:  1bbb387d6b6 midx: support custom `--base` for incremental MIDX writes
 7:  92aba3d366f =  7:  4a93adb3ad3 repack: track the ODB source via existing_packs
 8:  d3ac65c1f11 =  8:  8d1b8b1d301 midx: expose `midx_layer_contains_pack()`
 9:  1bd2f194c6f !  9:  42111e5f75d repack-midx: factor out `repack_prepare_midx_command()`

@@ Commit message
subcommands), so extract the common portions of the command setup into a reusable repack_prepare_midx_command() helper.

- The extracted helper sets git_cmd, pushes the multi-pack-index
- subcommand and verb, and handles --progress/--no-progress and
- --bitmap flags. The remaining arguments that are specific to the
- write subcommand (such as --stdin-packs) are left to the caller.
+ The extracted helper sets git_cmd, pushes multi-pack-index and a
+ subcommand, and handles --progress/--no-progress and --bitmap
+ flags. The remaining arguments that are specific to the write
+ subcommand (such as --stdin-packs) are left to the caller.

No functional changes are included in this patch.

@@ repack-midx.c: static void remove_redundant_bitmaps(struct string_list *include,

+static void repack_prepare_midx_command(struct child_process *cmd,
+ struct repack_write_midx_opts *opts,
-+ const char *verb)
++ const char *subcommand)
+{
+ cmd->git_cmd = 1;
+
-+ strvec_pushl(&cmd->args, "multi-pack-index", verb, NULL);
++ strvec_pushl(&cmd->args, "multi-pack-index", subcommand, NULL);
+
+ if (opts->show_progress)
+ strvec_push(&cmd->args, "--progress");

10:  2a87a1e4561 = 10:  ed76e6efd1c repack-midx: extract `repack_fill_midx_stdin_packs()`
11:  3d32b9c88da = 11:  9665f1b3a64 repack-geometry: prepare for incremental MIDX repacking
12:  1f7a5479bb8 = 12:  e0db62b9f10 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`
13:  b155f25d53c = 13:  c8c846b1ac1 packfile: ensure `close_pack_revindex()` frees in-memory revindex
14:  ef012314930 = 14:  78f1a98c1ea repack: implement incremental MIDX repacking
15:  04cfecd5136 ! 15:  95bf8b21fff repack: introduce `--write-midx=incremental`
    @@ midx.c: void clear_midx_file(struct repository *r)
     +void clear_incremental_midx_files(struct repository *r,
     +				  const struct strvec *keep_hashes)
     +{
    ++	struct odb_source *source = r->objects->sources;
     +	struct strbuf chain = STRBUF_INIT;
     +
    -+	get_midx_chain_filename(r->objects->sources, &chain);
    ++	get_midx_chain_filename(source, &chain);
     +
    -+	if (r->objects) {
    -+		struct odb_source *source = r->objects->sources;
    -+		for (source = r->objects->sources; source; source = source->next) {
    -+			struct odb_source_files *files = odb_source_files_downcast(source);
    -+			if (files->packed->midx)
    -+				close_midx(files->packed->midx);
    -+			files->packed->midx = NULL;
    -+		}
    ++	for (; source; source = source->next) {
    ++		struct odb_source_files *files = odb_source_files_downcast(source);
    ++		if (files->packed->midx)
    ++			close_midx(files->packed->midx);
    ++		files->packed->midx = NULL;
     +	}
     +
     +	if (!keep_hashes && remove_path(chain.buf))

16: 1c05dfce579 = 16: 8bd0ec98dc3 repack: allow --write-midx=incremental without --geometric

base-commit: 7bcaabddcf68bd0702697da5904c3b68c52f94cf

2.54.0.175.g8bd0ec98dc3