Skip to contentSkip to conversation

midx: implement a multi-pack reverse index

PATCH Superseded Taylor Blau 8 participants 171 messages Feb 10, 2021 – Sep 18, 2021 Documentation

lore
Share & tools
lore
#0 lore raw

cover midx: implement a multi-pack reverse index

This series describes and implements a reverse index for the multi-pack index, based on a "pseudo-pack" which can be uniquely described by the multi-pack index.

The details of the pseudo-pack, and multi-pack reverse index are laid out in detail in the sixth patch.

This is in support of multi-pack reachability bitmaps, which contain objects from the multi-pack index. Likewise, an object's bit position in a multi-pack reachability bitmap is determined by its position with that multi-pack index's pseudo pack.

In this series, there are no users of the multi-pack index, so this series is mainly about laying the groundwork for implementing multi-pack bitmaps. This series is the final prerequisite needed before we can implement multi-pack bitmaps, which will come in the next series[1].

Since tb/pack-revindex-on-disk is queued to be merged to 'master', but hasn't yet been merged, this series is based on that branch.

Thanks in advance for your review of this series, and all of the many other series in support of multi-pack bitmaps.

[1]: If you're curious, you can find the patches in the tb/multi-pack-bitmaps branch of my fork at https://github.com/ttaylorr/git.

Taylor Blau (9):
t/helper/test-read-midx.c: add '--show-objects'
midx: allow marking a pack as preferred
midx: don't free midx_name early
midx: keep track of the checksum
midx: make some functions non-static Documentation/technical: describe multi-pack reverse indexes
pack-revindex: read multi-pack reverse indexes pack-write.c: extract 'write_rev_file_order'
pack-revindex: write multi-pack reverse indexes

 Documentation/git-multi-pack-index.txt       |  11 +-
 Documentation/technical/multi-pack-index.txt |   5 +-
 Documentation/technical/pack-format.txt      |  83 +++++++
 builtin/multi-pack-index.c                   |  10 +-
 builtin/repack.c                             |   2 +-
 midx.c                                       | 239 ++++++++++++++++++-
 midx.h                                       |  11 +-
 pack-revindex.c                              | 112 +++++++++
 pack-revindex.h                              |  46 ++++
 pack-write.c                                 |  39 ++-
 pack.h                                       |   1 +
 packfile.c                                   |   3 +
 t/helper/test-read-midx.c                    |  24 +-
 t/t5319-multi-pack-index.sh                  |  39 +++
 14 files changed, 591 insertions(+), 34 deletions(-)
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#1 lore raw

PATCH 1/9 t/helper/test-read-midx.c: add '--show-objects'

The 'read-midx' helper is used in places like t5319 to display basic information about a multi-pack-index.

In the next patch, the MIDX writing machinery will learn a new way to choose from which pack an object is selected when multiple copies of that object exist.

To disambiguate which pack introduces an object so that this feature can be tested, add a '--show-objects' option which displays additional information about each object in the MIDX.

---
 t/helper/test-read-midx.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)
t/helper/test-read-midx.c +20 -4
index 2430880f78..4ec12f77a0 100644
@@ -4,7 +4,7 @@
 #include "repository.h"
 #include "object-store.h"
 
-static int read_midx_file(const char *object_dir)
+static int read_midx_file(const char *object_dir, int show_objects)
 {
 	uint32_t i;
 	struct multi_pack_index *m;
@@ -15,6 +15,20 @@ static int read_midx_file(const char *object_dir)
 	if (!m)
 		return 1;
 
+	if (show_objects) {
+		struct object_id oid;
+		struct pack_entry e;
+
+		for (i = 0; i < m->num_objects; i++) {
+			nth_midxed_object_oid(&oid, m, i);
+			fill_midx_entry(the_repository, &oid, &e, m);
+
+			printf("%s %"PRIu64"\t%s\n",
+			       oid_to_hex(&oid), e.offset, e.p->pack_name);
+		}
+		return 0;
+	}
+
 	printf("header: %08x %d %d %d %d\n",
 	       m->signature,
 	       m->version,
@@ -48,8 +62,10 @@ static int read_midx_file(const char *object_dir)
 
 int cmd__read_midx(int argc, const char **argv)
 {
-	if (argc != 2)
-		usage("read-midx <object-dir>");
+	if (!(argc == 2 || argc == 3))
+		usage("read-midx [--show-objects] <object-dir>");
 
-	return read_midx_file(argv[1]);
+	if (!strcmp(argv[1], "--show-objects"))
+		return read_midx_file(argv[2], 1);
+	return read_midx_file(argv[1], 0);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#2 lore raw

Re: [PATCH 1/9] t/helper/test-read-midx.c: add '--show-objects'

Quoted 7 lines · Taylor Blau
t/helper/test-read-midx.c +7 -0
+	if (show_objects) {
+		struct object_id oid;
+		struct pack_entry e;
+
+		for (i = 0; i < m->num_objects; i++) {
+			nth_midxed_object_oid(&oid, m, i);
+			fill_midx_entry(the_repository, &oid, &e, m);

t/helper/test-read-midx.c static int read_midx_file(const char *object_dir) · line 34

Quoted 10 lines
t/helper/test-read-midx.c +7 -0
+
+			printf("%s %"PRIu64"\t%s\n",
+			       oid_to_hex(&oid), e.offset, e.p->pack_name);
+		}
+		return 0;
+	}
+
 	printf("header: %08x %d %d %d %d\n",
 	       m->signature,
 	       m->version,

It seems a little odd to me that the list of objects happens after the header information. Probably doesn't matter in your test cases, but I sometimes use the test helpers to diagnose data during development and could see piping this output into 'less' and wanting the header at the top.

Thanks,
-Stolee

#3 lore raw

Re: [PATCH 1/9] t/helper/test-read-midx.c: add '--show-objects'

Quoted 16 lines · Derrick Stolee
Quoted 15 lines · Taylor Blau
t/helper/test-read-midx.c +14 -0
+	if (show_objects) {
+		struct object_id oid;
+		struct pack_entry e;
+
+		for (i = 0; i < m->num_objects; i++) {
+			nth_midxed_object_oid(&oid, m, i);
+			fill_midx_entry(the_repository, &oid, &e, m);
+
+			printf("%s %"PRIu64"\t%s\n",
+			       oid_to_hex(&oid), e.offset, e.p->pack_name);
+		}
+		return 0;
+	}
+
 	printf("header: %08x %d %d %d %d\n",
Quoted 8 lines

t/helper/test-read-midx.c static int read_midx_file(const char *object_dir) · line 34

Quoted 2 lines
t/helper/test-read-midx.c +0 -0
 	       m->signature,
 	       m->version,

It seems a little odd to me that the list of objects happens after the header information. Probably doesn't matter in your test cases, but I sometimes use the test helpers to diagnose data during development and could see piping this output into 'less' and wanting the header at the top.

Indeed. In theory you could pipe to tail instead (or to less and immediately hit 'G'), but I can't think of a good reason that this would have appeared above the header when I originally wrote the patch.

Anyway, it doesn't seem that the tests care about where this is (they're just looking for for a line that begins with the object id and ends with its offset), so I think this could probably be moved without thinking too hard about it.

Quoted 2 lines

Thanks,
-Stolee

Thanks,
Taylor

#4 lore raw

PATCH 2/9 midx: allow marking a pack as preferred

In reply to Taylor Blau

When multiple packs in the multi-pack index contain the same object, the MIDX machinery must make a choice about which pack it associates with that object. Prior to this patch, the lowest-ordered[1] pack was always selected.

Pack selection for duplicate objects is relatively unimportant today, but it will become important for multi-pack bitmaps. This is because we can only invoke the pack-reuse mechanism when all of the bits for reused objects come from the reuse pack (in order to ensure that all reused deltas can find their base objects in the same pack).

To encourage the pack selection process to prefer one pack over another (the pack to be preferred is the one a caller would like to later use as a reuse pack), introduce the concept of a "preferred pack". When provided, the MIDX code will always prefer an object found in a preferred pack over any other.

No format changes are required to store the preferred pack, since it will be able to be inferred with a corresponding MIDX bitmap, by looking up the pack associated with the object in the first bit position (this ordering is described in detail in a subsequent commit).

[1]: the ordering is specified by MIDX internals; for our purposes we can consider the "lowest ordered" pack to be "the one with the most-recent mtime.

---
 Documentation/git-multi-pack-index.txt       | 11 ++-
 Documentation/technical/multi-pack-index.txt |  5 +-
 builtin/multi-pack-index.c                   | 10 +-
 builtin/repack.c                             |  2 +-
 midx.c                                       | 97 ++++++++++++++++++--
 midx.h                                       |  2 +-
 t/t5319-multi-pack-index.sh                  | 39 ++++++++
 7 files changed, 151 insertions(+), 15 deletions(-)
Documentation/git-multi-pack-index.txt +10 -1
index eb0caa0439..dd14eab781 100644
@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes
 SYNOPSIS
 --------
 [verse]
-'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress] <subcommand>
+'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
+	[--preferred-pack=<pack>] <subcommand>
 
 DESCRIPTION
 -----------
@@ -27,6 +28,14 @@ OPTIONS
 	Turn progress on/off explicitly. If neither is specified, progress is
 	shown if standard error is connected to a terminal.
 
+--preferred-pack=<pack>::
+	When using the `write` subcommand, optionally specify the
+	tie-breaking pack used when multiple packs contain the same
+	object. Incompatible with other subcommands, including `repack`,
+	which may repack the pack marked as preferred. If not given, the
+	preferred pack is inferred from an existing `multi-pack-index`,
+	if one exists, otherwise the pack with the lowest mtime.
+
 The following subcommands are available:
 
 write::
Documentation/technical/multi-pack-index.txt +3 -2
index e8e377a59f..fb688976c4 100644
@@ -43,8 +43,9 @@ Design Details
   a change in format.
 
 - The MIDX keeps only one record per object ID. If an object appears
-  in multiple packfiles, then the MIDX selects the copy in the most-
-  recently modified packfile.
+  in multiple packfiles, then the MIDX selects the copy in the
+  preferred packfile, otherwise selecting from the most-recently
+  modified packfile.
 
 - If there exist packfiles in the pack directory not registered in
   the MIDX, then those packfiles are loaded into the `packed_git`
builtin/multi-pack-index.c +9 -1
index 5bf88cd2a8..4d1ea3fe84 100644
@@ -4,6 +4,7 @@
 #include "parse-options.h"
 #include "midx.h"
 #include "trace2.h"
+#include "object-store.h"
 
 static char const * const builtin_multi_pack_index_usage[] = {
 	N_("git multi-pack-index [<options>] (write|verify|expire|repack --batch-size=<size>)"),
@@ -12,6 +13,7 @@ static char const * const builtin_multi_pack_index_usage[] = {
 
 static struct opts_multi_pack_index {
 	const char *object_dir;
+	const char *preferred_pack;
 	unsigned long batch_size;
 	int progress;
 } opts;
@@ -24,6 +26,8 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
+		OPT_STRING(0, "preferred-pack", &opts.preferred_pack, N_("preferred-pack"),
+		  N_("pack for reuse when computing a multi-pack bitmap")),
 		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
@@ -51,6 +55,9 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		return 1;
 	}
 
+	if (strcmp(argv[0], "write") && opts.preferred_pack)
+		die(_("'--preferred-pack' requires 'write'"));
+
 	trace2_cmd_mode(argv[0]);
 
 	if (!strcmp(argv[0], "repack"))
@@ -60,7 +67,8 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		die(_("--batch-size option is only for 'repack' subcommand"));
 
 	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, flags);
+		return write_midx_file(opts.object_dir, opts.preferred_pack,
+				       flags);
 	if (!strcmp(argv[0], "verify"))
 		return verify_midx_file(the_repository, opts.object_dir, flags);
 	if (!strcmp(argv[0], "expire"))
builtin/repack.c +1 -1
index 01440de2d5..9f00806805 100644
@@ -523,7 +523,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 	remove_temporary_files();
 
 	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
-		write_midx_file(get_object_directory(), 0);
+		write_midx_file(get_object_directory(), NULL, 0);
 
 	string_list_clear(&names, 0);
 	string_list_clear(&rollback, 0);
midx.c +88 -9

194 lines not on this page · raw message

midx.h +1 -1
index b18cf53bc4..e7fea61109 100644
@@ -47,7 +47,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa
 int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);
 int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);
 
-int write_midx_file(const char *object_dir, unsigned flags);
+int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);
 void clear_midx_file(struct repository *r);
 int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);
 int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);
t/t5319-multi-pack-index.sh +39 -0
index 2fc3aadbd1..9304b33484 100755
@@ -31,6 +31,14 @@ midx_read_expect () {
 	test_cmp expect actual
 }
 
+midx_expect_object_offset () {
+	OID="$1"
+	OFFSET="$2"
+	OBJECT_DIR="$3"
+	test-tool read-midx --show-objects $OBJECT_DIR >actual &&
+	grep "^$OID $OFFSET" actual
+}
+
 test_expect_success 'setup' '
 	test_oid_cache <<-EOF
 	idxoff sha1:2999
@@ -234,6 +242,37 @@ test_expect_success 'warn on improper hash version' '
 	)
 '
 
+test_expect_success 'midx picks objects from preferred pack' '
+	test_when_finished rm -rf preferred.git &&
+	git init --bare preferred.git &&
+	(
+		cd preferred.git &&
+
+		a=$(echo "a" | git hash-object -w --stdin) &&
+		b=$(echo "b" | git hash-object -w --stdin) &&
+		c=$(echo "c" | git hash-object -w --stdin) &&
+
+		# Set up two packs, duplicating the object "B" at different
+		# offsets.
+		git pack-objects objects/pack/test-AB <<-EOF &&
+		$a
+		$b
+		EOF
+		bc=$(git pack-objects objects/pack/test-BC <<-EOF
+		$b
+		$c
+		EOF
+		) &&
+
+		git multi-pack-index --object-dir=objects \
+			--preferred-pack=test-BC-$bc.idx write 2>err &&
+		test_must_be_empty err &&
+
+		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |
+			cut -d" " -f1) &&
+		midx_expect_object_offset $b $ofs objects
+	)
+'
 
 test_expect_success 'verify multi-pack-index success' '
 	git multi-pack-index verify --object-dir=$objdir
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#5 lore raw

Re: [PATCH 2/9] midx: allow marking a pack as preferred

Quoted 15 lines · Taylor Blau
Documentation/git-multi-pack-index.txt +2 -1
index eb0caa0439..dd14eab781 100644
@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes
 SYNOPSIS
 --------
 [verse]
-'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress] <subcommand>
+'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
+	[--preferred-pack=<pack>] <subcommand>
 
 DESCRIPTION
 -----------

Documentation/git-multi-pack-index.txt OPTIONS · line 34

Quoted 7 lines
Documentation/git-multi-pack-index.txt +4 -0
@@ -27,6 +28,14 @@ OPTIONS
 	Turn progress on/off explicitly. If neither is specified, progress is
 	shown if standard error is connected to a terminal.
 
+--preferred-pack=<pack>::
+	When using the `write` subcommand, optionally specify the
+	tie-breaking pack used when multiple packs contain the same
+	object. Incompatible with other subcommands, including `repack`,

I think this shouldn't be an option of the 'git multi-pack-index' command but an option of its 'write' subcommand.

#6 lore raw

Re: [PATCH 2/9] midx: allow marking a pack as preferred

Quoted 7 lines · SZEDER Gábor

Documentation/git-multi-pack-index.txt OPTIONS · line 34

Quoted 4 lines

+--preferred-pack=<pack>::

Documentation/git-multi-pack-index.txt +3 -0
+	When using the `write` subcommand, optionally specify the
+	tie-breaking pack used when multiple packs contain the same
+	object. Incompatible with other subcommands, including `repack`,

I think this shouldn't be an option of the 'git multi-pack-index' command but an option of its 'write' subcommand.

:-/. I wrote a lengthy response on Friday, but Gmail must have eaten it.

The gist of my response was that the intermingling of sub-commands with options from other sub-commands goes deeper than just the documentation, since command-line arguments are only parsed once in builtin/multi-pack-index.c.

I explored whether or not it would be worth it to parse the common options first, and then have separate options for each of the sub-commands (as is done in the commit-graph builtin). But, this is tricky, since we accept common options on either side of the sub-command (i.e., we'd expect both 'git multi-pack-index --object-dir=... write' to behave the same as 'git multi-pack-index write --object-dir=...').

So you could let the first call to parse_options() parse all of the arguments, but then specialized arguments (e.g., 'repack --batch-size') would cause the parse-options API to barf because the first call to parse_options() doesn't recognize '--batch-size'.

I think the easiest way to do it would be to pass PARSE_OPT_STOP_AT_NON_OPTION, and then let the subsequent calls to parse_options() pass an array of option structs that also includes the common options so they can be parsed on either side of the sub-command.

Obviously this leads to a lot of rather unfortunate duplication. So, I'm content to leave it all as-is, and let the multi-pack-index builtin check the disallowed combinations itself (e.g., if you passed '--preferred-pack' but aren't in 'write' mode, then we should complain).

I can certainly move this piece of documentation into the 'write' section, though, which should alleviate your immediate concern.

Thanks,
Taylor

#7 lore raw

Re: [PATCH 2/9] midx: allow marking a pack as preferred

Quoted 33 lines · Taylor Blau
Quoted 7 lines · SZEDER Gábor

Documentation/git-multi-pack-index.txt OPTIONS · line 34

Quoted 4 lines

+--preferred-pack=<pack>::

Documentation/git-multi-pack-index.txt +3 -0
+	When using the `write` subcommand, optionally specify the
+	tie-breaking pack used when multiple packs contain the same
+	object. Incompatible with other subcommands, including `repack`,

I think this shouldn't be an option of the 'git multi-pack-index' command but an option of its 'write' subcommand.

:-/. I wrote a lengthy response on Friday, but Gmail must have eaten it.

The gist of my response was that the intermingling of sub-commands with options from other sub-commands goes deeper than just the documentation, since command-line arguments are only parsed once in builtin/multi-pack-index.c.

I explored whether or not it would be worth it to parse the common options first, and then have separate options for each of the sub-commands (as is done in the commit-graph builtin). But, this is tricky, since we accept common options on either side of the sub-command (i.e., we'd expect both 'git multi-pack-index --object-dir=... write' to behave the same as 'git multi-pack-index write --object-dir=...').

So you could let the first call to parse_options() parse all of the arguments, but then specialized arguments (e.g., 'repack --batch-size') would cause the parse-options API to barf because the first call to parse_options() doesn't recognize '--batch-size'.

I think the easiest way to do it would be to pass PARSE_OPT_STOP_AT_NON_OPTION, and then let the subsequent calls to parse_options() pass an array of option structs that also includes the common options so they can be parsed on either side of the sub-command.

Quoted 7 lines

Obviously this leads to a lot of rather unfortunate duplication. So, I'm content to leave it all as-is, and let the multi-pack-index builtin check the disallowed combinations itself (e.g., if you passed '--preferred-pack' but aren't in 'write' mode, then we should complain).

I can certainly move this piece of documentation into the 'write' section, though, which should alleviate your immediate concern.

I may be missing something, but...

It sounds to me like you're imagining this is more complex than it is because you don't know about some/all of parse_options_concat() or PARSE_OPT_KEEP_*.

See e.g. cmd_{switch,restore} in builti/checkout.c, or the entire family of diff-like commands where we do parse_options() followed by setup_revisions(). We've got a lot of commands that parse options in a piecemeal manner.

At no point do you need to re-parse the options. You just have the common command parse as far as it gets, and leave anything else in argv/argc for sub-commands like "write".

I think the problem is you read the builtin/commit-graph.c code, it could really benefit from using parse_options_concat(), now things like "object-directory" are copy/pasted in that file. See 2087182272 (checkout: split options[] array in three pieces, 2019-03-29) for a commit which simplified that sort of code.

In this case you'd share the "opts_multi_pack_index" struct between the various commands, it would just have unused fields for "write" that aren't used by "verify" or whatever.

The PARSE_OPT_STOP_AT_NON_OPTION flag isn't for what you're doing with "write" here, since as your test shows you're doing:

git multi-pack-index <ALL_OPTS> <SUBCOMMAND>

But PARSE_OPT_STOP_AT_NON_OPTION is for cases like "git-remote" that do:

git multi-pack-index <COMMOT_OPTS> <SUBCOMMAND> <SUBCOMMAND_OPTS>

(Or maybe you really want the latter, and the test change isn't representative).

So then we want to stop at the first non-option, i.e. the subcommand. I think it's good practice not to emulate how "git remote" works for new commands, which makes things a bit simpler to implement.

You say "since we accept common options on either side of the sub-command" but without PARSE_OPT_STOP_AT_NON_OPTION this works, since if you can parse everything you'll have "write" left, but if you truly have unknown options you'll have more than that in argv.

All of the above shouldn't be taken as a "your patch should change" comment, maybe it's fine as-is.

I just replied because it sounded like you didn't spot how to easily use parse_options() to do this sort of thing. It's actually rather easy.

#8 lore raw

cover commit-graph: parse_options() cleanup

A trivial cleanup series as a follow-up to my comments in
https://lore.kernel.org/git/[email protected]/

Ævar Arnfjörð Bjarmason (5):
commit-graph: define common usage with a macro
commit-graph: remove redundant handling of -h
commit-graph: use parse_options_concat()
commit-graph: refactor dispatch loop for style
commit-graph: show usage on "commit-graph [write|verify] garbage"

 builtin/commit-graph.c  | 102 ++++++++++++++++++++++------------------
 t/t5318-commit-graph.sh |   7 +++
 2 files changed, 62 insertions(+), 47 deletions(-)
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
#9 lore raw

PATCH 1/5 commit-graph: define common usage with a macro

In reply to Ævar Arnfjörð Bjarmason

Share the usage message between these three variables by using a macro. Before this new options needed to copy/paste the usage information, see e.g. 809e0327f5 (builtin/commit-graph.c: introduce '--max-new-filters=<n>', 2020-09-18).

---
 builtin/commit-graph.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)
builtin/commit-graph.c +15 -14
index cd86315221..c3fa4fde3e 100644
@@ -9,26 +9,27 @@
 #include "progress.h"
 #include "tag.h"
 
-static char const * const builtin_commit_graph_usage[] = {
-	N_("git commit-graph verify [--object-dir <objdir>] [--shallow] [--[no-]progress]"),
-	N_("git commit-graph write [--object-dir <objdir>] [--append] "
-	   "[--split[=<strategy>]] [--reachable|--stdin-packs|--stdin-commits] "
-	   "[--changed-paths] [--[no-]max-new-filters <n>] [--[no-]progress] "
-	   "<split options>"),
+static const char * builtin_commit_graph_verify_usage[] = {
+#define BUILTIN_COMMIT_GRAPH_VERIFY_USAGE \
+	N_("git commit-graph verify [--object-dir <objdir>] [--shallow] [--[no-]progress]")
+	BUILTIN_COMMIT_GRAPH_VERIFY_USAGE,
 	NULL
 };
 
-static const char * const builtin_commit_graph_verify_usage[] = {
-	N_("git commit-graph verify [--object-dir <objdir>] [--shallow] [--[no-]progress]"),
+static const char * builtin_commit_graph_write_usage[] = {
+#define BUILTIN_COMMIT_GRAPH_WRITE_USAGE \
+	N_("git commit-graph write [--object-dir <objdir>] [--append] " \
+	   "[--split[=<strategy>]] [--reachable|--stdin-packs|--stdin-commits] " \
+	   "[--changed-paths] [--[no-]max-new-filters <n>] [--[no-]progress] " \
+	   "<split options>")
+	BUILTIN_COMMIT_GRAPH_WRITE_USAGE,
 	NULL
 };
 
-static const char * const builtin_commit_graph_write_usage[] = {
-	N_("git commit-graph write [--object-dir <objdir>] [--append] "
-	   "[--split[=<strategy>]] [--reachable|--stdin-packs|--stdin-commits] "
-	   "[--changed-paths] [--[no-]max-new-filters <n>] [--[no-]progress] "
-	   "<split options>"),
-	NULL
+static char const * const builtin_commit_graph_usage[] = {
+	BUILTIN_COMMIT_GRAPH_VERIFY_USAGE,
+	BUILTIN_COMMIT_GRAPH_WRITE_USAGE,
+	NULL,
 };
 
 static struct opts_commit_graph {
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
#10 lore raw

Re: [PATCH 1/5] commit-graph: define common usage with a macro

Quoted 6 lines · Ævar Arnfjörð Bjarmason
builtin/commit-graph.c +4 -0
+static const char * builtin_commit_graph_verify_usage[] = {
+#define BUILTIN_COMMIT_GRAPH_VERIFY_USAGE \
+	N_("git commit-graph verify [--object-dir <objdir>] [--shallow] [--[no-]progress]")
+	BUILTIN_COMMIT_GRAPH_VERIFY_USAGE,
 	NULL
 };

builtin/commit-graph.c line 27

Quoted 10 lines
builtin/commit-graph.c +7 -0
 
+static const char * builtin_commit_graph_write_usage[] = {
+#define BUILTIN_COMMIT_GRAPH_WRITE_USAGE \
+	N_("git commit-graph write [--object-dir <objdir>] [--append] " \
+	   "[--split[=<strategy>]] [--reachable|--stdin-packs|--stdin-commits] " \
+	   "[--changed-paths] [--[no-]max-new-filters <n>] [--[no-]progress] " \
+	   "<split options>")
+	BUILTIN_COMMIT_GRAPH_WRITE_USAGE,
 	NULL
 };

This seemed very unnatural to me, but it all makes sense in the end:

builtin/commit-graph.c line 33

Quoted 5 lines
builtin/commit-graph.c +4 -0
+static char const * const builtin_commit_graph_usage[] = {
+	BUILTIN_COMMIT_GRAPH_VERIFY_USAGE,
+	BUILTIN_COMMIT_GRAPH_WRITE_USAGE,
+	NULL,
 };

Clever!

Thanks,
-Stolee

#11 lore raw

PATCH 2/5 commit-graph: remove redundant handling of -h

In reply to Ævar Arnfjörð Bjarmason

If we don't handle the -h option here like most parse_options() users we'll fall through and it'll do the right thing for us.

I think this code added in 4ce58ee38d (commit-graph: create git-commit-graph builtin, 2018-04-02) was always redundant, parse_options() did this at the time, and the commit-graph code never used PARSE_OPT_NO_INTERNAL_HELP.

---
 builtin/commit-graph.c  | 4 ----
 t/t5318-commit-graph.sh | 5 +++++
 2 files changed, 5 insertions(+), 4 deletions(-)
builtin/commit-graph.c +0 -4
index c3fa4fde3e..baead04a03 100644
@@ -319,10 +319,6 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)
 		OPT_END(),
 	};
 
-	if (argc == 2 && !strcmp(argv[1], "-h"))
-		usage_with_options(builtin_commit_graph_usage,
-				   builtin_commit_graph_options);
-
 	git_config(git_default_config, NULL);
 	argc = parse_options(argc, argv, prefix,
 			     builtin_commit_graph_options,
t/t5318-commit-graph.sh +5 -0
index 2ed0c1544d..567e68bd93 100755
@@ -5,6 +5,11 @@ test_description='commit graph'
 
 GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0
 
+test_expect_success 'usage' '
+	test_expect_code 129 git commit-graph -h 2>err &&
+	! grep error: err
+'
+
 test_expect_success 'setup full repo' '
 	mkdir full &&
 	cd "$TRASH_DIRECTORY/full" &&
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
#12 lore raw

Re: [PATCH 2/5] commit-graph: remove redundant handling of -h

t/t5318-commit-graph.sh test_description='commit graph' · line 11

Quoted 4 lines · Ævar Arnfjörð Bjarmason
t/t5318-commit-graph.sh +4 -0
+test_expect_success 'usage' '
+	test_expect_code 129 git commit-graph -h 2>err &&
+	! grep error: err
+'

I think this test already exists in t0012-help.sh, since it tests bogus options for all of the builtins. (I can guarantee that I wouldn't have thought to add the check without some instance like that.)

Thanks,
-Stolee

#13 lore raw

PATCH 3/5 commit-graph: use parse_options_concat()

In reply to Ævar Arnfjörð Bjarmason

Make use of the parse_options_concat() so we don't need to copy/paste common options like --object-dir. This is inspired by a similar change to "checkout" in 2087182272
(checkout: split options[] array in three pieces, 2019-03-29).

A minor behavior change here is that now we're going to list both --object-dir and --progress first, before we'd list --progress along with other options.

---
 builtin/commit-graph.c | 43 ++++++++++++++++++++++++------------------
 1 file changed, 25 insertions(+), 18 deletions(-)
builtin/commit-graph.c +25 -18
index baead04a03..a7718b2025 100644
@@ -44,6 +44,21 @@ static struct opts_commit_graph {
 	int enable_changed_paths;
 } opts;
 
+static struct option *add_common_options(struct option *prevopts)
+{
+	struct option options[] = {
+		OPT_STRING(0, "object-dir", &opts.obj_dir,
+			   N_("dir"),
+			   N_("the object directory to store the graph")),
+		OPT_BOOL(0, "progress", &opts.progress,
+			 N_("force progress reporting")),
+		OPT_END()
+	};
+	struct option *newopts = parse_options_concat(options, prevopts);
+	free(prevopts);
+	return newopts;
+}
+
 static struct object_directory *find_odb(struct repository *r,
 					 const char *obj_dir)
 {
@@ -75,22 +90,20 @@ static int graph_verify(int argc, const char **argv)
 	int fd;
 	struct stat st;
 	int flags = 0;
-
+	struct option *options = NULL;
 	static struct option builtin_commit_graph_verify_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			   N_("dir"),
-			   N_("the object directory to store the graph")),
 		OPT_BOOL(0, "shallow", &opts.shallow,
 			 N_("if the commit-graph is split, only verify the tip file")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
 		OPT_END(),
 	};
+	options = parse_options_dup(builtin_commit_graph_verify_options);
+	options = add_common_options(options);
 
 	trace2_cmd_mode("verify");
 
 	opts.progress = isatty(2);
 	argc = parse_options(argc, argv, NULL,
-			     builtin_commit_graph_verify_options,
+			     options,
 			     builtin_commit_graph_verify_usage, 0);
 
 	if (!opts.obj_dir)
@@ -205,11 +218,8 @@ static int graph_write(int argc, const char **argv)
 	int result = 0;
 	enum commit_graph_write_flags flags = 0;
 	struct progress *progress = NULL;
-
+	struct option *options = NULL;
 	static struct option builtin_commit_graph_write_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			N_("dir"),
-			N_("the object directory to store the graph")),
 		OPT_BOOL(0, "reachable", &opts.reachable,
 			N_("start walk at all refs")),
 		OPT_BOOL(0, "stdin-packs", &opts.stdin_packs,
@@ -220,7 +230,6 @@ static int graph_write(int argc, const char **argv)
 			N_("include all commits already in the commit-graph file")),
 		OPT_BOOL(0, "changed-paths", &opts.enable_changed_paths,
 			N_("enable computation for changed paths")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
 		OPT_CALLBACK_F(0, "split", &write_opts.split_flags, NULL,
 			N_("allow writing an incremental commit-graph file"),
 			PARSE_OPT_OPTARG | PARSE_OPT_NONEG,
@@ -236,6 +245,8 @@ static int graph_write(int argc, const char **argv)
 			0, write_option_max_new_filters),
 		OPT_END(),
 	};
+	options = parse_options_dup(builtin_commit_graph_write_options);
+	options = add_common_options(options);
 
 	opts.progress = isatty(2);
 	opts.enable_changed_paths = -1;
@@ -249,7 +260,7 @@ static int graph_write(int argc, const char **argv)
 	git_config(git_commit_graph_write_config, &opts);
 
 	argc = parse_options(argc, argv, NULL,
-			     builtin_commit_graph_write_options,
+			     options,
 			     builtin_commit_graph_write_usage, 0);
 
 	if (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)
@@ -312,12 +323,8 @@ static int graph_write(int argc, const char **argv)
 
 int cmd_commit_graph(int argc, const char **argv, const char *prefix)
 {
-	static struct option builtin_commit_graph_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			N_("dir"),
-			N_("the object directory to store the graph")),
-		OPT_END(),
-	};
+	struct option *no_options = parse_options_dup(NULL);
+	struct option *builtin_commit_graph_options = add_common_options(no_options);
 
 	git_config(git_default_config, NULL);
 	argc = parse_options(argc, argv, prefix,
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
#14 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

Quoted 8 lines · Ævar Arnfjörð Bjarmason

Make use of the parse_options_concat() so we don't need to copy/paste common options like --object-dir. This is inspired by a similar change to "checkout" in 2087182272
(checkout: split options[] array in three pieces, 2019-03-29).

A minor behavior change here is that now we're going to list both --object-dir and --progress first, before we'd list --progress along with other options.

"Behavior change" referring only to the output of git commit-graph -h, no?

Looking at the code (and understanding this whole situation a little bit better), I'd think that this wouldn't cause us to parse anything differently before or after this change, right?

Quoted 13 lines
---
 builtin/commit-graph.c | 43 ++++++++++++++++++++++++------------------
 1 file changed, 25 insertions(+), 18 deletions(-)
builtin/commit-graph.c +0 -0
index baead04a03..a7718b2025 100644
@@ -44,6 +44,21 @@ static struct opts_commit_graph {
 	int enable_changed_paths;
 } opts;
 

builtin/commit-graph.c static struct opts_commit_graph { · line 56

Quoted 10 lines
builtin/commit-graph.c +10 -0
+static struct option *add_common_options(struct option *prevopts)
+{
+	struct option options[] = {
+		OPT_STRING(0, "object-dir", &opts.obj_dir,
+			   N_("dir"),
+			   N_("the object directory to store the graph")),
+		OPT_BOOL(0, "progress", &opts.progress,
+			 N_("force progress reporting")),
+		OPT_END()
+	};

I'm nitpicking, but I wouldn't be sad to see this called "common" instead".

Can't this also be declared statically?

Quoted 14 lines
builtin/commit-graph.c +6 -1
+	struct option *newopts = parse_options_concat(options, prevopts);
+	free(prevopts);
+	return newopts;
+}
+
 static struct object_directory *find_odb(struct repository *r,
 					 const char *obj_dir)
 {
@@ -75,22 +90,20 @@ static int graph_verify(int argc, const char **argv)
 	int fd;
 	struct stat st;
 	int flags = 0;
-
+	struct option *options = NULL;

builtin/commit-graph.c static int graph_verify(int argc, const char **argv) · line 99

Quoted 10 lines
builtin/commit-graph.c +1 -4
 	static struct option builtin_commit_graph_verify_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			   N_("dir"),
-			   N_("the object directory to store the graph")),
 		OPT_BOOL(0, "shallow", &opts.shallow,
 			 N_("if the commit-graph is split, only verify the tip file")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
 		OPT_END(),
 	};
+	options = parse_options_dup(builtin_commit_graph_verify_options);

Another nitpick, but I'd rather see the initialization of "options" and its declaration be on the same line, after declaring builtin_commit_graph_verify_options.

Quoted 51 lines
builtin/commit-graph.c +6 -7
+	options = add_common_options(options);
 
 	trace2_cmd_mode("verify");
 
 	opts.progress = isatty(2);
 	argc = parse_options(argc, argv, NULL,
-			     builtin_commit_graph_verify_options,
+			     options,
 			     builtin_commit_graph_verify_usage, 0);
 
 	if (!opts.obj_dir)
@@ -205,11 +218,8 @@ static int graph_write(int argc, const char **argv)
 	int result = 0;
 	enum commit_graph_write_flags flags = 0;
 	struct progress *progress = NULL;
-
+	struct option *options = NULL;
 	static struct option builtin_commit_graph_write_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			N_("dir"),
-			N_("the object directory to store the graph")),
 		OPT_BOOL(0, "reachable", &opts.reachable,
 			N_("start walk at all refs")),
 		OPT_BOOL(0, "stdin-packs", &opts.stdin_packs,
@@ -220,7 +230,6 @@ static int graph_write(int argc, const char **argv)
 			N_("include all commits already in the commit-graph file")),
 		OPT_BOOL(0, "changed-paths", &opts.enable_changed_paths,
 			N_("enable computation for changed paths")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
 		OPT_CALLBACK_F(0, "split", &write_opts.split_flags, NULL,
 			N_("allow writing an incremental commit-graph file"),
 			PARSE_OPT_OPTARG | PARSE_OPT_NONEG,
@@ -236,6 +245,8 @@ static int graph_write(int argc, const char **argv)
 			0, write_option_max_new_filters),
 		OPT_END(),
 	};
+	options = parse_options_dup(builtin_commit_graph_write_options);
+	options = add_common_options(options);
 
 	opts.progress = isatty(2);
 	opts.enable_changed_paths = -1;
@@ -249,7 +260,7 @@ static int graph_write(int argc, const char **argv)
 	git_config(git_commit_graph_write_config, &opts);
 
 	argc = parse_options(argc, argv, NULL,
-			     builtin_commit_graph_write_options,
+			     options,
 			     builtin_commit_graph_write_usage, 0);
 
 	if (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)

builtin/commit-graph.c static int graph_write(int argc, const char **argv) · line 326

Quoted 10 lines
builtin/commit-graph.c +1 -6
@@ -312,12 +323,8 @@ static int graph_write(int argc, const char **argv)
 
 int cmd_commit_graph(int argc, const char **argv, const char *prefix)
 {
-	static struct option builtin_commit_graph_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			N_("dir"),
-			N_("the object directory to store the graph")),
-		OPT_END(),
-	};
+	struct option *no_options = parse_options_dup(NULL);

Hmm. Why bother calling add_common_options at all here?

Thanks,
Taylor

#15 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

Quoted 3 lines · Taylor Blau

Another nitpick, but I'd rather see the initialization of "options" and its declaration be on the same line, after declaring builtin_commit_graph_verify_options.

Ignore me; the NULL initialization is important.

Thanks,
Taylor

#16 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

In reply to Taylor Blau
Quoted 10 lines · Taylor Blau
Quoted 8 lines · Ævar Arnfjörð Bjarmason

Make use of the parse_options_concat() so we don't need to copy/paste common options like --object-dir. This is inspired by a similar change to "checkout" in 2087182272
(checkout: split options[] array in three pieces, 2019-03-29).

A minor behavior change here is that now we're going to list both --object-dir and --progress first, before we'd list --progress along with other options.

Quoted 6 lines

"Behavior change" referring only to the output of git commit-graph -h, no?

Looking at the code (and understanding this whole situation a little bit better), I'd think that this wouldn't cause us to parse anything differently before or after this change, right?

Indeed, I just mean the "-h" or "--invalid-opt" output changed in the order we show the options in.

Quoted 27 lines
Quoted 13 lines
---
 builtin/commit-graph.c | 43 ++++++++++++++++++++++++------------------
 1 file changed, 25 insertions(+), 18 deletions(-)
builtin/commit-graph.c +0 -0
index baead04a03..a7718b2025 100644
@@ -44,6 +44,21 @@ static struct opts_commit_graph {
 	int enable_changed_paths;
 } opts;
 

builtin/commit-graph.c static struct opts_commit_graph { · line 56

Quoted 10 lines
builtin/commit-graph.c +10 -0
+static struct option *add_common_options(struct option *prevopts)
+{
+	struct option options[] = {
+		OPT_STRING(0, "object-dir", &opts.obj_dir,
+			   N_("dir"),
+			   N_("the object directory to store the graph")),
+		OPT_BOOL(0, "progress", &opts.progress,
+			 N_("force progress reporting")),
+		OPT_END()
+	};

I'm nitpicking, but I wouldn't be sad to see this called "common" instead".

Quoted 1 line

Can't this also be declared statically?

It happens to work now to do that, but try it in builtin/checkout.c and you'll see it blows up with a wall of "initializer element is not constant".

Probably better to be consistent in parse_options() usage than make it safe for that sort of use...

Quoted 20 lines
Quoted 20 lines
builtin/commit-graph.c +6 -4
+	struct option *newopts = parse_options_concat(options, prevopts);
+	free(prevopts);
+	return newopts;
+}
+
 static struct object_directory *find_odb(struct repository *r,
 					 const char *obj_dir)
 {
@@ -75,22 +90,20 @@ static int graph_verify(int argc, const char **argv)
 	int fd;
 	struct stat st;
 	int flags = 0;
-
+	struct option *options = NULL;
 	static struct option builtin_commit_graph_verify_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			   N_("dir"),
-			   N_("the object directory to store the graph")),
 		OPT_BOOL(0, "shallow", &opts.shallow,
 			 N_("if the commit-graph is split, only verify the tip file")),
Quoted 8 lines

builtin/commit-graph.c static int graph_verify(int argc, const char **argv) · line 99

Quoted 4 lines
builtin/commit-graph.c +1 -1
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
 		OPT_END(),
 	};
+	options = parse_options_dup(builtin_commit_graph_verify_options);

Another nitpick, but I'd rather see the initialization of "options" and its declaration be on the same line, after declaring builtin_commit_graph_verify_options.

As you noted in your own reply "the NULL initialization is important", or more specifically: We're doing this dance here (and in other existing code, e.g. checkout.c) to trampoline from the stack ot the heap.

Quoted 52 lines
Quoted 52 lines
builtin/commit-graph.c +6 -7
+	options = add_common_options(options);
 
 	trace2_cmd_mode("verify");
 
 	opts.progress = isatty(2);
 	argc = parse_options(argc, argv, NULL,
-			     builtin_commit_graph_verify_options,
+			     options,
 			     builtin_commit_graph_verify_usage, 0);
 
 	if (!opts.obj_dir)
@@ -205,11 +218,8 @@ static int graph_write(int argc, const char **argv)
 	int result = 0;
 	enum commit_graph_write_flags flags = 0;
 	struct progress *progress = NULL;
-
+	struct option *options = NULL;
 	static struct option builtin_commit_graph_write_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			N_("dir"),
-			N_("the object directory to store the graph")),
 		OPT_BOOL(0, "reachable", &opts.reachable,
 			N_("start walk at all refs")),
 		OPT_BOOL(0, "stdin-packs", &opts.stdin_packs,
@@ -220,7 +230,6 @@ static int graph_write(int argc, const char **argv)
 			N_("include all commits already in the commit-graph file")),
 		OPT_BOOL(0, "changed-paths", &opts.enable_changed_paths,
 			N_("enable computation for changed paths")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
 		OPT_CALLBACK_F(0, "split", &write_opts.split_flags, NULL,
 			N_("allow writing an incremental commit-graph file"),
 			PARSE_OPT_OPTARG | PARSE_OPT_NONEG,
@@ -236,6 +245,8 @@ static int graph_write(int argc, const char **argv)
 			0, write_option_max_new_filters),
 		OPT_END(),
 	};
+	options = parse_options_dup(builtin_commit_graph_write_options);
+	options = add_common_options(options);
 
 	opts.progress = isatty(2);
 	opts.enable_changed_paths = -1;
@@ -249,7 +260,7 @@ static int graph_write(int argc, const char **argv)
 	git_config(git_commit_graph_write_config, &opts);
 
 	argc = parse_options(argc, argv, NULL,
-			     builtin_commit_graph_write_options,
+			     options,
 			     builtin_commit_graph_write_usage, 0);
 
 	if (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)
@@ -312,12 +323,8 @@ static int graph_write(int argc, const char **argv)
 
Quoted 11 lines

builtin/commit-graph.c static int graph_write(int argc, const char **argv) · line 326

Quoted 9 lines
builtin/commit-graph.c +1 -6
 int cmd_commit_graph(int argc, const char **argv, const char *prefix)
 {
-	static struct option builtin_commit_graph_options[] = {
-		OPT_STRING(0, "object-dir", &opts.obj_dir,
-			N_("dir"),
-			N_("the object directory to store the graph")),
-		OPT_END(),
-	};
+	struct option *no_options = parse_options_dup(NULL);

Hmm. Why bother calling add_common_options at all here?

I assume you mean in this line just below what you quoted:

    struct option *builtin_commit_graph_options = add_common_options(no_options);

Do you mean why not do the whole thing in graph_{verify,write}() and only show the usage if we fail here?

Yeah arguably that makes more sense, but I wanted to just focus on refactoring existing behavior & get rid of the copy/pasted options rather than start a bigger rewrite of "maybe we shouldn't show this rather useless help info if we die here....".

#17 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

Quoted 4 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line · Taylor Blau

On Mon, Feb 15, 2021 at 07:41:16PM +0100, Ævar Arnfjörð Bjarmason wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Make use of the parse_options_concat() so we don't need to copy/paste common options like --object-dir. This is inspired by a similar change to "checkout" in 2087182272
(checkout: split options[] array in three pieces, 2019-03-29).

A minor behavior change here is that now we're going to list both --object-dir and --progress first, before we'd list --progress along with other options.

The final version of this patch that was picked up is at

https://public-inbox.org/git/[email protected]/

I reply to this old version because of the following pieces of the discussion:

Quoted 9 lines
Quoted 6 lines

"Behavior change" referring only to the output of git commit-graph -h, no?

Looking at the code (and understanding this whole situation a little bit better), I'd think that this wouldn't cause us to parse anything differently before or after this change, right?

Indeed, I just mean the "-h" or "--invalid-opt" output changed in the order we show the options in.

[...]

Quoted 2 lines

but I wanted to just focus on
refactoring existing behavior & get rid of the copy/pasted options

No, there is more behavior change: since 84e4484f12 (commit-graph: use parse_options_concat(), 2021-08-23) the 'git commit-graph' command does accept the '--[no-]progress' options as well, but before that only its subcommands did, and 'git commit-graph --progress ...' errored out with "unknown option".

Worse, sometimes 'git commit-graph --progress ...' doesn't work as it's supposed to. The patch below descibes the problem and fixes it, but on second thought I don't think that it is the right approach.

In general, even when all subcommands of a git command understand a particular --option, that does not mean that it's a good idea to teach that option to that git command. E.g. what if we later add another subcommand for which that --option doesn't make any sense? And from the quoted discussion above it seems that teaching 'git commit-graph' the '--progress' option was not intentional at all.

I'm inclined to think that '--progress' should rather be removed from the common 'git commit-graph' options; luckily it's not too late, because it hasn't been released yet.

--- >8 ---

Subject: [PATCH] commit-graph: fix 'git commit-graph --[no-]progress ...'

Until recenly 'git commit-graph' didn't have a '--progress' option, only its subcommands did, but this changed with 84e4484f12 (commit-graph: use parse_options_concat(), 2021-08-23), and now the 'git commit-graph' command accepts the '--[no-]progress' options as well. Alas, they don't always works as they are supposed to, because the isatty(2) check is only performed in the subcommands, i.e. after the "main" 'git commit-graph' command has parsed its options, and it unconditionally overwrites whatever '--[no-]progress' option might have been given:

  $ GIT_PROGRESS_DELAY=0 git commit-graph --no-progress write --reachable
  Collecting referenced commits: 1617, done.
  Loading known commits in commit graph: 100% (1617/1617), done.
  [...]
  $ GIT_PROGRESS_DELAY=0 git commit-graph --progress write 2>out
  $ wc -c out
  0 out

Move the isatty(2) check to cmd_commit_graph(), before it calls parse_options(), so 'git commit-graph --[no-]progress' will be able to override it as well.

---
 builtin/commit-graph.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
builtin/commit-graph.c +1 -2
index 21fc6e934b..3a873ceaf6 100644
@@ -101,7 +101,6 @@ static int graph_verify(int argc, const char **argv)
 
 	trace2_cmd_mode("verify");
 
-	opts.progress = isatty(2);
 	argc = parse_options(argc, argv, NULL,
 			     options,
 			     builtin_commit_graph_verify_usage, 0);
@@ -250,7 +249,6 @@ static int graph_write(int argc, const char **argv)
 	};
 	struct option *options = add_common_options(builtin_commit_graph_write_options);
 
-	opts.progress = isatty(2);
 	opts.enable_changed_paths = -1;
 	write_opts.size_multiple = 2;
 	write_opts.max_commits = 0;
@@ -331,6 +329,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)
 	struct option *builtin_commit_graph_options = common_opts;
 
 	git_config(git_default_config, NULL);
+	opts.progress = isatty(2);
 	argc = parse_options(argc, argv, prefix,
 			     builtin_commit_graph_options,
 			     builtin_commit_graph_usage,
Signature · 3 lines
2.33.0.517.ga8dcee0d0a

  ---  8<  ---
#18 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

Quoted 11 lines · SZEDER Gábor

Worse, sometimes 'git commit-graph --progress ...' doesn't work as it's supposed to. The patch below descibes the problem and fixes it, but on second thought I don't think that it is the right approach.

In general, even when all subcommands of a git command understand a particular --option, that does not mean that it's a good idea to teach that option to that git command. E.g. what if we later add another subcommand for which that --option doesn't make any sense? And from the quoted discussion above it seems that teaching 'git commit-graph' the '--progress' option was not intentional at all.

Quoted 3 lines

I'm inclined to think that '--progress' should rather be removed from the common 'git commit-graph' options; luckily it's not too late, because it hasn't been released yet.

I wasn't following this series closely, but having seen your fix below, I'm inclined to agree with you. Just because we _can_ allow options before or after sub-commands does not necessarily make it a good idea.

There is a distinct meaning to options before/after the command for the base "git" command (e.g., "git -C foo branch" versus "git branch -C foo"), and I think that has been useful overall.

Quoted 3 lines

--- >8 ---

Subject: [PATCH] commit-graph: fix 'git commit-graph --[no-]progress ...'

This patch looks like a sensible fix if we don't simply remove the "git commit-graph --progress write" version.

-Peff

#19 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

Quoted 8 lines · Jeff King
Quoted 3 lines

I'm inclined to think that '--progress' should rather be removed from the common 'git commit-graph' options; luckily it's not too late, because it hasn't been released yet.

I wasn't following this series closely, but having seen your fix below, I'm inclined to agree with you. Just because we _can_ allow options before or after sub-commands does not necessarily make it a good idea.

I agree. Suppose we had a "git commit-graph remove" sub-command that removed the commit-graph file (ignoring that there are probably better hypothetical examples than this ;)). It's not obvious what --progress means in the context of that mode.

Here's a patch that does what you and Gábor are suggesting as an alternative. Unfortunately, we can't do the same for the multi-pack-index command, since the analogous change there is 60ca94769c (builtin/multi-pack-index.c: split sub-commands, 2021-03-30), which was released in 2.32.

Anyway, as promised:

--- 8< ---

Subject: [PATCH] builtin/commit-graph.c: don't accept common --[no-]progress

In 84e4484f12 (commit-graph: use parse_options_concat(), 2021-08-23) we unified common options of commit-graph's subcommands into a single "common_opts" array.

But 84e4484f12 introduced a behavior change which is to accept the "--[no-]progress" option before any sub-commands, e.g.,

git commit-graph --progress write ...

Prior to that commit, the above would error out with "unknown option".

There are two issues with this behavior change. First is that the top-level --[no-]progress is not always respected. This is because isatty(2) is performed in the sub-commands, which unconditionally overwrites any --[no-]progress that was given at the top-level.

But the second issue is that the existing sub-commands of commit-graph only happen to both have a sensible interpretation of what --progress or --no-progress means. If we ever added a sub-command which didn't have a notion of progress, we would be forced to ignore the top-level --[no-]progress altogether.

Since we haven't released a version of Git that supports --[no-]progress as a top-level option for git commit-graph, let's remove it.

---
 builtin/commit-graph.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
builtin/commit-graph.c +4 -2
index 21fc6e934b..067587a0fd 100644
@@ -50,8 +50,6 @@ static struct option common_opts[] = {
 	OPT_STRING(0, "object-dir", &opts.obj_dir,
 		   N_("dir"),
 		   N_("the object directory to store the graph")),
-	OPT_BOOL(0, "progress", &opts.progress,
-		 N_("force progress reporting")),
 	OPT_END()
 };
 
@@ -95,6 +93,8 @@ static int graph_verify(int argc, const char **argv)
 	static struct option builtin_commit_graph_verify_options[] = {
 		OPT_BOOL(0, "shallow", &opts.shallow,
 			 N_("if the commit-graph is split, only verify the tip file")),
+		OPT_BOOL(0, "progress", &opts.progress,
+			 N_("force progress reporting")),
 		OPT_END(),
 	};
 	struct option *options = add_common_options(builtin_commit_graph_verify_options);
@@ -246,6 +246,8 @@ static int graph_write(int argc, const char **argv)
 		OPT_CALLBACK_F(0, "max-new-filters", &write_opts.max_new_filters,
 			NULL, N_("maximum number of changed-path Bloom filters to compute"),
 			0, write_option_max_new_filters),
+		OPT_BOOL(0, "progress", &opts.progress,
+			 N_("force progress reporting")),
 		OPT_END(),
 	};
 	struct option *options = add_common_options(builtin_commit_graph_write_options);

--
2.33.0.96.g73915697e6

#20 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

Quoted 5 lines · Taylor Blau
Quoted 4 lines · Jeff King
Quoted 3 lines

I'm inclined to think that '--progress' should rather be removed from the common 'git commit-graph' options; luckily it's not too late, because it hasn't been released yet.

Quoted 8 lines
Quoted 4 lines

I wasn't following this series closely, but having seen your fix below, I'm inclined to agree with you. Just because we _can_ allow options before or after sub-commands does not necessarily make it a good idea.

I agree. Suppose we had a "git commit-graph remove" sub-command that removed the commit-graph file (ignoring that there are probably better hypothetical examples than this ;)). It's not obvious what --progress means in the context of that mode.

Well, you might have as many as tens of commit-graph files, and could be running this on AIX where I/O is apparently implemented in terms of carrier pigeon messaging judging by how slow it is :)

But as argued in
https://lore.kernel.org/git/[email protected]/ I don't see how it's going to be any more confusing to user than "git -c foo=bar version" (the -c does nothing there)>

Quoted 5 lines

Here's a patch that does what you and Gábor are suggesting as an alternative. Unfortunately, we can't do the same for the multi-pack-index command, since the analogous change there is 60ca94769c (builtin/multi-pack-index.c: split sub-commands, 2021-03-30), which was released in 2.32.

If we came up with some call about what we want subcommands in general to look like I'd think it would be fine to convert multi-pack-index to it, perhaps with some deprecation period where it would issue a warning() while it understood both forms.

Quoted 60 lines

Anyway, as promised:

--- 8< ---

Subject: [PATCH] builtin/commit-graph.c: don't accept common --[no-]progress

In 84e4484f12 (commit-graph: use parse_options_concat(), 2021-08-23) we unified common options of commit-graph's subcommands into a single "common_opts" array.

But 84e4484f12 introduced a behavior change which is to accept the "--[no-]progress" option before any sub-commands, e.g.,

git commit-graph --progress write ...

Prior to that commit, the above would error out with "unknown option".

There are two issues with this behavior change. First is that the top-level --[no-]progress is not always respected. This is because isatty(2) is performed in the sub-commands, which unconditionally overwrites any --[no-]progress that was given at the top-level.

But the second issue is that the existing sub-commands of commit-graph only happen to both have a sensible interpretation of what --progress or --no-progress means. If we ever added a sub-command which didn't have a notion of progress, we would be forced to ignore the top-level --[no-]progress altogether.

Since we haven't released a version of Git that supports --[no-]progress as a top-level option for git commit-graph, let's remove it.

---
 builtin/commit-graph.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
builtin/commit-graph.c +2 -2
index 21fc6e934b..067587a0fd 100644
@@ -50,8 +50,6 @@ static struct option common_opts[] = {
 	OPT_STRING(0, "object-dir", &opts.obj_dir,
 		   N_("dir"),
 		   N_("the object directory to store the graph")),
-	OPT_BOOL(0, "progress", &opts.progress,
-		 N_("force progress reporting")),
 	OPT_END()
 };
 
@@ -95,6 +93,8 @@ static int graph_verify(int argc, const char **argv)
 	static struct option builtin_commit_graph_verify_options[] = {
 		OPT_BOOL(0, "shallow", &opts.shallow,
 			 N_("if the commit-graph is split, only verify the tip file")),
+		OPT_BOOL(0, "progress", &opts.progress,
+			 N_("force progress reporting")),
 		OPT_END(),
 	};
 	struct option *options = add_common_options(builtin_commit_graph_verify_options);

builtin/commit-graph.c static int graph_write(int argc, const char **argv) · line 253

Quoted 8 lines
builtin/commit-graph.c +2 -0
@@ -246,6 +246,8 @@ static int graph_write(int argc, const char **argv)
 		OPT_CALLBACK_F(0, "max-new-filters", &write_opts.max_new_filters,
 			NULL, N_("maximum number of changed-path Bloom filters to compute"),
 			0, write_option_max_new_filters),
+		OPT_BOOL(0, "progress", &opts.progress,
+			 N_("force progress reporting")),
 		OPT_END(),
 	};
 	struct option *options = add_common_options(builtin_commit_graph_write_options);

This is a good change, but if you're up for bonus points leaves the docs in an odd where we (as noted in [1]) document the --object-dir and --progress options under OPTIONS, but now only take the former before the sub-command.

1. https://lore.kernel.org/git/[email protected]/

#21 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

Quoted 4 lines · Ævar Arnfjörð Bjarmason

If we came up with some call about what we want subcommands in general to look like I'd think it would be fine to convert multi-pack-index to it, perhaps with some deprecation period where it would issue a warning() while it understood both forms.

I'm not sure about what to do with the multi-pack-index command. Probably going through a deprecation process makes the most sense if we do want to get rid of the top-level --[no-]progress there, too. But let's have the discussion elsewhere and not buried in the MIDX bitmaps thread ;).

Quoted 4 lines

This is a good change, but if you're up for bonus points leaves the docs in an odd where we (as noted in [1]) document the --object-dir and --progress options under OPTIONS, but now only take the former before the sub-command.

Thanks for noticing. I got up and did something in between writing and sending this patch, and had a nagging feeling of forgetting something before I sent. But I couldn't figure out what ;).

I'll resubmit this patch (with the doc changes squashed in) as a new thread on the list so we can have the discussion not buried in another thread.

Thanks,
Taylor

#22 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

Quoted 8 lines · Taylor Blau
Quoted 4 lines

This is a good change, but if you're up for bonus points leaves the docs in an odd where we (as noted in [1]) document the --object-dir and --progress options under OPTIONS, but now only take the former before the sub-command.

Thanks for noticing. I got up and did something in between writing and sending this patch, and had a nagging feeling of forgetting something before I sent. But I couldn't figure out what ;).

Actually, I stand by the original patch. Yes, the top-level OPTIONS of git-commit-graph(1) mentions --[no-]progress, but the synopsis makes clear that those are only accepted after the sub-commands.

So I think it's fine as-is.

Thanks,
Taylor

#23 lore raw

Re: [PATCH 3/5] commit-graph: use parse_options_concat()

In reply to SZEDER Gábor
Quoted 110 lines · SZEDER Gábor
Quoted 4 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line · Taylor Blau

On Mon, Feb 15, 2021 at 07:41:16PM +0100, Ævar Arnfjörð Bjarmason wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Make use of the parse_options_concat() so we don't need to copy/paste common options like --object-dir. This is inspired by a similar change to "checkout" in 2087182272
(checkout: split options[] array in three pieces, 2019-03-29).

A minor behavior change here is that now we're going to list both --object-dir and --progress first, before we'd list --progress along with other options.

The final version of this patch that was picked up is at

https://public-inbox.org/git/[email protected]/

I reply to this old version because of the following pieces of the discussion:

Quoted 9 lines
Quoted 6 lines

"Behavior change" referring only to the output of git commit-graph -h, no?

Looking at the code (and understanding this whole situation a little bit better), I'd think that this wouldn't cause us to parse anything differently before or after this change, right?

Indeed, I just mean the "-h" or "--invalid-opt" output changed in the order we show the options in.

[...]

Quoted 2 lines

but I wanted to just focus on
refactoring existing behavior & get rid of the copy/pasted options

No, there is more behavior change: since 84e4484f12 (commit-graph: use parse_options_concat(), 2021-08-23) the 'git commit-graph' command does accept the '--[no-]progress' options as well, but before that only its subcommands did, and 'git commit-graph --progress ...' errored out with "unknown option".

Worse, sometimes 'git commit-graph --progress ...' doesn't work as it's supposed to. The patch below descibes the problem and fixes it, but on second thought I don't think that it is the right approach.

In general, even when all subcommands of a git command understand a particular --option, that does not mean that it's a good idea to teach that option to that git command. E.g. what if we later add another subcommand for which that --option doesn't make any sense? And from the quoted discussion above it seems that teaching 'git commit-graph' the '--progress' option was not intentional at all.

I'm inclined to think that '--progress' should rather be removed from the common 'git commit-graph' options; luckily it's not too late, because it hasn't been released yet.

--- >8 ---

Subject: [PATCH] commit-graph: fix 'git commit-graph --[no-]progress ...'

Until recenly 'git commit-graph' didn't have a '--progress' option, only its subcommands did, but this changed with 84e4484f12 (commit-graph: use parse_options_concat(), 2021-08-23), and now the 'git commit-graph' command accepts the '--[no-]progress' options as well. Alas, they don't always works as they are supposed to, because the isatty(2) check is only performed in the subcommands, i.e. after the "main" 'git commit-graph' command has parsed its options, and it unconditionally overwrites whatever '--[no-]progress' option might have been given:

  $ GIT_PROGRESS_DELAY=0 git commit-graph --no-progress write --reachable
  Collecting referenced commits: 1617, done.
  Loading known commits in commit graph: 100% (1617/1617), done.
  [...]
  $ GIT_PROGRESS_DELAY=0 git commit-graph --progress write 2>out
  $ wc -c out
  0 out

Move the isatty(2) check to cmd_commit_graph(), before it calls parse_options(), so 'git commit-graph --[no-]progress' will be able to override it as well.

---
 builtin/commit-graph.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
builtin/commit-graph.c +0 -2
index 21fc6e934b..3a873ceaf6 100644
@@ -101,7 +101,6 @@ static int graph_verify(int argc, const char **argv)
 
 	trace2_cmd_mode("verify");
 
-	opts.progress = isatty(2);
 	argc = parse_options(argc, argv, NULL,
 			     options,
 			     builtin_commit_graph_verify_usage, 0);
@@ -250,7 +249,6 @@ static int graph_write(int argc, const char **argv)
 	};
 	struct option *options = add_common_options(builtin_commit_graph_write_options);
 
-	opts.progress = isatty(2);
 	opts.enable_changed_paths = -1;
 	write_opts.size_multiple = 2;
 	write_opts.max_commits = 0;

builtin/commit-graph.c int cmd_commit_graph(int argc, const char **argv, const char *prefix) · line 335

Quoted 7 lines
builtin/commit-graph.c +1 -0
@@ -331,6 +329,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)
 	struct option *builtin_commit_graph_options = common_opts;
 
 	git_config(git_default_config, NULL);
+	opts.progress = isatty(2);
 	argc = parse_options(argc, argv, prefix,
 			     builtin_commit_graph_options,
 			     builtin_commit_graph_usage,

Yes, this was unintentional on my part, sorry, and thanks for cleaning up my mess.

However, I have wondered how we should be dealing with these sub-commands in general.

In the case of commit-graph we've always documented it at the top-level as OPTIONS, so even though the usage shows:

git commit-graph write <options>

We've always accepted "--object-dir" after "git commit-graph", and all the other options are documented in their per-subcommand sections.

So just from reading the documentation you might think that this (with your fix here) is intentional behavior, and we should just fix the synopsis.

Then we have the more recent multi-pack-index which *is* documented as:

    'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
            [--preferred-pack=<pack>] <subcommand>

So actually, the reason this crept in is probably because I was copying the pattern we've had there since 60ca94769ce (builtin/multi-pack-index.c: split sub-commands, 2021-03-30), my commit message says as much.

Given that and multi-pack-index's documented behavior I think that it probably makes sense to keep and document this, and as a follow-up (which I or Taylor could do) change the synopsis accordingly.

Aside from whatever bugs have crept or existing behavior, I think it makes sense as UI to do things like:

    git commit-graph --object-dir=<dir> write --reachable
    git commit-graph --progress write
    git commit-graph --progress verify

etc., as --progress is a not-subcommand-specific option, not really. We might have a subcommand that doesn't have progress output, but I still think it makes sense to have it in that position, maybe we'll end up adding it later.

Brian and I also had a discussion back in April[1] about --object-format, i.e. should we be making every single command support:

    git hash-object --object-format=sha256

Or (as I suggested) doesn't it make more sense to do:

    git --object-format=sha256 hash-object

Like the --progress option it does mean that you'll end up with commands for whom that'll just be ignored:

    git --object-format=sha256 version

But that's conceptually similar to repo settings, and I don't think it's confusing, the same can be said about e.g.:

git -c this.doesNotUse=thisConfig version

Having said that for --progress it probably makes sense to eventually have:

    git --progress commit-graph write

I.e. maybe we'd want a top-level option for it, given how many commands have that option and us needing to pass a "do_progress" flag all over the place.

Of course we'd need to (silently or not) support it also as:

    git commit-graph --progress write
    git commit-graph write --progress

Which is the case here.

1. https://lore.kernel.org/git/[email protected]/

#24 lore raw

PATCH 4/5 commit-graph: refactor dispatch loop for style

In reply to Ævar Arnfjörð Bjarmason

I think it's more readable to have one if/elsif/else chain here than the code this replaces.

---
 builtin/commit-graph.c | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)
builtin/commit-graph.c +7 -9
index a7718b2025..66fbdb7cb1 100644
@@ -334,13 +334,11 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)
 
 	save_commit_buffer = 0;
 
-	if (argc > 0) {
-		if (!strcmp(argv[0], "verify"))
-			return graph_verify(argc, argv);
-		if (!strcmp(argv[0], "write"))
-			return graph_write(argc, argv);
-	}
-
-	usage_with_options(builtin_commit_graph_usage,
-			   builtin_commit_graph_options);
+	if (argc && !strcmp(argv[0], "verify"))
+		return graph_verify(argc, argv);
+	else if (argc && !strcmp(argv[0], "write"))
+		return graph_write(argc, argv);
+	else
+		usage_with_options(builtin_commit_graph_usage,
+				   builtin_commit_graph_options);
 }
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
#25 lore raw

Re: [PATCH 4/5] commit-graph: refactor dispatch loop for style

Quoted 2 lines · Ævar Arnfjörð Bjarmason

I think it's more readable to have one if/elsif/else chain here than the code this replaces.

FWIW, I find the pre-image more readable than what you are proposing replacing it with here.

Of course, I have no doubts about the obvious correctness of this patch; I'm merely suggesting that I wouldn't be sad to see us apply the first three patches, and the fifth patch, but drop this one.

Thanks,
Taylor

#26 lore raw

Re: [PATCH 4/5] commit-graph: refactor dispatch loop for style

Quoted 10 lines · Taylor Blau
Quoted 2 lines · Ævar Arnfjörð Bjarmason

I think it's more readable to have one if/elsif/else chain here than the code this replaces.

FWIW, I find the pre-image more readable than what you are proposing replacing it with here.

Of course, I have no doubts about the obvious correctness of this patch; I'm merely suggesting that I wouldn't be sad to see us apply the first three patches, and the fifth patch, but drop this one.

I agree with all of your points here. I think that compared to the current code at-rest, the new version might be preferred. It's a little dense, which is my only complaint.

The issue comes for the future: what if we need to add a third verb to 'git commit-graph'? Then extending this new option looks worse since we would check 'argc' three times.

The other patches solve real readability problems or reorganize the code to use other concepts within the codebase. This one is much more optional.

Thanks,
-Stolee

#27 lore raw

Re: [PATCH 4/5] commit-graph: refactor dispatch loop for style

Quoted 16 lines · Derrick Stolee
Quoted 10 lines · Taylor Blau
Quoted 2 lines · Ævar Arnfjörð Bjarmason

I think it's more readable to have one if/elsif/else chain here than the code this replaces.

FWIW, I find the pre-image more readable than what you are proposing replacing it with here.

Of course, I have no doubts about the obvious correctness of this patch; I'm merely suggesting that I wouldn't be sad to see us apply the first three patches, and the fifth patch, but drop this one.

I agree with all of your points here. I think that compared to the current code at-rest, the new version might be preferred. It's a little dense, which is my only complaint.

Quoted 6 lines

The issue comes for the future: what if we need to add a third verb to 'git commit-graph'? Then extending this new option looks worse since we would check 'argc' three times.

The other patches solve real readability problems or reorganize the code to use other concepts within the codebase. This one is much more optional.

What do you think about
https://lore.kernel.org/git/[email protected]/ ? :)

#28 lore raw

Re: [PATCH 4/5] commit-graph: refactor dispatch loop for style

Quoted 7 lines · Ævar Arnfjörð Bjarmason
Quoted 2 lines · Derrick Stolee

The other patches solve real readability problems or reorganize the code to use other concepts within the codebase. This one is much more optional.

What do you think about
https://lore.kernel.org/git/[email protected]/ ? :)

Using a 'goto' is a fine way to avoid a nesting level, but I'm not sure it "improves readability." Having the tab level makes it clear that that code is executed only when some condition is met, in this case "if (argc)," while with the 'goto' we need to know that execution was redirected.

I don't feel too strongly either way. If the code was presented one way or the other, I probably wouldn't recommend changing to the other mode. In that sense, the change isn't necessary and causes me to break the tie in favor of leaving it where it is.

Of course, if someone else says "I like this and would prefer it be used as an example for future contributors" then the tie is broken in the other way.

Thanks,
-Stolee

#29 lore raw

PATCH 5/5 commit-graph: show usage on "commit-graph [write|verify] garbage"

In reply to Ævar Arnfjörð Bjarmason

Change the parse_options() invocation in the commit-graph code to make sense. We're calling it twice, once for common options parsing, and then for the sub-commands.

But we never checked if we had something leftover in argc in "write" or "verify", as a result we'd silently accept garbage in these subcommands. Let's not do that.

---
 builtin/commit-graph.c  | 10 ++++++++--
 t/t5318-commit-graph.sh |  4 +++-
 2 files changed, 11 insertions(+), 3 deletions(-)
builtin/commit-graph.c +8 -2
index 66fbdb7cb1..cb57771026 100644
@@ -104,7 +104,10 @@ static int graph_verify(int argc, const char **argv)
 	opts.progress = isatty(2);
 	argc = parse_options(argc, argv, NULL,
 			     options,
-			     builtin_commit_graph_verify_usage, 0);
+			     builtin_commit_graph_verify_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_commit_graph_verify_usage, options);
 
 	if (!opts.obj_dir)
 		opts.obj_dir = get_object_directory();
@@ -261,7 +264,10 @@ static int graph_write(int argc, const char **argv)
 
 	argc = parse_options(argc, argv, NULL,
 			     options,
-			     builtin_commit_graph_write_usage, 0);
+			     builtin_commit_graph_write_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_commit_graph_write_usage, options);
 
 	if (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)
 		die(_("use at most one of --reachable, --stdin-commits, or --stdin-packs"));
t/t5318-commit-graph.sh +3 -1
index 567e68bd93..3f1c6dbc8f 100755
@@ -7,7 +7,9 @@ GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0
 
 test_expect_success 'usage' '
 	test_expect_code 129 git commit-graph -h 2>err &&
-	! grep error: err
+	! grep error: err &&
+	test_expect_code 129 git commit-graph write blah &&
+	test_expect_code 129 git commit-graph write verify
 '
 
 test_expect_success 'setup full repo' '
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
#30 lore raw

Re: [PATCH 5/5] commit-graph: show usage on "commit-graph [write|verify] garbage"

Quoted 7 lines · Ævar Arnfjörð Bjarmason

Change the parse_options() invocation in the commit-graph code to make sense. We're calling it twice, once for common options parsing, and then for the sub-commands.

But we never checked if we had something leftover in argc in "write" or "verify", as a result we'd silently accept garbage in these subcommands. Let's not do that.

...Implicit in all of this is that we need to pass PARSE_OPT_KEEP_UNKNOWN to have the sub-commands' call to parse_options() leave extra cruft alone so we can check for its existence with an "if (argc)".

Makes sense, thanks.

Thanks,
Taylor

#31 lore raw

Re: [PATCH 5/5] commit-graph: show usage on "commit-graph [write|verify] garbage"

In reply to Ævar Arnfjörð Bjarmason
Quoted 7 lines · Ævar Arnfjörð Bjarmason

test_expect_success 'usage' '

t/t5318-commit-graph.sh +2 -1
 	test_expect_code 129 git commit-graph -h 2>err &&
-	! grep error: err
+	! grep error: err &&
+	test_expect_code 129 git commit-graph write blah &&

test_expect_code 129 git commit-graph write verify
'

This extension of the test justifies its existence, so ignore my earlier comment about it being redundant.

Thanks,
-Stolee

#32 lore raw

v2 cover midx: split out sub-commands

In reply to Ævar Arnfjörð Bjarmason

Here's a few patches that we could add to the beginning of this series, or queue up separately.

I think that these are all fairly straightforward, but it would be good to have Ævar take a look and make sure I'm not doing anything wrong here.

I'll plan to send a v2 of the reverse index series in a few days with these four new patches at the beginning.

Taylor Blau (4):
builtin/multi-pack-index.c: inline 'flags' with options builtin/multi-pack-index.c: don't handle 'progress' separately builtin/multi-pack-index.c: define common usage with a macro builtin/multi-pack-index.c: split sub-commands

 builtin/multi-pack-index.c | 155 +++++++++++++++++++++++++++++--------
 1 file changed, 124 insertions(+), 31 deletions(-)

--
2.30.0.667.g81c0cbc6fd

#33 lore raw

PATCH v2 1/4 builtin/multi-pack-index.c: inline 'flags' with options

Subcommands of the 'git multi-pack-index' command (e.g., 'write', 'verify', etc.) will want to optionally change a set of shared flags that are eventually passed to the MIDX libraries.

Right now, options and flags are handled separately. Inline them into the same structure so that sub-commands can more easily share the 'flags' data.

---
 builtin/multi-pack-index.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
builtin/multi-pack-index.c +6 -7
index 5bf88cd2a8..4a0ddb06c4 100644
@@ -14,13 +14,12 @@ static struct opts_multi_pack_index {
 	const char *object_dir;
 	unsigned long batch_size;
 	int progress;
+	unsigned flags;
 } opts;
 
 int cmd_multi_pack_index(int argc, const char **argv,
 			 const char *prefix)
 {
-	unsigned flags = 0;
-
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
@@ -40,7 +39,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
 	if (opts.progress)
-		flags |= MIDX_PROGRESS;
+		opts.flags |= MIDX_PROGRESS;
 
 	if (argc == 0)
 		usage_with_options(builtin_multi_pack_index_usage,
@@ -55,16 +54,16 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	if (!strcmp(argv[0], "repack"))
 		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, flags);
+			(size_t)opts.batch_size, opts.flags);
 	if (opts.batch_size)
 		die(_("--batch-size option is only for 'repack' subcommand"));
 
 	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, flags);
+		return write_midx_file(opts.object_dir, opts.flags);
 	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, flags);
+		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
 	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, flags);
+		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
 
 	die(_("unrecognized subcommand: %s"), argv[0]);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#34 lore raw

PATCH v2 2/4 builtin/multi-pack-index.c: don't handle 'progress' separately

In reply to Taylor Blau

Now that there is a shared 'flags' member in the options structure, there is no need to keep track of whether to force progress or not, since ultimately the decision of whether or not to show a progress meter is controlled by a bit in the flags member.

Manipulate that bit directly, and drop the now-unnecessary 'progress' field while we're at it.

---
 builtin/multi-pack-index.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
builtin/multi-pack-index.c +3 -5
index 4a0ddb06c4..c70f020d8f 100644
@@ -13,7 +13,6 @@ static char const * const builtin_multi_pack_index_usage[] = {
 static struct opts_multi_pack_index {
 	const char *object_dir;
 	unsigned long batch_size;
-	int progress;
 	unsigned flags;
 } opts;
 
@@ -23,7 +22,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
+		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	git_config(git_default_config, NULL);
 
-	opts.progress = isatty(2);
+	if (isatty(2))
+		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
 			     builtin_multi_pack_index_usage, 0);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
-	if (opts.progress)
-		opts.flags |= MIDX_PROGRESS;
 
 	if (argc == 0)
 		usage_with_options(builtin_multi_pack_index_usage,
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#35 lore raw

Re: [PATCH v2 2/4] builtin/multi-pack-index.c: don't handle 'progress' separately

Quoted 5 lines · Taylor Blau
builtin/multi-pack-index.c +0 -1
@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	git_config(git_default_config, NULL);
 
-	opts.progress = isatty(2);

builtin/multi-pack-index.c int cmd_multi_pack_index(int argc, const char **argv, · line 42

Quoted 10 lines
builtin/multi-pack-index.c +2 -2
+	if (isatty(2))
+		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
 			     builtin_multi_pack_index_usage, 0);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
-	if (opts.progress)
-		opts.flags |= MIDX_PROGRESS;

Funnily enough we could also just do:

    opts.flags = isatty(2);

Since there's a grand total of one flag it knows about, and MIDX_PROGRESS is defined as 1.

Not the problem of this series really, just a nit: In efbc3aee08d (midx:
add MIDX_PROGRESS flag, 2019-10-21) we added this flag, and around the same time the similar commit-graph code got refactored to have an enum of flags in 5af80394521 (commit-graph: collapse parameters into flags, 2019-06-12).

I prefer the commit-graph way of having a clean boundary between the two a bit more, and then just setting a flag based on an OPT_BOOL...

#36 lore raw

Re: [PATCH v2 2/4] builtin/multi-pack-index.c: don't handle 'progress' separately

Quoted 22 lines · Ævar Arnfjörð Bjarmason
Quoted 5 lines · Taylor Blau
builtin/multi-pack-index.c +0 -1
@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	git_config(git_default_config, NULL);
 
-	opts.progress = isatty(2);

builtin/multi-pack-index.c int cmd_multi_pack_index(int argc, const char **argv, · line 42

Quoted 10 lines
builtin/multi-pack-index.c +2 -2
+	if (isatty(2))
+		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
 			     builtin_multi_pack_index_usage, 0);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
-	if (opts.progress)
-		opts.flags |= MIDX_PROGRESS;

Funnily enough we could also just do:

Quoted 4 lines
    opts.flags = isatty(2);

Since there's a grand total of one flag it knows about, and MIDX_PROGRESS is defined as 1.

:-). I have a handful of branches that add some new flags (including the original series I sent down-thread), so I'm not sure that I'm in favor of this (admittedly cute) hack.

Quoted 5 lines

Not the problem of this series really, just a nit: In efbc3aee08d (midx:
add MIDX_PROGRESS flag, 2019-10-21) we added this flag, and around the same time the similar commit-graph code got refactored to have an enum of flags in 5af80394521 (commit-graph: collapse parameters into flags, 2019-06-12).

Hmm. I don't really have a strong opinion either way. I'd like to avoid steering too far away from my original goal of multi-pack reverse indexes, at least for now...

Quoted 2 lines

I prefer the commit-graph way of having a clean boundary between the two a bit more, and then just setting a flag based on an OPT_BOOL...

Me too. But if you can part ways with it, it cuts down on the code duplication (since callers in the sub-commands don't have to set that bit on the flags themselves).

OTOH, we could keep half of this change and store the flags in the options structure in addition to the progress field, then set the appropriate bit in "flags" in cmd_builtin_multi_pack_index().

But I think at that point you're already sharing the flags field everywhere, so you're just as well off to have something like what's written in this patch here.

I don't have strong feelings either way.

Thanks,
Taylor

#37 lore raw

Re: [PATCH v2 2/4] builtin/multi-pack-index.c: don't handle 'progress' separately

Quoted 11 lines · Taylor Blau
Quoted 6 lines · Ævar Arnfjörð Bjarmason

Funnily enough we could also just do:

    opts.flags = isatty(2);

Since there's a grand total of one flag it knows about, and MIDX_PROGRESS is defined as 1.

:-). I have a handful of branches that add some new flags (including the original series I sent down-thread), so I'm not sure that I'm in favor of this (admittedly cute) hack.

It's also _wrong_ if the user passes in '--progress' but redirects stderr to a file. I don't know why someone would want to do that, but they could, and we honor that throughout all commands.

Thanks,
-Stolee

#38 lore raw

PATCH v2 3/4 builtin/multi-pack-index.c: define common usage with a macro

In reply to Taylor Blau

Factor out the usage message into pieces corresponding to each mode. This avoids options specific to one sub-command from being shared with another in the usage.

A subsequent commit will use these #define macros to have usage variables for each sub-command without duplicating their contents.

---
 builtin/multi-pack-index.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
builtin/multi-pack-index.c +16 -1
index c70f020d8f..eea498e026 100644
@@ -5,8 +5,23 @@
 #include "midx.h"
 #include "trace2.h"
 
+#define BUILTIN_MIDX_WRITE_USAGE \
+	N_("git multi-pack-index [<options>] write")
+
+#define BUILTIN_MIDX_VERIFY_USAGE \
+	N_("git multi-pack-index [<options>] verify")
+
+#define BUILTIN_MIDX_EXPIRE_USAGE \
+	N_("git multi-pack-index [<options>] expire")
+
+#define BUILTIN_MIDX_REPACK_USAGE \
+	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
+
 static char const * const builtin_multi_pack_index_usage[] = {
-	N_("git multi-pack-index [<options>] (write|verify|expire|repack --batch-size=<size>)"),
+	BUILTIN_MIDX_WRITE_USAGE,
+	BUILTIN_MIDX_VERIFY_USAGE,
+	BUILTIN_MIDX_EXPIRE_USAGE,
+	BUILTIN_MIDX_REPACK_USAGE,
 	NULL
 };
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#39 lore raw

PATCH v2 4/4 builtin/multi-pack-index.c: split sub-commands

In reply to Taylor Blau

Handle sub-commands of the 'git multi-pack-index' builtin (e.g., "write", "repack", etc.) separately from one another. This allows sub-commands with unique options, without forcing cmd_multi_pack_index() to reject invalid combinations itself.

This comes at the cost of some duplication and boilerplate. Luckily, the duplication is reduced to a minimum, since common options are shared among sub-commands due to a suggestion by Ævar. (Sub-commands do have to retain the common options, too, since this builtin accepts common options on either side of the sub-command).

Roughly speaking, cmd_multi_pack_index() parses options (including common ones), and stops at the first non-option, which is the sub-command. It then dispatches to the appropriate sub-command, which parses the remaining options (also including common options).

Unknown options are kept by the sub-commands in order to detect their presence (and complain that too many arguments were given).

---
 builtin/multi-pack-index.c | 131 ++++++++++++++++++++++++++++++-------
 1 file changed, 106 insertions(+), 25 deletions(-)
builtin/multi-pack-index.c +106 -25
index eea498e026..caf0248a98 100644
@@ -5,17 +5,33 @@
 #include "midx.h"
 #include "trace2.h"
 
+static char const * const builtin_multi_pack_index_write_usage[] = {
 #define BUILTIN_MIDX_WRITE_USAGE \
 	N_("git multi-pack-index [<options>] write")
+	BUILTIN_MIDX_WRITE_USAGE,
+	NULL
+};
 
+static char const * const builtin_multi_pack_index_verify_usage[] = {
 #define BUILTIN_MIDX_VERIFY_USAGE \
 	N_("git multi-pack-index [<options>] verify")
+	BUILTIN_MIDX_VERIFY_USAGE,
+	NULL
+};
 
+static char const * const builtin_multi_pack_index_expire_usage[] = {
 #define BUILTIN_MIDX_EXPIRE_USAGE \
 	N_("git multi-pack-index [<options>] expire")
+	BUILTIN_MIDX_EXPIRE_USAGE,
+	NULL
+};
 
+static char const * const builtin_multi_pack_index_repack_usage[] = {
 #define BUILTIN_MIDX_REPACK_USAGE \
 	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
+	BUILTIN_MIDX_REPACK_USAGE,
+	NULL
+};
 
 static char const * const builtin_multi_pack_index_usage[] = {
 	BUILTIN_MIDX_WRITE_USAGE,
@@ -31,25 +47,99 @@ static struct opts_multi_pack_index {
 	unsigned flags;
 } opts;
 
-int cmd_multi_pack_index(int argc, const char **argv,
-			 const char *prefix)
+static struct option common_opts[] = {
+	OPT_FILENAME(0, "object-dir", &opts.object_dir,
+	  N_("object directory containing set of packfile and pack-index pairs")),
+	OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	OPT_END(),
+};
+
+static struct option *add_common_options(struct option *prev)
 {
-	static struct option builtin_multi_pack_index_options[] = {
-		OPT_FILENAME(0, "object-dir", &opts.object_dir,
-		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	struct option *with_common = parse_options_concat(common_opts, prev);
+	free(prev);
+	return with_common;
+}
+
+static int cmd_multi_pack_index_write(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_write_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_write_usage,
+				   options);
+
+	return write_midx_file(opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_verify(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_verify_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_verify_usage,
+				   options);
+
+	return verify_midx_file(the_repository, opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_expire(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_expire_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_expire_usage,
+				   options);
+
+	return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_repack(int argc, const char **argv)
+{
+	struct option *options;
+	static struct option builtin_multi_pack_index_repack_options[] = {
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
 	};
 
+	options = parse_options_dup(builtin_multi_pack_index_repack_options);
+	options = add_common_options(options);
+
+	argc = parse_options(argc, argv, NULL,
+			     options,
+			     builtin_multi_pack_index_repack_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_repack_usage,
+				   options);
+
+	return midx_repack(the_repository, opts.object_dir,
+			   (size_t)opts.batch_size, opts.flags);
+}
+
+int cmd_multi_pack_index(int argc, const char **argv,
+			 const char *prefix)
+{
+	struct option *builtin_multi_pack_index_options = common_opts;
+
 	git_config(git_default_config, NULL);
 
 	if (isatty(2))
 		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
-			     builtin_multi_pack_index_usage, 0);
+			     builtin_multi_pack_index_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
@@ -58,25 +148,16 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		usage_with_options(builtin_multi_pack_index_usage,
 				   builtin_multi_pack_index_options);
 
-	if (argc > 1) {
-		die(_("too many arguments"));
-		return 1;
-	}
-
 	trace2_cmd_mode(argv[0]);
 
 	if (!strcmp(argv[0], "repack"))
-		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, opts.flags);
-	if (opts.batch_size)
-		die(_("--batch-size option is only for 'repack' subcommand"));
-
-	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
-
-	die(_("unrecognized subcommand: %s"), argv[0]);
+		return cmd_multi_pack_index_repack(argc, argv);
+	else if (!strcmp(argv[0], "write"))
+		return cmd_multi_pack_index_write(argc, argv);
+	else if (!strcmp(argv[0], "verify"))
+		return cmd_multi_pack_index_verify(argc, argv);
+	else if (!strcmp(argv[0], "expire"))
+		return cmd_multi_pack_index_expire(argc, argv);
+	else
+		die(_("unrecognized subcommand: %s"), argv[0]);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#40 lore raw

Re: [PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands

Quoted 1 line · Taylor Blau

trace2_cmd_mode(argv[0]);

Not a new issue, but curious that in the commit-graph.c code we'll first validate, but here write garbage into the trace2_cmd_mode() before potentially dying.

Quoted 14 lines
(quoted diff) +0 -12
 	if (!strcmp(argv[0], "repack"))
-		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, opts.flags);
-	if (opts.batch_size)
-		die(_("--batch-size option is only for 'repack' subcommand"));
-
-	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
-

Review

Quoted 10 lines
(quoted diff) +9 -1
-	die(_("unrecognized subcommand: %s"), argv[0]);
+		return cmd_multi_pack_index_repack(argc, argv);
+	else if (!strcmp(argv[0], "write"))
+		return cmd_multi_pack_index_write(argc, argv);
+	else if (!strcmp(argv[0], "verify"))
+		return cmd_multi_pack_index_verify(argc, argv);
+	else if (!strcmp(argv[0], "expire"))
+		return cmd_multi_pack_index_expire(argc, argv);
+	else
+		die(_("unrecognized subcommand: %s"), argv[0]);

I realize this is the existing behavior, but let's just make this die() be the usage_with_options() we emit above in this case?

So maybe this on top?

builtin/multi-pack-index.c +14 -11
index caf0248a98..6f9223d538 100644
@@ -65,6 +65,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_write_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -79,6 +81,8 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_verify_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -93,6 +97,8 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_expire_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -112,6 +118,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)
 		OPT_END(),
 	};
 
+	trace2_cmd_mode(argv[0]);
+
 	options = parse_options_dup(builtin_multi_pack_index_repack_options);
 	options = add_common_options(options);
 
@@ -144,20 +152,15 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
 
-	if (argc == 0)
-		usage_with_options(builtin_multi_pack_index_usage,
-				   builtin_multi_pack_index_options);
-
-	trace2_cmd_mode(argv[0]);
-
-	if (!strcmp(argv[0], "repack"))
+	if (argc && !strcmp(argv[0], "repack"))
 		return cmd_multi_pack_index_repack(argc, argv);
-	else if (!strcmp(argv[0], "write"))
+	else if (argc && !strcmp(argv[0], "write"))
 		return cmd_multi_pack_index_write(argc, argv);
-	else if (!strcmp(argv[0], "verify"))
+	else if (argc && !strcmp(argv[0], "verify"))
 		return cmd_multi_pack_index_verify(argc, argv);
-	else if (!strcmp(argv[0], "expire"))
+	else if (argc && !strcmp(argv[0], "expire"))
 		return cmd_multi_pack_index_expire(argc, argv);
 	else
-		die(_("unrecognized subcommand: %s"), argv[0]);
+		usage_with_options(builtin_multi_pack_index_usage,
+				   builtin_multi_pack_index_options);
 }
#41 lore raw

Re: [PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands

Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line · Taylor Blau

trace2_cmd_mode(argv[0]);

Not a new issue, but curious that in the commit-graph.c code we'll first validate, but here write garbage into the trace2_cmd_mode() before potentially dying.

Yeah, that's a good catch.

Quoted 4 lines

I realize this is the existing behavior, but let's just make this die() be the usage_with_options() we emit above in this case?

So maybe this on top?

I split this into two patches: one to move the trace2_cmd_mode() calls around, and another to replace the final 'die()' with the usage text.

Like I said in my review of your patches to the commit-graph builtin here:

https://lore.kernel.org/git/[email protected]/

I don't find the 'if (argc && ...)' style more readable, so the second patch looks like this instead:

builtin/multi-pack-index.c +2 -1
index 5ab80bc722..ce4f1a0bcb 100644
@@ -165,5 +165,6 @@ int cmd_multi_pack_index(int argc, const char **argv,
        else if (!strcmp(argv[0], "expire"))
                return cmd_multi_pack_index_expire(argc, argv);
        else
-               die(_("unrecognized subcommand: %s"), argv[0]);
+               usage_with_options(builtin_multi_pack_index_usage,
+                                  builtin_multi_pack_index_options);
 }

Is it OK if I use your Signed-off-by on both of those two new patches?

Thanks,
Taylor

#42 lore raw

Re: [PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands

Quoted 12 lines · Taylor Blau
Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line · Taylor Blau

trace2_cmd_mode(argv[0]);

Not a new issue, but curious that in the commit-graph.c code we'll first validate, but here write garbage into the trace2_cmd_mode() before potentially dying.

Yeah, that's a good catch.

Quoted 7 lines
Quoted 4 lines

I realize this is the existing behavior, but let's just make this die() be the usage_with_options() we emit above in this case?

So maybe this on top?

I split this into two patches: one to move the trace2_cmd_mode() calls around, and another to replace the final 'die()' with the usage text.

Thanks for picking it up.

Quoted 7 lines

Like I said in my review of your patches to the commit-graph builtin here:

https://lore.kernel.org/git/[email protected]/

I don't find the 'if (argc && ...)' style more readable, so the second patch looks like this instead:

*Nod* FWIW (and this is getting way to nit-y) I don't disagree with you about the "argc &&" being not very readable,

I just lean more on the side of getting rid of duplicate branches, you'll still need the if (!argc) usage(...) case above without that pattern, or some replacement for it.

But we can have our cake (not re-check argc all the time) and eat it too (not copy/paste usage_with_options()). Isn't it beautiful?

    diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
    index caf0248a98..7ff50439f8 100644
    --- a/builtin/multi-pack-index.c
    +++ b/builtin/multi-pack-index.c
    @@ -144,12 +144,8 @@ int cmd_multi_pack_index(int argc, const char **argv,
     	if (!opts.object_dir)
     		opts.object_dir = get_object_directory();
     
    -	if (argc == 0)
    -		usage_with_options(builtin_multi_pack_index_usage,
    -				   builtin_multi_pack_index_options);
    -
    -	trace2_cmd_mode(argv[0]);
    -
    +	if (!argc)
    +		goto usage;
     	if (!strcmp(argv[0], "repack"))
     		return cmd_multi_pack_index_repack(argc, argv);
     	else if (!strcmp(argv[0], "write"))
    @@ -159,5 +155,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
     	else if (!strcmp(argv[0], "expire"))
     		return cmd_multi_pack_index_expire(argc, argv);
     	else
    -		die(_("unrecognized subcommand: %s"), argv[0]);
    +usage:
    +		usage_with_options(builtin_multi_pack_index_usage,
    +				   builtin_multi_pack_index_options);
     }

:)

Quoted 13 lines
builtin/multi-pack-index.c +2 -1
index 5ab80bc722..ce4f1a0bcb 100644
@@ -165,5 +165,6 @@ int cmd_multi_pack_index(int argc, const char **argv,
        else if (!strcmp(argv[0], "expire"))
                return cmd_multi_pack_index_expire(argc, argv);
        else
-               die(_("unrecognized subcommand: %s"), argv[0]);
+               usage_with_options(builtin_multi_pack_index_usage,
+                                  builtin_multi_pack_index_options);
 }
Quoted 1 line

Is it OK if I use your Signed-off-by on both of those two new patches?

Yes please, should have included it to begin with.

#43 lore raw

Re: [PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands

Quoted 4 lines · Ævar Arnfjörð Bjarmason
Quoted 2 lines

I split this into two patches: one to move the trace2_cmd_mode() calls around, and another to replace the final 'die()' with the usage text.

Thanks for picking it up.

Of course. This has been quite a fun digression :-).

Quoted 11 lines
Quoted 7 lines

Like I said in my review of your patches to the commit-graph builtin here:

https://lore.kernel.org/git/[email protected]/

I don't find the 'if (argc && ...)' style more readable, so the second patch looks like this instead:

*Nod* FWIW (and this is getting way to nit-y) I don't disagree with you about the "argc &&" being not very readable,

Quoted 6 lines

I just lean more on the side of getting rid of duplicate branches, you'll still need the if (!argc) usage(...) case above without that pattern, or some replacement for it.

But we can have our cake (not re-check argc all the time) and eat it too (not copy/paste usage_with_options()). Isn't it beautiful?

Heh; I'm not sure that I'd call adding a goto "beautiful", but I actually do find this one more readable. I dunno, honestly, I'm happy to squash it in to the last commit on top, but honestly I don't really care strongly one way or another ;).

Quoted 3 lines
Quoted 1 line

Is it OK if I use your Signed-off-by on both of those two new patches?

Yes please, should have included it to begin with.

Thanks, and no worries.

Thanks,
Taylor

#44 lore raw

Re: [PATCH v2 0/4] midx: split out sub-commands

In reply to Taylor Blau
Quoted 9 lines · Taylor Blau

Here's a few patches that we could add to the beginning of this series, or queue up separately.

I think that these are all fairly straightforward, but it would be good to have Ævar take a look and make sure I'm not doing anything wrong here.

I'll plan to send a v2 of the reverse index series in a few days with these four new patches at the beginning.

Thanks, both, for cleaning up a mess I made as a new contributor. These patches have been enlightening and definitely move the code into a cleaner and more extensible direction. Thanks!

-Stolee

#45 lore raw

Re: [PATCH v2 0/4] midx: split out sub-commands

Quoted 8 lines · Derrick Stolee
Quoted 7 lines · Taylor Blau

Here's a few patches that we could add to the beginning of this series, or queue up separately.

I think that these are all fairly straightforward, but it would be good to have Ævar take a look and make sure I'm not doing anything wrong here.

Quoted 6 lines
Quoted 2 lines

I'll plan to send a v2 of the reverse index series in a few days with these four new patches at the beginning.

Thanks, both, for cleaning up a mess I made as a new contributor. These patches have been enlightening and definitely move the code into a cleaner and more extensible direction. Thanks!

There was hardly a mess to clean-up, and clearly this pattern is new to me, too :).

I'm planning on resubmitting my tb/reverse-midx series as soon as it gets another set of reviewer eyes with these four or five patches as new at the beginning.

I do wonder about the merge conflicts caused between this and your chunk-format API series. I'd rather not create such conflicts for Junio, and last I recall there were still some outstanding comments on that series. So long as you don't think that you resolving those comments would cause new conflicts, I would assume that Junio's rerere cache would make applying both easy enough.

If you do think it would cause new conflicts, it may make sense for you to rebase your branch on mine, but I'm not sure if that's something you'd want to do or not.

Quoted 1 line

-Stolee

Thanks,
Taylor

#46 lore raw

PATCH 3/9 midx: don't free midx_name early

In reply to Taylor Blau

A subsequent patch will need to refer back to 'midx_name' later on in the function. In fact, this variable is already free()'d later on, so this makes the later free() no longer redundant.

---
 midx.c | 1 -
 1 file changed, 1 deletion(-)
midx.c +0 -1
index 064670c0c0..34fb9de3f3 100644
@@ -995,7 +995,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
-	FREE_AND_NULL(midx_name);
 
 	if (packs.m)
 		close_midx(packs.m);
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#47 lore raw

PATCH 4/9 midx: keep track of the checksum

In reply to Taylor Blau

write_midx_internal() uses a hashfile to write the multi-pack index, but discards its checksum. This makes sense, since nothing that takes place after writing the MIDX cares about its checksum.

That is about to change in a subsequent patch, when the optional reverse index corresponding to the MIDX will want to include the MIDX's checksum.

Store the checksum of the MIDX in preparation for that.

---
 midx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
midx.c +2 -1
index 34fb9de3f3..6e47c726af 100644
@@ -837,6 +837,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 {
 	unsigned char cur_chunk, num_chunks = 0;
 	char *midx_name;
+	unsigned char midx_hash[GIT_MAX_RAWSZ];
 	uint32_t i;
 	struct hashfile *f = NULL;
 	struct lock_file lk;
@@ -1098,7 +1099,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 		    written,
 		    chunk_offsets[num_chunks]);
 
-	finalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	commit_lock_file(&lk);
 
 cleanup:
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#48 lore raw

Re: [PATCH 4/9] midx: keep track of the checksum

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 840

Quoted 1 line · Taylor Blau
midx.c +1 -0
+	unsigned char midx_hash[GIT_MAX_RAWSZ];

I was initially thinking we should use something like 'struct object_id' here, but the hash we are storing doesn't correspond to an object, which would be confusing. I suppose this is the most correct thing to do.

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 1102

Quoted 2 lines
midx.c +1 -1
-	finalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);

And this is of course correct. The issue will be what happens to 'midx_hash' in the later changes. Will keep reading.

Thanks,
-Stolee

#49 lore raw

Re: [PATCH 4/9] midx: keep track of the checksum

Quoted 8 lines · Derrick Stolee

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 840

Quoted 1 line · Taylor Blau
midx.c +1 -0
+	unsigned char midx_hash[GIT_MAX_RAWSZ];

I was initially thinking we should use something like 'struct object_id' here, but the hash we are storing doesn't correspond to an object, which would be confusing. I suppose this is the most correct thing to do.

Yeah. There are a number of places that abuse the unsigned char array inside of object_id, but there's no good reason to.

Thanks,
Taylor

#50 lore raw

PATCH 5/9 midx: make some functions non-static

In reply to Taylor Blau

In a subsequent commit, pack-revindex.c will become responsible for sorting a list of objects in the "MIDX pack order" (which will be defined in the following patch). To do so, it will need to be know the pack identifier and offset within that pack for each object in the MIDX.

The MIDX code already has functions for doing just that (nth_midxed_offset() and nth_midxed_pack_int_id()), but they are statically declared.

Since there is no reason that they couldn't be exposed publicly, and because they are already doing exactly what the caller in pack-revindex.c will want, expose them publicly so that they can be reused there.

---
 midx.c | 4 ++--
 midx.h | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)
midx.c +2 -2
index 6e47c726af..bf258c4fde 100644
@@ -260,7 +260,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,
 	return oid;
 }
 
-static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
 {
 	const unsigned char *offset_data;
 	uint32_t offset32;
@@ -279,7 +279,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
 	return offset32;
 }
 
-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
 {
 	return get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);
 }
midx.h +2 -0
index e7fea61109..93bd68189e 100644
@@ -40,6 +40,8 @@ struct multi_pack_index {
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);
+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);
 struct object_id *nth_midxed_object_oid(struct object_id *oid,
 					struct multi_pack_index *m,
 					uint32_t n);
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#51 lore raw

PATCH 6/9 Documentation/technical: describe multi-pack reverse indexes

In reply to Taylor Blau

As a prerequisite to implementing multi-pack bitmaps, motivate and describe the format and ordering of the multi-pack reverse index.

The subsequent patch will implement reading this format, and the patch after that will implement writing it while producing a multi-pack index.

---
 Documentation/technical/pack-format.txt | 83 +++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
Documentation/technical/pack-format.txt +83 -0
index 8833b71c8b..a14722f119 100644
@@ -376,3 +376,86 @@ CHUNK DATA:
 TRAILER:
 
 	Index checksum of the above contents.
+
+== multi-pack-index reverse indexes
+
+Similar to the pack-based reverse index, the multi-pack index can also
+be used to generate a reverse index.
+
+Instead of mapping between offset, pack-, and index position, this
+reverse index maps between an object's position within the midx, and
+that object's position within a pseudo-pack that the midx describes.
+Crucially, the objects' positions within this pseudo-pack are the same
+as their bit positions in a multi-pack reachability bitmap.
+
+As a motivating example, consider the multi-pack reachability bitmap
+(which does not yet exist, but is what we are building towards here). We
+need each bit to correspond to an object covered by the midx, and we
+need to be able to convert bit positions back to index positions (from
+which we can get the oid, etc).
+
+One solution is to let each bit position in the index correspond to
+the same position in the oid-sorted index stored by the midx. But
+because oids are effectively random, there resulting reachability
+bitmaps would have no locality, and thus compress poorly. (This is the
+reason that single-pack bitmaps use the pack ordering, and not the .idx
+ordering, for the same purpose.)
+
+So we'd like to define an ordering for the whole midx based around
+pack ordering. We can think of it as a pseudo-pack created by the
+concatenation of all of the packs in the midx. E.g., if we had a midx
+with three packs (a, b, c), with 10, 15, and 20 objects respectively, we
+can imagine an ordering of the objects like:
+
+    |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|
+
+where the ordering of the packs is defined by the midx's pack list,
+and then the ordering of objects within each pack is the same as the
+order in the actual packfile.
+
+Given the list of packs and their counts of objects, you can
+na&iuml;vely reconstruct that pseudo-pack ordering (e.g., the object at
+position 27 must be (c,1) because packs "a" and "b" consumed 25 of the
+slots). But there's a catch. Objects may be duplicated between packs, in
+which case the midx only stores one pointer to the object (and thus we'd
+want only one slot in the bitmap).
+
+Callers could handle duplicates themselves by reading objects in order
+of their bit-position, but that's linear in the number of objects, and
+much too expensive for ordinary bitmap lookups. Building a reverse index
+solves this, since it is the logical inverse of the index, and that
+index has already removed duplicates. But, building a reverse index on
+the fly can be expensive. Since we already have an on-disk format for
+pack-based reverse indexes, let's reuse it for the midx's pseudo-pack,
+too.
+
+Objects from the midx are ordered as follows to string together the
+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected
+by the midx, and define an ordering of packs based on their numeric ID
+(as stored by the midx). Let _offset(o)_ return the object offset of _o_
+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:
+
+  - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other
+    is not, then the preferred one sorts first.
++
+(This is a detail that allows the midx bitmap to determine which
+pack should be used by the pack-reuse mechanism, since it can ask
+the midx for the pack containing the object at bit position 0).
+
+  - If _pack(o~1~) &ne; pack(o~2~)_, then sort the two objects in
+    descending order based on the pack ID.
+
+  - Otherwise, _pack(o~1~) &equals; pack(o~2~)_, and the objects are
+    sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly
+    when _offset(o~1~) &lt; offset(o~2~)_).
+
+In short, a midx's pseudo-pack is the de-duplicated concatenation of
+objects in packs stored by the midx, laid out in pack order, and the
+packs arranged in midx order (with the preferred pack coming first).
+
+Finally, note that the midx's reverse index is not stored as a chunk in
+the multi-pack-index itself. This is done because the reverse index
+includes the checksum of the pack or midx to which it belongs, which
+makes it impossible to write in the midx. To avoid races when rewriting
+the midx, a midx reverse index includes the midx's checksum in its
+filename (e.g., `multi-pack-index-xyz.rev`).
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#52 lore raw

Re: [PATCH 6/9] Documentation/technical: describe multi-pack reverse indexes

On 2/10/21 6:03 PM, Taylor Blau wrote:> +Instead of mapping between offset, pack-, and index position, this

The "pack-," should be paired with "index-position" or drop the hyphen in both cases. Perhaps just be explicit, especially since "position" doesn't match with "offset":

Instead of mapping between pack offset, pack position, and index position, ...

Documentation/technical/pack-format.txt CHUNK DATA: · line 387

Quoted 2 lines
Documentation/technical/pack-format.txt +2 -0
+reverse index maps between an object's position within the midx, and
+that object's position within a pseudo-pack that the midx describes.

nit: use multi-pack-index or MIDX, not lower-case 'midx'.

Documentation/technical/pack-format.txt CHUNK DATA: · line 395

Quoted 8 lines
Documentation/technical/pack-format.txt +8 -0
+Crucially, the objects' positions within this pseudo-pack are the same
+as their bit positions in a multi-pack reachability bitmap.
+
+As a motivating example, consider the multi-pack reachability bitmap
+(which does not yet exist, but is what we are building towards here). We
+need each bit to correspond to an object covered by the midx, and we
+need to be able to convert bit positions back to index positions (from
+which we can get the oid, etc).

These paragraphs are awkward. Instead of operating in the hypothetical world of reachability bitmaps, focus on the fact that bitmaps need a bidirectional mapping between "bit position" and an object ID.

Here is an attempt to reword some of the context you are using here. Feel free to take as much or as little as you want.

The multi-pack-index stores the object IDs in lexicographical order (lex-order) to allow binary search. To allow compressible reachability bitmaps to pair with a multi-pack-index, a different ordering is required. When paired with a single packfile, the order used is the object order within the packfile (called the pack-order). Construct a "pseudo-pack" by concatenating all tracked packfiles in the multi-pack-index. We now need a mapping between the lex-order and the pseudo-pack-order.

Quoted 55 lines
Documentation/technical/pack-format.txt +55 -0
+One solution is to let each bit position in the index correspond to
+the same position in the oid-sorted index stored by the midx. But
+because oids are effectively random, there resulting reachability
+bitmaps would have no locality, and thus compress poorly. (This is the
+reason that single-pack bitmaps use the pack ordering, and not the .idx
+ordering, for the same purpose.)
+
+So we'd like to define an ordering for the whole midx based around
+pack ordering. We can think of it as a pseudo-pack created by the
+concatenation of all of the packs in the midx. E.g., if we had a midx
+with three packs (a, b, c), with 10, 15, and 20 objects respectively, we
+can imagine an ordering of the objects like:
+
+    |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|
+
+where the ordering of the packs is defined by the midx's pack list,
+and then the ordering of objects within each pack is the same as the
+order in the actual packfile.
+
+Given the list of packs and their counts of objects, you can
+na&iuml;vely reconstruct that pseudo-pack ordering (e.g., the object at
+position 27 must be (c,1) because packs "a" and "b" consumed 25 of the
+slots). But there's a catch. Objects may be duplicated between packs, in
+which case the midx only stores one pointer to the object (and thus we'd
+want only one slot in the bitmap).
+
+Callers could handle duplicates themselves by reading objects in order
+of their bit-position, but that's linear in the number of objects, and
+much too expensive for ordinary bitmap lookups. Building a reverse index
+solves this, since it is the logical inverse of the index, and that
+index has already removed duplicates. But, building a reverse index on
+the fly can be expensive. Since we already have an on-disk format for
+pack-based reverse indexes, let's reuse it for the midx's pseudo-pack,
+too.
+
+Objects from the midx are ordered as follows to string together the
+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected
+by the midx, and define an ordering of packs based on their numeric ID
+(as stored by the midx). Let _offset(o)_ return the object offset of _o_
+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:
+
+  - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other
+    is not, then the preferred one sorts first.
++
+(This is a detail that allows the midx bitmap to determine which
+pack should be used by the pack-reuse mechanism, since it can ask
+the midx for the pack containing the object at bit position 0).
+
+  - If _pack(o~1~) &ne; pack(o~2~)_, then sort the two objects in
+    descending order based on the pack ID.
+
+  - Otherwise, _pack(o~1~) &equals; pack(o~2~)_, and the objects are
+    sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly
+    when _offset(o~1~) &lt; offset(o~2~)_).
+

Documentation/technical/pack-format.txt CHUNK DATA: · line 461

Quoted 10 lines
Documentation/technical/pack-format.txt +10 -0
+In short, a midx's pseudo-pack is the de-duplicated concatenation of
+objects in packs stored by the midx, laid out in pack order, and the
+packs arranged in midx order (with the preferred pack coming first).
+
+Finally, note that the midx's reverse index is not stored as a chunk in
+the multi-pack-index itself. This is done because the reverse index
+includes the checksum of the pack or midx to which it belongs, which
+makes it impossible to write in the midx. To avoid races when rewriting
+the midx, a midx reverse index includes the midx's checksum in its
+filename (e.g., `multi-pack-index-xyz.rev`).

The rest of these details make sense and sufficiently motivate the ordering, once the concept is clear.

Thanks,
-Stolee

#53 lore raw

Re: [PATCH 6/9] Documentation/technical: describe multi-pack reverse indexes

Quoted 1 line · Derrick Stolee

nit: use multi-pack-index or MIDX, not lower-case 'midx'.

Thanks.

Quoted 4 lines
Quoted 4 lines
Documentation/technical/pack-format.txt +4 -0
+Crucially, the objects' positions within this pseudo-pack are the same
+as their bit positions in a multi-pack reachability bitmap.
+
+As a motivating example, consider the multi-pack reachability bitmap
Quoted 8 lines

Documentation/technical/pack-format.txt CHUNK DATA: · line 395

Quoted 4 lines
Documentation/technical/pack-format.txt +4 -0
+(which does not yet exist, but is what we are building towards here). We
+need each bit to correspond to an object covered by the midx, and we
+need to be able to convert bit positions back to index positions (from
+which we can get the oid, etc).

These paragraphs are awkward. Instead of operating in the hypothetical world of reachability bitmaps, focus on the fact that bitmaps need a bidirectional mapping between "bit position" and an object ID.

Hmm. I could buy that these paragraphs are awkward, but I'm not sure that what you proposed makes it less so.

I may be a bad person to judge what you wrote, since I am familiar with the details of what it's describing. But my thoughts on that second and third paragraph are basically:

- define the valid orderings we might consider objects in a MIDX by, indicating which of those orderings we're going to use for multi-pack bitmaps

- motivate the need for a mapping between lexicographic order and pseudo-pack order

Quoted 11 lines

Here is an attempt to reword some of the context you are using here. Feel free to take as much or as little as you want.

The multi-pack-index stores the object IDs in lexicographical order (lex-order) to allow binary search. To allow compressible reachability bitmaps to pair with a multi-pack-index, a different ordering is required. When paired with a single packfile, the order used is the object order within the packfile (called the pack-order). Construct a "pseudo-pack" by concatenating all tracked packfiles in the multi-pack-index. We now need a mapping between the lex-order and the pseudo-pack-order.

I struggled with what you wrote because I couldn't seem to neatly place/replace that paragraph in with the existing text without referring to yet-undefined concepts.

Maybe the confusion lies in the fact that we stray too far from the point in the second and third paragraphs. What if we reordered the second, third, and fourth paragraph like this:

Instead of mapping between offset, pack-, and index position, this reverse index maps between an object's position within the MIDX, and that object's position within a pseudo-pack that the MIDX describes.

To clarify these three orderings, consider a multi-pack reachability bitmap (which does not yet exist, but is what we are building towards here). Each bit needs to correspond to an object in the MIDX, and so we need an efficient mapping from bit position to MIDX position.

One solution is to let bits occupy the same position in the oid-sorted index stored by the MIDX. But because oids are effectively random, there resulting reachability bitmaps would have no locality, and thus compress poorly. (This is the reason that single-pack bitmaps use the pack ordering, and not the .idx ordering, for the same purpose.)

So we'd like to define an ordering for the whole MIDX based around pack ordering, which has far better locality (and thus compresses more efficiently). We can think of a pseudo-pack created by the concatenation of all of the packs in the MIDX. E.g., if we had a MIDX with three packs (a, b, c), with 10, 15, and 20 objects respectively, we can imagine an ordering of the objects like:

Quoted 7 lines

[snip]

The rest of these details make sense and sufficiently motivate the ordering, once the concept is clear.

Thanks,
-Stolee

Thanks,
Taylor

#54 lore raw

PATCH 7/9 pack-revindex: read multi-pack reverse indexes

In reply to Taylor Blau

Implement reading for multi-pack reverse indexes, as described in the previous patch.

Note that these functions don't yet have any callers, and won't until multi-pack reachability bitmaps are introduced in a later patch series. In the meantime, this patch implements some of the infrastructure necessary to support multi-pack bitmaps.

There are three new functions exposed by the revindex API:

- load_midx_revindex(): loads the reverse index corresponding to the given multi-pack index.

- midx_to_pack_pos() and pack_pos_to_midx(): these convert between the multi-pack index and pseudo-pack order.

load_midx_revindex() and pack_pos_to_midx() are both relatively straightforward.

load_midx_revindex() needs a few functions to be exposed from the midx API. One to get the checksum of a midx, and another to get the .rev's filename. Similar to recent changes in the packed_git struct, three new fields are added to the multi_pack_index struct: one to keep track of the size, one to keep track of the mmap'd pointer, and another to point past the header and at the reverse index's data.

pack_pos_to_midx() simply reads the corresponding entry out of the table.

midx_to_pack_pos() is the trickiest, since it needs to find an object's position in the psuedo-pack order, but that order can only be recovered in the .rev file itself. This mapping can be implemented with a binary search, but note that the thing we're binary searching over isn't an array, but rather a _permutation_.

So, when comparing two items, it's helpful to keep in mind the difference. Instead of a traditional binary search, where you are comparing two things directly, here we're comparing a (pack, offset) tuple with an index into the multi-pack index. That index describes another (pack, offset) tuple, and it is _those_ two tuples that are compared.

---
 midx.c          |  11 +++++
 midx.h          |   6 +++
 pack-revindex.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++
 pack-revindex.h |  46 ++++++++++++++++++++
 packfile.c      |   3 ++
 5 files changed, 178 insertions(+)
midx.c +11 -0
index bf258c4fde..12bfce8bb1 100644
@@ -48,11 +48,22 @@ static uint8_t oid_version(void)
 	}
 }
 
+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)
+{
+	return m->data + m->data_len - the_hash_algo->rawsz;
+}
+
 static char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }
 
+char *get_midx_rev_filename(struct multi_pack_index *m)
+{
+	return xstrfmt("%s/pack/multi-pack-index-%s.rev",
+		       m->object_dir, hash_to_hex(get_midx_checksum(m)));
+}
+
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)
 {
 	struct multi_pack_index *m = NULL;
midx.h +6 -0
index 93bd68189e..0a8294d2ee 100644
@@ -15,6 +15,10 @@ struct multi_pack_index {
 	const unsigned char *data;
 	size_t data_len;
 
+	const uint32_t *revindex_data;
+	const uint32_t *revindex_map;
+	size_t revindex_len;
+
 	uint32_t signature;
 	unsigned char version;
 	unsigned char hash_len;
@@ -37,6 +41,8 @@ struct multi_pack_index {
 
 #define MIDX_PROGRESS     (1 << 0)
 
+char *get_midx_rev_filename(struct multi_pack_index *m);
+
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
pack-revindex.c +112 -0

127 lines not on this page · raw message

pack-revindex.h +46 -0

68 lines not on this page · raw message

packfile.c +3 -0
index 1fec12ac5f..82623e0cb4 100644
@@ -862,6 +862,9 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 
 	if (!strcmp(file_name, "multi-pack-index"))
 		return;
+	if (starts_with(file_name, "multi-pack-index") &&
+	    ends_with(file_name, ".rev"))
+		return;
 	if (ends_with(file_name, ".idx") ||
 	    ends_with(file_name, ".rev") ||
 	    ends_with(file_name, ".pack") ||
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#55 lore raw

Re: [PATCH 7/9] pack-revindex: read multi-pack reverse indexes

Quoted 56 lines · Taylor Blau

Implement reading for multi-pack reverse indexes, as described in the previous patch.

Note that these functions don't yet have any callers, and won't until multi-pack reachability bitmaps are introduced in a later patch series. In the meantime, this patch implements some of the infrastructure necessary to support multi-pack bitmaps.

There are three new functions exposed by the revindex API:

- load_midx_revindex(): loads the reverse index corresponding to the given multi-pack index.

- midx_to_pack_pos() and pack_pos_to_midx(): these convert between the multi-pack index and pseudo-pack order.

load_midx_revindex() and pack_pos_to_midx() are both relatively straightforward.

load_midx_revindex() needs a few functions to be exposed from the midx API. One to get the checksum of a midx, and another to get the .rev's filename. Similar to recent changes in the packed_git struct, three new fields are added to the multi_pack_index struct: one to keep track of the size, one to keep track of the mmap'd pointer, and another to point past the header and at the reverse index's data.

pack_pos_to_midx() simply reads the corresponding entry out of the table.

midx_to_pack_pos() is the trickiest, since it needs to find an object's position in the psuedo-pack order, but that order can only be recovered in the .rev file itself. This mapping can be implemented with a binary search, but note that the thing we're binary searching over isn't an array, but rather a _permutation_.

So, when comparing two items, it's helpful to keep in mind the difference. Instead of a traditional binary search, where you are comparing two things directly, here we're comparing a (pack, offset) tuple with an index into the multi-pack index. That index describes another (pack, offset) tuple, and it is _those_ two tuples that are compared.

---
 midx.c          |  11 +++++
 midx.h          |   6 +++
 pack-revindex.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++
 pack-revindex.h |  46 ++++++++++++++++++++
 packfile.c      |   3 ++
 5 files changed, 178 insertions(+)
midx.c +0 -0
index bf258c4fde..12bfce8bb1 100644

midx.c static uint8_t oid_version(void) · line 53

Quoted 6 lines
midx.c +3 -0
@@ -48,11 +48,22 @@ static uint8_t oid_version(void)
 	}
 }
 
+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)
+{
+	return m->data + m->data_len - the_hash_algo->rawsz;

'struct multi_pack_index' has a 'hash_len' member that you could use here. It would allow a different hash length in the stored file than the one required by the repository. Except...

midx.c static uint8_t oid_version(void) · line 64

Quoted 11 lines
midx.c +6 -0
+}
+
 static char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }
 
+char *get_midx_rev_filename(struct multi_pack_index *m)
+{
+	return xstrfmt("%s/pack/multi-pack-index-%s.rev",
+		       m->object_dir, hash_to_hex(get_midx_checksum(m)));

...this assumes the hash is of the same length as the_hash_algo, so you are doing the right thing. Currently, I think we check that 'm->hash_len == the_hash_algo->rawsz' on load. We'll need to check this again later when in the transition phase of the new hash work.

(No changes are needed to your patch.)

Thanks,
-Stolee

#56 lore raw

Re: [PATCH 7/9] pack-revindex: read multi-pack reverse indexes

Quoted 7 lines · Derrick Stolee

...this assumes the hash is of the same length as the_hash_algo, so you are doing the right thing. Currently, I think we check that 'm->hash_len == the_hash_algo->rawsz' on load. We'll need to check this again later when in the transition phase of the new hash work.

(No changes are needed to your patch.)

All makes sense, thanks.

Taylor

#57 lore raw

Re: [PATCH 7/9] pack-revindex: read multi-pack reverse indexes

In reply to Taylor Blau

pack-revindex.c line 6

Quoted 9 lines · Taylor Blau
pack-revindex.c +1 -0
index 83fe4de773..da4101f4b2 100644
@@ -3,6 +3,7 @@
 #include "object-store.h"
 #include "packfile.h"
 #include "config.h"
+#include "midx.h"

This seems to assume that the topic tb/pack-revindex-on-disk is already there?

Just trying to establish what dependencies the bunch of topics have among themselves.

Thanks.

#58 lore raw

Re: [PATCH 7/9] pack-revindex: read multi-pack reverse indexes

Quoted 14 lines · Junio C Hamano

pack-revindex.c line 6

Quoted 9 lines · Taylor Blau
pack-revindex.c +1 -0
index 83fe4de773..da4101f4b2 100644
@@ -3,6 +3,7 @@
 #include "object-store.h"
 #include "packfile.h"
 #include "config.h"
+#include "midx.h"

This seems to assume that the topic tb/pack-revindex-on-disk is already there?

Yeah, this topic depends on tb/pack-revindex-on-disk, which looks like will be merged to master according to your last What's Cooking email.

Quoted 2 lines

Just trying to establish what dependencies the bunch of topics have among themselves.

Thanks, I hope that it wasn't too much trouble.

Taylor

#59 lore raw

PATCH 8/9 pack-write.c: extract 'write_rev_file_order'

In reply to Taylor Blau

Existing callers provide the reverse index code with an array of 'struct pack_idx_entry *'s, which is then sorted by pack order (comparing the offsets of each object within the pack).

Prepare for the multi-pack index to write a .rev file by providing a way to write the reverse index without an array of pack_idx_entry (which the MIDX code does not have).

Instead, callers can invoke 'write_rev_index_positions()', which takes an array of uint32_t's. The ith entry in this array specifies the ith object's (in index order) position within the pack (in pack order).

Expose this new function for use in a later patch, and rewrite the existing write_rev_file() in terms of this new function.

---
 pack-write.c | 39 ++++++++++++++++++++++++++++-----------
 pack.h       |  1 +
 2 files changed, 29 insertions(+), 11 deletions(-)
pack-write.c +28 -11
index 680c36755d..75fcf70db1 100644
@@ -201,21 +201,12 @@ static void write_rev_header(struct hashfile *f)
 }
 
 static void write_rev_index_positions(struct hashfile *f,
-				      struct pack_idx_entry **objects,
+				      uint32_t *pack_order,
 				      uint32_t nr_objects)
 {
-	uint32_t *pack_order;
 	uint32_t i;
-
-	ALLOC_ARRAY(pack_order, nr_objects);
-	for (i = 0; i < nr_objects; i++)
-		pack_order[i] = i;
-	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
-
 	for (i = 0; i < nr_objects; i++)
 		hashwrite_be32(f, pack_order[i]);
-
-	free(pack_order);
 }
 
 static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)
@@ -228,6 +219,32 @@ const char *write_rev_file(const char *rev_name,
 			   uint32_t nr_objects,
 			   const unsigned char *hash,
 			   unsigned flags)
+{
+	uint32_t *pack_order;
+	uint32_t i;
+	const char *ret;
+
+	ALLOC_ARRAY(pack_order, nr_objects);
+	for (i = 0; i < nr_objects; i++)
+		pack_order[i] = i;
+	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
+
+	if (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))
+		return NULL;
+
+	ret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,
+				   flags);
+
+	free(pack_order);
+
+	return ret;
+}
+
+const char *write_rev_file_order(const char *rev_name,
+				 uint32_t *pack_order,
+				 uint32_t nr_objects,
+				 const unsigned char *hash,
+				 unsigned flags)
 {
 	struct hashfile *f;
 	int fd;
@@ -262,7 +279,7 @@ const char *write_rev_file(const char *rev_name,
 
 	write_rev_header(f);
 
-	write_rev_index_positions(f, objects, nr_objects);
+	write_rev_index_positions(f, pack_order, nr_objects);
 	write_rev_trailer(f, hash);
 
 	if (rev_name && adjust_shared_perm(rev_name) < 0)
pack.h +1 -0
index afdcf8f5c7..09c2a7dd3a 100644
@@ -94,6 +94,7 @@ struct ref;
 void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);
 
 const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);
+const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order, uint32_t nr_objects, const unsigned char *hash, unsigned flags);
 
 /*
  * The "hdr" output buffer should be at least this big, which will handle sizes
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#60 lore raw

PATCH 9/9 pack-revindex: write multi-pack reverse indexes

In reply to Taylor Blau

Implement the writing half of multi-pack reverse indexes. This is nothing more than the format describe a few patches ago, with a new set of helper functions that will be used to clear out stale .rev files corresponding to old MIDXs.

Unfortunately, a very similar comparison function as the one implemented recently in pack-revindex.c is reimplemented here, this time accepting a MIDX-internal type. An effort to DRY these up would create more indirection and overhead than is necessary, so it isn't pursued here.

Currently, there are no callers which pass the MIDX_WRITE_REV_INDEX flag, meaning that this is all dead code. But, that won't be the case for long, since subsequent patches will introduce the multi-pack bitmap, which will begin passing this field.

---
 midx.c | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 midx.h |   1 +
 2 files changed, 124 insertions(+)
midx.c +123 -0
index 12bfce8bb1..3d79a1930a 100644
@@ -11,6 +11,7 @@
 #include "trace2.h"
 #include "run-command.h"
 #include "repository.h"
+#include "pack.h"
 
 #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
 #define MIDX_VERSION 1
@@ -841,6 +842,78 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off
 	return written;
 }
 
+struct midx_pack_order_data {
+	struct pack_midx_entry *entries;
+	uint32_t *pack_perm;
+};
+
+static int midx_pack_order_cmp(const void *va, const void *vb, void *_data)
+{
+	struct midx_pack_order_data *data = _data;
+
+	struct pack_midx_entry *a = &data->entries[*(const uint32_t *)va];
+	struct pack_midx_entry *b = &data->entries[*(const uint32_t *)vb];
+
+	uint32_t perm_a = data->pack_perm[a->pack_int_id];
+	uint32_t perm_b = data->pack_perm[b->pack_int_id];
+
+	/* Sort objects in the preferred pack ahead of any others. */
+	if (a->preferred > b->preferred)
+		return -1;
+	if (a->preferred < b->preferred)
+		return 1;
+
+	/* Then, order objects by which packs they appear in. */
+	if (perm_a < perm_b)
+		return -1;
+	if (perm_a > perm_b)
+		return 1;
+
+	/* Then, disambiguate by their offset within each pack. */
+	if (a->offset < b->offset)
+		return -1;
+	if (a->offset > b->offset)
+		return 1;
+
+	return 0;
+}
+
+static uint32_t *midx_pack_order(struct pack_midx_entry *entries,
+				 uint32_t *pack_perm,
+				 uint32_t entries_nr)
+{
+	struct midx_pack_order_data data;
+	uint32_t *pack_order;
+	uint32_t i;
+
+	data.entries = entries;
+	data.pack_perm = pack_perm;
+
+	ALLOC_ARRAY(pack_order, entries_nr);
+	for (i = 0; i < entries_nr; i++)
+		pack_order[i] = i;
+	QSORT_S(pack_order, entries_nr, midx_pack_order_cmp, &data);
+
+	return pack_order;
+}
+
+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,
+				     uint32_t *pack_order,
+				     uint32_t entries_nr)
+{
+	struct strbuf buf = STRBUF_INIT;
+
+	strbuf_addf(&buf, "%s-%s.rev", midx_name, hash_to_hex(midx_hash));
+
+	write_rev_file_order(buf.buf, pack_order, entries_nr, midx_hash,
+			     WRITE_REV);
+
+	strbuf_release(&buf);
+}
+
+static void clear_midx_files_ext(struct repository *r, const char *ext,
+				 unsigned char *keep_hash);
+
 static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
 			       struct string_list *packs_to_drop,
 			       const char *preferred_pack_name,
@@ -854,6 +927,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	struct lock_file lk;
 	struct pack_list packs;
 	uint32_t *pack_perm = NULL;
+	uint32_t *pack_order = NULL;
 	uint64_t written = 0;
 	uint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];
 	uint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];
@@ -1111,6 +1185,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 		    chunk_offsets[num_chunks]);
 
 	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+	if (flags & MIDX_WRITE_REV_INDEX)
+		pack_order = midx_pack_order(entries, pack_perm, nr_entries);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		write_midx_reverse_index(midx_name, midx_hash, pack_order,
+					 nr_entries);
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+
 	commit_lock_file(&lk);
 
 cleanup:
@@ -1125,6 +1207,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	free(packs.info);
 	free(entries);
 	free(pack_perm);
+	free(pack_order);
 	free(midx_name);
 	return result;
 }
@@ -1137,6 +1220,44 @@ int write_midx_file(const char *object_dir,
 				   flags);
 }
 
+struct clear_midx_data {
+	char *keep;
+	const char *ext;
+};
+
+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,
+				const char *file_name, void *_data)
+{
+	struct clear_midx_data *data = _data;
+
+	if (!(starts_with(file_name, "multi-pack-index-") &&
+	      ends_with(file_name, data->ext)))
+		return;
+	if (data->keep && !strcmp(data->keep, file_name))
+		return;
+
+	if (unlink(full_path))
+		die_errno(_("failed to remove %s"), full_path);
+}
+
+static void clear_midx_files_ext(struct repository *r, const char *ext,
+				 unsigned char *keep_hash)
+{
+	struct clear_midx_data data;
+	memset(&data, 0, sizeof(struct clear_midx_data));
+
+	if (keep_hash)
+		data.keep = xstrfmt("multi-pack-index-%s%s",
+				    hash_to_hex(keep_hash), ext);
+	data.ext = ext;
+
+	for_each_file_in_pack_dir(r->objects->odb->path,
+				  clear_midx_file_ext,
+				  &data);
+
+	free(data.keep);
+}
+
 void clear_midx_file(struct repository *r)
 {
 	char *midx = get_midx_filename(r->objects->odb->path);
@@ -1149,6 +1270,8 @@ void clear_midx_file(struct repository *r)
 	if (remove_path(midx))
 		die(_("failed to clear multi-pack-index at %s"), midx);
 
+	clear_midx_files_ext(r, ".rev", NULL);
+
 	free(midx);
 }
 
midx.h +1 -0
index 0a8294d2ee..8684cf0fef 100644
@@ -40,6 +40,7 @@ struct multi_pack_index {
 };
 
 #define MIDX_PROGRESS     (1 << 0)
+#define MIDX_WRITE_REV_INDEX (1 << 1)
 
 char *get_midx_rev_filename(struct multi_pack_index *m);
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#61 lore raw
In reply to Taylor Blau
Quoted 11 lines · Taylor Blau

This series describes and implements a reverse index for the multi-pack index, based on a "pseudo-pack" which can be uniquely described by the multi-pack index.

The details of the pseudo-pack, and multi-pack reverse index are laid out in detail in the sixth patch.

This is in support of multi-pack reachability bitmaps, which contain objects from the multi-pack index. Likewise, an object's bit position in a multi-pack reachability bitmap is determined by its position with that multi-pack index's pseudo pack.

This has been a lot of work, but I'm impressed with the progress here.

This series is good prep, and my comments are very minor.

Since the need for these multi-pack-index-<hash>.rev files doesn't show up until the reachability bitmaps can be paired with MIDX files, it would make sense to hold this series in 'next' until that one also stabilizes and they merge to 'master' together.

Thanks,
-Stolee

#62 lore raw
Quoted 1 line · Derrick Stolee

This series is good prep, and my comments are very minor.

Thanks for your review, especially on a series like this one that adds lots of code without any users :).

Quoted 4 lines

Since the need for these multi-pack-index-<hash>.rev files doesn't show up until the reachability bitmaps can be paired with MIDX files, it would make sense to hold this series in 'next' until that one also stabilizes and they merge to 'master' together.

That's fine with me. It would be OK to merge this down to 'master', too, since this is all dead code. In fact, that may be easier to work with, since the next topic can be based directly off 'master' instead of having to keep this branch around forever.

Either is fine, though.

Thanks,
Taylor

#63 lore raw
In reply to Taylor Blau
Quoted 2 lines · Taylor Blau

Since tb/pack-revindex-on-disk is queued to be merged to 'master', but hasn't yet been merged, this series is based on that branch.

This seems to have a light conflict with Derrick's chunked file format work in midx.c where pack_info is renamed and extended so the new pack_order variable now needs to become a member in it.

I think I resolved it OK, but without any callers that actually utilize the new code or tests, it is almost impossible to have any confidence in the result of the conflict resolution X-<.

Could you two please look over to see if I made any silly mistakes, when I pushe it out later?

Thanks.

#64 lore raw
Quoted 9 lines · Junio C Hamano
Quoted 2 lines · Taylor Blau

Since tb/pack-revindex-on-disk is queued to be merged to 'master', but hasn't yet been merged, this series is based on that branch.

This seems to have a light conflict with Derrick's chunked file format work in midx.c where pack_info is renamed and extended so the new pack_order variable now needs to become a member in it.

Quoted 6 lines

I think I resolved it OK, but without any callers that actually utilize the new code or tests, it is almost impossible to have any confidence in the result of the conflict resolution X-<.

Could you two please look over to see if I made any silly mistakes, when I pushe it out later?

I reproduced your merge and got very similar results. The differences that happened between my result and yours are not meaningful in any way.

Definitely a very subtle merge, so thank you for doing that!

-Stolee

#65 lore raw
Quoted 8 lines · Derrick Stolee
Quoted 2 lines

Could you two please look over to see if I made any silly mistakes, when I pushe it out later?

I reproduced your merge and got very similar results. The differences that happened between my result and yours are not meaningful in any way.

Definitely a very subtle merge, so thank you for doing that!

Resolving this kind of conflict is like a quick quiz in the classroom to check my understanding of what is happening in all the topics involved. It is like solving a puzzle, and I do not dislike doing it (otherwise I won't be working as a maintainer), but having a second set of eyes, helping with independent verification, is always appreciated.

Thanks.

#66 lore raw

v2 cover midx: implement a multi-pack reverse index

In reply to Taylor Blau

Here is a reroll of my series to implement a reverse index in preparation for multi-pack reachability bitmaps. This version is based on a merge of 'ds/chunked-file-api' (currently on 'next') with the tip of 'master'.

The range-diff below looks complicated, but it isn't. The main changes from last time are:

- A handful of new patches at the front inspired by Ævar to handle sub-command options separately from the main 'git multi-pack-index' builtin.

- Minor tweaks to the '--show-objects' mode for the multi-pack index helper (the information now follows the existing output instead of preceeding it).

- Many fields have been folded into the new 'write_midx_context' struct, which allowed for some related cleanup.

- A new 'close_midx_revindex()' was added, which I discovered I need for the multi-pack bitmaps topic.

I know that we're getting close to an -rc0, so I'm torn about what should happen with this branch assuming that no major issues are uncovered during review. I would be happy if this were queued soon so that we can start working on the main topic which all of this is in support of, but I'm not sure how that would interact with the upcoming release.

In fact, I think that the first six patches couple make it into 2.31 as-is, since they have not much to do with the substance of this series and should be fairly uncontroversial.

In any case, thanks in advance for your review.

Taylor Blau (15):
builtin/multi-pack-index.c: inline 'flags' with options builtin/multi-pack-index.c: don't handle 'progress' separately builtin/multi-pack-index.c: define common usage with a macro builtin/multi-pack-index.c: split sub-commands builtin/multi-pack-index.c: don't enter bogus cmd_mode builtin/multi-pack-index.c: display usage on unrecognized command t/helper/test-read-midx.c: add '--show-objects'
midx: allow marking a pack as preferred
midx: don't free midx_name early
midx: keep track of the checksum
midx: make some functions non-static Documentation/technical: describe multi-pack reverse indexes
pack-revindex: read multi-pack reverse indexes pack-write.c: extract 'write_rev_file_order'
pack-revindex: write multi-pack reverse indexes

 Documentation/git-multi-pack-index.txt       |  14 +-
 Documentation/technical/multi-pack-index.txt |   5 +-
 Documentation/technical/pack-format.txt      |  80 +++++++
 builtin/multi-pack-index.c                   | 182 ++++++++++++---
 builtin/repack.c                             |   2 +-
 midx.c                                       | 229 +++++++++++++++++--
 midx.h                                       |  11 +-
 pack-revindex.c                              | 127 ++++++++++
 pack-revindex.h                              |  53 +++++
 pack-write.c                                 |  39 +++-
 pack.h                                       |   1 +
 packfile.c                                   |   3 +
 t/helper/test-read-midx.c                    |  24 +-
 t/t5319-multi-pack-index.sh                  |  39 ++++
 14 files changed, 740 insertions(+), 69 deletions(-)

Range-diff against v1:

 -:  ---------- >  1:  0527fa89a9 builtin/multi-pack-index.c: inline 'flags' with options
 -:  ---------- >  2:  a4e107b1f8 builtin/multi-pack-index.c: don't handle 'progress' separately
 -:  ---------- >  3:  8679dfd212 builtin/multi-pack-index.c: define common usage with a macro
 -:  ---------- >  4:  bc42b56ea2 builtin/multi-pack-index.c: split sub-commands
 -:  ---------- >  5:  5daa2946d3 builtin/multi-pack-index.c: don't enter bogus cmd_mode
 -:  ---------- >  6:  98d9ea0770 builtin/multi-pack-index.c: display usage on unrecognized command
 1:  e36acb005d !  7:  2fd9f4debf t/helper/test-read-midx.c: add '--show-objects'

@@ t/helper/test-read-midx.c
uint32_t i;
struct multi_pack_index *m;
@@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir)
- if (!m)
- return 1;
+
+ printf("object-dir: %s\n", m->object_dir);

+ if (show_objects) {
+ struct object_id oid;
@@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir)
+ return 0;
+ }
+
- printf("header: %08x %d %d %d %d\n",
- m->signature,
- m->version,
-@@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir)
+ return 0;
+ }

int cmd__read_midx(int argc, const char **argv)
{
2: 4a358d57cf ! 8: 223b899094 midx: allow marking a pack as preferred

    @@ Documentation/git-multi-pack-index.txt: git-multi-pack-index - Write and verify
      DESCRIPTION
      -----------
     @@ Documentation/git-multi-pack-index.txt: OPTIONS
    - 	Turn progress on/off explicitly. If neither is specified, progress is
    - 	shown if standard error is connected to a terminal.
    - 
    -+--preferred-pack=<pack>::
    -+	When using the `write` subcommand, optionally specify the
    -+	tie-breaking pack used when multiple packs contain the same
    -+	object. Incompatible with other subcommands, including `repack`,
    -+	which may repack the pack marked as preferred. If not given, the
    -+	preferred pack is inferred from an existing `multi-pack-index`,
    -+	if one exists, otherwise the pack with the lowest mtime.
    -+
      The following subcommands are available:
      
      write::
    +-	Write a new MIDX file.
    ++	Write a new MIDX file. The following options are available for
    ++	the `write` sub-command:
    +++
    ++--
    ++	--preferred-pack=<pack>::
    ++		Optionally specify the tie-breaking pack used when
    ++		multiple packs contain the same object. If not given,
    ++		ties are broken in favor of the pack with the lowest
    ++		mtime.
    ++--
    + 
    + verify::
    + 	Verify the contents of the MIDX file.
     
      ## Documentation/technical/multi-pack-index.txt ##
     @@ Documentation/technical/multi-pack-index.txt: Design Details
    @@ builtin/multi-pack-index.c
      #include "trace2.h"
     +#include "object-store.h"
      
    - static char const * const builtin_multi_pack_index_usage[] = {
    - 	N_("git multi-pack-index [<options>] (write|verify|expire|repack --batch-size=<size>)"),
    + static char const * const builtin_multi_pack_index_write_usage[] = {
    + #define BUILTIN_MIDX_WRITE_USAGE \
    +-	N_("git multi-pack-index [<options>] write")
    ++	N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]")
    + 	BUILTIN_MIDX_WRITE_USAGE,
    + 	NULL
    + };
     @@ builtin/multi-pack-index.c: static char const * const builtin_multi_pack_index_usage[] = {
      
      static struct opts_multi_pack_index {
      	const char *object_dir;
     +	const char *preferred_pack;
      	unsigned long batch_size;
    - 	int progress;
    + 	unsigned flags;
      } opts;
    -@@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv,
    - 	static struct option builtin_multi_pack_index_options[] = {
    - 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
    - 		  N_("object directory containing set of packfile and pack-index pairs")),
    -+		OPT_STRING(0, "preferred-pack", &opts.preferred_pack, N_("preferred-pack"),
    -+		  N_("pack for reuse when computing a multi-pack bitmap")),
    - 		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
    - 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
    - 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
    -@@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv,
    - 		return 1;
    - 	}
    +@@ builtin/multi-pack-index.c: static struct option *add_common_options(struct option *prev)
      
    -+	if (strcmp(argv[0], "write") && opts.preferred_pack)
    -+		die(_("'--preferred-pack' requires 'write'"));
    + static int cmd_multi_pack_index_write(int argc, const char **argv)
    + {
    +-	struct option *options = common_opts;
    ++	struct option *options;
    ++	static struct option builtin_multi_pack_index_write_options[] = {
    ++		OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
    ++			   N_("preferred-pack"),
    ++			   N_("pack for reuse when computing a multi-pack bitmap")),
    ++		OPT_END(),
    ++	};
     +
    ++	options = parse_options_dup(builtin_multi_pack_index_write_options);
    ++	options = add_common_options(options);
    + 
      	trace2_cmd_mode(argv[0]);
      
    - 	if (!strcmp(argv[0], "repack"))
    -@@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv,
    - 		die(_("--batch-size option is only for 'repack' subcommand"));
    +@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv)
    + 		usage_with_options(builtin_multi_pack_index_write_usage,
    + 				   options);
      
    - 	if (!strcmp(argv[0], "write"))
    --		return write_midx_file(opts.object_dir, flags);
    -+		return write_midx_file(opts.object_dir, opts.preferred_pack,
    -+				       flags);
    - 	if (!strcmp(argv[0], "verify"))
    - 		return verify_midx_file(the_repository, opts.object_dir, flags);
    - 	if (!strcmp(argv[0], "expire"))
    +-	return write_midx_file(opts.object_dir, opts.flags);
    ++	return write_midx_file(opts.object_dir, opts.preferred_pack,
    ++			       opts.flags);
    + }
    + 
    + static int cmd_multi_pack_index_verify(int argc, const char **argv)
     
      ## builtin/repack.c ##
     @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)
    @@ midx.c: static int pack_info_compare(const void *_a, const void *_b)
     +	return -1;
     +}
     +
    - struct pack_list {
    + struct write_midx_context {
      	struct pack_info *info;
      	uint32_t nr;
    +@@ midx.c: struct write_midx_context {
    + 	uint32_t *pack_perm;
    + 	unsigned large_offsets_needed:1;
    + 	uint32_t num_large_offsets;
    ++
    ++	int preferred_pack_idx;
    + };
    + 
    + static void add_pack_to_midx(const char *full_path, size_t full_path_len,
     @@ midx.c: struct pack_midx_entry {
      	uint32_t pack_int_id;
      	time_t pack_mtime;
    @@ midx.c: static struct pack_midx_entry *get_sorted_entries(struct multi_pack_inde
      				nr_fanout++;
      			}
      		}
    -@@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off
    +@@ midx.c: static int write_midx_large_offsets(struct hashfile *f,
      }
      
      static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
    @@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_l
     +			       const char *preferred_pack_name,
     +			       unsigned flags)
      {
    - 	unsigned char cur_chunk, num_chunks = 0;
      	char *midx_name;
    + 	uint32_t i;
     @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *
    - 	int pack_name_concat_len = 0;
    - 	int dropped_packs = 0;
    - 	int result = 0;
    -+	int preferred_pack_idx = -1;
    - 
    - 	midx_name = get_midx_filename(object_dir);
    - 	if (safe_create_leading_directories(midx_name))
    -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *
    - 	if (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)
    + 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)
      		goto cleanup;
      
    --	entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);
    +-	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);
     +	if (preferred_pack_name) {
    -+		for (i = 0; i < packs.nr; i++) {
    ++		for (i = 0; i < ctx.nr; i++) {
     +			if (!cmp_idx_or_pack_name(preferred_pack_name,
    -+						  packs.info[i].pack_name)) {
    -+				preferred_pack_idx = i;
    ++						  ctx.info[i].pack_name)) {
    ++				ctx.preferred_pack_idx = i;
     +				break;
     +			}
     +		}
    -+	}
    ++	} else
    ++		ctx.preferred_pack_idx = -1;
     +
    -+	entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries,
    -+				     preferred_pack_idx);
    ++	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,
    ++					 ctx.preferred_pack_idx);
      
    - 	for (i = 0; i < nr_entries; i++) {
    - 		if (entries[i].offset > 0x7fffffff)
    + 	ctx.large_offsets_needed = 0;
    + 	for (i = 0; i < ctx.entries_nr; i++) {
     @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *
    - 			pack_name_concat_len += strlen(packs.info[i].pack_name) + 1;
    + 			pack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;
      	}
      
     +	/*
     +	 * Recompute the preferred_pack_idx (if applicable) according to the
     +	 * permuted pack order.
     +	 */
    -+	preferred_pack_idx = -1;
    ++	ctx.preferred_pack_idx = -1;
     +	if (preferred_pack_name) {
    -+		preferred_pack_idx = lookup_idx_or_pack_name(packs.info,
    -+							     packs.nr,
    ++		ctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,
    ++							     ctx.nr,
     +							     preferred_pack_name);
    -+		if (preferred_pack_idx < 0)
    ++		if (ctx.preferred_pack_idx < 0)
     +			warning(_("unknown preferred pack: '%s'"),
     +				preferred_pack_name);
     +		else {
    -+			uint32_t orig = packs.info[preferred_pack_idx].orig_pack_int_id;
    -+			uint32_t perm = pack_perm[orig];
    ++			uint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;
    ++			uint32_t perm = ctx.pack_perm[orig];
     +
     +			if (perm == PACK_EXPIRED) {
     +				warning(_("preferred pack '%s' is expired"),
     +					preferred_pack_name);
    -+				preferred_pack_idx = -1;
    ++				ctx.preferred_pack_idx = -1;
     +			} else
    -+				preferred_pack_idx = perm;
    ++				ctx.preferred_pack_idx = perm;
     +		}
     +	}
     +
    @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'
     +		) &&
     +
     +		git multi-pack-index --object-dir=objects \
    -+			--preferred-pack=test-BC-$bc.idx write 2>err &&
    ++			write --preferred-pack=test-BC-$bc.idx 2>err &&
     +		test_must_be_empty err &&
     +
     +		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |

3: 218474158a ! 9: 976848bc4b midx: don't free midx_name early
@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
- FREE_AND_NULL(midx_name);

- if (packs.m)
- close_midx(packs.m);
+ if (ctx.m)
+ close_midx(ctx.m);

 4:  b4b842fa1e <  -:  ---------- midx: keep track of the checksum
 -:  ---------- > 10:  5ed47f7e3a midx: keep track of the checksum
 5:  953beabaa4 ! 11:  0292508e12 midx: make some functions non-static

@@ midx.c: static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos) +uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
{
- return get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);
- }
+ return get_be32(m->chunk_object_offsets +
+ (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);

## midx.h ##
@@ midx.h: struct multi_pack_index {

 6:  e64504bad6 <  -:  ---------- Documentation/technical: describe multi-pack reverse indexes
 -:  ---------- > 12:  404d730498 Documentation/technical: describe multi-pack reverse indexes
 7:  4c5e64c5fc ! 13:  d4e01a44e7 pack-revindex: read multi-pack reverse indexes
    @@ midx.c: static uint8_t oid_version(void)
     +		       m->object_dir, hash_to_hex(get_midx_checksum(m)));
     +}
     +
    - struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)
    + static int midx_read_oid_fanout(const unsigned char *chunk_start,
    + 				size_t chunk_size, void *data)
      {
    - 	struct multi_pack_index *m = NULL;
     
      ## midx.h ##
     @@ midx.h: struct multi_pack_index {
    @@ pack-revindex.c: int load_pack_revindex(struct packed_git *p)
     +	free(revindex_name);
     +	return ret;
     +}
    ++
    ++int close_midx_revindex(struct multi_pack_index *m)
    ++{
    ++	if (!m)
    ++		return 0;
    ++
    ++	if (munmap((void*)m->revindex_map, m->revindex_len))
    ++		return -1;
    ++
    ++	m->revindex_map = NULL;
    ++	m->revindex_data = NULL;
    ++	m->revindex_len = 0;
    ++
    ++	return 0;
    ++}
     +
      int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)
      {
    @@ pack-revindex.h: struct packed_git;
     + * A negative number is returned on error.
     + */
     +int load_midx_revindex(struct multi_pack_index *m);
    ++
    ++/*
    ++ * Frees resources associated with a multi-pack reverse index.
    ++ *
    ++ * A negative number is returned on error.
    ++ */
    ++int close_midx_revindex(struct multi_pack_index *m);
     +
      /*
       * offset_to_pack_pos converts an object offset to a pack position. This
 8:  4829b93f42 = 14:  ab7012b283 pack-write.c: extract 'write_rev_file_order'
 9:  fb5954b769 ! 15:  01bd6a35c6 pack-revindex: write multi-pack reverse indexes
    @@ Commit message
     
      ## midx.c ##
     @@
    - #include "trace2.h"
      #include "run-command.h"
      #include "repository.h"
    + #include "chunk-format.h"
     +#include "pack.h"
      
      #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
      #define MIDX_VERSION 1
    -@@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off
    - 	return written;
    +@@ midx.c: struct write_midx_context {
    + 	uint32_t entries_nr;
    + 
    + 	uint32_t *pack_perm;
    ++	uint32_t *pack_order;
    + 	unsigned large_offsets_needed:1;
    + 	uint32_t num_large_offsets;
    + 
    +@@ midx.c: static int write_midx_large_offsets(struct hashfile *f,
    + 	return 0;
      }
      
    -+struct midx_pack_order_data {
    -+	struct pack_midx_entry *entries;
    -+	uint32_t *pack_perm;
    -+};
    -+
    -+static int midx_pack_order_cmp(const void *va, const void *vb, void *_data)
    ++static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)
     +{
    -+	struct midx_pack_order_data *data = _data;
    ++	struct write_midx_context *ctx = _ctx;
     +
    -+	struct pack_midx_entry *a = &data->entries[*(const uint32_t *)va];
    -+	struct pack_midx_entry *b = &data->entries[*(const uint32_t *)vb];
    ++	struct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];
    ++	struct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];
     +
    -+	uint32_t perm_a = data->pack_perm[a->pack_int_id];
    -+	uint32_t perm_b = data->pack_perm[b->pack_int_id];
    ++	uint32_t perm_a = ctx->pack_perm[a->pack_int_id];
    ++	uint32_t perm_b = ctx->pack_perm[b->pack_int_id];
     +
     +	/* Sort objects in the preferred pack ahead of any others. */
     +	if (a->preferred > b->preferred)
    @@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_l
     +	return 0;
     +}
     +
    -+static uint32_t *midx_pack_order(struct pack_midx_entry *entries,
    -+				 uint32_t *pack_perm,
    -+				 uint32_t entries_nr)
    ++static uint32_t *midx_pack_order(struct write_midx_context *ctx)
     +{
    -+	struct midx_pack_order_data data;
     +	uint32_t *pack_order;
     +	uint32_t i;
     +
    -+	data.entries = entries;
    -+	data.pack_perm = pack_perm;
    -+
    -+	ALLOC_ARRAY(pack_order, entries_nr);
    -+	for (i = 0; i < entries_nr; i++)
    ++	ALLOC_ARRAY(pack_order, ctx->entries_nr);
    ++	for (i = 0; i < ctx->entries_nr; i++)
     +		pack_order[i] = i;
    -+	QSORT_S(pack_order, entries_nr, midx_pack_order_cmp, &data);
    ++	QSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);
     +
     +	return pack_order;
     +}
     +
     +static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,
    -+				     uint32_t *pack_order,
    -+				     uint32_t entries_nr)
    ++				     struct write_midx_context *ctx)
     +{
     +	struct strbuf buf = STRBUF_INIT;
     +
     +	strbuf_addf(&buf, "%s-%s.rev", midx_name, hash_to_hex(midx_hash));
     +
    -+	write_rev_file_order(buf.buf, pack_order, entries_nr, midx_hash,
    -+			     WRITE_REV);
    ++	write_rev_file_order(buf.buf, ctx->pack_order, ctx->entries_nr,
    ++			     midx_hash, WRITE_REV);
     +
     +	strbuf_release(&buf);
     +}
    @@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_l
      			       struct string_list *packs_to_drop,
      			       const char *preferred_pack_name,
     @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *
    - 	struct lock_file lk;
    - 	struct pack_list packs;
    - 	uint32_t *pack_perm = NULL;
    -+	uint32_t *pack_order = NULL;
    - 	uint64_t written = 0;
    - 	uint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];
    - 	uint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];
    -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *
    - 		    chunk_offsets[num_chunks]);
      
      	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
    + 	free_chunkfile(cf);
    ++
     +	if (flags & MIDX_WRITE_REV_INDEX)
    -+		pack_order = midx_pack_order(entries, pack_perm, nr_entries);
    ++		ctx.pack_order = midx_pack_order(&ctx);
     +
     +	if (flags & MIDX_WRITE_REV_INDEX)
    -+		write_midx_reverse_index(midx_name, midx_hash, pack_order,
    -+					 nr_entries);
    ++		write_midx_reverse_index(midx_name, midx_hash, &ctx);
     +	clear_midx_files_ext(the_repository, ".rev", midx_hash);
     +
      	commit_lock_file(&lk);
      
      cleanup:
     @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *
    - 	free(packs.info);
    - 	free(entries);
    - 	free(pack_perm);
    -+	free(pack_order);
    + 	free(ctx.info);
    + 	free(ctx.entries);
    + 	free(ctx.pack_perm);
    ++	free(ctx.pack_order);
      	free(midx_name);
      	return result;
      }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#67 lore raw

PATCH v2 1/15 builtin/multi-pack-index.c: inline 'flags' with options

Subcommands of the 'git multi-pack-index' command (e.g., 'write', 'verify', etc.) will want to optionally change a set of shared flags that are eventually passed to the MIDX libraries.

Right now, options and flags are handled separately. Inline them into the same structure so that sub-commands can more easily share the 'flags' data.

---
 builtin/multi-pack-index.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
builtin/multi-pack-index.c +6 -7
index 5bf88cd2a8..4a0ddb06c4 100644
@@ -14,13 +14,12 @@ static struct opts_multi_pack_index {
 	const char *object_dir;
 	unsigned long batch_size;
 	int progress;
+	unsigned flags;
 } opts;
 
 int cmd_multi_pack_index(int argc, const char **argv,
 			 const char *prefix)
 {
-	unsigned flags = 0;
-
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
@@ -40,7 +39,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
 	if (opts.progress)
-		flags |= MIDX_PROGRESS;
+		opts.flags |= MIDX_PROGRESS;
 
 	if (argc == 0)
 		usage_with_options(builtin_multi_pack_index_usage,
@@ -55,16 +54,16 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	if (!strcmp(argv[0], "repack"))
 		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, flags);
+			(size_t)opts.batch_size, opts.flags);
 	if (opts.batch_size)
 		die(_("--batch-size option is only for 'repack' subcommand"));
 
 	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, flags);
+		return write_midx_file(opts.object_dir, opts.flags);
 	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, flags);
+		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
 	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, flags);
+		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
 
 	die(_("unrecognized subcommand: %s"), argv[0]);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#68 lore raw

PATCH v2 2/15 builtin/multi-pack-index.c: don't handle 'progress' separately

In reply to Taylor Blau

Now that there is a shared 'flags' member in the options structure, there is no need to keep track of whether to force progress or not, since ultimately the decision of whether or not to show a progress meter is controlled by a bit in the flags member.

Manipulate that bit directly, and drop the now-unnecessary 'progress' field while we're at it.

---
 builtin/multi-pack-index.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
builtin/multi-pack-index.c +3 -5
index 4a0ddb06c4..c70f020d8f 100644
@@ -13,7 +13,6 @@ static char const * const builtin_multi_pack_index_usage[] = {
 static struct opts_multi_pack_index {
 	const char *object_dir;
 	unsigned long batch_size;
-	int progress;
 	unsigned flags;
 } opts;
 
@@ -23,7 +22,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
+		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	git_config(git_default_config, NULL);
 
-	opts.progress = isatty(2);
+	if (isatty(2))
+		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
 			     builtin_multi_pack_index_usage, 0);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
-	if (opts.progress)
-		opts.flags |= MIDX_PROGRESS;
 
 	if (argc == 0)
 		usage_with_options(builtin_multi_pack_index_usage,
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#69 lore raw

PATCH v2 3/15 builtin/multi-pack-index.c: define common usage with a macro

In reply to Taylor Blau

Factor out the usage message into pieces corresponding to each mode. This avoids options specific to one sub-command from being shared with another in the usage.

A subsequent commit will use these #define macros to have usage variables for each sub-command without duplicating their contents.

---
 builtin/multi-pack-index.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
builtin/multi-pack-index.c +16 -1
index c70f020d8f..eea498e026 100644
@@ -5,8 +5,23 @@
 #include "midx.h"
 #include "trace2.h"
 
+#define BUILTIN_MIDX_WRITE_USAGE \
+	N_("git multi-pack-index [<options>] write")
+
+#define BUILTIN_MIDX_VERIFY_USAGE \
+	N_("git multi-pack-index [<options>] verify")
+
+#define BUILTIN_MIDX_EXPIRE_USAGE \
+	N_("git multi-pack-index [<options>] expire")
+
+#define BUILTIN_MIDX_REPACK_USAGE \
+	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
+
 static char const * const builtin_multi_pack_index_usage[] = {
-	N_("git multi-pack-index [<options>] (write|verify|expire|repack --batch-size=<size>)"),
+	BUILTIN_MIDX_WRITE_USAGE,
+	BUILTIN_MIDX_VERIFY_USAGE,
+	BUILTIN_MIDX_EXPIRE_USAGE,
+	BUILTIN_MIDX_REPACK_USAGE,
 	NULL
 };
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#70 lore raw

PATCH v2 4/15 builtin/multi-pack-index.c: split sub-commands

In reply to Taylor Blau

Handle sub-commands of the 'git multi-pack-index' builtin (e.g., "write", "repack", etc.) separately from one another. This allows sub-commands with unique options, without forcing cmd_multi_pack_index() to reject invalid combinations itself.

This comes at the cost of some duplication and boilerplate. Luckily, the duplication is reduced to a minimum, since common options are shared among sub-commands due to a suggestion by Ævar. (Sub-commands do have to retain the common options, too, since this builtin accepts common options on either side of the sub-command).

Roughly speaking, cmd_multi_pack_index() parses options (including common ones), and stops at the first non-option, which is the sub-command. It then dispatches to the appropriate sub-command, which parses the remaining options (also including common options).

Unknown options are kept by the sub-commands in order to detect their presence (and complain that too many arguments were given).

---
 builtin/multi-pack-index.c | 131 ++++++++++++++++++++++++++++++-------
 1 file changed, 106 insertions(+), 25 deletions(-)
builtin/multi-pack-index.c +106 -25
index eea498e026..caf0248a98 100644
@@ -5,17 +5,33 @@
 #include "midx.h"
 #include "trace2.h"
 
+static char const * const builtin_multi_pack_index_write_usage[] = {
 #define BUILTIN_MIDX_WRITE_USAGE \
 	N_("git multi-pack-index [<options>] write")
+	BUILTIN_MIDX_WRITE_USAGE,
+	NULL
+};
 
+static char const * const builtin_multi_pack_index_verify_usage[] = {
 #define BUILTIN_MIDX_VERIFY_USAGE \
 	N_("git multi-pack-index [<options>] verify")
+	BUILTIN_MIDX_VERIFY_USAGE,
+	NULL
+};
 
+static char const * const builtin_multi_pack_index_expire_usage[] = {
 #define BUILTIN_MIDX_EXPIRE_USAGE \
 	N_("git multi-pack-index [<options>] expire")
+	BUILTIN_MIDX_EXPIRE_USAGE,
+	NULL
+};
 
+static char const * const builtin_multi_pack_index_repack_usage[] = {
 #define BUILTIN_MIDX_REPACK_USAGE \
 	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
+	BUILTIN_MIDX_REPACK_USAGE,
+	NULL
+};
 
 static char const * const builtin_multi_pack_index_usage[] = {
 	BUILTIN_MIDX_WRITE_USAGE,
@@ -31,25 +47,99 @@ static struct opts_multi_pack_index {
 	unsigned flags;
 } opts;
 
-int cmd_multi_pack_index(int argc, const char **argv,
-			 const char *prefix)
+static struct option common_opts[] = {
+	OPT_FILENAME(0, "object-dir", &opts.object_dir,
+	  N_("object directory containing set of packfile and pack-index pairs")),
+	OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	OPT_END(),
+};
+
+static struct option *add_common_options(struct option *prev)
 {
-	static struct option builtin_multi_pack_index_options[] = {
-		OPT_FILENAME(0, "object-dir", &opts.object_dir,
-		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	struct option *with_common = parse_options_concat(common_opts, prev);
+	free(prev);
+	return with_common;
+}
+
+static int cmd_multi_pack_index_write(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_write_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_write_usage,
+				   options);
+
+	return write_midx_file(opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_verify(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_verify_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_verify_usage,
+				   options);
+
+	return verify_midx_file(the_repository, opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_expire(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_expire_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_expire_usage,
+				   options);
+
+	return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_repack(int argc, const char **argv)
+{
+	struct option *options;
+	static struct option builtin_multi_pack_index_repack_options[] = {
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
 	};
 
+	options = parse_options_dup(builtin_multi_pack_index_repack_options);
+	options = add_common_options(options);
+
+	argc = parse_options(argc, argv, NULL,
+			     options,
+			     builtin_multi_pack_index_repack_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_repack_usage,
+				   options);
+
+	return midx_repack(the_repository, opts.object_dir,
+			   (size_t)opts.batch_size, opts.flags);
+}
+
+int cmd_multi_pack_index(int argc, const char **argv,
+			 const char *prefix)
+{
+	struct option *builtin_multi_pack_index_options = common_opts;
+
 	git_config(git_default_config, NULL);
 
 	if (isatty(2))
 		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
-			     builtin_multi_pack_index_usage, 0);
+			     builtin_multi_pack_index_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
@@ -58,25 +148,16 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		usage_with_options(builtin_multi_pack_index_usage,
 				   builtin_multi_pack_index_options);
 
-	if (argc > 1) {
-		die(_("too many arguments"));
-		return 1;
-	}
-
 	trace2_cmd_mode(argv[0]);
 
 	if (!strcmp(argv[0], "repack"))
-		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, opts.flags);
-	if (opts.batch_size)
-		die(_("--batch-size option is only for 'repack' subcommand"));
-
-	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
-
-	die(_("unrecognized subcommand: %s"), argv[0]);
+		return cmd_multi_pack_index_repack(argc, argv);
+	else if (!strcmp(argv[0], "write"))
+		return cmd_multi_pack_index_write(argc, argv);
+	else if (!strcmp(argv[0], "verify"))
+		return cmd_multi_pack_index_verify(argc, argv);
+	else if (!strcmp(argv[0], "expire"))
+		return cmd_multi_pack_index_expire(argc, argv);
+	else
+		die(_("unrecognized subcommand: %s"), argv[0]);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#71 lore raw

Re: [PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands

Quoted 5 lines
builtin/multi-pack-index.c +0 -0
index eea498e026..caf0248a98 100644

builtin/multi-pack-index.c line 13

Quoted 9 lines
builtin/multi-pack-index.c +4 -0
@@ -5,17 +5,33 @@
 #include "midx.h"
 #include "trace2.h"
 
+static char const * const builtin_multi_pack_index_write_usage[] = {
 #define BUILTIN_MIDX_WRITE_USAGE \
 	N_("git multi-pack-index [<options>] write")
+	BUILTIN_MIDX_WRITE_USAGE,
+	NULL
+};

I think this way of writing is vulnerable to confusing errors if a missing or extra backslash happens, so I would prefer the #define to be outside the variable declaration.

builtin/multi-pack-index.c static struct opts_multi_pack_index { · line 116

Quoted 11 lines
builtin/multi-pack-index.c +6 -0
+static int cmd_multi_pack_index_repack(int argc, const char **argv)
+{
+	struct option *options;
+	static struct option builtin_multi_pack_index_repack_options[] = {
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
 	};
 
+	options = parse_options_dup(builtin_multi_pack_index_repack_options);
+	options = add_common_options(options);

I looked for where this was freed, but I guess freeing this struct is not really something we're worried about (which makes sense).

The other patches up to this one look good.

#72 lore raw

Re: [PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands

Quoted 10 lines · Jonathan Tan

builtin/multi-pack-index.c line 13

Quoted 6 lines
builtin/multi-pack-index.c +4 -0
+static char const * const builtin_multi_pack_index_write_usage[] = {
 #define BUILTIN_MIDX_WRITE_USAGE \
 	N_("git multi-pack-index [<options>] write")
+	BUILTIN_MIDX_WRITE_USAGE,
+	NULL
+};

I think this way of writing is vulnerable to confusing errors if a missing or extra backslash happens, so I would prefer the #define to be outside the variable declaration.

Yeah, I can't say that I disagree with you. Of course, having the #define's outside of the declaration makes the whole thing a little more verbose, which isn't a huge deal.

But I was mirroring what Ævar was doing in the sub-thread he started at:

    https://public-inbox.org/git/[email protected]/

Unless you feel strongly, I think that what we have isn't so bad here.

Quoted 9 lines
Quoted 9 lines
builtin/multi-pack-index.c +4 -0
+static int cmd_multi_pack_index_repack(int argc, const char **argv)
+{
+	struct option *options;
+	static struct option builtin_multi_pack_index_repack_options[] = {
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
 	};
Quoted 5 lines

builtin/multi-pack-index.c static struct opts_multi_pack_index { · line 116

Quoted 2 lines
builtin/multi-pack-index.c +2 -0
+	options = parse_options_dup(builtin_multi_pack_index_repack_options);
+	options = add_common_options(options);

I looked for where this was freed, but I guess freeing this struct is not really something we're worried about (which makes sense).

Right.

Thanks,
Taylor

#73 lore raw

Re: [PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands

Quoted 12 lines · Jonathan Tan
Quoted 10 lines

builtin/multi-pack-index.c line 13

Quoted 6 lines
builtin/multi-pack-index.c +4 -0
+static char const * const builtin_multi_pack_index_write_usage[] = {
 #define BUILTIN_MIDX_WRITE_USAGE \
 	N_("git multi-pack-index [<options>] write")
+	BUILTIN_MIDX_WRITE_USAGE,
+	NULL
+};

I think this way of writing is vulnerable to confusing errors if a missing or extra backslash happens, so I would prefer the #define to be outside the variable declaration.

Quoted 3 lines

Yeah, I can't say that I disagree with you. Of course, having the #define's outside of the declaration makes the whole thing a little more verbose, which isn't a huge deal.

I think it's the same verbosity - you just need to move the lines?

Quoted 5 lines

But I was mirroring what Ævar was doing in the sub-thread he started at:

    https://public-inbox.org/git/[email protected]/

Unless you feel strongly, I think that what we have isn't so bad here.

Yeah I don't feel that strongly about it.

#74 lore raw

Re: [PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands

Quoted 12 lines · Jonathan Tan
Quoted 12 lines · Jonathan Tan
Quoted 10 lines

builtin/multi-pack-index.c line 13

Quoted 6 lines
builtin/multi-pack-index.c +4 -0
+static char const * const builtin_multi_pack_index_write_usage[] = {
 #define BUILTIN_MIDX_WRITE_USAGE \
 	N_("git multi-pack-index [<options>] write")
+	BUILTIN_MIDX_WRITE_USAGE,
+	NULL
+};

I think this way of writing is vulnerable to confusing errors if a missing or extra backslash happens, so I would prefer the #define to be outside the variable declaration.

Quoted 5 lines
Quoted 3 lines

Yeah, I can't say that I disagree with you. Of course, having the #define's outside of the declaration makes the whole thing a little more verbose, which isn't a huge deal.

I think it's the same verbosity - you just need to move the lines?

Yeah, you're right. I'm being too subjective, and I don't really feel strongly, either.

Quoted 8 lines
Quoted 6 lines

But I was mirroring what Ævar was doing in the sub-thread he started at:

    https://public-inbox.org/git/[email protected]/

Unless you feel strongly, I think that what we have isn't so bad here.

Yeah I don't feel that strongly about it.

I'll take your suggestion, thanks.

Thanks,
Taylor

#75 lore raw

PATCH v2 5/15 builtin/multi-pack-index.c: don't enter bogus cmd_mode

In reply to Taylor Blau

Even before the recent refactoring, 'git multi-pack-index' calls 'trace2_cmd_mode()' before verifying that the sub-command is recognized.

Push this call down into the individual sub-commands so that we don't enter a bogus command mode.

---
 builtin/multi-pack-index.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
builtin/multi-pack-index.c +8 -2
index caf0248a98..9fdfe168c2 100644
@@ -65,6 +65,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_write_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -79,6 +81,8 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_verify_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -93,6 +97,8 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_expire_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -115,6 +121,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)
 	options = parse_options_dup(builtin_multi_pack_index_repack_options);
 	options = add_common_options(options);
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options,
 			     builtin_multi_pack_index_repack_usage,
@@ -148,8 +156,6 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		usage_with_options(builtin_multi_pack_index_usage,
 				   builtin_multi_pack_index_options);
 
-	trace2_cmd_mode(argv[0]);
-
 	if (!strcmp(argv[0], "repack"))
 		return cmd_multi_pack_index_repack(argc, argv);
 	else if (!strcmp(argv[0], "write"))
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#76 lore raw

PATCH v2 6/15 builtin/multi-pack-index.c: display usage on unrecognized command

In reply to Taylor Blau

When given a sub-command that it doesn't understand, 'git multi-pack-index' dies with the following message:

    $ git multi-pack-index bogus
    fatal: unrecognized subcommand: bogus

Instead of 'die()'-ing, we can display the usage text, which is much more helpful:

    $ git.compile multi-pack-index bogus
    usage: git multi-pack-index [<options>] write
       or: git multi-pack-index [<options>] verify
       or: git multi-pack-index [<options>] expire
       or: git multi-pack-index [<options>] repack [--batch-size=<size>]

--object-dir <file> object directory containing set of packfile and pack-index pairs
--progress force progress reporting

While we're at it, clean up some duplication between the "no sub-command" and "unrecognized sub-command" conditionals.

---
 builtin/multi-pack-index.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
builtin/multi-pack-index.c +4 -3
index 9fdfe168c2..5b05e5ce39 100644
@@ -153,8 +153,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		opts.object_dir = get_object_directory();
 
 	if (argc == 0)
-		usage_with_options(builtin_multi_pack_index_usage,
-				   builtin_multi_pack_index_options);
+		goto usage;
 
 	if (!strcmp(argv[0], "repack"))
 		return cmd_multi_pack_index_repack(argc, argv);
@@ -165,5 +164,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	else if (!strcmp(argv[0], "expire"))
 		return cmd_multi_pack_index_expire(argc, argv);
 	else
-		die(_("unrecognized subcommand: %s"), argv[0]);
+usage:
+		usage_with_options(builtin_multi_pack_index_usage,
+				   builtin_multi_pack_index_options);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#77 lore raw

PATCH v2 7/15 t/helper/test-read-midx.c: add '--show-objects'

In reply to Taylor Blau

The 'read-midx' helper is used in places like t5319 to display basic information about a multi-pack-index.

In the next patch, the MIDX writing machinery will learn a new way to choose from which pack an object is selected when multiple copies of that object exist.

To disambiguate which pack introduces an object so that this feature can be tested, add a '--show-objects' option which displays additional information about each object in the MIDX.

---
 t/helper/test-read-midx.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)
t/helper/test-read-midx.c +20 -4
index 2430880f78..7c2eb11a8e 100644
@@ -4,7 +4,7 @@
 #include "repository.h"
 #include "object-store.h"
 
-static int read_midx_file(const char *object_dir)
+static int read_midx_file(const char *object_dir, int show_objects)
 {
 	uint32_t i;
 	struct multi_pack_index *m;
@@ -43,13 +43,29 @@ static int read_midx_file(const char *object_dir)
 
 	printf("object-dir: %s\n", m->object_dir);
 
+	if (show_objects) {
+		struct object_id oid;
+		struct pack_entry e;
+
+		for (i = 0; i < m->num_objects; i++) {
+			nth_midxed_object_oid(&oid, m, i);
+			fill_midx_entry(the_repository, &oid, &e, m);
+
+			printf("%s %"PRIu64"\t%s\n",
+			       oid_to_hex(&oid), e.offset, e.p->pack_name);
+		}
+		return 0;
+	}
+
 	return 0;
 }
 
 int cmd__read_midx(int argc, const char **argv)
 {
-	if (argc != 2)
-		usage("read-midx <object-dir>");
+	if (!(argc == 2 || argc == 3))
+		usage("read-midx [--show-objects] <object-dir>");
 
-	return read_midx_file(argv[1]);
+	if (!strcmp(argv[1], "--show-objects"))
+		return read_midx_file(argv[2], 1);
+	return read_midx_file(argv[1], 0);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#78 lore raw

PATCH v2 8/15 midx: allow marking a pack as preferred

In reply to Taylor Blau

When multiple packs in the multi-pack index contain the same object, the MIDX machinery must make a choice about which pack it associates with that object. Prior to this patch, the lowest-ordered[1] pack was always selected.

Pack selection for duplicate objects is relatively unimportant today, but it will become important for multi-pack bitmaps. This is because we can only invoke the pack-reuse mechanism when all of the bits for reused objects come from the reuse pack (in order to ensure that all reused deltas can find their base objects in the same pack).

To encourage the pack selection process to prefer one pack over another (the pack to be preferred is the one a caller would like to later use as a reuse pack), introduce the concept of a "preferred pack". When provided, the MIDX code will always prefer an object found in a preferred pack over any other.

No format changes are required to store the preferred pack, since it will be able to be inferred with a corresponding MIDX bitmap, by looking up the pack associated with the object in the first bit position (this ordering is described in detail in a subsequent commit).

[1]: the ordering is specified by MIDX internals; for our purposes we can consider the "lowest ordered" pack to be "the one with the most-recent mtime.

---
 Documentation/git-multi-pack-index.txt       | 14 ++-
 Documentation/technical/multi-pack-index.txt |  5 +-
 builtin/multi-pack-index.c                   | 18 +++-
 builtin/repack.c                             |  2 +-
 midx.c                                       | 99 ++++++++++++++++++--
 midx.h                                       |  2 +-
 t/t5319-multi-pack-index.sh                  | 39 ++++++++
 7 files changed, 161 insertions(+), 18 deletions(-)
Documentation/git-multi-pack-index.txt +12 -2
index eb0caa0439..ffd601bc17 100644
@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes
 SYNOPSIS
 --------
 [verse]
-'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress] <subcommand>
+'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
+	[--preferred-pack=<pack>] <subcommand>
 
 DESCRIPTION
 -----------
@@ -30,7 +31,16 @@ OPTIONS
 The following subcommands are available:
 
 write::
-	Write a new MIDX file.
+	Write a new MIDX file. The following options are available for
+	the `write` sub-command:
++
+--
+	--preferred-pack=<pack>::
+		Optionally specify the tie-breaking pack used when
+		multiple packs contain the same object. If not given,
+		ties are broken in favor of the pack with the lowest
+		mtime.
+--
 
 verify::
 	Verify the contents of the MIDX file.
Documentation/technical/multi-pack-index.txt +3 -2
index e8e377a59f..fb688976c4 100644
@@ -43,8 +43,9 @@ Design Details
   a change in format.
 
 - The MIDX keeps only one record per object ID. If an object appears
-  in multiple packfiles, then the MIDX selects the copy in the most-
-  recently modified packfile.
+  in multiple packfiles, then the MIDX selects the copy in the
+  preferred packfile, otherwise selecting from the most-recently
+  modified packfile.
 
 - If there exist packfiles in the pack directory not registered in
   the MIDX, then those packfiles are loaded into the `packed_git`
builtin/multi-pack-index.c +15 -3
index 5b05e5ce39..2329dc5ec0 100644
@@ -4,10 +4,11 @@
 #include "parse-options.h"
 #include "midx.h"
 #include "trace2.h"
+#include "object-store.h"
 
 static char const * const builtin_multi_pack_index_write_usage[] = {
 #define BUILTIN_MIDX_WRITE_USAGE \
-	N_("git multi-pack-index [<options>] write")
+	N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]")
 	BUILTIN_MIDX_WRITE_USAGE,
 	NULL
 };
@@ -43,6 +44,7 @@ static char const * const builtin_multi_pack_index_usage[] = {
 
 static struct opts_multi_pack_index {
 	const char *object_dir;
+	const char *preferred_pack;
 	unsigned long batch_size;
 	unsigned flags;
 } opts;
@@ -63,7 +65,16 @@ static struct option *add_common_options(struct option *prev)
 
 static int cmd_multi_pack_index_write(int argc, const char **argv)
 {
-	struct option *options = common_opts;
+	struct option *options;
+	static struct option builtin_multi_pack_index_write_options[] = {
+		OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
+			   N_("preferred-pack"),
+			   N_("pack for reuse when computing a multi-pack bitmap")),
+		OPT_END(),
+	};
+
+	options = parse_options_dup(builtin_multi_pack_index_write_options);
+	options = add_common_options(options);
 
 	trace2_cmd_mode(argv[0]);
 
@@ -74,7 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		usage_with_options(builtin_multi_pack_index_write_usage,
 				   options);
 
-	return write_midx_file(opts.object_dir, opts.flags);
+	return write_midx_file(opts.object_dir, opts.preferred_pack,
+			       opts.flags);
 }
 
 static int cmd_multi_pack_index_verify(int argc, const char **argv)
builtin/repack.c +1 -1
index 01440de2d5..9f00806805 100644
@@ -523,7 +523,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 	remove_temporary_files();
 
 	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
-		write_midx_file(get_object_directory(), 0);
+		write_midx_file(get_object_directory(), NULL, 0);
 
 	string_list_clear(&names, 0);
 	string_list_clear(&rollback, 0);
midx.c +90 -9

196 lines not on this page · raw message

midx.h +1 -1
index b18cf53bc4..e7fea61109 100644
@@ -47,7 +47,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa
 int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);
 int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);
 
-int write_midx_file(const char *object_dir, unsigned flags);
+int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);
 void clear_midx_file(struct repository *r);
 int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);
 int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);
t/t5319-multi-pack-index.sh +39 -0
index b4afab1dfc..fd94ba9053 100755
@@ -31,6 +31,14 @@ midx_read_expect () {
 	test_cmp expect actual
 }
 
+midx_expect_object_offset () {
+	OID="$1"
+	OFFSET="$2"
+	OBJECT_DIR="$3"
+	test-tool read-midx --show-objects $OBJECT_DIR >actual &&
+	grep "^$OID $OFFSET" actual
+}
+
 test_expect_success 'setup' '
 	test_oid_cache <<-EOF
 	idxoff sha1:2999
@@ -234,6 +242,37 @@ test_expect_success 'warn on improper hash version' '
 	)
 '
 
+test_expect_success 'midx picks objects from preferred pack' '
+	test_when_finished rm -rf preferred.git &&
+	git init --bare preferred.git &&
+	(
+		cd preferred.git &&
+
+		a=$(echo "a" | git hash-object -w --stdin) &&
+		b=$(echo "b" | git hash-object -w --stdin) &&
+		c=$(echo "c" | git hash-object -w --stdin) &&
+
+		# Set up two packs, duplicating the object "B" at different
+		# offsets.
+		git pack-objects objects/pack/test-AB <<-EOF &&
+		$a
+		$b
+		EOF
+		bc=$(git pack-objects objects/pack/test-BC <<-EOF
+		$b
+		$c
+		EOF
+		) &&
+
+		git multi-pack-index --object-dir=objects \
+			write --preferred-pack=test-BC-$bc.idx 2>err &&
+		test_must_be_empty err &&
+
+		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |
+			cut -d" " -f1) &&
+		midx_expect_object_offset $b $ofs objects
+	)
+'
 
 test_expect_success 'verify multi-pack-index success' '
 	git multi-pack-index verify --object-dir=$objdir
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#79 lore raw

Re: [PATCH v2 08/15] midx: allow marking a pack as preferred

Quoted 21 lines
midx.c +5 -0
@@ -589,12 +619,17 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
 				nth_midxed_pack_midx_entry(m,
 							   &entries_by_fanout[nr_fanout],
 							   cur_object);
+				if (nth_midxed_pack_int_id(m, cur_object) == preferred_pack)
+					entries_by_fanout[nr_fanout].preferred = 1;
+				else
+					entries_by_fanout[nr_fanout].preferred = 0;
 				nr_fanout++;
 			}
 		}
 
 		for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {
 			uint32_t start = 0, end;
+			int preferred = cur_pack == preferred_pack;
 
 			if (cur_fanout)
 				start = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);
@@ -602,7 +637,11 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
 
 			for (cur_object = start; cur_object < end; cur_object++) {

midx.c static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, · line 647

Quoted 10 lines
midx.c +5 -1
 				ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);
-				fill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);
+				fill_pack_entry(cur_pack,
+						info[cur_pack].p,
+						cur_object,
+						&entries_by_fanout[nr_fanout],
+						preferred);
 				nr_fanout++;
 			}
 		}

I was initially confused that "preferred" was set twice, but this makes sense - the first one is when an existing midx is reused, and the second one is for objects in packs that the midx (if it exists) does not cover.

Quoted 5 lines
midx.c +0 -1
@@ -828,7 +869,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)
 		goto cleanup;
 
-	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 881

Quoted 10 lines
midx.c +10 -0
+	if (preferred_pack_name) {
+		for (i = 0; i < ctx.nr; i++) {
+			if (!cmp_idx_or_pack_name(preferred_pack_name,
+						  ctx.info[i].pack_name)) {
+				ctx.preferred_pack_idx = i;
+				break;
+			}
+		}
+	} else
+		ctx.preferred_pack_idx = -1;

Looks safer to put "ctx.preferred_pack_idx = -1" before the "if", just in case the given pack name does not exist?

Quoted 18 lines
midx.c +14 -0
@@ -889,6 +942,31 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			pack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;
 	}
 
+	/*
+	 * Recompute the preferred_pack_idx (if applicable) according to the
+	 * permuted pack order.
+	 */
+	ctx.preferred_pack_idx = -1;
+	if (preferred_pack_name) {
+		ctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,
+							     ctx.nr,
+							     preferred_pack_name);
+		if (ctx.preferred_pack_idx < 0)
+			warning(_("unknown preferred pack: '%s'"),
+				preferred_pack_name);
+		else {
+			uint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 968

Quoted 10 lines
midx.c +10 -0
+			uint32_t perm = ctx.pack_perm[orig];
+
+			if (perm == PACK_EXPIRED) {
+				warning(_("preferred pack '%s' is expired"),
+					preferred_pack_name);
+				ctx.preferred_pack_idx = -1;
+			} else
+				ctx.preferred_pack_idx = perm;
+		}
+	}

I couldn't figure out why the preferred pack index needs to be recalculated here, since the pack entries would have already been sorted. Also, the tests still pass when I comment this part out. A comment describing what's going on would be helpful.

All previous patches look good to me.

#80 lore raw

Re: [PATCH v2 08/15] midx: allow marking a pack as preferred

Quoted 3 lines · Jonathan Tan

I was initially confused that "preferred" was set twice, but this makes sense - the first one is when an existing midx is reused, and the second one is for objects in packs that the midx (if it exists) does not cover.

Yep. Those two paths permeate a lot of the MIDX writer code, since it wants to reuse work from an existing MIDX if it can find one.

Quoted 4 lines
Quoted 4 lines
(quoted diff) +0 -0
@@ -828,7 +869,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)
 		goto cleanup;
 
Quoted 14 lines

Review static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 881

Quoted 11 lines
(quoted diff) +10 -1
-	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);
+	if (preferred_pack_name) {
+		for (i = 0; i < ctx.nr; i++) {
+			if (!cmp_idx_or_pack_name(preferred_pack_name,
+						  ctx.info[i].pack_name)) {
+				ctx.preferred_pack_idx = i;
+				break;
+			}
+		}
+	} else
+		ctx.preferred_pack_idx = -1;

Looks safer to put "ctx.preferred_pack_idx = -1" before the "if", just in case the given pack name does not exist?

Agreed.

Quoted 25 lines
Quoted 25 lines
(quoted diff) +21 -0
@@ -889,6 +942,31 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			pack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;
 	}
 
+	/*
+	 * Recompute the preferred_pack_idx (if applicable) according to the
+	 * permuted pack order.
+	 */
+	ctx.preferred_pack_idx = -1;
+	if (preferred_pack_name) {
+		ctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,
+							     ctx.nr,
+							     preferred_pack_name);
+		if (ctx.preferred_pack_idx < 0)
+			warning(_("unknown preferred pack: '%s'"),
+				preferred_pack_name);
+		else {
+			uint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;
+			uint32_t perm = ctx.pack_perm[orig];
+
+			if (perm == PACK_EXPIRED) {
+				warning(_("preferred pack '%s' is expired"),
+					preferred_pack_name);
+				ctx.preferred_pack_idx = -1;
+			} else
Quoted 8 lines

Review static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 968

Quoted 3 lines
(quoted diff) +3 -0
+				ctx.preferred_pack_idx = perm;
+		}
+	}

I couldn't figure out why the preferred pack index needs to be recalculated here, since the pack entries would have already been sorted. Also, the tests still pass when I comment this part out. A comment describing what's going on would be helpful.

Funny you mention that; I was wondering the same thing myself the other day when reading these patches again before deploying them to a couple of testing repositories at GitHub.

It is totally unnecessary: since we have already marked objects from the preferred pack in get_sorted_entries(), the rest of the code doesn't care if the preferred pack was permuted or not.

But we *do* care if the pack which was preferred expired. The 'git repack --geometric --write-midx' caller (which will appear in a later series) should never do that, so emitting a warning() is worthwhile. I think ultimately you want something like this squashed in:

--- >8 ---

midx.c +10 -17
index d2c56c4bc6..46f55ff6cf 100644
@@ -582,7 +582,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
 						  struct pack_info *info,
 						  uint32_t nr_packs,
 						  uint32_t *nr_objects,
-						  uint32_t preferred_pack)
+						  int preferred_pack)
 {
 	uint32_t cur_fanout, cur_pack, cur_object;
 	uint32_t alloc_fanout, alloc_objects, total_objects = 0;
@@ -869,6 +869,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)
 		goto cleanup;
 
+	ctx.preferred_pack_idx = -1;
 	if (preferred_pack_name) {
 		for (i = 0; i < ctx.nr; i++) {
 			if (!cmp_idx_or_pack_name(preferred_pack_name,
@@ -877,8 +878,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 				break;
 			}
 		}
-	} else
-		ctx.preferred_pack_idx = -1;
+	}
 
 	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,
 					 ctx.preferred_pack_idx);
@@ -942,28 +942,21 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			pack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;
 	}
 
-	/*
-	 * Recompute the preferred_pack_idx (if applicable) according to the
-	 * permuted pack order.
-	 */
-	ctx.preferred_pack_idx = -1;
+	/* Check that the preferred pack wasn't expired (if given). */
 	if (preferred_pack_name) {
-		ctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,
-							     ctx.nr,
-							     preferred_pack_name);
-		if (ctx.preferred_pack_idx < 0)
+		int preferred_idx = lookup_idx_or_pack_name(ctx.info,
+							    ctx.nr,
+							    preferred_pack_name);
+		if (preferred_idx < 0)
 			warning(_("unknown preferred pack: '%s'"),
 				preferred_pack_name);
 		else {
-			uint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;
+			uint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;
 			uint32_t perm = ctx.pack_perm[orig];
 
-			if (perm == PACK_EXPIRED) {
+			if (perm == PACK_EXPIRED)
 				warning(_("preferred pack '%s' is expired"),
 					preferred_pack_name);
-				ctx.preferred_pack_idx = -1;
-			} else
-				ctx.preferred_pack_idx = perm;
 		}
 	}
 
#81 lore raw

Re: [PATCH v2 08/15] midx: allow marking a pack as preferred

Quoted 11 lines

Funny you mention that; I was wondering the same thing myself the other day when reading these patches again before deploying them to a couple of testing repositories at GitHub.

It is totally unnecessary: since we have already marked objects from the preferred pack in get_sorted_entries(), the rest of the code doesn't care if the preferred pack was permuted or not.

But we *do* care if the pack which was preferred expired. The 'git repack --geometric --write-midx' caller (which will appear in a later series) should never do that, so emitting a warning() is worthwhile.

Ah, this makes sense.

Quoted 10 lines

I
think ultimately you want something like this squashed in:

--- >8 ---

midx.c +0 -0
index d2c56c4bc6..46f55ff6cf 100644

midx.c static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, · line 585

Quoted 5 lines
midx.c +1 -1
@@ -582,7 +582,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
 						  struct pack_info *info,
 						  uint32_t nr_packs,
 						  uint32_t *nr_objects,
-						  uint32_t preferred_pack)
+						  int preferred_pack)

Why this change?

Quoted 46 lines
midx.c +8 -12
 {
 	uint32_t cur_fanout, cur_pack, cur_object;
 	uint32_t alloc_fanout, alloc_objects, total_objects = 0;
@@ -869,6 +869,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)
 		goto cleanup;
 
+	ctx.preferred_pack_idx = -1;
 	if (preferred_pack_name) {
 		for (i = 0; i < ctx.nr; i++) {
 			if (!cmp_idx_or_pack_name(preferred_pack_name,
@@ -877,8 +878,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 				break;
 			}
 		}
-	} else
-		ctx.preferred_pack_idx = -1;
+	}
 
 	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,
 					 ctx.preferred_pack_idx);
@@ -942,28 +942,21 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			pack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;
 	}
 
-	/*
-	 * Recompute the preferred_pack_idx (if applicable) according to the
-	 * permuted pack order.
-	 */
-	ctx.preferred_pack_idx = -1;
+	/* Check that the preferred pack wasn't expired (if given). */
 	if (preferred_pack_name) {
-		ctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,
-							     ctx.nr,
-							     preferred_pack_name);
-		if (ctx.preferred_pack_idx < 0)
+		int preferred_idx = lookup_idx_or_pack_name(ctx.info,
+							    ctx.nr,
+							    preferred_pack_name);
+		if (preferred_idx < 0)
 			warning(_("unknown preferred pack: '%s'"),
 				preferred_pack_name);
 		else {
-			uint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;
+			uint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;
 			uint32_t perm = ctx.pack_perm[orig];

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 961

Quoted 10 lines
midx.c +1 -4
 
-			if (perm == PACK_EXPIRED) {
+			if (perm == PACK_EXPIRED)
 				warning(_("preferred pack '%s' is expired"),
 					preferred_pack_name);
-				ctx.preferred_pack_idx = -1;
-			} else
-				ctx.preferred_pack_idx = perm;
 		}
 	}

The rest makes sense.

Quoted 1 line
#82 lore raw

Re: [PATCH v2 08/15] midx: allow marking a pack as preferred

Quoted 5 lines · Jonathan Tan
Quoted 5 lines

I
think ultimately you want something like this squashed in:

--- >8 ---

Quoted 12 lines

midx.c static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, · line 585

Quoted 10 lines
midx.c +1 -1
index d2c56c4bc6..46f55ff6cf 100644
@@ -582,7 +582,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
 						  struct pack_info *info,
 						  uint32_t nr_packs,
 						  uint32_t *nr_objects,
-						  uint32_t preferred_pack)
+						  int preferred_pack)

Why this change?

This was wrong in the original patch: ctx.preferred_pack is an integer, and is set to -1 when no preferred pack was specified.

It's certainly unlikely that we'd have 2^31 packs, but silently converting a signed type to an unsigned one is misleading.

Quoted 1 line

The rest makes sense.

Thanks for taking a look.

Taylor

#83 lore raw

PATCH v2 9/15 midx: don't free midx_name early

In reply to Taylor Blau

A subsequent patch will need to refer back to 'midx_name' later on in the function. In fact, this variable is already free()'d later on, so this makes the later free() no longer redundant.

---
 midx.c | 1 -
 1 file changed, 1 deletion(-)
midx.c +0 -1
index d2c56c4bc6..db043d3e65 100644
@@ -973,7 +973,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
-	FREE_AND_NULL(midx_name);
 
 	if (ctx.m)
 		close_midx(ctx.m);
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#84 lore raw

PATCH v2 10/15 midx: keep track of the checksum

In reply to Taylor Blau

write_midx_internal() uses a hashfile to write the multi-pack index, but discards its checksum. This makes sense, since nothing that takes place after writing the MIDX cares about its checksum.

That is about to change in a subsequent patch, when the optional reverse index corresponding to the MIDX will want to include the MIDX's checksum.

Store the checksum of the MIDX in preparation for that.

---
 midx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
midx.c +2 -1
index db043d3e65..3ea795f416 100644
@@ -821,6 +821,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			       unsigned flags)
 {
 	char *midx_name;
+	unsigned char midx_hash[GIT_MAX_RAWSZ];
 	uint32_t i;
 	struct hashfile *f = NULL;
 	struct lock_file lk;
@@ -1004,7 +1005,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
 	write_chunkfile(cf, &ctx);
 
-	finalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
 	commit_lock_file(&lk);
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#85 lore raw

PATCH v2 11/15 midx: make some functions non-static

In reply to Taylor Blau

In a subsequent commit, pack-revindex.c will become responsible for sorting a list of objects in the "MIDX pack order" (which will be defined in the following patch). To do so, it will need to be know the pack identifier and offset within that pack for each object in the MIDX.

The MIDX code already has functions for doing just that (nth_midxed_offset() and nth_midxed_pack_int_id()), but they are statically declared.

Since there is no reason that they couldn't be exposed publicly, and because they are already doing exactly what the caller in pack-revindex.c will want, expose them publicly so that they can be reused there.

---
 midx.c | 4 ++--
 midx.h | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)
midx.c +2 -2
index 3ea795f416..27a8b76dfe 100644
@@ -239,7 +239,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,
 	return oid;
 }
 
-static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
 {
 	const unsigned char *offset_data;
 	uint32_t offset32;
@@ -258,7 +258,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
 	return offset32;
 }
 
-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
 {
 	return get_be32(m->chunk_object_offsets +
 			(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);
midx.h +2 -0
index e7fea61109..93bd68189e 100644
@@ -40,6 +40,8 @@ struct multi_pack_index {
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);
+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);
 struct object_id *nth_midxed_object_oid(struct object_id *oid,
 					struct multi_pack_index *m,
 					uint32_t n);
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#86 lore raw

PATCH v2 12/15 Documentation/technical: describe multi-pack reverse indexes

In reply to Taylor Blau

As a prerequisite to implementing multi-pack bitmaps, motivate and describe the format and ordering of the multi-pack reverse index.

The subsequent patch will implement reading this format, and the patch after that will implement writing it while producing a multi-pack index.

---
 Documentation/technical/pack-format.txt | 80 +++++++++++++++++++++++++
 1 file changed, 80 insertions(+)
Documentation/technical/pack-format.txt +80 -0
index 1faa949bf6..77eb591057 100644
@@ -379,3 +379,83 @@ CHUNK DATA:
 TRAILER:
 
 	Index checksum of the above contents.
+
+== multi-pack-index reverse indexes
+
+Similar to the pack-based reverse index, the multi-pack index can also
+be used to generate a reverse index.
+
+Instead of mapping between offset, pack-, and index position, this
+reverse index maps between an object's position within the MIDX, and
+that object's position within a pseudo-pack that the MIDX describes.
+
+To clarify these three orderings, consider a multi-pack reachability
+bitmap (which does not yet exist, but is what we are building towards
+here). Each bit needs to correspond to an object in the MIDX, and so we
+need an efficient mapping from bit position to MIDX position.
+
+One solution is to let bits occupy the same position in the oid-sorted
+index stored by the MIDX. But because oids are effectively random, there
+resulting reachability bitmaps would have no locality, and thus compress
+poorly. (This is the reason that single-pack bitmaps use the pack
+ordering, and not the .idx ordering, for the same purpose.)
+
+So we'd like to define an ordering for the whole MIDX based around
+pack ordering, which has far better locality (and thus compresses more
+efficiently). We can think of a pseudo-pack created by the concatenation
+of all of the packs in the MIDX. E.g., if we had a MIDX with three packs
+(a, b, c), with 10, 15, and 20 objects respectively, we can imagine an
+ordering of the objects like:
+
+    |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|
+
+where the ordering of the packs is defined by the MIDX's pack list,
+and then the ordering of objects within each pack is the same as the
+order in the actual packfile.
+
+Given the list of packs and their counts of objects, you can
+na&iuml;vely reconstruct that pseudo-pack ordering (e.g., the object at
+position 27 must be (c,1) because packs "a" and "b" consumed 25 of the
+slots). But there's a catch. Objects may be duplicated between packs, in
+which case the MIDX only stores one pointer to the object (and thus we'd
+want only one slot in the bitmap).
+
+Callers could handle duplicates themselves by reading objects in order
+of their bit-position, but that's linear in the number of objects, and
+much too expensive for ordinary bitmap lookups. Building a reverse index
+solves this, since it is the logical inverse of the index, and that
+index has already removed duplicates. But, building a reverse index on
+the fly can be expensive. Since we already have an on-disk format for
+pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,
+too.
+
+Objects from the MIDX are ordered as follows to string together the
+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected
+by the MIDX, and define an ordering of packs based on their numeric ID
+(as stored by the MIDX). Let _offset(o)_ return the object offset of _o_
+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:
+
+  - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other
+    is not, then the preferred one sorts first.
++
+(This is a detail that allows the MIDX bitmap to determine which
+pack should be used by the pack-reuse mechanism, since it can ask
+the MIDX for the pack containing the object at bit position 0).
+
+  - If _pack(o~1~) &ne; pack(o~2~)_, then sort the two objects in
+    descending order based on the pack ID.
+
+  - Otherwise, _pack(o~1~) &equals; pack(o~2~)_, and the objects are
+    sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly
+    when _offset(o~1~) &lt; offset(o~2~)_).
+
+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of
+objects in packs stored by the MIDX, laid out in pack order, and the
+packs arranged in MIDX order (with the preferred pack coming first).
+
+Finally, note that the MIDX's reverse index is not stored as a chunk in
+the multi-pack-index itself. This is done because the reverse index
+includes the checksum of the pack or MIDX to which it belongs, which
+makes it impossible to write in the MIDX. To avoid races when rewriting
+the MIDX, a MIDX reverse index includes the MIDX's checksum in its
+filename (e.g., `multi-pack-index-xyz.rev`).
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#87 lore raw

Re: [PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes

Documentation/technical/pack-format.txt CHUNK DATA:

Quoted 10 lines

+== multi-pack-index reverse indexes

Documentation/technical/pack-format.txt +9 -0
+
+Similar to the pack-based reverse index, the multi-pack index can also
+be used to generate a reverse index.
+
+Instead of mapping between offset, pack-, and index position, this
+reverse index maps between an object's position within the MIDX, and
+that object's position within a pseudo-pack that the MIDX describes.
+
+To clarify these three orderings

The paragraph seems to only describe 2 orderings - object's position within the MIDX and object's position within the pseudo-pack. (Is the third one the offset within the MIDX - which is, I believe, trivially computable from the position within the MIDX?)

Also, which are stored in the .rev file?

The previous patches look good to me, and I'll review the remaining patches hopefully tomorrow.

#88 lore raw

Re: [PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes

Quoted 2 lines · Jonathan Tan

The previous patches look good to me, and I'll review the remaining patches hopefully tomorrow.

Thanks; I am sorely behind recent activity on the list. I had a last-minute errand to run last weekend and I haven't managed to quite dig out of the hole I created for myself since then.

Incidentally, I have had this code (and the tb/multi-pack-bitmaps) running on a couple of high-traffic repositories internal to GitHub, and so have a couple of improvements that I was hoping to squash in, too.

Thanks,
Taylor

#89 lore raw

Re: [PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes

In reply to Jonathan Tan
Quoted 7 lines · Jonathan Tan
Quoted 7 lines

+== multi-pack-index reverse indexes

(quoted diff) +6 -0
+
+Similar to the pack-based reverse index, the multi-pack index can also
+be used to generate a reverse index.
+
+Instead of mapping between offset, pack-, and index position, this
+reverse index maps between an object's position within the MIDX, and
Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+that object's position within a pseudo-pack that the MIDX describes.
+
+To clarify these three orderings

The paragraph seems to only describe 2 orderings - object's position within the MIDX and object's position within the pseudo-pack. (Is the third one the offset within the MIDX - which is, I believe, trivially computable from the position within the MIDX?)

Sorry for the confusion. I was trying to distinguish between ordering based on object offset, pack position, and index position.

I guess you could count that as 2, 3, or 4 different orderings (if you classify "pack vs MIDX", "offset vs pack pos vs index pos" or the last three plus "vs MIDX pos").

But I think that all of that is needlessly confusing, so I'd much rather just say "To clarify the difference between these orderings".

Quoted 1 line

Also, which are stored in the .rev file?

The paragraph above describes it a little bit "this reverse index maps between ...", but I think it could be made clearer. (I was intentionally brief there since I wanted to not get too far into the details before explaining the relevant concepts, but I think I went too far).

How does this sound?

--- >8 ---

Documentation/technical/pack-format.txt +8 -5
index 77eb591057..4bbbb188a4 100644
@@ -387,12 +387,15 @@ be used to generate a reverse index.
 
 Instead of mapping between offset, pack-, and index position, this
 reverse index maps between an object's position within the MIDX, and
-that object's position within a pseudo-pack that the MIDX describes.
+that object's position within a pseudo-pack that the MIDX describes
+(i.e., the ith entry of the multi-pack reverse index holds the MIDX
+position of ith object in pseudo-pack order).
 
-To clarify these three orderings, consider a multi-pack reachability
-bitmap (which does not yet exist, but is what we are building towards
-here). Each bit needs to correspond to an object in the MIDX, and so we
-need an efficient mapping from bit position to MIDX position.
+To clarify the difference between these orderings, consider a multi-pack
+reachability bitmap (which does not yet exist, but is what we are
+building towards here). Each bit needs to correspond to an object in the
+MIDX, and so we need an efficient mapping from bit position to MIDX
+position.
 
 One solution is to let bits occupy the same position in the oid-sorted
 index stored by the MIDX. But because oids are effectively random, there
#90 lore raw

Re: [PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes

Quoted 15 lines
Documentation/technical/pack-format.txt +3 -3
index 77eb591057..4bbbb188a4 100644
@@ -387,12 +387,15 @@ be used to generate a reverse index.
 
 Instead of mapping between offset, pack-, and index position, this
 reverse index maps between an object's position within the MIDX, and
-that object's position within a pseudo-pack that the MIDX describes.
+that object's position within a pseudo-pack that the MIDX describes
+(i.e., the ith entry of the multi-pack reverse index holds the MIDX
+position of ith object in pseudo-pack order).
 
-To clarify these three orderings, consider a multi-pack reachability
-bitmap (which does not yet exist, but is what we are building towards

Documentation/technical/pack-format.txt be used to generate a reverse index. · line 401

Quoted 10 lines
Documentation/technical/pack-format.txt +5 -2
-here). Each bit needs to correspond to an object in the MIDX, and so we
-need an efficient mapping from bit position to MIDX position.
+To clarify the difference between these orderings, consider a multi-pack
+reachability bitmap (which does not yet exist, but is what we are
+building towards here). Each bit needs to correspond to an object in the
+MIDX, and so we need an efficient mapping from bit position to MIDX
+position.
 
 One solution is to let bits occupy the same position in the oid-sorted
 index stored by the MIDX. But because oids are effectively random, there

Thanks - this diff makes sense.

#91 lore raw

PATCH v2 13/15 pack-revindex: read multi-pack reverse indexes

In reply to Taylor Blau

Implement reading for multi-pack reverse indexes, as described in the previous patch.

Note that these functions don't yet have any callers, and won't until multi-pack reachability bitmaps are introduced in a later patch series. In the meantime, this patch implements some of the infrastructure necessary to support multi-pack bitmaps.

There are three new functions exposed by the revindex API:

- load_midx_revindex(): loads the reverse index corresponding to the given multi-pack index.

- midx_to_pack_pos() and pack_pos_to_midx(): these convert between the multi-pack index and pseudo-pack order.

load_midx_revindex() and pack_pos_to_midx() are both relatively straightforward.

load_midx_revindex() needs a few functions to be exposed from the midx API. One to get the checksum of a midx, and another to get the .rev's filename. Similar to recent changes in the packed_git struct, three new fields are added to the multi_pack_index struct: one to keep track of the size, one to keep track of the mmap'd pointer, and another to point past the header and at the reverse index's data.

pack_pos_to_midx() simply reads the corresponding entry out of the table.

midx_to_pack_pos() is the trickiest, since it needs to find an object's position in the psuedo-pack order, but that order can only be recovered in the .rev file itself. This mapping can be implemented with a binary search, but note that the thing we're binary searching over isn't an array, but rather a _permutation_.

So, when comparing two items, it's helpful to keep in mind the difference. Instead of a traditional binary search, where you are comparing two things directly, here we're comparing a (pack, offset) tuple with an index into the multi-pack index. That index describes another (pack, offset) tuple, and it is _those_ two tuples that are compared.

---
 midx.c          |  11 +++++
 midx.h          |   6 +++
 pack-revindex.c | 127 ++++++++++++++++++++++++++++++++++++++++++++++++
 pack-revindex.h |  53 ++++++++++++++++++++
 packfile.c      |   3 ++
 5 files changed, 200 insertions(+)
midx.c +11 -0
index 27a8b76dfe..8d7a8927b8 100644
@@ -47,11 +47,22 @@ static uint8_t oid_version(void)
 	}
 }
 
+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)
+{
+	return m->data + m->data_len - the_hash_algo->rawsz;
+}
+
 static char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }
 
+char *get_midx_rev_filename(struct multi_pack_index *m)
+{
+	return xstrfmt("%s/pack/multi-pack-index-%s.rev",
+		       m->object_dir, hash_to_hex(get_midx_checksum(m)));
+}
+
 static int midx_read_oid_fanout(const unsigned char *chunk_start,
 				size_t chunk_size, void *data)
 {
midx.h +6 -0
index 93bd68189e..0a8294d2ee 100644
@@ -15,6 +15,10 @@ struct multi_pack_index {
 	const unsigned char *data;
 	size_t data_len;
 
+	const uint32_t *revindex_data;
+	const uint32_t *revindex_map;
+	size_t revindex_len;
+
 	uint32_t signature;
 	unsigned char version;
 	unsigned char hash_len;
@@ -37,6 +41,8 @@ struct multi_pack_index {
 
 #define MIDX_PROGRESS     (1 << 0)
 
+char *get_midx_rev_filename(struct multi_pack_index *m);
+
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
pack-revindex.c +127 -0

142 lines not on this page · raw message

pack-revindex.h +53 -0

75 lines not on this page · raw message

packfile.c +3 -0
index 1fec12ac5f..82623e0cb4 100644
@@ -862,6 +862,9 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 
 	if (!strcmp(file_name, "multi-pack-index"))
 		return;
+	if (starts_with(file_name, "multi-pack-index") &&
+	    ends_with(file_name, ".rev"))
+		return;
 	if (ends_with(file_name, ".idx") ||
 	    ends_with(file_name, ".rev") ||
 	    ends_with(file_name, ".pack") ||
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#92 lore raw

Re: [PATCH v2 13/15] pack-revindex: read multi-pack reverse indexes

Quoted 6 lines

midx_to_pack_pos() is the trickiest, since it needs to find an object's position in the psuedo-pack order, but that order can only be recovered in the .rev file itself. This mapping can be implemented with a binary search, but note that the thing we're binary searching over isn't an array, but rather a _permutation_.

Quoted 6 lines

So, when comparing two items, it's helpful to keep in mind the difference. Instead of a traditional binary search, where you are comparing two things directly, here we're comparing a (pack, offset) tuple with an index into the multi-pack index. That index describes another (pack, offset) tuple, and it is _those_ two tuples that are compared.

Well, the binary search is indeed over an array :-)

I understood that the array we're searching over is an array of indexes into the MIDX in pack-pos order, so I understood what's written here. It might be easier for other readers if we just say that we're treating the elements of this array not as indexes into MIDX but as their corresponding (is-preferred-pack, pack number, offset) tuples. But I'm fine with retaining the existing wording too.

The patch itself looks good.

#93 lore raw

Re: [PATCH v2 13/15] pack-revindex: read multi-pack reverse indexes

Quoted 6 lines · Jonathan Tan
Quoted 6 lines

midx_to_pack_pos() is the trickiest, since it needs to find an object's position in the psuedo-pack order, but that order can only be recovered in the .rev file itself. This mapping can be implemented with a binary search, but note that the thing we're binary searching over isn't an array, but rather a _permutation_.

Quoted 8 lines
Quoted 6 lines

So, when comparing two items, it's helpful to keep in mind the difference. Instead of a traditional binary search, where you are comparing two things directly, here we're comparing a (pack, offset) tuple with an index into the multi-pack index. That index describes another (pack, offset) tuple, and it is _those_ two tuples that are compared.

Well, the binary search is indeed over an array :-)

:-). This might be more clearer as:

...isn't an array of values, but rather a permuted order of those values.

Quoted 1 line

The patch itself looks good.

Thanks.

Taylor

#94 lore raw

PATCH v2 14/15 pack-write.c: extract 'write_rev_file_order'

In reply to Taylor Blau

Existing callers provide the reverse index code with an array of 'struct pack_idx_entry *'s, which is then sorted by pack order (comparing the offsets of each object within the pack).

Prepare for the multi-pack index to write a .rev file by providing a way to write the reverse index without an array of pack_idx_entry (which the MIDX code does not have).

Instead, callers can invoke 'write_rev_index_positions()', which takes an array of uint32_t's. The ith entry in this array specifies the ith object's (in index order) position within the pack (in pack order).

Expose this new function for use in a later patch, and rewrite the existing write_rev_file() in terms of this new function.

---
 pack-write.c | 39 ++++++++++++++++++++++++++++-----------
 pack.h       |  1 +
 2 files changed, 29 insertions(+), 11 deletions(-)
pack-write.c +28 -11
index 680c36755d..75fcf70db1 100644
@@ -201,21 +201,12 @@ static void write_rev_header(struct hashfile *f)
 }
 
 static void write_rev_index_positions(struct hashfile *f,
-				      struct pack_idx_entry **objects,
+				      uint32_t *pack_order,
 				      uint32_t nr_objects)
 {
-	uint32_t *pack_order;
 	uint32_t i;
-
-	ALLOC_ARRAY(pack_order, nr_objects);
-	for (i = 0; i < nr_objects; i++)
-		pack_order[i] = i;
-	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
-
 	for (i = 0; i < nr_objects; i++)
 		hashwrite_be32(f, pack_order[i]);
-
-	free(pack_order);
 }
 
 static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)
@@ -228,6 +219,32 @@ const char *write_rev_file(const char *rev_name,
 			   uint32_t nr_objects,
 			   const unsigned char *hash,
 			   unsigned flags)
+{
+	uint32_t *pack_order;
+	uint32_t i;
+	const char *ret;
+
+	ALLOC_ARRAY(pack_order, nr_objects);
+	for (i = 0; i < nr_objects; i++)
+		pack_order[i] = i;
+	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
+
+	if (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))
+		return NULL;
+
+	ret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,
+				   flags);
+
+	free(pack_order);
+
+	return ret;
+}
+
+const char *write_rev_file_order(const char *rev_name,
+				 uint32_t *pack_order,
+				 uint32_t nr_objects,
+				 const unsigned char *hash,
+				 unsigned flags)
 {
 	struct hashfile *f;
 	int fd;
@@ -262,7 +279,7 @@ const char *write_rev_file(const char *rev_name,
 
 	write_rev_header(f);
 
-	write_rev_index_positions(f, objects, nr_objects);
+	write_rev_index_positions(f, pack_order, nr_objects);
 	write_rev_trailer(f, hash);
 
 	if (rev_name && adjust_shared_perm(rev_name) < 0)
pack.h +1 -0
index afdcf8f5c7..09c2a7dd3a 100644
@@ -94,6 +94,7 @@ struct ref;
 void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);
 
 const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);
+const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order, uint32_t nr_objects, const unsigned char *hash, unsigned flags);
 
 /*
  * The "hdr" output buffer should be at least this big, which will handle sizes
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#95 lore raw

PATCH v2 15/15 pack-revindex: write multi-pack reverse indexes

In reply to Taylor Blau

Implement the writing half of multi-pack reverse indexes. This is nothing more than the format describe a few patches ago, with a new set of helper functions that will be used to clear out stale .rev files corresponding to old MIDXs.

Unfortunately, a very similar comparison function as the one implemented recently in pack-revindex.c is reimplemented here, this time accepting a MIDX-internal type. An effort to DRY these up would create more indirection and overhead than is necessary, so it isn't pursued here.

Currently, there are no callers which pass the MIDX_WRITE_REV_INDEX flag, meaning that this is all dead code. But, that won't be the case for long, since subsequent patches will introduce the multi-pack bitmap, which will begin passing this field.

---
 midx.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 midx.h |   1 +
 2 files changed, 112 insertions(+)
midx.c +111 -0
index 8d7a8927b8..820276cc45 100644
@@ -12,6 +12,7 @@
 #include "run-command.h"
 #include "repository.h"
 #include "chunk-format.h"
+#include "pack.h"
 
 #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
 #define MIDX_VERSION 1
@@ -472,6 +473,7 @@ struct write_midx_context {
 	uint32_t entries_nr;
 
 	uint32_t *pack_perm;
+	uint32_t *pack_order;
 	unsigned large_offsets_needed:1;
 	uint32_t num_large_offsets;
 
@@ -826,6 +828,66 @@ static int write_midx_large_offsets(struct hashfile *f,
 	return 0;
 }
 
+static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)
+{
+	struct write_midx_context *ctx = _ctx;
+
+	struct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];
+	struct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];
+
+	uint32_t perm_a = ctx->pack_perm[a->pack_int_id];
+	uint32_t perm_b = ctx->pack_perm[b->pack_int_id];
+
+	/* Sort objects in the preferred pack ahead of any others. */
+	if (a->preferred > b->preferred)
+		return -1;
+	if (a->preferred < b->preferred)
+		return 1;
+
+	/* Then, order objects by which packs they appear in. */
+	if (perm_a < perm_b)
+		return -1;
+	if (perm_a > perm_b)
+		return 1;
+
+	/* Then, disambiguate by their offset within each pack. */
+	if (a->offset < b->offset)
+		return -1;
+	if (a->offset > b->offset)
+		return 1;
+
+	return 0;
+}
+
+static uint32_t *midx_pack_order(struct write_midx_context *ctx)
+{
+	uint32_t *pack_order;
+	uint32_t i;
+
+	ALLOC_ARRAY(pack_order, ctx->entries_nr);
+	for (i = 0; i < ctx->entries_nr; i++)
+		pack_order[i] = i;
+	QSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);
+
+	return pack_order;
+}
+
+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,
+				     struct write_midx_context *ctx)
+{
+	struct strbuf buf = STRBUF_INIT;
+
+	strbuf_addf(&buf, "%s-%s.rev", midx_name, hash_to_hex(midx_hash));
+
+	write_rev_file_order(buf.buf, ctx->pack_order, ctx->entries_nr,
+			     midx_hash, WRITE_REV);
+
+	strbuf_release(&buf);
+}
+
+static void clear_midx_files_ext(struct repository *r, const char *ext,
+				 unsigned char *keep_hash);
+
 static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
 			       struct string_list *packs_to_drop,
 			       const char *preferred_pack_name,
@@ -1018,6 +1080,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		ctx.pack_order = midx_pack_order(&ctx);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		write_midx_reverse_index(midx_name, midx_hash, &ctx);
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+
 	commit_lock_file(&lk);
 
 cleanup:
@@ -1032,6 +1102,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	free(ctx.info);
 	free(ctx.entries);
 	free(ctx.pack_perm);
+	free(ctx.pack_order);
 	free(midx_name);
 	return result;
 }
@@ -1044,6 +1115,44 @@ int write_midx_file(const char *object_dir,
 				   flags);
 }
 
+struct clear_midx_data {
+	char *keep;
+	const char *ext;
+};
+
+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,
+				const char *file_name, void *_data)
+{
+	struct clear_midx_data *data = _data;
+
+	if (!(starts_with(file_name, "multi-pack-index-") &&
+	      ends_with(file_name, data->ext)))
+		return;
+	if (data->keep && !strcmp(data->keep, file_name))
+		return;
+
+	if (unlink(full_path))
+		die_errno(_("failed to remove %s"), full_path);
+}
+
+static void clear_midx_files_ext(struct repository *r, const char *ext,
+				 unsigned char *keep_hash)
+{
+	struct clear_midx_data data;
+	memset(&data, 0, sizeof(struct clear_midx_data));
+
+	if (keep_hash)
+		data.keep = xstrfmt("multi-pack-index-%s%s",
+				    hash_to_hex(keep_hash), ext);
+	data.ext = ext;
+
+	for_each_file_in_pack_dir(r->objects->odb->path,
+				  clear_midx_file_ext,
+				  &data);
+
+	free(data.keep);
+}
+
 void clear_midx_file(struct repository *r)
 {
 	char *midx = get_midx_filename(r->objects->odb->path);
@@ -1056,6 +1165,8 @@ void clear_midx_file(struct repository *r)
 	if (remove_path(midx))
 		die(_("failed to clear multi-pack-index at %s"), midx);
 
+	clear_midx_files_ext(r, ".rev", NULL);
+
 	free(midx);
 }
 
midx.h +1 -0
index 0a8294d2ee..8684cf0fef 100644
@@ -40,6 +40,7 @@ struct multi_pack_index {
 };
 
 #define MIDX_PROGRESS     (1 << 0)
+#define MIDX_WRITE_REV_INDEX (1 << 1)
 
 char *get_midx_rev_filename(struct multi_pack_index *m);
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#96 lore raw

Re: [PATCH v2 15/15] pack-revindex: write multi-pack reverse indexes

Quoted 5 lines
midx.c +1 -0
@@ -1018,6 +1080,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
+

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 1093

Quoted 10 lines
midx.c +7 -0
+	if (flags & MIDX_WRITE_REV_INDEX)
+		ctx.pack_order = midx_pack_order(&ctx);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		write_midx_reverse_index(midx_name, midx_hash, &ctx);
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+
 	commit_lock_file(&lk);
 
 cleanup:

Any reason why we're using 2 separate "if" statements?

Other than that, this patch and patch 14 look good. Besides all my minor comments, I think the overall patch set is in good shape and ready to be merged. It's great that we could reuse some of the individual-pack reverse index concepts and code too.

#97 lore raw

Re: [PATCH v2 15/15] pack-revindex: write multi-pack reverse indexes

Quoted 14 lines · Jonathan Tan
Quoted 14 lines
midx.c +8 -0
@@ -1018,6 +1080,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		ctx.pack_order = midx_pack_order(&ctx);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		write_midx_reverse_index(midx_name, midx_hash, &ctx);
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+
 	commit_lock_file(&lk);
 
Quoted 3 lines

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 1093

Quoted 1 line
midx.c +0 -0
 cleanup:

Any reason why we're using 2 separate "if" statements?

Yeah. This first if statement will turn into:

if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))

so that the pack order is computed in either case (since both the existing write_midx_reverse_index() and the eventual write_midx_bitmap() will be able to use the pack order).

Arguably there is never a practical reason to write one without the other (and writing a MIDX bitmap without a reverse index is a bug), so perhaps these options should be consolidated.

But that's cleanup that I'd rather do after all of this has settled (since it'd be weird to say: "here's the option to write bitmaps, except we can't write multi-pack bitmaps yet, but setting it actually writes this other thing").

Quoted 4 lines

Other than that, this patch and patch 14 look good. Besides all my minor comments, I think the overall patch set is in good shape and ready to be merged. It's great that we could reuse some of the individual-pack reverse index concepts and code too.

Thanks, I am really glad that you had a chance to take a look at it. I always find your review quite helpful.

Thanks,
Taylor

#98 lore raw

Re: [PATCH v2 15/15] pack-revindex: write multi-pack reverse indexes

Quoted 19 lines · Jonathan Tan
Quoted 14 lines
Quoted 14 lines
midx.c +8 -0
@@ -1018,6 +1080,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		ctx.pack_order = midx_pack_order(&ctx);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		write_midx_reverse_index(midx_name, midx_hash, &ctx);
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+
 	commit_lock_file(&lk);
 
Quoted 3 lines

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 1093

Quoted 1 line
midx.c +0 -0
 cleanup:

Any reason why we're using 2 separate "if" statements?

Quoted 7 lines

Yeah. This first if statement will turn into:

if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))

so that the pack order is computed in either case (since both the existing write_midx_reverse_index() and the eventual write_midx_bitmap() will be able to use the pack order).

Ah, OK. That's what I was thinking of, but nice to have confirmation. Maybe write in the commit message that these are separated because in the future, one of the conditions will change.

#99 lore raw

Re: [PATCH v2 15/15] pack-revindex: write multi-pack reverse indexes

Quoted 6 lines · Jonathan Tan
Quoted 6 lines
Quoted 1 line

Any reason why we're using 2 separate "if" statements?

Yeah. This first if statement will turn into:

if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))

Quoted 7 lines
Quoted 3 lines

so that the pack order is computed in either case (since both the existing write_midx_reverse_index() and the eventual write_midx_bitmap() will be able to use the pack order).

Ah, OK. That's what I was thinking of, but nice to have confirmation. Maybe write in the commit message that these are separated because in the future, one of the conditions will change.

Thanks; that's a great idea.

Thanks,
Taylor

#100 lore raw

v3 cover midx: implement a multi-pack reverse index

In reply to Taylor Blau

Here is another reroll of my series to implement a reverse index in preparation for multi-pack reachability bitmaps. The previous version was based on 'ds/chunked-file-api', but that topic has since been merged to 'master'. This series is now built directly on top of 'master'.

Not much has changed since last time. Jonathan Tan reviewed the previous version, and I incorporated feedback from his review:

- The usage macros in builtin/multi-pack-index.c were pulled out and defined separately.
- Some sloppiness with converting a signed index referring to the preferred pack into an unsigned value was cleaned up.
- Documentation clean-up, particularly in patches 12 and 13.

There are a couple of new things that we found while testing this out at GitHub.

- We now call finalize_object_file() on the multi-pack reverse index to set the correct permissions.
- Patch 14 removed a stray hunk that introduced a memory leak.
- Patch 16 (courtesy of Peff) is new. It improves the cache locality of midx_pack_order_cmp(), which has a substantial impact on repositories with many objects.

Thanks in advance for your review.

Jeff King (1):
midx.c: improve cache locality in midx_pack_order_cmp()

Taylor Blau (15):
builtin/multi-pack-index.c: inline 'flags' with options builtin/multi-pack-index.c: don't handle 'progress' separately builtin/multi-pack-index.c: define common usage with a macro builtin/multi-pack-index.c: split sub-commands builtin/multi-pack-index.c: don't enter bogus cmd_mode builtin/multi-pack-index.c: display usage on unrecognized command t/helper/test-read-midx.c: add '--show-objects'
midx: allow marking a pack as preferred
midx: don't free midx_name early
midx: keep track of the checksum
midx: make some functions non-static Documentation/technical: describe multi-pack reverse indexes
pack-revindex: read multi-pack reverse indexes pack-write.c: extract 'write_rev_file_order'
pack-revindex: write multi-pack reverse indexes

 Documentation/git-multi-pack-index.txt       |  14 +-
 Documentation/technical/multi-pack-index.txt |   5 +-
 Documentation/technical/pack-format.txt      |  83 +++++++
 builtin/multi-pack-index.c                   | 182 ++++++++++++---
 builtin/repack.c                             |   2 +-
 midx.c                                       | 229 +++++++++++++++++--
 midx.h                                       |  11 +-
 pack-revindex.c                              | 127 ++++++++++
 pack-revindex.h                              |  53 +++++
 pack-write.c                                 |  36 ++-
 pack.h                                       |   1 +
 packfile.c                                   |   3 +
 t/helper/test-read-midx.c                    |  24 +-
 t/t5319-multi-pack-index.sh                  |  39 ++++
 14 files changed, 740 insertions(+), 69 deletions(-)

Range-diff against v2:

 1:  0527fa89a9 =  1:  43fc0ad276 builtin/multi-pack-index.c: inline 'flags' with options
 2:  a4e107b1f8 =  2:  181f11e4c5 builtin/multi-pack-index.c: don't handle 'progress' separately
 3:  8679dfd212 =  3:  94c498f0e2 builtin/multi-pack-index.c: define common usage with a macro
 4:  bc42b56ea2 !  4:  d084f90466 builtin/multi-pack-index.c: split sub-commands
    @@ Commit message
     
      ## builtin/multi-pack-index.c ##
     @@
    - #include "midx.h"
    - #include "trace2.h"
    + #define BUILTIN_MIDX_REPACK_USAGE \
    + 	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
      
     +static char const * const builtin_multi_pack_index_write_usage[] = {
    - #define BUILTIN_MIDX_WRITE_USAGE \
    - 	N_("git multi-pack-index [<options>] write")
     +	BUILTIN_MIDX_WRITE_USAGE,
     +	NULL
     +};
    - 
     +static char const * const builtin_multi_pack_index_verify_usage[] = {
    - #define BUILTIN_MIDX_VERIFY_USAGE \
    - 	N_("git multi-pack-index [<options>] verify")
     +	BUILTIN_MIDX_VERIFY_USAGE,
     +	NULL
     +};
    - 
     +static char const * const builtin_multi_pack_index_expire_usage[] = {
    - #define BUILTIN_MIDX_EXPIRE_USAGE \
    - 	N_("git multi-pack-index [<options>] expire")
     +	BUILTIN_MIDX_EXPIRE_USAGE,
     +	NULL
     +};
    - 
     +static char const * const builtin_multi_pack_index_repack_usage[] = {
    - #define BUILTIN_MIDX_REPACK_USAGE \
    - 	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
     +	BUILTIN_MIDX_REPACK_USAGE,
     +	NULL
     +};
    - 
      static char const * const builtin_multi_pack_index_usage[] = {
      	BUILTIN_MIDX_WRITE_USAGE,
    + 	BUILTIN_MIDX_VERIFY_USAGE,
     @@ builtin/multi-pack-index.c: static struct opts_multi_pack_index {
      	unsigned flags;
      } opts;
 5:  5daa2946d3 =  5:  bc3b6837f2 builtin/multi-pack-index.c: don't enter bogus cmd_mode
 6:  98d9ea0770 =  6:  f117e442c3 builtin/multi-pack-index.c: display usage on unrecognized command
 7:  2fd9f4debf =  7:  ae85a68ef2 t/helper/test-read-midx.c: add '--show-objects'
 8:  223b899094 !  8:  30194a6786 midx: allow marking a pack as preferred
    @@ builtin/multi-pack-index.c
      #include "trace2.h"
     +#include "object-store.h"
      
    - static char const * const builtin_multi_pack_index_write_usage[] = {
      #define BUILTIN_MIDX_WRITE_USAGE \
     -	N_("git multi-pack-index [<options>] write")
     +	N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]")
    - 	BUILTIN_MIDX_WRITE_USAGE,
    - 	NULL
    - };
    + 
    + #define BUILTIN_MIDX_VERIFY_USAGE \
    + 	N_("git multi-pack-index [<options>] verify")
     @@ builtin/multi-pack-index.c: static char const * const builtin_multi_pack_index_usage[] = {
      
      static struct opts_multi_pack_index {
    @@ midx.c: static void fill_pack_entry(uint32_t pack_int_id,
      						  uint32_t nr_packs,
     -						  uint32_t *nr_objects)
     +						  uint32_t *nr_objects,
    -+						  uint32_t preferred_pack)
    ++						  int preferred_pack)
      {
      	uint32_t cur_fanout, cur_pack, cur_object;
      	uint32_t alloc_fanout, alloc_objects, total_objects = 0;
    @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
      		goto cleanup;
      
     -	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);
    ++	ctx.preferred_pack_idx = -1;
     +	if (preferred_pack_name) {
     +		for (i = 0; i < ctx.nr; i++) {
     +			if (!cmp_idx_or_pack_name(preferred_pack_name,
    @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
     +				break;
     +			}
     +		}
    -+	} else
    -+		ctx.preferred_pack_idx = -1;
    ++	}
     +
     +	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,
     +					 ctx.preferred_pack_idx);
    @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
      			pack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;
      	}
      
    -+	/*
    -+	 * Recompute the preferred_pack_idx (if applicable) according to the
    -+	 * permuted pack order.
    -+	 */
    -+	ctx.preferred_pack_idx = -1;
    ++	/* Check that the preferred pack wasn't expired (if given). */
     +	if (preferred_pack_name) {
    -+		ctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,
    -+							     ctx.nr,
    -+							     preferred_pack_name);
    -+		if (ctx.preferred_pack_idx < 0)
    ++		int preferred_idx = lookup_idx_or_pack_name(ctx.info,
    ++							    ctx.nr,
    ++							    preferred_pack_name);
    ++		if (preferred_idx < 0)
     +			warning(_("unknown preferred pack: '%s'"),
     +				preferred_pack_name);
     +		else {
    -+			uint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;
    ++			uint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;
     +			uint32_t perm = ctx.pack_perm[orig];
     +
    -+			if (perm == PACK_EXPIRED) {
    ++			if (perm == PACK_EXPIRED)
     +				warning(_("preferred pack '%s' is expired"),
     +					preferred_pack_name);
    -+				ctx.preferred_pack_idx = -1;
    -+			} else
    -+				ctx.preferred_pack_idx = perm;
     +		}
     +	}
     +
 9:  976848bc4b =  9:  5c5aca761a midx: don't free midx_name early
10:  5ed47f7e3a = 10:  a22a1463a5 midx: keep track of the checksum
11:  0292508e12 = 11:  efa54479b1 midx: make some functions non-static
12:  404d730498 ! 12:  4745bb8590 Documentation/technical: describe multi-pack reverse indexes

@@ Documentation/technical/pack-format.txt: CHUNK DATA:
+
+Instead of mapping between offset, pack-, and index position, this +reverse index maps between an object's position within the MIDX, and
-+that object's position within a pseudo-pack that the MIDX describes. ++that object's position within a pseudo-pack that the MIDX describes ++(i.e., the ith entry of the multi-pack reverse index holds the MIDX ++position of ith object in pseudo-pack order).
+
-+To clarify these three orderings, consider a multi-pack reachability -+bitmap (which does not yet exist, but is what we are building towards -+here). Each bit needs to correspond to an object in the MIDX, and so we -+need an efficient mapping from bit position to MIDX position. ++To clarify the difference between these orderings, consider a multi-pack ++reachability bitmap (which does not yet exist, but is what we are ++building towards here). Each bit needs to correspond to an object in the ++MIDX, and so we need an efficient mapping from bit position to MIDX ++position.
+
+One solution is to let bits occupy the same position in the oid-sorted +index stored by the MIDX. But because oids are effectively random, there
13: d4e01a44e7 ! 13: a6ebd4be91 pack-revindex: read multi-pack reverse indexes
@@ Commit message
position in the psuedo-pack order, but that order can only be recovered in the .rev file itself. This mapping can be implemented with a binary search, but note that the thing we're binary searching over isn't an
- array, but rather a _permutation_.
+ array of values, but rather a permuted order of those values.

So, when comparing two items, it's helpful to keep in mind the difference. Instead of a traditional binary search, where you are
14: ab7012b283 ! 14: f5314f1822 pack-write.c: extract 'write_rev_file_order'

    @@ pack-write.c: const char *write_rev_file(const char *rev_name,
     +		pack_order[i] = i;
     +	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
     +
    -+	if (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))
    -+		return NULL;
    -+
     +	ret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,
     +				   flags);
     +
15:  01bd6a35c6 ! 15:  fa3acb5d5a pack-revindex: write multi-pack reverse indexes
    @@ Commit message
         for long, since subsequent patches will introduce the multi-pack bitmap,
         which will begin passing this field.
     
    +    (In midx.c:write_midx_internal(), the two adjacent if statements share a
    +    conditional, but are written separately since the first one will
    +    eventually also handle the MIDX_WRITE_BITMAP flag, which does not yet
    +    exist.)
    +
         Signed-off-by: Taylor Blau <[email protected]>
     
      ## midx.c ##
    @@ midx.c: static int write_midx_large_offsets(struct hashfile *f,
     +				     struct write_midx_context *ctx)
     +{
     +	struct strbuf buf = STRBUF_INIT;
    ++	const char *tmp_file;
     +
     +	strbuf_addf(&buf, "%s-%s.rev", midx_name, hash_to_hex(midx_hash));
     +
    -+	write_rev_file_order(buf.buf, ctx->pack_order, ctx->entries_nr,
    -+			     midx_hash, WRITE_REV);
    ++	tmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,
    ++					midx_hash, WRITE_REV);
    ++
    ++	if (finalize_object_file(tmp_file, buf.buf))
    ++		die(_("cannot store reverse index file"));
     +
     +	strbuf_release(&buf);
     +}

-: ---------- > 16: 550e785f10 midx.c: improve cache locality in midx_pack_order_cmp()

Signature · 1 line
2.30.0.667.g81c0cbc6fd
#101 lore raw

PATCH v3 1/16 builtin/multi-pack-index.c: inline 'flags' with options

Subcommands of the 'git multi-pack-index' command (e.g., 'write', 'verify', etc.) will want to optionally change a set of shared flags that are eventually passed to the MIDX libraries.

Right now, options and flags are handled separately. Inline them into the same structure so that sub-commands can more easily share the 'flags' data.

---
 builtin/multi-pack-index.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
builtin/multi-pack-index.c +6 -7
index 5bf88cd2a8..4a0ddb06c4 100644
@@ -14,13 +14,12 @@ static struct opts_multi_pack_index {
 	const char *object_dir;
 	unsigned long batch_size;
 	int progress;
+	unsigned flags;
 } opts;
 
 int cmd_multi_pack_index(int argc, const char **argv,
 			 const char *prefix)
 {
-	unsigned flags = 0;
-
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
@@ -40,7 +39,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
 	if (opts.progress)
-		flags |= MIDX_PROGRESS;
+		opts.flags |= MIDX_PROGRESS;
 
 	if (argc == 0)
 		usage_with_options(builtin_multi_pack_index_usage,
@@ -55,16 +54,16 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	if (!strcmp(argv[0], "repack"))
 		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, flags);
+			(size_t)opts.batch_size, opts.flags);
 	if (opts.batch_size)
 		die(_("--batch-size option is only for 'repack' subcommand"));
 
 	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, flags);
+		return write_midx_file(opts.object_dir, opts.flags);
 	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, flags);
+		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
 	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, flags);
+		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
 
 	die(_("unrecognized subcommand: %s"), argv[0]);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#102 lore raw

Re: [PATCH v3 01/16] builtin/multi-pack-index.c: inline 'flags' with options

Quoted 7 lines · Taylor Blau

Subcommands of the 'git multi-pack-index' command (e.g., 'write', 'verify', etc.) will want to optionally change a set of shared flags that are eventually passed to the MIDX libraries.

Right now, options and flags are handled separately. Inline them into the same structure so that sub-commands can more easily share the 'flags' data.

This "opts" struct is kind of funny. It is used to collect the options in cmd_multi_pack_index(), but nobody ever passes it anywhere! Instead, we pass individual components of it around.

So I'm not sure I buy "...so that sub-commands can more easily share the flags data", since either way they are all receiving the individual flags field already. And your patch 2 could just as easily do the same simplification by modifying the function-local "flags" variable.

But. I think things get more interesting when you later introduce common_opts, because now those options have to refer back to actuals storage for each item. Which means that "flags" would have to become a global variable. And there it's nicer to have all of the options stuffed into a struct, even if it is a single global struct.

So I think this is the right direction, but it took me a minute to realize quite why.

-Peff

#103 lore raw

PATCH v3 2/16 builtin/multi-pack-index.c: don't handle 'progress' separately

In reply to Taylor Blau

Now that there is a shared 'flags' member in the options structure, there is no need to keep track of whether to force progress or not, since ultimately the decision of whether or not to show a progress meter is controlled by a bit in the flags member.

Manipulate that bit directly, and drop the now-unnecessary 'progress' field while we're at it.

---
 builtin/multi-pack-index.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
builtin/multi-pack-index.c +3 -5
index 4a0ddb06c4..c70f020d8f 100644
@@ -13,7 +13,6 @@ static char const * const builtin_multi_pack_index_usage[] = {
 static struct opts_multi_pack_index {
 	const char *object_dir;
 	unsigned long batch_size;
-	int progress;
 	unsigned flags;
 } opts;
 
@@ -23,7 +22,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
+		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	git_config(git_default_config, NULL);
 
-	opts.progress = isatty(2);
+	if (isatty(2))
+		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
 			     builtin_multi_pack_index_usage, 0);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
-	if (opts.progress)
-		opts.flags |= MIDX_PROGRESS;
 
 	if (argc == 0)
 		usage_with_options(builtin_multi_pack_index_usage,
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#104 lore raw

Re: [PATCH v3 02/16] builtin/multi-pack-index.c: don't handle 'progress' separately

Quoted 4 lines · Taylor Blau

Now that there is a shared 'flags' member in the options structure, there is no need to keep track of whether to force progress or not, since ultimately the decision of whether or not to show a progress meter is controlled by a bit in the flags member.

Just going back to what I wrote for patch 1, I think this "now that there is a shared flags..." bit is what misled me.

You can easily have done this patch by just manipulating the local "flags" variable. And the rationale for the patch is "we can get rid of opts.progress, because nobody ever reads it except to set a bit opts.flags".

Definitely not worth re-rolling or anything; I'm just explaining my earlier comments. :)

-Peff

#105 lore raw

PATCH v3 3/16 builtin/multi-pack-index.c: define common usage with a macro

In reply to Taylor Blau

Factor out the usage message into pieces corresponding to each mode. This avoids options specific to one sub-command from being shared with another in the usage.

A subsequent commit will use these #define macros to have usage variables for each sub-command without duplicating their contents.

---
 builtin/multi-pack-index.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
builtin/multi-pack-index.c +16 -1
index c70f020d8f..eea498e026 100644
@@ -5,8 +5,23 @@
 #include "midx.h"
 #include "trace2.h"
 
+#define BUILTIN_MIDX_WRITE_USAGE \
+	N_("git multi-pack-index [<options>] write")
+
+#define BUILTIN_MIDX_VERIFY_USAGE \
+	N_("git multi-pack-index [<options>] verify")
+
+#define BUILTIN_MIDX_EXPIRE_USAGE \
+	N_("git multi-pack-index [<options>] expire")
+
+#define BUILTIN_MIDX_REPACK_USAGE \
+	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
+
 static char const * const builtin_multi_pack_index_usage[] = {
-	N_("git multi-pack-index [<options>] (write|verify|expire|repack --batch-size=<size>)"),
+	BUILTIN_MIDX_WRITE_USAGE,
+	BUILTIN_MIDX_VERIFY_USAGE,
+	BUILTIN_MIDX_EXPIRE_USAGE,
+	BUILTIN_MIDX_REPACK_USAGE,
 	NULL
 };
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#106 lore raw

PATCH v3 4/16 builtin/multi-pack-index.c: split sub-commands

In reply to Taylor Blau

Handle sub-commands of the 'git multi-pack-index' builtin (e.g., "write", "repack", etc.) separately from one another. This allows sub-commands with unique options, without forcing cmd_multi_pack_index() to reject invalid combinations itself.

This comes at the cost of some duplication and boilerplate. Luckily, the duplication is reduced to a minimum, since common options are shared among sub-commands due to a suggestion by Ævar. (Sub-commands do have to retain the common options, too, since this builtin accepts common options on either side of the sub-command).

Roughly speaking, cmd_multi_pack_index() parses options (including common ones), and stops at the first non-option, which is the sub-command. It then dispatches to the appropriate sub-command, which parses the remaining options (also including common options).

Unknown options are kept by the sub-commands in order to detect their presence (and complain that too many arguments were given).

---
 builtin/multi-pack-index.c | 131 ++++++++++++++++++++++++++++++-------
 1 file changed, 106 insertions(+), 25 deletions(-)
builtin/multi-pack-index.c +106 -25
index eea498e026..23e51dfeb4 100644
@@ -17,6 +17,22 @@
 #define BUILTIN_MIDX_REPACK_USAGE \
 	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
 
+static char const * const builtin_multi_pack_index_write_usage[] = {
+	BUILTIN_MIDX_WRITE_USAGE,
+	NULL
+};
+static char const * const builtin_multi_pack_index_verify_usage[] = {
+	BUILTIN_MIDX_VERIFY_USAGE,
+	NULL
+};
+static char const * const builtin_multi_pack_index_expire_usage[] = {
+	BUILTIN_MIDX_EXPIRE_USAGE,
+	NULL
+};
+static char const * const builtin_multi_pack_index_repack_usage[] = {
+	BUILTIN_MIDX_REPACK_USAGE,
+	NULL
+};
 static char const * const builtin_multi_pack_index_usage[] = {
 	BUILTIN_MIDX_WRITE_USAGE,
 	BUILTIN_MIDX_VERIFY_USAGE,
@@ -31,25 +47,99 @@ static struct opts_multi_pack_index {
 	unsigned flags;
 } opts;
 
-int cmd_multi_pack_index(int argc, const char **argv,
-			 const char *prefix)
+static struct option common_opts[] = {
+	OPT_FILENAME(0, "object-dir", &opts.object_dir,
+	  N_("object directory containing set of packfile and pack-index pairs")),
+	OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	OPT_END(),
+};
+
+static struct option *add_common_options(struct option *prev)
 {
-	static struct option builtin_multi_pack_index_options[] = {
-		OPT_FILENAME(0, "object-dir", &opts.object_dir,
-		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	struct option *with_common = parse_options_concat(common_opts, prev);
+	free(prev);
+	return with_common;
+}
+
+static int cmd_multi_pack_index_write(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_write_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_write_usage,
+				   options);
+
+	return write_midx_file(opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_verify(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_verify_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_verify_usage,
+				   options);
+
+	return verify_midx_file(the_repository, opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_expire(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_expire_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_expire_usage,
+				   options);
+
+	return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_repack(int argc, const char **argv)
+{
+	struct option *options;
+	static struct option builtin_multi_pack_index_repack_options[] = {
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
 	};
 
+	options = parse_options_dup(builtin_multi_pack_index_repack_options);
+	options = add_common_options(options);
+
+	argc = parse_options(argc, argv, NULL,
+			     options,
+			     builtin_multi_pack_index_repack_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_repack_usage,
+				   options);
+
+	return midx_repack(the_repository, opts.object_dir,
+			   (size_t)opts.batch_size, opts.flags);
+}
+
+int cmd_multi_pack_index(int argc, const char **argv,
+			 const char *prefix)
+{
+	struct option *builtin_multi_pack_index_options = common_opts;
+
 	git_config(git_default_config, NULL);
 
 	if (isatty(2))
 		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
-			     builtin_multi_pack_index_usage, 0);
+			     builtin_multi_pack_index_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
@@ -58,25 +148,16 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		usage_with_options(builtin_multi_pack_index_usage,
 				   builtin_multi_pack_index_options);
 
-	if (argc > 1) {
-		die(_("too many arguments"));
-		return 1;
-	}
-
 	trace2_cmd_mode(argv[0]);
 
 	if (!strcmp(argv[0], "repack"))
-		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, opts.flags);
-	if (opts.batch_size)
-		die(_("--batch-size option is only for 'repack' subcommand"));
-
-	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
-
-	die(_("unrecognized subcommand: %s"), argv[0]);
+		return cmd_multi_pack_index_repack(argc, argv);
+	else if (!strcmp(argv[0], "write"))
+		return cmd_multi_pack_index_write(argc, argv);
+	else if (!strcmp(argv[0], "verify"))
+		return cmd_multi_pack_index_verify(argc, argv);
+	else if (!strcmp(argv[0], "expire"))
+		return cmd_multi_pack_index_expire(argc, argv);
+	else
+		die(_("unrecognized subcommand: %s"), argv[0]);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#107 lore raw

Re: [PATCH v3 04/16] builtin/multi-pack-index.c: split sub-commands

Quoted 11 lines · Taylor Blau

Handle sub-commands of the 'git multi-pack-index' builtin (e.g., "write", "repack", etc.) separately from one another. This allows sub-commands with unique options, without forcing cmd_multi_pack_index() to reject invalid combinations itself.

This comes at the cost of some duplication and boilerplate. Luckily, the duplication is reduced to a minimum, since common options are shared among sub-commands due to a suggestion by Ævar. (Sub-commands do have to retain the common options, too, since this builtin accepts common options on either side of the sub-command).

Quoted 7 lines

Roughly speaking, cmd_multi_pack_index() parses options (including common ones), and stops at the first non-option, which is the sub-command. It then dispatches to the appropriate sub-command, which parses the remaining options (also including common options).

Unknown options are kept by the sub-commands in order to detect their presence (and complain that too many arguments were given).

Makes sense, and the implementation looks pretty clean.

A few small nits:

builtin/multi-pack-index.c static struct opts_multi_pack_index { · line 62

Quoted 10 lines
builtin/multi-pack-index.c +5 -4
+static struct option *add_common_options(struct option *prev)
 {
-	static struct option builtin_multi_pack_index_options[] = {
-		OPT_FILENAME(0, "object-dir", &opts.object_dir,
-		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	struct option *with_common = parse_options_concat(common_opts, prev);
+	free(prev);
+	return with_common;
+}

This free(prev) pattern is copied from builtin/checkout.c, where we have multiple layers of options, each added by a function. So it requires that callers duplicate the base set of options, and each subsequent "add_foo_options()" concatenates that and frees the old one.

But here, we only have one layer, so in the caller which uses it:

builtin/multi-pack-index.c static struct opts_multi_pack_index { · line 116

Quoted 5 lines
builtin/multi-pack-index.c +4 -0
+static int cmd_multi_pack_index_repack(int argc, const char **argv)
+{
[..]
+	options = parse_options_dup(builtin_multi_pack_index_repack_options);
+	options = add_common_options(options);

we do a rather pointless dup() followed by free(). Perhaps not that big a deal, and this would naturally extend to adding other option sets, so it may even be considered future-proofing. But it did confuse me for a moment.

However, we do end up leaking the return value from add_common_options() at the end of the function:

Quoted 4 lines
builtin/multi-pack-index.c +4 -0
+	options = parse_options_dup(builtin_multi_pack_index_repack_options);
+	options = add_common_options(options);
+
+	argc = parse_options(argc, argv, NULL,

builtin/multi-pack-index.c static struct opts_multi_pack_index { · line 128

Quoted 10 lines
builtin/multi-pack-index.c +10 -0
+			     options,
+			     builtin_multi_pack_index_repack_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_repack_usage,
+				   options);
+
+	return midx_repack(the_repository, opts.object_dir,
+			   (size_t)opts.batch_size, opts.flags);
+}

This is definitely a harmless leak in the sense that we are going to exit the program after midx_repack() returns anyway. But it might be worth keeping things tidy, as we've recently seen a renewed effort to do some leak-checking of the test suite. I _think_ we can just free the options struct (even though we are still using the values themselves, we don't care about the "struct options" anymore). But even if not, an UNLEAK(options) annotation would do it.

(This doesn't apply to the other functions, because they just use common_opts directly).

-Peff

#108 lore raw

Re: [PATCH v3 04/16] builtin/multi-pack-index.c: split sub-commands

Quoted 7 lines · Jeff King

This is definitely a harmless leak in the sense that we are going to exit the program after midx_repack() returns anyway. But it might be worth keeping things tidy, as we've recently seen a renewed effort to do some leak-checking of the test suite. I _think_ we can just free the options struct (even though we are still using the values themselves, we don't care about the "struct options" anymore). But even if not, an UNLEAK(options) annotation would do it.

I see what you're saying. Let me make sure that I got the right idea in mind after reading your email. I'm thinking of squashing the following diff into this patch. For what it's worth, it causes 'valgrind --leak-check=full ./git-multi-pack-index repack' to exit cleanly (when it didn't before).

Does this match your expectations?

--- >8 ---

builtin/multi-pack-index.c +4 -5
index 23e51dfeb4..a78640c061 100644
@@ -56,9 +56,7 @@ static struct option common_opts[] = {
 
 static struct option *add_common_options(struct option *prev)
 {
-	struct option *with_common = parse_options_concat(common_opts, prev);
-	free(prev);
-	return with_common;
+	return parse_options_concat(common_opts, prev);
 }
 
 static int cmd_multi_pack_index_write(int argc, const char **argv)
@@ -112,8 +110,7 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)
 		OPT_END(),
 	};
 
-	options = parse_options_dup(builtin_multi_pack_index_repack_options);
-	options = add_common_options(options);
+	options = add_common_options(builtin_multi_pack_index_repack_options);
 
 	argc = parse_options(argc, argv, NULL,
 			     options,
@@ -123,6 +120,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)
 		usage_with_options(builtin_multi_pack_index_repack_usage,
 				   options);
 
+	FREE_AND_NULL(options);
+
 	return midx_repack(the_repository, opts.object_dir,
 			   (size_t)opts.batch_size, opts.flags);
 }
#109 lore raw

Re: [PATCH v3 04/16] builtin/multi-pack-index.c: split sub-commands

Quoted 9 lines · Taylor Blau
Quoted 7 lines · Jeff King

This is definitely a harmless leak in the sense that we are going to exit the program after midx_repack() returns anyway. But it might be worth keeping things tidy, as we've recently seen a renewed effort to do some leak-checking of the test suite. I _think_ we can just free the options struct (even though we are still using the values themselves, we don't care about the "struct options" anymore). But even if not, an UNLEAK(options) annotation would do it.

Quoted 7 lines

I see what you're saying. Let me make sure that I got the right idea in mind after reading your email. I'm thinking of squashing the following diff into this patch. For what it's worth, it causes 'valgrind --leak-check=full ./git-multi-pack-index repack' to exit cleanly (when it didn't before).

Does this match your expectations?

Yes, though...

builtin/multi-pack-index.c static struct option common_opts[] = { · line 60

Quoted 13 lines
builtin/multi-pack-index.c +1 -3
index 23e51dfeb4..a78640c061 100644
@@ -56,9 +56,7 @@ static struct option common_opts[] = {
 
 static struct option *add_common_options(struct option *prev)
 {
-	struct option *with_common = parse_options_concat(common_opts, prev);
-	free(prev);
-	return with_common;
+	return parse_options_concat(common_opts, prev);
 }

This simplification is orthogonal to the leak, and I'd be OK if you wanted to retain it as it was before (because it future-proofs against adding more add_foo_options() later, though for now it is a useless dup/free pair).

builtin/multi-pack-index.c static int cmd_multi_pack_index_repack(int argc, const char **argv) · line 124

Quoted 6 lines
builtin/multi-pack-index.c +2 -0
@@ -123,6 +120,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)
 		usage_with_options(builtin_multi_pack_index_repack_usage,
 				   options);
 
+	FREE_AND_NULL(options);
+

And this is the leak fix I care about. We'd want the same thing in the later caller that adds another use of add_common_options(), of course.

-Peff

#110 lore raw

PATCH v3 5/16 builtin/multi-pack-index.c: don't enter bogus cmd_mode

In reply to Taylor Blau

Even before the recent refactoring, 'git multi-pack-index' calls 'trace2_cmd_mode()' before verifying that the sub-command is recognized.

Push this call down into the individual sub-commands so that we don't enter a bogus command mode.

---
 builtin/multi-pack-index.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
builtin/multi-pack-index.c +8 -2
index 23e51dfeb4..b5678cc2bb 100644
@@ -65,6 +65,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_write_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -79,6 +81,8 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_verify_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -93,6 +97,8 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_expire_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -115,6 +121,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)
 	options = parse_options_dup(builtin_multi_pack_index_repack_options);
 	options = add_common_options(options);
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options,
 			     builtin_multi_pack_index_repack_usage,
@@ -148,8 +156,6 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		usage_with_options(builtin_multi_pack_index_usage,
 				   builtin_multi_pack_index_options);
 
-	trace2_cmd_mode(argv[0]);
-
 	if (!strcmp(argv[0], "repack"))
 		return cmd_multi_pack_index_repack(argc, argv);
 	else if (!strcmp(argv[0], "write"))
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#111 lore raw

PATCH v3 6/16 builtin/multi-pack-index.c: display usage on unrecognized command

In reply to Taylor Blau

When given a sub-command that it doesn't understand, 'git multi-pack-index' dies with the following message:

    $ git multi-pack-index bogus
    fatal: unrecognized subcommand: bogus

Instead of 'die()'-ing, we can display the usage text, which is much more helpful:

    $ git.compile multi-pack-index bogus
    usage: git multi-pack-index [<options>] write
       or: git multi-pack-index [<options>] verify
       or: git multi-pack-index [<options>] expire
       or: git multi-pack-index [<options>] repack [--batch-size=<size>]

--object-dir <file> object directory containing set of packfile and pack-index pairs
--progress force progress reporting

While we're at it, clean up some duplication between the "no sub-command" and "unrecognized sub-command" conditionals.

---
 builtin/multi-pack-index.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
builtin/multi-pack-index.c +4 -3
index b5678cc2bb..243b6ccc7c 100644
@@ -153,8 +153,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		opts.object_dir = get_object_directory();
 
 	if (argc == 0)
-		usage_with_options(builtin_multi_pack_index_usage,
-				   builtin_multi_pack_index_options);
+		goto usage;
 
 	if (!strcmp(argv[0], "repack"))
 		return cmd_multi_pack_index_repack(argc, argv);
@@ -165,5 +164,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	else if (!strcmp(argv[0], "expire"))
 		return cmd_multi_pack_index_expire(argc, argv);
 	else
-		die(_("unrecognized subcommand: %s"), argv[0]);
+usage:
+		usage_with_options(builtin_multi_pack_index_usage,
+				   builtin_multi_pack_index_options);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#112 lore raw

Re: [PATCH v3 06/16] builtin/multi-pack-index.c: display usage on unrecognized command

Quoted 15 lines · Taylor Blau

When given a sub-command that it doesn't understand, 'git multi-pack-index' dies with the following message:

    $ git multi-pack-index bogus
    fatal: unrecognized subcommand: bogus

Instead of 'die()'-ing, we can display the usage text, which is much more helpful:

    $ git.compile multi-pack-index bogus
    usage: git multi-pack-index [<options>] write
       or: git multi-pack-index [<options>] verify
       or: git multi-pack-index [<options>] expire
       or: git multi-pack-index [<options>] repack [--batch-size=<size>]
Quoted 5 lines

--object-dir <file> object directory containing set of packfile and pack-index pairs
--progress force progress reporting

While we're at it, clean up some duplication between the "no sub-command" and "unrecognized sub-command" conditionals.

I agree that it's much nicer to give the usage. But my preference in general for cases like this is to _also_ explain what we found wrong with the options we were given.

E.g., with a bogus option, we say so:

  $ git multi-pack-index --foo
  error: unknown option `foo'
  usage: git multi-pack-index [<options>] write [--preferred-pack=<pack>]
  [etc...]

but with a bogus sub-command, we get just the usage string:

  $ git multi-pack-index foo
  usage: git multi-pack-index [<options>] write [--preferred-pack=<pack>]
  [etc...]

Sometimes it is quote obvious what is wrong, but sometimes typos can be hard to spot, especially because the usage message is so long.

I.e., I'd suggest changing this:

builtin/multi-pack-index.c int cmd_multi_pack_index(int argc, const char **argv, · line 169

Quoted 5 lines
builtin/multi-pack-index.c +3 -1
 	else
-		die(_("unrecognized subcommand: %s"), argv[0]);
+usage:
+		usage_with_options(builtin_multi_pack_index_usage,
+				   builtin_multi_pack_index_options);

to:

  error(_("unrecognized subcommand: %s"), argv[0]);
  usage_with_options(...);

-Peff

#113 lore raw

Re: [PATCH v3 06/16] builtin/multi-pack-index.c: display usage on unrecognized command

Quoted 10 lines · Jeff King

I.e., I'd suggest changing this:

builtin/multi-pack-index.c int cmd_multi_pack_index(int argc, const char **argv, · line 169

Quoted 5 lines
builtin/multi-pack-index.c +3 -1
 	else
-		die(_("unrecognized subcommand: %s"), argv[0]);
+usage:
+		usage_with_options(builtin_multi_pack_index_usage,
+				   builtin_multi_pack_index_options);

to:

Quoted 2 lines
  error(_("unrecognized subcommand: %s"), argv[0]);
  usage_with_options(...);

Thanks, that's a helpful suggestion (and pretty easy to change, too).

Thanks,
Taylor

#114 lore raw

PATCH v3 7/16 t/helper/test-read-midx.c: add '--show-objects'

In reply to Taylor Blau

The 'read-midx' helper is used in places like t5319 to display basic information about a multi-pack-index.

In the next patch, the MIDX writing machinery will learn a new way to choose from which pack an object is selected when multiple copies of that object exist.

To disambiguate which pack introduces an object so that this feature can be tested, add a '--show-objects' option which displays additional information about each object in the MIDX.

---
 t/helper/test-read-midx.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)
t/helper/test-read-midx.c +20 -4
index 2430880f78..7c2eb11a8e 100644
@@ -4,7 +4,7 @@
 #include "repository.h"
 #include "object-store.h"
 
-static int read_midx_file(const char *object_dir)
+static int read_midx_file(const char *object_dir, int show_objects)
 {
 	uint32_t i;
 	struct multi_pack_index *m;
@@ -43,13 +43,29 @@ static int read_midx_file(const char *object_dir)
 
 	printf("object-dir: %s\n", m->object_dir);
 
+	if (show_objects) {
+		struct object_id oid;
+		struct pack_entry e;
+
+		for (i = 0; i < m->num_objects; i++) {
+			nth_midxed_object_oid(&oid, m, i);
+			fill_midx_entry(the_repository, &oid, &e, m);
+
+			printf("%s %"PRIu64"\t%s\n",
+			       oid_to_hex(&oid), e.offset, e.p->pack_name);
+		}
+		return 0;
+	}
+
 	return 0;
 }
 
 int cmd__read_midx(int argc, const char **argv)
 {
-	if (argc != 2)
-		usage("read-midx <object-dir>");
+	if (!(argc == 2 || argc == 3))
+		usage("read-midx [--show-objects] <object-dir>");
 
-	return read_midx_file(argv[1]);
+	if (!strcmp(argv[1], "--show-objects"))
+		return read_midx_file(argv[2], 1);
+	return read_midx_file(argv[1], 0);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#115 lore raw

PATCH v3 8/16 midx: allow marking a pack as preferred

In reply to Taylor Blau

When multiple packs in the multi-pack index contain the same object, the MIDX machinery must make a choice about which pack it associates with that object. Prior to this patch, the lowest-ordered[1] pack was always selected.

Pack selection for duplicate objects is relatively unimportant today, but it will become important for multi-pack bitmaps. This is because we can only invoke the pack-reuse mechanism when all of the bits for reused objects come from the reuse pack (in order to ensure that all reused deltas can find their base objects in the same pack).

To encourage the pack selection process to prefer one pack over another (the pack to be preferred is the one a caller would like to later use as a reuse pack), introduce the concept of a "preferred pack". When provided, the MIDX code will always prefer an object found in a preferred pack over any other.

No format changes are required to store the preferred pack, since it will be able to be inferred with a corresponding MIDX bitmap, by looking up the pack associated with the object in the first bit position (this ordering is described in detail in a subsequent commit).

[1]: the ordering is specified by MIDX internals; for our purposes we can consider the "lowest ordered" pack to be "the one with the most-recent mtime.

---
 Documentation/git-multi-pack-index.txt       | 14 ++-
 Documentation/technical/multi-pack-index.txt |  5 +-
 builtin/multi-pack-index.c                   | 18 +++-
 builtin/repack.c                             |  2 +-
 midx.c                                       | 92 ++++++++++++++++++--
 midx.h                                       |  2 +-
 t/t5319-multi-pack-index.sh                  | 39 +++++++++
 7 files changed, 154 insertions(+), 18 deletions(-)
Documentation/git-multi-pack-index.txt +12 -2
index eb0caa0439..ffd601bc17 100644
@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes
 SYNOPSIS
 --------
 [verse]
-'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress] <subcommand>
+'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
+	[--preferred-pack=<pack>] <subcommand>
 
 DESCRIPTION
 -----------
@@ -30,7 +31,16 @@ OPTIONS
 The following subcommands are available:
 
 write::
-	Write a new MIDX file.
+	Write a new MIDX file. The following options are available for
+	the `write` sub-command:
++
+--
+	--preferred-pack=<pack>::
+		Optionally specify the tie-breaking pack used when
+		multiple packs contain the same object. If not given,
+		ties are broken in favor of the pack with the lowest
+		mtime.
+--
 
 verify::
 	Verify the contents of the MIDX file.
Documentation/technical/multi-pack-index.txt +3 -2
index e8e377a59f..fb688976c4 100644
@@ -43,8 +43,9 @@ Design Details
   a change in format.
 
 - The MIDX keeps only one record per object ID. If an object appears
-  in multiple packfiles, then the MIDX selects the copy in the most-
-  recently modified packfile.
+  in multiple packfiles, then the MIDX selects the copy in the
+  preferred packfile, otherwise selecting from the most-recently
+  modified packfile.
 
 - If there exist packfiles in the pack directory not registered in
   the MIDX, then those packfiles are loaded into the `packed_git`
builtin/multi-pack-index.c +15 -3
index 243b6ccc7c..92f358f212 100644
@@ -4,9 +4,10 @@
 #include "parse-options.h"
 #include "midx.h"
 #include "trace2.h"
+#include "object-store.h"
 
 #define BUILTIN_MIDX_WRITE_USAGE \
-	N_("git multi-pack-index [<options>] write")
+	N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]")
 
 #define BUILTIN_MIDX_VERIFY_USAGE \
 	N_("git multi-pack-index [<options>] verify")
@@ -43,6 +44,7 @@ static char const * const builtin_multi_pack_index_usage[] = {
 
 static struct opts_multi_pack_index {
 	const char *object_dir;
+	const char *preferred_pack;
 	unsigned long batch_size;
 	unsigned flags;
 } opts;
@@ -63,7 +65,16 @@ static struct option *add_common_options(struct option *prev)
 
 static int cmd_multi_pack_index_write(int argc, const char **argv)
 {
-	struct option *options = common_opts;
+	struct option *options;
+	static struct option builtin_multi_pack_index_write_options[] = {
+		OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
+			   N_("preferred-pack"),
+			   N_("pack for reuse when computing a multi-pack bitmap")),
+		OPT_END(),
+	};
+
+	options = parse_options_dup(builtin_multi_pack_index_write_options);
+	options = add_common_options(options);
 
 	trace2_cmd_mode(argv[0]);
 
@@ -74,7 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		usage_with_options(builtin_multi_pack_index_write_usage,
 				   options);
 
-	return write_midx_file(opts.object_dir, opts.flags);
+	return write_midx_file(opts.object_dir, opts.preferred_pack,
+			       opts.flags);
 }
 
 static int cmd_multi_pack_index_verify(int argc, const char **argv)
builtin/repack.c +1 -1
index 01440de2d5..9f00806805 100644
@@ -523,7 +523,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 	remove_temporary_files();
 
 	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
-		write_midx_file(get_object_directory(), 0);
+		write_midx_file(get_object_directory(), NULL, 0);
 
 	string_list_clear(&names, 0);
 	string_list_clear(&rollback, 0);
midx.c +83 -9

189 lines not on this page · raw message

midx.h +1 -1
index b18cf53bc4..e7fea61109 100644
@@ -47,7 +47,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa
 int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);
 int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);
 
-int write_midx_file(const char *object_dir, unsigned flags);
+int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);
 void clear_midx_file(struct repository *r);
 int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);
 int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);
t/t5319-multi-pack-index.sh +39 -0
index b4afab1dfc..fd94ba9053 100755
@@ -31,6 +31,14 @@ midx_read_expect () {
 	test_cmp expect actual
 }
 
+midx_expect_object_offset () {
+	OID="$1"
+	OFFSET="$2"
+	OBJECT_DIR="$3"
+	test-tool read-midx --show-objects $OBJECT_DIR >actual &&
+	grep "^$OID $OFFSET" actual
+}
+
 test_expect_success 'setup' '
 	test_oid_cache <<-EOF
 	idxoff sha1:2999
@@ -234,6 +242,37 @@ test_expect_success 'warn on improper hash version' '
 	)
 '
 
+test_expect_success 'midx picks objects from preferred pack' '
+	test_when_finished rm -rf preferred.git &&
+	git init --bare preferred.git &&
+	(
+		cd preferred.git &&
+
+		a=$(echo "a" | git hash-object -w --stdin) &&
+		b=$(echo "b" | git hash-object -w --stdin) &&
+		c=$(echo "c" | git hash-object -w --stdin) &&
+
+		# Set up two packs, duplicating the object "B" at different
+		# offsets.
+		git pack-objects objects/pack/test-AB <<-EOF &&
+		$a
+		$b
+		EOF
+		bc=$(git pack-objects objects/pack/test-BC <<-EOF
+		$b
+		$c
+		EOF
+		) &&
+
+		git multi-pack-index --object-dir=objects \
+			write --preferred-pack=test-BC-$bc.idx 2>err &&
+		test_must_be_empty err &&
+
+		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |
+			cut -d" " -f1) &&
+		midx_expect_object_offset $b $ofs objects
+	)
+'
 
 test_expect_success 'verify multi-pack-index success' '
 	git multi-pack-index verify --object-dir=$objdir
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#116 lore raw

Re: [PATCH v3 08/16] midx: allow marking a pack as preferred

Quoted 10 lines · Taylor Blau

To encourage the pack selection process to prefer one pack over another (the pack to be preferred is the one a caller would like to later use as a reuse pack), introduce the concept of a "preferred pack". When provided, the MIDX code will always prefer an object found in a preferred pack over any other.

No format changes are required to store the preferred pack, since it will be able to be inferred with a corresponding MIDX bitmap, by looking up the pack associated with the object in the first bit position (this ordering is described in detail in a subsequent commit).

I think in the long run we may want to add a midx chunk that gives the order of the packs (and likewise allow the caller of "midx write" to specify the exact order), since that may allow correlating locality between history and object order within the .rev/.bitmap files.

But I think this is a nice stopping point for this series, since we're not having to introduce any new on-disk formats to do it, and it seems to give pretty good results in practice. I guess we'll have to support --preferred-pack forever, but that's OK. Even if we do eventually support arbitrary orderings, it's just a simple subset of that functionality.

Quoted 17 lines

static int cmd_multi_pack_index_write(int argc, const char **argv) {

builtin/multi-pack-index.c +10 -1
-	struct option *options = common_opts;
+	struct option *options;
+	static struct option builtin_multi_pack_index_write_options[] = {
+		OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
+			   N_("preferred-pack"),
+			   N_("pack for reuse when computing a multi-pack bitmap")),
+		OPT_END(),
+	};
+
+	options = parse_options_dup(builtin_multi_pack_index_write_options);
+	options = add_common_options(options);
 
 	trace2_cmd_mode(argv[0]);

builtin/multi-pack-index.c static int cmd_multi_pack_index_write(int argc, const char **argv) · line 90

Quoted 7 lines
builtin/multi-pack-index.c +2 -1
@@ -74,7 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		usage_with_options(builtin_multi_pack_index_write_usage,
 				   options);
 
-	return write_midx_file(opts.object_dir, opts.flags);
+	return write_midx_file(opts.object_dir, opts.preferred_pack,
+			       opts.flags);
 }

This has the same leak of "options" that I mentioned in the earlier patch.

Quoted 15 lines
midx.c +7 -0
index 971faa8cfc..46f55ff6cf 100644
@@ -431,6 +431,24 @@ static int pack_info_compare(const void *_a, const void *_b)
 	return strcmp(a->pack_name, b->pack_name);
 }
 
+static int lookup_idx_or_pack_name(struct pack_info *info,
+				   uint32_t nr,
+				   const char *pack_name)
+{
+	uint32_t lo = 0, hi = nr;
+	while (lo < hi) {
+		uint32_t mi = lo + (hi - lo) / 2;

midx.c static int pack_info_compare(const void *_a, const void *_b) · line 450

Quoted 10 lines
midx.c +10 -0
+		int cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);
+		if (cmp < 0)
+			hi = mi;
+		else if (cmp > 0)
+			lo = mi + 1;
+		else
+			return mi;
+	}
+	return -1;
+}

Could this just be replaced with bsearch() in the caller?

Quoted 11 lines
midx.c +11 -0
+test_expect_success 'midx picks objects from preferred pack' '
+	test_when_finished rm -rf preferred.git &&
+	git init --bare preferred.git &&
+	(
+		cd preferred.git &&
+
+		a=$(echo "a" | git hash-object -w --stdin) &&
+		b=$(echo "b" | git hash-object -w --stdin) &&
+		c=$(echo "c" | git hash-object -w --stdin) &&
+
+		# Set up two packs, duplicating the object "B" at different

midx.c static int pack_info_compare(const void *_a, const void *_b)

Quoted 10 lines
midx.c +10 -0
+		# offsets.
+		git pack-objects objects/pack/test-AB <<-EOF &&
+		$a
+		$b
+		EOF
+		bc=$(git pack-objects objects/pack/test-BC <<-EOF
+		$b
+		$c
+		EOF
+		) &&

I don't think pack-objects guarantees that the pack ordering matches the input it received. compute_write_order() uses a variety of heuristics to reorder things. I think this will work in practice with the current code, because the objects have the same type, there are no deltas, and the fallback ordering is input-order (or traversal order, if --revs is used).

So it's probably OK in practice, though if we wanted to be paranoid we could check that show-index produces different results for the $b entry of both packs. That said...

midx.c

Quoted 8 lines
midx.c +8 -0
+		git multi-pack-index --object-dir=objects \
+			write --preferred-pack=test-BC-$bc.idx 2>err &&
+		test_must_be_empty err &&
+
+		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |
+			cut -d" " -f1) &&
+		midx_expect_object_offset $b $ofs objects
+	)

...what we really care about is that the object came from BC. And we are just using the offset as a proxy for that. But doesn't "test-tool read-midx" give us the actual pack name? We could just be checking that.

I also wondered if we should confirm that without the --preferred-pack option, we choose the other pack. I think it will always be true because the default order is to sort them lexically. A comment to that effect might be worth it (near the "set up two packs" comment).

-Peff

#117 lore raw

Re: [PATCH v3 08/16] midx: allow marking a pack as preferred

Quoted 11 lines · Jeff King

I think in the long run we may want to add a midx chunk that gives the order of the packs (and likewise allow the caller of "midx write" to specify the exact order), since that may allow correlating locality between history and object order within the .rev/.bitmap files.

But I think this is a nice stopping point for this series, since we're not having to introduce any new on-disk formats to do it, and it seems to give pretty good results in practice. I guess we'll have to support --preferred-pack forever, but that's OK. Even if we do eventually support arbitrary orderings, it's just a simple subset of that functionality.

To add a little bit of extra detail, I think what you're getting at here is that it would be nice to let the order of the packs be dictated by mtime, not the order they appear in the MIDX (which is lexicographic by their hash, and thus effectively random).

The reason there being the same as you pointed out in

https://lore.kernel.org/git/YDRdmh8oS5%[email protected]/

which is that it effectively would lay objects out from newest to oldest.

But, there's a problem, which is that the MIDX doesn't store the packs' mtimes. That's fine for writing, since we can just look that information up ourselves. But the reading side can get broken. That's because the reader also has to know the pack order to go from MIDX- to bit-position.

So if a third party goes and touches some of the packs after the .rev file was written, then the reader is going to think the packs ought to appear in a different order than they actually do. So relying on having to look up the mtimes again later on isn't good enough.

There are two solutions to the problem:

- You could write the mtimes in the MIDX itself. This would give you a single point of reference, and resolve the TOCTOU race I just described.

- Or, you could forget about mtimes entirely and let the MIDX dictate the pack ordering itself. That resolves the race in a similar-but-different way.

Of the two, I prefer the latter, but I think it introduces functionality that we don't necessarily need yet. That's because the objects within the packs are still ordered as such, and so the compression we get in the packs is just as good as it is for single-pack bitmaps. It's only at the objects between pack boundaries that any runs of 1s or 0s might be interrupted, but there are far fewer pack boundaries than objects, so it doesn't seem to matter in practice.

Anyway, I think that you know all of that already (mostly because we thought aloud together when I originally brought this up), but I figure that this detail may be interesting for other readers, too.

Quoted 11 lines

builtin/multi-pack-index.c static int cmd_multi_pack_index_write(int argc, const char **argv) · line 90

Quoted 8 lines
builtin/multi-pack-index.c +2 -1
@@ -74,7 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		usage_with_options(builtin_multi_pack_index_write_usage,
 				   options);
 
-	return write_midx_file(opts.object_dir, opts.flags);
+	return write_midx_file(opts.object_dir, opts.preferred_pack,
+			       opts.flags);
 }

This has the same leak of "options" that I mentioned in the earlier patch.

Yup, thanks for pointing it out.

Quoted 19 lines
Quoted 19 lines
midx.c +11 -0
index 971faa8cfc..46f55ff6cf 100644
@@ -431,6 +431,24 @@ static int pack_info_compare(const void *_a, const void *_b)
 	return strcmp(a->pack_name, b->pack_name);
 }
 
+static int lookup_idx_or_pack_name(struct pack_info *info,
+				   uint32_t nr,
+				   const char *pack_name)
+{
+	uint32_t lo = 0, hi = nr;
+	while (lo < hi) {
+		uint32_t mi = lo + (hi - lo) / 2;
+		int cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);
+		if (cmp < 0)
+			hi = mi;
+		else if (cmp > 0)
Quoted 8 lines

midx.c static int pack_info_compare(const void *_a, const void *_b) · line 450

Quoted 6 lines
midx.c +6 -0
+			lo = mi + 1;
+		else
+			return mi;
+	}
+	return -1;
+}

Could this just be replaced with bsearch() in the caller?

Great suggestion. Yes, it can be. FWIW, I think that I may have originally thought that it couldn't be since we were comparing a fixed string to an array of structs (each having a field which holds the value we actually want to compare). But bsearch() always passes the key as the first argument to the comparator, so this is possible to do.

Quoted 13 lines

midx.c static int pack_info_compare(const void *_a, const void *_b)

Quoted 8 lines
midx.c +8 -0
+		git multi-pack-index --object-dir=objects \
+			write --preferred-pack=test-BC-$bc.idx 2>err &&
+		test_must_be_empty err &&
+
+		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |
+			cut -d" " -f1) &&
+		midx_expect_object_offset $b $ofs objects
+	)

...what we really care about is that the object came from BC. And we are just using the offset as a proxy for that. But doesn't "test-tool read-midx" give us the actual pack name? We could just be checking that.

Quoted 4 lines

I also wondered if we should confirm that without the --preferred-pack option, we choose the other pack. I think it will always be true because the default order is to sort them lexically. A comment to that effect might be worth it (near the "set up two packs" comment).

Both great points, thanks.

Quoted 1 line

-Peff

Thanks,
Taylor

#118 lore raw

Re: [PATCH v3 08/16] midx: allow marking a pack as preferred

Quoted 6 lines · Taylor Blau

There are two solutions to the problem:

- You could write the mtimes in the MIDX itself. This would give you a single point of reference, and resolve the TOCTOU race I just described.

Quoted 6 lines

- Or, you could forget about mtimes entirely and let the MIDX dictate the pack ordering itself. That resolves the race in a similar-but-different way.

Of the two, I prefer the latter, but I think it introduces functionality that we don't necessarily need yet.

Yeah, I'd strongly favor the latter over the former. The reason to go with the solution you have in this series is that it doesn't require changing anything in the on-disk midx format, and we think it is good enough. But once we are going to change the on-disk format, we might as well give the writing side as much flexibility as possible.

Of course the mtimes themselves are really just numbers, so in a sense the two are really equivalent. ;)

Quoted 6 lines

That's because the objects within
the packs are still ordered as such, and so the compression we get in the packs is just as good as it is for single-pack bitmaps. It's only at the objects between pack boundaries that any runs of 1s or 0s might be interrupted, but there are far fewer pack boundaries than objects, so it doesn't seem to matter in practice.

Right. The absolute worst case is a large number of single-object packs, in which case the bitmap order becomes essentially random with respect to history (because it would be sorted by sha1 of the packs).

The effect _might_ be measurable in more real-world cases, like say one big pack and 100 pushes each with a handful of commits. The big pack would be in good shape, but you have a lot of extra pack boundaries that hurt the bitmap compression.

But in practice, generating bitmaps is expensive enough that you'd probably want to roll up some of the packs anyway (and that is certainly what we are doing at GitHub, using your "repack --geometric"). So you'd end usually with one big pack representing most of history, and then a handful of roll-up packs.

So I'm a little curious whether one could even measure the impact of, say, 100 little packs. But not enough to even run the experiment, because even that is not a case that is really that interesting.

Quoted 3 lines

Anyway, I think that you know all of that already (mostly because we thought aloud together when I originally brought this up), but I figure that this detail may be interesting for other readers, too.

Indeed. And I know that you know everything I just wrote, but I agree it's nice to get a record of these discussions onto the list. :)

-Peff

#119 lore raw

PATCH v3 9/16 midx: don't free midx_name early

In reply to Taylor Blau

A subsequent patch will need to refer back to 'midx_name' later on in the function. In fact, this variable is already free()'d later on, so this makes the later free() no longer redundant.

---
 midx.c | 1 -
 1 file changed, 1 deletion(-)
midx.c +0 -1
index 46f55ff6cf..e0009d3314 100644
@@ -966,7 +966,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
-	FREE_AND_NULL(midx_name);
 
 	if (ctx.m)
 		close_midx(ctx.m);
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#120 lore raw

PATCH v3 10/16 midx: keep track of the checksum

In reply to Taylor Blau

write_midx_internal() uses a hashfile to write the multi-pack index, but discards its checksum. This makes sense, since nothing that takes place after writing the MIDX cares about its checksum.

That is about to change in a subsequent patch, when the optional reverse index corresponding to the MIDX will want to include the MIDX's checksum.

Store the checksum of the MIDX in preparation for that.

---
 midx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
midx.c +2 -1
index e0009d3314..31e6d3d2df 100644
@@ -821,6 +821,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			       unsigned flags)
 {
 	char *midx_name;
+	unsigned char midx_hash[GIT_MAX_RAWSZ];
 	uint32_t i;
 	struct hashfile *f = NULL;
 	struct lock_file lk;
@@ -997,7 +998,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
 	write_chunkfile(cf, &ctx);
 
-	finalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
 	commit_lock_file(&lk);
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#121 lore raw

PATCH v3 11/16 midx: make some functions non-static

In reply to Taylor Blau

In a subsequent commit, pack-revindex.c will become responsible for sorting a list of objects in the "MIDX pack order" (which will be defined in the following patch). To do so, it will need to be know the pack identifier and offset within that pack for each object in the MIDX.

The MIDX code already has functions for doing just that (nth_midxed_offset() and nth_midxed_pack_int_id()), but they are statically declared.

Since there is no reason that they couldn't be exposed publicly, and because they are already doing exactly what the caller in pack-revindex.c will want, expose them publicly so that they can be reused there.

---
 midx.c | 4 ++--
 midx.h | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)
midx.c +2 -2
index 31e6d3d2df..0a5da49ed6 100644
@@ -239,7 +239,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,
 	return oid;
 }
 
-static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
 {
 	const unsigned char *offset_data;
 	uint32_t offset32;
@@ -258,7 +258,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
 	return offset32;
 }
 
-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
 {
 	return get_be32(m->chunk_object_offsets +
 			(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);
midx.h +2 -0
index e7fea61109..93bd68189e 100644
@@ -40,6 +40,8 @@ struct multi_pack_index {
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);
+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);
 struct object_id *nth_midxed_object_oid(struct object_id *oid,
 					struct multi_pack_index *m,
 					uint32_t n);
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#122 lore raw

PATCH v3 12/16 Documentation/technical: describe multi-pack reverse indexes

In reply to Taylor Blau

As a prerequisite to implementing multi-pack bitmaps, motivate and describe the format and ordering of the multi-pack reverse index.

The subsequent patch will implement reading this format, and the patch after that will implement writing it while producing a multi-pack index.

---
 Documentation/technical/pack-format.txt | 83 +++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
Documentation/technical/pack-format.txt +83 -0
index 1faa949bf6..4bbbb188a4 100644
@@ -379,3 +379,86 @@ CHUNK DATA:
 TRAILER:
 
 	Index checksum of the above contents.
+
+== multi-pack-index reverse indexes
+
+Similar to the pack-based reverse index, the multi-pack index can also
+be used to generate a reverse index.
+
+Instead of mapping between offset, pack-, and index position, this
+reverse index maps between an object's position within the MIDX, and
+that object's position within a pseudo-pack that the MIDX describes
+(i.e., the ith entry of the multi-pack reverse index holds the MIDX
+position of ith object in pseudo-pack order).
+
+To clarify the difference between these orderings, consider a multi-pack
+reachability bitmap (which does not yet exist, but is what we are
+building towards here). Each bit needs to correspond to an object in the
+MIDX, and so we need an efficient mapping from bit position to MIDX
+position.
+
+One solution is to let bits occupy the same position in the oid-sorted
+index stored by the MIDX. But because oids are effectively random, there
+resulting reachability bitmaps would have no locality, and thus compress
+poorly. (This is the reason that single-pack bitmaps use the pack
+ordering, and not the .idx ordering, for the same purpose.)
+
+So we'd like to define an ordering for the whole MIDX based around
+pack ordering, which has far better locality (and thus compresses more
+efficiently). We can think of a pseudo-pack created by the concatenation
+of all of the packs in the MIDX. E.g., if we had a MIDX with three packs
+(a, b, c), with 10, 15, and 20 objects respectively, we can imagine an
+ordering of the objects like:
+
+    |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|
+
+where the ordering of the packs is defined by the MIDX's pack list,
+and then the ordering of objects within each pack is the same as the
+order in the actual packfile.
+
+Given the list of packs and their counts of objects, you can
+na&iuml;vely reconstruct that pseudo-pack ordering (e.g., the object at
+position 27 must be (c,1) because packs "a" and "b" consumed 25 of the
+slots). But there's a catch. Objects may be duplicated between packs, in
+which case the MIDX only stores one pointer to the object (and thus we'd
+want only one slot in the bitmap).
+
+Callers could handle duplicates themselves by reading objects in order
+of their bit-position, but that's linear in the number of objects, and
+much too expensive for ordinary bitmap lookups. Building a reverse index
+solves this, since it is the logical inverse of the index, and that
+index has already removed duplicates. But, building a reverse index on
+the fly can be expensive. Since we already have an on-disk format for
+pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,
+too.
+
+Objects from the MIDX are ordered as follows to string together the
+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected
+by the MIDX, and define an ordering of packs based on their numeric ID
+(as stored by the MIDX). Let _offset(o)_ return the object offset of _o_
+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:
+
+  - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other
+    is not, then the preferred one sorts first.
++
+(This is a detail that allows the MIDX bitmap to determine which
+pack should be used by the pack-reuse mechanism, since it can ask
+the MIDX for the pack containing the object at bit position 0).
+
+  - If _pack(o~1~) &ne; pack(o~2~)_, then sort the two objects in
+    descending order based on the pack ID.
+
+  - Otherwise, _pack(o~1~) &equals; pack(o~2~)_, and the objects are
+    sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly
+    when _offset(o~1~) &lt; offset(o~2~)_).
+
+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of
+objects in packs stored by the MIDX, laid out in pack order, and the
+packs arranged in MIDX order (with the preferred pack coming first).
+
+Finally, note that the MIDX's reverse index is not stored as a chunk in
+the multi-pack-index itself. This is done because the reverse index
+includes the checksum of the pack or MIDX to which it belongs, which
+makes it impossible to write in the MIDX. To avoid races when rewriting
+the MIDX, a MIDX reverse index includes the MIDX's checksum in its
+filename (e.g., `multi-pack-index-xyz.rev`).
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#123 lore raw

Re: [PATCH v3 12/16] Documentation/technical: describe multi-pack reverse indexes

Quoted 2 lines · Taylor Blau

As a prerequisite to implementing multi-pack bitmaps, motivate and describe the format and ordering of the multi-pack reverse index.

Nicely written overall. I found a few typos / formatting issues.

Documentation/technical/pack-format.txt CHUNK DATA: · line 401

Quoted 2 lines
Documentation/technical/pack-format.txt +2 -0
+One solution is to let bits occupy the same position in the oid-sorted
+index stored by the MIDX. But because oids are effectively random, there

s/there/their/

Documentation/technical/pack-format.txt CHUNK DATA: · line 420

Quoted 2 lines
Documentation/technical/pack-format.txt +2 -0
+Given the list of packs and their counts of objects, you can
+na&iuml;vely reconstruct that pseudo-pack ordering (e.g., the object at

An HTML entity seems to have snuck in. The source is utf8, so we can just say ï.

Documentation/technical/pack-format.txt CHUNK DATA: · line 433

Quoted 13 lines
Documentation/technical/pack-format.txt +13 -0
+position 27 must be (c,1) because packs "a" and "b" consumed 25 of the
+slots). But there's a catch. Objects may be duplicated between packs, in
+which case the MIDX only stores one pointer to the object (and thus we'd
+want only one slot in the bitmap).
+
+Callers could handle duplicates themselves by reading objects in order
+of their bit-position, but that's linear in the number of objects, and
+much too expensive for ordinary bitmap lookups. Building a reverse index
+solves this, since it is the logical inverse of the index, and that
+index has already removed duplicates. But, building a reverse index on
+the fly can be expensive. Since we already have an on-disk format for
+pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,
+too.

Yep, I think this nicely builds up the logic explaining the need for the midx .rev file.

Documentation/technical/pack-format.txt CHUNK DATA: · line 439

Quoted 5 lines
Documentation/technical/pack-format.txt +5 -0
+Objects from the MIDX are ordered as follows to string together the
+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected
+by the MIDX, and define an ordering of packs based on their numeric ID
+(as stored by the MIDX). Let _offset(o)_ return the object offset of _o_
+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:

I guess the asciidoc-formatted version of this makes these nicely italicized and subscripted. Personally I think pack(o) and o1 would be more readable in the source (which is what I would tend to read). Or maybe backticks if you want to be fancy.

Documentation/technical/pack-format.txt CHUNK DATA: · line 453

Quoted 6 lines
Documentation/technical/pack-format.txt +6 -0
+  - If _pack(o~1~) &ne; pack(o~2~)_, then sort the two objects in
+    descending order based on the pack ID.
+
+  - Otherwise, _pack(o~1~) &equals; pack(o~2~)_, and the objects are
+    sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly
+    when _offset(o~1~) &lt; offset(o~2~)_).

A few more HTML bits in the comparison operators.

-Peff

#124 lore raw

Re: [PATCH v3 12/16] Documentation/technical: describe multi-pack reverse indexes

Quoted 6 lines · Jeff King
Quoted 2 lines · Taylor Blau

As a prerequisite to implementing multi-pack bitmaps, motivate and describe the format and ordering of the multi-pack reverse index.

Nicely written overall. I found a few typos / formatting issues.

Thanks for the attention to detail. Everything you wrote makes sense to me (including a quite-embarrassing mistake to switch "their" with "there").

Thanks,
Taylor

#125 lore raw

PATCH v3 13/16 pack-revindex: read multi-pack reverse indexes

In reply to Taylor Blau

Implement reading for multi-pack reverse indexes, as described in the previous patch.

Note that these functions don't yet have any callers, and won't until multi-pack reachability bitmaps are introduced in a later patch series. In the meantime, this patch implements some of the infrastructure necessary to support multi-pack bitmaps.

There are three new functions exposed by the revindex API:

- load_midx_revindex(): loads the reverse index corresponding to the given multi-pack index.

- midx_to_pack_pos() and pack_pos_to_midx(): these convert between the multi-pack index and pseudo-pack order.

load_midx_revindex() and pack_pos_to_midx() are both relatively straightforward.

load_midx_revindex() needs a few functions to be exposed from the midx API. One to get the checksum of a midx, and another to get the .rev's filename. Similar to recent changes in the packed_git struct, three new fields are added to the multi_pack_index struct: one to keep track of the size, one to keep track of the mmap'd pointer, and another to point past the header and at the reverse index's data.

pack_pos_to_midx() simply reads the corresponding entry out of the table.

midx_to_pack_pos() is the trickiest, since it needs to find an object's position in the psuedo-pack order, but that order can only be recovered in the .rev file itself. This mapping can be implemented with a binary search, but note that the thing we're binary searching over isn't an array of values, but rather a permuted order of those values.

So, when comparing two items, it's helpful to keep in mind the difference. Instead of a traditional binary search, where you are comparing two things directly, here we're comparing a (pack, offset) tuple with an index into the multi-pack index. That index describes another (pack, offset) tuple, and it is _those_ two tuples that are compared.

---
 midx.c          |  11 +++++
 midx.h          |   6 +++
 pack-revindex.c | 127 ++++++++++++++++++++++++++++++++++++++++++++++++
 pack-revindex.h |  53 ++++++++++++++++++++
 packfile.c      |   3 ++
 5 files changed, 200 insertions(+)
midx.c +11 -0
index 0a5da49ed6..55f4567fca 100644
@@ -47,11 +47,22 @@ static uint8_t oid_version(void)
 	}
 }
 
+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)
+{
+	return m->data + m->data_len - the_hash_algo->rawsz;
+}
+
 static char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }
 
+char *get_midx_rev_filename(struct multi_pack_index *m)
+{
+	return xstrfmt("%s/pack/multi-pack-index-%s.rev",
+		       m->object_dir, hash_to_hex(get_midx_checksum(m)));
+}
+
 static int midx_read_oid_fanout(const unsigned char *chunk_start,
 				size_t chunk_size, void *data)
 {
midx.h +6 -0
index 93bd68189e..0a8294d2ee 100644
@@ -15,6 +15,10 @@ struct multi_pack_index {
 	const unsigned char *data;
 	size_t data_len;
 
+	const uint32_t *revindex_data;
+	const uint32_t *revindex_map;
+	size_t revindex_len;
+
 	uint32_t signature;
 	unsigned char version;
 	unsigned char hash_len;
@@ -37,6 +41,8 @@ struct multi_pack_index {
 
 #define MIDX_PROGRESS     (1 << 0)
 
+char *get_midx_rev_filename(struct multi_pack_index *m);
+
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
pack-revindex.c +127 -0

142 lines not on this page · raw message

pack-revindex.h +53 -0

75 lines not on this page · raw message

packfile.c +3 -0
index 1fec12ac5f..82623e0cb4 100644
@@ -862,6 +862,9 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 
 	if (!strcmp(file_name, "multi-pack-index"))
 		return;
+	if (starts_with(file_name, "multi-pack-index") &&
+	    ends_with(file_name, ".rev"))
+		return;
 	if (ends_with(file_name, ".idx") ||
 	    ends_with(file_name, ".rev") ||
 	    ends_with(file_name, ".pack") ||
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#126 lore raw

Re: [PATCH v3 13/16] pack-revindex: read multi-pack reverse indexes

Quoted 2 lines · Taylor Blau

Implement reading for multi-pack reverse indexes, as described in the previous patch.

Looks good overall. I found a few tiny nits below.

Quoted 5 lines
pack-revindex.c +5 -0
+int load_midx_revindex(struct multi_pack_index *m)
+{
+	char *revindex_name;
+	int ret;
+	if (m->revindex_data)

pack-revindex.c int load_pack_revindex(struct packed_git *p) · line 310

Quoted 10 lines
pack-revindex.c +10 -0
+		return 0;
+
+	revindex_name = get_midx_rev_filename(m);
+
+	ret = load_revindex_from_disk(revindex_name,
+				      m->num_objects,
+				      &m->revindex_map,
+				      &m->revindex_len);
+	if (ret)
+		goto cleanup;

On error, I wondered if m->revindex_map, etc, would be modified. But it looks like no, load_revindex_from_disk() is careful not to touch them unless it sees a valid revindex. Good.

Quoted 4 lines
pack-revindex.c +4 -0
+int close_midx_revindex(struct multi_pack_index *m)
+{
+	if (!m)
+		return 0;

pack-revindex.c int load_pack_revindex(struct packed_git *p) · line 332

Quoted 10 lines
pack-revindex.c +10 -0
+
+	if (munmap((void*)m->revindex_map, m->revindex_len))
+		return -1;
+
+	m->revindex_map = NULL;
+	m->revindex_data = NULL;
+	m->revindex_len = 0;
+
+	return 0;
+}

It's hard to imagine why munmap() would fail. But if it does, we should probably clear the struct fields anyway. I note that the matching code for a "struct packed_git" does not bother even checking the return value of munmap. Perhaps we should just do the same here.

The packed_git version also returned early if revindex_map is NULL. Here the burden is placed on the caller (it's hard to tell if that matters since there aren't any callers yet, but it probably makes sense to push the check down into this function).

pack-revindex.c off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos) · line 396

Quoted 8 lines
pack-revindex.c +8 -0
+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)
+{
+	if (!m->revindex_data)
+		BUG("pack_pos_to_midx: reverse index not yet loaded");
+	if (m->num_objects <= pos)
+		BUG("pack_pos_to_midx: out-of-bounds object at %"PRIu32, pos);
+	return get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));
+}

OK, this one is just a direct read of the .rev data, like pack_pos_to_index() is. I think the final line can be simplified to:

  return get_be32(m->revindex_data + pos);

just like pack_pos_to_index(). (I suspect this is a leftover from the earlier version of your .rev series where the pointer was still a "void *").

Quoted 16 lines
pack-revindex.c +16 -0
+int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)
+{
+	struct midx_pack_key key;
+	uint32_t *found;
+
+	if (!m->revindex_data)
+		BUG("midx_to_pack_pos: reverse index not yet loaded");
+	if (m->num_objects <= at)
+		BUG("midx_to_pack_pos: out-of-bounds object at %"PRIu32, at);
+
+	key.pack = nth_midxed_pack_int_id(m, at);
+	key.offset = nth_midxed_offset(m, at);
+	key.midx = m;
+	/*
+	 * The preferred pack sorts first, so determine its identifier by
+	 * looking at the first object in pseudo-pack order.

pack-revindex.c off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos) · line 468

Quoted 10 lines
pack-revindex.c +10 -0
+	 *
+	 * Note that if no --preferred-pack is explicitly given when writing a
+	 * multi-pack index, then whichever pack has the lowest identifier
+	 * implicitly is preferred (and includes all its objects, since ties are
+	 * broken first by pack identifier).
+	 */
+	key.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));
+
+	found = bsearch(&key, m->revindex_data, m->num_objects,
+			sizeof(uint32_t), midx_pack_order_cmp);

OK, this one is _roughly_ equivalent to offset_to_pack_pos(), in that we have to binary search within the pack-ordered list to find the entry. Makes sense.

Probably sizeof(*m->revindex_data) would be slightly nicer in the bsearch call (again, I suspect a holdover from when that was a void pointer).

-Peff

#127 lore raw

Re: [PATCH v3 13/16] pack-revindex: read multi-pack reverse indexes

Quoted 18 lines · Jeff King
Quoted 2 lines · Taylor Blau

Implement reading for multi-pack reverse indexes, as described in the previous patch.

Looks good overall. I found a few tiny nits below.

Quoted 11 lines
pack-revindex.c +11 -0
+int load_midx_revindex(struct multi_pack_index *m)
+{
+	char *revindex_name;
+	int ret;
+	if (m->revindex_data)
+		return 0;
+
+	revindex_name = get_midx_rev_filename(m);
+
+	ret = load_revindex_from_disk(revindex_name,
+				      m->num_objects,
Quoted 8 lines

pack-revindex.c int load_pack_revindex(struct packed_git *p) · line 310

Quoted 4 lines
pack-revindex.c +4 -0
+				      &m->revindex_map,
+				      &m->revindex_len);
+	if (ret)
+		goto cleanup;

On error, I wondered if m->revindex_map, etc, would be modified. But it looks like no, load_revindex_from_disk() is careful not to touch them unless it sees a valid revindex. Good.

Yep, that was intentional. Thanks.

Quoted 11 lines
Quoted 11 lines
pack-revindex.c +11 -0
+int close_midx_revindex(struct multi_pack_index *m)
+{
+	if (!m)
+		return 0;
+
+	if (munmap((void*)m->revindex_map, m->revindex_len))
+		return -1;
+
+	m->revindex_map = NULL;
+	m->revindex_data = NULL;
+	m->revindex_len = 0;
Quoted 8 lines

pack-revindex.c int load_pack_revindex(struct packed_git *p) · line 332

Quoted 3 lines
pack-revindex.c +3 -0
+
+	return 0;
+}

It's hard to imagine why munmap() would fail. But if it does, we should probably clear the struct fields anyway. I note that the matching code for a "struct packed_git" does not bother even checking the return value of munmap. Perhaps we should just do the same here.

I tend to agree that we should match the behavior of "packfile.c:close_pack_revindex()" and just not check the return value of munmap. Either the call to munmap() worked, and we shouldn't be reading revindex_map anymore, or it didn't, and something else is probably wrong enough with the original mmap call that we probably also shouldn't be reading it.

Quoted 4 lines

The packed_git version also returned early if revindex_map is NULL. Here the burden is placed on the caller (it's hard to tell if that matters since there aren't any callers yet, but it probably makes sense to push the check down into this function).

Yeah, I think that that function actually is doing the worst of both worlds (which is to check p->revindex_map, but not p itself).

I modified the MIDX version to check both m and m->revindex_map (but I agree it's hard to tell with the caller coming in a later series).

Quoted 13 lines

pack-revindex.c off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos) · line 396

Quoted 9 lines
pack-revindex.c +8 -0
+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)
+{
+	if (!m->revindex_data)
+		BUG("pack_pos_to_midx: reverse index not yet loaded");
+	if (m->num_objects <= pos)
+		BUG("pack_pos_to_midx: out-of-bounds object at %"PRIu32, pos);
+	return get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));
+}

OK, this one is just a direct read of the .rev data, like pack_pos_to_index() is. I think the final line can be simplified to:

Quoted 5 lines
  return get_be32(m->revindex_data + pos);

just like pack_pos_to_index(). (I suspect this is a leftover from the earlier version of your .rev series where the pointer was still a "void *").

Yes, definitely.

Quoted 3 lines

Probably sizeof(*m->revindex_data) would be slightly nicer in the bsearch call (again, I suspect a holdover from when that was a void pointer).

Yes, exactly.

Thanks,
Taylor

#128 lore raw

PATCH v3 14/16 pack-write.c: extract 'write_rev_file_order'

In reply to Taylor Blau

Existing callers provide the reverse index code with an array of 'struct pack_idx_entry *'s, which is then sorted by pack order (comparing the offsets of each object within the pack).

Prepare for the multi-pack index to write a .rev file by providing a way to write the reverse index without an array of pack_idx_entry (which the MIDX code does not have).

Instead, callers can invoke 'write_rev_index_positions()', which takes an array of uint32_t's. The ith entry in this array specifies the ith object's (in index order) position within the pack (in pack order).

Expose this new function for use in a later patch, and rewrite the existing write_rev_file() in terms of this new function.

---
 pack-write.c | 36 +++++++++++++++++++++++++-----------
 pack.h       |  1 +
 2 files changed, 26 insertions(+), 11 deletions(-)
pack-write.c +25 -11
index 2ca85a9d16..f1fc3ecafa 100644
@@ -201,21 +201,12 @@ static void write_rev_header(struct hashfile *f)
 }
 
 static void write_rev_index_positions(struct hashfile *f,
-				      struct pack_idx_entry **objects,
+				      uint32_t *pack_order,
 				      uint32_t nr_objects)
 {
-	uint32_t *pack_order;
 	uint32_t i;
-
-	ALLOC_ARRAY(pack_order, nr_objects);
-	for (i = 0; i < nr_objects; i++)
-		pack_order[i] = i;
-	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
-
 	for (i = 0; i < nr_objects; i++)
 		hashwrite_be32(f, pack_order[i]);
-
-	free(pack_order);
 }
 
 static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)
@@ -228,6 +219,29 @@ const char *write_rev_file(const char *rev_name,
 			   uint32_t nr_objects,
 			   const unsigned char *hash,
 			   unsigned flags)
+{
+	uint32_t *pack_order;
+	uint32_t i;
+	const char *ret;
+
+	ALLOC_ARRAY(pack_order, nr_objects);
+	for (i = 0; i < nr_objects; i++)
+		pack_order[i] = i;
+	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
+
+	ret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,
+				   flags);
+
+	free(pack_order);
+
+	return ret;
+}
+
+const char *write_rev_file_order(const char *rev_name,
+				 uint32_t *pack_order,
+				 uint32_t nr_objects,
+				 const unsigned char *hash,
+				 unsigned flags)
 {
 	struct hashfile *f;
 	int fd;
@@ -262,7 +276,7 @@ const char *write_rev_file(const char *rev_name,
 
 	write_rev_header(f);
 
-	write_rev_index_positions(f, objects, nr_objects);
+	write_rev_index_positions(f, pack_order, nr_objects);
 	write_rev_trailer(f, hash);
 
 	if (rev_name && adjust_shared_perm(rev_name) < 0)
pack.h +1 -0
index 857cbd5bd4..fa13954526 100644
@@ -94,6 +94,7 @@ struct ref;
 void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);
 
 const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);
+const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order, uint32_t nr_objects, const unsigned char *hash, unsigned flags);
 
 /*
  * The "hdr" output buffer should be at least this big, which will handle sizes
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#129 lore raw

PATCH v3 15/16 pack-revindex: write multi-pack reverse indexes

In reply to Taylor Blau

Implement the writing half of multi-pack reverse indexes. This is nothing more than the format describe a few patches ago, with a new set of helper functions that will be used to clear out stale .rev files corresponding to old MIDXs.

Unfortunately, a very similar comparison function as the one implemented recently in pack-revindex.c is reimplemented here, this time accepting a MIDX-internal type. An effort to DRY these up would create more indirection and overhead than is necessary, so it isn't pursued here.

Currently, there are no callers which pass the MIDX_WRITE_REV_INDEX flag, meaning that this is all dead code. But, that won't be the case for long, since subsequent patches will introduce the multi-pack bitmap, which will begin passing this field.

(In midx.c:write_midx_internal(), the two adjacent if statements share a conditional, but are written separately since the first one will eventually also handle the MIDX_WRITE_BITMAP flag, which does not yet exist.)

---
 midx.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 midx.h |   1 +
 2 files changed, 116 insertions(+)
midx.c +115 -0
index 55f4567fca..eea9574d92 100644
@@ -12,6 +12,7 @@
 #include "run-command.h"
 #include "repository.h"
 #include "chunk-format.h"
+#include "pack.h"
 
 #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
 #define MIDX_VERSION 1
@@ -472,6 +473,7 @@ struct write_midx_context {
 	uint32_t entries_nr;
 
 	uint32_t *pack_perm;
+	uint32_t *pack_order;
 	unsigned large_offsets_needed:1;
 	uint32_t num_large_offsets;
 
@@ -826,6 +828,70 @@ static int write_midx_large_offsets(struct hashfile *f,
 	return 0;
 }
 
+static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)
+{
+	struct write_midx_context *ctx = _ctx;
+
+	struct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];
+	struct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];
+
+	uint32_t perm_a = ctx->pack_perm[a->pack_int_id];
+	uint32_t perm_b = ctx->pack_perm[b->pack_int_id];
+
+	/* Sort objects in the preferred pack ahead of any others. */
+	if (a->preferred > b->preferred)
+		return -1;
+	if (a->preferred < b->preferred)
+		return 1;
+
+	/* Then, order objects by which packs they appear in. */
+	if (perm_a < perm_b)
+		return -1;
+	if (perm_a > perm_b)
+		return 1;
+
+	/* Then, disambiguate by their offset within each pack. */
+	if (a->offset < b->offset)
+		return -1;
+	if (a->offset > b->offset)
+		return 1;
+
+	return 0;
+}
+
+static uint32_t *midx_pack_order(struct write_midx_context *ctx)
+{
+	uint32_t *pack_order;
+	uint32_t i;
+
+	ALLOC_ARRAY(pack_order, ctx->entries_nr);
+	for (i = 0; i < ctx->entries_nr; i++)
+		pack_order[i] = i;
+	QSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);
+
+	return pack_order;
+}
+
+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,
+				     struct write_midx_context *ctx)
+{
+	struct strbuf buf = STRBUF_INIT;
+	const char *tmp_file;
+
+	strbuf_addf(&buf, "%s-%s.rev", midx_name, hash_to_hex(midx_hash));
+
+	tmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,
+					midx_hash, WRITE_REV);
+
+	if (finalize_object_file(tmp_file, buf.buf))
+		die(_("cannot store reverse index file"));
+
+	strbuf_release(&buf);
+}
+
+static void clear_midx_files_ext(struct repository *r, const char *ext,
+				 unsigned char *keep_hash);
+
 static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
 			       struct string_list *packs_to_drop,
 			       const char *preferred_pack_name,
@@ -1011,6 +1077,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		ctx.pack_order = midx_pack_order(&ctx);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		write_midx_reverse_index(midx_name, midx_hash, &ctx);
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+
 	commit_lock_file(&lk);
 
 cleanup:
@@ -1025,6 +1099,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	free(ctx.info);
 	free(ctx.entries);
 	free(ctx.pack_perm);
+	free(ctx.pack_order);
 	free(midx_name);
 	return result;
 }
@@ -1037,6 +1112,44 @@ int write_midx_file(const char *object_dir,
 				   flags);
 }
 
+struct clear_midx_data {
+	char *keep;
+	const char *ext;
+};
+
+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,
+				const char *file_name, void *_data)
+{
+	struct clear_midx_data *data = _data;
+
+	if (!(starts_with(file_name, "multi-pack-index-") &&
+	      ends_with(file_name, data->ext)))
+		return;
+	if (data->keep && !strcmp(data->keep, file_name))
+		return;
+
+	if (unlink(full_path))
+		die_errno(_("failed to remove %s"), full_path);
+}
+
+static void clear_midx_files_ext(struct repository *r, const char *ext,
+				 unsigned char *keep_hash)
+{
+	struct clear_midx_data data;
+	memset(&data, 0, sizeof(struct clear_midx_data));
+
+	if (keep_hash)
+		data.keep = xstrfmt("multi-pack-index-%s%s",
+				    hash_to_hex(keep_hash), ext);
+	data.ext = ext;
+
+	for_each_file_in_pack_dir(r->objects->odb->path,
+				  clear_midx_file_ext,
+				  &data);
+
+	free(data.keep);
+}
+
 void clear_midx_file(struct repository *r)
 {
 	char *midx = get_midx_filename(r->objects->odb->path);
@@ -1049,6 +1162,8 @@ void clear_midx_file(struct repository *r)
 	if (remove_path(midx))
 		die(_("failed to clear multi-pack-index at %s"), midx);
 
+	clear_midx_files_ext(r, ".rev", NULL);
+
 	free(midx);
 }
 
midx.h +1 -0
index 0a8294d2ee..8684cf0fef 100644
@@ -40,6 +40,7 @@ struct multi_pack_index {
 };
 
 #define MIDX_PROGRESS     (1 << 0)
+#define MIDX_WRITE_REV_INDEX (1 << 1)
 
 char *get_midx_rev_filename(struct multi_pack_index *m);
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#130 lore raw

Re: [PATCH v3 15/16] pack-revindex: write multi-pack reverse indexes

Quoted 4 lines · Taylor Blau

Implement the writing half of multi-pack reverse indexes. This is nothing more than the format describe a few patches ago, with a new set of helper functions that will be used to clear out stale .rev files corresponding to old MIDXs.

Looks good.

midx.c int write_midx_file(const char *object_dir, · line 1121

Quoted 7 lines
midx.c +7 -0
+struct clear_midx_data {
+	char *keep;
+	const char *ext;
+};
+
+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,
+				const char *file_name, void *_data)

This will clean up _any_ stale midx .rev file. So even if we miss one when writing a new midx (due to a bug, race, power loss, etc), we'll catch it later.

We _might_ want to also teach various tempfile-cleanup code run by gc to likewise look for unattached midx .rev files, but I don't think we necessarily have to do it now.

midx.c void clear_midx_file(struct repository *r) · line 1167

Quoted 10 lines

void clear_midx_file(struct repository *r)
{

midx.c +2 -0
 	char *midx = get_midx_filename(r->objects->odb->path);
@@ -1049,6 +1162,8 @@ void clear_midx_file(struct repository *r)
 	if (remove_path(midx))
 		die(_("failed to clear multi-pack-index at %s"), midx);
 
+	clear_midx_files_ext(r, ".rev", NULL);
+
 	free(midx);

The sole caller now doesn't pass the "keep" hash, so we'd always delete all of them. I guess we'll see that change once somebody starts actually writing them.

-Peff

#131 lore raw

Re: [PATCH v3 15/16] pack-revindex: write multi-pack reverse indexes

Quoted 21 lines · Jeff King
Quoted 4 lines · Taylor Blau

Implement the writing half of multi-pack reverse indexes. This is nothing more than the format describe a few patches ago, with a new set of helper functions that will be used to clear out stale .rev files corresponding to old MIDXs.

Looks good.

midx.c int write_midx_file(const char *object_dir, · line 1121

Quoted 7 lines
midx.c +7 -0
+struct clear_midx_data {
+	char *keep;
+	const char *ext;
+};
+
+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,
+				const char *file_name, void *_data)

This will clean up _any_ stale midx .rev file. So even if we miss one when writing a new midx (due to a bug, race, power loss, etc), we'll catch it later.

Quoted 3 lines

We _might_ want to also teach various tempfile-cleanup code run by gc to likewise look for unattached midx .rev files, but I don't think we necessarily have to do it now.

Agreed there on both counts.

Quoted 7 lines
Quoted 7 lines

void clear_midx_file(struct repository *r)
{

midx.c +0 -0
 	char *midx = get_midx_filename(r->objects->odb->path);
@@ -1049,6 +1162,8 @@ void clear_midx_file(struct repository *r)
 	if (remove_path(midx))
 		die(_("failed to clear multi-pack-index at %s"), midx);
 
Quoted 7 lines

midx.c void clear_midx_file(struct repository *r) · line 1167

Quoted 3 lines
midx.c +2 -0
+	clear_midx_files_ext(r, ".rev", NULL);
+
 	free(midx);

The sole caller now doesn't pass the "keep" hash, so we'd always delete all of them. I guess we'll see that change once somebody starts actually writing them.

That's right. I hope that the benefits of splitting the MIDX bitmaps topic into two series has generally outweighed the drawbacks, but in instances like these it can be kind of annoying.

Thanks,
Taylor

#132 lore raw

PATCH v3 16/16 midx.c: improve cache locality in midx_pack_order_cmp()

In reply to Taylor Blau

From: Jeff King <[email protected]>

There is a lot of pointer dereferencing in the pre-image version of 'midx_pack_order_cmp()', which this patch gets rid of.

Instead of comparing the pack preferred-ness and then the pack id, both of these checks are done at the same time by using the high-order bit of the pack id to represent whether it's preferred. Then the pack id and offset are compared as usual.

This produces the same result so long as there are less than 2^31 packs, which seems like a likely assumption to make in practice.

---
 midx.c | 55 +++++++++++++++++++++++++++++--------------------------
 1 file changed, 29 insertions(+), 26 deletions(-)
midx.c +29 -26
index eea9574d92..4835cc13d1 100644
@@ -828,46 +828,49 @@ static int write_midx_large_offsets(struct hashfile *f,
 	return 0;
 }
 
-static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)
+struct midx_pack_order_data {
+	uint32_t nr;
+	uint32_t pack;
+	off_t offset;
+};
+
+static int midx_pack_order_cmp(const void *va, const void *vb)
 {
-	struct write_midx_context *ctx = _ctx;
-
-	struct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];
-	struct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];
-
-	uint32_t perm_a = ctx->pack_perm[a->pack_int_id];
-	uint32_t perm_b = ctx->pack_perm[b->pack_int_id];
-
-	/* Sort objects in the preferred pack ahead of any others. */
-	if (a->preferred > b->preferred)
+	const struct midx_pack_order_data *a = va, *b = vb;
+	if (a->pack < b->pack)
 		return -1;
-	if (a->preferred < b->preferred)
+	else if (a->pack > b->pack)
 		return 1;
-
-	/* Then, order objects by which packs they appear in. */
-	if (perm_a < perm_b)
+	else if (a->offset < b->offset)
 		return -1;
-	if (perm_a > perm_b)
+	else if (a->offset > b->offset)
 		return 1;
-
-	/* Then, disambiguate by their offset within each pack. */
-	if (a->offset < b->offset)
-		return -1;
-	if (a->offset > b->offset)
-		return 1;
-
-	return 0;
+	else
+		return 0;
 }
 
 static uint32_t *midx_pack_order(struct write_midx_context *ctx)
 {
+	struct midx_pack_order_data *data;
 	uint32_t *pack_order;
 	uint32_t i;
 
+	ALLOC_ARRAY(data, ctx->entries_nr);
+	for (i = 0; i < ctx->entries_nr; i++) {
+		struct pack_midx_entry *e = &ctx->entries[i];
+		data[i].nr = i;
+		data[i].pack = ctx->pack_perm[e->pack_int_id];
+		if (!e->preferred)
+			data[i].pack |= (1U << 31);
+		data[i].offset = e->offset;
+	}
+
+	QSORT(data, ctx->entries_nr, midx_pack_order_cmp);
+
 	ALLOC_ARRAY(pack_order, ctx->entries_nr);
 	for (i = 0; i < ctx->entries_nr; i++)
-		pack_order[i] = i;
-	QSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);
+		pack_order[i] = data[i].nr;
+	free(data);
 
 	return pack_order;
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#133 lore raw

Re: [PATCH v3 16/16] midx.c: improve cache locality in midx_pack_order_cmp()

Quoted 5 lines · Taylor Blau

From: Jeff King <[email protected]>

There is a lot of pointer dereferencing in the pre-image version of 'midx_pack_order_cmp()', which this patch gets rid of.

Quoted 7 lines

Instead of comparing the pack preferred-ness and then the pack id, both of these checks are done at the same time by using the high-order bit of the pack id to represent whether it's preferred. Then the pack id and offset are compared as usual.

This produces the same result so long as there are less than 2^31 packs, which seems like a likely assumption to make in practice.

Obviously this patch is brilliant. ;)

Did we record any numbers to show the improvement here? I don't think it can be demonstrated with this series (since most of the code is dead), but I recall that this was motivated by a noticeable slowdown.

I briefly wondered whether the complicated midx_pack_order_cmp() in pack-revindex.c, which is used for the bsearch() there, could benefit from the same speedup. It's only log(n), of course, instead of n*log(n), but one might imagine making "n" calls to it. I don't think it makes sense, though. The pointer dereferencing there is into the midx mmap itself. Creating an auxiliary array would defeat the purpose.

-Peff

#134 lore raw

Re: [PATCH v3 16/16] midx.c: improve cache locality in midx_pack_order_cmp()

Quoted 12 lines · Jeff King
Quoted 10 lines · Taylor Blau

From: Jeff King <[email protected]>

There is a lot of pointer dereferencing in the pre-image version of 'midx_pack_order_cmp()', which this patch gets rid of.

Instead of comparing the pack preferred-ness and then the pack id, both of these checks are done at the same time by using the high-order bit of the pack id to represent whether it's preferred. Then the pack id and offset are compared as usual.

Quoted 4 lines
Quoted 2 lines

This produces the same result so long as there are less than 2^31 packs, which seems like a likely assumption to make in practice.

Obviously this patch is brilliant. ;)

Obviously.

Quoted 3 lines

Did we record any numbers to show the improvement here? I don't think it can be demonstrated with this series (since most of the code is dead), but I recall that this was motivated by a noticeable slowdown.

Looking through our messages, you wrote that this seemed to produce a .8 second speed-up on a large-ish repository that we were testing. That's not significant overall, the fact that we were spending so long probably caught our attention when looking at a profiler.

I could go either way on mentioning it. It does feel a little like cheating to say, "well, if you applied these other patches it would make it about this much faster". So I'm mostly happy to just keep it vague and say that it makes things a little faster, unless you feel strongly otherwise.

Quoted 6 lines

I briefly wondered whether the complicated midx_pack_order_cmp() in pack-revindex.c, which is used for the bsearch() there, could benefit from the same speedup. It's only log(n), of course, instead of n*log(n), but one might imagine making "n" calls to it. I don't think it makes sense, though. The pointer dereferencing there is into the midx mmap itself. Creating an auxiliary array would defeat the purpose.

Right.

Quoted 1 line

-Peff

Thanks,
Taylor

#135 lore raw

Re: [PATCH v3 16/16] midx.c: improve cache locality in midx_pack_order_cmp()

Quoted 8 lines · Taylor Blau
Quoted 3 lines

Did we record any numbers to show the improvement here? I don't think it can be demonstrated with this series (since most of the code is dead), but I recall that this was motivated by a noticeable slowdown.

Looking through our messages, you wrote that this seemed to produce a .8 second speed-up on a large-ish repository that we were testing. That's not significant overall, the fact that we were spending so long probably caught our attention when looking at a profiler.

That sounds about right from my recollection.

Quoted 5 lines

I could go either way on mentioning it. It does feel a little like cheating to say, "well, if you applied these other patches it would make it about this much faster". So I'm mostly happy to just keep it vague and say that it makes things a little faster, unless you feel strongly otherwise.

No, I don't feel strongly. I just wanted to give people reading a sense of what to expect. Now we have.

-Peff

#136 lore raw
In reply to Taylor Blau
Quoted 17 lines · Taylor Blau

Here is another reroll of my series to implement a reverse index in preparation for multi-pack reachability bitmaps. The previous version was based on 'ds/chunked-file-api', but that topic has since been merged to 'master'. This series is now built directly on top of 'master'.

Not much has changed since last time. Jonathan Tan reviewed the previous version, and I incorporated feedback from his review:

- The usage macros in builtin/multi-pack-index.c were pulled out and defined separately.
- Some sloppiness with converting a signed index referring to the preferred pack into an unsigned value was cleaned up.
- Documentation clean-up, particularly in patches 12 and 13.

There are a couple of new things that we found while testing this out at GitHub.

Quoted 8 lines

- We now call finalize_object_file() on the multi-pack reverse index to set the correct permissions.
- Patch 14 removed a stray hunk that introduced a memory leak.
- Patch 16 (courtesy of Peff) is new. It improves the cache locality of midx_pack_order_cmp(), which has a substantial impact on repositories with many objects.

Thanks in advance for your review.

I've reviewed the changes since my last review and this one looks good, including that new patch from Peff.

Thanks,
-Stolee

#137 lore raw
In reply to Taylor Blau
Quoted 4 lines · Taylor Blau

Here is another reroll of my series to implement a reverse index in preparation for multi-pack reachability bitmaps. The previous version was based on 'ds/chunked-file-api', but that topic has since been merged to 'master'. This series is now built directly on top of 'master'.

I gave the whole thing another careful read. Most of what I found were small nits, but enough that I think one more re-roll is worth it.

The biggest question is what we want to happen next. As you note, the concept of a midx .rev file is useless until we have the matching .bitmap file. So we _could_ let this sit in next while the dependent bitmap topic is reviewed, and then merge them down together. But I'm inclined to treat this as an independent topic that can get merged to master on its own, since the early cleanups are valuable on their own, and the .rev parts at the end, even if dead, won't hurt anything.

If we did want to break it up, the useful line would be after "allow marking a pack as preferred" (while it is mostly intended for the bitmap selection, it is theoretically useful on its own to make it more likely to find a copy of an object with a useful delta).

-Peff

#138 lore raw
Quoted 9 lines · Jeff King
Quoted 4 lines · Taylor Blau

Here is another reroll of my series to implement a reverse index in preparation for multi-pack reachability bitmaps. The previous version was based on 'ds/chunked-file-api', but that topic has since been merged to 'master'. This series is now built directly on top of 'master'.

I gave the whole thing another careful read. Most of what I found were small nits, but enough that I think one more re-roll is worth it.

Thanks.

Quoted 7 lines

The biggest question is what we want to happen next. As you note, the concept of a midx .rev file is useless until we have the matching .bitmap file. So we _could_ let this sit in next while the dependent bitmap topic is reviewed, and then merge them down together. But I'm inclined to treat this as an independent topic that can get merged to master on its own, since the early cleanups are valuable on their own, and the .rev parts at the end, even if dead, won't hurt anything.

It was my impression as well that the early clean-ups are worth on their own.

#139 lore raw
In reply to Jeff King
Quoted 9 lines · Jeff King
Quoted 4 lines · Taylor Blau

Here is another reroll of my series to implement a reverse index in preparation for multi-pack reachability bitmaps. The previous version was based on 'ds/chunked-file-api', but that topic has since been merged to 'master'. This series is now built directly on top of 'master'.

I gave the whole thing another careful read. Most of what I found were small nits, but enough that I think one more re-roll is worth it.

Thanks. I agree that another re-roll is worth it. I have one prepared locally, and I just had one outstanding question in:

https://lore.kernel.org/git/[email protected]/

that I'll wait on your reply to before sending a reroll.

Quoted 7 lines

The biggest question is what we want to happen next. As you note, the concept of a midx .rev file is useless until we have the matching .bitmap file. So we _could_ let this sit in next while the dependent bitmap topic is reviewed, and then merge them down together. But I'm inclined to treat this as an independent topic that can get merged to master on its own, since the early cleanups are valuable on their own, and the .rev parts at the end, even if dead, won't hurt anything.

That matches what I was hoping for. I think the clean-ups are worth it on their own, but I also think it's a good idea to take the whole series, since it means there's one less long-running branch in flight while we review the MIDX bitmaps topic.

(FWIW, I can also see an argument in the other direction along the lines of "we may discover something later on that requires us to change the way multi-pack .rev files work". I think that such an outcome is fairly unlikely, but worth considering anyway).

Thanks,
Taylor

#140 lore raw
Quoted 13 lines · Taylor Blau
Quoted 7 lines

The biggest question is what we want to happen next. As you note, the concept of a midx .rev file is useless until we have the matching .bitmap file. So we _could_ let this sit in next while the dependent bitmap topic is reviewed, and then merge them down together. But I'm inclined to treat this as an independent topic that can get merged to master on its own, since the early cleanups are valuable on their own, and the .rev parts at the end, even if dead, won't hurt anything.

That matches what I was hoping for. I think the clean-ups are worth it on their own, but I also think it's a good idea to take the whole series, since it means there's one less long-running branch in flight while we review the MIDX bitmaps topic.

Quoted 4 lines

(FWIW, I can also see an argument in the other direction along the lines of "we may discover something later on that requires us to change the way multi-pack .rev files work". I think that such an outcome is fairly unlikely, but worth considering anyway).

That would be my general worry, too, but in this case I am not too concerned because I know the code has received substantial exercise already on real-world production servers. So while we may clean up some cosmetic bits or respond to review as it goes upstream, I'm much less worried about seeing some brown-paper-bag bug that would be sufficient to make us want to re-roll these .rev commits. And hopefully the existing rounds have addressed the cosmetic/review bits.

-Peff

#141 lore raw
Quoted 6 lines · Jeff King
Quoted 4 lines · Taylor Blau

(FWIW, I can also see an argument in the other direction along the lines of "we may discover something later on that requires us to change the way multi-pack .rev files work". I think that such an outcome is fairly unlikely, but worth considering anyway).

Quoted 7 lines

That would be my general worry, too, but in this case I am not too concerned because I know the code has received substantial exercise already on real-world production servers. So while we may clean up some cosmetic bits or respond to review as it goes upstream, I'm much less worried about seeing some brown-paper-bag bug that would be sufficient to make us want to re-roll these .rev commits. And hopefully the existing rounds have addressed the cosmetic/review bits.

Yes. Another benefit is that it should give us substantial confidence in the correctness not just of this topic, but of the multi-pack bitmaps that are built on top, too.

Thanks,
Taylor

#142 lore raw

v4 cover midx: implement a multi-pack reverse index

In reply to Taylor Blau

Here is another reroll of my series to implement a reverse index in preparation for multi-pack reachability bitmaps.

This reroll differs only in the feedback I incorporated from Peff's review. They are mostly cosmetic; the most substantial change being that the --preferred-pack code now uses bsearch() to locate the name of the preferred pack (instead of implementing a binary search itself).

I think that this version is ready to go. I would hope that it can head for 'master' and avoid sitting in 'next' forever (since it has some worthwhile cleanups outside of preparing for MIDX bitmaps).

But in either case, this is the last prereq series before MIDX bitmaps, which I'll send shortly (based on this one).

Jeff King (1):
midx.c: improve cache locality in midx_pack_order_cmp()

Taylor Blau (15):
builtin/multi-pack-index.c: inline 'flags' with options builtin/multi-pack-index.c: don't handle 'progress' separately builtin/multi-pack-index.c: define common usage with a macro builtin/multi-pack-index.c: split sub-commands builtin/multi-pack-index.c: don't enter bogus cmd_mode builtin/multi-pack-index.c: display usage on unrecognized command t/helper/test-read-midx.c: add '--show-objects'
midx: allow marking a pack as preferred
midx: don't free midx_name early
midx: keep track of the checksum
midx: make some functions non-static Documentation/technical: describe multi-pack reverse indexes
pack-revindex: read multi-pack reverse indexes pack-write.c: extract 'write_rev_file_order'
pack-revindex: write multi-pack reverse indexes

 Documentation/git-multi-pack-index.txt       |  14 +-
 Documentation/technical/multi-pack-index.txt |   5 +-
 Documentation/technical/pack-format.txt      |  83 +++++++
 builtin/multi-pack-index.c                   | 182 ++++++++++++---
 builtin/repack.c                             |   2 +-
 midx.c                                       | 219 +++++++++++++++++--
 midx.h                                       |  11 +-
 pack-revindex.c                              | 126 +++++++++++
 pack-revindex.h                              |  53 +++++
 pack-write.c                                 |  36 ++-
 pack.h                                       |   1 +
 packfile.c                                   |   3 +
 t/helper/test-read-midx.c                    |  24 +-
 t/t5319-multi-pack-index.sh                  |  43 ++++
 14 files changed, 734 insertions(+), 68 deletions(-)

Range-diff against v3:
1: 43fc0ad276 ! 1: 90e021725f builtin/multi-pack-index.c: inline 'flags' with options
@@ Commit message
'verify', etc.) will want to optionally change a set of shared flags that are eventually passed to the MIDX libraries.

- Right now, options and flags are handled separately. Inline them into
- the same structure so that sub-commands can more easily share the
- 'flags' data.
+ Right now, options and flags are handled separately. That's fine, since
+ the options structure is never passed around. But a future patch will
+ make it so that common options shared by all sub-commands are defined in
+ a common location. That means that "flags" would have to become a global
+ variable.
+
+ Group it with the options structure so that we reduce the number of
+ global variables we have overall.

 2:  181f11e4c5 =  2:  130c191b80 builtin/multi-pack-index.c: don't handle 'progress' separately
 3:  94c498f0e2 =  3:  5a274b9096 builtin/multi-pack-index.c: define common usage with a macro
 4:  d084f90466 !  4:  b8c89cc239 builtin/multi-pack-index.c: split sub-commands
    @@ builtin/multi-pack-index.c: static struct opts_multi_pack_index {
     -		OPT_FILENAME(0, "object-dir", &opts.object_dir,
     -		  N_("object directory containing set of packfile and pack-index pairs")),
     -		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
    -+	struct option *with_common = parse_options_concat(common_opts, prev);
    -+	free(prev);
    -+	return with_common;
    ++	return parse_options_concat(common_opts, prev);
     +}
     +
     +static int cmd_multi_pack_index_write(int argc, const char **argv)
    @@ builtin/multi-pack-index.c: static struct opts_multi_pack_index {
      		OPT_END(),
      	};
      
    -+	options = parse_options_dup(builtin_multi_pack_index_repack_options);
    -+	options = add_common_options(options);
    ++	options = add_common_options(builtin_multi_pack_index_repack_options);
     +
     +	argc = parse_options(argc, argv, NULL,
     +			     options,
    @@ builtin/multi-pack-index.c: static struct opts_multi_pack_index {
     +		usage_with_options(builtin_multi_pack_index_repack_usage,
     +				   options);
     +
    ++	FREE_AND_NULL(options);
    ++
     +	return midx_repack(the_repository, opts.object_dir,
     +			   (size_t)opts.batch_size, opts.flags);
     +}

5: bc3b6837f2 ! 5: d817920e2a builtin/multi-pack-index.c: don't enter bogus cmd_mode
@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_expire(int argc, con
options, builtin_multi_pack_index_expire_usage, PARSE_OPT_KEEP_UNKNOWN);
@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_repack(int argc, const char **argv)
- options = parse_options_dup(builtin_multi_pack_index_repack_options);
- options = add_common_options(options);
+
+ options = add_common_options(builtin_multi_pack_index_repack_options);

+ trace2_cmd_mode(argv[0]);
+
6: f117e442c3 ! 6: 604a02ce85 builtin/multi-pack-index.c: display usage on unrecognized command

    @@ Commit message
         more helpful:
     
             $ git.compile multi-pack-index bogus
    +        error: unrecognized subcommand: bogus
             usage: git multi-pack-index [<options>] write
                or: git multi-pack-index [<options>] verify
                or: git multi-pack-index [<options>] expire
    @@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv
      	if (!strcmp(argv[0], "repack"))
      		return cmd_multi_pack_index_repack(argc, argv);
     @@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv,
    + 		return cmd_multi_pack_index_verify(argc, argv);
      	else if (!strcmp(argv[0], "expire"))
      		return cmd_multi_pack_index_expire(argc, argv);
    - 	else
    +-	else
     -		die(_("unrecognized subcommand: %s"), argv[0]);
    ++	else {
     +usage:
    ++		error(_("unrecognized subcommand: %s"), argv[0]);
     +		usage_with_options(builtin_multi_pack_index_usage,
     +				   builtin_multi_pack_index_options);
    ++	}
      }
 7:  ae85a68ef2 =  7:  37e073ea27 t/helper/test-read-midx.c: add '--show-objects'
 8:  30194a6786 !  8:  d061828e7e midx: allow marking a pack as preferred
    @@ builtin/multi-pack-index.c: static struct option *add_common_options(struct opti
     +		OPT_END(),
     +	};
     +
    -+	options = parse_options_dup(builtin_multi_pack_index_write_options);
    -+	options = add_common_options(options);
    ++	options = add_common_options(builtin_multi_pack_index_write_options);
      
      	trace2_cmd_mode(argv[0]);
      
    @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons
      				   options);
      
     -	return write_midx_file(opts.object_dir, opts.flags);
    ++	FREE_AND_NULL(options);
    ++
     +	return write_midx_file(opts.object_dir, opts.preferred_pack,
     +			       opts.flags);
      }
    @@ midx.c: static int pack_info_compare(const void *_a, const void *_b)
      	return strcmp(a->pack_name, b->pack_name);
      }
      
    -+static int lookup_idx_or_pack_name(struct pack_info *info,
    -+				   uint32_t nr,
    -+				   const char *pack_name)
    ++static int idx_or_pack_name_cmp(const void *_va, const void *_vb)
     +{
    -+	uint32_t lo = 0, hi = nr;
    -+	while (lo < hi) {
    -+		uint32_t mi = lo + (hi - lo) / 2;
    -+		int cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);
    -+		if (cmp < 0)
    -+			hi = mi;
    -+		else if (cmp > 0)
    -+			lo = mi + 1;
    -+		else
    -+			return mi;
    -+	}
    -+	return -1;
    ++	const char *pack_name = _va;
    ++	const struct pack_info *compar = _vb;
    ++
    ++	return cmp_idx_or_pack_name(pack_name, compar->pack_name);
     +}
     +
      struct write_midx_context {
    @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
      
     +	/* Check that the preferred pack wasn't expired (if given). */
     +	if (preferred_pack_name) {
    -+		int preferred_idx = lookup_idx_or_pack_name(ctx.info,
    -+							    ctx.nr,
    -+							    preferred_pack_name);
    -+		if (preferred_idx < 0)
    ++		struct pack_info *preferred = bsearch(preferred_pack_name,
    ++						      ctx.info, ctx.nr,
    ++						      sizeof(*ctx.info),
    ++						      idx_or_pack_name_cmp);
    ++
    ++		if (!preferred)
     +			warning(_("unknown preferred pack: '%s'"),
     +				preferred_pack_name);
     +		else {
    -+			uint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;
    -+			uint32_t perm = ctx.pack_perm[orig];
    -+
    ++			uint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];
     +			if (perm == PACK_EXPIRED)
     +				warning(_("preferred pack '%s' is expired"),
     +					preferred_pack_name);
    @@ midx.h: int fill_midx_entry(struct repository *r, const struct object_id *oid, s
      int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);
     
      ## t/t5319-multi-pack-index.sh ##
    -@@ t/t5319-multi-pack-index.sh: midx_read_expect () {
    - 	test_cmp expect actual
    - }
    - 
    -+midx_expect_object_offset () {
    -+	OID="$1"
    -+	OFFSET="$2"
    -+	OBJECT_DIR="$3"
    -+	test-tool read-midx --show-objects $OBJECT_DIR >actual &&
    -+	grep "^$OID $OFFSET" actual
    -+}
    -+
    - test_expect_success 'setup' '
    - 	test_oid_cache <<-EOF
    - 	idxoff sha1:2999
     @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version' '
      	)
      '
    @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'
     +
     +		# Set up two packs, duplicating the object "B" at different
     +		# offsets.
    ++		#
    ++		# Note that the "BC" pack (the one we choose as preferred) sorts
    ++		# lexically after the "AB" pack, meaning that omitting the
    ++		# --preferred-pack argument would cause this test to fail (since
    ++		# the MIDX code would select the copy of "b" in the "AB" pack).
     +		git pack-objects objects/pack/test-AB <<-EOF &&
     +		$a
     +		$b
    @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'
     +			write --preferred-pack=test-BC-$bc.idx 2>err &&
     +		test_must_be_empty err &&
     +
    ++		echo hi &&
    ++		test-tool read-midx --show-objects objects >out &&
    ++
     +		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |
     +			cut -d" " -f1) &&
    -+		midx_expect_object_offset $b $ofs objects
    ++		printf "%s %s\tobjects/pack/test-BC-%s.pack\n" \
    ++			"$b" "$ofs" "$bc" >expect &&
    ++		grep ^$b out >actual &&
    ++
    ++		test_cmp expect actual
     +	)
     +'
 9:  5c5aca761a =  9:  33b8af97e7 midx: don't free midx_name early
10:  a22a1463a5 = 10:  3fc9b83dc6 midx: keep track of the checksum
11:  efa54479b1 = 11:  2ada397320 midx: make some functions non-static
12:  4745bb8590 ! 12:  8bb3dd24a7 Documentation/technical: describe multi-pack reverse indexes

@@ Documentation/technical/pack-format.txt: CHUNK DATA:
+position.
+
+One solution is to let bits occupy the same position in the oid-sorted
-+index stored by the MIDX. But because oids are effectively random, there ++index stored by the MIDX. But because oids are effectively random, their
+resulting reachability bitmaps would have no locality, and thus compress +poorly. (This is the reason that single-pack bitmaps use the pack +ordering, and not the .idx ordering, for the same purpose.)
@@ Documentation/technical/pack-format.txt: CHUNK DATA:
+order in the actual packfile.
+
+Given the list of packs and their counts of objects, you can
-+na&iuml;vely reconstruct that pseudo-pack ordering (e.g., the object at ++naïvely reconstruct that pseudo-pack ordering (e.g., the object at
+position 27 must be (c,1) because packs "a" and "b" consumed 25 of the +slots). But there's a catch. Objects may be duplicated between packs, in +which case the MIDX only stores one pointer to the object (and thus we'd
@@ Documentation/technical/pack-format.txt: CHUNK DATA:
+too.
+
+Objects from the MIDX are ordered as follows to string together the
-+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected ++pseudo-pack. Let pack(o) return the pack from which o was selected
+by the MIDX, and define an ordering of packs based on their numeric ID
-+(as stored by the MIDX). Let _offset(o)_ return the object offset of _o_ -+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:
++(as stored by the MIDX). Let offset(o) return the object offset of o ++within pack(o). Then, compare o1 and o2 as follows:
+
-+ - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other ++ - If one of pack(o1) and pack(o2) is preferred and the other
+ is not, then the preferred one sorts first.
++
+(This is a detail that allows the MIDX bitmap to determine which +pack should be used by the pack-reuse mechanism, since it can ask +the MIDX for the pack containing the object at bit position 0). +
-+ - If _pack(o~1~) &ne; pack(o~2~)_, then sort the two objects in
-+ descending order based on the pack ID.
++ - If pack(o1) ≠ pack(o2), then sort the two objects in descending
++ order based on the pack ID.
+
-+ - Otherwise, _pack(o~1~) &equals; pack(o~2~)_, and the objects are
-+ sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly
-+ when _offset(o~1~) &lt; offset(o~2~)_).
++ - Otherwise, pack(o1) = pack(o2), and the objects are sorted in
++ pack-order (i.e., o1 sorts ahead of o2 exactly when `offset(o1)
++ < offset(o2)`).
+
+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of +objects in packs stored by the MIDX, laid out in pack order, and the
13: a6ebd4be91 ! 13: c070b9c99f pack-revindex: read multi-pack reverse indexes

    @@ pack-revindex.c: int load_pack_revindex(struct packed_git *p)
     +
     +int close_midx_revindex(struct multi_pack_index *m)
     +{
    -+	if (!m)
    ++	if (!m || !m->revindex_map)
     +		return 0;
     +
    -+	if (munmap((void*)m->revindex_map, m->revindex_len))
    -+		return -1;
    ++	munmap((void*)m->revindex_map, m->revindex_len);
     +
     +	m->revindex_map = NULL;
     +	m->revindex_data = NULL;
    @@ pack-revindex.c: off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)
     +		BUG("pack_pos_to_midx: reverse index not yet loaded");
     +	if (m->num_objects <= pos)
     +		BUG("pack_pos_to_midx: out-of-bounds object at %"PRIu32, pos);
    -+	return get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));
    ++	return get_be32(m->revindex_data + pos);
     +}
     +
     +struct midx_pack_key {
    @@ pack-revindex.c: off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)
     +	key.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));
     +
     +	found = bsearch(&key, m->revindex_data, m->num_objects,
    -+			sizeof(uint32_t), midx_pack_order_cmp);
    ++			sizeof(*m->revindex_data), midx_pack_order_cmp);
     +
     +	if (!found)
     +		return error("bad offset for revindex");
14:  f5314f1822 = 14:  9f40019eb3 pack-write.c: extract 'write_rev_file_order'
15:  fa3acb5d5a = 15:  47409cc508 pack-revindex: write multi-pack reverse indexes
16:  550e785f10 = 16:  7b793e7d09 midx.c: improve cache locality in midx_pack_order_cmp()
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#143 lore raw

PATCH v4 1/16 builtin/multi-pack-index.c: inline 'flags' with options

Subcommands of the 'git multi-pack-index' command (e.g., 'write', 'verify', etc.) will want to optionally change a set of shared flags that are eventually passed to the MIDX libraries.

Right now, options and flags are handled separately. That's fine, since the options structure is never passed around. But a future patch will make it so that common options shared by all sub-commands are defined in a common location. That means that "flags" would have to become a global variable.

Group it with the options structure so that we reduce the number of global variables we have overall.

---
 builtin/multi-pack-index.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
builtin/multi-pack-index.c +6 -7
index 5bf88cd2a8..4a0ddb06c4 100644
@@ -14,13 +14,12 @@ static struct opts_multi_pack_index {
 	const char *object_dir;
 	unsigned long batch_size;
 	int progress;
+	unsigned flags;
 } opts;
 
 int cmd_multi_pack_index(int argc, const char **argv,
 			 const char *prefix)
 {
-	unsigned flags = 0;
-
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
@@ -40,7 +39,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
 	if (opts.progress)
-		flags |= MIDX_PROGRESS;
+		opts.flags |= MIDX_PROGRESS;
 
 	if (argc == 0)
 		usage_with_options(builtin_multi_pack_index_usage,
@@ -55,16 +54,16 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	if (!strcmp(argv[0], "repack"))
 		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, flags);
+			(size_t)opts.batch_size, opts.flags);
 	if (opts.batch_size)
 		die(_("--batch-size option is only for 'repack' subcommand"));
 
 	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, flags);
+		return write_midx_file(opts.object_dir, opts.flags);
 	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, flags);
+		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
 	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, flags);
+		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
 
 	die(_("unrecognized subcommand: %s"), argv[0]);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#144 lore raw

PATCH v4 2/16 builtin/multi-pack-index.c: don't handle 'progress' separately

In reply to Taylor Blau

Now that there is a shared 'flags' member in the options structure, there is no need to keep track of whether to force progress or not, since ultimately the decision of whether or not to show a progress meter is controlled by a bit in the flags member.

Manipulate that bit directly, and drop the now-unnecessary 'progress' field while we're at it.

---
 builtin/multi-pack-index.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
builtin/multi-pack-index.c +3 -5
index 4a0ddb06c4..c70f020d8f 100644
@@ -13,7 +13,6 @@ static char const * const builtin_multi_pack_index_usage[] = {
 static struct opts_multi_pack_index {
 	const char *object_dir;
 	unsigned long batch_size;
-	int progress;
 	unsigned flags;
 } opts;
 
@@ -23,7 +22,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 	static struct option builtin_multi_pack_index_options[] = {
 		OPT_FILENAME(0, "object-dir", &opts.object_dir,
 		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BOOL(0, "progress", &opts.progress, N_("force progress reporting")),
+		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,
 
 	git_config(git_default_config, NULL);
 
-	opts.progress = isatty(2);
+	if (isatty(2))
+		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
 			     builtin_multi_pack_index_usage, 0);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
-	if (opts.progress)
-		opts.flags |= MIDX_PROGRESS;
 
 	if (argc == 0)
 		usage_with_options(builtin_multi_pack_index_usage,
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#145 lore raw

PATCH v4 3/16 builtin/multi-pack-index.c: define common usage with a macro

In reply to Taylor Blau

Factor out the usage message into pieces corresponding to each mode. This avoids options specific to one sub-command from being shared with another in the usage.

A subsequent commit will use these #define macros to have usage variables for each sub-command without duplicating their contents.

---
 builtin/multi-pack-index.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
builtin/multi-pack-index.c +16 -1
index c70f020d8f..eea498e026 100644
@@ -5,8 +5,23 @@
 #include "midx.h"
 #include "trace2.h"
 
+#define BUILTIN_MIDX_WRITE_USAGE \
+	N_("git multi-pack-index [<options>] write")
+
+#define BUILTIN_MIDX_VERIFY_USAGE \
+	N_("git multi-pack-index [<options>] verify")
+
+#define BUILTIN_MIDX_EXPIRE_USAGE \
+	N_("git multi-pack-index [<options>] expire")
+
+#define BUILTIN_MIDX_REPACK_USAGE \
+	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
+
 static char const * const builtin_multi_pack_index_usage[] = {
-	N_("git multi-pack-index [<options>] (write|verify|expire|repack --batch-size=<size>)"),
+	BUILTIN_MIDX_WRITE_USAGE,
+	BUILTIN_MIDX_VERIFY_USAGE,
+	BUILTIN_MIDX_EXPIRE_USAGE,
+	BUILTIN_MIDX_REPACK_USAGE,
 	NULL
 };
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#146 lore raw

PATCH v4 4/16 builtin/multi-pack-index.c: split sub-commands

In reply to Taylor Blau

Handle sub-commands of the 'git multi-pack-index' builtin (e.g., "write", "repack", etc.) separately from one another. This allows sub-commands with unique options, without forcing cmd_multi_pack_index() to reject invalid combinations itself.

This comes at the cost of some duplication and boilerplate. Luckily, the duplication is reduced to a minimum, since common options are shared among sub-commands due to a suggestion by Ævar. (Sub-commands do have to retain the common options, too, since this builtin accepts common options on either side of the sub-command).

Roughly speaking, cmd_multi_pack_index() parses options (including common ones), and stops at the first non-option, which is the sub-command. It then dispatches to the appropriate sub-command, which parses the remaining options (also including common options).

Unknown options are kept by the sub-commands in order to detect their presence (and complain that too many arguments were given).

---
 builtin/multi-pack-index.c | 130 ++++++++++++++++++++++++++++++-------
 1 file changed, 105 insertions(+), 25 deletions(-)
builtin/multi-pack-index.c +105 -25
index eea498e026..a78640c061 100644
@@ -17,6 +17,22 @@
 #define BUILTIN_MIDX_REPACK_USAGE \
 	N_("git multi-pack-index [<options>] repack [--batch-size=<size>]")
 
+static char const * const builtin_multi_pack_index_write_usage[] = {
+	BUILTIN_MIDX_WRITE_USAGE,
+	NULL
+};
+static char const * const builtin_multi_pack_index_verify_usage[] = {
+	BUILTIN_MIDX_VERIFY_USAGE,
+	NULL
+};
+static char const * const builtin_multi_pack_index_expire_usage[] = {
+	BUILTIN_MIDX_EXPIRE_USAGE,
+	NULL
+};
+static char const * const builtin_multi_pack_index_repack_usage[] = {
+	BUILTIN_MIDX_REPACK_USAGE,
+	NULL
+};
 static char const * const builtin_multi_pack_index_usage[] = {
 	BUILTIN_MIDX_WRITE_USAGE,
 	BUILTIN_MIDX_VERIFY_USAGE,
@@ -31,25 +47,98 @@ static struct opts_multi_pack_index {
 	unsigned flags;
 } opts;
 
-int cmd_multi_pack_index(int argc, const char **argv,
-			 const char *prefix)
+static struct option common_opts[] = {
+	OPT_FILENAME(0, "object-dir", &opts.object_dir,
+	  N_("object directory containing set of packfile and pack-index pairs")),
+	OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	OPT_END(),
+};
+
+static struct option *add_common_options(struct option *prev)
 {
-	static struct option builtin_multi_pack_index_options[] = {
-		OPT_FILENAME(0, "object-dir", &opts.object_dir,
-		  N_("object directory containing set of packfile and pack-index pairs")),
-		OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS),
+	return parse_options_concat(common_opts, prev);
+}
+
+static int cmd_multi_pack_index_write(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_write_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_write_usage,
+				   options);
+
+	return write_midx_file(opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_verify(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_verify_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_verify_usage,
+				   options);
+
+	return verify_midx_file(the_repository, opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_expire(int argc, const char **argv)
+{
+	struct option *options = common_opts;
+
+	argc = parse_options(argc, argv, NULL,
+			     options, builtin_multi_pack_index_expire_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_expire_usage,
+				   options);
+
+	return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
+}
+
+static int cmd_multi_pack_index_repack(int argc, const char **argv)
+{
+	struct option *options;
+	static struct option builtin_multi_pack_index_repack_options[] = {
 		OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
 		  N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
 		OPT_END(),
 	};
 
+	options = add_common_options(builtin_multi_pack_index_repack_options);
+
+	argc = parse_options(argc, argv, NULL,
+			     options,
+			     builtin_multi_pack_index_repack_usage,
+			     PARSE_OPT_KEEP_UNKNOWN);
+	if (argc)
+		usage_with_options(builtin_multi_pack_index_repack_usage,
+				   options);
+
+	FREE_AND_NULL(options);
+
+	return midx_repack(the_repository, opts.object_dir,
+			   (size_t)opts.batch_size, opts.flags);
+}
+
+int cmd_multi_pack_index(int argc, const char **argv,
+			 const char *prefix)
+{
+	struct option *builtin_multi_pack_index_options = common_opts;
+
 	git_config(git_default_config, NULL);
 
 	if (isatty(2))
 		opts.flags |= MIDX_PROGRESS;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_multi_pack_index_options,
-			     builtin_multi_pack_index_usage, 0);
+			     builtin_multi_pack_index_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
 
 	if (!opts.object_dir)
 		opts.object_dir = get_object_directory();
@@ -58,25 +147,16 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		usage_with_options(builtin_multi_pack_index_usage,
 				   builtin_multi_pack_index_options);
 
-	if (argc > 1) {
-		die(_("too many arguments"));
-		return 1;
-	}
-
 	trace2_cmd_mode(argv[0]);
 
 	if (!strcmp(argv[0], "repack"))
-		return midx_repack(the_repository, opts.object_dir,
-			(size_t)opts.batch_size, opts.flags);
-	if (opts.batch_size)
-		die(_("--batch-size option is only for 'repack' subcommand"));
-
-	if (!strcmp(argv[0], "write"))
-		return write_midx_file(opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "verify"))
-		return verify_midx_file(the_repository, opts.object_dir, opts.flags);
-	if (!strcmp(argv[0], "expire"))
-		return expire_midx_packs(the_repository, opts.object_dir, opts.flags);
-
-	die(_("unrecognized subcommand: %s"), argv[0]);
+		return cmd_multi_pack_index_repack(argc, argv);
+	else if (!strcmp(argv[0], "write"))
+		return cmd_multi_pack_index_write(argc, argv);
+	else if (!strcmp(argv[0], "verify"))
+		return cmd_multi_pack_index_verify(argc, argv);
+	else if (!strcmp(argv[0], "expire"))
+		return cmd_multi_pack_index_expire(argc, argv);
+	else
+		die(_("unrecognized subcommand: %s"), argv[0]);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#147 lore raw

PATCH v4 5/16 builtin/multi-pack-index.c: don't enter bogus cmd_mode

In reply to Taylor Blau

Even before the recent refactoring, 'git multi-pack-index' calls 'trace2_cmd_mode()' before verifying that the sub-command is recognized.

Push this call down into the individual sub-commands so that we don't enter a bogus command mode.

---
 builtin/multi-pack-index.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
builtin/multi-pack-index.c +8 -2
index a78640c061..b590c4fc88 100644
@@ -63,6 +63,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_write_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -77,6 +79,8 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_verify_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -91,6 +95,8 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv)
 {
 	struct option *options = common_opts;
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options, builtin_multi_pack_index_expire_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
@@ -112,6 +118,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)
 
 	options = add_common_options(builtin_multi_pack_index_repack_options);
 
+	trace2_cmd_mode(argv[0]);
+
 	argc = parse_options(argc, argv, NULL,
 			     options,
 			     builtin_multi_pack_index_repack_usage,
@@ -147,8 +155,6 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		usage_with_options(builtin_multi_pack_index_usage,
 				   builtin_multi_pack_index_options);
 
-	trace2_cmd_mode(argv[0]);
-
 	if (!strcmp(argv[0], "repack"))
 		return cmd_multi_pack_index_repack(argc, argv);
 	else if (!strcmp(argv[0], "write"))
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#148 lore raw

PATCH v4 6/16 builtin/multi-pack-index.c: display usage on unrecognized command

In reply to Taylor Blau

When given a sub-command that it doesn't understand, 'git multi-pack-index' dies with the following message:

    $ git multi-pack-index bogus
    fatal: unrecognized subcommand: bogus

Instead of 'die()'-ing, we can display the usage text, which is much more helpful:

    $ git.compile multi-pack-index bogus
    error: unrecognized subcommand: bogus
    usage: git multi-pack-index [<options>] write
       or: git multi-pack-index [<options>] verify
       or: git multi-pack-index [<options>] expire
       or: git multi-pack-index [<options>] repack [--batch-size=<size>]

--object-dir <file> object directory containing set of packfile and pack-index pairs
--progress force progress reporting

While we're at it, clean up some duplication between the "no sub-command" and "unrecognized sub-command" conditionals.

---
 builtin/multi-pack-index.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
builtin/multi-pack-index.c +7 -4
index b590c4fc88..8711174fae 100644
@@ -152,8 +152,7 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		opts.object_dir = get_object_directory();
 
 	if (argc == 0)
-		usage_with_options(builtin_multi_pack_index_usage,
-				   builtin_multi_pack_index_options);
+		goto usage;
 
 	if (!strcmp(argv[0], "repack"))
 		return cmd_multi_pack_index_repack(argc, argv);
@@ -163,6 +162,10 @@ int cmd_multi_pack_index(int argc, const char **argv,
 		return cmd_multi_pack_index_verify(argc, argv);
 	else if (!strcmp(argv[0], "expire"))
 		return cmd_multi_pack_index_expire(argc, argv);
-	else
-		die(_("unrecognized subcommand: %s"), argv[0]);
+	else {
+usage:
+		error(_("unrecognized subcommand: %s"), argv[0]);
+		usage_with_options(builtin_multi_pack_index_usage,
+				   builtin_multi_pack_index_options);
+	}
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#149 lore raw

PATCH v4 7/16 t/helper/test-read-midx.c: add '--show-objects'

In reply to Taylor Blau

The 'read-midx' helper is used in places like t5319 to display basic information about a multi-pack-index.

In the next patch, the MIDX writing machinery will learn a new way to choose from which pack an object is selected when multiple copies of that object exist.

To disambiguate which pack introduces an object so that this feature can be tested, add a '--show-objects' option which displays additional information about each object in the MIDX.

---
 t/helper/test-read-midx.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)
t/helper/test-read-midx.c +20 -4
index 2430880f78..7c2eb11a8e 100644
@@ -4,7 +4,7 @@
 #include "repository.h"
 #include "object-store.h"
 
-static int read_midx_file(const char *object_dir)
+static int read_midx_file(const char *object_dir, int show_objects)
 {
 	uint32_t i;
 	struct multi_pack_index *m;
@@ -43,13 +43,29 @@ static int read_midx_file(const char *object_dir)
 
 	printf("object-dir: %s\n", m->object_dir);
 
+	if (show_objects) {
+		struct object_id oid;
+		struct pack_entry e;
+
+		for (i = 0; i < m->num_objects; i++) {
+			nth_midxed_object_oid(&oid, m, i);
+			fill_midx_entry(the_repository, &oid, &e, m);
+
+			printf("%s %"PRIu64"\t%s\n",
+			       oid_to_hex(&oid), e.offset, e.p->pack_name);
+		}
+		return 0;
+	}
+
 	return 0;
 }
 
 int cmd__read_midx(int argc, const char **argv)
 {
-	if (argc != 2)
-		usage("read-midx <object-dir>");
+	if (!(argc == 2 || argc == 3))
+		usage("read-midx [--show-objects] <object-dir>");
 
-	return read_midx_file(argv[1]);
+	if (!strcmp(argv[1], "--show-objects"))
+		return read_midx_file(argv[2], 1);
+	return read_midx_file(argv[1], 0);
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#150 lore raw

PATCH v4 8/16 midx: allow marking a pack as preferred

In reply to Taylor Blau

When multiple packs in the multi-pack index contain the same object, the MIDX machinery must make a choice about which pack it associates with that object. Prior to this patch, the lowest-ordered[1] pack was always selected.

Pack selection for duplicate objects is relatively unimportant today, but it will become important for multi-pack bitmaps. This is because we can only invoke the pack-reuse mechanism when all of the bits for reused objects come from the reuse pack (in order to ensure that all reused deltas can find their base objects in the same pack).

To encourage the pack selection process to prefer one pack over another (the pack to be preferred is the one a caller would like to later use as a reuse pack), introduce the concept of a "preferred pack". When provided, the MIDX code will always prefer an object found in a preferred pack over any other.

No format changes are required to store the preferred pack, since it will be able to be inferred with a corresponding MIDX bitmap, by looking up the pack associated with the object in the first bit position (this ordering is described in detail in a subsequent commit).

[1]: the ordering is specified by MIDX internals; for our purposes we can consider the "lowest ordered" pack to be "the one with the most-recent mtime.

---
 Documentation/git-multi-pack-index.txt       | 14 +++-
 Documentation/technical/multi-pack-index.txt |  5 +-
 builtin/multi-pack-index.c                   | 19 ++++-
 builtin/repack.c                             |  2 +-
 midx.c                                       | 82 +++++++++++++++++---
 midx.h                                       |  2 +-
 t/t5319-multi-pack-index.sh                  | 43 ++++++++++
 7 files changed, 149 insertions(+), 18 deletions(-)
Documentation/git-multi-pack-index.txt +12 -2
index eb0caa0439..ffd601bc17 100644
@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes
 SYNOPSIS
 --------
 [verse]
-'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress] <subcommand>
+'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
+	[--preferred-pack=<pack>] <subcommand>
 
 DESCRIPTION
 -----------
@@ -30,7 +31,16 @@ OPTIONS
 The following subcommands are available:
 
 write::
-	Write a new MIDX file.
+	Write a new MIDX file. The following options are available for
+	the `write` sub-command:
++
+--
+	--preferred-pack=<pack>::
+		Optionally specify the tie-breaking pack used when
+		multiple packs contain the same object. If not given,
+		ties are broken in favor of the pack with the lowest
+		mtime.
+--
 
 verify::
 	Verify the contents of the MIDX file.
Documentation/technical/multi-pack-index.txt +3 -2
index e8e377a59f..fb688976c4 100644
@@ -43,8 +43,9 @@ Design Details
   a change in format.
 
 - The MIDX keeps only one record per object ID. If an object appears
-  in multiple packfiles, then the MIDX selects the copy in the most-
-  recently modified packfile.
+  in multiple packfiles, then the MIDX selects the copy in the
+  preferred packfile, otherwise selecting from the most-recently
+  modified packfile.
 
 - If there exist packfiles in the pack directory not registered in
   the MIDX, then those packfiles are loaded into the `packed_git`
builtin/multi-pack-index.c +16 -3
index 8711174fae..5d3ea445fd 100644
@@ -4,9 +4,10 @@
 #include "parse-options.h"
 #include "midx.h"
 #include "trace2.h"
+#include "object-store.h"
 
 #define BUILTIN_MIDX_WRITE_USAGE \
-	N_("git multi-pack-index [<options>] write")
+	N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]")
 
 #define BUILTIN_MIDX_VERIFY_USAGE \
 	N_("git multi-pack-index [<options>] verify")
@@ -43,6 +44,7 @@ static char const * const builtin_multi_pack_index_usage[] = {
 
 static struct opts_multi_pack_index {
 	const char *object_dir;
+	const char *preferred_pack;
 	unsigned long batch_size;
 	unsigned flags;
 } opts;
@@ -61,7 +63,15 @@ static struct option *add_common_options(struct option *prev)
 
 static int cmd_multi_pack_index_write(int argc, const char **argv)
 {
-	struct option *options = common_opts;
+	struct option *options;
+	static struct option builtin_multi_pack_index_write_options[] = {
+		OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
+			   N_("preferred-pack"),
+			   N_("pack for reuse when computing a multi-pack bitmap")),
+		OPT_END(),
+	};
+
+	options = add_common_options(builtin_multi_pack_index_write_options);
 
 	trace2_cmd_mode(argv[0]);
 
@@ -72,7 +82,10 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		usage_with_options(builtin_multi_pack_index_write_usage,
 				   options);
 
-	return write_midx_file(opts.object_dir, opts.flags);
+	FREE_AND_NULL(options);
+
+	return write_midx_file(opts.object_dir, opts.preferred_pack,
+			       opts.flags);
 }
 
 static int cmd_multi_pack_index_verify(int argc, const char **argv)
builtin/repack.c +1 -1
index 6ce2556c9e..2847fdfbab 100644
@@ -721,7 +721,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 	remove_temporary_files();
 
 	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
-		write_midx_file(get_object_directory(), 0);
+		write_midx_file(get_object_directory(), NULL, 0);
 
 	string_list_clear(&names, 0);
 	string_list_clear(&rollback, 0);
midx.c +73 -9

179 lines not on this page · raw message

midx.h +1 -1
index b18cf53bc4..e7fea61109 100644
@@ -47,7 +47,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa
 int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);
 int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);
 
-int write_midx_file(const char *object_dir, unsigned flags);
+int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);
 void clear_midx_file(struct repository *r);
 int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);
 int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);
t/t5319-multi-pack-index.sh +43 -0

49 lines not on this page · raw message

Signature · 1 line
2.30.0.667.g81c0cbc6fd
#151 lore raw

Re: [PATCH v4 08/16] midx: allow marking a pack as preferred

Junio,

On Tue, Mar 30, 2021 at 11:04:11AM -0400, Taylor Blau wrote:

I accidentally left a stray debugging line in here, and managed to skip over it when reading the range-diff. It's right...

Quoted 30 lines
t/t5319-multi-pack-index.sh +22 -0
index b4afab1dfc..031a5570c0 100755
@@ -234,6 +234,49 @@ test_expect_success 'warn on improper hash version' '
 	)
 '
 
+test_expect_success 'midx picks objects from preferred pack' '
+	test_when_finished rm -rf preferred.git &&
+	git init --bare preferred.git &&
+	(
+		cd preferred.git &&
+
+		a=$(echo "a" | git hash-object -w --stdin) &&
+		b=$(echo "b" | git hash-object -w --stdin) &&
+		c=$(echo "c" | git hash-object -w --stdin) &&
+
+		# Set up two packs, duplicating the object "B" at different
+		# offsets.
+		#
+		# Note that the "BC" pack (the one we choose as preferred) sorts
+		# lexically after the "AB" pack, meaning that omitting the
+		# --preferred-pack argument would cause this test to fail (since
+		# the MIDX code would select the copy of "b" in the "AB" pack).
+		git pack-objects objects/pack/test-AB <<-EOF &&
+		$a
+		$b
+		EOF
+		bc=$(git pack-objects objects/pack/test-BC <<-EOF

t/t5319-multi-pack-index.sh test_expect_success 'warn on improper hash version' ' · line 268

Quoted 10 lines
t/t5319-multi-pack-index.sh +10 -0
+		$b
+		$c
+		EOF
+		) &&
+
+		git multi-pack-index --object-dir=objects \
+			write --preferred-pack=test-BC-$bc.idx 2>err &&
+		test_must_be_empty err &&
+
+		echo hi &&

...here. Would you mind fixing it up locally before applying this to next?

Sorry for the trouble.

Thanks,
Taylor

#152 lore raw

PATCH v4 9/16 midx: don't free midx_name early

In reply to Taylor Blau

A subsequent patch will need to refer back to 'midx_name' later on in the function. In fact, this variable is already free()'d later on, so this makes the later free() no longer redundant.

---
 midx.c | 1 -
 1 file changed, 1 deletion(-)
midx.c +0 -1
index 4a9d84352c..3edde2b68d 100644
@@ -956,7 +956,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
-	FREE_AND_NULL(midx_name);
 
 	if (ctx.m)
 		close_midx(ctx.m);
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#153 lore raw

PATCH v4 10/16 midx: keep track of the checksum

In reply to Taylor Blau

write_midx_internal() uses a hashfile to write the multi-pack index, but discards its checksum. This makes sense, since nothing that takes place after writing the MIDX cares about its checksum.

That is about to change in a subsequent patch, when the optional reverse index corresponding to the MIDX will want to include the MIDX's checksum.

Store the checksum of the MIDX in preparation for that.

---
 midx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
midx.c +2 -1
index 3edde2b68d..526795ff0e 100644
@@ -811,6 +811,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			       unsigned flags)
 {
 	char *midx_name;
+	unsigned char midx_hash[GIT_MAX_RAWSZ];
 	uint32_t i;
 	struct hashfile *f = NULL;
 	struct lock_file lk;
@@ -987,7 +988,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
 	write_chunkfile(cf, &ctx);
 
-	finalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
 	commit_lock_file(&lk);
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#154 lore raw

PATCH v4 11/16 midx: make some functions non-static

In reply to Taylor Blau

In a subsequent commit, pack-revindex.c will become responsible for sorting a list of objects in the "MIDX pack order" (which will be defined in the following patch). To do so, it will need to be know the pack identifier and offset within that pack for each object in the MIDX.

The MIDX code already has functions for doing just that (nth_midxed_offset() and nth_midxed_pack_int_id()), but they are statically declared.

Since there is no reason that they couldn't be exposed publicly, and because they are already doing exactly what the caller in pack-revindex.c will want, expose them publicly so that they can be reused there.

---
 midx.c | 4 ++--
 midx.h | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)
midx.c +2 -2
index 526795ff0e..2173a9b45c 100644
@@ -239,7 +239,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,
 	return oid;
 }
 
-static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
 {
 	const unsigned char *offset_data;
 	uint32_t offset32;
@@ -258,7 +258,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
 	return offset32;
 }
 
-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
 {
 	return get_be32(m->chunk_object_offsets +
 			(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);
midx.h +2 -0
index e7fea61109..93bd68189e 100644
@@ -40,6 +40,8 @@ struct multi_pack_index {
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);
+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);
 struct object_id *nth_midxed_object_oid(struct object_id *oid,
 					struct multi_pack_index *m,
 					uint32_t n);
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#155 lore raw

PATCH v4 12/16 Documentation/technical: describe multi-pack reverse indexes

In reply to Taylor Blau

As a prerequisite to implementing multi-pack bitmaps, motivate and describe the format and ordering of the multi-pack reverse index.

The subsequent patch will implement reading this format, and the patch after that will implement writing it while producing a multi-pack index.

---
 Documentation/technical/pack-format.txt | 83 +++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
Documentation/technical/pack-format.txt +83 -0
index 1faa949bf6..8d2f42f29e 100644
@@ -379,3 +379,86 @@ CHUNK DATA:
 TRAILER:
 
 	Index checksum of the above contents.
+
+== multi-pack-index reverse indexes
+
+Similar to the pack-based reverse index, the multi-pack index can also
+be used to generate a reverse index.
+
+Instead of mapping between offset, pack-, and index position, this
+reverse index maps between an object's position within the MIDX, and
+that object's position within a pseudo-pack that the MIDX describes
+(i.e., the ith entry of the multi-pack reverse index holds the MIDX
+position of ith object in pseudo-pack order).
+
+To clarify the difference between these orderings, consider a multi-pack
+reachability bitmap (which does not yet exist, but is what we are
+building towards here). Each bit needs to correspond to an object in the
+MIDX, and so we need an efficient mapping from bit position to MIDX
+position.
+
+One solution is to let bits occupy the same position in the oid-sorted
+index stored by the MIDX. But because oids are effectively random, their
+resulting reachability bitmaps would have no locality, and thus compress
+poorly. (This is the reason that single-pack bitmaps use the pack
+ordering, and not the .idx ordering, for the same purpose.)
+
+So we'd like to define an ordering for the whole MIDX based around
+pack ordering, which has far better locality (and thus compresses more
+efficiently). We can think of a pseudo-pack created by the concatenation
+of all of the packs in the MIDX. E.g., if we had a MIDX with three packs
+(a, b, c), with 10, 15, and 20 objects respectively, we can imagine an
+ordering of the objects like:
+
+    |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|
+
+where the ordering of the packs is defined by the MIDX's pack list,
+and then the ordering of objects within each pack is the same as the
+order in the actual packfile.
+
+Given the list of packs and their counts of objects, you can
+naïvely reconstruct that pseudo-pack ordering (e.g., the object at
+position 27 must be (c,1) because packs "a" and "b" consumed 25 of the
+slots). But there's a catch. Objects may be duplicated between packs, in
+which case the MIDX only stores one pointer to the object (and thus we'd
+want only one slot in the bitmap).
+
+Callers could handle duplicates themselves by reading objects in order
+of their bit-position, but that's linear in the number of objects, and
+much too expensive for ordinary bitmap lookups. Building a reverse index
+solves this, since it is the logical inverse of the index, and that
+index has already removed duplicates. But, building a reverse index on
+the fly can be expensive. Since we already have an on-disk format for
+pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,
+too.
+
+Objects from the MIDX are ordered as follows to string together the
+pseudo-pack. Let `pack(o)` return the pack from which `o` was selected
+by the MIDX, and define an ordering of packs based on their numeric ID
+(as stored by the MIDX). Let `offset(o)` return the object offset of `o`
+within `pack(o)`. Then, compare `o1` and `o2` as follows:
+
+  - If one of `pack(o1)` and `pack(o2)` is preferred and the other
+    is not, then the preferred one sorts first.
++
+(This is a detail that allows the MIDX bitmap to determine which
+pack should be used by the pack-reuse mechanism, since it can ask
+the MIDX for the pack containing the object at bit position 0).
+
+  - If `pack(o1) ≠ pack(o2)`, then sort the two objects in descending
+    order based on the pack ID.
+
+  - Otherwise, `pack(o1) = pack(o2)`, and the objects are sorted in
+    pack-order (i.e., `o1` sorts ahead of `o2` exactly when `offset(o1)
+    < offset(o2)`).
+
+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of
+objects in packs stored by the MIDX, laid out in pack order, and the
+packs arranged in MIDX order (with the preferred pack coming first).
+
+Finally, note that the MIDX's reverse index is not stored as a chunk in
+the multi-pack-index itself. This is done because the reverse index
+includes the checksum of the pack or MIDX to which it belongs, which
+makes it impossible to write in the MIDX. To avoid races when rewriting
+the MIDX, a MIDX reverse index includes the MIDX's checksum in its
+filename (e.g., `multi-pack-index-xyz.rev`).
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#156 lore raw

PATCH v4 13/16 pack-revindex: read multi-pack reverse indexes

In reply to Taylor Blau

Implement reading for multi-pack reverse indexes, as described in the previous patch.

Note that these functions don't yet have any callers, and won't until multi-pack reachability bitmaps are introduced in a later patch series. In the meantime, this patch implements some of the infrastructure necessary to support multi-pack bitmaps.

There are three new functions exposed by the revindex API:

- load_midx_revindex(): loads the reverse index corresponding to the given multi-pack index.

- midx_to_pack_pos() and pack_pos_to_midx(): these convert between the multi-pack index and pseudo-pack order.

load_midx_revindex() and pack_pos_to_midx() are both relatively straightforward.

load_midx_revindex() needs a few functions to be exposed from the midx API. One to get the checksum of a midx, and another to get the .rev's filename. Similar to recent changes in the packed_git struct, three new fields are added to the multi_pack_index struct: one to keep track of the size, one to keep track of the mmap'd pointer, and another to point past the header and at the reverse index's data.

pack_pos_to_midx() simply reads the corresponding entry out of the table.

midx_to_pack_pos() is the trickiest, since it needs to find an object's position in the psuedo-pack order, but that order can only be recovered in the .rev file itself. This mapping can be implemented with a binary search, but note that the thing we're binary searching over isn't an array of values, but rather a permuted order of those values.

So, when comparing two items, it's helpful to keep in mind the difference. Instead of a traditional binary search, where you are comparing two things directly, here we're comparing a (pack, offset) tuple with an index into the multi-pack index. That index describes another (pack, offset) tuple, and it is _those_ two tuples that are compared.

---
 midx.c          |  11 +++++
 midx.h          |   6 +++
 pack-revindex.c | 126 ++++++++++++++++++++++++++++++++++++++++++++++++
 pack-revindex.h |  53 ++++++++++++++++++++
 packfile.c      |   3 ++
 5 files changed, 199 insertions(+)
midx.c +11 -0
index 2173a9b45c..c04e794888 100644
@@ -47,11 +47,22 @@ static uint8_t oid_version(void)
 	}
 }
 
+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)
+{
+	return m->data + m->data_len - the_hash_algo->rawsz;
+}
+
 static char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }
 
+char *get_midx_rev_filename(struct multi_pack_index *m)
+{
+	return xstrfmt("%s/pack/multi-pack-index-%s.rev",
+		       m->object_dir, hash_to_hex(get_midx_checksum(m)));
+}
+
 static int midx_read_oid_fanout(const unsigned char *chunk_start,
 				size_t chunk_size, void *data)
 {
midx.h +6 -0
index 93bd68189e..0a8294d2ee 100644
@@ -15,6 +15,10 @@ struct multi_pack_index {
 	const unsigned char *data;
 	size_t data_len;
 
+	const uint32_t *revindex_data;
+	const uint32_t *revindex_map;
+	size_t revindex_len;
+
 	uint32_t signature;
 	unsigned char version;
 	unsigned char hash_len;
@@ -37,6 +41,8 @@ struct multi_pack_index {
 
 #define MIDX_PROGRESS     (1 << 0)
 
+char *get_midx_rev_filename(struct multi_pack_index *m);
+
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
 int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
pack-revindex.c +126 -0

141 lines not on this page · raw message

pack-revindex.h +53 -0

75 lines not on this page · raw message

packfile.c +3 -0
index 6661f3325a..8668345d93 100644
@@ -862,6 +862,9 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 
 	if (!strcmp(file_name, "multi-pack-index"))
 		return;
+	if (starts_with(file_name, "multi-pack-index") &&
+	    ends_with(file_name, ".rev"))
+		return;
 	if (ends_with(file_name, ".idx") ||
 	    ends_with(file_name, ".rev") ||
 	    ends_with(file_name, ".pack") ||
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#157 lore raw

PATCH v4 14/16 pack-write.c: extract 'write_rev_file_order'

In reply to Taylor Blau

Existing callers provide the reverse index code with an array of 'struct pack_idx_entry *'s, which is then sorted by pack order (comparing the offsets of each object within the pack).

Prepare for the multi-pack index to write a .rev file by providing a way to write the reverse index without an array of pack_idx_entry (which the MIDX code does not have).

Instead, callers can invoke 'write_rev_index_positions()', which takes an array of uint32_t's. The ith entry in this array specifies the ith object's (in index order) position within the pack (in pack order).

Expose this new function for use in a later patch, and rewrite the existing write_rev_file() in terms of this new function.

---
 pack-write.c | 36 +++++++++++++++++++++++++-----------
 pack.h       |  1 +
 2 files changed, 26 insertions(+), 11 deletions(-)
pack-write.c +25 -11
index 2ca85a9d16..f1fc3ecafa 100644
@@ -201,21 +201,12 @@ static void write_rev_header(struct hashfile *f)
 }
 
 static void write_rev_index_positions(struct hashfile *f,
-				      struct pack_idx_entry **objects,
+				      uint32_t *pack_order,
 				      uint32_t nr_objects)
 {
-	uint32_t *pack_order;
 	uint32_t i;
-
-	ALLOC_ARRAY(pack_order, nr_objects);
-	for (i = 0; i < nr_objects; i++)
-		pack_order[i] = i;
-	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
-
 	for (i = 0; i < nr_objects; i++)
 		hashwrite_be32(f, pack_order[i]);
-
-	free(pack_order);
 }
 
 static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)
@@ -228,6 +219,29 @@ const char *write_rev_file(const char *rev_name,
 			   uint32_t nr_objects,
 			   const unsigned char *hash,
 			   unsigned flags)
+{
+	uint32_t *pack_order;
+	uint32_t i;
+	const char *ret;
+
+	ALLOC_ARRAY(pack_order, nr_objects);
+	for (i = 0; i < nr_objects; i++)
+		pack_order[i] = i;
+	QSORT_S(pack_order, nr_objects, pack_order_cmp, objects);
+
+	ret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,
+				   flags);
+
+	free(pack_order);
+
+	return ret;
+}
+
+const char *write_rev_file_order(const char *rev_name,
+				 uint32_t *pack_order,
+				 uint32_t nr_objects,
+				 const unsigned char *hash,
+				 unsigned flags)
 {
 	struct hashfile *f;
 	int fd;
@@ -262,7 +276,7 @@ const char *write_rev_file(const char *rev_name,
 
 	write_rev_header(f);
 
-	write_rev_index_positions(f, objects, nr_objects);
+	write_rev_index_positions(f, pack_order, nr_objects);
 	write_rev_trailer(f, hash);
 
 	if (rev_name && adjust_shared_perm(rev_name) < 0)
pack.h +1 -0
index 857cbd5bd4..fa13954526 100644
@@ -94,6 +94,7 @@ struct ref;
 void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);
 
 const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);
+const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order, uint32_t nr_objects, const unsigned char *hash, unsigned flags);
 
 /*
  * The "hdr" output buffer should be at least this big, which will handle sizes
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#158 lore raw

PATCH 1/1 pack-write: skip *.rev work when not writing *.rev

Fix a performance regression introduced in a587b5a786 (pack-write.c:
extract 'write_rev_file_order', 2021-03-30) and stop needlessly allocating the "pack_order" array and sorting it with "pack_order_cmp()", only to throw that work away when we discover that we're not writing *.rev files after all.

This redundant work was not present in the original version of this code added in 8ef50d9958 (pack-write.c: prepare to write 'pack-*.rev' files, 2021-01-25). There we'd call write_rev_file() from e.g. finish_tmp_packfile(), but we'd "return NULL" early in write_rev_file() if not doing a "WRITE_REV" or "WRITE_REV_VERIFY".

---
 pack-write.c | 3 +++
 1 file changed, 3 insertions(+)
pack-write.c +3 -0
index f1fc3ecafa..1883848e7c 100644
@@ -224,6 +224,9 @@ const char *write_rev_file(const char *rev_name,
 	uint32_t i;
 	const char *ret;
 
+	if (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))
+		return NULL;
+
 	ALLOC_ARRAY(pack_order, nr_objects);
 	for (i = 0; i < nr_objects; i++)
 		pack_order[i] = i;
Signature · 1 line
2.33.0.819.gea1b153a43c
#159 lore raw

Re: [PATCH] pack-write: skip *.rev work when not writing *.rev

pack-write.c const char *write_rev_file(const char *rev_name, · line 228

Quoted 2 lines · Ævar Arnfjörð Bjarmason
pack-write.c +2 -0
+       if (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))
+               return NULL;

I see this expression matches exactly the logic from 8ef50d9958 which is why I presume
you used it, but the simpler (and logically equivalent[1]) :

  if !((flags & WRITE_REV) || (flags & WRITE_REV_VERIFY))

is easier to read IMHO

Carlo

[1] https://en.wikipedia.org/wiki/De_Morgan%27s_laws

#160 lore raw

Re: [PATCH] pack-write: skip *.rev work when not writing *.rev

Quoted 9 lines · Carlo Arenas

pack-write.c const char *write_rev_file(const char *rev_name, · line 228

Quoted 2 lines · Ævar Arnfjörð Bjarmason
pack-write.c +2 -0
+       if (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))
+               return NULL;

I see this expression matches exactly the logic from 8ef50d9958 which is why I presume
you used it, but the simpler (and logically equivalent[1]) :

  if !((flags & WRITE_REV) || (flags & WRITE_REV_VERIFY))

Even simpler would be:

    if (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))

although with optimization flags other than -O0, it seems that each of these three produce the same result [1], so I don't think that it matters much either way ;-).

Thanks,
Taylor

[1]: https://godbolt.org/z/fxxhzEz79

#161 lore raw

Re: [PATCH] pack-write: skip *.rev work when not writing *.rev

Quoted 11 lines · Taylor Blau
Quoted 9 lines · Carlo Arenas

pack-write.c const char *write_rev_file(const char *rev_name, · line 228

Quoted 2 lines · Ævar Arnfjörð Bjarmason
pack-write.c +2 -0
+       if (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))
+               return NULL;

I see this expression matches exactly the logic from 8ef50d9958 which is why I presume
you used it, but the simpler (and logically equivalent[1]) :

  if !((flags & WRITE_REV) || (flags & WRITE_REV_VERIFY))
Quoted 7 lines

Even simpler would be:

    if (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))

although with optimization flags other than -O0, it seems that each of these three produce the same result [1], so I don't think that it matters much either way ;-).

If all result in the same binary, the only deciding factor would be how readable the code is to human readers.

I too find that your version, "we care about these two bits---if flags does not have either of them, then...", the easiest to follow. But the original is not unreadable and is good enough once it has already been written.

Thanks.

#162 lore raw

Re: [PATCH] pack-write: skip *.rev work when not writing *.rev

In reply to Ævar Arnfjörð Bjarmason
Quoted 22 lines · Ævar Arnfjörð Bjarmason

Fix a performance regression introduced in a587b5a786 (pack-write.c:
extract 'write_rev_file_order', 2021-03-30) and stop needlessly allocating the "pack_order" array and sorting it with "pack_order_cmp()", only to throw that work away when we discover that we're not writing *.rev files after all.

This redundant work was not present in the original version of this code added in 8ef50d9958 (pack-write.c: prepare to write 'pack-*.rev' files, 2021-01-25). There we'd call write_rev_file() from e.g. finish_tmp_packfile(), but we'd "return NULL" early in write_rev_file() if not doing a "WRITE_REV" or "WRITE_REV_VERIFY".

---
 pack-write.c | 3 +++
 1 file changed, 3 insertions(+)
pack-write.c +0 -0
index f1fc3ecafa..1883848e7c 100644

pack-write.c const char *write_rev_file(const char *rev_name, · line 229

Quoted 6 lines
pack-write.c +3 -0
@@ -224,6 +224,9 @@ const char *write_rev_file(const char *rev_name,
 	uint32_t i;
 	const char *ret;
 
+	if (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))
+		return NULL;
+

Great catch! This fix as-is is obviously correct, but it does make the same checks in write_rev_file_order redundant as a result.

If we call write_rev_file() without WRITE_REV, then we'll never call write_rev_file_order(). The other caller of write_rev_file_order() is from midx.c:write_midx_reverse_index(), which is only called by write_midx_internal() where it is guarded by a similar conditional.

So I think we could probably either:

- remove the check from write_rev_file_order() altogether, moving it to write_rev_file() like you did here, or

- remove the check from write_rev_file_order() and elevate it to the caller which is missing the check in finish_tmp_packfile()

Of the two, I think the former is more appealing (since no other functions called by finish_tmp_packfile() are guarded like that; they conditionally behave as noops depending on flags).

But what you wrote here is just fine as-is, so the above are just some optional ideas for potential improvements.

Thanks,
Taylor

#163 lore raw

Re: [PATCH] pack-write: skip *.rev work when not writing *.rev

Quoted 3 lines · Taylor Blau

Of the two, I think the former is more appealing (since no other functions called by finish_tmp_packfile() are guarded like that; they conditionally behave as noops depending on flags).

Sorry; this is nonsensical. The only other function we call is write_idx_file() which merely changes its behavior based on flags, but it never behaves as a noop.

That doesn't change my thinking about preferring the former of my two suggestions, but just wanted to correct my error.

Thanks,
Taylor

#164 lore raw

Re: [PATCH] pack-write: skip *.rev work when not writing *.rev

Quoted 11 lines · Taylor Blau
Quoted 3 lines · Taylor Blau

Of the two, I think the former is more appealing (since no other functions called by finish_tmp_packfile() are guarded like that; they conditionally behave as noops depending on flags).

Sorry; this is nonsensical. The only other function we call is write_idx_file() which merely changes its behavior based on flags, but it never behaves as a noop.

That doesn't change my thinking about preferring the former of my two suggestions, but just wanted to correct my error.

I agree that this code is very confusing overall, but would prefer to wait on refactoring further until the two topics in flight (this and the other pack-write topic) settle.

As shown in
https://lore.kernel.org/git/[email protected]/ I think the best thing to do is neither of the narrow fixes you suggest, but to more deeply untangle the whole mess around how we choose to write these files & with what options. A lot of it is bit-twiddling back and forth for no real reason.

Once we do that it becomes impossible to land in a mode where these functions need to in principle deal with writing a "real" file and the "verify" mode, which as noted in the linked E-Mail is the case now, we just need/want these "is more than one set?" checks & assertions because we've made the interface overly confusing/general.

#165 lore raw

Re: [PATCH] pack-write: skip *.rev work when not writing *.rev

Quoted 12 lines · Ævar Arnfjörð Bjarmason
Quoted 9 lines · Taylor Blau
Quoted 3 lines · Taylor Blau

Of the two, I think the former is more appealing (since no other functions called by finish_tmp_packfile() are guarded like that; they conditionally behave as noops depending on flags).

Sorry; this is nonsensical. The only other function we call is write_idx_file() which merely changes its behavior based on flags, but it never behaves as a noop.

Quoted 6 lines
Quoted 2 lines

That doesn't change my thinking about preferring the former of my two suggestions, but just wanted to correct my error.

I agree that this code is very confusing overall, but would prefer to wait on refactoring further until the two topics in flight (this and the other pack-write topic) settle.

I'm fine to wait on any further refactorings. And I agree that this code is confusing, since when I read it last night I thought that the check in write_rev_file_order() was a duplicate of the one you introduced, but it is not:

    if ((flags & WRITE_REV) && (flags & WRITE_REV_VERIFY))
      die(_("cannot both write and verify reverse index"));

and that check is different than the one you added, which I think is appropriate.

So this patch looks good to me, and sorry for the confusion.

Thanks,
Taylor

#166 lore raw

PATCH v4 15/16 pack-revindex: write multi-pack reverse indexes

In reply to Taylor Blau

Implement the writing half of multi-pack reverse indexes. This is nothing more than the format describe a few patches ago, with a new set of helper functions that will be used to clear out stale .rev files corresponding to old MIDXs.

Unfortunately, a very similar comparison function as the one implemented recently in pack-revindex.c is reimplemented here, this time accepting a MIDX-internal type. An effort to DRY these up would create more indirection and overhead than is necessary, so it isn't pursued here.

Currently, there are no callers which pass the MIDX_WRITE_REV_INDEX flag, meaning that this is all dead code. But, that won't be the case for long, since subsequent patches will introduce the multi-pack bitmap, which will begin passing this field.

(In midx.c:write_midx_internal(), the two adjacent if statements share a conditional, but are written separately since the first one will eventually also handle the MIDX_WRITE_BITMAP flag, which does not yet exist.)

---
 midx.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 midx.h |   1 +
 2 files changed, 116 insertions(+)
midx.c +115 -0
index c04e794888..b96eaa12fb 100644
@@ -12,6 +12,7 @@
 #include "run-command.h"
 #include "repository.h"
 #include "chunk-format.h"
+#include "pack.h"
 
 #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
 #define MIDX_VERSION 1
@@ -462,6 +463,7 @@ struct write_midx_context {
 	uint32_t entries_nr;
 
 	uint32_t *pack_perm;
+	uint32_t *pack_order;
 	unsigned large_offsets_needed:1;
 	uint32_t num_large_offsets;
 
@@ -816,6 +818,70 @@ static int write_midx_large_offsets(struct hashfile *f,
 	return 0;
 }
 
+static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)
+{
+	struct write_midx_context *ctx = _ctx;
+
+	struct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];
+	struct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];
+
+	uint32_t perm_a = ctx->pack_perm[a->pack_int_id];
+	uint32_t perm_b = ctx->pack_perm[b->pack_int_id];
+
+	/* Sort objects in the preferred pack ahead of any others. */
+	if (a->preferred > b->preferred)
+		return -1;
+	if (a->preferred < b->preferred)
+		return 1;
+
+	/* Then, order objects by which packs they appear in. */
+	if (perm_a < perm_b)
+		return -1;
+	if (perm_a > perm_b)
+		return 1;
+
+	/* Then, disambiguate by their offset within each pack. */
+	if (a->offset < b->offset)
+		return -1;
+	if (a->offset > b->offset)
+		return 1;
+
+	return 0;
+}
+
+static uint32_t *midx_pack_order(struct write_midx_context *ctx)
+{
+	uint32_t *pack_order;
+	uint32_t i;
+
+	ALLOC_ARRAY(pack_order, ctx->entries_nr);
+	for (i = 0; i < ctx->entries_nr; i++)
+		pack_order[i] = i;
+	QSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);
+
+	return pack_order;
+}
+
+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,
+				     struct write_midx_context *ctx)
+{
+	struct strbuf buf = STRBUF_INIT;
+	const char *tmp_file;
+
+	strbuf_addf(&buf, "%s-%s.rev", midx_name, hash_to_hex(midx_hash));
+
+	tmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,
+					midx_hash, WRITE_REV);
+
+	if (finalize_object_file(tmp_file, buf.buf))
+		die(_("cannot store reverse index file"));
+
+	strbuf_release(&buf);
+}
+
+static void clear_midx_files_ext(struct repository *r, const char *ext,
+				 unsigned char *keep_hash);
+
 static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
 			       struct string_list *packs_to_drop,
 			       const char *preferred_pack_name,
@@ -1001,6 +1067,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
 	free_chunkfile(cf);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		ctx.pack_order = midx_pack_order(&ctx);
+
+	if (flags & MIDX_WRITE_REV_INDEX)
+		write_midx_reverse_index(midx_name, midx_hash, &ctx);
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+
 	commit_lock_file(&lk);
 
 cleanup:
@@ -1015,6 +1089,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	free(ctx.info);
 	free(ctx.entries);
 	free(ctx.pack_perm);
+	free(ctx.pack_order);
 	free(midx_name);
 	return result;
 }
@@ -1027,6 +1102,44 @@ int write_midx_file(const char *object_dir,
 				   flags);
 }
 
+struct clear_midx_data {
+	char *keep;
+	const char *ext;
+};
+
+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,
+				const char *file_name, void *_data)
+{
+	struct clear_midx_data *data = _data;
+
+	if (!(starts_with(file_name, "multi-pack-index-") &&
+	      ends_with(file_name, data->ext)))
+		return;
+	if (data->keep && !strcmp(data->keep, file_name))
+		return;
+
+	if (unlink(full_path))
+		die_errno(_("failed to remove %s"), full_path);
+}
+
+static void clear_midx_files_ext(struct repository *r, const char *ext,
+				 unsigned char *keep_hash)
+{
+	struct clear_midx_data data;
+	memset(&data, 0, sizeof(struct clear_midx_data));
+
+	if (keep_hash)
+		data.keep = xstrfmt("multi-pack-index-%s%s",
+				    hash_to_hex(keep_hash), ext);
+	data.ext = ext;
+
+	for_each_file_in_pack_dir(r->objects->odb->path,
+				  clear_midx_file_ext,
+				  &data);
+
+	free(data.keep);
+}
+
 void clear_midx_file(struct repository *r)
 {
 	char *midx = get_midx_filename(r->objects->odb->path);
@@ -1039,6 +1152,8 @@ void clear_midx_file(struct repository *r)
 	if (remove_path(midx))
 		die(_("failed to clear multi-pack-index at %s"), midx);
 
+	clear_midx_files_ext(r, ".rev", NULL);
+
 	free(midx);
 }
 
midx.h +1 -0
index 0a8294d2ee..8684cf0fef 100644
@@ -40,6 +40,7 @@ struct multi_pack_index {
 };
 
 #define MIDX_PROGRESS     (1 << 0)
+#define MIDX_WRITE_REV_INDEX (1 << 1)
 
 char *get_midx_rev_filename(struct multi_pack_index *m);
 
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#167 lore raw

PATCH v4 16/16 midx.c: improve cache locality in midx_pack_order_cmp()

In reply to Taylor Blau

From: Jeff King <[email protected]>

There is a lot of pointer dereferencing in the pre-image version of 'midx_pack_order_cmp()', which this patch gets rid of.

Instead of comparing the pack preferred-ness and then the pack id, both of these checks are done at the same time by using the high-order bit of the pack id to represent whether it's preferred. Then the pack id and offset are compared as usual.

This produces the same result so long as there are less than 2^31 packs, which seems like a likely assumption to make in practice.

---
 midx.c | 55 +++++++++++++++++++++++++++++--------------------------
 1 file changed, 29 insertions(+), 26 deletions(-)
midx.c +29 -26
index b96eaa12fb..9e86583172 100644
@@ -818,46 +818,49 @@ static int write_midx_large_offsets(struct hashfile *f,
 	return 0;
 }
 
-static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)
+struct midx_pack_order_data {
+	uint32_t nr;
+	uint32_t pack;
+	off_t offset;
+};
+
+static int midx_pack_order_cmp(const void *va, const void *vb)
 {
-	struct write_midx_context *ctx = _ctx;
-
-	struct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];
-	struct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];
-
-	uint32_t perm_a = ctx->pack_perm[a->pack_int_id];
-	uint32_t perm_b = ctx->pack_perm[b->pack_int_id];
-
-	/* Sort objects in the preferred pack ahead of any others. */
-	if (a->preferred > b->preferred)
+	const struct midx_pack_order_data *a = va, *b = vb;
+	if (a->pack < b->pack)
 		return -1;
-	if (a->preferred < b->preferred)
+	else if (a->pack > b->pack)
 		return 1;
-
-	/* Then, order objects by which packs they appear in. */
-	if (perm_a < perm_b)
+	else if (a->offset < b->offset)
 		return -1;
-	if (perm_a > perm_b)
+	else if (a->offset > b->offset)
 		return 1;
-
-	/* Then, disambiguate by their offset within each pack. */
-	if (a->offset < b->offset)
-		return -1;
-	if (a->offset > b->offset)
-		return 1;
-
-	return 0;
+	else
+		return 0;
 }
 
 static uint32_t *midx_pack_order(struct write_midx_context *ctx)
 {
+	struct midx_pack_order_data *data;
 	uint32_t *pack_order;
 	uint32_t i;
 
+	ALLOC_ARRAY(data, ctx->entries_nr);
+	for (i = 0; i < ctx->entries_nr; i++) {
+		struct pack_midx_entry *e = &ctx->entries[i];
+		data[i].nr = i;
+		data[i].pack = ctx->pack_perm[e->pack_int_id];
+		if (!e->preferred)
+			data[i].pack |= (1U << 31);
+		data[i].offset = e->offset;
+	}
+
+	QSORT(data, ctx->entries_nr, midx_pack_order_cmp);
+
 	ALLOC_ARRAY(pack_order, ctx->entries_nr);
 	for (i = 0; i < ctx->entries_nr; i++)
-		pack_order[i] = i;
-	QSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);
+		pack_order[i] = data[i].nr;
+	free(data);
 
 	return pack_order;
 }
Signature · 1 line
2.30.0.667.g81c0cbc6fd
#168 lore raw
In reply to Taylor Blau
Quoted 2 lines · Taylor Blau

Here is another reroll of my series to implement a reverse index in preparation for multi-pack reachability bitmaps.

Thanks, this addresses all of my comments from the last round.

Quoted 4 lines

This reroll differs only in the feedback I incorporated from Peff's review. They are mostly cosmetic; the most substantial change being that the --preferred-pack code now uses bsearch() to locate the name of the preferred pack (instead of implementing a binary search itself).

Yeah, I read over this part carefully, since it's actual new code (that isn't run yet!), but I think it is correct.

One minor observation:

Quoted 9 lines
    @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'
     +			write --preferred-pack=test-BC-$bc.idx 2>err &&
     +		test_must_be_empty err &&
     +
    ++		echo hi &&
    ++		test-tool read-midx --show-objects objects >out &&
    ++
     +		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |
     +			cut -d" " -f1) &&
Quoted 8 lines
    -+		midx_expect_object_offset $b $ofs objects
    ++		printf "%s %s\tobjects/pack/test-BC-%s.pack\n" \
    ++			"$b" "$ofs" "$bc" >expect &&
    ++		grep ^$b out >actual &&
    ++
    ++		test_cmp expect actual
     +	)
     +'

I'd probably have just skipped show-index entirely, and done:

grep "^$b .* objects/pack/test-BC" actual

which expresses the intent ($b came from that pack). But I don't mind the more exacting version (and certainly it is not worth a re-roll even if you prefer mine).

-Peff

#169 lore raw
Quoted 7 lines · Jeff King
Quoted 4 lines

This reroll differs only in the feedback I incorporated from Peff's review. They are mostly cosmetic; the most substantial change being that the --preferred-pack code now uses bsearch() to locate the name of the preferred pack (instead of implementing a binary search itself).

Yeah, I read over this part carefully, since it's actual new code (that isn't run yet!), but I think it is correct.

Thankfully this does have coverage via any test that passes --preferred-pack (like the one below).

Quoted 20 lines

One minor observation:

Quoted 9 lines
    @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'
     +			write --preferred-pack=test-BC-$bc.idx 2>err &&
     +		test_must_be_empty err &&
     +
    ++		echo hi &&
    ++		test-tool read-midx --show-objects objects >out &&
    ++
     +		ofs=$(git show-index <objects/pack/test-BC-$bc.idx | grep $b |
     +			cut -d" " -f1) &&
Quoted 8 lines
    -+		midx_expect_object_offset $b $ofs objects
    ++		printf "%s %s\tobjects/pack/test-BC-%s.pack\n" \
    ++			"$b" "$ofs" "$bc" >expect &&
    ++		grep ^$b out >actual &&
    ++
    ++		test_cmp expect actual
     +	)
     +'
Quoted 7 lines

I'd probably have just skipped show-index entirely, and done:

grep "^$b .* objects/pack/test-BC" actual

which expresses the intent ($b came from that pack). But I don't mind the more exacting version (and certainly it is not worth a re-roll even if you prefer mine).

I originally wrote it that way, but decided to write both expect and actual to make debugging easier if this ever regresses. Not like it's that hard to run the test-tool yourself in the trash directory, but having a snapshot of that object from the MIDX's perspective might make things a little easier.

Anyway, I agree with you that it doesn't probably matter a ton either way.

Quoted 1 line

-Peff

Thanks,
Taylor

#170 lore raw
Quoted 11 lines · Taylor Blau
Quoted 7 lines · Jeff King
Quoted 4 lines

This reroll differs only in the feedback I incorporated from Peff's review. They are mostly cosmetic; the most substantial change being that the --preferred-pack code now uses bsearch() to locate the name of the preferred pack (instead of implementing a binary search itself).

Yeah, I read over this part carefully, since it's actual new code (that isn't run yet!), but I think it is correct.

Thankfully this does have coverage via any test that passes --preferred-pack (like the one below).

Oh right, I forgot this was touching that early part. So now I'm doubly confident in it.

-Peff