Skip to contentSkip to conversation

multi-pack reachability bitmaps

PATCH Superseded Taylor Blau 8 participants 273 messages Apr 9, 2021 – Sep 2, 2021 Documentation Page 2 of 2

lore
Share & tools
lore
#144 lore raw

Re: [PATCH v3 09/25] midx: avoid opening multiple MIDXs when writing

In reply to Jeff King
Quoted 19 lines · Jeff King

I think we'd need something like this:

midx.c +1 -2
index 6dfafe7a8c..bfb6afea2e 100644
@@ -1123,8 +1123,7 @@ static int write_midx_internal(const char *object_dir,
 	hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
 
-	if (ctx.m)
-		close_midx(ctx.m);
+	close_object_store(the_repository->objects);
 
 	if (ctx.nr - dropped_packs == 0) {
 		error(_("no pack files to index."));

though I'm not sure:

Quoted 8 lines

- if this should be unconditional or dependent on ctx.m (I think the latter, because if we are renaming over any open midx, we would have filled in ctx.m earlier).

- if this should go below the "no pack files to index" check (i.e., is there any point in closing if we know we will not write?). In fact, its purpose might be more obvious right before finalize_hashfile(), but I am OK either way on that.

Ah, this close_midx() actually gets moved and made unconditional later in the series. But it still needs to be close_object_store() instead.

Also, my mention of finalize_hashfile() is wrong. It's commit_lock_file() that does the actual rename, and indeed, that's where you moved it to in the end, which is good.

-Peff

Quoted 2 lines

-Peff

#145 lore raw

Re: [PATCH v3 09/25] midx: avoid opening multiple MIDXs when writing

Quoted 25 lines · Jeff King
Quoted 23 lines · Jeff King

I think we'd need something like this:

midx.c +1 -2
index 6dfafe7a8c..bfb6afea2e 100644
@@ -1123,8 +1123,7 @@ static int write_midx_internal(const char *object_dir,
 	hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
 
-	if (ctx.m)
-		close_midx(ctx.m);
+	close_object_store(the_repository->objects);
 
 	if (ctx.nr - dropped_packs == 0) {
 		error(_("no pack files to index."));

though I'm not sure:

- if this should be unconditional or dependent on ctx.m (I think the latter, because if we are renaming over any open midx, we would have filled in ctx.m earlier).

Quoted 7 lines
Quoted 4 lines

- if this should go below the "no pack files to index" check (i.e., is there any point in closing if we know we will not write?). In fact, its purpose might be more obvious right before finalize_hashfile(), but I am OK either way on that.

Ah, this close_midx() actually gets moved and made unconditional later in the series. But it still needs to be close_object_store() instead.

Exactly; this first patch should read:

    if (ctx.m)
      close_object_store(the_repository->objects);

and then the latter patch (15/25) we drop the conditional and move our call down until after the MIDX bitmap is written, but before we call commit_lock_file().

Thanks,
Taylor

#146 lore raw

PATCH v3 10/25 pack-bitmap.c: introduce 'bitmap_num_objects()'

In reply to Taylor Blau

A subsequent patch to support reading MIDX bitmaps will be less noisy after extracting a generic function to return how many objects are contained in a bitmap.

---
 pack-bitmap.c | 37 +++++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 16 deletions(-)
pack-bitmap.c +21 -16
index a73960a55d..54dc4f7915 100644
@@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)
 	return b;
 }
 
+static uint32_t bitmap_num_objects(struct bitmap_index *index)
+{
+	return index->pack->num_objects;
+}
+
 static int load_bitmap_header(struct bitmap_index *index)
 {
 	struct bitmap_disk_header *header = (void *)index->map;
@@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index)
 	/* Parse known bitmap format options */
 	{
 		uint32_t flags = ntohs(header->options);
-		size_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));
+		size_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t));
 		unsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;
 
 		if ((flags & BITMAP_OPT_FULL_DAG) == 0)
@@ -399,7 +404,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,
 
 	if (pos < kh_end(positions)) {
 		int bitmap_pos = kh_value(positions, pos);
-		return bitmap_pos + bitmap_git->pack->num_objects;
+		return bitmap_pos + bitmap_num_objects(bitmap_git);
 	}
 
 	return -1;
@@ -451,7 +456,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,
 		bitmap_pos = kh_value(eindex->positions, hash_pos);
 	}
 
-	return bitmap_pos + bitmap_git->pack->num_objects;
+	return bitmap_pos + bitmap_num_objects(bitmap_git);
 }
 
 struct bitmap_show_data {
@@ -668,7 +673,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,
 	for (i = 0; i < eindex->count; ++i) {
 		struct object *obj;
 
-		if (!bitmap_get(objects, bitmap_git->pack->num_objects + i))
+		if (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))
 			continue;
 
 		obj = eindex->objects[i];
@@ -826,7 +831,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,
 	 * individually.
 	 */
 	for (i = 0; i < eindex->count; i++) {
-		uint32_t pos = i + bitmap_git->pack->num_objects;
+		uint32_t pos = i + bitmap_num_objects(bitmap_git);
 		if (eindex->objects[i]->type == type &&
 		    bitmap_get(to_filter, pos) &&
 		    !bitmap_get(tips, pos))
@@ -853,7 +858,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 
 	oi.sizep = &size;
 
-	if (pos < pack->num_objects) {
+	if (pos < bitmap_num_objects(bitmap_git)) {
 		off_t ofs = pack_pos_to_offset(pack, pos);
 		if (packed_object_info(the_repository, pack, ofs, &oi) < 0) {
 			struct object_id oid;
@@ -863,7 +868,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 		}
 	} else {
 		struct eindex *eindex = &bitmap_git->ext_index;
-		struct object *obj = eindex->objects[pos - pack->num_objects];
+		struct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];
 		if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)
 			die(_("unable to get size of %s"), oid_to_hex(&obj->oid));
 	}
@@ -905,7 +910,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,
 	}
 
 	for (i = 0; i < eindex->count; i++) {
-		uint32_t pos = i + bitmap_git->pack->num_objects;
+		uint32_t pos = i + bitmap_num_objects(bitmap_git);
 		if (eindex->objects[i]->type == OBJ_BLOB &&
 		    bitmap_get(to_filter, pos) &&
 		    !bitmap_get(tips, pos) &&
@@ -1131,8 +1136,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 	enum object_type type;
 	unsigned long size;
 
-	if (pos >= bitmap_git->pack->num_objects)
-		return; /* not actually in the pack */
+	if (pos >= bitmap_num_objects(bitmap_git))
+		return; /* not actually in the pack or MIDX */
 
 	offset = header = pack_pos_to_offset(bitmap_git->pack, pos);
 	type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);
@@ -1198,6 +1203,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 	struct pack_window *w_curs = NULL;
 	size_t i = 0;
 	uint32_t offset;
+	uint32_t objects_nr = bitmap_num_objects(bitmap_git);
 
 	assert(result);
 
@@ -1205,8 +1211,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 		i++;
 
 	/* Don't mark objects not in the packfile */
-	if (i > bitmap_git->pack->num_objects / BITS_IN_EWORD)
-		i = bitmap_git->pack->num_objects / BITS_IN_EWORD;
+	if (i > objects_nr / BITS_IN_EWORD)
+		i = objects_nr / BITS_IN_EWORD;
 
 	reuse = bitmap_word_alloc(i);
 	memset(reuse->words, 0xFF, i * sizeof(eword_t));
@@ -1290,7 +1296,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,
 
 	for (i = 0; i < eindex->count; ++i) {
 		if (eindex->objects[i]->type == type &&
-			bitmap_get(objects, bitmap_git->pack->num_objects + i))
+			bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))
 			count++;
 	}
 
@@ -1511,7 +1517,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,
 	uint32_t i, num_objects;
 	uint32_t *reposition;
 
-	num_objects = bitmap_git->pack->num_objects;
+	num_objects = bitmap_num_objects(bitmap_git);
 	CALLOC_ARRAY(reposition, num_objects);
 
 	for (i = 0; i < num_objects; ++i) {
@@ -1594,7 +1600,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,
 static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)
 {
 	struct bitmap *result = bitmap_git->result;
-	struct packed_git *pack = bitmap_git->pack;
 	struct eindex *eindex = &bitmap_git->ext_index;
 	off_t total = 0;
 	struct object_info oi = OBJECT_INFO_INIT;
@@ -1606,7 +1611,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)
 	for (i = 0; i < eindex->count; i++) {
 		struct object *obj = eindex->objects[i];
 
-		if (!bitmap_get(result, pack->num_objects + i))
+		if (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i))
 			continue;
 
 		if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)
Signature · 1 line
2.31.1.163.ga65ce7f831
#147 lore raw

PATCH v3 11/25 pack-bitmap.c: introduce 'nth_bitmap_object_oid()'

In reply to Taylor Blau

A subsequent patch to support reading MIDX bitmaps will be less noisy after extracting a generic function to fetch the nth OID contained in the bitmap.

---
 pack-bitmap.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
pack-bitmap.c +10 -3
index 54dc4f7915..9d0dd1cde7 100644
@@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)
 
 #define MAX_XOR_OFFSET 160
 
+static int nth_bitmap_object_oid(struct bitmap_index *index,
+				 struct object_id *oid,
+				 uint32_t n)
+{
+	return nth_packed_object_id(oid, index->pack, n);
+}
+
 static int load_bitmap_entries_v1(struct bitmap_index *index)
 {
 	uint32_t i;
@@ -242,7 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
 		xor_offset = read_u8(index->map, &index->map_pos);
 		flags = read_u8(index->map, &index->map_pos);
 
-		if (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)
+		if (nth_bitmap_object_oid(index, &oid, commit_idx_pos) < 0)
 			return error("corrupt ewah bitmap: commit index %u out of range",
 				     (unsigned)commit_idx_pos);
 
@@ -862,8 +869,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 		off_t ofs = pack_pos_to_offset(pack, pos);
 		if (packed_object_info(the_repository, pack, ofs, &oi) < 0) {
 			struct object_id oid;
-			nth_packed_object_id(&oid, pack,
-					     pack_pos_to_index(pack, pos));
+			nth_bitmap_object_oid(bitmap_git, &oid,
+					      pack_pos_to_index(pack, pos));
 			die(_("unable to get size of %s"), oid_to_hex(&oid));
 		}
 	} else {
Signature · 1 line
2.31.1.163.ga65ce7f831
#148 lore raw

PATCH v3 12/25 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'

In reply to Taylor Blau

In a recent commit, pack-objects learned support for the 'pack.preferBitmapTips' configuration. This patch prepares the multi-pack bitmap code to respect this configuration, too.

The yet-to-be implemented code will find that it is more efficient to check whether each reference contains a prefix found in the configured set of values rather than doing an additional traversal.

Implement a function 'bitmap_is_preferred_refname()' which will perform that check. Its caller will be added in a subsequent patch.

---
 pack-bitmap.c | 16 ++++++++++++++++
 pack-bitmap.h |  1 +
 2 files changed, 17 insertions(+)
pack-bitmap.c +16 -0
index 9d0dd1cde7..6b12c96e32 100644
@@ -1652,3 +1652,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r)
 {
 	return repo_config_get_value_multi(r, "pack.preferbitmaptips");
 }
+
+int bitmap_is_preferred_refname(struct repository *r, const char *refname)
+{
+	const struct string_list *preferred_tips = bitmap_preferred_tips(r);
+	struct string_list_item *item;
+
+	if (!preferred_tips)
+		return 0;
+
+	for_each_string_list_item(item, preferred_tips) {
+		if (starts_with(refname, item->string))
+			return 1;
+	}
+
+	return 0;
+}
pack-bitmap.h +1 -0
index 020cd8d868..52ea10de51 100644
@@ -94,5 +94,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint16_t options);
 
 const struct string_list *bitmap_preferred_tips(struct repository *r);
+int bitmap_is_preferred_refname(struct repository *r, const char *refname);
 
 #endif
Signature · 1 line
2.31.1.163.ga65ce7f831
#149 lore raw

PATCH v3 13/25 pack-bitmap.c: avoid redundant calls to try_partial_reuse

In reply to Taylor Blau

try_partial_reuse() is used to mark any bits in the beginning of a bitmap whose objects can be reused verbatim from the pack they came from.

Currently this function returns void, and signals nothing to the caller when bits could not be reused. But multi-pack bitmaps would benefit from having such a signal, because they may try to pass objects which are in bounds, but from a pack other than the preferred one.

Any extra calls are noops because of a conditional in reuse_partial_packfile_from_bitmap(), but those loop iterations can be avoided by letting try_partial_reuse() indicate when it can't accept any more bits for reuse, and then listening to that signal.

---
 pack-bitmap.c | 40 +++++++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 11 deletions(-)
pack-bitmap.c +29 -11
index 6b12c96e32..1442f0c8f2 100644
@@ -1134,22 +1134,26 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,
 	return NULL;
 }
 
-static void try_partial_reuse(struct bitmap_index *bitmap_git,
-			      size_t pos,
-			      struct bitmap *reuse,
-			      struct pack_window **w_curs)
+/*
+ * -1 means "stop trying further objects"; 0 means we may or may not have
+ * reused, but you can keep feeding bits.
+ */
+static int try_partial_reuse(struct bitmap_index *bitmap_git,
+			     size_t pos,
+			     struct bitmap *reuse,
+			     struct pack_window **w_curs)
 {
 	off_t offset, header;
 	enum object_type type;
 	unsigned long size;
 
 	if (pos >= bitmap_num_objects(bitmap_git))
-		return; /* not actually in the pack or MIDX */
+		return -1; /* not actually in the pack or MIDX */
 
 	offset = header = pack_pos_to_offset(bitmap_git->pack, pos);
 	type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);
 	if (type < 0)
-		return; /* broken packfile, punt */
+		return -1; /* broken packfile, punt */
 
 	if (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {
 		off_t base_offset;
@@ -1166,9 +1170,9 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		base_offset = get_delta_base(bitmap_git->pack, w_curs,
 					     &offset, type, header);
 		if (!base_offset)
-			return;
+			return 0;
 		if (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)
-			return;
+			return 0;
 
 		/*
 		 * We assume delta dependencies always point backwards. This
@@ -1180,7 +1184,7 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		 * odd parameters.
 		 */
 		if (base_pos >= pos)
-			return;
+			return 0;
 
 		/*
 		 * And finally, if we're not sending the base as part of our
@@ -1191,13 +1195,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		 * object_entry code path handle it.
 		 */
 		if (!bitmap_get(reuse, base_pos))
-			return;
+			return 0;
 	}
 
 	/*
 	 * If we got here, then the object is OK to reuse. Mark it.
 	 */
 	bitmap_set(reuse, pos);
+	return 0;
 }
 
 int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
@@ -1233,10 +1238,23 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 				break;
 
 			offset += ewah_bit_ctz64(word >> offset);
-			try_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);
+			if (try_partial_reuse(bitmap_git, pos + offset, reuse,
+					      &w_curs) < 0) {
+				/*
+				 * try_partial_reuse indicated we couldn't reuse
+				 * any bits, so there is no point in trying more
+				 * bits in the current word, or any other words
+				 * in result.
+				 *
+				 * Jump out of both loops to avoid future
+				 * unnecessary calls to try_partial_reuse.
+				 */
+				goto done;
+			}
 		}
 	}
 
+done:
 	unuse_pack(&w_curs);
 
 	*entries = bitmap_popcount(reuse);
Signature · 1 line
2.31.1.163.ga65ce7f831
#150 lore raw

PATCH v3 14/25 pack-bitmap: read multi-pack bitmaps

In reply to Taylor Blau

This prepares the code in pack-bitmap to interpret the new multi-pack bitmaps described in Documentation/technical/bitmap-format.txt, which mostly involves converting bit positions to accommodate looking them up in a MIDX.

Note that there are currently no writers who write multi-pack bitmaps, and that this will be implemented in the subsequent commit. Note also that get_midx_checksum() and get_midx_filename() are made non-static so they can be called from pack-bitmap.c.

---
 builtin/pack-objects.c |   5 +
 midx.c                 |   4 +-
 midx.h                 |   2 +
 pack-bitmap-write.c    |   2 +-
 pack-bitmap.c          | 357 ++++++++++++++++++++++++++++++++++++-----
 pack-bitmap.h          |   6 +
 packfile.c             |   2 +-
 7 files changed, 336 insertions(+), 42 deletions(-)
builtin/pack-objects.c +5 -0
index 8a523624a1..e11d3ac2e5 100644
@@ -1124,6 +1124,11 @@ static void write_reused_pack(struct hashfile *f)
 				break;
 
 			offset += ewah_bit_ctz64(word >> offset);
+			/*
+			 * Can use bit positions directly, even for MIDX
+			 * bitmaps. See comment in try_partial_reuse()
+			 * for why.
+			 */
 			write_reused_pack_one(pos + offset, f, &w_curs);
 			display_progress(progress_state, ++written);
 		}
midx.c +2 -2
index d67d7f383d..db21727c62 100644
@@ -48,12 +48,12 @@ static uint8_t oid_version(void)
 	}
 }
 
-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)
+const unsigned char *get_midx_checksum(struct multi_pack_index *m)
 {
 	return m->data + m->data_len - the_hash_algo->rawsz;
 }
 
-static char *get_midx_filename(const char *object_dir)
+char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }
midx.h +2 -0
index 8684cf0fef..1172df1a71 100644
@@ -42,6 +42,8 @@ struct multi_pack_index {
 #define MIDX_PROGRESS     (1 << 0)
 #define MIDX_WRITE_REV_INDEX (1 << 1)
 
+const unsigned char *get_midx_checksum(struct multi_pack_index *m);
+char *get_midx_filename(const char *object_dir);
 char *get_midx_rev_filename(struct multi_pack_index *m);
 
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
pack-bitmap-write.c +1 -1
index 142fd0adb8..9c55c1531e 100644
@@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show)
 }
 
 /**
- * Build the initial type index for the packfile
+ * Build the initial type index for the packfile or multi-pack-index
  */
 void bitmap_writer_build_type_index(struct packing_data *to_pack,
 				    struct pack_idx_entry **index,
pack-bitmap.c +319 -38

584 lines not on this page · raw message

pack-bitmap.h +6 -0
index 52ea10de51..81664f933f 100644
@@ -44,6 +44,8 @@ typedef int (*show_reachable_fn)(
 struct bitmap_index;
 
 struct bitmap_index *prepare_bitmap_git(struct repository *r);
+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,
+					     struct multi_pack_index *midx);
 void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,
 			      uint32_t *trees, uint32_t *blobs, uint32_t *tags);
 void traverse_bitmap_commit_list(struct bitmap_index *,
@@ -92,6 +94,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint32_t index_nr,
 			  const char *filename,
 			  uint16_t options);
+char *midx_bitmap_filename(struct multi_pack_index *midx);
+char *pack_bitmap_filename(struct packed_git *p);
+
+int bitmap_is_midx(struct bitmap_index *bitmap_git);
 
 const struct string_list *bitmap_preferred_tips(struct repository *r);
 int bitmap_is_preferred_refname(struct repository *r, const char *refname);
packfile.c +1 -1
index 9ef6d98292..371f5488cf 100644
@@ -860,7 +860,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 	if (!strcmp(file_name, "multi-pack-index"))
 		return;
 	if (starts_with(file_name, "multi-pack-index") &&
-	    ends_with(file_name, ".rev"))
+	    (ends_with(file_name, ".bitmap") || ends_with(file_name, ".rev")))
 		return;
 	if (ends_with(file_name, ".idx") ||
 	    ends_with(file_name, ".rev") ||
Signature · 1 line
2.31.1.163.ga65ce7f831
#151 lore raw

PATCH v3 15/25 pack-bitmap: write multi-pack bitmaps

In reply to Taylor Blau

Write multi-pack bitmaps in the format described by Documentation/technical/bitmap-format.txt, inferring their presence with the absence of '--bitmap'.

To write a multi-pack bitmap, this patch attempts to reuse as much of the existing machinery from pack-objects as possible. Specifically, the MIDX code prepares a packing_data struct that pretends as if a single packfile has been generated containing all of the objects contained within the MIDX.

---
 Documentation/git-multi-pack-index.txt |  12 +-
 builtin/multi-pack-index.c             |   2 +
 midx.c                                 | 208 ++++++++++++++++++++++++-
 midx.h                                 |   1 +
 4 files changed, 214 insertions(+), 9 deletions(-)
Documentation/git-multi-pack-index.txt +11 -1
index c9b063d31e..ed52459a9d 100644
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [verse]
 'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
-	[--preferred-pack=<pack>] <subcommand>
+	[--preferred-pack=<pack>] [--[no-]bitmap] <subcommand>
 
 DESCRIPTION
 -----------
@@ -40,6 +40,9 @@ write::
 		multiple packs contain the same object. `<pack>` must
 		contain at least one object. If not given, ties are
 		broken in favor of the pack with the lowest mtime.
+
+	--[no-]bitmap::
+		Control whether or not a multi-pack bitmap is written.
 --
 
 verify::
@@ -81,6 +84,13 @@ EXAMPLES
 $ git multi-pack-index write
 -----------------------------------------------
 
+* Write a MIDX file for the packfiles in the current .git folder with a
+corresponding bitmap.
++
+-------------------------------------------------------------
+$ git multi-pack-index write --preferred-pack=<pack> --bitmap
+-------------------------------------------------------------
+
 * Write a MIDX file for the packfiles in an alternate object store.
 +
 -----------------------------------------------
builtin/multi-pack-index.c +2 -0
index 5d3ea445fd..bf6fa982e3 100644
@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
 			   N_("preferred-pack"),
 			   N_("pack for reuse when computing a multi-pack bitmap")),
+		OPT_BIT(0, "bitmap", &opts.flags, N_("write multi-pack bitmap"),
+			MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),
 		OPT_END(),
 	};
 
midx.c +200 -8

269 lines not on this page · raw message

midx.h +1 -0
index 1172df1a71..350f4d0a7b 100644
@@ -41,6 +41,7 @@ struct multi_pack_index {
 
 #define MIDX_PROGRESS     (1 << 0)
 #define MIDX_WRITE_REV_INDEX (1 << 1)
+#define MIDX_WRITE_BITMAP (1 << 2)
 
 const unsigned char *get_midx_checksum(struct multi_pack_index *m);
 char *get_midx_filename(const char *object_dir);
Signature · 1 line
2.31.1.163.ga65ce7f831
#152 lore raw

PATCH v3 16/25 t5310: move some tests to lib-bitmap.sh

In reply to Taylor Blau

We'll soon be adding a test script that will cover many of the same bitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many of the applicable tests as we can so we don't have to rewrite them.

There should be no functional change to t5310; we still run the same operations in the same order.

---
 t/lib-bitmap.sh         | 236 ++++++++++++++++++++++++++++++++++++++++
 t/t5310-pack-bitmaps.sh | 227 +-------------------------------------
 2 files changed, 240 insertions(+), 223 deletions(-)
t/lib-bitmap.sh +236 -0
index fe3f98be24..ecb5d0e05d 100644
@@ -1,3 +1,6 @@
+# Helpers for scripts testing bitamp functionality; see t5310 for
+# example usage.
+
 # Compare a file containing rev-list bitmap traversal output to its non-bitmap
 # counterpart. You can't just use test_cmp for this, because the two produce
 # subtly different output:
@@ -24,3 +27,236 @@ test_bitmap_traversal () {
 	test_cmp "$1.normalized" "$2.normalized" &&
 	rm -f "$1.normalized" "$2.normalized"
 }
+
+# To ensure the logic for "maximal commits" is exercised, make
+# the repository a bit more complicated.
+#
+#    other                         second
+#      *                             *
+# (99 commits)                  (99 commits)
+#      *                             *
+#      |\                           /|
+#      | * octo-other  octo-second * |
+#      |/|\_________  ____________/|\|
+#      | \          \/  __________/  |
+#      |  | ________/\ /             |
+#      *  |/          * merge-right  *
+#      | _|__________/ \____________ |
+#      |/ |                         \|
+# (l1) *  * merge-left               * (r1)
+#      | / \________________________ |
+#      |/                           \|
+# (l2) *                             * (r2)
+#       \___________________________ |
+#                                   \|
+#                                    * (base)
+#
+# We only push bits down the first-parent history, which
+# makes some of these commits unimportant!
+#
+# The important part for the maximal commit algorithm is how
+# the bitmasks are extended. Assuming starting bit positions
+# for second (bit 0) and other (bit 1), the bitmasks at the
+# end should be:
+#
+#      second: 1       (maximal, selected)
+#       other: 01      (maximal, selected)
+#      (base): 11 (maximal)
+#
+# This complicated history was important for a previous
+# version of the walk that guarantees never walking a
+# commit multiple times. That goal might be important
+# again, so preserve this complicated case. For now, this
+# test will guarantee that the bitmaps are computed
+# correctly, even with the repeat calculations.
+setup_bitmap_history() {
+	test_expect_success 'setup repo with moderate-sized history' '
+		test_commit_bulk --id=file 10 &&
+		git branch -M second &&
+		git checkout -b other HEAD~5 &&
+		test_commit_bulk --id=side 10 &&
+
+		# add complicated history setup, including merges and
+		# ambiguous merge-bases
+
+		git checkout -b merge-left other~2 &&
+		git merge second~2 -m "merge-left" &&
+
+		git checkout -b merge-right second~1 &&
+		git merge other~1 -m "merge-right" &&
+
+		git checkout -b octo-second second &&
+		git merge merge-left merge-right -m "octopus-second" &&
+
+		git checkout -b octo-other other &&
+		git merge merge-left merge-right -m "octopus-other" &&
+
+		git checkout other &&
+		git merge octo-other -m "pull octopus" &&
+
+		git checkout second &&
+		git merge octo-second -m "pull octopus" &&
+
+		# Remove these branches so they are not selected
+		# as bitmap tips
+		git branch -D merge-left &&
+		git branch -D merge-right &&
+		git branch -D octo-other &&
+		git branch -D octo-second &&
+
+		# add padding to make these merges less interesting
+		# and avoid having them selected for bitmaps
+		test_commit_bulk --id=file 100 &&
+		git checkout other &&
+		test_commit_bulk --id=side 100 &&
+		git checkout second &&
+
+		bitmaptip=$(git rev-parse second) &&
+		blob=$(echo tagged-blob | git hash-object -w --stdin) &&
+		git tag tagged-blob $blob
+	'
+}
+
+rev_list_tests_head () {
+	test_expect_success "counting commits via bitmap ($state, $branch)" '
+		git rev-list --count $branch >expect &&
+		git rev-list --use-bitmap-index --count $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting partial commits via bitmap ($state, $branch)" '
+		git rev-list --count $branch~5..$branch >expect &&
+		git rev-list --use-bitmap-index --count $branch~5..$branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting commits with limit ($state, $branch)" '
+		git rev-list --count -n 1 $branch >expect &&
+		git rev-list --use-bitmap-index --count -n 1 $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting non-linear history ($state, $branch)" '
+		git rev-list --count other...second >expect &&
+		git rev-list --use-bitmap-index --count other...second >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting commits with limiting ($state, $branch)" '
+		git rev-list --count $branch -- 1.t >expect &&
+		git rev-list --use-bitmap-index --count $branch -- 1.t >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting objects via bitmap ($state, $branch)" '
+		git rev-list --count --objects $branch >expect &&
+		git rev-list --use-bitmap-index --count --objects $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "enumerate commits ($state, $branch)" '
+		git rev-list --use-bitmap-index $branch >actual &&
+		git rev-list $branch >expect &&
+		test_bitmap_traversal --no-confirm-bitmaps expect actual
+	'
+
+	test_expect_success "enumerate --objects ($state, $branch)" '
+		git rev-list --objects --use-bitmap-index $branch >actual &&
+		git rev-list --objects $branch >expect &&
+		test_bitmap_traversal expect actual
+	'
+
+	test_expect_success "bitmap --objects handles non-commit objects ($state, $branch)" '
+		git rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&
+		grep $blob actual
+	'
+}
+
+rev_list_tests () {
+	state=$1
+
+	for branch in "second" "other"
+	do
+		rev_list_tests_head
+	done
+}
+
+basic_bitmap_tests () {
+	tip="$1"
+	test_expect_success 'rev-list --test-bitmap verifies bitmaps' "
+		git rev-list --test-bitmap "${tip:-HEAD}"
+	"
+
+	rev_list_tests 'full bitmap'
+
+	test_expect_success 'clone from bitmapped repository' '
+		rm -fr clone.git &&
+		git clone --no-local --bare . clone.git &&
+		git rev-parse HEAD >expect &&
+		git --git-dir=clone.git rev-parse HEAD >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success 'partial clone from bitmapped repository' '
+		test_config uploadpack.allowfilter true &&
+		rm -fr partial-clone.git &&
+		git clone --no-local --bare --filter=blob:none . partial-clone.git &&
+		(
+			cd partial-clone.git &&
+			pack=$(echo objects/pack/*.pack) &&
+			git verify-pack -v "$pack" >have &&
+			awk "/blob/ { print \$1 }" <have >blobs &&
+			# we expect this single blob because of the direct ref
+			git rev-parse refs/tags/tagged-blob >expect &&
+			test_cmp expect blobs
+		)
+	'
+
+	test_expect_success 'setup further non-bitmapped commits' '
+		test_commit_bulk --id=further 10
+	'
+
+	rev_list_tests 'partial bitmap'
+
+	test_expect_success 'fetch (partial bitmap)' '
+		git --git-dir=clone.git fetch origin second:second &&
+		git rev-parse HEAD >expect &&
+		git --git-dir=clone.git rev-parse HEAD >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success 'enumerating progress counts pack-reused objects' '
+		count=$(git rev-list --objects --all --count) &&
+		git repack -adb &&
+
+		# check first with only reused objects; confirm that our
+		# progress showed the right number, and also that we did
+		# pack-reuse as expected.  Check only the final "done"
+		# line of the meter (there may be an arbitrary number of
+		# intermediate lines ending with CR).
+		GIT_PROGRESS_DELAY=0 \
+			git pack-objects --all --stdout --progress \
+			</dev/null >/dev/null 2>stderr &&
+		grep "Enumerating objects: $count, done" stderr &&
+		grep "pack-reused $count" stderr &&
+
+		# now the same but with one non-reused object
+		git commit --allow-empty -m "an extra commit object" &&
+		GIT_PROGRESS_DELAY=0 \
+			git pack-objects --all --stdout --progress \
+			</dev/null >/dev/null 2>stderr &&
+		grep "Enumerating objects: $((count+1)), done" stderr &&
+		grep "pack-reused $count" stderr
+	'
+}
+
+# have_delta <obj> <expected_base>
+#
+# Note that because this relies on cat-file, it might find _any_ copy of an
+# object in the repository. The caller is responsible for making sure
+# there's only one (e.g., via "repack -ad", or having just fetched a copy).
+have_delta () {
+	echo $2 >expect &&
+	echo $1 | git cat-file --batch-check="%(deltabase)" >actual &&
+	test_cmp expect actual
+}
t/t5310-pack-bitmaps.sh +4 -223

246 lines not on this page · raw message

Signature · 1 line
2.31.1.163.ga65ce7f831
#153 lore raw

Re: [PATCH v3 16/25] t5310: move some tests to lib-bitmap.sh

t/lib-bitmap.sh line 2

Quoted 7 lines · Taylor Blau
t/lib-bitmap.sh +2 -0
index fe3f98be24..ecb5d0e05d 100644
@@ -1,3 +1,6 @@
+# Helpers for scripts testing bitamp functionality; see t5310 for
+# example usage.

Bitamp. :) Not worth a re-roll on its own, but I think we'll want one more round to fix the close_object_store() stuff earlier in the series.

The rest of the patch looks good.

-Peff

#154 lore raw

PATCH v3 17/25 t/helper/test-read-midx.c: add --checksum mode

In reply to Taylor Blau

Subsequent tests will want to check for the existence of a multi-pack bitmap which matches the multi-pack-index stored in the pack directory.

The multi-pack bitmap includes the hex checksum of the MIDX it corresponds to in its filename (for example, '$packdir/multi-pack-index-<checksum>.bitmap'). As a result, some tests want a way to learn what '<checksum>' is.

This helper addresses that need by printing the checksum of the repository's multi-pack-index.

---
 t/helper/test-read-midx.c | 16 +++++++++++++++-
 t/lib-bitmap.sh           |  4 ++++
 2 files changed, 19 insertions(+), 1 deletion(-)
t/helper/test-read-midx.c +15 -1
index 7c2eb11a8e..cb0d27049a 100644
@@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects)
 	return 0;
 }
 
+static int read_midx_checksum(const char *object_dir)
+{
+	struct multi_pack_index *m;
+
+	setup_git_directory();
+	m = load_multi_pack_index(object_dir, 1);
+	if (!m)
+		return 1;
+	printf("%s\n", hash_to_hex(get_midx_checksum(m)));
+	return 0;
+}
+
 int cmd__read_midx(int argc, const char **argv)
 {
 	if (!(argc == 2 || argc == 3))
-		usage("read-midx [--show-objects] <object-dir>");
+		usage("read-midx [--show-objects|--checksum] <object-dir>");
 
 	if (!strcmp(argv[1], "--show-objects"))
 		return read_midx_file(argv[2], 1);
+	else if (!strcmp(argv[1], "--checksum"))
+		return read_midx_checksum(argv[2]);
 	return read_midx_file(argv[1], 0);
 }
t/lib-bitmap.sh +4 -0
index ecb5d0e05d..09cd036f4d 100644
@@ -260,3 +260,7 @@ have_delta () {
 	echo $1 | git cat-file --batch-check="%(deltabase)" >actual &&
 	test_cmp expect actual
 }
+
+midx_checksum () {
+	test-tool read-midx --checksum "${1:-.git/objects}"
+}
Signature · 1 line
2.31.1.163.ga65ce7f831
#155 lore raw

Re: [PATCH v3 17/25] t/helper/test-read-midx.c: add --checksum mode

Quoted 10 lines · Taylor Blau

Subsequent tests will want to check for the existence of a multi-pack bitmap which matches the multi-pack-index stored in the pack directory.

The multi-pack bitmap includes the hex checksum of the MIDX it corresponds to in its filename (for example, '$packdir/multi-pack-index-<checksum>.bitmap'). As a result, some tests want a way to learn what '<checksum>' is.

This helper addresses that need by printing the checksum of the repository's multi-pack-index.

Makes sense. It might be nice to have a generic tool for pulling hashes out of checksum files. Perhaps even a tool that is shipped with Git for operating on such files (for in-the-field debugging and diagnosis). But that can definitely be separate from this series (if ever).

Quoted 3 lines
 t/helper/test-read-midx.c | 16 +++++++++++++++-
 t/lib-bitmap.sh           |  4 ++++
 2 files changed, 19 insertions(+), 1 deletion(-)

The patch itself looks fine to me. One curiosity:

t/lib-bitmap.sh have_delta () { · line 266

Quoted 12 lines
t/lib-bitmap.sh +4 -0
index ecb5d0e05d..09cd036f4d 100644
@@ -260,3 +260,7 @@ have_delta () {
 	echo $1 | git cat-file --batch-check="%(deltabase)" >actual &&
 	test_cmp expect actual
 }
+
+midx_checksum () {
+	test-tool read-midx --checksum "${1:-.git/objects}"
+}

This default ".git/objects" will only _usually_ be the right thing. :) If the actual C code accepted a missing object-dir, it could use the correct object directory discovered by setup_git_directory().

Probably not a big deal either way, though.

-Peff

#156 lore raw

Re: [PATCH v3 17/25] t/helper/test-read-midx.c: add --checksum mode

Quoted 10 lines · Jeff King
Quoted 8 lines · Taylor Blau

Subsequent tests will want to check for the existence of a multi-pack bitmap which matches the multi-pack-index stored in the pack directory.

The multi-pack bitmap includes the hex checksum of the MIDX it corresponds to in its filename (for example, '$packdir/multi-pack-index-<checksum>.bitmap'). As a result, some tests want a way to learn what '<checksum>' is.

Quoted 7 lines
Quoted 2 lines

This helper addresses that need by printing the checksum of the repository's multi-pack-index.

Makes sense. It might be nice to have a generic tool for pulling hashes out of checksum files. Perhaps even a tool that is shipped with Git for operating on such files (for in-the-field debugging and diagnosis). But that can definitely be separate from this series (if ever).

Yeah. That would definitely be in the spirit of "we should have more test-tool-like helpers exposed via user-facing plumbing". And I agree that it would be nice, but I definitely agree that it's a topic for a later date ;).

Quoted 23 lines
Quoted 3 lines
 t/helper/test-read-midx.c | 16 +++++++++++++++-
 t/lib-bitmap.sh           |  4 ++++
 2 files changed, 19 insertions(+), 1 deletion(-)

The patch itself looks fine to me. One curiosity:

t/lib-bitmap.sh have_delta () { · line 266

Quoted 12 lines
t/lib-bitmap.sh +4 -0
index ecb5d0e05d..09cd036f4d 100644
@@ -260,3 +260,7 @@ have_delta () {
 	echo $1 | git cat-file --batch-check="%(deltabase)" >actual &&
 	test_cmp expect actual
 }
+
+midx_checksum () {
+	test-tool read-midx --checksum "${1:-.git/objects}"
+}

This default ".git/objects" will only _usually_ be the right thing. :) If the actual C code accepted a missing object-dir, it could use the correct object directory discovered by setup_git_directory().

Quoted 1 line

Probably not a big deal either way, though.

Yeah. We could just sidestep the whole thing by not having the .git/objects default, since all callers of midx_checksum pass an argument, so that fallback is dead code anyway. Thanks for noting, I'll remove it for the next round.

Thanks,
Taylor

#157 lore raw

PATCH v3 18/25 t5326: test multi-pack bitmap behavior

In reply to Taylor Blau

This patch introduces a new test, t5326, which tests the basic functionality of multi-pack bitmaps.

Some trivial behavior is tested, such as:

- Whether bitmaps can be generated with more than one pack.
- Whether clones can be served with all objects in the bitmap.
- Whether follow-up fetches can be served with some objects outside of the server's bitmap

These use lib-bitmap's tests (which in turn were pulled from t5310), and we cover cases where the MIDX represents both a single pack and multiple packs.

In addition, some non-trivial and MIDX-specific behavior is tested, too, including:

- Whether multi-pack bitmaps behave correctly with respect to the pack-reuse machinery when the base for some object is selected from a different pack than the delta.
- Whether multi-pack bitmaps correctly respect the pack.preferBitmapTips configuration.

---
 t/t5326-multi-pack-bitmaps.sh | 277 ++++++++++++++++++++++++++++++++++
 1 file changed, 277 insertions(+)
 create mode 100755 t/t5326-multi-pack-bitmaps.sh
t/t5326-multi-pack-bitmaps.sh new file +277 -0
new file mode 100755
index 0000000000..c1b7d633e2
@@ -0,0 +1,277 @@
+#!/bin/sh
+
+test_description='exercise basic multi-pack bitmap functionality'
+. ./test-lib.sh
+. "${TEST_DIRECTORY}/lib-bitmap.sh"
+
+# We'll be writing our own midx and bitmaps, so avoid getting confused by the
+# automatic ones.
+GIT_TEST_MULTI_PACK_INDEX=0
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
+objdir=.git/objects
+midx=$objdir/pack/multi-pack-index
+
+# midx_pack_source <obj>
+midx_pack_source () {
+	test-tool read-midx --show-objects .git/objects | grep "^$1 " | cut -f2
+}
+
+setup_bitmap_history
+
+test_expect_success 'enable core.multiPackIndex' '
+	git config core.multiPackIndex true
+'
+
+test_expect_success 'create single-pack midx with bitmaps' '
+	git repack -ad &&
+	git multi-pack-index write --bitmap &&
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
+'
+
+basic_bitmap_tests
+
+test_expect_success 'create new additional packs' '
+	for i in $(test_seq 1 16)
+	do
+		test_commit "$i" &&
+		git repack -d
+	done &&
+
+	git checkout -b other2 HEAD~8 &&
+	for i in $(test_seq 1 8)
+	do
+		test_commit "side-$i" &&
+		git repack -d
+	done &&
+	git checkout second
+'
+
+test_expect_success 'create multi-pack midx with bitmaps' '
+	git multi-pack-index write --bitmap &&
+
+	ls $objdir/pack/pack-*.pack >packs &&
+	test_line_count = 25 packs &&
+
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
+'
+
+basic_bitmap_tests
+
+test_expect_success '--no-bitmap is respected when bitmaps exist' '
+	git multi-pack-index write --bitmap &&
+
+	test_commit respect--no-bitmap &&
+	GIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&
+
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+
+	git multi-pack-index write --no-bitmap &&
+
+	test_path_is_file $midx &&
+	test_path_is_missing $midx-$(midx_checksum $objdir).bitmap
+'
+
+test_expect_success 'setup midx with base from later pack' '
+	# Write a and b so that "a" is a delta on top of base "b", since Git
+	# prefers to delete contents out of a base rather than add to a shorter
+	# object.
+	test_seq 1 128 >a &&
+	test_seq 1 130 >b &&
+
+	git add a b &&
+	git commit -m "initial commit" &&
+
+	a=$(git rev-parse HEAD:a) &&
+	b=$(git rev-parse HEAD:b) &&
+
+	# In the first pack, "a" is stored as a delta to "b".
+	p1=$(git pack-objects .git/objects/pack/pack <<-EOF
+	$a
+	$b
+	EOF
+	) &&
+
+	# In the second pack, "a" is missing, and "b" is not a delta nor base to
+	# any other object.
+	p2=$(git pack-objects .git/objects/pack/pack <<-EOF
+	$b
+	$(git rev-parse HEAD)
+	$(git rev-parse HEAD^{tree})
+	EOF
+	) &&
+
+	git prune-packed &&
+	# Use the second pack as the preferred source, so that "b" occurs
+	# earlier in the MIDX object order, rendering "a" unusable for pack
+	# reuse.
+	git multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&
+
+	have_delta $a $b &&
+	test $(midx_pack_source $a) != $(midx_pack_source $b)
+'
+
+rev_list_tests 'full bitmap with backwards delta'
+
+test_expect_success 'clone with bitmaps enabled' '
+	git clone --no-local --bare . clone-reverse-delta.git &&
+	test_when_finished "rm -fr clone-reverse-delta.git" &&
+
+	git rev-parse HEAD >expect &&
+	git --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&
+	test_cmp expect actual
+'
+
+bitmap_reuse_tests() {
+	from=$1
+	to=$2
+
+	test_expect_success "setup pack reuse tests ($from -> $to)" '
+		rm -fr repo &&
+		git init repo &&
+		(
+			cd repo &&
+			test_commit_bulk 16 &&
+			git tag old-tip &&
+
+			git config core.multiPackIndex true &&
+			if test "MIDX" = "$from"
+			then
+				GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad &&
+				git multi-pack-index write --bitmap
+			else
+				GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb
+			fi
+		)
+	'
+
+	test_expect_success "build bitmap from existing ($from -> $to)" '
+		(
+			cd repo &&
+			test_commit_bulk --id=further 16 &&
+			git tag new-tip &&
+
+			if test "MIDX" = "$to"
+			then
+				GIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&
+				git multi-pack-index write --bitmap
+			else
+				GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb
+			fi
+		)
+	'
+
+	test_expect_success "verify resulting bitmaps ($from -> $to)" '
+		(
+			cd repo &&
+			git for-each-ref &&
+			git rev-list --test-bitmap refs/tags/old-tip &&
+			git rev-list --test-bitmap refs/tags/new-tip
+		)
+	'
+}
+
+bitmap_reuse_tests 'pack' 'MIDX'
+bitmap_reuse_tests 'MIDX' 'pack'
+bitmap_reuse_tests 'MIDX' 'MIDX'
+
+test_expect_success 'missing object closure fails gracefully' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit loose &&
+		test_commit packed &&
+
+		# Do not pass "--revs"; we want a pack without the "loose"
+		# commit.
+		git pack-objects $objdir/pack/pack <<-EOF &&
+		$(git rev-parse packed)
+		EOF
+
+		test_must_fail git multi-pack-index write --bitmap 2>err &&
+		grep "doesn.t have full closure" err &&
+		test_path_is_missing $midx
+	)
+'
+
+test_expect_success 'setup partial bitmaps' '
+	test_commit packed &&
+	git repack &&
+	test_commit loose &&
+	git multi-pack-index write --bitmap 2>err &&
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
+'
+
+basic_bitmap_tests HEAD~
+
+test_expect_success 'removing a MIDX clears stale bitmaps' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+		test_commit base &&
+		git repack &&
+		git multi-pack-index write --bitmap &&
+
+		# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.
+		stale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&
+		rm $midx &&
+
+		# Then write a new MIDX.
+		test_commit new &&
+		git repack &&
+		git multi-pack-index write --bitmap &&
+
+		test_path_is_file $midx &&
+		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+		test_path_is_missing $stale_bitmap
+	)
+'
+
+test_expect_success 'pack.preferBitmapTips' '
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit_bulk --message="%s" 103 &&
+
+		git log --format="%H" >commits.raw &&
+		sort <commits.raw >commits &&
+
+		git log --format="create refs/tags/%s %H" HEAD >refs &&
+		git update-ref --stdin <refs &&
+
+		git multi-pack-index write --bitmap &&
+		test_path_is_file $midx &&
+		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+
+		test-tool bitmap list-commits | sort >bitmaps &&
+		comm -13 bitmaps commits >before &&
+		test_line_count = 1 before &&
+
+		perl -ne "printf(\"create refs/tags/include/%d \", $.); print" \
+			<before | git update-ref --stdin &&
+
+		rm -fr $midx-$(midx_checksum $objdir).bitmap &&
+		rm -fr $midx-$(midx_checksum $objdir).rev &&
+		rm -fr $midx &&
+
+		git -c pack.preferBitmapTips=refs/tags/include \
+			multi-pack-index write --bitmap &&
+		test-tool bitmap list-commits | sort >bitmaps &&
+		comm -13 bitmaps commits >after &&
+
+		! test_cmp before after
+	)
+'
+
+test_done
Signature · 1 line
2.31.1.163.ga65ce7f831
#158 lore raw

Re: [PATCH v3 18/25] t5326: test multi-pack bitmap behavior

Quoted 6 lines · Taylor Blau
t/t5326-multi-pack-bitmaps.sh new file +0 -0
new file mode 100755
index 0000000000..c1b7d633e2

t/t5326-multi-pack-bitmaps.sh line 10

Quoted 10 lines
t/t5326-multi-pack-bitmaps.sh new file +10 -0
@@ -0,0 +1,277 @@
+#!/bin/sh
+
+test_description='exercise basic multi-pack bitmap functionality'
+. ./test-lib.sh
+. "${TEST_DIRECTORY}/lib-bitmap.sh"
+
+# We'll be writing our own midx and bitmaps, so avoid getting confused by the
+# automatic ones.
+GIT_TEST_MULTI_PACK_INDEX=0
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0

This latter variable doesn't do anything at this point in the series. Probably not a big deal (it is simply a noop until then), but if it's not hard, it may make sense to bump the "respect ... WRITE_BITMAP" patch earlier in the series.

t/t5326-multi-pack-bitmaps.sh line 33

Quoted 8 lines
t/t5326-multi-pack-bitmaps.sh new file +8 -0
+test_expect_success 'create single-pack midx with bitmaps' '
+	git repack -ad &&
+	git multi-pack-index write --bitmap &&
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
+'
+
+basic_bitmap_tests

We can't use a midx bitmap without a .rev file. The basic_bitmap_tests function covers that, but I wonder if we should also check:

  test_path_is_file $midx-$(midx_checksum $objdir).rev

in that first test.

t/t5326-multi-pack-bitmaps.sh line 40

Quoted 6 lines
t/t5326-multi-pack-bitmaps.sh new file +6 -0
+test_expect_success 'create new additional packs' '
+	for i in $(test_seq 1 16)
+	do
+		test_commit "$i" &&
+		git repack -d
+	done &&

This loop needs an "|| return 1" inside to catch &&-chain problems (not that we expect "repack -d" to fail, but just on principle).

t/t5326-multi-pack-bitmaps.sh line 47

Quoted 6 lines
t/t5326-multi-pack-bitmaps.sh new file +6 -0
+	git checkout -b other2 HEAD~8 &&
+	for i in $(test_seq 1 8)
+	do
+		test_commit "side-$i" &&
+		git repack -d
+	done &&

Ditto here.

t/t5326-multi-pack-bitmaps.sh line 59

Quoted 9 lines
t/t5326-multi-pack-bitmaps.sh new file +9 -0
+test_expect_success 'create multi-pack midx with bitmaps' '
+	git multi-pack-index write --bitmap &&
+
+	ls $objdir/pack/pack-*.pack >packs &&
+	test_line_count = 25 packs &&
+
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
+'

Possible spot for checking the .rev file again (though really, it is belt-and-suspenders at this point).

t/t5326-multi-pack-bitmaps.sh line 40

Quoted 1 line
t/t5326-multi-pack-bitmaps.sh new file +1 -0
+basic_bitmap_tests

I love how the earlier refactoring made it easy to test the single- and multi-pack cases thoroughly.

t/t5326-multi-pack-bitmaps.sh line 67

Quoted 5 lines
t/t5326-multi-pack-bitmaps.sh new file +5 -0
+test_expect_success '--no-bitmap is respected when bitmaps exist' '
+	git multi-pack-index write --bitmap &&
+
+	test_commit respect--no-bitmap &&
+	GIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&

Do we need to set this env variable? We've already set it to 0 at the top of the script.

t/t5326-multi-pack-bitmaps.sh line 76

Quoted 8 lines
t/t5326-multi-pack-bitmaps.sh new file +8 -0
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+
+	git multi-pack-index write --no-bitmap &&
+
+	test_path_is_file $midx &&
+	test_path_is_missing $midx-$(midx_checksum $objdir).bitmap
+'

OK, so we expect "--no-bitmap" to drop the bitmap (just like it does for a regular pack bitmap). Makes sense. We probably should check:

  test_path_is_missing $midx-$(midx_checksum $objdir).rev

here, too (unlike the other spots, it isn't redundant; we could leave a stale file around and likely nobody would notice).

Quoted 9 lines
t/t5326-multi-pack-bitmaps.sh new file +9 -0
+test_expect_success 'setup midx with base from later pack' '
+	# Write a and b so that "a" is a delta on top of base "b", since Git
+	# prefers to delete contents out of a base rather than add to a shorter
+	# object.
+	test_seq 1 128 >a &&
+	test_seq 1 130 >b &&
+
+	git add a b &&
+	git commit -m "initial commit" &&

t/t5326-multi-pack-bitmaps.sh line 96

Quoted 10 lines
t/t5326-multi-pack-bitmaps.sh new file +10 -0
+
+	a=$(git rev-parse HEAD:a) &&
+	b=$(git rev-parse HEAD:b) &&
+
+	# In the first pack, "a" is stored as a delta to "b".
+	p1=$(git pack-objects .git/objects/pack/pack <<-EOF
+	$a
+	$b
+	EOF
+	) &&

This is brittle with respect to Git's delta heuristics, of course, but I don't think there's a better way to do it with pack-objects. And this is not the first test to make similar assumptions. I think you can construct a known set of deltas using lib-pack.sh. It may get a bit complicated. As an alternative, maybe it makes sense to confirm that the deltas are set up as expected? You can do it with cat-file --batch-check.

Quoted 13 lines
t/t5326-multi-pack-bitmaps.sh new file +13 -0
+test_expect_success 'removing a MIDX clears stale bitmaps' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+		test_commit base &&
+		git repack &&
+		git multi-pack-index write --bitmap &&
+
+		# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.
+		stale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&
+		rm $midx &&

t/t5326-multi-pack-bitmaps.sh line 236

Quoted 10 lines
t/t5326-multi-pack-bitmaps.sh new file +10 -0
+
+		# Then write a new MIDX.
+		test_commit new &&
+		git repack &&
+		git multi-pack-index write --bitmap &&
+
+		test_path_is_file $midx &&
+		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+		test_path_is_missing $stale_bitmap
+	)

Another spot where we might want to check that the stale .rev file has gone away (and optionally that the new one was written; I haven't noted all of those, though).

Quoted 27 lines
t/t5326-multi-pack-bitmaps.sh new file +27 -0
+test_expect_success 'pack.preferBitmapTips' '
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit_bulk --message="%s" 103 &&
+
+		git log --format="%H" >commits.raw &&
+		sort <commits.raw >commits &&
+
+		git log --format="create refs/tags/%s %H" HEAD >refs &&
+		git update-ref --stdin <refs &&
+
+		git multi-pack-index write --bitmap &&
+		test_path_is_file $midx &&
+		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+
+		test-tool bitmap list-commits | sort >bitmaps &&
+		comm -13 bitmaps commits >before &&
+		test_line_count = 1 before &&
+
+		perl -ne "printf(\"create refs/tags/include/%d \", $.); print" \
+			<before | git update-ref --stdin &&
+
+		rm -fr $midx-$(midx_checksum $objdir).bitmap &&
+		rm -fr $midx-$(midx_checksum $objdir).rev &&

t/t5326-multi-pack-bitmaps.sh line 275

Quoted 10 lines
t/t5326-multi-pack-bitmaps.sh new file +10 -0
+		rm -fr $midx &&
+
+		git -c pack.preferBitmapTips=refs/tags/include \
+			multi-pack-index write --bitmap &&
+		test-tool bitmap list-commits | sort >bitmaps &&
+		comm -13 bitmaps commits >after &&
+
+		! test_cmp before after
+	)
+'

OK, so we are not depending on any _specific_ commits to get bitmapped, but just confirming that we have some impact. That may be the best we can do given that we are subject to the bitmap code's heuristics (and anyway, this is exactly what the pack version does).

Any other parts of the patch that I didn't quote looked very good to me. I'm happy to have such a thorough set of tests.

-Peff

#159 lore raw

Re: [PATCH v3 18/25] t5326: test multi-pack bitmap behavior

Quoted 9 lines · Jeff King

t/t5326-multi-pack-bitmaps.sh line 10

Quoted 4 lines
t/t5326-multi-pack-bitmaps.sh new file +4 -0
+# We'll be writing our own midx and bitmaps, so avoid getting confused by the
+# automatic ones.
+GIT_TEST_MULTI_PACK_INDEX=0
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0

This latter variable doesn't do anything at this point in the series. Probably not a big deal (it is simply a noop until then), but if it's not hard, it may make sense to bump the "respect ... WRITE_BITMAP" patch earlier in the series.

Reading the other patches, I guess you ordering was to "fix" each of the tests preemptively, and then add the knob at the end. That's OK by me. For an alternate test-mode like this, I usually wouldn't worry about bisectability, but it doesn't hurt. Somebody reading the commits later won't have any trouble finding the definition of the WRITE_BITMAP variable added in the subsequent patch.

Quoted 8 lines

t/t5326-multi-pack-bitmaps.sh line 67

Quoted 5 lines
t/t5326-multi-pack-bitmaps.sh new file +5 -0
+test_expect_success '--no-bitmap is respected when bitmaps exist' '
+	git multi-pack-index write --bitmap &&
+
+	test_commit respect--no-bitmap &&
+	GIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&

Do we need to set this env variable? We've already set it to 0 at the top of the script.

By the way, there were a few more of these later in the script that could be cleaned up, too.

-Peff

#160 lore raw

Re: [PATCH v3 18/25] t5326: test multi-pack bitmap behavior

In reply to Jeff King
Quoted 15 lines · Jeff King
Quoted 13 lines · Taylor Blau
t/t5326-multi-pack-bitmaps.sh new file +7 -0
new file mode 100755
index 0000000000..c1b7d633e2
@@ -0,0 +1,277 @@
+#!/bin/sh
+
+test_description='exercise basic multi-pack bitmap functionality'
+. ./test-lib.sh
+. "${TEST_DIRECTORY}/lib-bitmap.sh"
+
+# We'll be writing our own midx and bitmaps, so avoid getting confused by the
Quoted 8 lines

t/t5326-multi-pack-bitmaps.sh line 10

Quoted 3 lines
t/t5326-multi-pack-bitmaps.sh new file +3 -0
+# automatic ones.
+GIT_TEST_MULTI_PACK_INDEX=0
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0

This latter variable doesn't do anything at this point in the series. Probably not a big deal (it is simply a noop until then), but if it's not hard, it may make sense to bump the "respect ... WRITE_BITMAP" patch earlier in the series.

If my memory serves me correctly, I think the very first version of this patch didn't have a GIT_TEST_MULTI_PACK_INDEX{,_WRITE_BITMAP}=0 at the top, and so individual invocations needed to set it in their own environment. Presumably at some point I added this, but forgot to clean up the redundant ones. I removed the ones you mentioned in your response, and a few others.

Quoted 12 lines

t/t5326-multi-pack-bitmaps.sh line 33

Quoted 8 lines
t/t5326-multi-pack-bitmaps.sh new file +8 -0
+test_expect_success 'create single-pack midx with bitmaps' '
+	git repack -ad &&
+	git multi-pack-index write --bitmap &&
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
+'
+
+basic_bitmap_tests

We can't use a midx bitmap without a .rev file. The basic_bitmap_tests function covers that, but I wonder if we should also check:

Quoted 3 lines
  test_path_is_file $midx-$(midx_checksum $objdir).rev

in that first test.

Good idea. These tests probably preceded the invention of .rev files, so a lot of them needed updating. I made sure to add them where appropriate.

Quoted 9 lines

t/t5326-multi-pack-bitmaps.sh line 40

Quoted 6 lines
t/t5326-multi-pack-bitmaps.sh new file +6 -0
+test_expect_success 'create new additional packs' '
+	for i in $(test_seq 1 16)
+	do
+		test_commit "$i" &&
+		git repack -d
+	done &&

This loop needs an "|| return 1" inside to catch &&-chain problems (not that we expect "repack -d" to fail, but just on principle).

Nice catch, thanks.

Quoted 2 lines

I love how the earlier refactoring made it easy to test the single- and multi-pack cases thoroughly.

Likewise :-).

Quoted 20 lines
Quoted 9 lines
t/t5326-multi-pack-bitmaps.sh new file +9 -0
+test_expect_success 'setup midx with base from later pack' '
+	# Write a and b so that "a" is a delta on top of base "b", since Git
+	# prefers to delete contents out of a base rather than add to a shorter
+	# object.
+	test_seq 1 128 >a &&
+	test_seq 1 130 >b &&
+
+	git add a b &&
+	git commit -m "initial commit" &&

t/t5326-multi-pack-bitmaps.sh line 96

Quoted 10 lines
t/t5326-multi-pack-bitmaps.sh new file +10 -0
+
+	a=$(git rev-parse HEAD:a) &&
+	b=$(git rev-parse HEAD:b) &&
+
+	# In the first pack, "a" is stored as a delta to "b".
+	p1=$(git pack-objects .git/objects/pack/pack <<-EOF
+	$a
+	$b
+	EOF
+	) &&
Quoted 7 lines

This is brittle with respect to Git's delta heuristics, of course, but I don't think there's a better way to do it with pack-objects. And this is not the first test to make similar assumptions. I think you can construct a known set of deltas using lib-pack.sh. It may get a bit complicated. As an alternative, maybe it makes sense to confirm that the deltas are set up as expected? You can do it with cat-file --batch-check.

Yeah, I definitely agree that this test is brittle. But it would fail if our assumptions about what gets delta'd with what changes, because we do check that 'a' is a delta on top of 'b' (see the call to have_delta towards the end of this test). That have_delta helper does use --batch-check=%(deltabase), which is (I think) the cat-file invocation you're mentioning.

Thanks,
Taylor

#161 lore raw

Re: [PATCH v3 18/25] t5326: test multi-pack bitmap behavior

Quoted 8 lines · Taylor Blau
Quoted 7 lines

This is brittle with respect to Git's delta heuristics, of course, but I don't think there's a better way to do it with pack-objects. And this is not the first test to make similar assumptions. I think you can construct a known set of deltas using lib-pack.sh. It may get a bit complicated. As an alternative, maybe it makes sense to confirm that the deltas are set up as expected? You can do it with cat-file --batch-check.

Quoted 6 lines

Yeah, I definitely agree that this test is brittle. But it would fail if our assumptions about what gets delta'd with what changes, because we do check that 'a' is a delta on top of 'b' (see the call to have_delta towards the end of this test). That have_delta helper does use --batch-check=%(deltabase), which is (I think) the cat-file invocation you're mentioning.

Doh, I totally missed that. I was expecting to verify it earlier in the test as a pre-condition, but it works just fine where it is. So yeah, you are already doing the thing I was suggesting.

-Peff

#162 lore raw

PATCH v3 19/25 t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

In reply to Taylor Blau

From: Jeff King <[email protected]>

Generating a MIDX bitmap causes tests which repack in a partial clone to fail because they are missing objects. Missing objects is an expected component of tests in t0410, so disable this knob altogether. Graceful degradation when writing a bitmap with missing objects is tested in t5326.

---
 t/t0410-partial-clone.sh | 3 +++
 1 file changed, 3 insertions(+)
t/t0410-partial-clone.sh +3 -0
index bbcc51ee8e..bba679685f 100755
@@ -4,6 +4,9 @@ test_description='partial clone'
 
 . ./test-lib.sh
 
+# missing promisor objects cause repacks which write bitmaps to fail
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
 delete_object () {
 	rm $1/.git/objects/$(echo $2 | sed -e 's|^..|&/|')
 }
Signature · 1 line
2.31.1.163.ga65ce7f831
#163 lore raw

PATCH v3 20/25 t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

In reply to Taylor Blau

From: Jeff King <[email protected]>

Generating a MIDX bitmap confuses many of the tests in t5310, which expect to control whether and how bitmaps are written. Since the relevant MIDX-bitmap tests here are covered already in t5326, let's just disable the flag for the whole t5310 script.

---
 t/t5310-pack-bitmaps.sh | 4 ++++
 1 file changed, 4 insertions(+)
t/t5310-pack-bitmaps.sh +4 -0
index 4318f84d53..673baa5c3c 100755
@@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 . "$TEST_DIRECTORY"/lib-bundle.sh
 . "$TEST_DIRECTORY"/lib-bitmap.sh
 
+# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in
+# their place.
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
 objpath () {
 	echo ".git/objects/$(echo "$1" | sed -e 's|\(..\)|\1/|')"
 }
Signature · 1 line
2.31.1.163.ga65ce7f831
#164 lore raw

PATCH v3 21/25 t5319: don't write MIDX bitmaps in t5319

In reply to Taylor Blau

This test is specifically about generating a midx still respecting a pack-based bitmap file. Generating a MIDX bitmap would confuse the test. Let's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to make sure we don't do so.

---
 t/t5319-multi-pack-index.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
t/t5319-multi-pack-index.sh +2 -1
index 1f0a2ae852..7b685957c6 100755
@@ -504,7 +504,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '
 compare_results_with_midx "after repack"
 
 test_expect_success 'multi-pack-index and pack-bitmap' '
-	git -c repack.writeBitmaps=true repack -ad &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writeBitmaps=true repack -ad &&
 	git multi-pack-index write &&
 	git rev-list --test-bitmap HEAD
 '
Signature · 1 line
2.31.1.163.ga65ce7f831
#165 lore raw

PATCH v3 22/25 t7700: update to work with MIDX bitmap test knob

In reply to Taylor Blau

A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary.

---
 t/t7700-repack.sh | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
t/t7700-repack.sh +12 -6
index 25b235c063..98eda3bfeb 100755
@@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' '
 
 test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '
 	# build on $oid, $packid, and .keep state from previous
-	git repack -Adbl &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&
 	test_has_duplicate_object true
 '
 
 test_expect_success 'writing bitmaps via config can duplicate .keep objects' '
 	# build on $oid, $packid, and .keep state from previous
-	git -c repack.writebitmaps=true repack -Adl &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writebitmaps=true repack -Adl &&
 	test_has_duplicate_object true
 '
 
@@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' '
 
 test_expect_success 'bitmaps are created by default in bare repos' '
 	git clone --bare .git bare.git &&
-	git -C bare.git repack -ad &&
+	rm -f bare.git/objects/pack/*.bitmap &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad &&
 	bitmap=$(ls bare.git/objects/pack/*.bitmap) &&
 	test_path_is_file "$bitmap"
 '
@@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' '
 '
 
 test_expect_success 'bitmaps can be disabled on bare repos' '
-	git -c repack.writeBitmaps=false -C bare.git repack -ad &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writeBitmaps=false -C bare.git repack -ad &&
 	bitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&
 	test -z "$bitmap"
 '
@@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' '
 	keep=${pack%.pack}.keep &&
 	test_when_finished "rm -f \"\$keep\"" &&
 	>"$keep" &&
-	git -C bare.git repack -ad 2>stderr &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad 2>stderr &&
 	test_must_be_empty stderr &&
 	find bare.git/objects/pack/ -type f -name "*.bitmap" >actual &&
 	test_must_be_empty actual
@@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '
 	blob=$(test-tool genrandom big $((1024*1024)) |
 	       git -C bare.git hash-object -w --stdin) &&
 	git -C bare.git update-ref refs/tags/big $blob &&
-	git -C bare.git repack -ad 2>stderr &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad 2>stderr &&
 	test_must_be_empty stderr &&
 	find bare.git/objects/pack -type f -name "*.bitmap" >actual &&
 	test_must_be_empty actual
Signature · 1 line
2.31.1.163.ga65ce7f831
#166 lore raw

PATCH v3 23/25 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'

In reply to Taylor Blau

Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable to also write a multi-pack bitmap when 'GIT_TEST_MULTI_PACK_INDEX' is set.

---
 builtin/repack.c          | 12 ++++++++++--
 ci/run-build-and-tests.sh |  1 +
 midx.h                    |  2 ++
 t/README                  |  4 ++++
 4 files changed, 17 insertions(+), 2 deletions(-)
builtin/repack.c +10 -2
index 5f9bc74adc..82ab668272 100644
@@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		if (!(pack_everything & ALL_INTO_ONE) ||
 		    !is_bare_repository())
 			write_bitmaps = 0;
+	} else if (write_bitmaps &&
+		   git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&
+		   git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {
+		write_bitmaps = 0;
 	}
 	if (pack_kept_objects < 0)
 		pack_kept_objects = write_bitmaps > 0;
@@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		update_server_info(0);
 	remove_temporary_files();
 
-	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
-		write_midx_file(get_object_directory(), NULL, 0);
+	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {
+		unsigned flags = 0;
+		if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))
+			flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;
+		write_midx_file(get_object_directory(), NULL, flags);
+	}
 
 	string_list_clear(&names, 0);
 	string_list_clear(&rollback, 0);
ci/run-build-and-tests.sh +1 -0
index 3ce81ffee9..7ee9ba9325 100755
@@ -23,6 +23,7 @@ linux-gcc)
 	export GIT_TEST_COMMIT_GRAPH=1
 	export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1
 	export GIT_TEST_MULTI_PACK_INDEX=1
+	export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1
 	export GIT_TEST_ADD_I_USE_BUILTIN=1
 	export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master
 	export GIT_TEST_WRITE_REV_INDEX=1
midx.h +2 -0
index 350f4d0a7b..aa3da557bb 100644
@@ -8,6 +8,8 @@ struct pack_entry;
 struct repository;
 
 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX"
+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \
+	"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP"
 
 struct multi_pack_index {
 	struct multi_pack_index *next;
t/README +4 -0
index 9e70122302..12014aa988 100644
@@ -425,6 +425,10 @@ GIT_TEST_MULTI_PACK_INDEX=<boolean>, when true, forces the multi-pack-
 index to be written after every 'git repack' command, and overrides the
 'core.multiPackIndex' setting to true.
 
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=<boolean>, when true, sets the
+'--bitmap' option on all invocations of 'git multi-pack-index write',
+and ignores pack-objects' '--write-bitmap-index'.
+
 GIT_TEST_SIDEBAND_ALL=<boolean>, when true, overrides the
 'uploadpack.allowSidebandAll' setting to true, and when false, forces
 fetch-pack to not request sideband-all (even if the server advertises
Signature · 1 line
2.31.1.163.ga65ce7f831
#167 lore raw

Re: [PATCH v3 23/25] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'

builtin/repack.c int cmd_repack(int argc, const char **argv, const char *prefix) · line 522

Quoted 13 lines · Taylor Blau
builtin/repack.c +4 -0
index 5f9bc74adc..82ab668272 100644
@@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		if (!(pack_everything & ALL_INTO_ONE) ||
 		    !is_bare_repository())
 			write_bitmaps = 0;
+	} else if (write_bitmaps &&
+		   git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&
+		   git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {
+		write_bitmaps = 0;
 	}

This hunk confused me for a minute, since we are turning write_bitmaps _off_ if we see a positive "write midx bitmap". But I guess the point is to turn off the pack bitmap, and then in the later hunk:

builtin/repack.c int cmd_repack(int argc, const char **argv, const char *prefix) · line 737

Quoted 12 lines
builtin/repack.c +6 -2
@@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		update_server_info(0);
 	remove_temporary_files();
 
-	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
-		write_midx_file(get_object_directory(), NULL, 0);
+	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {
+		unsigned flags = 0;
+		if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))
+			flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;
+		write_midx_file(get_object_directory(), NULL, flags);
+	}

...we'd turn on the midx one. Makes sense.

-Peff

#168 lore raw

PATCH v3 24/25 p5310: extract full and partial bitmap tests

In reply to Taylor Blau

A new p5326 introduced by the next patch will want these same tests, interjecting its own setup in between. Move them out so that both perf tests can reuse them.

---
 t/perf/lib-bitmap.sh         | 69 ++++++++++++++++++++++++++++++++++++
 t/perf/p5310-pack-bitmaps.sh | 65 ++-------------------------------
 2 files changed, 72 insertions(+), 62 deletions(-)
 create mode 100644 t/perf/lib-bitmap.sh
t/perf/lib-bitmap.sh new file +69 -0
new file mode 100644
index 0000000000..63d3bc7cec
@@ -0,0 +1,69 @@
+# Helper functions for testing bitmap performance; see p5310.
+
+test_full_bitmap () {
+	test_perf 'simulated clone' '
+		git pack-objects --stdout --all </dev/null >/dev/null
+	'
+
+	test_perf 'simulated fetch' '
+		have=$(git rev-list HEAD~100 -1) &&
+		{
+			echo HEAD &&
+			echo ^$have
+		} | git pack-objects --revs --stdout >/dev/null
+	'
+
+	test_perf 'pack to file (bitmap)' '
+		git pack-objects --use-bitmap-index --all pack1b </dev/null >/dev/null
+	'
+
+	test_perf 'rev-list (commits)' '
+		git rev-list --all --use-bitmap-index >/dev/null
+	'
+
+	test_perf 'rev-list (objects)' '
+		git rev-list --all --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list with tag negated via --not --all (objects)' '
+		git rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list with negative tag (objects)' '
+		git rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list count with blob:none' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=blob:none >/dev/null
+	'
+
+	test_perf 'rev-list count with blob:limit=1k' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=blob:limit=1k >/dev/null
+	'
+
+	test_perf 'rev-list count with tree:0' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=tree:0 >/dev/null
+	'
+
+	test_perf 'simulated partial clone' '
+		git pack-objects --stdout --all --filter=blob:none </dev/null >/dev/null
+	'
+}
+
+test_partial_bitmap () {
+	test_perf 'clone (partial bitmap)' '
+		git pack-objects --stdout --all </dev/null >/dev/null
+	'
+
+	test_perf 'pack to file (partial bitmap)' '
+		git pack-objects --use-bitmap-index --all pack2b </dev/null >/dev/null
+	'
+
+	test_perf 'rev-list with tree filter (partial bitmap)' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=tree:0 >/dev/null
+	'
+}
t/perf/p5310-pack-bitmaps.sh +3 -62

82 lines not on this page · raw message

Signature · 1 line
2.31.1.163.ga65ce7f831
#169 lore raw

PATCH v3 25/25 p5326: perf tests for MIDX bitmaps

In reply to Taylor Blau

These new performance tests demonstrate effectively the same behavior as p5310, but use a multi-pack bitmap instead of a single-pack one.

Notably, p5326 does not create a MIDX bitmap with multiple packs. This is so we can measure a direct comparison between it and p5310. Any difference between the two is measuring just the overhead of using MIDX bitmaps.

Here are the results of p5310 and p5326 together, measured at the same time and on the same machine (using a Xenon W-2255 CPU):

    Test                                                  HEAD
    ------------------------------------------------------------------------
    5310.2: repack to disk                                96.78(93.39+11.33)
    5310.3: simulated clone                               9.98(9.79+0.19)
    5310.4: simulated fetch                               1.75(4.26+0.19)
    5310.5: pack to file (bitmap)                         28.20(27.87+8.70)
    5310.6: rev-list (commits)                            0.41(0.36+0.05)
    5310.7: rev-list (objects)                            1.61(1.54+0.07)
    5310.8: rev-list count with blob:none                 0.25(0.21+0.04)
    5310.9: rev-list count with blob:limit=1k             2.65(2.54+0.10)
    5310.10: rev-list count with tree:0                   0.23(0.19+0.04)
    5310.11: simulated partial clone                      4.34(4.21+0.12)
    5310.13: clone (partial bitmap)                       11.05(12.21+0.48)
    5310.14: pack to file (partial bitmap)                31.25(34.22+3.70)
    5310.15: rev-list with tree filter (partial bitmap)   0.26(0.22+0.04)

versus the same tests (this time using a multi-pack index):

    Test                                                  HEAD
    ------------------------------------------------------------------------
    5326.2: setup multi-pack index                        78.99(75.29+11.58)
    5326.3: simulated clone                               11.78(11.56+0.22)
    5326.4: simulated fetch                               1.70(4.49+0.13)
    5326.5: pack to file (bitmap)                         28.02(27.72+8.76)
    5326.6: rev-list (commits)                            0.42(0.36+0.06)
    5326.7: rev-list (objects)                            1.65(1.58+0.06)
    5326.8: rev-list count with blob:none                 0.26(0.21+0.05)
    5326.9: rev-list count with blob:limit=1k             2.97(2.86+0.10)
    5326.10: rev-list count with tree:0                   0.25(0.20+0.04)
    5326.11: simulated partial clone                      5.65(5.49+0.16)
    5326.13: clone (partial bitmap)                       12.22(13.43+0.38)
    5326.14: pack to file (partial bitmap)                30.05(31.57+7.25)
    5326.15: rev-list with tree filter (partial bitmap)   0.24(0.20+0.04)

There is slight overhead in "simulated clone", "simulated partial clone", and "clone (partial bitmap)". Unsurprisingly, that overhead is due to using the MIDX's reverse index to map between bit positions and MIDX positions.

This can be reproduced by running "git repack -adb" along with "git multi-pack-index write --bitmap" in a large-ish repository. Then run:

    $ perf record -o pack.perf git -c core.multiPackIndex=false \
      pack-objects --all --stdout >/dev/null </dev/null
    $ perf record -o midx.perf git -c core.multiPackIndex=true \
      pack-objects --all --stdout >/dev/null </dev/null

and compare the two with "perf diff -c delta -o 1 pack.perf midx.perf". The most notable results are below (the next largest positive delta is +0.14%):

# Event 'cycles'
#
# Baseline Delta Shared Object Symbol
# ........ ....... .................. .......................... #
+5.86% git [.] nth_midxed_offset
+5.24% git [.] nth_midxed_pack_int_id
3.45% +0.97% git [.] offset_to_pack_pos
3.30% +0.57% git [.] pack_pos_to_offset
+0.30% git [.] pack_pos_to_midx

---
 t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh
t/perf/p5326-multi-pack-bitmaps.sh new file +43 -0
new file mode 100755
index 0000000000..5845109ac7
@@ -0,0 +1,43 @@
+#!/bin/sh
+
+test_description='Tests performance using midx bitmaps'
+. ./perf-lib.sh
+. "${TEST_DIRECTORY}/perf/lib-bitmap.sh"
+
+test_perf_large_repo
+
+test_expect_success 'enable multi-pack index' '
+	git config core.multiPackIndex true
+'
+
+test_perf 'setup multi-pack index' '
+	git repack -ad &&
+	git multi-pack-index write --bitmap
+'
+
+test_full_bitmap
+
+test_expect_success 'create partial bitmap state' '
+	# pick a commit to represent the repo tip in the past
+	cutoff=$(git rev-list HEAD~100 -1) &&
+	orig_tip=$(git rev-parse HEAD) &&
+
+	# now pretend we have just one tip
+	rm -rf .git/logs .git/refs/* .git/packed-refs &&
+	git update-ref HEAD $cutoff &&
+
+	# and then repack, which will leave us with a nice
+	# big bitmap pack of the "old" history, and all of
+	# the new history will be loose, as if it had been pushed
+	# up incrementally and exploded via unpack-objects
+	git repack -Ad &&
+	git multi-pack-index write --bitmap &&
+
+	# and now restore our original tip, as if the pushes
+	# had happened
+	git update-ref HEAD $orig_tip
+'
+
+test_partial_bitmap
+
+test_done
Signature · 1 line
2.31.1.163.ga65ce7f831
#170 lore raw

Re: [PATCH v3 25/25] p5326: perf tests for MIDX bitmaps

Quoted 10 lines · Taylor Blau

These new performance tests demonstrate effectively the same behavior as p5310, but use a multi-pack bitmap instead of a single-pack one.

Notably, p5326 does not create a MIDX bitmap with multiple packs. This is so we can measure a direct comparison between it and p5310. Any difference between the two is measuring just the overhead of using MIDX bitmaps.

Here are the results of p5310 and p5326 together, measured at the same time and on the same machine (using a Xenon W-2255 CPU):

Neat. I think having separate perf regression tests for regular and mix bitmaps will be useful, but being able to compare the pack and mix versions is a cherry on top.

There was one funny number:

Quoted 2 lines
    5310.2: repack to disk                                96.78(93.39+11.33)
    5326.2: setup multi-pack index                        78.99(75.29+11.58)

In p5310, that step is repacking and writing bitmaps. With the midx, it's repacking, then writing a midx with bitmaps. I'd expect the latter to be strictly slower than the former, but here it's faster.

Running the code locally, I got similar results (with p5310 just a tiny bit faster). So it may have just been noise or some other timing issue.

As an aside, I think that test is a little bit bogus due to GIT_PERF_REPEAT_COUNT; the first trial will generate bitmaps from scratch, and then subsequent runs will reuse partial results. It probably should "rm -f .git/objects/*.bitmap" within the test. We can deal with that separately, though.

-Peff

#171 lore raw
In reply to Taylor Blau
Quoted 3 lines · Taylor Blau

Thanks in advance for your review. I think Peff still wanted to read through patches 16-25, but that the first 15 or so should be in pretty good shape by now.

I think this is looking pretty close. There's the close_midx() thing discussed in patch 9 that I think we need to deal with. In the tests, I found some little nits. Nothing serious, but some of it at least is worth fixing.

So I think with one more fairly trivial re-roll, we can think about merging this to 'next'.

Thanks for your patience with my slow reviews, and for all your work on this. It's really quite a complicated topic. :)

-Peff

#172 lore raw
Quoted 10 lines · Jeff King
Quoted 3 lines · Taylor Blau

Thanks in advance for your review. I think Peff still wanted to read through patches 16-25, but that the first 15 or so should be in pretty good shape by now.

I think this is looking pretty close. There's the close_midx() thing discussed in patch 9 that I think we need to deal with. In the tests, I found some little nits. Nothing serious, but some of it at least is worth fixing.

Thanks; I think I fixed everything up that you mentioned. The big deals were about close_object_store() versus close_midx(), and the changes to t5326 in patch 18/25. I think the remaining comments on patches 19-25 were thinking aloud instead of recommending any changes.

Quoted 2 lines

So I think with one more fairly trivial re-roll, we can think about merging this to 'next'.

Great. I have all of that prepared locally, but I'll wait until after Monday to send it since I don't want to dilute the conversation away from release hardening (I certainly don't mind your review, I just figure folks would appreciate me *not* sending 25 new messages to their inboxes ;-)).

Quoted 2 lines

Thanks for your patience with my slow reviews, and for all your work on this. It's really quite a complicated topic. :)

Thanks for reviewing. I tried my best to break this series up from all of the ones that it depends on, but there was only so much I could do to isolate the complexity. I'm glad that it had a thorough set of eyes on it.

I'll look forward to sending a hopefully-final reroll shortly after the release so we can move onto a few more cosmetic topics on top to integrate MIDX bitmaps more closely with repack and so on.

Thanks,
Taylor

#173 lore raw

v4 cover multi-pack reachability bitmaps

In reply to Taylor Blau

Here is what I anticipate to be a final reroll of my series to implement multi-pack reachability bitmaps, based on review feedback from Peff.

Most of the change since last time are cosmetic test clean-ups. The previous reroll of this series incorporated feedback from a discussion[1] surrounding the multi-pack-index builtin's --object-dir argument. This reroll fixes a bug discussed here[2] where we should have been calling close_object_store() but weren't; the remainder of that bug has already been dealt with.

Thanks everybody for dealing with multiple versions of this quite lengthy and complicated series. Hopefully we are done in this round and can move on to integrating this with git repack, which will complete the MIDX bitmaps topic.

[1]: https://lore.kernel.org/git/YQMFIljXl7sAAA%[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/

Jeff King (2):
t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

Taylor Blau (23):
pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps pack-bitmap-write.c: gracefully fail to write non-closed bitmaps pack-bitmap-write.c: free existing bitmaps
Documentation: describe MIDX-based bitmaps
midx: clear auxiliary .rev after replacing the MIDX
midx: reject empty --preferred-pack's
midx: infer preferred pack when not given one
midx: close linked MIDXs, avoid leaking memory
midx: avoid opening multiple MIDXs when writing pack-bitmap.c: introduce 'bitmap_num_objects()' pack-bitmap.c: introduce 'nth_bitmap_object_oid()' pack-bitmap.c: introduce 'bitmap_is_preferred_refname()' pack-bitmap.c: avoid redundant calls to try_partial_reuse
pack-bitmap: read multi-pack bitmaps
pack-bitmap: write multi-pack bitmaps
t5310: move some tests to lib-bitmap.sh t/helper/test-read-midx.c: add --checksum mode
t5326: test multi-pack bitmap behavior
t5319: don't write MIDX bitmaps in t5319
t7700: update to work with MIDX bitmap test knob
midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'
p5310: extract full and partial bitmap tests
p5326: perf tests for MIDX bitmaps

 Documentation/git-multi-pack-index.txt       |  18 +-
 Documentation/technical/bitmap-format.txt    |  71 ++-
 Documentation/technical/multi-pack-index.txt |  10 +-
 builtin/multi-pack-index.c                   |   2 +
 builtin/pack-objects.c                       |   8 +-
 builtin/repack.c                             |  12 +-
 ci/run-build-and-tests.sh                    |   1 +
 midx.c                                       | 319 +++++++++++-
 midx.h                                       |   5 +
 pack-bitmap-write.c                          |  79 ++-
 pack-bitmap.c                                | 499 ++++++++++++++++---
 pack-bitmap.h                                |   9 +-
 packfile.c                                   |   2 +-
 t/README                                     |   4 +
 t/helper/test-read-midx.c                    |  16 +-
 t/lib-bitmap.sh                              | 240 +++++++++
 t/perf/lib-bitmap.sh                         |  69 +++
 t/perf/p5310-pack-bitmaps.sh                 |  65 +--
 t/perf/p5326-multi-pack-bitmaps.sh           |  43 ++
 t/t0410-partial-clone.sh                     |  12 +-
 t/t5310-pack-bitmaps.sh                      | 231 +--------
 t/t5319-multi-pack-index.sh                  |  20 +-
 t/t5326-multi-pack-bitmaps.sh                | 286 +++++++++++
 t/t7700-repack.sh                            |  18 +-
 24 files changed, 1603 insertions(+), 436 deletions(-)
 create mode 100644 t/perf/lib-bitmap.sh
 create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh
 create mode 100755 t/t5326-multi-pack-bitmaps.sh

Range-diff against v3:

 1:  fa4cbed48e =  1:  92dc0bbc0d pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps
 2:  2b15c1fc5c =  2:  979276bc74 pack-bitmap-write.c: gracefully fail to write non-closed bitmaps
 3:  2ad513a230 =  3:  8f00493955 pack-bitmap-write.c: free existing bitmaps
 4:  8da5de7c24 =  4:  bc7db926d8 Documentation: describe MIDX-based bitmaps
 5:  49297f57ed =  5:  771741844b midx: clear auxiliary .rev after replacing the MIDX
 6:  c5513f2a75 =  6:  dab5dbf228 midx: reject empty `--preferred-pack`'s
 7:  53ef0a6d67 =  7:  31f4517de0 midx: infer preferred pack when not given one
 8:  114773d9cd =  8:  aa3bd96d9b midx: close linked MIDXs, avoid leaking memory
 9:  40cff5beb5 !  9:  c9fea31fa8 midx: avoid opening multiple MIDXs when writing

@@ Commit message
one and should invalidate the object store's memory of any MIDX that might have existed beforehand.

+ Note that this now forbids passing object directories that don't belong
+ to alternate repositories over --object-dir, since before we would
+ have happily opened a MIDX in any directory, but now restrict ourselves
+ to only those reachable by r->objects->multi_pack_index (and alternate
+ MIDXs that we can see by walking the next pointer).
+
+ As far as I can tell, supporting arbitrary directories with
+ --object-dir was a historical accident, since even the documentation
+ says <alt> when referring to the value passed to this option.
+
+ A future patch could clean this up and provide a warning() when a
+ non-alternate directory was given, since we'll still write a new MIDX
+ there, we just won't reuse any MIDX that might happen to already exist
+ in that directory.
+

## midx.c ##
@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
+ break;
+ }
+ }
-+ if (!ctx.m)
-+ ctx.m = get_local_multi_pack_index(the_repository);

if (ctx.m && !midx_checksum_valid(ctx.m)) {
warning(_("ignoring existing multi-pack-index; checksum mismatch"));
+@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *
+ f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
+
+ if (ctx.m)
+- close_midx(ctx.m);
++ close_object_store(the_repository->objects);
+
+ if (ctx.nr - dropped_packs == 0) {
+ error(_("no pack files to index."));
@@ midx.c: int write_midx_file(const char *object_dir,
const char *preferred_pack_name,
unsigned flags)
10: ca7f726abf ! 10: ee72fb7e38 pack-bitmap.c: introduce 'bitmap_num_objects()'
@@ pack-bitmap.c: static void show_extended_objects(struct bitmap_index *bitmap_git

obj = eindex->objects[i];
@@ pack-bitmap.c: static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,
- * individually.
+ * them individually.
*/
for (i = 0; i < eindex->count; i++) {
- uint32_t pos = i + bitmap_git->pack->num_objects;

11:  67e6897a34 = 11:  ede0bf1ce1 pack-bitmap.c: introduce 'nth_bitmap_object_oid()'
12:  743a1a138e = 12:  df6844def0 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'
13:  a3b641b3e6 = 13:  4e06f051a7 pack-bitmap.c: avoid redundant calls to try_partial_reuse
14:  141ff83275 = 14:  a0d73eb3d3 pack-bitmap: read multi-pack bitmaps
15:  54600b5814 ! 15:  9d83ad77ab pack-bitmap: write multi-pack bitmaps
    @@ midx.c: static int write_midx_internal(const char *object_dir,
      	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
      
     -	if (ctx.m)
    --		close_midx(ctx.m);
    +-		close_object_store(the_repository->objects);
     -
      	if (ctx.nr - dropped_packs == 0) {
      		error(_("no pack files to index."));
    @@ midx.c: static int write_midx_internal(const char *object_dir,
     +		}
     +	}
     +
    -+	close_midx(ctx.m);
    ++	close_object_store(the_repository->objects);
      
      	commit_lock_file(&lk);

16: 168b7b0976 ! 16: a92af89884 t5310: move some tests to lib-bitmap.sh
@@ Commit message

## t/lib-bitmap.sh ##
@@
-+# Helpers for scripts testing bitamp functionality; see t5310 for ++# Helpers for scripts testing bitmap functionality; see t5310 for
+# example usage.
+
# Compare a file containing rev-list bitmap traversal output to its non-bitmap
17: 60ec8b3466 ! 17: d47aa4a919 t/helper/test-read-midx.c: add --checksum mode

    @@ t/lib-bitmap.sh: have_delta () {
      }
     +
     +midx_checksum () {
    -+	test-tool read-midx --checksum "${1:-.git/objects}"
    ++	test-tool read-midx --checksum "$1"
     +}
18:  3258ccfc1c ! 18:  9d9d9f28a6 t5326: test multi-pack bitmap behavior
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +	git repack -ad &&
     +	git multi-pack-index write --bitmap &&
     +	test_path_is_file $midx &&
    -+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
    ++	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
    ++	test_path_is_file $midx-$(midx_checksum $objdir).rev
     +'
     +
     +basic_bitmap_tests
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +	for i in $(test_seq 1 16)
     +	do
     +		test_commit "$i" &&
    -+		git repack -d
    ++		git repack -d || return 1
     +	done &&
     +
     +	git checkout -b other2 HEAD~8 &&
     +	for i in $(test_seq 1 8)
     +	do
     +		test_commit "side-$i" &&
    -+		git repack -d
    ++		git repack -d || return 1
     +	done &&
     +	git checkout second
     +'
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +	test_line_count = 25 packs &&
     +
     +	test_path_is_file $midx &&
    -+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
    ++	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
    ++	test_path_is_file $midx-$(midx_checksum $objdir).rev
     +'
     +
     +basic_bitmap_tests
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +	git multi-pack-index write --bitmap &&
     +
     +	test_commit respect--no-bitmap &&
    -+	GIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&
    ++	git repack -d &&
     +
     +	test_path_is_file $midx &&
     +	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
    ++	test_path_is_file $midx-$(midx_checksum $objdir).rev &&
     +
     +	git multi-pack-index write --no-bitmap &&
     +
     +	test_path_is_file $midx &&
    -+	test_path_is_missing $midx-$(midx_checksum $objdir).bitmap
    ++	test_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&
    ++	test_path_is_missing $midx-$(midx_checksum $objdir).rev
     +'
     +
     +test_expect_success 'setup midx with base from later pack' '
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +			git config core.multiPackIndex true &&
     +			if test "MIDX" = "$from"
     +			then
    -+				GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad &&
    ++				git repack -Ad &&
     +				git multi-pack-index write --bitmap
     +			else
    -+				GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb
    ++				git repack -Adb
     +			fi
     +		)
     +	'
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +
     +			if test "MIDX" = "$to"
     +			then
    -+				GIT_TEST_MULTI_PACK_INDEX=0 git repack -d &&
    ++				git repack -d &&
     +				git multi-pack-index write --bitmap
     +			else
    -+				GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb
    ++				git repack -Adb
     +			fi
     +		)
     +	'
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +	test_commit loose &&
     +	git multi-pack-index write --bitmap 2>err &&
     +	test_path_is_file $midx &&
    -+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap
    ++	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
    ++	test_path_is_file $midx-$(midx_checksum $objdir).rev
     +'
     +
     +basic_bitmap_tests HEAD~
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +
     +		# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.
     +		stale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&
    ++		stale_rev=$midx-$(midx_checksum $objdir).rev &&
     +		rm $midx &&
     +
     +		# Then write a new MIDX.
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +
     +		test_path_is_file $midx &&
     +		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
    -+		test_path_is_missing $stale_bitmap
    ++		test_path_is_file $midx-$(midx_checksum $objdir).rev &&
    ++		test_path_is_missing $stale_bitmap &&
    ++		test_path_is_missing $stale_rev
     +	)
     +'
     +
    @@ t/t5326-multi-pack-bitmaps.sh (new)
     +		git multi-pack-index write --bitmap &&
     +		test_path_is_file $midx &&
     +		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
    ++		test_path_is_file $midx-$(midx_checksum $objdir).rev &&
     +
     +		test-tool bitmap list-commits | sort >bitmaps &&
     +		comm -13 bitmaps commits >before &&
19:  47c7e6bb9b = 19:  3e0da7e5ed t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
20:  6a708858b1 = 20:  4e0d49a2dd t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
21:  1eaa744b24 = 21:  47eba8ecf9 t5319: don't write MIDX bitmaps in t5319
22:  a4a899e31f = 22:  3d78afa2ad t7700: update to work with MIDX bitmap test knob
23:  50865e52a3 = 23:  c2f94e033d midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'
24:  0f1fd6e7d4 = 24:  6b03016c99 p5310: extract full and partial bitmap tests
25:  82e8133bf4 = 25:  d98faa4c2c p5326: perf tests for MIDX bitmaps
Signature · 1 line
2.31.1.163.ga65ce7f831
#174 lore raw

PATCH v4 1/25 pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps

The special --test-bitmap mode of git rev-list is used to compare the result of an object traversal with a bitmap to check its integrity. This mode does not, however, assert that the types of reachable objects are stored correctly.

Harden this mode by teaching it to also check that each time an object's bit is marked, the corresponding bit should be set in exactly one of the type bitmaps (whose type matches the object's true type).

---
 pack-bitmap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
pack-bitmap.c +48 -0
index d999616c9e..9b11af87aa 100644
@@ -1325,10 +1325,52 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git,
 struct bitmap_test_data {
 	struct bitmap_index *bitmap_git;
 	struct bitmap *base;
+	struct bitmap *commits;
+	struct bitmap *trees;
+	struct bitmap *blobs;
+	struct bitmap *tags;
 	struct progress *prg;
 	size_t seen;
 };
 
+static void test_bitmap_type(struct bitmap_test_data *tdata,
+			     struct object *obj, int pos)
+{
+	enum object_type bitmap_type = OBJ_NONE;
+	int bitmaps_nr = 0;
+
+	if (bitmap_get(tdata->commits, pos)) {
+		bitmap_type = OBJ_COMMIT;
+		bitmaps_nr++;
+	}
+	if (bitmap_get(tdata->trees, pos)) {
+		bitmap_type = OBJ_TREE;
+		bitmaps_nr++;
+	}
+	if (bitmap_get(tdata->blobs, pos)) {
+		bitmap_type = OBJ_BLOB;
+		bitmaps_nr++;
+	}
+	if (bitmap_get(tdata->tags, pos)) {
+		bitmap_type = OBJ_TAG;
+		bitmaps_nr++;
+	}
+
+	if (bitmap_type == OBJ_NONE)
+		die("object %s not found in type bitmaps",
+		    oid_to_hex(&obj->oid));
+
+	if (bitmaps_nr > 1)
+		die("object %s does not have a unique type",
+		    oid_to_hex(&obj->oid));
+
+	if (bitmap_type != obj->type)
+		die("object %s: real type %s, expected: %s",
+		    oid_to_hex(&obj->oid),
+		    type_name(obj->type),
+		    type_name(bitmap_type));
+}
+
 static void test_show_object(struct object *object, const char *name,
 			     void *data)
 {
@@ -1338,6 +1380,7 @@ static void test_show_object(struct object *object, const char *name,
 	bitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);
 	if (bitmap_pos < 0)
 		die("Object not in bitmap: %s\n", oid_to_hex(&object->oid));
+	test_bitmap_type(tdata, object, bitmap_pos);
 
 	bitmap_set(tdata->base, bitmap_pos);
 	display_progress(tdata->prg, ++tdata->seen);
@@ -1352,6 +1395,7 @@ static void test_show_commit(struct commit *commit, void *data)
 				     &commit->object.oid);
 	if (bitmap_pos < 0)
 		die("Object not in bitmap: %s\n", oid_to_hex(&commit->object.oid));
+	test_bitmap_type(tdata, &commit->object, bitmap_pos);
 
 	bitmap_set(tdata->base, bitmap_pos);
 	display_progress(tdata->prg, ++tdata->seen);
@@ -1399,6 +1443,10 @@ void test_bitmap_walk(struct rev_info *revs)
 
 	tdata.bitmap_git = bitmap_git;
 	tdata.base = bitmap_new();
+	tdata.commits = ewah_to_bitmap(bitmap_git->commits);
+	tdata.trees = ewah_to_bitmap(bitmap_git->trees);
+	tdata.blobs = ewah_to_bitmap(bitmap_git->blobs);
+	tdata.tags = ewah_to_bitmap(bitmap_git->tags);
 	tdata.prg = start_progress("Verifying bitmap entries", result_popcnt);
 	tdata.seen = 0;
 
Signature · 1 line
2.31.1.163.ga65ce7f831
#175 lore raw

PATCH v4 2/25 pack-bitmap-write.c: gracefully fail to write non-closed bitmaps

In reply to Taylor Blau

The set of objects covered by a bitmap must be closed under reachability, since it must be the case that there is a valid bit position assigned for every possible reachable object (otherwise the bitmaps would be incomplete).

Pack bitmaps are never written from 'git repack' unless repacking all-into-one, and so we never write non-closed bitmaps (except in the case of partial clones where we aren't guaranteed to have all objects).

But multi-pack bitmaps change this, since it isn't known whether the set of objects in the MIDX is closed under reachability until walking them. Plumb through a bit that is set when a reachable object isn't found.

As soon as a reachable object isn't found in the set of objects to include in the bitmap, bitmap_writer_build() knows that the set is not closed, and so it now fails gracefully.

A test is added in t0410 to trigger a bitmap write without full reachability closure by removing local copies of some reachable objects from a promisor remote.

---
 builtin/pack-objects.c   |  3 +-
 pack-bitmap-write.c      | 76 ++++++++++++++++++++++++++++------------
 pack-bitmap.h            |  2 +-
 t/t0410-partial-clone.sh |  9 ++++-
 4 files changed, 64 insertions(+), 26 deletions(-)
builtin/pack-objects.c +2 -1
index de00adbb9e..8a523624a1 100644
@@ -1256,7 +1256,8 @@ static void write_pack_file(void)
 
 				bitmap_writer_show_progress(progress);
 				bitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);
-				bitmap_writer_build(&to_pack);
+				if (bitmap_writer_build(&to_pack) < 0)
+					die(_("failed to write bitmap index"));
 				bitmap_writer_finish(written_list, nr_written,
 						     tmpname.buf, write_bitmap_options);
 				write_bitmap_index = 0;
pack-bitmap-write.c +53 -23

159 lines not on this page · raw message

pack-bitmap.h +1 -1
index 99d733eb26..020cd8d868 100644
@@ -87,7 +87,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,
 				      struct commit *commit);
 void bitmap_writer_select_commits(struct commit **indexed_commits,
 		unsigned int indexed_commits_nr, int max_bitmaps);
-void bitmap_writer_build(struct packing_data *to_pack);
+int bitmap_writer_build(struct packing_data *to_pack);
 void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint32_t index_nr,
 			  const char *filename,
t/t0410-partial-clone.sh +8 -1
index a211a66c67..bbcc51ee8e 100755
@@ -536,7 +536,13 @@ test_expect_success 'gc does not repack promisor objects if there are none' '
 repack_and_check () {
 	rm -rf repo2 &&
 	cp -r repo repo2 &&
-	git -C repo2 repack $1 -d &&
+	if test x"$1" = "x--must-fail"
+	then
+		shift
+		test_must_fail git -C repo2 repack $1 -d
+	else
+		git -C repo2 repack $1 -d
+	fi &&
 	git -C repo2 fsck &&
 
 	git -C repo2 cat-file -e $2 &&
@@ -561,6 +567,7 @@ test_expect_success 'repack -d does not irreversibly delete promisor objects' '
 	printf "$THREE\n" | pack_as_from_promisor &&
 	delete_object repo "$ONE" &&
 
+	repack_and_check --must-fail -ab "$TWO" "$THREE" &&
 	repack_and_check -a "$TWO" "$THREE" &&
 	repack_and_check -A "$TWO" "$THREE" &&
 	repack_and_check -l "$TWO" "$THREE"
Signature · 1 line
2.31.1.163.ga65ce7f831
#176 lore raw

PATCH v4 3/25 pack-bitmap-write.c: free existing bitmaps

In reply to Taylor Blau

When writing a new bitmap, the bitmap writer code attempts to read the existing bitmap (if one is present). This is done in order to quickly permute the bits of any bitmaps for commits which appear in the existing bitmap, and were also selected for the new bitmap.

But since this code was added in 341fa34887 (pack-bitmap-write: use existing bitmaps, 2020-12-08), the resources associated with opening an existing bitmap were never released.

It's fine to ignore this, but it's bad hygiene. It will also cause a problem for the multi-pack-index builtin, which will be responsible not only for writing bitmaps, but also for expiring any old multi-pack bitmaps.

If an existing bitmap was reused here, it will also be expired. That will cause a problem on platforms which require file resources to be closed before unlinking them, like Windows. Avoid this by ensuring we close reused bitmaps with free_bitmap_index() before removing them.

---
 pack-bitmap-write.c | 1 +
 1 file changed, 1 insertion(+)
pack-bitmap-write.c +1 -0
index d374f7884b..142fd0adb8 100644
@@ -520,6 +520,7 @@ int bitmap_writer_build(struct packing_data *to_pack)
 	clear_prio_queue(&queue);
 	clear_prio_queue(&tree_queue);
 	bitmap_builder_clear(&bb);
+	free_bitmap_index(old_bitmap);
 	free(mapping);
 
 	trace2_region_leave("pack-bitmap-write", "building_bitmaps_total",
Signature · 1 line
2.31.1.163.ga65ce7f831
#177 lore raw

PATCH v4 4/25 Documentation: describe MIDX-based bitmaps

In reply to Taylor Blau

Update the technical documentation to describe the multi-pack bitmap format. This patch merely introduces the new format, and describes its high-level ideas. Git does not yet know how to read nor write these multi-pack variants, and so the subsequent patches will:

- Introduce code to interpret multi-pack bitmaps, according to this document.

- Then, introduce code to write multi-pack bitmaps from the 'git multi-pack-index write' sub-command.

Finally, the implementation will gain tests in subsequent patches (as opposed to inline with the patch teaching Git how to write multi-pack bitmaps) to avoid a cyclic dependency.

---
 Documentation/technical/bitmap-format.txt    | 71 ++++++++++++++++----
 Documentation/technical/multi-pack-index.txt | 10 +--
 2 files changed, 60 insertions(+), 21 deletions(-)
Documentation/technical/bitmap-format.txt +57 -14
index f8c18a0f7a..04b3ec2178 100644
@@ -1,6 +1,44 @@
 GIT bitmap v1 format
 ====================
 
+== Pack and multi-pack bitmaps
+
+Bitmaps store reachability information about the set of objects in a packfile,
+or a multi-pack index (MIDX). The former is defined obviously, and the latter is
+defined as the union of objects in packs contained in the MIDX.
+
+A bitmap may belong to either one pack, or the repository's multi-pack index (if
+it exists). A repository may have at most one bitmap.
+
+An object is uniquely described by its bit position within a bitmap:
+
+	- If the bitmap belongs to a packfile, the __n__th bit corresponds to
+	the __n__th object in pack order. For a function `offset` which maps
+	objects to their byte offset within a pack, pack order is defined as
+	follows:
+
+		o1 <= o2 <==> offset(o1) <= offset(o2)
+
+	- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the
+	__n__th object in MIDX order. With an additional function `pack` which
+	maps objects to the pack they were selected from by the MIDX, MIDX order
+	is defined as follows:
+
+		o1 <= o2 <==> pack(o1) <= pack(o2) /\ offset(o1) <= offset(o2)
+
+	The ordering between packs is done according to the MIDX's .rev file.
+	Notably, the preferred pack sorts ahead of all other packs.
+
+The on-disk representation (described below) of a bitmap is the same regardless
+of whether or not that bitmap belongs to a packfile or a MIDX. The only
+difference is the interpretation of the bits, which is described above.
+
+Certain bitmap extensions are supported (see: Appendix B). No extensions are
+required for bitmaps corresponding to packfiles. For bitmaps that correspond to
+MIDXs, both the bit-cache and rev-cache extensions are required.
+
+== On-disk format
+
 	- A header appears at the beginning:
 
 		4-byte signature: {'B', 'I', 'T', 'M'}
@@ -14,17 +52,19 @@ GIT bitmap v1 format
 			The following flags are supported:
 
 			- BITMAP_OPT_FULL_DAG (0x1) REQUIRED
-			This flag must always be present. It implies that the bitmap
-			index has been generated for a packfile with full closure
-			(i.e. where every single object in the packfile can find
-			 its parent links inside the same packfile). This is a
-			requirement for the bitmap index format, also present in JGit,
-			that greatly reduces the complexity of the implementation.
+			This flag must always be present. It implies that the
+			bitmap index has been generated for a packfile or
+			multi-pack index (MIDX) with full closure (i.e. where
+			every single object in the packfile/MIDX can find its
+			parent links inside the same packfile/MIDX). This is a
+			requirement for the bitmap index format, also present in
+			JGit, that greatly reduces the complexity of the
+			implementation.
 
 			- BITMAP_OPT_HASH_CACHE (0x4)
 			If present, the end of the bitmap file contains
 			`N` 32-bit name-hash values, one per object in the
-			pack. The format and meaning of the name-hash is
+			pack/MIDX. The format and meaning of the name-hash is
 			described below.
 
 		4-byte entry count (network byte order)
@@ -33,7 +73,8 @@ GIT bitmap v1 format
 
 		20-byte checksum
 
-			The SHA1 checksum of the pack this bitmap index belongs to.
+			The SHA1 checksum of the pack/MIDX this bitmap index
+			belongs to.
 
 	- 4 EWAH bitmaps that act as type indexes
 
@@ -50,7 +91,7 @@ GIT bitmap v1 format
 			- Tags
 
 		In each bitmap, the `n`th bit is set to true if the `n`th object
-		in the packfile is of that type.
+		in the packfile or multi-pack index is of that type.
 
 		The obvious consequence is that the OR of all 4 bitmaps will result
 		in a full set (all bits set), and the AND of all 4 bitmaps will
@@ -62,8 +103,9 @@ GIT bitmap v1 format
 		Each entry contains the following:
 
 		- 4-byte object position (network byte order)
-			The position **in the index for the packfile** where the
-			bitmap for this commit is found.
+			The position **in the index for the packfile or
+			multi-pack index** where the bitmap for this commit is
+			found.
 
 		- 1-byte XOR-offset
 			The xor offset used to compress this bitmap. For an entry
@@ -146,10 +188,11 @@ Name-hash cache
 ---------------
 
 If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains
-a cache of 32-bit values, one per object in the pack. The value at
+a cache of 32-bit values, one per object in the pack/MIDX. The value at
 position `i` is the hash of the pathname at which the `i`th object
-(counting in index order) in the pack can be found.  This can be fed
-into the delta heuristics to compare objects with similar pathnames.
+(counting in index or multi-pack index order) in the pack/MIDX can be found.
+This can be fed into the delta heuristics to compare objects with similar
+pathnames.
 
 The hash algorithm used is:
 
Documentation/technical/multi-pack-index.txt +3 -7
index fb688976c4..1a73c3ee20 100644
@@ -71,14 +71,10 @@ Future Work
   still reducing the number of binary searches required for object
   lookups.
 
-- The reachability bitmap is currently paired directly with a single
-  packfile, using the pack-order as the object order to hopefully
-  compress the bitmaps well using run-length encoding. This could be
-  extended to pair a reachability bitmap with a multi-pack-index. If
-  the multi-pack-index is extended to store a "stable object order"
+- If the multi-pack-index is extended to store a "stable object order"
   (a function Order(hash) = integer that is constant for a given hash,
-  even as the multi-pack-index is updated) then a reachability bitmap
-  could point to a multi-pack-index and be updated independently.
+  even as the multi-pack-index is updated) then MIDX bitmaps could be
+  updated independently of the MIDX.
 
 - Packfiles can be marked as "special" using empty files that share
   the initial name but replace ".pack" with ".keep" or ".promisor".
Signature · 1 line
2.31.1.163.ga65ce7f831
#178 lore raw

PATCH v4 5/25 midx: clear auxiliary .rev after replacing the MIDX

In reply to Taylor Blau

When writing a new multi-pack index, write_midx_internal() attempts to clean up any auxiliary files (currently just the MIDX's .rev file, but soon to include a .bitmap, too) corresponding to the MIDX it's replacing.

This step should happen after the new MIDX is written into place, since doing so beforehand means that the old MIDX could be read without its corresponding .rev file.

---
 midx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
midx.c +2 -1
index 321c6fdd2f..73b199ca49 100644
@@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	if (flags & MIDX_WRITE_REV_INDEX)
 		write_midx_reverse_index(midx_name, midx_hash, &ctx);
-	clear_midx_files_ext(the_repository, ".rev", midx_hash);
 
 	commit_lock_file(&lk);
 
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+
 cleanup:
 	for (i = 0; i < ctx.nr; i++) {
 		if (ctx.info[i].p) {
Signature · 1 line
2.31.1.163.ga65ce7f831
#179 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 19 lines · Taylor Blau

When writing a new multi-pack index, write_midx_internal() attempts to clean up any auxiliary files (currently just the MIDX's .rev file, but soon to include a .bitmap, too) corresponding to the MIDX it's replacing.

This step should happen after the new MIDX is written into place, since doing so beforehand means that the old MIDX could be read without its corresponding .rev file.

---
 midx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
midx.c +0 -0
index 321c6fdd2f..73b199ca49 100644

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 1092

Quoted 8 lines
midx.c +1 -1
@@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	if (flags & MIDX_WRITE_REV_INDEX)
 		write_midx_reverse_index(midx_name, midx_hash, &ctx);
-	clear_midx_files_ext(the_repository, ".rev", midx_hash);
 
 	commit_lock_file(&lk);
 
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);

This needs to take object_dir into account, no?

There are a few more calls to clear_midx_files_ext() added in 15/25 and they use the_repository, too.

Quoted 3 lines
midx.c +0 -0
 cleanup:
 	for (i = 0; i < ctx.nr; i++) {
 		if (ctx.info[i].p) {
#180 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 26 lines · Junio C Hamano
Quoted 24 lines · Taylor Blau

When writing a new multi-pack index, write_midx_internal() attempts to clean up any auxiliary files (currently just the MIDX's .rev file, but soon to include a .bitmap, too) corresponding to the MIDX it's replacing.

This step should happen after the new MIDX is written into place, since doing so beforehand means that the old MIDX could be read without its corresponding .rev file.

---
 midx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
midx.c +0 -1
index 321c6fdd2f..73b199ca49 100644
@@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	if (flags & MIDX_WRITE_REV_INDEX)
 		write_midx_reverse_index(midx_name, midx_hash, &ctx);
-	clear_midx_files_ext(the_repository, ".rev", midx_hash);
 
Quoted 5 lines

midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 1092

Quoted 3 lines
midx.c +1 -0
 	commit_lock_file(&lk);
 
+	clear_midx_files_ext(the_repository, ".rev", midx_hash);

This needs to take object_dir into account, no?

Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct repository' until we pick up [1].

I asked for some changes in the latest version that Johannes posted. So I'd be OK to live with this behavior for the time being, and then I can send another patch on top that fixes the new and existing callers (incorporating [1] with some new tests).

Or we can hold one up and expedite the other. I would suggest that we pick up this series to next if you're otherwise happy with it and then I can send the trivial fixes on top.

Thanks,
Taylor

[1]: https://lore.kernel.org/git/[email protected]/

#181 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 4 lines · Taylor Blau
Quoted 1 line

This needs to take object_dir into account, no?

Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct repository' until we pick up [1].

I was hoping that [1] will become part of this series as a trivial clean-up and bugfix, perhaps in its early part.

#182 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 9 lines · Junio C Hamano
Quoted 4 lines · Taylor Blau
Quoted 1 line

This needs to take object_dir into account, no?

Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct repository' until we pick up [1].

I was hoping that [1] will become part of this series as a trivial clean-up and bugfix, perhaps in its early part.

Sure, that works even better. I'll send a reroll incorporating it as soon as I finish re-testing.

Thanks,
Taylor

#183 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 5 lines · Taylor Blau
Quoted 4 lines · Junio C Hamano
Quoted 2 lines · Taylor Blau
Quoted 1 line

This needs to take object_dir into account, no?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct repository' until we pick up [1].

I was hoping that [1] will become part of this series as a trivial clean-up and bugfix, perhaps in its early part.

Sure, that works even better. I'll send a reroll incorporating it as soon as I finish re-testing.

Hmm, this got me wondering: what should be the behavior be when we run the multi-pack-index command outside of a Git repository? For example, in patch 15 we do:

    for (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {
      if (!strcmp(object_dir, cur->object_dir)) {
        ctx.m = cur;
        break;
      }
    }

but obviously get_multi_pack_index(the_repository) will fail when there is no repository to begin with.

The real question is whether we should allow munging arbitrary MIDXs, or restrict the ones we can modify to just our alternates. If we allow the former, then that code needs to be tweaked. If not, and we only allow touching alternates, then we need to require a repository for the multi-pack-index builtin.

Thanks,
Taylor

#184 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

In reply to Taylor Blau
Quoted 5 lines · Taylor Blau
Quoted 4 lines · Junio C Hamano
Quoted 2 lines · Taylor Blau
Quoted 1 line

This needs to take object_dir into account, no?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Yes and no; clear_midx_files_ext() still takes a pointer to a 'struct repository' until we pick up [1].

I was hoping that [1] will become part of this series as a trivial clean-up and bugfix, perhaps in its early part.

Sure, that works even better. I'll send a reroll incorporating it as soon as I finish re-testing.

FWIW, here is what I have somewhere in 'seen' where two topics meet.

midx.c +0 -0
diff --cc midx.c
index c0209751b5,4574e6d411..0000000000
@@@ -1090,6 -1351,9 +1351,9 @@@ static int write_midx_internal(const ch
  
  	commit_lock_file(&lk);
  
 -	clear_midx_files_ext(the_repository, ".bitmap", midx_hash);
 -	clear_midx_files_ext(the_repository, ".rev", midx_hash);
++	clear_midx_files_ext(object_dir, ".bitmap", midx_hash);
++	clear_midx_files_ext(object_dir, ".rev", midx_hash);
+ 
  cleanup:
  	for (i = 0; i < ctx.nr; i++) {
  		if (ctx.info[i].p) {
@@@ -1165,7 -1429,8 +1429,8 @@@ void clear_midx_file(struct repository 
  	if (remove_path(midx))
  		die(_("failed to clear multi-pack-index at %s"), midx);
  
 -	clear_midx_files_ext(r, ".bitmap", NULL);
 -	clear_midx_files_ext(r, ".rev", NULL);
++	clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL);
 +	clear_midx_files_ext(r->objects->odb->path, ".rev", NULL);
  
  	free(midx);
  }
#185 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 1 line · Junio C Hamano

FWIW, here is what I have somewhere in 'seen' where two topics meet.

Oops, one change missed.

midx.c +0 -0
diff --cc midx.c
index c0209751b5,4574e6d411..0000000000
@@@ -947,11 -1136,29 +1136,29 @@@ static int write_midx_internal(const ch
  	for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);
  	stop_progress(&ctx.progress);
  
- 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)
- 		goto cleanup;
+ 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {
+ 		struct bitmap_index *bitmap_git;
+ 		int bitmap_exists;
+ 		int want_bitmap = flags & MIDX_WRITE_BITMAP;
+ 
+ 		bitmap_git = prepare_midx_bitmap_git(the_repository, ctx.m);
+ 		bitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);
+ 		free_bitmap_index(bitmap_git);
+ 
+ 		if (bitmap_exists || !want_bitmap) {
+ 			/*
+ 			 * The correct MIDX already exists, and so does a
+ 			 * corresponding bitmap (or one wasn't requested).
+ 			 */
+ 			if (!want_bitmap)
 -				clear_midx_files_ext(the_repository, ".bitmap",
++				clear_midx_files_ext(object_dir, ".bitmap",
+ 						     NULL);
+ 			goto cleanup;
+ 		}
+ 	}
  
- 	ctx.preferred_pack_idx = -1;
  	if (preferred_pack_name) {
+ 		int found = 0;
  		for (i = 0; i < ctx.nr; i++) {
  			if (!cmp_idx_or_pack_name(preferred_pack_name,
  						  ctx.info[i].pack_name)) {
@@@ -1090,6 -1351,9 +1351,9 @@@
  
  	commit_lock_file(&lk);
  
 -	clear_midx_files_ext(the_repository, ".bitmap", midx_hash);
 -	clear_midx_files_ext(the_repository, ".rev", midx_hash);
++	clear_midx_files_ext(object_dir, ".bitmap", midx_hash);
++	clear_midx_files_ext(object_dir, ".rev", midx_hash);
+ 
  cleanup:
  	for (i = 0; i < ctx.nr; i++) {
  		if (ctx.info[i].p) {
@@@ -1165,7 -1429,8 +1429,8 @@@ void clear_midx_file(struct repository 
  	if (remove_path(midx))
  		die(_("failed to clear multi-pack-index at %s"), midx);
  
 -	clear_midx_files_ext(r, ".bitmap", NULL);
 -	clear_midx_files_ext(r, ".rev", NULL);
++	clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL);
 +	clear_midx_files_ext(r->objects->odb->path, ".rev", NULL);
  
  	free(midx);
  }
#186 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 5 lines · Junio C Hamano
Quoted 1 line · Junio C Hamano

FWIW, here is what I have somewhere in 'seen' where two topics meet.

Oops, one change missed.

Thanks; that matches my own resolution. I noticed that it does fail the new test in t5319, since writing a MIDX wants to make sure that we are only touching an alternate's object directory (which will fail if we are running git multi-pack-index from outside of a repository).

My opinion is that we should require being inside of a repository to run the MIDX builtin. Otherwise we're allowing that command to modify any old MIDX, which doesn't make sense.

I think we probably need a single unifying topic, so I'm happy if you want to discard one of our two topics from seen in the meantime.

Thanks,
Taylor

#187 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 12 lines · Taylor Blau
Quoted 5 lines · Junio C Hamano
Quoted 1 line · Junio C Hamano

FWIW, here is what I have somewhere in 'seen' where two topics meet.

Oops, one change missed.

Thanks; that matches my own resolution. I noticed that it does fail the new test in t5319, since writing a MIDX wants to make sure that we are only touching an alternate's object directory (which will fail if we are running git multi-pack-index from outside of a repository).

Quoted 6 lines

My opinion is that we should require being inside of a repository to run the MIDX builtin. Otherwise we're allowing that command to modify any old MIDX, which doesn't make sense.

I think we probably need a single unifying topic, so I'm happy if you want to discard one of our two topics from seen in the meantime.

It seems that the *.rev test (probably added by the other topic that is a single patch fix) fails under sha256 hash. I am not going to dig it any further myself, but for the interested, CI breakage is here:

https://github.com/git/git/runs/3440068613?check_suite_focus=true#step:5:1219

Thanks.

#188 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 8 lines · Junio C Hamano

It seems that the *.rev test (probably added by the other topic that is a single patch fix) fails under sha256 hash. I am not going to dig it any further myself, but for the interested, CI breakage is here:

https://github.com/git/git/runs/3440068613?check_suite_focus=true#step:5:1219

Thanks.

I saw the same error myself when integrating that patch into my series. I discussed it more in [1], but the failure is basically caused by the midx code using the_hash_algo even when operating in a different repository via --object-dir.

If the_hash_algo doesn't match (as is the case when using --object-dir to point at a SHA-256 repository when invoking the builtin from a repository using SHA-1 or outside of a repository altogether), then we'll fail when trying to open the pack indexes.

Thanks,
Taylor

[1]: https://lore.kernel.org/git/[email protected]/

#189 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 15 lines · Taylor Blau
Quoted 8 lines · Junio C Hamano

It seems that the *.rev test (probably added by the other topic that is a single patch fix) fails under sha256 hash. I am not going to dig it any further myself, but for the interested, CI breakage is here:

https://github.com/git/git/runs/3440068613?check_suite_focus=true#step:5:1219

Thanks.

I saw the same error myself when integrating that patch into my series. I discussed it more in [1], but the failure is basically caused by the midx code using the_hash_algo even when operating in a different repository via --object-dir.

Quoted 4 lines

If the_hash_algo doesn't match (as is the case when using --object-dir to point at a SHA-256 repository when invoking the builtin from a repository using SHA-1 or outside of a repository altogether), then we'll fail when trying to open the pack indexes.

My recollection is that "--object-dir" is mostly about the alternate odb usecase---am I correct? It is unfortunate that we didn't start with "alternate repository" and said "we only care about the objects in the object store they have, and we do not have to care what refs they point into their object database or what configuration they have" instead.

I wonder if it is safe to assume that in practice a directory given to the "--object-dir" option is always the "objects" subdirectory in a repository, and it is an error if there is no "config" file next to the directory. Then, we could check ../config relative to the given directory and error out if they use different hash.

I do not recall offhand how careful link_alt_odb_entries() is, but I suspect it isn't at all (back when I invented it, there weren't need for configuration to switch between hashes, and since then I do not recall seeing any heavy update to the alternate odb code). Perhaps we should tighten it so that we check the accompanying "config" file first and ignore the entry with incompatible "hash" (and we may later discover other trait on a repository that is incompatible with the current one)?

Thanks.

#190 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 2 lines · Junio C Hamano

My recollection is that "--object-dir" is mostly about the alternate odb usecase---am I correct?

That matches my understanding. The documentation refers to the value of this flag as <alt>, making me think that supporting non-alternates is a historical accident.

Quoted 5 lines

I wonder if it is safe to assume that in practice a directory given to the "--object-dir" option is always the "objects" subdirectory in a repository, and it is an error if there is no "config" file next to the directory. Then, we could check ../config relative to the given directory and error out if they use different hash.

Maybe... although I have to admit to not being very excited about it. Is the idea to read ../config to try and check for any incompatibilities between the in-core state and the target repository's settings? If so, this seems like a recipe for catching bugs too late.

For e.g., catching the_hash_algo != target_repository->hash would definitely squash the bug you saw when integrating, but we would have to remember to update this spot later on if, say, the target repository started using a different reference storage backend (since bitmap generation necessarily iterates the references to figure out which commits should receive coverage).

Quoted 8 lines

I do not recall offhand how careful link_alt_odb_entries() is, but I suspect it isn't at all (back when I invented it, there weren't need for configuration to switch between hashes, and since then I do not recall seeing any heavy update to the alternate odb code). Perhaps we should tighten it so that we check the accompanying "config" file first and ignore the entry with incompatible "hash" (and we may later discover other trait on a repository that is incompatible with the current one)?

Or are you saying you're concerned about an alternates chain which don't all use the same object format?

If the former, then I would say:

"Supporting arbitrary --object-dir when invoked from outside a repository is a bug that happened to not cause any problems, but is surprising, error-prone, and should fall outside of the burden of backwards compatibility, so we should get rid of it."

If the latter, then I agree we could and should do better at detecting it and providing a helpful error message, but I don't see how doing so now or later would affect this series. Even if we just disallow --object-dir pointing at a non-alternate repository, we would still have the issue of having alternate chains which don't all have the same object format.

So that makes me feel like the latter is a problem outside of this series that can be dealt with later.

I'm admittedly a little unsure of how to progress here. Given that this series has received positive review over the complicated parts, it seems that it is getting stuck on how to deal with --object-dir, especially when invoked outside of a Git repository. My inclination would be to send a new version that simply requires the MIDX builtin to be run from within a repository (as well as the cleanups from Johannes).

Does that seem like a good direction forward to you? If not, let me know if there's another issue that we should deal with first and I'd be happy to start there.

Quoted 1 line

Thanks.

Thanks,
Taylor

#191 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 4 lines · Taylor Blau

now or later would affect this series. Even if we just disallow --object-dir pointing at a non-alternate repository, we would still have the issue of having alternate chains which don't all have the same object format.

Exactly. That is why I feel that it probably needs to be dealt with before doing anything else. The alternate mechanism pulling in an object store that uses incompatible hash algo would break not just the multi-pack-index but probably the basic object access layer as well, which would be more grave problem, no?

Quoted 3 lines

My inclination would be to
send a new version that simply requires the MIDX builtin to be run from within a repository (as well as the cleanups from Johannes).

Sounds like a good first step.

#192 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 7 lines · Junio C Hamano
Quoted 4 lines · Taylor Blau

now or later would affect this series. Even if we just disallow --object-dir pointing at a non-alternate repository, we would still have the issue of having alternate chains which don't all have the same object format.

Quoted 5 lines

Exactly. That is why I feel that it probably needs to be dealt with before doing anything else. The alternate mechanism pulling in an object store that uses incompatible hash algo would break not just the multi-pack-index but probably the basic object access layer as well, which would be more grave problem, no?

Yeah; it does. Maybe I'm holding it wrong (and brian, cc'd, can help me), but this is an easy way to see the problem:

git init repo
git init alternate

  git -C repo commit --allow-empty -m foo
  ( cd repo/.git/objects && pwd ) >alternate/.git/objects/info/alternates
  git -C alternate rev-list --objects --alternate-refs

which will produce:

    $ git rev-list --objects --alternate-refs
    warning: invalid line while parsing alternate refs: <sha256 id>

But I don't know if I quite understand your "probably needs to be dealt with before doing anything else". I think we can proceed with this series and deal with the alternate object-format thing separately, no?

Thanks,
Taylor

#193 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 24 lines · Taylor Blau
Quoted 7 lines · Junio C Hamano
Quoted 4 lines · Taylor Blau

now or later would affect this series. Even if we just disallow --object-dir pointing at a non-alternate repository, we would still have the issue of having alternate chains which don't all have the same object format.

Quoted 5 lines

Exactly. That is why I feel that it probably needs to be dealt with before doing anything else. The alternate mechanism pulling in an object store that uses incompatible hash algo would break not just the multi-pack-index but probably the basic object access layer as well, which would be more grave problem, no?

Yeah; it does. Maybe I'm holding it wrong (and brian, cc'd, can help me), but this is an easy way to see the problem:

git init repo
git init alternate

  git -C repo commit --allow-empty -m foo
  ( cd repo/.git/objects && pwd ) >alternate/.git/objects/info/alternates
  git -C alternate rev-list --objects --alternate-refs
Quoted 8 lines

which will produce:

    $ git rev-list --objects --alternate-refs
    warning: invalid line while parsing alternate refs: <sha256 id>

But I don't know if I quite understand your "probably needs to be dealt with before doing anything else". I think we can proceed with this series and deal with the alternate object-format thing separately, no?

Yeah, this is a possible problem. You can also see it when using git index-pack outside of a repository with an incorrect --object-format option.

I'm not sure how folks want to deal with that; I'm just fine saying, "Well, don't do that," but other folks may have different opinions.

Signature · 2 lines
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
#194 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 6 lines · brian m. carlson

Yeah, this is a possible problem. You can also see it when using git index-pack outside of a repository with an incorrect --object-format option.

I'm not sure how folks want to deal with that; I'm just fine saying, "Well, don't do that," but other folks may have different opinions.

OK, so if we go back to the original breakage of the test script that triggered this discussion, the right solution would be to make sure both test repositories/object stores are prepared with the algorithm specified with GIT_TEST_DEFAULT_HASH?

Thanks.

#195 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 6 lines · Junio C Hamano
Quoted 4 lines · brian m. carlson

Yeah, this is a possible problem. You can also see it when using git index-pack outside of a repository with an incorrect --object-format option.

Quoted 7 lines
Quoted 2 lines

I'm not sure how folks want to deal with that; I'm just fine saying, "Well, don't do that," but other folks may have different opinions.

OK, so if we go back to the original breakage of the test script that triggered this discussion, the right solution would be to make sure both test repositories/object stores are prepared with the algorithm specified with GIT_TEST_DEFAULT_HASH?

Just to make sure do you still see this as a separate issue from running the midx builtin outside of a repository?

I.e., if we require the midx builtin to be run in a repository, it side-steps this issue (but presumably not completely, and so we should deal with both eventually). I want to make sure that I'm on the same page before I drop 25+ emails on the list.

Thanks,
Taylor

#196 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 10 lines · Taylor Blau
Quoted 9 lines · Junio C Hamano
Quoted 6 lines · brian m. carlson

Yeah, this is a possible problem. You can also see it when using git index-pack outside of a repository with an incorrect --object-format option.

I'm not sure how folks want to deal with that; I'm just fine saying, "Well, don't do that," but other folks may have different opinions.

Quoted 7 lines
Quoted 4 lines

OK, so if we go back to the original breakage of the test script that triggered this discussion, the right solution would be to make sure both test repositories/object stores are prepared with the algorithm specified with GIT_TEST_DEFAULT_HASH?

Just to make sure do you still see this as a separate issue from running the midx builtin outside of a repository?

Adding my two cents: yes, I think it most definitely should be a separate issue. As you demonstrated, differing config between alternates and repos that point to them is not specific to the midx code. I agree with brian's "well, don't do that". But _if_ we want to try to behave better in such a case, whatever we changes we make would then naturally apply to the midx code as well.

The two midx-specific things we have to care about are:

- is it OK for the midx command to refuse to operate when we are not in a repository at all? I think yes; we can't even know which hash is being used, along with who knows what other lurking complications.

- is it OK to restrict the midx command's --object-dir to only operate on a directory which is an alternate of the current repo? I think yes again. If it _isn't_ related, we have all the lurking problems from the first point, but even worse (because we use config, refs, and other information from our current repo with the _totally unrelated_ object dir).

So I'm all in favor of locking those down now before things get any more complicated. If we later want to make the object store more aware of of differences between alternates and the main store (like say, the object hash in use), then we could consider loosening using the same mechanism.

-Peff

#197 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

In reply to Taylor Blau
Quoted 10 lines · Taylor Blau
Quoted 9 lines · Junio C Hamano
Quoted 6 lines · brian m. carlson

Yeah, this is a possible problem. You can also see it when using git index-pack outside of a repository with an incorrect --object-format option.

I'm not sure how folks want to deal with that; I'm just fine saying, "Well, don't do that," but other folks may have different opinions.

Quoted 7 lines
Quoted 4 lines

OK, so if we go back to the original breakage of the test script that triggered this discussion, the right solution would be to make sure both test repositories/object stores are prepared with the algorithm specified with GIT_TEST_DEFAULT_HASH?

Just to make sure do you still see this as a separate issue from running the midx builtin outside of a repository?

They are separate issues, but the .midx issue has a small overlap with the much bigger "do not mix repositories and object stores with different hashes" issue.

The users of raw object stores (e.g. $GIT_OBJECT_DIRECTORIES, "--object-dir", there may be others) need to be updated so that the code paths involved can reliably learn what hash algorithm is used and other traits that may not be available in the object store alone (e.g. refs might be relevant if the using code needs to learn which objects are still reachable) for the latter. It would need a couple of things that are fairly isolated to solve, I would imagine:

(1) convention to either tie a raw object store with its repository or declare a raw object store is unusable because "other traits" are not found for it.

(2) given a repository, inspect it and decide if it is "compatible" with the current repository.

(3) update code paths involved in prepare_alt_odb() to use (1) and
(2) to inspect and reject incompatible object store as alternate.

And once we have that, "git multi-pack-index --object-dir=X" can use
(1) and (2) for the same "Is this other object store compatible with the current repository?" check, no?

The other side of the coin is that midx needs to do equivalents of
(1) and (2) anyway, and the required amount of the work for (3) smells a lot smaller than work for (1) and (2). (3) may be just a matter of "add a call to is_odb_compatible(dir) for the directory being added as an alt odb", and the same single validation call may be all it needs on the --object-dir argument on the midx side.

I think it makes sense for the midx command to require being in a repository to run (to establish what "the current repository" is) and insist on the other object store given with --object-dir to be "compatible" with the current repository (i.e. the same hash algorithm, there may be others). I am a bit fuzzy why we want it to be already our alternate.

Thanks.

#198 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 17 lines · Junio C Hamano
Quoted 15 lines · Taylor Blau
Quoted 13 lines · Junio C Hamano
Quoted 6 lines · brian m. carlson

Yeah, this is a possible problem. You can also see it when using git index-pack outside of a repository with an incorrect --object-format option.

I'm not sure how folks want to deal with that; I'm just fine saying, "Well, don't do that," but other folks may have different opinions.

OK, so if we go back to the original breakage of the test script that triggered this discussion, the right solution would be to make sure both test repositories/object stores are prepared with the algorithm specified with GIT_TEST_DEFAULT_HASH?

Quoted 6 lines
Quoted 2 lines

Just to make sure do you still see this as a separate issue from running the midx builtin outside of a repository?

They are separate issues, but the .midx issue has a small overlap with the much bigger "do not mix repositories and object stores with different hashes" issue.

OK, good. Everything you wrote below (which I snipped off in my reply) makes sense to me, and seems like a worthwhile direction to pursue outside of this series, especially as more users start using sha256 repositories.

Quoted 6 lines

I think it makes sense for the midx command to require being in a repository to run (to establish what "the current repository" is) and insist on the other object store given with --object-dir to be "compatible" with the current repository (i.e. the same hash algorithm, there may be others). I am a bit fuzzy why we want it to be already our alternate.

I don't think there's any strict requirement to the other repository being our alternate, other than touching arbitrary repositories is a surprising behavior that appears (to me, at least) to be inconsistent with the rest of Git.

After (the rerolled version of) this series, we'll be in a state where:

- git multi-pack-index will not run when outside of a Git repository.
- The --object-dir argument will only recognize object directories belonging to an alternate of the current repository.
- Using --object-dir to point to a repository which uses a different hash than the repository in the current working directory will continue to not work (as was the case before this series).

I think(?) that there is consensus for that approach, so patches incoming...

Quoted 1 line

Thanks.

(Thank you, by the way, for clarifying this all in so much detail. I would much rather just have code to talk about, but it feels particularly important to be on the same page beforehand in this instance, since there is *so much* code, and this discussion is centered around so little of it).

Thanks,
Taylor

#199 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 9 lines · Taylor Blau

After (the rerolled version of) this series, we'll be in a state where:

- git multi-pack-index will not run when outside of a Git repository.
- The --object-dir argument will only recognize object directories belonging to an alternate of the current repository.
- Using --object-dir to point to a repository which uses a different hash than the repository in the current working directory will continue to not work (as was the case before this series).

Hmph, re-reading the document for midx:

--object-dir=<dir>::
Use given directory for the location of Git objects. We check <dir>/packs/multi-pack-index for the current MIDX file, and <dir>/packs for the pack-files to index.

why does it matter if we are in a repository in the first place? It's not like we combine the objects from the specified object dir and our local object store (if that were the case, these two object stores must be compatible).

How old is --object-dir option and how widely is it used? Can we just remove it and have users go to the repository that uses it as its object store with "git -C <there>" mechanism, or have we come too far with this (apparently broken) design to make such a fix infeasible?

Thanks.

#200 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 19 lines · Junio C Hamano
Quoted 9 lines · Taylor Blau

After (the rerolled version of) this series, we'll be in a state where:

- git multi-pack-index will not run when outside of a Git repository.
- The --object-dir argument will only recognize object directories belonging to an alternate of the current repository.
- Using --object-dir to point to a repository which uses a different hash than the repository in the current working directory will continue to not work (as was the case before this series).

Hmph, re-reading the document for midx:

--object-dir=<dir>::
Use given directory for the location of Git objects. We check <dir>/packs/multi-pack-index for the current MIDX file, and <dir>/packs for the pack-files to index.

Quoted 4 lines

why does it matter if we are in a repository in the first place? It's not like we combine the objects from the specified object dir and our local object store (if that were the case, these two object stores must be compatible).

It shouldn't matter, but the use-case is described in [1] by Stolee. He explains it in detail, but I do think we have to live with --object-dir in one way or another. He does say it'd be OK to only be able to invoke it from within a repository, and to only be able to reference alternates, though.

Thanks,
Taylor

[1]: https://lore.kernel.org/git/[email protected]/

#201 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

In reply to Taylor Blau
Quoted 7 lines · Taylor Blau
Quoted 2 lines · Junio C Hamano

My recollection is that "--object-dir" is mostly about the alternate odb usecase---am I correct?

That matches my understanding. The documentation refers to the value of this flag as <alt>, making me think that supporting non-alternates is a historical accident.

Yes, supporting non-alternates is a historical accident. Supporting alternates that are not actually the core object database of a full repository is on purpose.

So, hopefully the remaining discussion that I am seeing can be solved by a decision such as:

"If we add the restriction that the builtin always runs with a
repository and --object-dir always points to its objects dir or one of its registered alternates, then we have access to a local config file to learn how to interpret that object directory."

Quoted 5 lines
Quoted 5 lines

I wonder if it is safe to assume that in practice a directory given to the "--object-dir" option is always the "objects" subdirectory in a repository, and it is an error if there is no "config" file next to the directory. Then, we could check ../config relative to the given directory and error out if they use different hash.

I would say that is not always the case, and we should not error out.

I think taking a look to see if ../config exists to use the data might be helpful for some cases, but should not be a blocker for completing the requested operation. The config from the non-alternate repo should be sufficient for this (somewhat strange) case.

Quoted 10 lines

I'm admittedly a little unsure of how to progress here. Given that this series has received positive review over the complicated parts, it seems that it is getting stuck on how to deal with --object-dir, especially when invoked outside of a Git repository. My inclination would be to send a new version that simply requires the MIDX builtin to be run from within a repository (as well as the cleanups from Johannes).

Does that seem like a good direction forward to you? If not, let me know if there's another issue that we should deal with first and I'd be happy to start there.

I think it is sensible to restrict 'git multi-pack-index' to run inside a repository on its own merits. It happens to also solve some tricky problems that have come up since its creation.

Sorry I'm so late to this thread. I gave most of the messages in this chain a quick read and this seemed like the best place to chime in. Hopefully this isn't too much of a re-tread of things covered elsewhere.

Thanks,
-Stolee

#202 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 11 lines · Derrick Stolee

Yes, supporting non-alternates is a historical accident. Supporting alternates that are not actually the core object database of a full repository is on purpose.

So, hopefully the remaining discussion that I am seeing can be solved by a decision such as:

"If we add the restriction that the builtin always runs with a
repository and --object-dir always points to its objects dir or one of its registered alternates, then we have access to a local config file to learn how to interpret that object directory."

I left a similar comment in the other part of the thread. :)

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

I wonder if it is safe to assume that in practice a directory given to the "--object-dir" option is always the "objects" subdirectory in a repository, and it is an error if there is no "config" file next to the directory. Then, we could check ../config relative to the given directory and error out if they use different hash.

Quoted 6 lines

I would say that is not always the case, and we should not error out.

I think taking a look to see if ../config exists to use the data might be helpful for some cases, but should not be a blocker for completing the requested operation. The config from the non-alternate repo should be sufficient for this (somewhat strange) case.

Yes, agreed. We have long supported these kind of "bare" alternates, and I wouldn't be surprised if they are in wide use (though I do wonder how folks actually modify them, since most commands that touch objects really do want to be in a repository).

In other cases where we may benefit from their being a containing repo (e.g., accessing the ref tips of the alternate), we speculatively look at ".." and see if there are any refs. See refs_from_alternate_cb()[0].

The natural extension for the hash-format problem would probably be to call check_repository_format_gently() on the parent directory of the alternate-objects dir. If it succeeds, then we can pull out the hash_algo parameter from its repository_format struct. And if not, then we just assume it matches the main repo.

But I suspect all of this is moot for now, beyond being able to return a nicer error message. The rest of the code is not at all ready to handle packs with two different hashes in the same process. And I suspect it would take a reasonable amount of refactoring to make it so. If somebody wants to work on that, I won't stop them, but I kind of doubt it is worth anybody's time.

[0] Looking at refs_from_alternate_cb(), I did wonder if it would work at all with a reftable alternate, but I suspect it would. I think we ended up still having a "refs/" directory in that case, so we'd recognize it as a repo (though really, it ought to be using is_git_directory() instead of its hacky check). And then we farm out the actual ref iteration to a separate for-each-ref process, passing along --git-dir, which will read that alternate repo's config. So it should Just Work, even with a different ref backend. It's almost certainly broken if the hash algorithms don't match, though, because we'd get oddly sized results from for-each-ref's output.

That's all just interesting tangent, though. :)

-Peff

#203 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 9 lines · Jeff King
Quoted 4 lines

I think taking a look to see if ../config exists to use the data might be helpful for some cases, but should not be a blocker for completing the requested operation. The config from the non-alternate repo should be sufficient for this (somewhat strange) case.

Yes, agreed. We have long supported these kind of "bare" alternates, and I wouldn't be surprised if they are in wide use (though I do wonder how folks actually modify them, since most commands that touch objects really do want to be in a repository).

I kind of find the above two somewhat surprising, but I am willing to go with the less safer option if that is what people want.

It has been perfectly OK in the pre-alternative-hash-algorithms world, but we no longer live in such a world, so we'd need to come up with a way to keep using alternates in a safer way.

I do not see the reasoning behind "should not be a blocker" from Derrick substantiated. What's the reason why that raw object store cannot come from an existing repository, and what's the benefit we get from not having to have a repository there?

Quoted 9 lines

The natural extension for the hash-format problem would probably be to call check_repository_format_gently() on the parent directory of the alternate-objects dir. If it succeeds, then we can pull out the hash_algo parameter from its repository_format struct. And if not, then we just assume it matches the main repo.

But I suspect all of this is moot for now, beyond being able to return a nicer error message. The rest of the code is not at all ready to handle packs with two different hashes in the same process.

I do not think it is all that urgent to make it possible for packs with different algorithms to be used. It is sufficient to _ignore_ (or error out) configured odb that is incompatible with the current repository.

Thanks.

#204 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 4 lines · Junio C Hamano

I do not see the reasoning behind "should not be a blocker" from Derrick substantiated. What's the reason why that raw object store cannot come from an existing repository, and what's the benefit we get from not having to have a repository there?

I also didn't find the reasoning spelled out in his response, but I have definitely had off-list discussions with Stolee where it was important to be able to pass a value to --object-dir which does *not* belong to a Git repository (but is used as a dumping ground for packs, a MIDX, and loose objects).

It may be worthwhile to recapitulate that discussion here on the list. (I'm hoping that Stolee won't mind filling in the details, since I seem to have forgotten most of them).

Thanks,
Taylor

#205 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

Quoted 12 lines · Taylor Blau
Quoted 4 lines · Junio C Hamano

I do not see the reasoning behind "should not be a blocker" from Derrick substantiated. What's the reason why that raw object store cannot come from an existing repository, and what's the benefit we get from not having to have a repository there?

I also didn't find the reasoning spelled out in his response, but I have definitely had off-list discussions with Stolee where it was important to be able to pass a value to --object-dir which does *not* belong to a Git repository (but is used as a dumping ground for packs, a MIDX, and loose objects).

Quoted 3 lines

It may be worthwhile to recapitulate that discussion here on the list. (I'm hoping that Stolee won't mind filling in the details, since I seem to have forgotten most of them).

The way we have been using alternates in VFS for Git and Scalar is as a "shared object cache" that is shared across multiple full Git repositories with their own working trees. The shared object cache is located in a location that can be found during "scalar clone" such as

	~/.scalarCache/url_<hash-of-URL>/

This directory contains the same data as a .git/objects directory would.

Data is added to that cache using hooks during 'git fetch' or other requests for remote data. This means that the second "scalar clone" command is much faster than the first, because it already has most of the commit and tree data required to satisfy the partial clone.

(Note: this feature does not exist in the current Scalar CLI RFC, but would be contributed later.)

These caches were designed before the multi-pack-index -- in fact, they were an inspiration for them because now deleting a repo would not clean up old pack-files. The data would be added as a raw pack-file that is processed with 'git index-pack' or as loose objects. The --object-dir option was directly created as a way to target the creation and maintenance of a multi-pack-index within one of these caches that don't exist as full repositories. Clearly, there were some gaps in that implementation and I regret creating those gaps.

If I were to redesign the shared object cache, then I would have created the cache directories as bare repos and then create the "clone" repo as a worktree linked to that base. That would allow all objects and refs to be shared, achieving the same goals and an even better user experience.

I'm advocating for the position to continue allowing this feature to exist without a necessary on-upgrade conversion of these non-repos to full repos. Maybe that is the best thing to do in the long-term, but will take some time to do. Keeping compatibility for now seems like it won't hurt too much.

Thanks,
-Stolee

#206 lore raw

Re: [PATCH v4 05/25] midx: clear auxiliary .rev after replacing the MIDX

In reply to Junio C Hamano
Quoted 12 lines · Junio C Hamano
Quoted 9 lines · Jeff King
Quoted 4 lines

I think taking a look to see if ../config exists to use the data might be helpful for some cases, but should not be a blocker for completing the requested operation. The config from the non-alternate repo should be sufficient for this (somewhat strange) case.

Yes, agreed. We have long supported these kind of "bare" alternates, and I wouldn't be surprised if they are in wide use (though I do wonder how folks actually modify them, since most commands that touch objects really do want to be in a repository).

Quoted 6 lines

I kind of find the above two somewhat surprising, but I am willing to go with the less safer option if that is what people want.

It has been perfectly OK in the pre-alternative-hash-algorithms world, but we no longer live in such a world, so we'd need to come up with a way to keep using alternates in a safer way.

I think the point is that most people _do_ still live in that world. They have not started using the new hash algorithm yet, and what they have been doing for years will continue to work. Likewise, once they switch, things will continue to work as long as each repo's alternates use the same hash.

So my reasoning was less "this is useful, and a good idea" and more "it works now, and will probably continue to work OK in practice, so taking it away will probably bother people".

Now if somebody wants to make an argument that they are not actually workable now, I could buy that. ;) You cannot even run "pack-objects" without a repository, though it is not too hard to copy the result around.

Quoted 8 lines
Quoted 3 lines

But I suspect all of this is moot for now, beyond being able to return a nicer error message. The rest of the code is not at all ready to handle packs with two different hashes in the same process.

I do not think it is all that urgent to make it possible for packs with different algorithms to be used. It is sufficient to _ignore_ (or error out) configured odb that is incompatible with the current repository.

Yes, I think that would be an improvement. I just don't find it all that urgent, since they're likely to get an error anyway (just probably one that is more mysterious). Given the work involved to even detect the situation, it doesn't seem like that high a priority to me.

-Peff

#207 lore raw

PATCH v4 6/25 midx: reject empty `--preferred-pack`'s

In reply to Taylor Blau

The soon-to-be-implemented multi-pack bitmap treats object in the first bit position specially by assuming that all objects in the pack it was selected from are also represented from that pack in the MIDX. In other words, the pack from which the first object was selected must also have all of its other objects selected from that same pack in the MIDX in case of any duplicates.

But this assumption relies on the fact that there is at least one object in that pack to begin with; otherwise the object in the first bit position isn't from a preferred pack, in which case we can no longer assume that all objects in that pack were also selected from the same pack.

Guard this assumption by checking the number of objects in the given preferred pack, and failing if the given pack is empty.

To make sure we can safely perform this check, open any packs which are contained in an existing MIDX via prepare_midx_pack(). The same is done for new packs via the add_pack_to_midx() callback, but packs picked up from a previous MIDX will not yet have these opened.

---
 Documentation/git-multi-pack-index.txt |  6 +++---
 midx.c                                 | 29 ++++++++++++++++++++++++++
 t/t5319-multi-pack-index.sh            | 17 +++++++++++++++
 3 files changed, 49 insertions(+), 3 deletions(-)
Documentation/git-multi-pack-index.txt +3 -3
index ffd601bc17..c9b063d31e 100644
@@ -37,9 +37,9 @@ write::
 --
 	--preferred-pack=<pack>::
 		Optionally specify the tie-breaking pack used when
-		multiple packs contain the same object. If not given,
-		ties are broken in favor of the pack with the lowest
-		mtime.
+		multiple packs contain the same object. `<pack>` must
+		contain at least one object. If not given, ties are
+		broken in favor of the pack with the lowest mtime.
 --
 
 verify::
midx.c +29 -0
index 73b199ca49..551e5c2ee5 100644
@@ -934,6 +934,25 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			ctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);
 			ctx.info[ctx.nr].p = NULL;
 			ctx.info[ctx.nr].expired = 0;
+
+			if (flags & MIDX_WRITE_REV_INDEX) {
+				/*
+				 * If generating a reverse index, need to have
+				 * packed_git's loaded to compare their
+				 * mtimes and object count.
+				 */
+				if (prepare_midx_pack(the_repository, ctx.m, i)) {
+					error(_("could not load pack"));
+					result = 1;
+					goto cleanup;
+				}
+
+				if (open_pack_index(ctx.m->packs[i]))
+					die(_("could not open index for %s"),
+					    ctx.m->packs[i]->pack_name);
+				ctx.info[ctx.nr].p = ctx.m->packs[i];
+			}
+
 			ctx.nr++;
 		}
 	}
@@ -961,6 +980,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 		}
 	}
 
+	if (ctx.preferred_pack_idx > -1) {
+		struct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;
+		if (!preferred->num_objects) {
+			error(_("cannot select preferred pack %s with no objects"),
+			      preferred->pack_name);
+			result = 1;
+			goto cleanup;
+		}
+	}
+
 	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,
 					 ctx.preferred_pack_idx);
 
t/t5319-multi-pack-index.sh +17 -0
index 3d4d9f10c3..9b184bd45e 100755
@@ -277,6 +277,23 @@ test_expect_success 'midx picks objects from preferred pack' '
 	)
 '
 
+test_expect_success 'preferred packs must be non-empty' '
+	test_when_finished rm -rf preferred.git &&
+	git init preferred.git &&
+	(
+		cd preferred.git &&
+
+		test_commit base &&
+		git repack -ad &&
+
+		empty="$(git pack-objects $objdir/pack/pack </dev/null)" &&
+
+		test_must_fail git multi-pack-index write \
+			--preferred-pack=pack-$empty.pack 2>err &&
+		grep "with no objects" err
+	)
+'
+
 test_expect_success 'verify multi-pack-index success' '
 	git multi-pack-index verify --object-dir=$objdir
 '
Signature · 1 line
2.31.1.163.ga65ce7f831
#208 lore raw

PATCH v4 7/25 midx: infer preferred pack when not given one

In reply to Taylor Blau

In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30), the multi-pack index code learned how to select a pack which all duplicate objects are selected from. That is, if an object appears in multiple packs, select the copy in the preferred pack before breaking ties according to the other rules like pack mtime and readdir() order.

Not specifying a preferred pack can cause serious problems with multi-pack reachability bitmaps, because these bitmaps rely on having at least one pack from which all duplicates are selected. Not having such a pack causes problems with the code in pack-objects to reuse packs verbatim (e.g., that code assumes that a delta object in a chunk of pack sent verbatim will have its base object sent from the same pack).

So why does not marking a pack preferred cause problems here? The reason is roughly as follows:

- Ties are broken (when handling duplicate objects) by sorting according to midx_oid_compare(), which sorts objects by OID, preferred-ness, pack mtime, and finally pack ID (more on that later).

- The psuedo pack-order (described in Documentation/technical/pack-format.txt under the section "multi-pack-index reverse indexes") is computed by midx_pack_order(), and sorts by pack ID and pack offset, with preferred packs sorting first.

- But! Pack IDs come from incrementing the pack count in add_pack_to_midx(), which is a callback to for_each_file_in_pack_dir(), meaning that pack IDs are assigned in readdir() order.

When specifying a preferred pack, all of that works fine, because duplicate objects are correctly resolved in favor of the copy in the preferred pack, and the preferred pack sorts first in the object order.

"Sorting first" is critical, because the bitmap code relies on finding out which pack holds the first object in the MIDX's pseudo pack-order to determine which pack is preferred.

But if we didn't specify a preferred pack, and the pack which comes first in readdir() order does not also have the lowest timestamp, then it's possible that that pack (the one that sorts first in pseudo-pack order, which the bitmap code will treat as the preferred one) did *not* have all duplicate objects resolved in its favor, resulting in breakage.

The fix is simple: pick a (semi-arbitrary, non-empty) preferred pack when none was specified. This forces that pack to have duplicates resolved in its favor, and (critically) to sort first in pseudo-pack order. Unfortunately, testing this behavior portably isn't possible, since it depends on readdir() order which isn't guaranteed by POSIX.

(Note that multi-pack reachability bitmaps have yet to be implemented; so in that sense this patch is fixing a bug which does not yet exist. But by having this patch beforehand, we can prevent the bug from ever materializing.)

---
 midx.c | 50 ++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 44 insertions(+), 6 deletions(-)
midx.c +44 -6
index 551e5c2ee5..e5b17483af 100644
@@ -969,15 +969,57 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)
 		goto cleanup;
 
-	ctx.preferred_pack_idx = -1;
 	if (preferred_pack_name) {
+		int found = 0;
 		for (i = 0; i < ctx.nr; i++) {
 			if (!cmp_idx_or_pack_name(preferred_pack_name,
 						  ctx.info[i].pack_name)) {
 				ctx.preferred_pack_idx = i;
+				found = 1;
 				break;
 			}
 		}
+
+		if (!found)
+			warning(_("unknown preferred pack: '%s'"),
+				preferred_pack_name);
+	} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {
+		struct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;
+		ctx.preferred_pack_idx = 0;
+
+		if (packs_to_drop && packs_to_drop->nr)
+			BUG("cannot write a MIDX bitmap during expiration");
+
+		/*
+		 * set a preferred pack when writing a bitmap to ensure that
+		 * the pack from which the first object is selected in pseudo
+		 * pack-order has all of its objects selected from that pack
+		 * (and not another pack containing a duplicate)
+		 */
+		for (i = 1; i < ctx.nr; i++) {
+			struct packed_git *p = ctx.info[i].p;
+
+			if (!oldest->num_objects || p->mtime < oldest->mtime) {
+				oldest = p;
+				ctx.preferred_pack_idx = i;
+			}
+		}
+
+		if (!oldest->num_objects) {
+			/*
+			 * If all packs are empty; unset the preferred index.
+			 * This is acceptable since there will be no duplicate
+			 * objects to resolve, so the preferred value doesn't
+			 * matter.
+			 */
+			ctx.preferred_pack_idx = -1;
+		}
+	} else {
+		/*
+		 * otherwise don't mark any pack as preferred to avoid
+		 * interfering with expiration logic below
+		 */
+		ctx.preferred_pack_idx = -1;
 	}
 
 	if (ctx.preferred_pack_idx > -1) {
@@ -1058,11 +1100,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 						      ctx.info, ctx.nr,
 						      sizeof(*ctx.info),
 						      idx_or_pack_name_cmp);
-
-		if (!preferred)
-			warning(_("unknown preferred pack: '%s'"),
-				preferred_pack_name);
-		else {
+		if (preferred) {
 			uint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];
 			if (perm == PACK_EXPIRED)
 				warning(_("preferred pack '%s' is expired"),
Signature · 1 line
2.31.1.163.ga65ce7f831
#209 lore raw

PATCH v4 8/25 midx: close linked MIDXs, avoid leaking memory

In reply to Taylor Blau

When a repository has at least one alternate, the MIDX belonging to each alternate is accessed through the next pointer on the main object store's copy of the MIDX. close_midx() didn't bother to close any of the linked MIDXs. It likewise didn't free the memory pointed to by m, leaving uninitialized bytes with live pointers to them left around in the heap.

Clean this up by closing linked MIDXs, and freeing up the memory pointed to by each of them. When callers call close_midx(), then they can discard the entire linked list of MIDXs and set their pointer to the head of that list to NULL.

This isn't strictly required for the upcoming patches, but it makes it much more difficult (though still possible, for e.g., by calling close_midx(m->next) which leaves m->next pointing at uninitialized bytes) to have pointers to uninitialized memory.

---
 midx.c | 3 +++
 1 file changed, 3 insertions(+)
midx.c +3 -0
index e5b17483af..0a515d8711 100644
@@ -195,6 +195,8 @@ void close_midx(struct multi_pack_index *m)
 	if (!m)
 		return;
 
+	close_midx(m->next);
+
 	munmap((unsigned char *)m->data, m->data_len);
 
 	for (i = 0; i < m->num_packs; i++) {
@@ -203,6 +205,7 @@ void close_midx(struct multi_pack_index *m)
 	}
 	FREE_AND_NULL(m->packs);
 	FREE_AND_NULL(m->pack_names);
+	free(m);
 }
 
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)
Signature · 1 line
2.31.1.163.ga65ce7f831
#210 lore raw

PATCH v4 9/25 midx: avoid opening multiple MIDXs when writing

In reply to Taylor Blau

Opening multiple instance of the same MIDX can lead to problems like two separate packed_git structures which represent the same pack being added to the repository's object store.

The above scenario can happen because prepare_midx_pack() checks if m->packs[pack_int_id] is NULL in order to determine if a pack has been opened and installed in the repository before. But a caller can construct two copies of the same MIDX by calling get_multi_pack_index() and load_multi_pack_index() since the former manipulates the object store directly but the latter is a lower-level routine which allocates a new MIDX for each call.

So if prepare_midx_pack() is called on multiple MIDXs with the same pack_int_id, then that pack will be installed twice in the object store's packed_git pointer.

This can lead to problems in, for e.g., the pack-bitmap code, which does something like the following (in pack-bitmap.c:open_pack_bitmap()):

    struct bitmap_index *bitmap_git = ...;
    for (p = get_all_packs(r); p; p = p->next) {
      if (open_pack_bitmap_1(bitmap_git, p) == 0)
        ret = 0;
    }

which is a problem if two copies of the same pack exist in the packed_git list because pack-bitmap.c:open_pack_bitmap_1() contains a conditional like the following:

    if (bitmap_git->pack || bitmap_git->midx) {
      /* ignore extra bitmap file; we can only handle one */
      warning("ignoring extra bitmap file: %s", packfile->pack_name);
      close(fd);
      return -1;
    }

Avoid this scenario by not letting write_midx_internal() open a MIDX that isn't also pointed at by the object store. So long as this is the case, other routines should prefer to open MIDXs with get_multi_pack_index() or reprepare_packed_git() instead of creating instances on their own. Because get_multi_pack_index() returns r->object_store->multi_pack_index if it is non-NULL, we'll only have one instance of a MIDX open at one time, avoiding these problems.

To encourage this, drop the struct multi_pack_index * parameter from write_midx_internal(), and rely instead on the object_dir to find (or initialize) the correct MIDX instance.

Likewise, replace the call to close_midx() with close_object_store(), since we're about to replace the MIDX with a new one and should invalidate the object store's memory of any MIDX that might have existed beforehand.

Note that this now forbids passing object directories that don't belong to alternate repositories over --object-dir, since before we would have happily opened a MIDX in any directory, but now restrict ourselves to only those reachable by r->objects->multi_pack_index (and alternate MIDXs that we can see by walking the next pointer).

As far as I can tell, supporting arbitrary directories with --object-dir was a historical accident, since even the documentation says <alt> when referring to the value passed to this option.

A future patch could clean this up and provide a warning() when a non-alternate directory was given, since we'll still write a new MIDX there, we just won't reuse any MIDX that might happen to already exist in that directory.

---
 midx.c | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)
midx.c +15 -11
index 0a515d8711..3dacb31f9d 100644
@@ -893,7 +893,7 @@ static int midx_checksum_valid(struct multi_pack_index *m)
 	return hashfile_checksum_valid(m->data, m->data_len);
 }
 
-static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
+static int write_midx_internal(const char *object_dir,
 			       struct string_list *packs_to_drop,
 			       const char *preferred_pack_name,
 			       unsigned flags)
@@ -904,6 +904,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	struct hashfile *f = NULL;
 	struct lock_file lk;
 	struct write_midx_context ctx = { 0 };
+	struct multi_pack_index *cur;
 	int pack_name_concat_len = 0;
 	int dropped_packs = 0;
 	int result = 0;
@@ -914,10 +915,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 		die_errno(_("unable to create leading directories of %s"),
 			  midx_name);
 
-	if (m)
-		ctx.m = m;
-	else
-		ctx.m = load_multi_pack_index(object_dir, 1);
+	for (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {
+		if (!strcmp(object_dir, cur->object_dir)) {
+			ctx.m = cur;
+			break;
+		}
+	}
 
 	if (ctx.m && !midx_checksum_valid(ctx.m)) {
 		warning(_("ignoring existing multi-pack-index; checksum mismatch"));
@@ -1119,7 +1122,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
 
 	if (ctx.m)
-		close_midx(ctx.m);
+		close_object_store(the_repository->objects);
 
 	if (ctx.nr - dropped_packs == 0) {
 		error(_("no pack files to index."));
@@ -1182,8 +1185,7 @@ int write_midx_file(const char *object_dir,
 		    const char *preferred_pack_name,
 		    unsigned flags)
 {
-	return write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,
-				   flags);
+	return write_midx_internal(object_dir, NULL, preferred_pack_name, flags);
 }
 
 struct clear_midx_data {
@@ -1461,8 +1463,10 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla
 
 	free(count);
 
-	if (packs_to_drop.nr)
-		result = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);
+	if (packs_to_drop.nr) {
+		result = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);
+		m = NULL;
+	}
 
 	string_list_clear(&packs_to_drop, 0);
 	return result;
@@ -1651,7 +1655,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,
 		goto cleanup;
 	}
 
-	result = write_midx_internal(object_dir, m, NULL, NULL, flags);
+	result = write_midx_internal(object_dir, NULL, NULL, flags);
 	m = NULL;
 
 cleanup:
Signature · 1 line
2.31.1.163.ga65ce7f831
#211 lore raw

PATCH v4 10/25 pack-bitmap.c: introduce 'bitmap_num_objects()'

In reply to Taylor Blau

A subsequent patch to support reading MIDX bitmaps will be less noisy after extracting a generic function to return how many objects are contained in a bitmap.

---
 pack-bitmap.c | 37 +++++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 16 deletions(-)
pack-bitmap.c +21 -16
index 9b11af87aa..65356f9657 100644
@@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)
 	return b;
 }
 
+static uint32_t bitmap_num_objects(struct bitmap_index *index)
+{
+	return index->pack->num_objects;
+}
+
 static int load_bitmap_header(struct bitmap_index *index)
 {
 	struct bitmap_disk_header *header = (void *)index->map;
@@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index)
 	/* Parse known bitmap format options */
 	{
 		uint32_t flags = ntohs(header->options);
-		size_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));
+		size_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t));
 		unsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;
 
 		if ((flags & BITMAP_OPT_FULL_DAG) == 0)
@@ -404,7 +409,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,
 
 	if (pos < kh_end(positions)) {
 		int bitmap_pos = kh_value(positions, pos);
-		return bitmap_pos + bitmap_git->pack->num_objects;
+		return bitmap_pos + bitmap_num_objects(bitmap_git);
 	}
 
 	return -1;
@@ -456,7 +461,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,
 		bitmap_pos = kh_value(eindex->positions, hash_pos);
 	}
 
-	return bitmap_pos + bitmap_git->pack->num_objects;
+	return bitmap_pos + bitmap_num_objects(bitmap_git);
 }
 
 struct bitmap_show_data {
@@ -673,7 +678,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,
 	for (i = 0; i < eindex->count; ++i) {
 		struct object *obj;
 
-		if (!bitmap_get(objects, bitmap_git->pack->num_objects + i))
+		if (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))
 			continue;
 
 		obj = eindex->objects[i];
@@ -832,7 +837,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,
 	 * them individually.
 	 */
 	for (i = 0; i < eindex->count; i++) {
-		uint32_t pos = i + bitmap_git->pack->num_objects;
+		uint32_t pos = i + bitmap_num_objects(bitmap_git);
 		if (eindex->objects[i]->type == type &&
 		    bitmap_get(to_filter, pos) &&
 		    !bitmap_get(tips, pos))
@@ -859,7 +864,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 
 	oi.sizep = &size;
 
-	if (pos < pack->num_objects) {
+	if (pos < bitmap_num_objects(bitmap_git)) {
 		off_t ofs = pack_pos_to_offset(pack, pos);
 		if (packed_object_info(the_repository, pack, ofs, &oi) < 0) {
 			struct object_id oid;
@@ -869,7 +874,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 		}
 	} else {
 		struct eindex *eindex = &bitmap_git->ext_index;
-		struct object *obj = eindex->objects[pos - pack->num_objects];
+		struct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];
 		if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)
 			die(_("unable to get size of %s"), oid_to_hex(&obj->oid));
 	}
@@ -911,7 +916,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,
 	}
 
 	for (i = 0; i < eindex->count; i++) {
-		uint32_t pos = i + bitmap_git->pack->num_objects;
+		uint32_t pos = i + bitmap_num_objects(bitmap_git);
 		if (eindex->objects[i]->type == OBJ_BLOB &&
 		    bitmap_get(to_filter, pos) &&
 		    !bitmap_get(tips, pos) &&
@@ -1137,8 +1142,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 	enum object_type type;
 	unsigned long size;
 
-	if (pos >= bitmap_git->pack->num_objects)
-		return; /* not actually in the pack */
+	if (pos >= bitmap_num_objects(bitmap_git))
+		return; /* not actually in the pack or MIDX */
 
 	offset = header = pack_pos_to_offset(bitmap_git->pack, pos);
 	type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);
@@ -1204,6 +1209,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 	struct pack_window *w_curs = NULL;
 	size_t i = 0;
 	uint32_t offset;
+	uint32_t objects_nr = bitmap_num_objects(bitmap_git);
 
 	assert(result);
 
@@ -1211,8 +1217,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 		i++;
 
 	/* Don't mark objects not in the packfile */
-	if (i > bitmap_git->pack->num_objects / BITS_IN_EWORD)
-		i = bitmap_git->pack->num_objects / BITS_IN_EWORD;
+	if (i > objects_nr / BITS_IN_EWORD)
+		i = objects_nr / BITS_IN_EWORD;
 
 	reuse = bitmap_word_alloc(i);
 	memset(reuse->words, 0xFF, i * sizeof(eword_t));
@@ -1296,7 +1302,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,
 
 	for (i = 0; i < eindex->count; ++i) {
 		if (eindex->objects[i]->type == type &&
-			bitmap_get(objects, bitmap_git->pack->num_objects + i))
+			bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))
 			count++;
 	}
 
@@ -1517,7 +1523,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,
 	uint32_t i, num_objects;
 	uint32_t *reposition;
 
-	num_objects = bitmap_git->pack->num_objects;
+	num_objects = bitmap_num_objects(bitmap_git);
 	CALLOC_ARRAY(reposition, num_objects);
 
 	for (i = 0; i < num_objects; ++i) {
@@ -1600,7 +1606,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,
 static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)
 {
 	struct bitmap *result = bitmap_git->result;
-	struct packed_git *pack = bitmap_git->pack;
 	struct eindex *eindex = &bitmap_git->ext_index;
 	off_t total = 0;
 	struct object_info oi = OBJECT_INFO_INIT;
@@ -1612,7 +1617,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)
 	for (i = 0; i < eindex->count; i++) {
 		struct object *obj = eindex->objects[i];
 
-		if (!bitmap_get(result, pack->num_objects + i))
+		if (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i))
 			continue;
 
 		if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)
Signature · 1 line
2.31.1.163.ga65ce7f831
#212 lore raw

PATCH v4 11/25 pack-bitmap.c: introduce 'nth_bitmap_object_oid()'

In reply to Taylor Blau

A subsequent patch to support reading MIDX bitmaps will be less noisy after extracting a generic function to fetch the nth OID contained in the bitmap.

---
 pack-bitmap.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
pack-bitmap.c +10 -3
index 65356f9657..612f62da97 100644
@@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)
 
 #define MAX_XOR_OFFSET 160
 
+static int nth_bitmap_object_oid(struct bitmap_index *index,
+				 struct object_id *oid,
+				 uint32_t n)
+{
+	return nth_packed_object_id(oid, index->pack, n);
+}
+
 static int load_bitmap_entries_v1(struct bitmap_index *index)
 {
 	uint32_t i;
@@ -242,7 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
 		xor_offset = read_u8(index->map, &index->map_pos);
 		flags = read_u8(index->map, &index->map_pos);
 
-		if (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)
+		if (nth_bitmap_object_oid(index, &oid, commit_idx_pos) < 0)
 			return error("corrupt ewah bitmap: commit index %u out of range",
 				     (unsigned)commit_idx_pos);
 
@@ -868,8 +875,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 		off_t ofs = pack_pos_to_offset(pack, pos);
 		if (packed_object_info(the_repository, pack, ofs, &oi) < 0) {
 			struct object_id oid;
-			nth_packed_object_id(&oid, pack,
-					     pack_pos_to_index(pack, pos));
+			nth_bitmap_object_oid(bitmap_git, &oid,
+					      pack_pos_to_index(pack, pos));
 			die(_("unable to get size of %s"), oid_to_hex(&oid));
 		}
 	} else {
Signature · 1 line
2.31.1.163.ga65ce7f831
#213 lore raw

PATCH v4 12/25 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'

In reply to Taylor Blau

In a recent commit, pack-objects learned support for the 'pack.preferBitmapTips' configuration. This patch prepares the multi-pack bitmap code to respect this configuration, too.

The yet-to-be implemented code will find that it is more efficient to check whether each reference contains a prefix found in the configured set of values rather than doing an additional traversal.

Implement a function 'bitmap_is_preferred_refname()' which will perform that check. Its caller will be added in a subsequent patch.

---
 pack-bitmap.c | 16 ++++++++++++++++
 pack-bitmap.h |  1 +
 2 files changed, 17 insertions(+)
pack-bitmap.c +16 -0
index 612f62da97..d5296750eb 100644
@@ -1658,3 +1658,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r)
 {
 	return repo_config_get_value_multi(r, "pack.preferbitmaptips");
 }
+
+int bitmap_is_preferred_refname(struct repository *r, const char *refname)
+{
+	const struct string_list *preferred_tips = bitmap_preferred_tips(r);
+	struct string_list_item *item;
+
+	if (!preferred_tips)
+		return 0;
+
+	for_each_string_list_item(item, preferred_tips) {
+		if (starts_with(refname, item->string))
+			return 1;
+	}
+
+	return 0;
+}
pack-bitmap.h +1 -0
index 020cd8d868..52ea10de51 100644
@@ -94,5 +94,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint16_t options);
 
 const struct string_list *bitmap_preferred_tips(struct repository *r);
+int bitmap_is_preferred_refname(struct repository *r, const char *refname);
 
 #endif
Signature · 1 line
2.31.1.163.ga65ce7f831
#214 lore raw

PATCH v4 13/25 pack-bitmap.c: avoid redundant calls to try_partial_reuse

In reply to Taylor Blau

try_partial_reuse() is used to mark any bits in the beginning of a bitmap whose objects can be reused verbatim from the pack they came from.

Currently this function returns void, and signals nothing to the caller when bits could not be reused. But multi-pack bitmaps would benefit from having such a signal, because they may try to pass objects which are in bounds, but from a pack other than the preferred one.

Any extra calls are noops because of a conditional in reuse_partial_packfile_from_bitmap(), but those loop iterations can be avoided by letting try_partial_reuse() indicate when it can't accept any more bits for reuse, and then listening to that signal.

---
 pack-bitmap.c | 40 +++++++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 11 deletions(-)
pack-bitmap.c +29 -11
index d5296750eb..4e37f5d574 100644
@@ -1140,22 +1140,26 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,
 	return NULL;
 }
 
-static void try_partial_reuse(struct bitmap_index *bitmap_git,
-			      size_t pos,
-			      struct bitmap *reuse,
-			      struct pack_window **w_curs)
+/*
+ * -1 means "stop trying further objects"; 0 means we may or may not have
+ * reused, but you can keep feeding bits.
+ */
+static int try_partial_reuse(struct bitmap_index *bitmap_git,
+			     size_t pos,
+			     struct bitmap *reuse,
+			     struct pack_window **w_curs)
 {
 	off_t offset, header;
 	enum object_type type;
 	unsigned long size;
 
 	if (pos >= bitmap_num_objects(bitmap_git))
-		return; /* not actually in the pack or MIDX */
+		return -1; /* not actually in the pack or MIDX */
 
 	offset = header = pack_pos_to_offset(bitmap_git->pack, pos);
 	type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);
 	if (type < 0)
-		return; /* broken packfile, punt */
+		return -1; /* broken packfile, punt */
 
 	if (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {
 		off_t base_offset;
@@ -1172,9 +1176,9 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		base_offset = get_delta_base(bitmap_git->pack, w_curs,
 					     &offset, type, header);
 		if (!base_offset)
-			return;
+			return 0;
 		if (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)
-			return;
+			return 0;
 
 		/*
 		 * We assume delta dependencies always point backwards. This
@@ -1186,7 +1190,7 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		 * odd parameters.
 		 */
 		if (base_pos >= pos)
-			return;
+			return 0;
 
 		/*
 		 * And finally, if we're not sending the base as part of our
@@ -1197,13 +1201,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		 * object_entry code path handle it.
 		 */
 		if (!bitmap_get(reuse, base_pos))
-			return;
+			return 0;
 	}
 
 	/*
 	 * If we got here, then the object is OK to reuse. Mark it.
 	 */
 	bitmap_set(reuse, pos);
+	return 0;
 }
 
 int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
@@ -1239,10 +1244,23 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 				break;
 
 			offset += ewah_bit_ctz64(word >> offset);
-			try_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);
+			if (try_partial_reuse(bitmap_git, pos + offset, reuse,
+					      &w_curs) < 0) {
+				/*
+				 * try_partial_reuse indicated we couldn't reuse
+				 * any bits, so there is no point in trying more
+				 * bits in the current word, or any other words
+				 * in result.
+				 *
+				 * Jump out of both loops to avoid future
+				 * unnecessary calls to try_partial_reuse.
+				 */
+				goto done;
+			}
 		}
 	}
 
+done:
 	unuse_pack(&w_curs);
 
 	*entries = bitmap_popcount(reuse);
Signature · 1 line
2.31.1.163.ga65ce7f831
#215 lore raw

PATCH v4 14/25 pack-bitmap: read multi-pack bitmaps

In reply to Taylor Blau

This prepares the code in pack-bitmap to interpret the new multi-pack bitmaps described in Documentation/technical/bitmap-format.txt, which mostly involves converting bit positions to accommodate looking them up in a MIDX.

Note that there are currently no writers who write multi-pack bitmaps, and that this will be implemented in the subsequent commit. Note also that get_midx_checksum() and get_midx_filename() are made non-static so they can be called from pack-bitmap.c.

---
 builtin/pack-objects.c |   5 +
 midx.c                 |   4 +-
 midx.h                 |   2 +
 pack-bitmap-write.c    |   2 +-
 pack-bitmap.c          | 357 ++++++++++++++++++++++++++++++++++++-----
 pack-bitmap.h          |   6 +
 packfile.c             |   2 +-
 7 files changed, 336 insertions(+), 42 deletions(-)
builtin/pack-objects.c +5 -0
index 8a523624a1..e11d3ac2e5 100644
@@ -1124,6 +1124,11 @@ static void write_reused_pack(struct hashfile *f)
 				break;
 
 			offset += ewah_bit_ctz64(word >> offset);
+			/*
+			 * Can use bit positions directly, even for MIDX
+			 * bitmaps. See comment in try_partial_reuse()
+			 * for why.
+			 */
 			write_reused_pack_one(pos + offset, f, &w_curs);
 			display_progress(progress_state, ++written);
 		}
midx.c +2 -2
index 3dacb31f9d..2dceaf9565 100644
@@ -48,12 +48,12 @@ static uint8_t oid_version(void)
 	}
 }
 
-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)
+const unsigned char *get_midx_checksum(struct multi_pack_index *m)
 {
 	return m->data + m->data_len - the_hash_algo->rawsz;
 }
 
-static char *get_midx_filename(const char *object_dir)
+char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }
midx.h +2 -0
index 8684cf0fef..1172df1a71 100644
@@ -42,6 +42,8 @@ struct multi_pack_index {
 #define MIDX_PROGRESS     (1 << 0)
 #define MIDX_WRITE_REV_INDEX (1 << 1)
 
+const unsigned char *get_midx_checksum(struct multi_pack_index *m);
+char *get_midx_filename(const char *object_dir);
 char *get_midx_rev_filename(struct multi_pack_index *m);
 
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
pack-bitmap-write.c +1 -1
index 142fd0adb8..9c55c1531e 100644
@@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show)
 }
 
 /**
- * Build the initial type index for the packfile
+ * Build the initial type index for the packfile or multi-pack-index
  */
 void bitmap_writer_build_type_index(struct packing_data *to_pack,
 				    struct pack_idx_entry **index,
pack-bitmap.c +319 -38

584 lines not on this page · raw message

pack-bitmap.h +6 -0
index 52ea10de51..81664f933f 100644
@@ -44,6 +44,8 @@ typedef int (*show_reachable_fn)(
 struct bitmap_index;
 
 struct bitmap_index *prepare_bitmap_git(struct repository *r);
+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,
+					     struct multi_pack_index *midx);
 void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,
 			      uint32_t *trees, uint32_t *blobs, uint32_t *tags);
 void traverse_bitmap_commit_list(struct bitmap_index *,
@@ -92,6 +94,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint32_t index_nr,
 			  const char *filename,
 			  uint16_t options);
+char *midx_bitmap_filename(struct multi_pack_index *midx);
+char *pack_bitmap_filename(struct packed_git *p);
+
+int bitmap_is_midx(struct bitmap_index *bitmap_git);
 
 const struct string_list *bitmap_preferred_tips(struct repository *r);
 int bitmap_is_preferred_refname(struct repository *r, const char *refname);
packfile.c +1 -1
index 9ef6d98292..371f5488cf 100644
@@ -860,7 +860,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 	if (!strcmp(file_name, "multi-pack-index"))
 		return;
 	if (starts_with(file_name, "multi-pack-index") &&
-	    ends_with(file_name, ".rev"))
+	    (ends_with(file_name, ".bitmap") || ends_with(file_name, ".rev")))
 		return;
 	if (ends_with(file_name, ".idx") ||
 	    ends_with(file_name, ".rev") ||
Signature · 1 line
2.31.1.163.ga65ce7f831
#216 lore raw

PATCH v4 15/25 pack-bitmap: write multi-pack bitmaps

In reply to Taylor Blau

Write multi-pack bitmaps in the format described by Documentation/technical/bitmap-format.txt, inferring their presence with the absence of '--bitmap'.

To write a multi-pack bitmap, this patch attempts to reuse as much of the existing machinery from pack-objects as possible. Specifically, the MIDX code prepares a packing_data struct that pretends as if a single packfile has been generated containing all of the objects contained within the MIDX.

---
 Documentation/git-multi-pack-index.txt |  12 +-
 builtin/multi-pack-index.c             |   2 +
 midx.c                                 | 208 ++++++++++++++++++++++++-
 midx.h                                 |   1 +
 4 files changed, 214 insertions(+), 9 deletions(-)
Documentation/git-multi-pack-index.txt +11 -1
index c9b063d31e..ed52459a9d 100644
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [verse]
 'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
-	[--preferred-pack=<pack>] <subcommand>
+	[--preferred-pack=<pack>] [--[no-]bitmap] <subcommand>
 
 DESCRIPTION
 -----------
@@ -40,6 +40,9 @@ write::
 		multiple packs contain the same object. `<pack>` must
 		contain at least one object. If not given, ties are
 		broken in favor of the pack with the lowest mtime.
+
+	--[no-]bitmap::
+		Control whether or not a multi-pack bitmap is written.
 --
 
 verify::
@@ -81,6 +84,13 @@ EXAMPLES
 $ git multi-pack-index write
 -----------------------------------------------
 
+* Write a MIDX file for the packfiles in the current .git folder with a
+corresponding bitmap.
++
+-------------------------------------------------------------
+$ git multi-pack-index write --preferred-pack=<pack> --bitmap
+-------------------------------------------------------------
+
 * Write a MIDX file for the packfiles in an alternate object store.
 +
 -----------------------------------------------
builtin/multi-pack-index.c +2 -0
index 8ff0dee2ec..73c0113b48 100644
@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
 			   N_("preferred-pack"),
 			   N_("pack for reuse when computing a multi-pack bitmap")),
+		OPT_BIT(0, "bitmap", &opts.flags, N_("write multi-pack bitmap"),
+			MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),
 		OPT_END(),
 	};
 
midx.c +200 -8

269 lines not on this page · raw message

midx.h +1 -0
index 1172df1a71..350f4d0a7b 100644
@@ -41,6 +41,7 @@ struct multi_pack_index {
 
 #define MIDX_PROGRESS     (1 << 0)
 #define MIDX_WRITE_REV_INDEX (1 << 1)
+#define MIDX_WRITE_BITMAP (1 << 2)
 
 const unsigned char *get_midx_checksum(struct multi_pack_index *m);
 char *get_midx_filename(const char *object_dir);
Signature · 1 line
2.31.1.163.ga65ce7f831
#217 lore raw

PATCH v4 16/25 t5310: move some tests to lib-bitmap.sh

In reply to Taylor Blau

We'll soon be adding a test script that will cover many of the same bitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many of the applicable tests as we can so we don't have to rewrite them.

There should be no functional change to t5310; we still run the same operations in the same order.

---
 t/lib-bitmap.sh         | 236 ++++++++++++++++++++++++++++++++++++++++
 t/t5310-pack-bitmaps.sh | 227 +-------------------------------------
 2 files changed, 240 insertions(+), 223 deletions(-)
t/lib-bitmap.sh +236 -0
index fe3f98be24..77464da6fd 100644
@@ -1,3 +1,6 @@
+# Helpers for scripts testing bitmap functionality; see t5310 for
+# example usage.
+
 # Compare a file containing rev-list bitmap traversal output to its non-bitmap
 # counterpart. You can't just use test_cmp for this, because the two produce
 # subtly different output:
@@ -24,3 +27,236 @@ test_bitmap_traversal () {
 	test_cmp "$1.normalized" "$2.normalized" &&
 	rm -f "$1.normalized" "$2.normalized"
 }
+
+# To ensure the logic for "maximal commits" is exercised, make
+# the repository a bit more complicated.
+#
+#    other                         second
+#      *                             *
+# (99 commits)                  (99 commits)
+#      *                             *
+#      |\                           /|
+#      | * octo-other  octo-second * |
+#      |/|\_________  ____________/|\|
+#      | \          \/  __________/  |
+#      |  | ________/\ /             |
+#      *  |/          * merge-right  *
+#      | _|__________/ \____________ |
+#      |/ |                         \|
+# (l1) *  * merge-left               * (r1)
+#      | / \________________________ |
+#      |/                           \|
+# (l2) *                             * (r2)
+#       \___________________________ |
+#                                   \|
+#                                    * (base)
+#
+# We only push bits down the first-parent history, which
+# makes some of these commits unimportant!
+#
+# The important part for the maximal commit algorithm is how
+# the bitmasks are extended. Assuming starting bit positions
+# for second (bit 0) and other (bit 1), the bitmasks at the
+# end should be:
+#
+#      second: 1       (maximal, selected)
+#       other: 01      (maximal, selected)
+#      (base): 11 (maximal)
+#
+# This complicated history was important for a previous
+# version of the walk that guarantees never walking a
+# commit multiple times. That goal might be important
+# again, so preserve this complicated case. For now, this
+# test will guarantee that the bitmaps are computed
+# correctly, even with the repeat calculations.
+setup_bitmap_history() {
+	test_expect_success 'setup repo with moderate-sized history' '
+		test_commit_bulk --id=file 10 &&
+		git branch -M second &&
+		git checkout -b other HEAD~5 &&
+		test_commit_bulk --id=side 10 &&
+
+		# add complicated history setup, including merges and
+		# ambiguous merge-bases
+
+		git checkout -b merge-left other~2 &&
+		git merge second~2 -m "merge-left" &&
+
+		git checkout -b merge-right second~1 &&
+		git merge other~1 -m "merge-right" &&
+
+		git checkout -b octo-second second &&
+		git merge merge-left merge-right -m "octopus-second" &&
+
+		git checkout -b octo-other other &&
+		git merge merge-left merge-right -m "octopus-other" &&
+
+		git checkout other &&
+		git merge octo-other -m "pull octopus" &&
+
+		git checkout second &&
+		git merge octo-second -m "pull octopus" &&
+
+		# Remove these branches so they are not selected
+		# as bitmap tips
+		git branch -D merge-left &&
+		git branch -D merge-right &&
+		git branch -D octo-other &&
+		git branch -D octo-second &&
+
+		# add padding to make these merges less interesting
+		# and avoid having them selected for bitmaps
+		test_commit_bulk --id=file 100 &&
+		git checkout other &&
+		test_commit_bulk --id=side 100 &&
+		git checkout second &&
+
+		bitmaptip=$(git rev-parse second) &&
+		blob=$(echo tagged-blob | git hash-object -w --stdin) &&
+		git tag tagged-blob $blob
+	'
+}
+
+rev_list_tests_head () {
+	test_expect_success "counting commits via bitmap ($state, $branch)" '
+		git rev-list --count $branch >expect &&
+		git rev-list --use-bitmap-index --count $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting partial commits via bitmap ($state, $branch)" '
+		git rev-list --count $branch~5..$branch >expect &&
+		git rev-list --use-bitmap-index --count $branch~5..$branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting commits with limit ($state, $branch)" '
+		git rev-list --count -n 1 $branch >expect &&
+		git rev-list --use-bitmap-index --count -n 1 $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting non-linear history ($state, $branch)" '
+		git rev-list --count other...second >expect &&
+		git rev-list --use-bitmap-index --count other...second >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting commits with limiting ($state, $branch)" '
+		git rev-list --count $branch -- 1.t >expect &&
+		git rev-list --use-bitmap-index --count $branch -- 1.t >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting objects via bitmap ($state, $branch)" '
+		git rev-list --count --objects $branch >expect &&
+		git rev-list --use-bitmap-index --count --objects $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "enumerate commits ($state, $branch)" '
+		git rev-list --use-bitmap-index $branch >actual &&
+		git rev-list $branch >expect &&
+		test_bitmap_traversal --no-confirm-bitmaps expect actual
+	'
+
+	test_expect_success "enumerate --objects ($state, $branch)" '
+		git rev-list --objects --use-bitmap-index $branch >actual &&
+		git rev-list --objects $branch >expect &&
+		test_bitmap_traversal expect actual
+	'
+
+	test_expect_success "bitmap --objects handles non-commit objects ($state, $branch)" '
+		git rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&
+		grep $blob actual
+	'
+}
+
+rev_list_tests () {
+	state=$1
+
+	for branch in "second" "other"
+	do
+		rev_list_tests_head
+	done
+}
+
+basic_bitmap_tests () {
+	tip="$1"
+	test_expect_success 'rev-list --test-bitmap verifies bitmaps' "
+		git rev-list --test-bitmap "${tip:-HEAD}"
+	"
+
+	rev_list_tests 'full bitmap'
+
+	test_expect_success 'clone from bitmapped repository' '
+		rm -fr clone.git &&
+		git clone --no-local --bare . clone.git &&
+		git rev-parse HEAD >expect &&
+		git --git-dir=clone.git rev-parse HEAD >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success 'partial clone from bitmapped repository' '
+		test_config uploadpack.allowfilter true &&
+		rm -fr partial-clone.git &&
+		git clone --no-local --bare --filter=blob:none . partial-clone.git &&
+		(
+			cd partial-clone.git &&
+			pack=$(echo objects/pack/*.pack) &&
+			git verify-pack -v "$pack" >have &&
+			awk "/blob/ { print \$1 }" <have >blobs &&
+			# we expect this single blob because of the direct ref
+			git rev-parse refs/tags/tagged-blob >expect &&
+			test_cmp expect blobs
+		)
+	'
+
+	test_expect_success 'setup further non-bitmapped commits' '
+		test_commit_bulk --id=further 10
+	'
+
+	rev_list_tests 'partial bitmap'
+
+	test_expect_success 'fetch (partial bitmap)' '
+		git --git-dir=clone.git fetch origin second:second &&
+		git rev-parse HEAD >expect &&
+		git --git-dir=clone.git rev-parse HEAD >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success 'enumerating progress counts pack-reused objects' '
+		count=$(git rev-list --objects --all --count) &&
+		git repack -adb &&
+
+		# check first with only reused objects; confirm that our
+		# progress showed the right number, and also that we did
+		# pack-reuse as expected.  Check only the final "done"
+		# line of the meter (there may be an arbitrary number of
+		# intermediate lines ending with CR).
+		GIT_PROGRESS_DELAY=0 \
+			git pack-objects --all --stdout --progress \
+			</dev/null >/dev/null 2>stderr &&
+		grep "Enumerating objects: $count, done" stderr &&
+		grep "pack-reused $count" stderr &&
+
+		# now the same but with one non-reused object
+		git commit --allow-empty -m "an extra commit object" &&
+		GIT_PROGRESS_DELAY=0 \
+			git pack-objects --all --stdout --progress \
+			</dev/null >/dev/null 2>stderr &&
+		grep "Enumerating objects: $((count+1)), done" stderr &&
+		grep "pack-reused $count" stderr
+	'
+}
+
+# have_delta <obj> <expected_base>
+#
+# Note that because this relies on cat-file, it might find _any_ copy of an
+# object in the repository. The caller is responsible for making sure
+# there's only one (e.g., via "repack -ad", or having just fetched a copy).
+have_delta () {
+	echo $2 >expect &&
+	echo $1 | git cat-file --batch-check="%(deltabase)" >actual &&
+	test_cmp expect actual
+}
t/t5310-pack-bitmaps.sh +4 -223

246 lines not on this page · raw message

Signature · 1 line
2.31.1.163.ga65ce7f831
#218 lore raw

PATCH v4 17/25 t/helper/test-read-midx.c: add --checksum mode

In reply to Taylor Blau

Subsequent tests will want to check for the existence of a multi-pack bitmap which matches the multi-pack-index stored in the pack directory.

The multi-pack bitmap includes the hex checksum of the MIDX it corresponds to in its filename (for example, '$packdir/multi-pack-index-<checksum>.bitmap'). As a result, some tests want a way to learn what '<checksum>' is.

This helper addresses that need by printing the checksum of the repository's multi-pack-index.

---
 t/helper/test-read-midx.c | 16 +++++++++++++++-
 t/lib-bitmap.sh           |  4 ++++
 2 files changed, 19 insertions(+), 1 deletion(-)
t/helper/test-read-midx.c +15 -1
index 7c2eb11a8e..cb0d27049a 100644
@@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects)
 	return 0;
 }
 
+static int read_midx_checksum(const char *object_dir)
+{
+	struct multi_pack_index *m;
+
+	setup_git_directory();
+	m = load_multi_pack_index(object_dir, 1);
+	if (!m)
+		return 1;
+	printf("%s\n", hash_to_hex(get_midx_checksum(m)));
+	return 0;
+}
+
 int cmd__read_midx(int argc, const char **argv)
 {
 	if (!(argc == 2 || argc == 3))
-		usage("read-midx [--show-objects] <object-dir>");
+		usage("read-midx [--show-objects|--checksum] <object-dir>");
 
 	if (!strcmp(argv[1], "--show-objects"))
 		return read_midx_file(argv[2], 1);
+	else if (!strcmp(argv[1], "--checksum"))
+		return read_midx_checksum(argv[2]);
 	return read_midx_file(argv[1], 0);
 }
t/lib-bitmap.sh +4 -0
index 77464da6fd..21d0392dda 100644
@@ -260,3 +260,7 @@ have_delta () {
 	echo $1 | git cat-file --batch-check="%(deltabase)" >actual &&
 	test_cmp expect actual
 }
+
+midx_checksum () {
+	test-tool read-midx --checksum "$1"
+}
Signature · 1 line
2.31.1.163.ga65ce7f831
#219 lore raw

PATCH v4 18/25 t5326: test multi-pack bitmap behavior

In reply to Taylor Blau

This patch introduces a new test, t5326, which tests the basic functionality of multi-pack bitmaps.

Some trivial behavior is tested, such as:

- Whether bitmaps can be generated with more than one pack.
- Whether clones can be served with all objects in the bitmap.
- Whether follow-up fetches can be served with some objects outside of the server's bitmap

These use lib-bitmap's tests (which in turn were pulled from t5310), and we cover cases where the MIDX represents both a single pack and multiple packs.

In addition, some non-trivial and MIDX-specific behavior is tested, too, including:

- Whether multi-pack bitmaps behave correctly with respect to the pack-reuse machinery when the base for some object is selected from a different pack than the delta.
- Whether multi-pack bitmaps correctly respect the pack.preferBitmapTips configuration.

---
 t/t5326-multi-pack-bitmaps.sh | 286 ++++++++++++++++++++++++++++++++++
 1 file changed, 286 insertions(+)
 create mode 100755 t/t5326-multi-pack-bitmaps.sh
t/t5326-multi-pack-bitmaps.sh new file +286 -0
new file mode 100755
index 0000000000..4ad7c2c969
@@ -0,0 +1,286 @@
+#!/bin/sh
+
+test_description='exercise basic multi-pack bitmap functionality'
+. ./test-lib.sh
+. "${TEST_DIRECTORY}/lib-bitmap.sh"
+
+# We'll be writing our own midx and bitmaps, so avoid getting confused by the
+# automatic ones.
+GIT_TEST_MULTI_PACK_INDEX=0
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
+objdir=.git/objects
+midx=$objdir/pack/multi-pack-index
+
+# midx_pack_source <obj>
+midx_pack_source () {
+	test-tool read-midx --show-objects .git/objects | grep "^$1 " | cut -f2
+}
+
+setup_bitmap_history
+
+test_expect_success 'enable core.multiPackIndex' '
+	git config core.multiPackIndex true
+'
+
+test_expect_success 'create single-pack midx with bitmaps' '
+	git repack -ad &&
+	git multi-pack-index write --bitmap &&
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_file $midx-$(midx_checksum $objdir).rev
+'
+
+basic_bitmap_tests
+
+test_expect_success 'create new additional packs' '
+	for i in $(test_seq 1 16)
+	do
+		test_commit "$i" &&
+		git repack -d || return 1
+	done &&
+
+	git checkout -b other2 HEAD~8 &&
+	for i in $(test_seq 1 8)
+	do
+		test_commit "side-$i" &&
+		git repack -d || return 1
+	done &&
+	git checkout second
+'
+
+test_expect_success 'create multi-pack midx with bitmaps' '
+	git multi-pack-index write --bitmap &&
+
+	ls $objdir/pack/pack-*.pack >packs &&
+	test_line_count = 25 packs &&
+
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_file $midx-$(midx_checksum $objdir).rev
+'
+
+basic_bitmap_tests
+
+test_expect_success '--no-bitmap is respected when bitmaps exist' '
+	git multi-pack-index write --bitmap &&
+
+	test_commit respect--no-bitmap &&
+	git repack -d &&
+
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_file $midx-$(midx_checksum $objdir).rev &&
+
+	git multi-pack-index write --no-bitmap &&
+
+	test_path_is_file $midx &&
+	test_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_missing $midx-$(midx_checksum $objdir).rev
+'
+
+test_expect_success 'setup midx with base from later pack' '
+	# Write a and b so that "a" is a delta on top of base "b", since Git
+	# prefers to delete contents out of a base rather than add to a shorter
+	# object.
+	test_seq 1 128 >a &&
+	test_seq 1 130 >b &&
+
+	git add a b &&
+	git commit -m "initial commit" &&
+
+	a=$(git rev-parse HEAD:a) &&
+	b=$(git rev-parse HEAD:b) &&
+
+	# In the first pack, "a" is stored as a delta to "b".
+	p1=$(git pack-objects .git/objects/pack/pack <<-EOF
+	$a
+	$b
+	EOF
+	) &&
+
+	# In the second pack, "a" is missing, and "b" is not a delta nor base to
+	# any other object.
+	p2=$(git pack-objects .git/objects/pack/pack <<-EOF
+	$b
+	$(git rev-parse HEAD)
+	$(git rev-parse HEAD^{tree})
+	EOF
+	) &&
+
+	git prune-packed &&
+	# Use the second pack as the preferred source, so that "b" occurs
+	# earlier in the MIDX object order, rendering "a" unusable for pack
+	# reuse.
+	git multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&
+
+	have_delta $a $b &&
+	test $(midx_pack_source $a) != $(midx_pack_source $b)
+'
+
+rev_list_tests 'full bitmap with backwards delta'
+
+test_expect_success 'clone with bitmaps enabled' '
+	git clone --no-local --bare . clone-reverse-delta.git &&
+	test_when_finished "rm -fr clone-reverse-delta.git" &&
+
+	git rev-parse HEAD >expect &&
+	git --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&
+	test_cmp expect actual
+'
+
+bitmap_reuse_tests() {
+	from=$1
+	to=$2
+
+	test_expect_success "setup pack reuse tests ($from -> $to)" '
+		rm -fr repo &&
+		git init repo &&
+		(
+			cd repo &&
+			test_commit_bulk 16 &&
+			git tag old-tip &&
+
+			git config core.multiPackIndex true &&
+			if test "MIDX" = "$from"
+			then
+				git repack -Ad &&
+				git multi-pack-index write --bitmap
+			else
+				git repack -Adb
+			fi
+		)
+	'
+
+	test_expect_success "build bitmap from existing ($from -> $to)" '
+		(
+			cd repo &&
+			test_commit_bulk --id=further 16 &&
+			git tag new-tip &&
+
+			if test "MIDX" = "$to"
+			then
+				git repack -d &&
+				git multi-pack-index write --bitmap
+			else
+				git repack -Adb
+			fi
+		)
+	'
+
+	test_expect_success "verify resulting bitmaps ($from -> $to)" '
+		(
+			cd repo &&
+			git for-each-ref &&
+			git rev-list --test-bitmap refs/tags/old-tip &&
+			git rev-list --test-bitmap refs/tags/new-tip
+		)
+	'
+}
+
+bitmap_reuse_tests 'pack' 'MIDX'
+bitmap_reuse_tests 'MIDX' 'pack'
+bitmap_reuse_tests 'MIDX' 'MIDX'
+
+test_expect_success 'missing object closure fails gracefully' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit loose &&
+		test_commit packed &&
+
+		# Do not pass "--revs"; we want a pack without the "loose"
+		# commit.
+		git pack-objects $objdir/pack/pack <<-EOF &&
+		$(git rev-parse packed)
+		EOF
+
+		test_must_fail git multi-pack-index write --bitmap 2>err &&
+		grep "doesn.t have full closure" err &&
+		test_path_is_missing $midx
+	)
+'
+
+test_expect_success 'setup partial bitmaps' '
+	test_commit packed &&
+	git repack &&
+	test_commit loose &&
+	git multi-pack-index write --bitmap 2>err &&
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_file $midx-$(midx_checksum $objdir).rev
+'
+
+basic_bitmap_tests HEAD~
+
+test_expect_success 'removing a MIDX clears stale bitmaps' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+		test_commit base &&
+		git repack &&
+		git multi-pack-index write --bitmap &&
+
+		# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.
+		stale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&
+		stale_rev=$midx-$(midx_checksum $objdir).rev &&
+		rm $midx &&
+
+		# Then write a new MIDX.
+		test_commit new &&
+		git repack &&
+		git multi-pack-index write --bitmap &&
+
+		test_path_is_file $midx &&
+		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+		test_path_is_file $midx-$(midx_checksum $objdir).rev &&
+		test_path_is_missing $stale_bitmap &&
+		test_path_is_missing $stale_rev
+	)
+'
+
+test_expect_success 'pack.preferBitmapTips' '
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit_bulk --message="%s" 103 &&
+
+		git log --format="%H" >commits.raw &&
+		sort <commits.raw >commits &&
+
+		git log --format="create refs/tags/%s %H" HEAD >refs &&
+		git update-ref --stdin <refs &&
+
+		git multi-pack-index write --bitmap &&
+		test_path_is_file $midx &&
+		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+		test_path_is_file $midx-$(midx_checksum $objdir).rev &&
+
+		test-tool bitmap list-commits | sort >bitmaps &&
+		comm -13 bitmaps commits >before &&
+		test_line_count = 1 before &&
+
+		perl -ne "printf(\"create refs/tags/include/%d \", $.); print" \
+			<before | git update-ref --stdin &&
+
+		rm -fr $midx-$(midx_checksum $objdir).bitmap &&
+		rm -fr $midx-$(midx_checksum $objdir).rev &&
+		rm -fr $midx &&
+
+		git -c pack.preferBitmapTips=refs/tags/include \
+			multi-pack-index write --bitmap &&
+		test-tool bitmap list-commits | sort >bitmaps &&
+		comm -13 bitmaps commits >after &&
+
+		! test_cmp before after
+	)
+'
+
+test_done
Signature · 1 line
2.31.1.163.ga65ce7f831
#220 lore raw

PATCH v4 19/25 t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

In reply to Taylor Blau

From: Jeff King <[email protected]>

Generating a MIDX bitmap causes tests which repack in a partial clone to fail because they are missing objects. Missing objects is an expected component of tests in t0410, so disable this knob altogether. Graceful degradation when writing a bitmap with missing objects is tested in t5326.

---
 t/t0410-partial-clone.sh | 3 +++
 1 file changed, 3 insertions(+)
t/t0410-partial-clone.sh +3 -0
index bbcc51ee8e..bba679685f 100755
@@ -4,6 +4,9 @@ test_description='partial clone'
 
 . ./test-lib.sh
 
+# missing promisor objects cause repacks which write bitmaps to fail
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
 delete_object () {
 	rm $1/.git/objects/$(echo $2 | sed -e 's|^..|&/|')
 }
Signature · 1 line
2.31.1.163.ga65ce7f831
#221 lore raw

PATCH v4 20/25 t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

In reply to Taylor Blau

From: Jeff King <[email protected]>

Generating a MIDX bitmap confuses many of the tests in t5310, which expect to control whether and how bitmaps are written. Since the relevant MIDX-bitmap tests here are covered already in t5326, let's just disable the flag for the whole t5310 script.

---
 t/t5310-pack-bitmaps.sh | 4 ++++
 1 file changed, 4 insertions(+)
t/t5310-pack-bitmaps.sh +4 -0
index 4318f84d53..673baa5c3c 100755
@@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 . "$TEST_DIRECTORY"/lib-bundle.sh
 . "$TEST_DIRECTORY"/lib-bitmap.sh
 
+# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in
+# their place.
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
 objpath () {
 	echo ".git/objects/$(echo "$1" | sed -e 's|\(..\)|\1/|')"
 }
Signature · 1 line
2.31.1.163.ga65ce7f831
#222 lore raw

PATCH v4 21/25 t5319: don't write MIDX bitmaps in t5319

In reply to Taylor Blau

This test is specifically about generating a midx still respecting a pack-based bitmap file. Generating a MIDX bitmap would confuse the test. Let's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to make sure we don't do so.

---
 t/t5319-multi-pack-index.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
t/t5319-multi-pack-index.sh +2 -1
index 9b184bd45e..a81375d920 100755
@@ -504,7 +504,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '
 compare_results_with_midx "after repack"
 
 test_expect_success 'multi-pack-index and pack-bitmap' '
-	git -c repack.writeBitmaps=true repack -ad &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writeBitmaps=true repack -ad &&
 	git multi-pack-index write &&
 	git rev-list --test-bitmap HEAD
 '
Signature · 1 line
2.31.1.163.ga65ce7f831
#223 lore raw

PATCH v4 22/25 t7700: update to work with MIDX bitmap test knob

In reply to Taylor Blau

A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary.

---
 t/t7700-repack.sh | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
t/t7700-repack.sh +12 -6
index 25b235c063..98eda3bfeb 100755
@@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' '
 
 test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '
 	# build on $oid, $packid, and .keep state from previous
-	git repack -Adbl &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&
 	test_has_duplicate_object true
 '
 
 test_expect_success 'writing bitmaps via config can duplicate .keep objects' '
 	# build on $oid, $packid, and .keep state from previous
-	git -c repack.writebitmaps=true repack -Adl &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writebitmaps=true repack -Adl &&
 	test_has_duplicate_object true
 '
 
@@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' '
 
 test_expect_success 'bitmaps are created by default in bare repos' '
 	git clone --bare .git bare.git &&
-	git -C bare.git repack -ad &&
+	rm -f bare.git/objects/pack/*.bitmap &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad &&
 	bitmap=$(ls bare.git/objects/pack/*.bitmap) &&
 	test_path_is_file "$bitmap"
 '
@@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' '
 '
 
 test_expect_success 'bitmaps can be disabled on bare repos' '
-	git -c repack.writeBitmaps=false -C bare.git repack -ad &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writeBitmaps=false -C bare.git repack -ad &&
 	bitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&
 	test -z "$bitmap"
 '
@@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' '
 	keep=${pack%.pack}.keep &&
 	test_when_finished "rm -f \"\$keep\"" &&
 	>"$keep" &&
-	git -C bare.git repack -ad 2>stderr &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad 2>stderr &&
 	test_must_be_empty stderr &&
 	find bare.git/objects/pack/ -type f -name "*.bitmap" >actual &&
 	test_must_be_empty actual
@@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '
 	blob=$(test-tool genrandom big $((1024*1024)) |
 	       git -C bare.git hash-object -w --stdin) &&
 	git -C bare.git update-ref refs/tags/big $blob &&
-	git -C bare.git repack -ad 2>stderr &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad 2>stderr &&
 	test_must_be_empty stderr &&
 	find bare.git/objects/pack -type f -name "*.bitmap" >actual &&
 	test_must_be_empty actual
Signature · 1 line
2.31.1.163.ga65ce7f831
#224 lore raw

PATCH v4 23/25 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'

In reply to Taylor Blau

Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable to also write a multi-pack bitmap when 'GIT_TEST_MULTI_PACK_INDEX' is set.

---
 builtin/repack.c          | 12 ++++++++++--
 ci/run-build-and-tests.sh |  1 +
 midx.h                    |  2 ++
 t/README                  |  4 ++++
 4 files changed, 17 insertions(+), 2 deletions(-)
builtin/repack.c +10 -2
index 5f9bc74adc..82ab668272 100644
@@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		if (!(pack_everything & ALL_INTO_ONE) ||
 		    !is_bare_repository())
 			write_bitmaps = 0;
+	} else if (write_bitmaps &&
+		   git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&
+		   git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {
+		write_bitmaps = 0;
 	}
 	if (pack_kept_objects < 0)
 		pack_kept_objects = write_bitmaps > 0;
@@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		update_server_info(0);
 	remove_temporary_files();
 
-	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
-		write_midx_file(get_object_directory(), NULL, 0);
+	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {
+		unsigned flags = 0;
+		if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))
+			flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;
+		write_midx_file(get_object_directory(), NULL, flags);
+	}
 
 	string_list_clear(&names, 0);
 	string_list_clear(&rollback, 0);
ci/run-build-and-tests.sh +1 -0
index 3ce81ffee9..7ee9ba9325 100755
@@ -23,6 +23,7 @@ linux-gcc)
 	export GIT_TEST_COMMIT_GRAPH=1
 	export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1
 	export GIT_TEST_MULTI_PACK_INDEX=1
+	export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1
 	export GIT_TEST_ADD_I_USE_BUILTIN=1
 	export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master
 	export GIT_TEST_WRITE_REV_INDEX=1
midx.h +2 -0
index 350f4d0a7b..aa3da557bb 100644
@@ -8,6 +8,8 @@ struct pack_entry;
 struct repository;
 
 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX"
+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \
+	"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP"
 
 struct multi_pack_index {
 	struct multi_pack_index *next;
t/README +4 -0
index 9e70122302..12014aa988 100644
@@ -425,6 +425,10 @@ GIT_TEST_MULTI_PACK_INDEX=<boolean>, when true, forces the multi-pack-
 index to be written after every 'git repack' command, and overrides the
 'core.multiPackIndex' setting to true.
 
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=<boolean>, when true, sets the
+'--bitmap' option on all invocations of 'git multi-pack-index write',
+and ignores pack-objects' '--write-bitmap-index'.
+
 GIT_TEST_SIDEBAND_ALL=<boolean>, when true, overrides the
 'uploadpack.allowSidebandAll' setting to true, and when false, forces
 fetch-pack to not request sideband-all (even if the server advertises
Signature · 1 line
2.31.1.163.ga65ce7f831
#225 lore raw

PATCH v4 24/25 p5310: extract full and partial bitmap tests

In reply to Taylor Blau

A new p5326 introduced by the next patch will want these same tests, interjecting its own setup in between. Move them out so that both perf tests can reuse them.

---
 t/perf/lib-bitmap.sh         | 69 ++++++++++++++++++++++++++++++++++++
 t/perf/p5310-pack-bitmaps.sh | 65 ++-------------------------------
 2 files changed, 72 insertions(+), 62 deletions(-)
 create mode 100644 t/perf/lib-bitmap.sh
t/perf/lib-bitmap.sh new file +69 -0
new file mode 100644
index 0000000000..63d3bc7cec
@@ -0,0 +1,69 @@
+# Helper functions for testing bitmap performance; see p5310.
+
+test_full_bitmap () {
+	test_perf 'simulated clone' '
+		git pack-objects --stdout --all </dev/null >/dev/null
+	'
+
+	test_perf 'simulated fetch' '
+		have=$(git rev-list HEAD~100 -1) &&
+		{
+			echo HEAD &&
+			echo ^$have
+		} | git pack-objects --revs --stdout >/dev/null
+	'
+
+	test_perf 'pack to file (bitmap)' '
+		git pack-objects --use-bitmap-index --all pack1b </dev/null >/dev/null
+	'
+
+	test_perf 'rev-list (commits)' '
+		git rev-list --all --use-bitmap-index >/dev/null
+	'
+
+	test_perf 'rev-list (objects)' '
+		git rev-list --all --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list with tag negated via --not --all (objects)' '
+		git rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list with negative tag (objects)' '
+		git rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list count with blob:none' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=blob:none >/dev/null
+	'
+
+	test_perf 'rev-list count with blob:limit=1k' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=blob:limit=1k >/dev/null
+	'
+
+	test_perf 'rev-list count with tree:0' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=tree:0 >/dev/null
+	'
+
+	test_perf 'simulated partial clone' '
+		git pack-objects --stdout --all --filter=blob:none </dev/null >/dev/null
+	'
+}
+
+test_partial_bitmap () {
+	test_perf 'clone (partial bitmap)' '
+		git pack-objects --stdout --all </dev/null >/dev/null
+	'
+
+	test_perf 'pack to file (partial bitmap)' '
+		git pack-objects --use-bitmap-index --all pack2b </dev/null >/dev/null
+	'
+
+	test_perf 'rev-list with tree filter (partial bitmap)' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=tree:0 >/dev/null
+	'
+}
t/perf/p5310-pack-bitmaps.sh +3 -62

82 lines not on this page · raw message

Signature · 1 line
2.31.1.163.ga65ce7f831
#226 lore raw

PATCH v4 25/25 p5326: perf tests for MIDX bitmaps

In reply to Taylor Blau

These new performance tests demonstrate effectively the same behavior as p5310, but use a multi-pack bitmap instead of a single-pack one.

Notably, p5326 does not create a MIDX bitmap with multiple packs. This is so we can measure a direct comparison between it and p5310. Any difference between the two is measuring just the overhead of using MIDX bitmaps.

Here are the results of p5310 and p5326 together, measured at the same time and on the same machine (using a Xenon W-2255 CPU):

    Test                                                  HEAD
    ------------------------------------------------------------------------
    5310.2: repack to disk                                96.78(93.39+11.33)
    5310.3: simulated clone                               9.98(9.79+0.19)
    5310.4: simulated fetch                               1.75(4.26+0.19)
    5310.5: pack to file (bitmap)                         28.20(27.87+8.70)
    5310.6: rev-list (commits)                            0.41(0.36+0.05)
    5310.7: rev-list (objects)                            1.61(1.54+0.07)
    5310.8: rev-list count with blob:none                 0.25(0.21+0.04)
    5310.9: rev-list count with blob:limit=1k             2.65(2.54+0.10)
    5310.10: rev-list count with tree:0                   0.23(0.19+0.04)
    5310.11: simulated partial clone                      4.34(4.21+0.12)
    5310.13: clone (partial bitmap)                       11.05(12.21+0.48)
    5310.14: pack to file (partial bitmap)                31.25(34.22+3.70)
    5310.15: rev-list with tree filter (partial bitmap)   0.26(0.22+0.04)

versus the same tests (this time using a multi-pack index):

    Test                                                  HEAD
    ------------------------------------------------------------------------
    5326.2: setup multi-pack index                        78.99(75.29+11.58)
    5326.3: simulated clone                               11.78(11.56+0.22)
    5326.4: simulated fetch                               1.70(4.49+0.13)
    5326.5: pack to file (bitmap)                         28.02(27.72+8.76)
    5326.6: rev-list (commits)                            0.42(0.36+0.06)
    5326.7: rev-list (objects)                            1.65(1.58+0.06)
    5326.8: rev-list count with blob:none                 0.26(0.21+0.05)
    5326.9: rev-list count with blob:limit=1k             2.97(2.86+0.10)
    5326.10: rev-list count with tree:0                   0.25(0.20+0.04)
    5326.11: simulated partial clone                      5.65(5.49+0.16)
    5326.13: clone (partial bitmap)                       12.22(13.43+0.38)
    5326.14: pack to file (partial bitmap)                30.05(31.57+7.25)
    5326.15: rev-list with tree filter (partial bitmap)   0.24(0.20+0.04)

There is slight overhead in "simulated clone", "simulated partial clone", and "clone (partial bitmap)". Unsurprisingly, that overhead is due to using the MIDX's reverse index to map between bit positions and MIDX positions.

This can be reproduced by running "git repack -adb" along with "git multi-pack-index write --bitmap" in a large-ish repository. Then run:

    $ perf record -o pack.perf git -c core.multiPackIndex=false \
      pack-objects --all --stdout >/dev/null </dev/null
    $ perf record -o midx.perf git -c core.multiPackIndex=true \
      pack-objects --all --stdout >/dev/null </dev/null

and compare the two with "perf diff -c delta -o 1 pack.perf midx.perf". The most notable results are below (the next largest positive delta is +0.14%):

# Event 'cycles'
#
# Baseline Delta Shared Object Symbol
# ........ ....... .................. .......................... #
+5.86% git [.] nth_midxed_offset
+5.24% git [.] nth_midxed_pack_int_id
3.45% +0.97% git [.] offset_to_pack_pos
3.30% +0.57% git [.] pack_pos_to_offset
+0.30% git [.] pack_pos_to_midx

---
 t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh
t/perf/p5326-multi-pack-bitmaps.sh new file +43 -0
new file mode 100755
index 0000000000..5845109ac7
@@ -0,0 +1,43 @@
+#!/bin/sh
+
+test_description='Tests performance using midx bitmaps'
+. ./perf-lib.sh
+. "${TEST_DIRECTORY}/perf/lib-bitmap.sh"
+
+test_perf_large_repo
+
+test_expect_success 'enable multi-pack index' '
+	git config core.multiPackIndex true
+'
+
+test_perf 'setup multi-pack index' '
+	git repack -ad &&
+	git multi-pack-index write --bitmap
+'
+
+test_full_bitmap
+
+test_expect_success 'create partial bitmap state' '
+	# pick a commit to represent the repo tip in the past
+	cutoff=$(git rev-list HEAD~100 -1) &&
+	orig_tip=$(git rev-parse HEAD) &&
+
+	# now pretend we have just one tip
+	rm -rf .git/logs .git/refs/* .git/packed-refs &&
+	git update-ref HEAD $cutoff &&
+
+	# and then repack, which will leave us with a nice
+	# big bitmap pack of the "old" history, and all of
+	# the new history will be loose, as if it had been pushed
+	# up incrementally and exploded via unpack-objects
+	git repack -Ad &&
+	git multi-pack-index write --bitmap &&
+
+	# and now restore our original tip, as if the pushes
+	# had happened
+	git update-ref HEAD $orig_tip
+'
+
+test_partial_bitmap
+
+test_done
Signature · 1 line
2.31.1.163.ga65ce7f831
#227 lore raw
In reply to Taylor Blau
Quoted 7 lines · Taylor Blau

Range-diff against v3:
[...]
9: 40cff5beb5 ! 9: c9fea31fa8 midx: avoid opening multiple MIDXs when writing
@@ Commit message
one and should invalidate the object store's memory of any MIDX that might have existed beforehand.

Quoted 15 lines

+ Note that this now forbids passing object directories that don't belong
+ to alternate repositories over --object-dir, since before we would
+ have happily opened a MIDX in any directory, but now restrict ourselves
+ to only those reachable by r->objects->multi_pack_index (and alternate
+ MIDXs that we can see by walking the next pointer).
+
+ As far as I can tell, supporting arbitrary directories with
+ --object-dir was a historical accident, since even the documentation
+ says <alt> when referring to the value passed to this option.
+
+ A future patch could clean this up and provide a warning() when a
+ non-alternate directory was given, since we'll still write a new MIDX
+ there, we just won't reuse any MIDX that might happen to already exist
+ in that directory.
+

So this is definitely fixed as we discussed. But since that discussion, we've had the thread over in:

https://lore.kernel.org/git/[email protected]/

and its siblings:

https://lore.kernel.org/git/[email protected]/

https://lore.kernel.org/git/[email protected]/

It's not clear to me that we have a resolution on whether calling "cd .. && git multi-pack-index write --object-dir repo.git" is supposed to work.

It has traditionally worked (at least for trivial cases, AFAICT), but I find the behavior surprising and unlike most of the rest of Git, and I'm not at all certain that there aren't subtle bugs lurking (basically anything that wants to do object lookup, like oh say, a bitmap generator).

But if we do want to support it, then we have to find a different solution here, don't we? I think the least-painful version of that is probably recording _whether_ we found ctx.m in the_repository's object_store, and switching behavior based on that (e.g., calling close_midx() versus close_object_store() depending).

-Peff

#228 lore raw
Quoted 32 lines · Jeff King
Quoted 7 lines · Taylor Blau

Range-diff against v3:
[...]
9: 40cff5beb5 ! 9: c9fea31fa8 midx: avoid opening multiple MIDXs when writing
@@ Commit message
one and should invalidate the object store's memory of any MIDX that might have existed beforehand.

Quoted 15 lines

+ Note that this now forbids passing object directories that don't belong
+ to alternate repositories over --object-dir, since before we would
+ have happily opened a MIDX in any directory, but now restrict ourselves
+ to only those reachable by r->objects->multi_pack_index (and alternate
+ MIDXs that we can see by walking the next pointer).
+
+ As far as I can tell, supporting arbitrary directories with
+ --object-dir was a historical accident, since even the documentation
+ says <alt> when referring to the value passed to this option.
+
+ A future patch could clean this up and provide a warning() when a
+ non-alternate directory was given, since we'll still write a new MIDX
+ there, we just won't reuse any MIDX that might happen to already exist
+ in that directory.
+

So this is definitely fixed as we discussed. But since that discussion, we've had the thread over in:

https://lore.kernel.org/git/[email protected]/

and its siblings:

Quoted 7 lines

https://lore.kernel.org/git/[email protected]/

https://lore.kernel.org/git/[email protected]/

It's not clear to me that we have a resolution on whether calling "cd .. && git multi-pack-index write --object-dir repo.git" is supposed to work.

My recommendation would be to do the following things, all in a reroll of this series:

- Fix the bug by which we would delete a .rev or .bitmap file out of a different object store than we were working in (when the caller passes --object-dir).

- Disallow running git multi-pack-index outside of a Git repository.

- Restrict --object-dir to only work with alternates of the repository in the current working directory.

To me, that seems like both the least-surprising behavior, and what would lend itself to the easiest implementation. I would probably argue that the existing behavior (where --object-dir would work against arbitrary repositories) is a bug, and shouldn't continue to be supported.

So my plan would be to do that, which would generate something like the following range-diff. If nobody has any objections, I'd like to send what I currently have in ttaylorr/git on GitHub in the tb/multi-pack-bitmaps branch as a reroll of this series, and then merge that early in the cycle to give it a chance to be tested before we cut 2.34.

Thanks,
Taylor

#229 lore raw
Quoted 2 lines · Taylor Blau

My recommendation would be to do the following things, all in a reroll of this series:

For what it's worth, the substantive changes (which I have not figured out how to include in a range-diff since they are entirely new patches) are these:

- Replacing Johannes' patch with:

    https://github.com/ttaylorr/git/commit/2b1afbd516a75bb43a8aae6ff1cac6a83ed7f589,

- and then adding another patch immediately after it:

    https://github.com/git/git/commit/0a2d4d8dbf3c50eb3e2b659d1dcdf432d3b4d223

...and otherwise keeping the remainder of the series unchanged.

Thanks,
Taylor

#230 lore raw
In reply to Taylor Blau
Quoted 13 lines · Taylor Blau
Quoted 3 lines

It's not clear to me that we have a resolution on whether calling "cd .. && git multi-pack-index write --object-dir repo.git" is supposed to work.

My recommendation would be to do the following things, all in a reroll of this series:

- Fix the bug by which we would delete a .rev or .bitmap file out of a different object store than we were working in (when the caller passes --object-dir).

- Disallow running git multi-pack-index outside of a Git repository.

Quoted 8 lines

- Restrict --object-dir to only work with alternates of the repository in the current working directory.

To me, that seems like both the least-surprising behavior, and what would lend itself to the easiest implementation. I would probably argue that the existing behavior (where --object-dir would work against arbitrary repositories) is a bug, and shouldn't continue to be supported.

All of those seem reasonable to me, and are what I would suggest if we were starting from scratch. My only hesitation is whether people are using the weird behavior of --object-dir in the wild (e.g., are bup folks relying on it).

Johannes, is this something you're using _now_, and it works, or something you hoped to use in the future?

In a sense, "hope to use" does not make you any less disappointed. ;) But what I'm wondering is whether using --object-dir from outside a repo entirely is actually something that even works. I.e., would we be disabling a behavior that was not intended, but does happen to work? Or are we closing off a possibly buggy and half-working part of the system?

-Peff

#231 lore raw
Quoted 6 lines · Jeff King
Quoted 4 lines · Taylor Blau
Quoted 3 lines

It's not clear to me that we have a resolution on whether calling "cd .. && git multi-pack-index write --object-dir repo.git" is supposed to work.

Quoted 6 lines
Quoted 6 lines

My recommendation would be to do the following things, all in a reroll of this series:

  - Fix the bug by which we would delete a .rev or .bitmap file out of a different object store than we were working in (when the caller passes --object-dir).

That was what my patch did, afaict.

Quoted 11 lines
Quoted 10 lines

  - Disallow running git multi-pack-index outside of a Git repository.

  - Restrict --object-dir to only work with alternates of the repository in the current working directory.

To me, that seems like both the least-surprising behavior, and what would lend itself to the easiest implementation. I would probably argue that the existing behavior (where --object-dir would work against arbitrary repositories) is a bug, and shouldn't continue to be supported.

Quoted 7 lines

All of those seem reasonable to me, and are what I would suggest if we were starting from scratch. My only hesitation is whether people are using the weird behavior of --object-dir in the wild (e.g., are bup folks relying on it).

Johannes, is this something you're using _now_, and it works, or something you hoped to use in the future?

I was "hoping" to use

	git multi-pack-index --object-dir=... write

but never

	$ git multi-pack-index write --object-dir=...

which almost seems like it really is more like

	$ git -C ... multi-pack-index write

anyway, because you specify a repo? At least per the above example, I never tried.

As I started playing with that again (I had done before, and it worked) I noticed the segfault, hence my previous patch.

However, what I was thinking of doing is more outlined in this thread:
https://lore.kernel.org/git/[email protected]/

And essentially, as I described later in
https://lore.kernel.org/git/[email protected]/

I have two only vaguely overlapping use cases.

One of them doesn't need "--object-dir", and the other requires that [RFC PATCH] to be applied as well, which would basically let me use only the small subset of git that is "git multi-pack-index" as machinery to *just* do indexing, *without* really ever having a real "repository" that git could otherwise operate on and worry about the actual objects etc.

I might resend that with the code style issues fixed, but the objects seemed more fundamental.

Quoted 4 lines

But what I'm wondering is whether using --object-dir from outside a repo entirely is actually something that even works. I.e., would we be disabling a behavior that was not intended, but does happen to work? Or are we closing off a possibly buggy and half-working part of the system?

Well, it does work now, modulo the segfault, but that is actually a very recent addition, I'd tried this before :)

johannes

#232 lore raw
In reply to Jeff King
Quoted 24 lines · Jeff King
Quoted 13 lines · Taylor Blau
Quoted 3 lines

It's not clear to me that we have a resolution on whether calling "cd .. && git multi-pack-index write --object-dir repo.git" is supposed to work.

My recommendation would be to do the following things, all in a reroll of this series:

- Fix the bug by which we would delete a .rev or .bitmap file out of a different object store than we were working in (when the caller passes --object-dir).

- Disallow running git multi-pack-index outside of a Git repository.

Quoted 8 lines

- Restrict --object-dir to only work with alternates of the repository in the current working directory.

To me, that seems like both the least-surprising behavior, and what would lend itself to the easiest implementation. I would probably argue that the existing behavior (where --object-dir would work against arbitrary repositories) is a bug, and shouldn't continue to be supported.

Quoted 7 lines

All of those seem reasonable to me, and are what I would suggest if we were starting from scratch. My only hesitation is whether people are using the weird behavior of --object-dir in the wild (e.g., are bup folks relying on it).

Johannes, is this something you're using _now_, and it works, or something you hoped to use in the future?

I did some research[1] on what parts of --object-dir have worked (and not worked) in the past, and came to the conclusion that although this behavior is surprising, we do bear the responsibility of continuing to maintain it.

And in that sense, I agree with your "only call close_object_store() if the MIDX we are using came from the object store, or otherwise call close_midx() if it didn't", so that's what I did in the tb/multi-pack-bitmaps branch of my fork[2].

I think that this is the most reasonable path forward, since it resolves Johannes' concerns while also not breaking any existing functionality in the meantime as we add new features on top. It has the added benefit of closing some holes that were open in the past, so I think that it's worth doing.

Before I drop 27 patches onto the inboxes of list subscribers, would you mind taking a look at [1] (and the rest of the patches in [2]) to make sure that you're OK with the approach too?

Thanks,
Taylor

[1]: https://github.com/ttaylorr/git/commit/a24290489c2b30f3caed7e33fe8f85226a12778f
[2]: https://github.com/ttaylorr/git/compare/tb/multi-pack-bitmaps

#233 lore raw
Quoted 30 lines · Taylor Blau
Quoted 29 lines · Jeff King
Quoted 13 lines · Taylor Blau
Quoted 3 lines

It's not clear to me that we have a resolution on whether calling "cd .. && git multi-pack-index write --object-dir repo.git" is supposed to work.

My recommendation would be to do the following things, all in a reroll of this series:

- Fix the bug by which we would delete a .rev or .bitmap file out of a different object store than we were working in (when the caller passes --object-dir).

- Disallow running git multi-pack-index outside of a Git repository.

Quoted 8 lines

- Restrict --object-dir to only work with alternates of the repository in the current working directory.

To me, that seems like both the least-surprising behavior, and what would lend itself to the easiest implementation. I would probably argue that the existing behavior (where --object-dir would work against arbitrary repositories) is a bug, and shouldn't continue to be supported.

All of those seem reasonable to me, and are what I would suggest if we were starting from scratch. My only hesitation is whether people are using the weird behavior of --object-dir in the wild (e.g., are bup folks relying on it).

Quoted 7 lines
Quoted 2 lines

Johannes, is this something you're using _now_, and it works, or something you hoped to use in the future?

I did some research[1] on what parts of --object-dir have worked (and not worked) in the past, and came to the conclusion that although this behavior is surprising, we do bear the responsibility of continuing to maintain it.

Hmm. Upon thinking on in more, here is some evidence to the contrary. The new test, specifically this snippet:

    git init repo &&
    test_when_finished "rm -fr repo" &&
    (
      cd repo &&
      test_commit base &&
      git repack -d
    ) &&

    nongit git multi-pack-index --object-dir=$(pwd)/repo/.git/objects write

will fail with GIT_TEST_DEFAULT_HASH=sha256, since the MIDX internals settle on the hash size via the_hash_algo which doesn't respect the hash algorithm used by the target repository.

And that seems like it never could have worked. Try this at your shell to observe the failure:

    git init --object-format=sha256 repo &&
    git -C repo commit --allow-empty -m initial &&
    git -C repo repack -d &&

    git multi-pack-index write --object-dir=$(pwd)/repo/.git/objects

and get:

error: wrong index v2 file size in /home/ttaylorr/repo/.git/objects/pack/pack-9f08dc78ae6f37407a5acad69e3fdf5a1887eb7da5c043a1ddedc56ea7160814.idx
warning: failed to open pack-index '/home/ttaylorr/repo/.git/objects/pack/pack-9f08dc78ae6f37407a5acad69e3fdf5a1887eb7da5c043a1ddedc56ea7160814.idx'

since we're trying to open a sha256 index with the_hash_algo in sha1-mode.

The question is do we consider this to be a bug in the existing behavior that we should patch, or an indication that the feature shouldn't exist in the first place?

I think that I tend to agree more with the latter, so I'm inclined to drop support for it (where "it" is running the midx command outside of a repository) in this series (i.e., by making the midx builtin have the RUN_SETUP flag instead of RUN_SETUP_GENTLY).

Thoughts?

Thanks,
Taylor

#234 lore raw
Quoted 18 lines · Taylor Blau
Quoted 4 lines

I did some research[1] on what parts of --object-dir have worked (and not worked) in the past, and came to the conclusion that although this behavior is surprising, we do bear the responsibility of continuing to maintain it.

Hmm. Upon thinking on in more, here is some evidence to the contrary. The new test, specifically this snippet:

    git init repo &&
    test_when_finished "rm -fr repo" &&
    (
      cd repo &&
      test_commit base &&
      git repack -d
    ) &&

    nongit git multi-pack-index --object-dir=$(pwd)/repo/.git/objects write
Quoted 3 lines

will fail with GIT_TEST_DEFAULT_HASH=sha256, since the MIDX internals settle on the hash size via the_hash_algo which doesn't respect the hash algorithm used by the target repository.

Yeah, I think this is a good example of the class of things that might
fail: anything that requires the repo config to behave correctly.

I do think the hash format is somewhat unusual here. Most of the changes to the on-disk files are reflected in the files themselves (e.g., pack index v2 is chosen by config at _write_ time, but readers can interpret the file stand-alone).

There may be other config that could influence the writing of the midx, and we'd skip it in this kind of non-repo setup. An example here is repack.usedeltabaseoffset, which midx_repack() tries to respect. Ignoring that doesn't produce a nonsense result, but it doesn't follow what would happen if run from inside the repo.

The other class of problems I'd expect is where part of the midx operation needs to look at other parts of the repo. Bitmap generation is an obvious one there, since we'd want to look at refs to find the reachable tips. Now obviously that's a new feature we're trying to introduce here, so it can't be an existing breakage. But it does make me wonder what other problems might be lurking.

So I dunno. Even if it mostly works now, I'm not sure it's something that I'm all that happy about supporting going forward. It seems like a recipe for subtle bugs where the midx code calls into other library code that assumes that it can look at the repository struct.

-Peff

#235 lore raw
Quoted 37 lines · Jeff King
Quoted 12 lines

The new test, specifically this snippet:

    git init repo &&
    test_when_finished "rm -fr repo" &&
    (
      cd repo &&
      test_commit base &&
      git repack -d
    ) &&

    nongit git multi-pack-index --object-dir=$(pwd)/repo/.git/objects write
Quoted 3 lines

will fail with GIT_TEST_DEFAULT_HASH=sha256, since the MIDX internals settle on the hash size via the_hash_algo which doesn't respect the hash algorithm used by the target repository.

Yeah, I think this is a good example of the class of things that might
fail: anything that requires the repo config to behave correctly.

I do think the hash format is somewhat unusual here. Most of the changes to the on-disk files are reflected in the files themselves (e.g., pack index v2 is chosen by config at _write_ time, but readers can interpret the file stand-alone).

There may be other config that could influence the writing of the midx, and we'd skip it in this kind of non-repo setup. An example here is repack.usedeltabaseoffset, which midx_repack() tries to respect. Ignoring that doesn't produce a nonsense result, but it doesn't follow what would happen if run from inside the repo.

The other class of problems I'd expect is where part of the midx operation needs to look at other parts of the repo. Bitmap generation is an obvious one there, since we'd want to look at refs to find the reachable tips. Now obviously that's a new feature we're trying to introduce here, so it can't be an existing breakage. But it does make me wonder what other problems might be lurking.

Quoted 4 lines

So I dunno. Even if it mostly works now, I'm not sure it's something that I'm all that happy about supporting going forward. It seems like a recipe for subtle bugs where the midx code calls into other library code that assumes that it can look at the repository struct.

I tend to agree that we should first disallow things that are not what we know we definitely need, and the non-repo setup is something we would want to punt on to make sure we have a solid support for the mainstream usecase.

Thanks.

#236 lore raw

v5 cover multi-pack reachability bitmaps

In reply to Taylor Blau

Here is another version of the multi-pack reachability bitmaps series. It is virtually unchanged since last time.

The changes that did occur is that I integrated Johannes' patch from [1] to fix cleaning up MIDX .rev and .bitmap files when using --object-dir. That inspired a lengthy discussion [2] about --object-dir, alternates, object-format and running the MIDX builtin outside of a Git repository.

This series resolves that discussion by leaving everything as-is, and only changing the following:

- git multi-pack-index will not run when outside of a Git repository.

- The --object-dir argument will only recognize object directories belonging to an alternate of the current repository.

- Using --object-dir to point to a repository which uses a different hash than the repository in the current working directory will continue to not work (as was the case before this series).

And because this incorporates [1], we will also not accidentally clean .rev files from the wrong object directory.

I think that this version is ready-to-go, and that we can turn our attention to squashing some of these cross-alternate buglets, and integrating MIDX bitmaps with git repack.

[1]: https://lore.kernel.org/git/[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/

Jeff King (2):
t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

Taylor Blau (25):
pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps pack-bitmap-write.c: gracefully fail to write non-closed bitmaps pack-bitmap-write.c: free existing bitmaps
Documentation: describe MIDX-based bitmaps
midx: disallow running outside of a repository
midx: fix *.rev cleanups with --object-dir
midx: clear auxiliary .rev after replacing the MIDX
midx: reject empty --preferred-pack's
midx: infer preferred pack when not given one
midx: close linked MIDXs, avoid leaking memory
midx: avoid opening multiple MIDXs when writing pack-bitmap.c: introduce 'bitmap_num_objects()' pack-bitmap.c: introduce 'nth_bitmap_object_oid()' pack-bitmap.c: introduce 'bitmap_is_preferred_refname()' pack-bitmap.c: avoid redundant calls to try_partial_reuse
pack-bitmap: read multi-pack bitmaps
pack-bitmap: write multi-pack bitmaps
t5310: move some tests to lib-bitmap.sh t/helper/test-read-midx.c: add --checksum mode
t5326: test multi-pack bitmap behavior
t5319: don't write MIDX bitmaps in t5319
t7700: update to work with MIDX bitmap test knob
midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'
p5310: extract full and partial bitmap tests
p5326: perf tests for MIDX bitmaps

 Documentation/git-multi-pack-index.txt       |  20 +-
 Documentation/technical/bitmap-format.txt    |  71 ++-
 Documentation/technical/multi-pack-index.txt |  10 +-
 builtin/multi-pack-index.c                   |   2 +
 builtin/pack-objects.c                       |   8 +-
 builtin/repack.c                             |  12 +-
 ci/run-build-and-tests.sh                    |   1 +
 git.c                                        |   2 +-
 midx.c                                       | 328 ++++++++++--
 midx.h                                       |   5 +
 pack-bitmap-write.c                          |  79 ++-
 pack-bitmap.c                                | 499 ++++++++++++++++---
 pack-bitmap.h                                |   9 +-
 packfile.c                                   |   2 +-
 t/README                                     |   4 +
 t/helper/test-read-midx.c                    |  16 +-
 t/lib-bitmap.sh                              | 240 +++++++++
 t/perf/lib-bitmap.sh                         |  69 +++
 t/perf/p5310-pack-bitmaps.sh                 |  65 +--
 t/perf/p5326-multi-pack-bitmaps.sh           |  43 ++
 t/t0410-partial-clone.sh                     |  12 +-
 t/t5310-pack-bitmaps.sh                      | 231 +--------
 t/t5319-multi-pack-index.sh                  |  53 +-
 t/t5326-multi-pack-bitmaps.sh                | 286 +++++++++++
 t/t7700-repack.sh                            |  18 +-
 25 files changed, 1644 insertions(+), 441 deletions(-)
 create mode 100644 t/perf/lib-bitmap.sh
 create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh
 create mode 100755 t/t5326-multi-pack-bitmaps.sh

Range-diff against v4:

 1:  92dc0bbc0d =  1:  7815d9929d pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps
 2:  979276bc74 =  2:  629171115a pack-bitmap-write.c: gracefully fail to write non-closed bitmaps
 3:  8f00493955 =  3:  d469c1d8f6 pack-bitmap-write.c: free existing bitmaps
 4:  bc7db926d8 =  4:  158ff797c4 Documentation: describe MIDX-based bitmaps
 -:  ---------- >  5:  5f24be8985 midx: disallow running outside of a repository
 -:  ---------- >  6:  0aacaa9283 midx: fix `*.rev` cleanups with `--object-dir`
 5:  771741844b !  7:  d30e6fe9a5 midx: clear auxiliary .rev after replacing the MIDX
    @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack
      
      	if (flags & MIDX_WRITE_REV_INDEX)
      		write_midx_reverse_index(midx_name, midx_hash, &ctx);
    --	clear_midx_files_ext(the_repository, ".rev", midx_hash);
    +-	clear_midx_files_ext(object_dir, ".rev", midx_hash);
      
      	commit_lock_file(&lk);
      
    -+	clear_midx_files_ext(the_repository, ".rev", midx_hash);
    ++	clear_midx_files_ext(object_dir, ".rev", midx_hash);
     +
      cleanup:
      	for (i = 0; i < ctx.nr; i++) {
 6:  dab5dbf228 =  8:  db2a24a8ae midx: reject empty `--preferred-pack`'s
 7:  31f4517de0 =  9:  059c583e34 midx: infer preferred pack when not given one
 8:  aa3bd96d9b = 10:  6f5ca446f3 midx: close linked MIDXs, avoid leaking memory
 9:  c9fea31fa8 ! 11:  4656608f73 midx: avoid opening multiple MIDXs when writing

@@ Commit message

+ ## Documentation/git-multi-pack-index.txt ##
+@@ Documentation/git-multi-pack-index.txt: OPTIONS
+ Use given directory for the location of Git objects. We check
+ <dir>/packs/multi-pack-index for the current MIDX file, and
+ <dir>/packs for the pack-files to index.
+++
++<dir> must be an alternate of the current repository. +
+ --[no-]progress::
+ Turn progress on/off explicitly. If neither is specified, progress is
+
## midx.c ##
@@ midx.c: static int midx_checksum_valid(struct multi_pack_index *m)
return hashfile_checksum_valid(m->data, m->data_len);

10:  ee72fb7e38 = 12:  4c793df9d1 pack-bitmap.c: introduce 'bitmap_num_objects()'
11:  ede0bf1ce1 = 13:  9f165037ce pack-bitmap.c: introduce 'nth_bitmap_object_oid()'
12:  df6844def0 = 14:  ba5fd71fb3 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'
13:  4e06f051a7 = 15:  06db8dbbc1 pack-bitmap.c: avoid redundant calls to try_partial_reuse
14:  a0d73eb3d3 = 16:  61798853b6 pack-bitmap: read multi-pack bitmaps
15:  9d83ad77ab ! 17:  4968229663 pack-bitmap: write multi-pack bitmaps
    @@ midx.c: static int write_midx_internal(const char *object_dir,
     +			 * corresponding bitmap (or one wasn't requested).
     +			 */
     +			if (!want_bitmap)
    -+				clear_midx_files_ext(the_repository, ".bitmap",
    ++				clear_midx_files_ext(object_dir, ".bitmap",
     +						     NULL);
     +			goto cleanup;
     +		}
    @@ midx.c: static int write_midx_internal(const char *object_dir,
     +		}
     +	}
     +
    -+	close_object_store(the_repository->objects);
    ++	if (ctx.m)
    ++		close_object_store(the_repository->objects);
      
      	commit_lock_file(&lk);
      
    -+	clear_midx_files_ext(the_repository, ".bitmap", midx_hash);
    - 	clear_midx_files_ext(the_repository, ".rev", midx_hash);
    ++	clear_midx_files_ext(object_dir, ".bitmap", midx_hash);
    + 	clear_midx_files_ext(object_dir, ".rev", midx_hash);
      
      cleanup:
     @@ midx.c: static int write_midx_internal(const char *object_dir,
    @@ midx.c: void clear_midx_file(struct repository *r)
      	if (remove_path(midx))
      		die(_("failed to clear multi-pack-index at %s"), midx);
      
    -+	clear_midx_files_ext(r, ".bitmap", NULL);
    - 	clear_midx_files_ext(r, ".rev", NULL);
    ++	clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL);
    + 	clear_midx_files_ext(r->objects->odb->path, ".rev", NULL);
      
      	free(midx);
16:  a92af89884 = 18:  5d60b07e2e t5310: move some tests to lib-bitmap.sh
17:  d47aa4a919 = 19:  1a9c3538db t/helper/test-read-midx.c: add --checksum mode
18:  9d9d9f28a6 = 20:  8895114ace t5326: test multi-pack bitmap behavior
19:  3e0da7e5ed = 21:  94b1317e0c t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
20:  4e0d49a2dd = 22:  a4f4d90bba t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP
21:  47eba8ecf9 = 23:  92a6370e77 t5319: don't write MIDX bitmaps in t5319
22:  3d78afa2ad = 24:  c49dc46fb2 t7700: update to work with MIDX bitmap test knob
23:  c2f94e033d = 25:  44a4800756 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'
24:  6b03016c99 = 26:  bf0981b606 p5310: extract full and partial bitmap tests
25:  d98faa4c2c = 27:  6888fe01aa p5326: perf tests for MIDX bitmaps
Signature · 1 line
2.33.0.96.g73915697e6
#237 lore raw

PATCH v5 1/27 pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps

The special --test-bitmap mode of git rev-list is used to compare the result of an object traversal with a bitmap to check its integrity. This mode does not, however, assert that the types of reachable objects are stored correctly.

Harden this mode by teaching it to also check that each time an object's bit is marked, the corresponding bit should be set in exactly one of the type bitmaps (whose type matches the object's true type).

---
 pack-bitmap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
pack-bitmap.c +48 -0
index d999616c9e..9b11af87aa 100644
@@ -1325,10 +1325,52 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git,
 struct bitmap_test_data {
 	struct bitmap_index *bitmap_git;
 	struct bitmap *base;
+	struct bitmap *commits;
+	struct bitmap *trees;
+	struct bitmap *blobs;
+	struct bitmap *tags;
 	struct progress *prg;
 	size_t seen;
 };
 
+static void test_bitmap_type(struct bitmap_test_data *tdata,
+			     struct object *obj, int pos)
+{
+	enum object_type bitmap_type = OBJ_NONE;
+	int bitmaps_nr = 0;
+
+	if (bitmap_get(tdata->commits, pos)) {
+		bitmap_type = OBJ_COMMIT;
+		bitmaps_nr++;
+	}
+	if (bitmap_get(tdata->trees, pos)) {
+		bitmap_type = OBJ_TREE;
+		bitmaps_nr++;
+	}
+	if (bitmap_get(tdata->blobs, pos)) {
+		bitmap_type = OBJ_BLOB;
+		bitmaps_nr++;
+	}
+	if (bitmap_get(tdata->tags, pos)) {
+		bitmap_type = OBJ_TAG;
+		bitmaps_nr++;
+	}
+
+	if (bitmap_type == OBJ_NONE)
+		die("object %s not found in type bitmaps",
+		    oid_to_hex(&obj->oid));
+
+	if (bitmaps_nr > 1)
+		die("object %s does not have a unique type",
+		    oid_to_hex(&obj->oid));
+
+	if (bitmap_type != obj->type)
+		die("object %s: real type %s, expected: %s",
+		    oid_to_hex(&obj->oid),
+		    type_name(obj->type),
+		    type_name(bitmap_type));
+}
+
 static void test_show_object(struct object *object, const char *name,
 			     void *data)
 {
@@ -1338,6 +1380,7 @@ static void test_show_object(struct object *object, const char *name,
 	bitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);
 	if (bitmap_pos < 0)
 		die("Object not in bitmap: %s\n", oid_to_hex(&object->oid));
+	test_bitmap_type(tdata, object, bitmap_pos);
 
 	bitmap_set(tdata->base, bitmap_pos);
 	display_progress(tdata->prg, ++tdata->seen);
@@ -1352,6 +1395,7 @@ static void test_show_commit(struct commit *commit, void *data)
 				     &commit->object.oid);
 	if (bitmap_pos < 0)
 		die("Object not in bitmap: %s\n", oid_to_hex(&commit->object.oid));
+	test_bitmap_type(tdata, &commit->object, bitmap_pos);
 
 	bitmap_set(tdata->base, bitmap_pos);
 	display_progress(tdata->prg, ++tdata->seen);
@@ -1399,6 +1443,10 @@ void test_bitmap_walk(struct rev_info *revs)
 
 	tdata.bitmap_git = bitmap_git;
 	tdata.base = bitmap_new();
+	tdata.commits = ewah_to_bitmap(bitmap_git->commits);
+	tdata.trees = ewah_to_bitmap(bitmap_git->trees);
+	tdata.blobs = ewah_to_bitmap(bitmap_git->blobs);
+	tdata.tags = ewah_to_bitmap(bitmap_git->tags);
 	tdata.prg = start_progress("Verifying bitmap entries", result_popcnt);
 	tdata.seen = 0;
 
Signature · 1 line
2.33.0.96.g73915697e6
#238 lore raw

PATCH v5 2/27 pack-bitmap-write.c: gracefully fail to write non-closed bitmaps

In reply to Taylor Blau

The set of objects covered by a bitmap must be closed under reachability, since it must be the case that there is a valid bit position assigned for every possible reachable object (otherwise the bitmaps would be incomplete).

Pack bitmaps are never written from 'git repack' unless repacking all-into-one, and so we never write non-closed bitmaps (except in the case of partial clones where we aren't guaranteed to have all objects).

But multi-pack bitmaps change this, since it isn't known whether the set of objects in the MIDX is closed under reachability until walking them. Plumb through a bit that is set when a reachable object isn't found.

As soon as a reachable object isn't found in the set of objects to include in the bitmap, bitmap_writer_build() knows that the set is not closed, and so it now fails gracefully.

A test is added in t0410 to trigger a bitmap write without full reachability closure by removing local copies of some reachable objects from a promisor remote.

---
 builtin/pack-objects.c   |  3 +-
 pack-bitmap-write.c      | 76 ++++++++++++++++++++++++++++------------
 pack-bitmap.h            |  2 +-
 t/t0410-partial-clone.sh |  9 ++++-
 4 files changed, 64 insertions(+), 26 deletions(-)
builtin/pack-objects.c +2 -1
index df49f656b9..b63e06e46c 100644
@@ -1256,7 +1256,8 @@ static void write_pack_file(void)
 
 				bitmap_writer_show_progress(progress);
 				bitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);
-				bitmap_writer_build(&to_pack);
+				if (bitmap_writer_build(&to_pack) < 0)
+					die(_("failed to write bitmap index"));
 				bitmap_writer_finish(written_list, nr_written,
 						     tmpname.buf, write_bitmap_options);
 				write_bitmap_index = 0;
pack-bitmap-write.c +53 -23

159 lines not on this page · raw message

pack-bitmap.h +1 -1
index 99d733eb26..020cd8d868 100644
@@ -87,7 +87,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,
 				      struct commit *commit);
 void bitmap_writer_select_commits(struct commit **indexed_commits,
 		unsigned int indexed_commits_nr, int max_bitmaps);
-void bitmap_writer_build(struct packing_data *to_pack);
+int bitmap_writer_build(struct packing_data *to_pack);
 void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint32_t index_nr,
 			  const char *filename,
t/t0410-partial-clone.sh +8 -1
index a211a66c67..bbcc51ee8e 100755
@@ -536,7 +536,13 @@ test_expect_success 'gc does not repack promisor objects if there are none' '
 repack_and_check () {
 	rm -rf repo2 &&
 	cp -r repo repo2 &&
-	git -C repo2 repack $1 -d &&
+	if test x"$1" = "x--must-fail"
+	then
+		shift
+		test_must_fail git -C repo2 repack $1 -d
+	else
+		git -C repo2 repack $1 -d
+	fi &&
 	git -C repo2 fsck &&
 
 	git -C repo2 cat-file -e $2 &&
@@ -561,6 +567,7 @@ test_expect_success 'repack -d does not irreversibly delete promisor objects' '
 	printf "$THREE\n" | pack_as_from_promisor &&
 	delete_object repo "$ONE" &&
 
+	repack_and_check --must-fail -ab "$TWO" "$THREE" &&
 	repack_and_check -a "$TWO" "$THREE" &&
 	repack_and_check -A "$TWO" "$THREE" &&
 	repack_and_check -l "$TWO" "$THREE"
Signature · 1 line
2.33.0.96.g73915697e6
#239 lore raw

PATCH v5 3/27 pack-bitmap-write.c: free existing bitmaps

In reply to Taylor Blau

When writing a new bitmap, the bitmap writer code attempts to read the existing bitmap (if one is present). This is done in order to quickly permute the bits of any bitmaps for commits which appear in the existing bitmap, and were also selected for the new bitmap.

But since this code was added in 341fa34887 (pack-bitmap-write: use existing bitmaps, 2020-12-08), the resources associated with opening an existing bitmap were never released.

It's fine to ignore this, but it's bad hygiene. It will also cause a problem for the multi-pack-index builtin, which will be responsible not only for writing bitmaps, but also for expiring any old multi-pack bitmaps.

If an existing bitmap was reused here, it will also be expired. That will cause a problem on platforms which require file resources to be closed before unlinking them, like Windows. Avoid this by ensuring we close reused bitmaps with free_bitmap_index() before removing them.

---
 pack-bitmap-write.c | 1 +
 1 file changed, 1 insertion(+)
pack-bitmap-write.c +1 -0
index d374f7884b..142fd0adb8 100644
@@ -520,6 +520,7 @@ int bitmap_writer_build(struct packing_data *to_pack)
 	clear_prio_queue(&queue);
 	clear_prio_queue(&tree_queue);
 	bitmap_builder_clear(&bb);
+	free_bitmap_index(old_bitmap);
 	free(mapping);
 
 	trace2_region_leave("pack-bitmap-write", "building_bitmaps_total",
Signature · 1 line
2.33.0.96.g73915697e6
#240 lore raw

PATCH v5 4/27 Documentation: describe MIDX-based bitmaps

In reply to Taylor Blau

Update the technical documentation to describe the multi-pack bitmap format. This patch merely introduces the new format, and describes its high-level ideas. Git does not yet know how to read nor write these multi-pack variants, and so the subsequent patches will:

- Introduce code to interpret multi-pack bitmaps, according to this document.

- Then, introduce code to write multi-pack bitmaps from the 'git multi-pack-index write' sub-command.

Finally, the implementation will gain tests in subsequent patches (as opposed to inline with the patch teaching Git how to write multi-pack bitmaps) to avoid a cyclic dependency.

---
 Documentation/technical/bitmap-format.txt    | 71 ++++++++++++++++----
 Documentation/technical/multi-pack-index.txt | 10 +--
 2 files changed, 60 insertions(+), 21 deletions(-)
Documentation/technical/bitmap-format.txt +57 -14
index f8c18a0f7a..04b3ec2178 100644
@@ -1,6 +1,44 @@
 GIT bitmap v1 format
 ====================
 
+== Pack and multi-pack bitmaps
+
+Bitmaps store reachability information about the set of objects in a packfile,
+or a multi-pack index (MIDX). The former is defined obviously, and the latter is
+defined as the union of objects in packs contained in the MIDX.
+
+A bitmap may belong to either one pack, or the repository's multi-pack index (if
+it exists). A repository may have at most one bitmap.
+
+An object is uniquely described by its bit position within a bitmap:
+
+	- If the bitmap belongs to a packfile, the __n__th bit corresponds to
+	the __n__th object in pack order. For a function `offset` which maps
+	objects to their byte offset within a pack, pack order is defined as
+	follows:
+
+		o1 <= o2 <==> offset(o1) <= offset(o2)
+
+	- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the
+	__n__th object in MIDX order. With an additional function `pack` which
+	maps objects to the pack they were selected from by the MIDX, MIDX order
+	is defined as follows:
+
+		o1 <= o2 <==> pack(o1) <= pack(o2) /\ offset(o1) <= offset(o2)
+
+	The ordering between packs is done according to the MIDX's .rev file.
+	Notably, the preferred pack sorts ahead of all other packs.
+
+The on-disk representation (described below) of a bitmap is the same regardless
+of whether or not that bitmap belongs to a packfile or a MIDX. The only
+difference is the interpretation of the bits, which is described above.
+
+Certain bitmap extensions are supported (see: Appendix B). No extensions are
+required for bitmaps corresponding to packfiles. For bitmaps that correspond to
+MIDXs, both the bit-cache and rev-cache extensions are required.
+
+== On-disk format
+
 	- A header appears at the beginning:
 
 		4-byte signature: {'B', 'I', 'T', 'M'}
@@ -14,17 +52,19 @@ GIT bitmap v1 format
 			The following flags are supported:
 
 			- BITMAP_OPT_FULL_DAG (0x1) REQUIRED
-			This flag must always be present. It implies that the bitmap
-			index has been generated for a packfile with full closure
-			(i.e. where every single object in the packfile can find
-			 its parent links inside the same packfile). This is a
-			requirement for the bitmap index format, also present in JGit,
-			that greatly reduces the complexity of the implementation.
+			This flag must always be present. It implies that the
+			bitmap index has been generated for a packfile or
+			multi-pack index (MIDX) with full closure (i.e. where
+			every single object in the packfile/MIDX can find its
+			parent links inside the same packfile/MIDX). This is a
+			requirement for the bitmap index format, also present in
+			JGit, that greatly reduces the complexity of the
+			implementation.
 
 			- BITMAP_OPT_HASH_CACHE (0x4)
 			If present, the end of the bitmap file contains
 			`N` 32-bit name-hash values, one per object in the
-			pack. The format and meaning of the name-hash is
+			pack/MIDX. The format and meaning of the name-hash is
 			described below.
 
 		4-byte entry count (network byte order)
@@ -33,7 +73,8 @@ GIT bitmap v1 format
 
 		20-byte checksum
 
-			The SHA1 checksum of the pack this bitmap index belongs to.
+			The SHA1 checksum of the pack/MIDX this bitmap index
+			belongs to.
 
 	- 4 EWAH bitmaps that act as type indexes
 
@@ -50,7 +91,7 @@ GIT bitmap v1 format
 			- Tags
 
 		In each bitmap, the `n`th bit is set to true if the `n`th object
-		in the packfile is of that type.
+		in the packfile or multi-pack index is of that type.
 
 		The obvious consequence is that the OR of all 4 bitmaps will result
 		in a full set (all bits set), and the AND of all 4 bitmaps will
@@ -62,8 +103,9 @@ GIT bitmap v1 format
 		Each entry contains the following:
 
 		- 4-byte object position (network byte order)
-			The position **in the index for the packfile** where the
-			bitmap for this commit is found.
+			The position **in the index for the packfile or
+			multi-pack index** where the bitmap for this commit is
+			found.
 
 		- 1-byte XOR-offset
 			The xor offset used to compress this bitmap. For an entry
@@ -146,10 +188,11 @@ Name-hash cache
 ---------------
 
 If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains
-a cache of 32-bit values, one per object in the pack. The value at
+a cache of 32-bit values, one per object in the pack/MIDX. The value at
 position `i` is the hash of the pathname at which the `i`th object
-(counting in index order) in the pack can be found.  This can be fed
-into the delta heuristics to compare objects with similar pathnames.
+(counting in index or multi-pack index order) in the pack/MIDX can be found.
+This can be fed into the delta heuristics to compare objects with similar
+pathnames.
 
 The hash algorithm used is:
 
Documentation/technical/multi-pack-index.txt +3 -7
index fb688976c4..1a73c3ee20 100644
@@ -71,14 +71,10 @@ Future Work
   still reducing the number of binary searches required for object
   lookups.
 
-- The reachability bitmap is currently paired directly with a single
-  packfile, using the pack-order as the object order to hopefully
-  compress the bitmaps well using run-length encoding. This could be
-  extended to pair a reachability bitmap with a multi-pack-index. If
-  the multi-pack-index is extended to store a "stable object order"
+- If the multi-pack-index is extended to store a "stable object order"
   (a function Order(hash) = integer that is constant for a given hash,
-  even as the multi-pack-index is updated) then a reachability bitmap
-  could point to a multi-pack-index and be updated independently.
+  even as the multi-pack-index is updated) then MIDX bitmaps could be
+  updated independently of the MIDX.
 
 - Packfiles can be marked as "special" using empty files that share
   the initial name but replace ".pack" with ".keep" or ".promisor".
Signature · 1 line
2.33.0.96.g73915697e6
#241 lore raw

PATCH v5 5/27 midx: disallow running outside of a repository

In reply to Taylor Blau

The multi-pack-index command supports working with arbitrary object directories via the --object-dir flag. Though this has historically worked in arbitrary repositories (including when the command itself was run outside of a Git repository), this has been somewhat of an accident.

For example, running:

    git multi-pack-index write --object-dir=/path/to/repo/objects

outside of a Git repository causes a BUG(). This is because the top-level cmd_multi_pack_index() function stops parsing when it sees "write", and then fills in the default object directory (the result of calling get_object_directory()) before handing off to cmd_multi_pack_index_write(). But there is no repository to initialize, and so calling get_object_directory() results in a BUG() (indicating that the current repository is not initialized).

Another case where this doesn't quite work as expected is when operating in a SHA-256 repository. To see the failure, try this in your shell:

    git init --object-format=sha256 repo
    git -C repo commit --allow-empty base
    git -C repo repack -d

    git multi-pack-index --object-dir=$(pwd)/repo/.git/objects write

and observe that we cannot open the .idx file in "repo", because the outermost process assumes that any repository that it works in also uses the default value of the_hash_algo (at the time of writing, SHA-1).

There may be compelling reasons for trying to work around these bugs, but working in arbitrary --object-dir's is non-standard enough (and likewise, these bugs prevalent enough) that I don't think any workflows would be broken by abandoning this behavior.

Accordingly, restrict the multi-pack-index builtin to only work when inside of a Git repository (i.e., its main utility becomes selecting which alternate to operate in), which avoids both of the bugs above.

(Note that you can still trigger a bug when writing a MIDX in an alternate which does not use the same object format as the repository which it is an alternate of, but that is an unrelated bug to this one).

---
 git.c                       | 2 +-
 t/t5319-multi-pack-index.sh | 5 +++++
 2 files changed, 6 insertions(+), 1 deletion(-)
git.c +1 -1
index 18bed9a996..60c2784be4 100644
@@ -561,7 +561,7 @@ static struct cmd_struct commands[] = {
 	{ "merge-tree", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },
 	{ "mktag", cmd_mktag, RUN_SETUP | NO_PARSEOPT },
 	{ "mktree", cmd_mktree, RUN_SETUP },
-	{ "multi-pack-index", cmd_multi_pack_index, RUN_SETUP_GENTLY },
+	{ "multi-pack-index", cmd_multi_pack_index, RUN_SETUP },
 	{ "mv", cmd_mv, RUN_SETUP | NEED_WORK_TREE },
 	{ "name-rev", cmd_name_rev, RUN_SETUP },
 	{ "notes", cmd_notes, RUN_SETUP },
t/t5319-multi-pack-index.sh +5 -0
index 3d4d9f10c3..9034e94c0a 100755
@@ -842,4 +842,9 @@ test_expect_success 'usage shown without sub-command' '
 	! test_i18ngrep "unrecognized subcommand" err
 '
 
+test_expect_success 'complains when run outside of a repository' '
+	nongit test_must_fail git multi-pack-index write 2>err &&
+	grep "not a git repository" err
+'
+
 test_done
Signature · 1 line
2.33.0.96.g73915697e6
#242 lore raw

PATCH v5 6/27 midx: fix `*.rev` cleanups with `--object-dir`

In reply to Taylor Blau

If using --object-dir to point into an object directory which belongs to a different repository than the one in the current working directory, such as:

git init repo
git -C repo ... # add some objects
cd alternate
git multi-pack-index --object-dir ../repo/.git/objects write

the binary will segfault trying to access the object-dir via the repo it found, but that's not fully initialized. Worse, if we later call clear_midx_files_ext(), we will use the_repository and remove files out of the wrong object directory.

Fix this by using the given object_dir (or the object directory of the_repository if --object-dir wasn't given) to properly to clean up the *.rev files, avoiding the crash.

---
 midx.c                      | 10 +++++-----
 t/t5319-multi-pack-index.sh | 28 ++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+), 5 deletions(-)
midx.c +5 -5
index 321c6fdd2f..902e1a7a7d 100644
@@ -882,7 +882,7 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,
 	strbuf_release(&buf);
 }
 
-static void clear_midx_files_ext(struct repository *r, const char *ext,
+static void clear_midx_files_ext(const char *object_dir, const char *ext,
 				 unsigned char *keep_hash);
 
 static int midx_checksum_valid(struct multi_pack_index *m)
@@ -1086,7 +1086,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	if (flags & MIDX_WRITE_REV_INDEX)
 		write_midx_reverse_index(midx_name, midx_hash, &ctx);
-	clear_midx_files_ext(the_repository, ".rev", midx_hash);
+	clear_midx_files_ext(object_dir, ".rev", midx_hash);
 
 	commit_lock_file(&lk);
 
@@ -1135,7 +1135,7 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len,
 		die_errno(_("failed to remove %s"), full_path);
 }
 
-static void clear_midx_files_ext(struct repository *r, const char *ext,
+static void clear_midx_files_ext(const char *object_dir, const char *ext,
 				 unsigned char *keep_hash)
 {
 	struct clear_midx_data data;
@@ -1146,7 +1146,7 @@ static void clear_midx_files_ext(struct repository *r, const char *ext,
 				    hash_to_hex(keep_hash), ext);
 	data.ext = ext;
 
-	for_each_file_in_pack_dir(r->objects->odb->path,
+	for_each_file_in_pack_dir(object_dir,
 				  clear_midx_file_ext,
 				  &data);
 
@@ -1165,7 +1165,7 @@ void clear_midx_file(struct repository *r)
 	if (remove_path(midx))
 		die(_("failed to clear multi-pack-index at %s"), midx);
 
-	clear_midx_files_ext(r, ".rev", NULL);
+	clear_midx_files_ext(r->objects->odb->path, ".rev", NULL);
 
 	free(midx);
 }
t/t5319-multi-pack-index.sh +28 -0
index 9034e94c0a..e953cdd6d1 100755
@@ -201,6 +201,34 @@ test_expect_success 'write midx with twelve packs' '
 
 compare_results_with_midx "twelve packs"
 
+test_expect_success 'multi-pack-index *.rev cleanup with --object-dir' '
+	git init repo &&
+	git clone -s repo alternate &&
+
+	test_when_finished "rm -rf repo alternate" &&
+
+	(
+		cd repo &&
+		test_commit base &&
+		git repack -d
+	) &&
+
+	ours="alternate/.git/objects/pack/multi-pack-index-123.rev" &&
+	theirs="repo/.git/objects/pack/multi-pack-index-abc.rev" &&
+	touch "$ours" "$theirs" &&
+
+	(
+		cd alternate &&
+		git multi-pack-index --object-dir ../repo/.git/objects write
+	) &&
+
+	# writing a midx in "repo" should not remove the .rev file in the
+	# alternate
+	test_path_is_file repo/.git/objects/pack/multi-pack-index &&
+	test_path_is_file $ours &&
+	test_path_is_missing $theirs
+'
+
 test_expect_success 'warn on improper hash version' '
 	git init --object-format=sha1 sha1 &&
 	(
Signature · 1 line
2.33.0.96.g73915697e6
#243 lore raw

PATCH v5 7/27 midx: clear auxiliary .rev after replacing the MIDX

In reply to Taylor Blau

When writing a new multi-pack index, write_midx_internal() attempts to clean up any auxiliary files (currently just the MIDX's .rev file, but soon to include a .bitmap, too) corresponding to the MIDX it's replacing.

This step should happen after the new MIDX is written into place, since doing so beforehand means that the old MIDX could be read without its corresponding .rev file.

---
 midx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
midx.c +2 -1
index 902e1a7a7d..0bcb403bae 100644
@@ -1086,10 +1086,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 
 	if (flags & MIDX_WRITE_REV_INDEX)
 		write_midx_reverse_index(midx_name, midx_hash, &ctx);
-	clear_midx_files_ext(object_dir, ".rev", midx_hash);
 
 	commit_lock_file(&lk);
 
+	clear_midx_files_ext(object_dir, ".rev", midx_hash);
+
 cleanup:
 	for (i = 0; i < ctx.nr; i++) {
 		if (ctx.info[i].p) {
Signature · 1 line
2.33.0.96.g73915697e6
#244 lore raw

PATCH v5 8/27 midx: reject empty `--preferred-pack`'s

In reply to Taylor Blau

The soon-to-be-implemented multi-pack bitmap treats object in the first bit position specially by assuming that all objects in the pack it was selected from are also represented from that pack in the MIDX. In other words, the pack from which the first object was selected must also have all of its other objects selected from that same pack in the MIDX in case of any duplicates.

But this assumption relies on the fact that there is at least one object in that pack to begin with; otherwise the object in the first bit position isn't from a preferred pack, in which case we can no longer assume that all objects in that pack were also selected from the same pack.

Guard this assumption by checking the number of objects in the given preferred pack, and failing if the given pack is empty.

To make sure we can safely perform this check, open any packs which are contained in an existing MIDX via prepare_midx_pack(). The same is done for new packs via the add_pack_to_midx() callback, but packs picked up from a previous MIDX will not yet have these opened.

---
 Documentation/git-multi-pack-index.txt |  6 +++---
 midx.c                                 | 29 ++++++++++++++++++++++++++
 t/t5319-multi-pack-index.sh            | 17 +++++++++++++++
 3 files changed, 49 insertions(+), 3 deletions(-)
Documentation/git-multi-pack-index.txt +3 -3
index ffd601bc17..c9b063d31e 100644
@@ -37,9 +37,9 @@ write::
 --
 	--preferred-pack=<pack>::
 		Optionally specify the tie-breaking pack used when
-		multiple packs contain the same object. If not given,
-		ties are broken in favor of the pack with the lowest
-		mtime.
+		multiple packs contain the same object. `<pack>` must
+		contain at least one object. If not given, ties are
+		broken in favor of the pack with the lowest mtime.
 --
 
 verify::
midx.c +29 -0
index 0bcb403bae..26089ec9c7 100644
@@ -934,6 +934,25 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			ctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);
 			ctx.info[ctx.nr].p = NULL;
 			ctx.info[ctx.nr].expired = 0;
+
+			if (flags & MIDX_WRITE_REV_INDEX) {
+				/*
+				 * If generating a reverse index, need to have
+				 * packed_git's loaded to compare their
+				 * mtimes and object count.
+				 */
+				if (prepare_midx_pack(the_repository, ctx.m, i)) {
+					error(_("could not load pack"));
+					result = 1;
+					goto cleanup;
+				}
+
+				if (open_pack_index(ctx.m->packs[i]))
+					die(_("could not open index for %s"),
+					    ctx.m->packs[i]->pack_name);
+				ctx.info[ctx.nr].p = ctx.m->packs[i];
+			}
+
 			ctx.nr++;
 		}
 	}
@@ -961,6 +980,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 		}
 	}
 
+	if (ctx.preferred_pack_idx > -1) {
+		struct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;
+		if (!preferred->num_objects) {
+			error(_("cannot select preferred pack %s with no objects"),
+			      preferred->pack_name);
+			result = 1;
+			goto cleanup;
+		}
+	}
+
 	ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,
 					 ctx.preferred_pack_idx);
 
t/t5319-multi-pack-index.sh +17 -0
index e953cdd6d1..d7e4988f2b 100755
@@ -305,6 +305,23 @@ test_expect_success 'midx picks objects from preferred pack' '
 	)
 '
 
+test_expect_success 'preferred packs must be non-empty' '
+	test_when_finished rm -rf preferred.git &&
+	git init preferred.git &&
+	(
+		cd preferred.git &&
+
+		test_commit base &&
+		git repack -ad &&
+
+		empty="$(git pack-objects $objdir/pack/pack </dev/null)" &&
+
+		test_must_fail git multi-pack-index write \
+			--preferred-pack=pack-$empty.pack 2>err &&
+		grep "with no objects" err
+	)
+'
+
 test_expect_success 'verify multi-pack-index success' '
 	git multi-pack-index verify --object-dir=$objdir
 '
Signature · 1 line
2.33.0.96.g73915697e6
#245 lore raw

PATCH v5 9/27 midx: infer preferred pack when not given one

In reply to Taylor Blau

In 9218c6a40c (midx: allow marking a pack as preferred, 2021-03-30), the multi-pack index code learned how to select a pack which all duplicate objects are selected from. That is, if an object appears in multiple packs, select the copy in the preferred pack before breaking ties according to the other rules like pack mtime and readdir() order.

Not specifying a preferred pack can cause serious problems with multi-pack reachability bitmaps, because these bitmaps rely on having at least one pack from which all duplicates are selected. Not having such a pack causes problems with the code in pack-objects to reuse packs verbatim (e.g., that code assumes that a delta object in a chunk of pack sent verbatim will have its base object sent from the same pack).

So why does not marking a pack preferred cause problems here? The reason is roughly as follows:

- Ties are broken (when handling duplicate objects) by sorting according to midx_oid_compare(), which sorts objects by OID, preferred-ness, pack mtime, and finally pack ID (more on that later).

- The psuedo pack-order (described in Documentation/technical/pack-format.txt under the section "multi-pack-index reverse indexes") is computed by midx_pack_order(), and sorts by pack ID and pack offset, with preferred packs sorting first.

- But! Pack IDs come from incrementing the pack count in add_pack_to_midx(), which is a callback to for_each_file_in_pack_dir(), meaning that pack IDs are assigned in readdir() order.

When specifying a preferred pack, all of that works fine, because duplicate objects are correctly resolved in favor of the copy in the preferred pack, and the preferred pack sorts first in the object order.

"Sorting first" is critical, because the bitmap code relies on finding out which pack holds the first object in the MIDX's pseudo pack-order to determine which pack is preferred.

But if we didn't specify a preferred pack, and the pack which comes first in readdir() order does not also have the lowest timestamp, then it's possible that that pack (the one that sorts first in pseudo-pack order, which the bitmap code will treat as the preferred one) did *not* have all duplicate objects resolved in its favor, resulting in breakage.

The fix is simple: pick a (semi-arbitrary, non-empty) preferred pack when none was specified. This forces that pack to have duplicates resolved in its favor, and (critically) to sort first in pseudo-pack order. Unfortunately, testing this behavior portably isn't possible, since it depends on readdir() order which isn't guaranteed by POSIX.

(Note that multi-pack reachability bitmaps have yet to be implemented; so in that sense this patch is fixing a bug which does not yet exist. But by having this patch beforehand, we can prevent the bug from ever materializing.)

---
 midx.c | 50 ++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 44 insertions(+), 6 deletions(-)
midx.c +44 -6
index 26089ec9c7..67de1dbaeb 100644
@@ -969,15 +969,57 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)
 		goto cleanup;
 
-	ctx.preferred_pack_idx = -1;
 	if (preferred_pack_name) {
+		int found = 0;
 		for (i = 0; i < ctx.nr; i++) {
 			if (!cmp_idx_or_pack_name(preferred_pack_name,
 						  ctx.info[i].pack_name)) {
 				ctx.preferred_pack_idx = i;
+				found = 1;
 				break;
 			}
 		}
+
+		if (!found)
+			warning(_("unknown preferred pack: '%s'"),
+				preferred_pack_name);
+	} else if (ctx.nr && (flags & MIDX_WRITE_REV_INDEX)) {
+		struct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;
+		ctx.preferred_pack_idx = 0;
+
+		if (packs_to_drop && packs_to_drop->nr)
+			BUG("cannot write a MIDX bitmap during expiration");
+
+		/*
+		 * set a preferred pack when writing a bitmap to ensure that
+		 * the pack from which the first object is selected in pseudo
+		 * pack-order has all of its objects selected from that pack
+		 * (and not another pack containing a duplicate)
+		 */
+		for (i = 1; i < ctx.nr; i++) {
+			struct packed_git *p = ctx.info[i].p;
+
+			if (!oldest->num_objects || p->mtime < oldest->mtime) {
+				oldest = p;
+				ctx.preferred_pack_idx = i;
+			}
+		}
+
+		if (!oldest->num_objects) {
+			/*
+			 * If all packs are empty; unset the preferred index.
+			 * This is acceptable since there will be no duplicate
+			 * objects to resolve, so the preferred value doesn't
+			 * matter.
+			 */
+			ctx.preferred_pack_idx = -1;
+		}
+	} else {
+		/*
+		 * otherwise don't mark any pack as preferred to avoid
+		 * interfering with expiration logic below
+		 */
+		ctx.preferred_pack_idx = -1;
 	}
 
 	if (ctx.preferred_pack_idx > -1) {
@@ -1058,11 +1100,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 						      ctx.info, ctx.nr,
 						      sizeof(*ctx.info),
 						      idx_or_pack_name_cmp);
-
-		if (!preferred)
-			warning(_("unknown preferred pack: '%s'"),
-				preferred_pack_name);
-		else {
+		if (preferred) {
 			uint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];
 			if (perm == PACK_EXPIRED)
 				warning(_("preferred pack '%s' is expired"),
Signature · 1 line
2.33.0.96.g73915697e6
#246 lore raw

PATCH v5 10/27 midx: close linked MIDXs, avoid leaking memory

In reply to Taylor Blau

When a repository has at least one alternate, the MIDX belonging to each alternate is accessed through the next pointer on the main object store's copy of the MIDX. close_midx() didn't bother to close any of the linked MIDXs. It likewise didn't free the memory pointed to by m, leaving uninitialized bytes with live pointers to them left around in the heap.

Clean this up by closing linked MIDXs, and freeing up the memory pointed to by each of them. When callers call close_midx(), then they can discard the entire linked list of MIDXs and set their pointer to the head of that list to NULL.

This isn't strictly required for the upcoming patches, but it makes it much more difficult (though still possible, for e.g., by calling close_midx(m->next) which leaves m->next pointing at uninitialized bytes) to have pointers to uninitialized memory.

---
 midx.c | 3 +++
 1 file changed, 3 insertions(+)
midx.c +3 -0
index 67de1dbaeb..e83f22b5ee 100644
@@ -195,6 +195,8 @@ void close_midx(struct multi_pack_index *m)
 	if (!m)
 		return;
 
+	close_midx(m->next);
+
 	munmap((unsigned char *)m->data, m->data_len);
 
 	for (i = 0; i < m->num_packs; i++) {
@@ -203,6 +205,7 @@ void close_midx(struct multi_pack_index *m)
 	}
 	FREE_AND_NULL(m->packs);
 	FREE_AND_NULL(m->pack_names);
+	free(m);
 }
 
 int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)
Signature · 1 line
2.33.0.96.g73915697e6
#247 lore raw

PATCH v5 11/27 midx: avoid opening multiple MIDXs when writing

In reply to Taylor Blau

Opening multiple instance of the same MIDX can lead to problems like two separate packed_git structures which represent the same pack being added to the repository's object store.

The above scenario can happen because prepare_midx_pack() checks if m->packs[pack_int_id] is NULL in order to determine if a pack has been opened and installed in the repository before. But a caller can construct two copies of the same MIDX by calling get_multi_pack_index() and load_multi_pack_index() since the former manipulates the object store directly but the latter is a lower-level routine which allocates a new MIDX for each call.

So if prepare_midx_pack() is called on multiple MIDXs with the same pack_int_id, then that pack will be installed twice in the object store's packed_git pointer.

This can lead to problems in, for e.g., the pack-bitmap code, which does something like the following (in pack-bitmap.c:open_pack_bitmap()):

    struct bitmap_index *bitmap_git = ...;
    for (p = get_all_packs(r); p; p = p->next) {
      if (open_pack_bitmap_1(bitmap_git, p) == 0)
        ret = 0;
    }

which is a problem if two copies of the same pack exist in the packed_git list because pack-bitmap.c:open_pack_bitmap_1() contains a conditional like the following:

    if (bitmap_git->pack || bitmap_git->midx) {
      /* ignore extra bitmap file; we can only handle one */
      warning("ignoring extra bitmap file: %s", packfile->pack_name);
      close(fd);
      return -1;
    }

Avoid this scenario by not letting write_midx_internal() open a MIDX that isn't also pointed at by the object store. So long as this is the case, other routines should prefer to open MIDXs with get_multi_pack_index() or reprepare_packed_git() instead of creating instances on their own. Because get_multi_pack_index() returns r->object_store->multi_pack_index if it is non-NULL, we'll only have one instance of a MIDX open at one time, avoiding these problems.

To encourage this, drop the struct multi_pack_index * parameter from write_midx_internal(), and rely instead on the object_dir to find (or initialize) the correct MIDX instance.

Likewise, replace the call to close_midx() with close_object_store(), since we're about to replace the MIDX with a new one and should invalidate the object store's memory of any MIDX that might have existed beforehand.

Note that this now forbids passing object directories that don't belong to alternate repositories over --object-dir, since before we would have happily opened a MIDX in any directory, but now restrict ourselves to only those reachable by r->objects->multi_pack_index (and alternate MIDXs that we can see by walking the next pointer).

As far as I can tell, supporting arbitrary directories with --object-dir was a historical accident, since even the documentation says <alt> when referring to the value passed to this option.

A future patch could clean this up and provide a warning() when a non-alternate directory was given, since we'll still write a new MIDX there, we just won't reuse any MIDX that might happen to already exist in that directory.

---
 Documentation/git-multi-pack-index.txt |  2 ++
 midx.c                                 | 26 +++++++++++++++-----------
 2 files changed, 17 insertions(+), 11 deletions(-)
Documentation/git-multi-pack-index.txt +2 -0
index c9b063d31e..0af6beb2dd 100644
@@ -23,6 +23,8 @@ OPTIONS
 	Use given directory for the location of Git objects. We check
 	`<dir>/packs/multi-pack-index` for the current MIDX file, and
 	`<dir>/packs` for the pack-files to index.
++
+`<dir>` must be an alternate of the current repository.
 
 --[no-]progress::
 	Turn progress on/off explicitly. If neither is specified, progress is
midx.c +15 -11
index e83f22b5ee..43510290dc 100644
@@ -893,7 +893,7 @@ static int midx_checksum_valid(struct multi_pack_index *m)
 	return hashfile_checksum_valid(m->data, m->data_len);
 }
 
-static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
+static int write_midx_internal(const char *object_dir,
 			       struct string_list *packs_to_drop,
 			       const char *preferred_pack_name,
 			       unsigned flags)
@@ -904,6 +904,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	struct hashfile *f = NULL;
 	struct lock_file lk;
 	struct write_midx_context ctx = { 0 };
+	struct multi_pack_index *cur;
 	int pack_name_concat_len = 0;
 	int dropped_packs = 0;
 	int result = 0;
@@ -914,10 +915,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 		die_errno(_("unable to create leading directories of %s"),
 			  midx_name);
 
-	if (m)
-		ctx.m = m;
-	else
-		ctx.m = load_multi_pack_index(object_dir, 1);
+	for (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {
+		if (!strcmp(object_dir, cur->object_dir)) {
+			ctx.m = cur;
+			break;
+		}
+	}
 
 	if (ctx.m && !midx_checksum_valid(ctx.m)) {
 		warning(_("ignoring existing multi-pack-index; checksum mismatch"));
@@ -1119,7 +1122,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
 
 	if (ctx.m)
-		close_midx(ctx.m);
+		close_object_store(the_repository->objects);
 
 	if (ctx.nr - dropped_packs == 0) {
 		error(_("no pack files to index."));
@@ -1182,8 +1185,7 @@ int write_midx_file(const char *object_dir,
 		    const char *preferred_pack_name,
 		    unsigned flags)
 {
-	return write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,
-				   flags);
+	return write_midx_internal(object_dir, NULL, preferred_pack_name, flags);
 }
 
 struct clear_midx_data {
@@ -1461,8 +1463,10 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla
 
 	free(count);
 
-	if (packs_to_drop.nr)
-		result = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);
+	if (packs_to_drop.nr) {
+		result = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);
+		m = NULL;
+	}
 
 	string_list_clear(&packs_to_drop, 0);
 	return result;
@@ -1651,7 +1655,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,
 		goto cleanup;
 	}
 
-	result = write_midx_internal(object_dir, m, NULL, NULL, flags);
+	result = write_midx_internal(object_dir, NULL, NULL, flags);
 	m = NULL;
 
 cleanup:
Signature · 1 line
2.33.0.96.g73915697e6
#248 lore raw

PATCH v5 12/27 pack-bitmap.c: introduce 'bitmap_num_objects()'

In reply to Taylor Blau

A subsequent patch to support reading MIDX bitmaps will be less noisy after extracting a generic function to return how many objects are contained in a bitmap.

---
 pack-bitmap.c | 37 +++++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 16 deletions(-)
pack-bitmap.c +21 -16
index 9b11af87aa..65356f9657 100644
@@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)
 	return b;
 }
 
+static uint32_t bitmap_num_objects(struct bitmap_index *index)
+{
+	return index->pack->num_objects;
+}
+
 static int load_bitmap_header(struct bitmap_index *index)
 {
 	struct bitmap_disk_header *header = (void *)index->map;
@@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index)
 	/* Parse known bitmap format options */
 	{
 		uint32_t flags = ntohs(header->options);
-		size_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t));
+		size_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t));
 		unsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz;
 
 		if ((flags & BITMAP_OPT_FULL_DAG) == 0)
@@ -404,7 +409,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,
 
 	if (pos < kh_end(positions)) {
 		int bitmap_pos = kh_value(positions, pos);
-		return bitmap_pos + bitmap_git->pack->num_objects;
+		return bitmap_pos + bitmap_num_objects(bitmap_git);
 	}
 
 	return -1;
@@ -456,7 +461,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,
 		bitmap_pos = kh_value(eindex->positions, hash_pos);
 	}
 
-	return bitmap_pos + bitmap_git->pack->num_objects;
+	return bitmap_pos + bitmap_num_objects(bitmap_git);
 }
 
 struct bitmap_show_data {
@@ -673,7 +678,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,
 	for (i = 0; i < eindex->count; ++i) {
 		struct object *obj;
 
-		if (!bitmap_get(objects, bitmap_git->pack->num_objects + i))
+		if (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))
 			continue;
 
 		obj = eindex->objects[i];
@@ -832,7 +837,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,
 	 * them individually.
 	 */
 	for (i = 0; i < eindex->count; i++) {
-		uint32_t pos = i + bitmap_git->pack->num_objects;
+		uint32_t pos = i + bitmap_num_objects(bitmap_git);
 		if (eindex->objects[i]->type == type &&
 		    bitmap_get(to_filter, pos) &&
 		    !bitmap_get(tips, pos))
@@ -859,7 +864,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 
 	oi.sizep = &size;
 
-	if (pos < pack->num_objects) {
+	if (pos < bitmap_num_objects(bitmap_git)) {
 		off_t ofs = pack_pos_to_offset(pack, pos);
 		if (packed_object_info(the_repository, pack, ofs, &oi) < 0) {
 			struct object_id oid;
@@ -869,7 +874,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 		}
 	} else {
 		struct eindex *eindex = &bitmap_git->ext_index;
-		struct object *obj = eindex->objects[pos - pack->num_objects];
+		struct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];
 		if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)
 			die(_("unable to get size of %s"), oid_to_hex(&obj->oid));
 	}
@@ -911,7 +916,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,
 	}
 
 	for (i = 0; i < eindex->count; i++) {
-		uint32_t pos = i + bitmap_git->pack->num_objects;
+		uint32_t pos = i + bitmap_num_objects(bitmap_git);
 		if (eindex->objects[i]->type == OBJ_BLOB &&
 		    bitmap_get(to_filter, pos) &&
 		    !bitmap_get(tips, pos) &&
@@ -1137,8 +1142,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 	enum object_type type;
 	unsigned long size;
 
-	if (pos >= bitmap_git->pack->num_objects)
-		return; /* not actually in the pack */
+	if (pos >= bitmap_num_objects(bitmap_git))
+		return; /* not actually in the pack or MIDX */
 
 	offset = header = pack_pos_to_offset(bitmap_git->pack, pos);
 	type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);
@@ -1204,6 +1209,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 	struct pack_window *w_curs = NULL;
 	size_t i = 0;
 	uint32_t offset;
+	uint32_t objects_nr = bitmap_num_objects(bitmap_git);
 
 	assert(result);
 
@@ -1211,8 +1217,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 		i++;
 
 	/* Don't mark objects not in the packfile */
-	if (i > bitmap_git->pack->num_objects / BITS_IN_EWORD)
-		i = bitmap_git->pack->num_objects / BITS_IN_EWORD;
+	if (i > objects_nr / BITS_IN_EWORD)
+		i = objects_nr / BITS_IN_EWORD;
 
 	reuse = bitmap_word_alloc(i);
 	memset(reuse->words, 0xFF, i * sizeof(eword_t));
@@ -1296,7 +1302,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,
 
 	for (i = 0; i < eindex->count; ++i) {
 		if (eindex->objects[i]->type == type &&
-			bitmap_get(objects, bitmap_git->pack->num_objects + i))
+			bitmap_get(objects, bitmap_num_objects(bitmap_git) + i))
 			count++;
 	}
 
@@ -1517,7 +1523,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,
 	uint32_t i, num_objects;
 	uint32_t *reposition;
 
-	num_objects = bitmap_git->pack->num_objects;
+	num_objects = bitmap_num_objects(bitmap_git);
 	CALLOC_ARRAY(reposition, num_objects);
 
 	for (i = 0; i < num_objects; ++i) {
@@ -1600,7 +1606,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,
 static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)
 {
 	struct bitmap *result = bitmap_git->result;
-	struct packed_git *pack = bitmap_git->pack;
 	struct eindex *eindex = &bitmap_git->ext_index;
 	off_t total = 0;
 	struct object_info oi = OBJECT_INFO_INIT;
@@ -1612,7 +1617,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)
 	for (i = 0; i < eindex->count; i++) {
 		struct object *obj = eindex->objects[i];
 
-		if (!bitmap_get(result, pack->num_objects + i))
+		if (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i))
 			continue;
 
 		if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0)
Signature · 1 line
2.33.0.96.g73915697e6
#249 lore raw

PATCH v5 13/27 pack-bitmap.c: introduce 'nth_bitmap_object_oid()'

In reply to Taylor Blau

A subsequent patch to support reading MIDX bitmaps will be less noisy after extracting a generic function to fetch the nth OID contained in the bitmap.

---
 pack-bitmap.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
pack-bitmap.c +10 -3
index 65356f9657..612f62da97 100644
@@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)
 
 #define MAX_XOR_OFFSET 160
 
+static int nth_bitmap_object_oid(struct bitmap_index *index,
+				 struct object_id *oid,
+				 uint32_t n)
+{
+	return nth_packed_object_id(oid, index->pack, n);
+}
+
 static int load_bitmap_entries_v1(struct bitmap_index *index)
 {
 	uint32_t i;
@@ -242,7 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
 		xor_offset = read_u8(index->map, &index->map_pos);
 		flags = read_u8(index->map, &index->map_pos);
 
-		if (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0)
+		if (nth_bitmap_object_oid(index, &oid, commit_idx_pos) < 0)
 			return error("corrupt ewah bitmap: commit index %u out of range",
 				     (unsigned)commit_idx_pos);
 
@@ -868,8 +875,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
 		off_t ofs = pack_pos_to_offset(pack, pos);
 		if (packed_object_info(the_repository, pack, ofs, &oi) < 0) {
 			struct object_id oid;
-			nth_packed_object_id(&oid, pack,
-					     pack_pos_to_index(pack, pos));
+			nth_bitmap_object_oid(bitmap_git, &oid,
+					      pack_pos_to_index(pack, pos));
 			die(_("unable to get size of %s"), oid_to_hex(&oid));
 		}
 	} else {
Signature · 1 line
2.33.0.96.g73915697e6
#250 lore raw

PATCH v5 14/27 pack-bitmap.c: introduce 'bitmap_is_preferred_refname()'

In reply to Taylor Blau

In a recent commit, pack-objects learned support for the 'pack.preferBitmapTips' configuration. This patch prepares the multi-pack bitmap code to respect this configuration, too.

The yet-to-be implemented code will find that it is more efficient to check whether each reference contains a prefix found in the configured set of values rather than doing an additional traversal.

Implement a function 'bitmap_is_preferred_refname()' which will perform that check. Its caller will be added in a subsequent patch.

---
 pack-bitmap.c | 16 ++++++++++++++++
 pack-bitmap.h |  1 +
 2 files changed, 17 insertions(+)
pack-bitmap.c +16 -0
index 612f62da97..d5296750eb 100644
@@ -1658,3 +1658,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r)
 {
 	return repo_config_get_value_multi(r, "pack.preferbitmaptips");
 }
+
+int bitmap_is_preferred_refname(struct repository *r, const char *refname)
+{
+	const struct string_list *preferred_tips = bitmap_preferred_tips(r);
+	struct string_list_item *item;
+
+	if (!preferred_tips)
+		return 0;
+
+	for_each_string_list_item(item, preferred_tips) {
+		if (starts_with(refname, item->string))
+			return 1;
+	}
+
+	return 0;
+}
pack-bitmap.h +1 -0
index 020cd8d868..52ea10de51 100644
@@ -94,5 +94,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint16_t options);
 
 const struct string_list *bitmap_preferred_tips(struct repository *r);
+int bitmap_is_preferred_refname(struct repository *r, const char *refname);
 
 #endif
Signature · 1 line
2.33.0.96.g73915697e6
#251 lore raw

PATCH v5 15/27 pack-bitmap.c: avoid redundant calls to try_partial_reuse

In reply to Taylor Blau

try_partial_reuse() is used to mark any bits in the beginning of a bitmap whose objects can be reused verbatim from the pack they came from.

Currently this function returns void, and signals nothing to the caller when bits could not be reused. But multi-pack bitmaps would benefit from having such a signal, because they may try to pass objects which are in bounds, but from a pack other than the preferred one.

Any extra calls are noops because of a conditional in reuse_partial_packfile_from_bitmap(), but those loop iterations can be avoided by letting try_partial_reuse() indicate when it can't accept any more bits for reuse, and then listening to that signal.

---
 pack-bitmap.c | 40 +++++++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 11 deletions(-)
pack-bitmap.c +29 -11
index d5296750eb..4e37f5d574 100644
@@ -1140,22 +1140,26 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,
 	return NULL;
 }
 
-static void try_partial_reuse(struct bitmap_index *bitmap_git,
-			      size_t pos,
-			      struct bitmap *reuse,
-			      struct pack_window **w_curs)
+/*
+ * -1 means "stop trying further objects"; 0 means we may or may not have
+ * reused, but you can keep feeding bits.
+ */
+static int try_partial_reuse(struct bitmap_index *bitmap_git,
+			     size_t pos,
+			     struct bitmap *reuse,
+			     struct pack_window **w_curs)
 {
 	off_t offset, header;
 	enum object_type type;
 	unsigned long size;
 
 	if (pos >= bitmap_num_objects(bitmap_git))
-		return; /* not actually in the pack or MIDX */
+		return -1; /* not actually in the pack or MIDX */
 
 	offset = header = pack_pos_to_offset(bitmap_git->pack, pos);
 	type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);
 	if (type < 0)
-		return; /* broken packfile, punt */
+		return -1; /* broken packfile, punt */
 
 	if (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {
 		off_t base_offset;
@@ -1172,9 +1176,9 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		base_offset = get_delta_base(bitmap_git->pack, w_curs,
 					     &offset, type, header);
 		if (!base_offset)
-			return;
+			return 0;
 		if (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)
-			return;
+			return 0;
 
 		/*
 		 * We assume delta dependencies always point backwards. This
@@ -1186,7 +1190,7 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		 * odd parameters.
 		 */
 		if (base_pos >= pos)
-			return;
+			return 0;
 
 		/*
 		 * And finally, if we're not sending the base as part of our
@@ -1197,13 +1201,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,
 		 * object_entry code path handle it.
 		 */
 		if (!bitmap_get(reuse, base_pos))
-			return;
+			return 0;
 	}
 
 	/*
 	 * If we got here, then the object is OK to reuse. Mark it.
 	 */
 	bitmap_set(reuse, pos);
+	return 0;
 }
 
 int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
@@ -1239,10 +1244,23 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 				break;
 
 			offset += ewah_bit_ctz64(word >> offset);
-			try_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs);
+			if (try_partial_reuse(bitmap_git, pos + offset, reuse,
+					      &w_curs) < 0) {
+				/*
+				 * try_partial_reuse indicated we couldn't reuse
+				 * any bits, so there is no point in trying more
+				 * bits in the current word, or any other words
+				 * in result.
+				 *
+				 * Jump out of both loops to avoid future
+				 * unnecessary calls to try_partial_reuse.
+				 */
+				goto done;
+			}
 		}
 	}
 
+done:
 	unuse_pack(&w_curs);
 
 	*entries = bitmap_popcount(reuse);
Signature · 1 line
2.33.0.96.g73915697e6
#252 lore raw

PATCH v5 16/27 pack-bitmap: read multi-pack bitmaps

In reply to Taylor Blau

This prepares the code in pack-bitmap to interpret the new multi-pack bitmaps described in Documentation/technical/bitmap-format.txt, which mostly involves converting bit positions to accommodate looking them up in a MIDX.

Note that there are currently no writers who write multi-pack bitmaps, and that this will be implemented in the subsequent commit. Note also that get_midx_checksum() and get_midx_filename() are made non-static so they can be called from pack-bitmap.c.

---
 builtin/pack-objects.c |   5 +
 midx.c                 |   4 +-
 midx.h                 |   2 +
 pack-bitmap-write.c    |   2 +-
 pack-bitmap.c          | 357 ++++++++++++++++++++++++++++++++++++-----
 pack-bitmap.h          |   6 +
 packfile.c             |   2 +-
 7 files changed, 336 insertions(+), 42 deletions(-)
builtin/pack-objects.c +5 -0
index b63e06e46c..c26dedfe5d 100644
@@ -1124,6 +1124,11 @@ static void write_reused_pack(struct hashfile *f)
 				break;
 
 			offset += ewah_bit_ctz64(word >> offset);
+			/*
+			 * Can use bit positions directly, even for MIDX
+			 * bitmaps. See comment in try_partial_reuse()
+			 * for why.
+			 */
 			write_reused_pack_one(pos + offset, f, &w_curs);
 			display_progress(progress_state, ++written);
 		}
midx.c +2 -2
index 43510290dc..6a10f7a042 100644
@@ -48,12 +48,12 @@ static uint8_t oid_version(void)
 	}
 }
 
-static const unsigned char *get_midx_checksum(struct multi_pack_index *m)
+const unsigned char *get_midx_checksum(struct multi_pack_index *m)
 {
 	return m->data + m->data_len - the_hash_algo->rawsz;
 }
 
-static char *get_midx_filename(const char *object_dir)
+char *get_midx_filename(const char *object_dir)
 {
 	return xstrfmt("%s/pack/multi-pack-index", object_dir);
 }
midx.h +2 -0
index 8684cf0fef..1172df1a71 100644
@@ -42,6 +42,8 @@ struct multi_pack_index {
 #define MIDX_PROGRESS     (1 << 0)
 #define MIDX_WRITE_REV_INDEX (1 << 1)
 
+const unsigned char *get_midx_checksum(struct multi_pack_index *m);
+char *get_midx_filename(const char *object_dir);
 char *get_midx_rev_filename(struct multi_pack_index *m);
 
 struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
pack-bitmap-write.c +1 -1
index 142fd0adb8..9c55c1531e 100644
@@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show)
 }
 
 /**
- * Build the initial type index for the packfile
+ * Build the initial type index for the packfile or multi-pack-index
  */
 void bitmap_writer_build_type_index(struct packing_data *to_pack,
 				    struct pack_idx_entry **index,
pack-bitmap.c +319 -38

584 lines not on this page · raw message

pack-bitmap.h +6 -0
index 52ea10de51..81664f933f 100644
@@ -44,6 +44,8 @@ typedef int (*show_reachable_fn)(
 struct bitmap_index;
 
 struct bitmap_index *prepare_bitmap_git(struct repository *r);
+struct bitmap_index *prepare_midx_bitmap_git(struct repository *r,
+					     struct multi_pack_index *midx);
 void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,
 			      uint32_t *trees, uint32_t *blobs, uint32_t *tags);
 void traverse_bitmap_commit_list(struct bitmap_index *,
@@ -92,6 +94,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint32_t index_nr,
 			  const char *filename,
 			  uint16_t options);
+char *midx_bitmap_filename(struct multi_pack_index *midx);
+char *pack_bitmap_filename(struct packed_git *p);
+
+int bitmap_is_midx(struct bitmap_index *bitmap_git);
 
 const struct string_list *bitmap_preferred_tips(struct repository *r);
 int bitmap_is_preferred_refname(struct repository *r, const char *refname);
packfile.c +1 -1
index 9ef6d98292..371f5488cf 100644
@@ -860,7 +860,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 	if (!strcmp(file_name, "multi-pack-index"))
 		return;
 	if (starts_with(file_name, "multi-pack-index") &&
-	    ends_with(file_name, ".rev"))
+	    (ends_with(file_name, ".bitmap") || ends_with(file_name, ".rev")))
 		return;
 	if (ends_with(file_name, ".idx") ||
 	    ends_with(file_name, ".rev") ||
Signature · 1 line
2.33.0.96.g73915697e6
#253 lore raw

PATCH v5 17/27 pack-bitmap: write multi-pack bitmaps

In reply to Taylor Blau

Write multi-pack bitmaps in the format described by Documentation/technical/bitmap-format.txt, inferring their presence with the absence of '--bitmap'.

To write a multi-pack bitmap, this patch attempts to reuse as much of the existing machinery from pack-objects as possible. Specifically, the MIDX code prepares a packing_data struct that pretends as if a single packfile has been generated containing all of the objects contained within the MIDX.

---
 Documentation/git-multi-pack-index.txt |  12 +-
 builtin/multi-pack-index.c             |   2 +
 midx.c                                 | 209 ++++++++++++++++++++++++-
 midx.h                                 |   1 +
 4 files changed, 215 insertions(+), 9 deletions(-)
Documentation/git-multi-pack-index.txt +11 -1
index 0af6beb2dd..a9df3dbd32 100644
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [verse]
 'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
-	[--preferred-pack=<pack>] <subcommand>
+	[--preferred-pack=<pack>] [--[no-]bitmap] <subcommand>
 
 DESCRIPTION
 -----------
@@ -42,6 +42,9 @@ write::
 		multiple packs contain the same object. `<pack>` must
 		contain at least one object. If not given, ties are
 		broken in favor of the pack with the lowest mtime.
+
+	--[no-]bitmap::
+		Control whether or not a multi-pack bitmap is written.
 --
 
 verify::
@@ -83,6 +86,13 @@ EXAMPLES
 $ git multi-pack-index write
 -----------------------------------------------
 
+* Write a MIDX file for the packfiles in the current .git folder with a
+corresponding bitmap.
++
+-------------------------------------------------------------
+$ git multi-pack-index write --preferred-pack=<pack> --bitmap
+-------------------------------------------------------------
+
 * Write a MIDX file for the packfiles in an alternate object store.
 +
 -----------------------------------------------
builtin/multi-pack-index.c +2 -0
index 8ff0dee2ec..73c0113b48 100644
@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
 			   N_("preferred-pack"),
 			   N_("pack for reuse when computing a multi-pack bitmap")),
+		OPT_BIT(0, "bitmap", &opts.flags, N_("write multi-pack bitmap"),
+			MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),
 		OPT_END(),
 	};
 
midx.c +201 -8

270 lines not on this page · raw message

midx.h +1 -0
index 1172df1a71..350f4d0a7b 100644
@@ -41,6 +41,7 @@ struct multi_pack_index {
 
 #define MIDX_PROGRESS     (1 << 0)
 #define MIDX_WRITE_REV_INDEX (1 << 1)
+#define MIDX_WRITE_BITMAP (1 << 2)
 
 const unsigned char *get_midx_checksum(struct multi_pack_index *m);
 char *get_midx_filename(const char *object_dir);
Signature · 1 line
2.33.0.96.g73915697e6
#254 lore raw

PATCH v5 18/27 t5310: move some tests to lib-bitmap.sh

In reply to Taylor Blau

We'll soon be adding a test script that will cover many of the same bitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many of the applicable tests as we can so we don't have to rewrite them.

There should be no functional change to t5310; we still run the same operations in the same order.

---
 t/lib-bitmap.sh         | 236 ++++++++++++++++++++++++++++++++++++++++
 t/t5310-pack-bitmaps.sh | 227 +-------------------------------------
 2 files changed, 240 insertions(+), 223 deletions(-)
t/lib-bitmap.sh +236 -0
index fe3f98be24..77464da6fd 100644
@@ -1,3 +1,6 @@
+# Helpers for scripts testing bitmap functionality; see t5310 for
+# example usage.
+
 # Compare a file containing rev-list bitmap traversal output to its non-bitmap
 # counterpart. You can't just use test_cmp for this, because the two produce
 # subtly different output:
@@ -24,3 +27,236 @@ test_bitmap_traversal () {
 	test_cmp "$1.normalized" "$2.normalized" &&
 	rm -f "$1.normalized" "$2.normalized"
 }
+
+# To ensure the logic for "maximal commits" is exercised, make
+# the repository a bit more complicated.
+#
+#    other                         second
+#      *                             *
+# (99 commits)                  (99 commits)
+#      *                             *
+#      |\                           /|
+#      | * octo-other  octo-second * |
+#      |/|\_________  ____________/|\|
+#      | \          \/  __________/  |
+#      |  | ________/\ /             |
+#      *  |/          * merge-right  *
+#      | _|__________/ \____________ |
+#      |/ |                         \|
+# (l1) *  * merge-left               * (r1)
+#      | / \________________________ |
+#      |/                           \|
+# (l2) *                             * (r2)
+#       \___________________________ |
+#                                   \|
+#                                    * (base)
+#
+# We only push bits down the first-parent history, which
+# makes some of these commits unimportant!
+#
+# The important part for the maximal commit algorithm is how
+# the bitmasks are extended. Assuming starting bit positions
+# for second (bit 0) and other (bit 1), the bitmasks at the
+# end should be:
+#
+#      second: 1       (maximal, selected)
+#       other: 01      (maximal, selected)
+#      (base): 11 (maximal)
+#
+# This complicated history was important for a previous
+# version of the walk that guarantees never walking a
+# commit multiple times. That goal might be important
+# again, so preserve this complicated case. For now, this
+# test will guarantee that the bitmaps are computed
+# correctly, even with the repeat calculations.
+setup_bitmap_history() {
+	test_expect_success 'setup repo with moderate-sized history' '
+		test_commit_bulk --id=file 10 &&
+		git branch -M second &&
+		git checkout -b other HEAD~5 &&
+		test_commit_bulk --id=side 10 &&
+
+		# add complicated history setup, including merges and
+		# ambiguous merge-bases
+
+		git checkout -b merge-left other~2 &&
+		git merge second~2 -m "merge-left" &&
+
+		git checkout -b merge-right second~1 &&
+		git merge other~1 -m "merge-right" &&
+
+		git checkout -b octo-second second &&
+		git merge merge-left merge-right -m "octopus-second" &&
+
+		git checkout -b octo-other other &&
+		git merge merge-left merge-right -m "octopus-other" &&
+
+		git checkout other &&
+		git merge octo-other -m "pull octopus" &&
+
+		git checkout second &&
+		git merge octo-second -m "pull octopus" &&
+
+		# Remove these branches so they are not selected
+		# as bitmap tips
+		git branch -D merge-left &&
+		git branch -D merge-right &&
+		git branch -D octo-other &&
+		git branch -D octo-second &&
+
+		# add padding to make these merges less interesting
+		# and avoid having them selected for bitmaps
+		test_commit_bulk --id=file 100 &&
+		git checkout other &&
+		test_commit_bulk --id=side 100 &&
+		git checkout second &&
+
+		bitmaptip=$(git rev-parse second) &&
+		blob=$(echo tagged-blob | git hash-object -w --stdin) &&
+		git tag tagged-blob $blob
+	'
+}
+
+rev_list_tests_head () {
+	test_expect_success "counting commits via bitmap ($state, $branch)" '
+		git rev-list --count $branch >expect &&
+		git rev-list --use-bitmap-index --count $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting partial commits via bitmap ($state, $branch)" '
+		git rev-list --count $branch~5..$branch >expect &&
+		git rev-list --use-bitmap-index --count $branch~5..$branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting commits with limit ($state, $branch)" '
+		git rev-list --count -n 1 $branch >expect &&
+		git rev-list --use-bitmap-index --count -n 1 $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting non-linear history ($state, $branch)" '
+		git rev-list --count other...second >expect &&
+		git rev-list --use-bitmap-index --count other...second >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting commits with limiting ($state, $branch)" '
+		git rev-list --count $branch -- 1.t >expect &&
+		git rev-list --use-bitmap-index --count $branch -- 1.t >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "counting objects via bitmap ($state, $branch)" '
+		git rev-list --count --objects $branch >expect &&
+		git rev-list --use-bitmap-index --count --objects $branch >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success "enumerate commits ($state, $branch)" '
+		git rev-list --use-bitmap-index $branch >actual &&
+		git rev-list $branch >expect &&
+		test_bitmap_traversal --no-confirm-bitmaps expect actual
+	'
+
+	test_expect_success "enumerate --objects ($state, $branch)" '
+		git rev-list --objects --use-bitmap-index $branch >actual &&
+		git rev-list --objects $branch >expect &&
+		test_bitmap_traversal expect actual
+	'
+
+	test_expect_success "bitmap --objects handles non-commit objects ($state, $branch)" '
+		git rev-list --objects --use-bitmap-index $branch tagged-blob >actual &&
+		grep $blob actual
+	'
+}
+
+rev_list_tests () {
+	state=$1
+
+	for branch in "second" "other"
+	do
+		rev_list_tests_head
+	done
+}
+
+basic_bitmap_tests () {
+	tip="$1"
+	test_expect_success 'rev-list --test-bitmap verifies bitmaps' "
+		git rev-list --test-bitmap "${tip:-HEAD}"
+	"
+
+	rev_list_tests 'full bitmap'
+
+	test_expect_success 'clone from bitmapped repository' '
+		rm -fr clone.git &&
+		git clone --no-local --bare . clone.git &&
+		git rev-parse HEAD >expect &&
+		git --git-dir=clone.git rev-parse HEAD >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success 'partial clone from bitmapped repository' '
+		test_config uploadpack.allowfilter true &&
+		rm -fr partial-clone.git &&
+		git clone --no-local --bare --filter=blob:none . partial-clone.git &&
+		(
+			cd partial-clone.git &&
+			pack=$(echo objects/pack/*.pack) &&
+			git verify-pack -v "$pack" >have &&
+			awk "/blob/ { print \$1 }" <have >blobs &&
+			# we expect this single blob because of the direct ref
+			git rev-parse refs/tags/tagged-blob >expect &&
+			test_cmp expect blobs
+		)
+	'
+
+	test_expect_success 'setup further non-bitmapped commits' '
+		test_commit_bulk --id=further 10
+	'
+
+	rev_list_tests 'partial bitmap'
+
+	test_expect_success 'fetch (partial bitmap)' '
+		git --git-dir=clone.git fetch origin second:second &&
+		git rev-parse HEAD >expect &&
+		git --git-dir=clone.git rev-parse HEAD >actual &&
+		test_cmp expect actual
+	'
+
+	test_expect_success 'enumerating progress counts pack-reused objects' '
+		count=$(git rev-list --objects --all --count) &&
+		git repack -adb &&
+
+		# check first with only reused objects; confirm that our
+		# progress showed the right number, and also that we did
+		# pack-reuse as expected.  Check only the final "done"
+		# line of the meter (there may be an arbitrary number of
+		# intermediate lines ending with CR).
+		GIT_PROGRESS_DELAY=0 \
+			git pack-objects --all --stdout --progress \
+			</dev/null >/dev/null 2>stderr &&
+		grep "Enumerating objects: $count, done" stderr &&
+		grep "pack-reused $count" stderr &&
+
+		# now the same but with one non-reused object
+		git commit --allow-empty -m "an extra commit object" &&
+		GIT_PROGRESS_DELAY=0 \
+			git pack-objects --all --stdout --progress \
+			</dev/null >/dev/null 2>stderr &&
+		grep "Enumerating objects: $((count+1)), done" stderr &&
+		grep "pack-reused $count" stderr
+	'
+}
+
+# have_delta <obj> <expected_base>
+#
+# Note that because this relies on cat-file, it might find _any_ copy of an
+# object in the repository. The caller is responsible for making sure
+# there's only one (e.g., via "repack -ad", or having just fetched a copy).
+have_delta () {
+	echo $2 >expect &&
+	echo $1 | git cat-file --batch-check="%(deltabase)" >actual &&
+	test_cmp expect actual
+}
t/t5310-pack-bitmaps.sh +4 -223

246 lines not on this page · raw message

Signature · 1 line
2.33.0.96.g73915697e6
#255 lore raw

PATCH v5 19/27 t/helper/test-read-midx.c: add --checksum mode

In reply to Taylor Blau

Subsequent tests will want to check for the existence of a multi-pack bitmap which matches the multi-pack-index stored in the pack directory.

The multi-pack bitmap includes the hex checksum of the MIDX it corresponds to in its filename (for example, '$packdir/multi-pack-index-<checksum>.bitmap'). As a result, some tests want a way to learn what '<checksum>' is.

This helper addresses that need by printing the checksum of the repository's multi-pack-index.

---
 t/helper/test-read-midx.c | 16 +++++++++++++++-
 t/lib-bitmap.sh           |  4 ++++
 2 files changed, 19 insertions(+), 1 deletion(-)
t/helper/test-read-midx.c +15 -1
index 7c2eb11a8e..cb0d27049a 100644
@@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects)
 	return 0;
 }
 
+static int read_midx_checksum(const char *object_dir)
+{
+	struct multi_pack_index *m;
+
+	setup_git_directory();
+	m = load_multi_pack_index(object_dir, 1);
+	if (!m)
+		return 1;
+	printf("%s\n", hash_to_hex(get_midx_checksum(m)));
+	return 0;
+}
+
 int cmd__read_midx(int argc, const char **argv)
 {
 	if (!(argc == 2 || argc == 3))
-		usage("read-midx [--show-objects] <object-dir>");
+		usage("read-midx [--show-objects|--checksum] <object-dir>");
 
 	if (!strcmp(argv[1], "--show-objects"))
 		return read_midx_file(argv[2], 1);
+	else if (!strcmp(argv[1], "--checksum"))
+		return read_midx_checksum(argv[2]);
 	return read_midx_file(argv[1], 0);
 }
t/lib-bitmap.sh +4 -0
index 77464da6fd..21d0392dda 100644
@@ -260,3 +260,7 @@ have_delta () {
 	echo $1 | git cat-file --batch-check="%(deltabase)" >actual &&
 	test_cmp expect actual
 }
+
+midx_checksum () {
+	test-tool read-midx --checksum "$1"
+}
Signature · 1 line
2.33.0.96.g73915697e6
#256 lore raw

PATCH v5 20/27 t5326: test multi-pack bitmap behavior

In reply to Taylor Blau

This patch introduces a new test, t5326, which tests the basic functionality of multi-pack bitmaps.

Some trivial behavior is tested, such as:

- Whether bitmaps can be generated with more than one pack.
- Whether clones can be served with all objects in the bitmap.
- Whether follow-up fetches can be served with some objects outside of the server's bitmap

These use lib-bitmap's tests (which in turn were pulled from t5310), and we cover cases where the MIDX represents both a single pack and multiple packs.

In addition, some non-trivial and MIDX-specific behavior is tested, too, including:

- Whether multi-pack bitmaps behave correctly with respect to the pack-reuse machinery when the base for some object is selected from a different pack than the delta.
- Whether multi-pack bitmaps correctly respect the pack.preferBitmapTips configuration.

---
 t/t5326-multi-pack-bitmaps.sh | 286 ++++++++++++++++++++++++++++++++++
 1 file changed, 286 insertions(+)
 create mode 100755 t/t5326-multi-pack-bitmaps.sh
t/t5326-multi-pack-bitmaps.sh new file +286 -0
new file mode 100755
index 0000000000..4ad7c2c969
@@ -0,0 +1,286 @@
+#!/bin/sh
+
+test_description='exercise basic multi-pack bitmap functionality'
+. ./test-lib.sh
+. "${TEST_DIRECTORY}/lib-bitmap.sh"
+
+# We'll be writing our own midx and bitmaps, so avoid getting confused by the
+# automatic ones.
+GIT_TEST_MULTI_PACK_INDEX=0
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
+objdir=.git/objects
+midx=$objdir/pack/multi-pack-index
+
+# midx_pack_source <obj>
+midx_pack_source () {
+	test-tool read-midx --show-objects .git/objects | grep "^$1 " | cut -f2
+}
+
+setup_bitmap_history
+
+test_expect_success 'enable core.multiPackIndex' '
+	git config core.multiPackIndex true
+'
+
+test_expect_success 'create single-pack midx with bitmaps' '
+	git repack -ad &&
+	git multi-pack-index write --bitmap &&
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_file $midx-$(midx_checksum $objdir).rev
+'
+
+basic_bitmap_tests
+
+test_expect_success 'create new additional packs' '
+	for i in $(test_seq 1 16)
+	do
+		test_commit "$i" &&
+		git repack -d || return 1
+	done &&
+
+	git checkout -b other2 HEAD~8 &&
+	for i in $(test_seq 1 8)
+	do
+		test_commit "side-$i" &&
+		git repack -d || return 1
+	done &&
+	git checkout second
+'
+
+test_expect_success 'create multi-pack midx with bitmaps' '
+	git multi-pack-index write --bitmap &&
+
+	ls $objdir/pack/pack-*.pack >packs &&
+	test_line_count = 25 packs &&
+
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_file $midx-$(midx_checksum $objdir).rev
+'
+
+basic_bitmap_tests
+
+test_expect_success '--no-bitmap is respected when bitmaps exist' '
+	git multi-pack-index write --bitmap &&
+
+	test_commit respect--no-bitmap &&
+	git repack -d &&
+
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_file $midx-$(midx_checksum $objdir).rev &&
+
+	git multi-pack-index write --no-bitmap &&
+
+	test_path_is_file $midx &&
+	test_path_is_missing $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_missing $midx-$(midx_checksum $objdir).rev
+'
+
+test_expect_success 'setup midx with base from later pack' '
+	# Write a and b so that "a" is a delta on top of base "b", since Git
+	# prefers to delete contents out of a base rather than add to a shorter
+	# object.
+	test_seq 1 128 >a &&
+	test_seq 1 130 >b &&
+
+	git add a b &&
+	git commit -m "initial commit" &&
+
+	a=$(git rev-parse HEAD:a) &&
+	b=$(git rev-parse HEAD:b) &&
+
+	# In the first pack, "a" is stored as a delta to "b".
+	p1=$(git pack-objects .git/objects/pack/pack <<-EOF
+	$a
+	$b
+	EOF
+	) &&
+
+	# In the second pack, "a" is missing, and "b" is not a delta nor base to
+	# any other object.
+	p2=$(git pack-objects .git/objects/pack/pack <<-EOF
+	$b
+	$(git rev-parse HEAD)
+	$(git rev-parse HEAD^{tree})
+	EOF
+	) &&
+
+	git prune-packed &&
+	# Use the second pack as the preferred source, so that "b" occurs
+	# earlier in the MIDX object order, rendering "a" unusable for pack
+	# reuse.
+	git multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx &&
+
+	have_delta $a $b &&
+	test $(midx_pack_source $a) != $(midx_pack_source $b)
+'
+
+rev_list_tests 'full bitmap with backwards delta'
+
+test_expect_success 'clone with bitmaps enabled' '
+	git clone --no-local --bare . clone-reverse-delta.git &&
+	test_when_finished "rm -fr clone-reverse-delta.git" &&
+
+	git rev-parse HEAD >expect &&
+	git --git-dir=clone-reverse-delta.git rev-parse HEAD >actual &&
+	test_cmp expect actual
+'
+
+bitmap_reuse_tests() {
+	from=$1
+	to=$2
+
+	test_expect_success "setup pack reuse tests ($from -> $to)" '
+		rm -fr repo &&
+		git init repo &&
+		(
+			cd repo &&
+			test_commit_bulk 16 &&
+			git tag old-tip &&
+
+			git config core.multiPackIndex true &&
+			if test "MIDX" = "$from"
+			then
+				git repack -Ad &&
+				git multi-pack-index write --bitmap
+			else
+				git repack -Adb
+			fi
+		)
+	'
+
+	test_expect_success "build bitmap from existing ($from -> $to)" '
+		(
+			cd repo &&
+			test_commit_bulk --id=further 16 &&
+			git tag new-tip &&
+
+			if test "MIDX" = "$to"
+			then
+				git repack -d &&
+				git multi-pack-index write --bitmap
+			else
+				git repack -Adb
+			fi
+		)
+	'
+
+	test_expect_success "verify resulting bitmaps ($from -> $to)" '
+		(
+			cd repo &&
+			git for-each-ref &&
+			git rev-list --test-bitmap refs/tags/old-tip &&
+			git rev-list --test-bitmap refs/tags/new-tip
+		)
+	'
+}
+
+bitmap_reuse_tests 'pack' 'MIDX'
+bitmap_reuse_tests 'MIDX' 'pack'
+bitmap_reuse_tests 'MIDX' 'MIDX'
+
+test_expect_success 'missing object closure fails gracefully' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit loose &&
+		test_commit packed &&
+
+		# Do not pass "--revs"; we want a pack without the "loose"
+		# commit.
+		git pack-objects $objdir/pack/pack <<-EOF &&
+		$(git rev-parse packed)
+		EOF
+
+		test_must_fail git multi-pack-index write --bitmap 2>err &&
+		grep "doesn.t have full closure" err &&
+		test_path_is_missing $midx
+	)
+'
+
+test_expect_success 'setup partial bitmaps' '
+	test_commit packed &&
+	git repack &&
+	test_commit loose &&
+	git multi-pack-index write --bitmap 2>err &&
+	test_path_is_file $midx &&
+	test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+	test_path_is_file $midx-$(midx_checksum $objdir).rev
+'
+
+basic_bitmap_tests HEAD~
+
+test_expect_success 'removing a MIDX clears stale bitmaps' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+		test_commit base &&
+		git repack &&
+		git multi-pack-index write --bitmap &&
+
+		# Write a MIDX and bitmap; remove the MIDX but leave the bitmap.
+		stale_bitmap=$midx-$(midx_checksum $objdir).bitmap &&
+		stale_rev=$midx-$(midx_checksum $objdir).rev &&
+		rm $midx &&
+
+		# Then write a new MIDX.
+		test_commit new &&
+		git repack &&
+		git multi-pack-index write --bitmap &&
+
+		test_path_is_file $midx &&
+		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+		test_path_is_file $midx-$(midx_checksum $objdir).rev &&
+		test_path_is_missing $stale_bitmap &&
+		test_path_is_missing $stale_rev
+	)
+'
+
+test_expect_success 'pack.preferBitmapTips' '
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit_bulk --message="%s" 103 &&
+
+		git log --format="%H" >commits.raw &&
+		sort <commits.raw >commits &&
+
+		git log --format="create refs/tags/%s %H" HEAD >refs &&
+		git update-ref --stdin <refs &&
+
+		git multi-pack-index write --bitmap &&
+		test_path_is_file $midx &&
+		test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
+		test_path_is_file $midx-$(midx_checksum $objdir).rev &&
+
+		test-tool bitmap list-commits | sort >bitmaps &&
+		comm -13 bitmaps commits >before &&
+		test_line_count = 1 before &&
+
+		perl -ne "printf(\"create refs/tags/include/%d \", $.); print" \
+			<before | git update-ref --stdin &&
+
+		rm -fr $midx-$(midx_checksum $objdir).bitmap &&
+		rm -fr $midx-$(midx_checksum $objdir).rev &&
+		rm -fr $midx &&
+
+		git -c pack.preferBitmapTips=refs/tags/include \
+			multi-pack-index write --bitmap &&
+		test-tool bitmap list-commits | sort >bitmaps &&
+		comm -13 bitmaps commits >after &&
+
+		! test_cmp before after
+	)
+'
+
+test_done
Signature · 1 line
2.33.0.96.g73915697e6
#257 lore raw

PATCH v5 21/27 t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

In reply to Taylor Blau

From: Jeff King <[email protected]>

Generating a MIDX bitmap causes tests which repack in a partial clone to fail because they are missing objects. Missing objects is an expected component of tests in t0410, so disable this knob altogether. Graceful degradation when writing a bitmap with missing objects is tested in t5326.

---
 t/t0410-partial-clone.sh | 3 +++
 1 file changed, 3 insertions(+)
t/t0410-partial-clone.sh +3 -0
index bbcc51ee8e..bba679685f 100755
@@ -4,6 +4,9 @@ test_description='partial clone'
 
 . ./test-lib.sh
 
+# missing promisor objects cause repacks which write bitmaps to fail
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
 delete_object () {
 	rm $1/.git/objects/$(echo $2 | sed -e 's|^..|&/|')
 }
Signature · 1 line
2.33.0.96.g73915697e6
#258 lore raw

PATCH v5 22/27 t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP

In reply to Taylor Blau

From: Jeff King <[email protected]>

Generating a MIDX bitmap confuses many of the tests in t5310, which expect to control whether and how bitmaps are written. Since the relevant MIDX-bitmap tests here are covered already in t5326, let's just disable the flag for the whole t5310 script.

---
 t/t5310-pack-bitmaps.sh | 4 ++++
 1 file changed, 4 insertions(+)
t/t5310-pack-bitmaps.sh +4 -0
index 4318f84d53..673baa5c3c 100755
@@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 . "$TEST_DIRECTORY"/lib-bundle.sh
 . "$TEST_DIRECTORY"/lib-bitmap.sh
 
+# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in
+# their place.
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
+
 objpath () {
 	echo ".git/objects/$(echo "$1" | sed -e 's|\(..\)|\1/|')"
 }
Signature · 1 line
2.33.0.96.g73915697e6
#259 lore raw

PATCH v5 23/27 t5319: don't write MIDX bitmaps in t5319

In reply to Taylor Blau

This test is specifically about generating a midx still respecting a pack-based bitmap file. Generating a MIDX bitmap would confuse the test. Let's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to make sure we don't do so.

---
 t/t5319-multi-pack-index.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
t/t5319-multi-pack-index.sh +2 -1
index d7e4988f2b..b3f9f3969d 100755
@@ -532,7 +532,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '
 compare_results_with_midx "after repack"
 
 test_expect_success 'multi-pack-index and pack-bitmap' '
-	git -c repack.writeBitmaps=true repack -ad &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writeBitmaps=true repack -ad &&
 	git multi-pack-index write &&
 	git rev-list --test-bitmap HEAD
 '
Signature · 1 line
2.33.0.96.g73915697e6
#260 lore raw

PATCH v5 24/27 t7700: update to work with MIDX bitmap test knob

In reply to Taylor Blau

A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary.

---
 t/t7700-repack.sh | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
t/t7700-repack.sh +12 -6
index 25b235c063..98eda3bfeb 100755
@@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' '
 
 test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '
 	# build on $oid, $packid, and .keep state from previous
-	git repack -Adbl &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&
 	test_has_duplicate_object true
 '
 
 test_expect_success 'writing bitmaps via config can duplicate .keep objects' '
 	# build on $oid, $packid, and .keep state from previous
-	git -c repack.writebitmaps=true repack -Adl &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writebitmaps=true repack -Adl &&
 	test_has_duplicate_object true
 '
 
@@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' '
 
 test_expect_success 'bitmaps are created by default in bare repos' '
 	git clone --bare .git bare.git &&
-	git -C bare.git repack -ad &&
+	rm -f bare.git/objects/pack/*.bitmap &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad &&
 	bitmap=$(ls bare.git/objects/pack/*.bitmap) &&
 	test_path_is_file "$bitmap"
 '
@@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' '
 '
 
 test_expect_success 'bitmaps can be disabled on bare repos' '
-	git -c repack.writeBitmaps=false -C bare.git repack -ad &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -c repack.writeBitmaps=false -C bare.git repack -ad &&
 	bitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&
 	test -z "$bitmap"
 '
@@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' '
 	keep=${pack%.pack}.keep &&
 	test_when_finished "rm -f \"\$keep\"" &&
 	>"$keep" &&
-	git -C bare.git repack -ad 2>stderr &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad 2>stderr &&
 	test_must_be_empty stderr &&
 	find bare.git/objects/pack/ -type f -name "*.bitmap" >actual &&
 	test_must_be_empty actual
@@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '
 	blob=$(test-tool genrandom big $((1024*1024)) |
 	       git -C bare.git hash-object -w --stdin) &&
 	git -C bare.git update-ref refs/tags/big $blob &&
-	git -C bare.git repack -ad 2>stderr &&
+	GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
+		git -C bare.git repack -ad 2>stderr &&
 	test_must_be_empty stderr &&
 	find bare.git/objects/pack -type f -name "*.bitmap" >actual &&
 	test_must_be_empty actual
Signature · 1 line
2.33.0.96.g73915697e6
#261 lore raw

PATCH v5 25/27 midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP'

In reply to Taylor Blau

Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable to also write a multi-pack bitmap when 'GIT_TEST_MULTI_PACK_INDEX' is set.

---
 builtin/repack.c          | 12 ++++++++++--
 ci/run-build-and-tests.sh |  1 +
 midx.h                    |  2 ++
 t/README                  |  4 ++++
 4 files changed, 17 insertions(+), 2 deletions(-)
builtin/repack.c +10 -2
index 5f9bc74adc..82ab668272 100644
@@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		if (!(pack_everything & ALL_INTO_ONE) ||
 		    !is_bare_repository())
 			write_bitmaps = 0;
+	} else if (write_bitmaps &&
+		   git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&
+		   git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {
+		write_bitmaps = 0;
 	}
 	if (pack_kept_objects < 0)
 		pack_kept_objects = write_bitmaps > 0;
@@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		update_server_info(0);
 	remove_temporary_files();
 
-	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
-		write_midx_file(get_object_directory(), NULL, 0);
+	if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {
+		unsigned flags = 0;
+		if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))
+			flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;
+		write_midx_file(get_object_directory(), NULL, flags);
+	}
 
 	string_list_clear(&names, 0);
 	string_list_clear(&rollback, 0);
ci/run-build-and-tests.sh +1 -0
index 3ce81ffee9..7ee9ba9325 100755
@@ -23,6 +23,7 @@ linux-gcc)
 	export GIT_TEST_COMMIT_GRAPH=1
 	export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1
 	export GIT_TEST_MULTI_PACK_INDEX=1
+	export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1
 	export GIT_TEST_ADD_I_USE_BUILTIN=1
 	export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master
 	export GIT_TEST_WRITE_REV_INDEX=1
midx.h +2 -0
index 350f4d0a7b..aa3da557bb 100644
@@ -8,6 +8,8 @@ struct pack_entry;
 struct repository;
 
 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX"
+#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \
+	"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP"
 
 struct multi_pack_index {
 	struct multi_pack_index *next;
t/README +4 -0
index 9e70122302..12014aa988 100644
@@ -425,6 +425,10 @@ GIT_TEST_MULTI_PACK_INDEX=<boolean>, when true, forces the multi-pack-
 index to be written after every 'git repack' command, and overrides the
 'core.multiPackIndex' setting to true.
 
+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=<boolean>, when true, sets the
+'--bitmap' option on all invocations of 'git multi-pack-index write',
+and ignores pack-objects' '--write-bitmap-index'.
+
 GIT_TEST_SIDEBAND_ALL=<boolean>, when true, overrides the
 'uploadpack.allowSidebandAll' setting to true, and when false, forces
 fetch-pack to not request sideband-all (even if the server advertises
Signature · 1 line
2.33.0.96.g73915697e6
#262 lore raw

PATCH v5 26/27 p5310: extract full and partial bitmap tests

In reply to Taylor Blau

A new p5326 introduced by the next patch will want these same tests, interjecting its own setup in between. Move them out so that both perf tests can reuse them.

---
 t/perf/lib-bitmap.sh         | 69 ++++++++++++++++++++++++++++++++++++
 t/perf/p5310-pack-bitmaps.sh | 65 ++-------------------------------
 2 files changed, 72 insertions(+), 62 deletions(-)
 create mode 100644 t/perf/lib-bitmap.sh
t/perf/lib-bitmap.sh new file +69 -0
new file mode 100644
index 0000000000..63d3bc7cec
@@ -0,0 +1,69 @@
+# Helper functions for testing bitmap performance; see p5310.
+
+test_full_bitmap () {
+	test_perf 'simulated clone' '
+		git pack-objects --stdout --all </dev/null >/dev/null
+	'
+
+	test_perf 'simulated fetch' '
+		have=$(git rev-list HEAD~100 -1) &&
+		{
+			echo HEAD &&
+			echo ^$have
+		} | git pack-objects --revs --stdout >/dev/null
+	'
+
+	test_perf 'pack to file (bitmap)' '
+		git pack-objects --use-bitmap-index --all pack1b </dev/null >/dev/null
+	'
+
+	test_perf 'rev-list (commits)' '
+		git rev-list --all --use-bitmap-index >/dev/null
+	'
+
+	test_perf 'rev-list (objects)' '
+		git rev-list --all --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list with tag negated via --not --all (objects)' '
+		git rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list with negative tag (objects)' '
+		git rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null
+	'
+
+	test_perf 'rev-list count with blob:none' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=blob:none >/dev/null
+	'
+
+	test_perf 'rev-list count with blob:limit=1k' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=blob:limit=1k >/dev/null
+	'
+
+	test_perf 'rev-list count with tree:0' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=tree:0 >/dev/null
+	'
+
+	test_perf 'simulated partial clone' '
+		git pack-objects --stdout --all --filter=blob:none </dev/null >/dev/null
+	'
+}
+
+test_partial_bitmap () {
+	test_perf 'clone (partial bitmap)' '
+		git pack-objects --stdout --all </dev/null >/dev/null
+	'
+
+	test_perf 'pack to file (partial bitmap)' '
+		git pack-objects --use-bitmap-index --all pack2b </dev/null >/dev/null
+	'
+
+	test_perf 'rev-list with tree filter (partial bitmap)' '
+		git rev-list --use-bitmap-index --count --objects --all \
+			--filter=tree:0 >/dev/null
+	'
+}
t/perf/p5310-pack-bitmaps.sh +3 -62

82 lines not on this page · raw message

Signature · 1 line
2.33.0.96.g73915697e6
#263 lore raw

PATCH v5 27/27 p5326: perf tests for MIDX bitmaps

In reply to Taylor Blau

These new performance tests demonstrate effectively the same behavior as p5310, but use a multi-pack bitmap instead of a single-pack one.

Notably, p5326 does not create a MIDX bitmap with multiple packs. This is so we can measure a direct comparison between it and p5310. Any difference between the two is measuring just the overhead of using MIDX bitmaps.

Here are the results of p5310 and p5326 together, measured at the same time and on the same machine (using a Xenon W-2255 CPU):

    Test                                                  HEAD
    ------------------------------------------------------------------------
    5310.2: repack to disk                                96.78(93.39+11.33)
    5310.3: simulated clone                               9.98(9.79+0.19)
    5310.4: simulated fetch                               1.75(4.26+0.19)
    5310.5: pack to file (bitmap)                         28.20(27.87+8.70)
    5310.6: rev-list (commits)                            0.41(0.36+0.05)
    5310.7: rev-list (objects)                            1.61(1.54+0.07)
    5310.8: rev-list count with blob:none                 0.25(0.21+0.04)
    5310.9: rev-list count with blob:limit=1k             2.65(2.54+0.10)
    5310.10: rev-list count with tree:0                   0.23(0.19+0.04)
    5310.11: simulated partial clone                      4.34(4.21+0.12)
    5310.13: clone (partial bitmap)                       11.05(12.21+0.48)
    5310.14: pack to file (partial bitmap)                31.25(34.22+3.70)
    5310.15: rev-list with tree filter (partial bitmap)   0.26(0.22+0.04)

versus the same tests (this time using a multi-pack index):

    Test                                                  HEAD
    ------------------------------------------------------------------------
    5326.2: setup multi-pack index                        78.99(75.29+11.58)
    5326.3: simulated clone                               11.78(11.56+0.22)
    5326.4: simulated fetch                               1.70(4.49+0.13)
    5326.5: pack to file (bitmap)                         28.02(27.72+8.76)
    5326.6: rev-list (commits)                            0.42(0.36+0.06)
    5326.7: rev-list (objects)                            1.65(1.58+0.06)
    5326.8: rev-list count with blob:none                 0.26(0.21+0.05)
    5326.9: rev-list count with blob:limit=1k             2.97(2.86+0.10)
    5326.10: rev-list count with tree:0                   0.25(0.20+0.04)
    5326.11: simulated partial clone                      5.65(5.49+0.16)
    5326.13: clone (partial bitmap)                       12.22(13.43+0.38)
    5326.14: pack to file (partial bitmap)                30.05(31.57+7.25)
    5326.15: rev-list with tree filter (partial bitmap)   0.24(0.20+0.04)

There is slight overhead in "simulated clone", "simulated partial clone", and "clone (partial bitmap)". Unsurprisingly, that overhead is due to using the MIDX's reverse index to map between bit positions and MIDX positions.

This can be reproduced by running "git repack -adb" along with "git multi-pack-index write --bitmap" in a large-ish repository. Then run:

    $ perf record -o pack.perf git -c core.multiPackIndex=false \
      pack-objects --all --stdout >/dev/null </dev/null
    $ perf record -o midx.perf git -c core.multiPackIndex=true \
      pack-objects --all --stdout >/dev/null </dev/null

and compare the two with "perf diff -c delta -o 1 pack.perf midx.perf". The most notable results are below (the next largest positive delta is +0.14%):

# Event 'cycles'
#
# Baseline Delta Shared Object Symbol
# ........ ....... .................. .......................... #
+5.86% git [.] nth_midxed_offset
+5.24% git [.] nth_midxed_pack_int_id
3.45% +0.97% git [.] offset_to_pack_pos
3.30% +0.57% git [.] pack_pos_to_offset
+0.30% git [.] pack_pos_to_midx

---
 t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh
t/perf/p5326-multi-pack-bitmaps.sh new file +43 -0
new file mode 100755
index 0000000000..5845109ac7
@@ -0,0 +1,43 @@
+#!/bin/sh
+
+test_description='Tests performance using midx bitmaps'
+. ./perf-lib.sh
+. "${TEST_DIRECTORY}/perf/lib-bitmap.sh"
+
+test_perf_large_repo
+
+test_expect_success 'enable multi-pack index' '
+	git config core.multiPackIndex true
+'
+
+test_perf 'setup multi-pack index' '
+	git repack -ad &&
+	git multi-pack-index write --bitmap
+'
+
+test_full_bitmap
+
+test_expect_success 'create partial bitmap state' '
+	# pick a commit to represent the repo tip in the past
+	cutoff=$(git rev-list HEAD~100 -1) &&
+	orig_tip=$(git rev-parse HEAD) &&
+
+	# now pretend we have just one tip
+	rm -rf .git/logs .git/refs/* .git/packed-refs &&
+	git update-ref HEAD $cutoff &&
+
+	# and then repack, which will leave us with a nice
+	# big bitmap pack of the "old" history, and all of
+	# the new history will be loose, as if it had been pushed
+	# up incrementally and exploded via unpack-objects
+	git repack -Ad &&
+	git multi-pack-index write --bitmap &&
+
+	# and now restore our original tip, as if the pushes
+	# had happened
+	git update-ref HEAD $orig_tip
+'
+
+test_partial_bitmap
+
+test_done
Signature · 1 line
2.33.0.96.g73915697e6
#264 lore raw
In reply to Taylor Blau
Quoted 21 lines · Taylor Blau

Here is another version of the multi-pack reachability bitmaps series. It is virtually unchanged since last time.

The changes that did occur is that I integrated Johannes' patch from [1] to fix cleaning up MIDX .rev and .bitmap files when using --object-dir. That inspired a lengthy discussion [2] about --object-dir, alternates, object-format and running the MIDX builtin outside of a Git repository.

This series resolves that discussion by leaving everything as-is, and only changing the following:

- git multi-pack-index will not run when outside of a Git repository.

- The --object-dir argument will only recognize object directories belonging to an alternate of the current repository.

- Using --object-dir to point to a repository which uses a different hash than the repository in the current working directory will continue to not work (as was the case before this series).

Quoted 6 lines

And because this incorporates [1], we will also not accidentally clean .rev files from the wrong object directory.

I think that this version is ready-to-go, and that we can turn our attention to squashing some of these cross-alternate buglets, and integrating MIDX bitmaps with git repack.

Thanks.

Quoted 6 lines

+@@ Documentation/git-multi-pack-index.txt: OPTIONS
+ Use given directory for the location of Git objects. We check
+ <dir>/packs/multi-pack-index for the current MIDX file, and
+ <dir>/packs for the pack-files to index.
+++
++<dir> must be an alternate of the current repository.

After replacing the previous round with this round and running "git diff @{1}" on the branch, I noticed this documentation update, but did't find any new code that tries to ensure that the requirement is met. It's a bit curious omission.

I think it is OK to allow running this command on <dir> and then add it as a new alternate (iow, the <dir> being an alternate is not a strict requirement for correct computation and writing of the midx, even though it may be a requirement for correct use of the resulting midx), so perhaps that is where the lack of validation comes from?

THanks.

#265 lore raw
Quoted 13 lines · Junio C Hamano
Quoted 6 lines · Taylor Blau

+@@ Documentation/git-multi-pack-index.txt: OPTIONS
+ Use given directory for the location of Git objects. We check
+ <dir>/packs/multi-pack-index for the current MIDX file, and
+ <dir>/packs for the pack-files to index.
+++
++<dir> must be an alternate of the current repository.

After replacing the previous round with this round and running "git diff @{1}" on the branch, I noticed this documentation update, but did't find any new code that tries to ensure that the requirement is met. It's a bit curious omission.

Quoted 5 lines

I think it is OK to allow running this command on <dir> and then add it as a new alternate (iow, the <dir> being an alternate is not a strict requirement for correct computation and writing of the midx, even though it may be a requirement for correct use of the resulting midx), so perhaps that is where the lack of validation comes from?

I wasn't sure whether to include it or not, since we technically will still write a MIDX in that object directory (alternate or not), but we won't load up an existing MIDX that is already there to reference. So we'll get the same result, just slower.

I'm comfortable with saying what's written in the documentation, since even though it happens to work today, we should leave ourselves open to not supporting directories which aren't alternates.

But I'm equally OK if you would rather drop this hunk from the documentation when staging.

Thanks,
Taylor

#266 lore raw
Quoted 6 lines · Taylor Blau

I'm comfortable with saying what's written in the documentation, since even though it happens to work today, we should leave ourselves open to not supporting directories which aren't alternates.

But I'm equally OK if you would rather drop this hunk from the documentation when staging.

Oh, no, don't get me wrong. I am comfortable with the documented limitation, as that is what the area experts have agreed that is reasonable given the expected use case.

I however am much less comfortable with a documented limitation that we make no attempt to enforce, and that is why the first thing I looked for after seeing the documentation update was new code to make sure we reject a random directory that is not our alternate object store.

Thanks.

#267 lore raw
Quoted 13 lines · Junio C Hamano
Quoted 6 lines · Taylor Blau

I'm comfortable with saying what's written in the documentation, since even though it happens to work today, we should leave ourselves open to not supporting directories which aren't alternates.

But I'm equally OK if you would rather drop this hunk from the documentation when staging.

Oh, no, don't get me wrong. I am comfortable with the documented limitation, as that is what the area experts have agreed that is reasonable given the expected use case.

Quoted 5 lines

I however am much less comfortable with a documented limitation that we make no attempt to enforce, and that is why the first thing I looked for after seeing the documentation update was new code to make sure we reject a random directory that is not our alternate object store.

Sure, I don't mind getting more strict here in this series. If you want, the below could be queued instead of the original 11/27:

--- 8< ---

Subject: [PATCH] midx: avoid opening multiple MIDXs when writing

Opening multiple instance of the same MIDX can lead to problems like two separate packed_git structures which represent the same pack being added to the repository's object store.

The above scenario can happen because prepare_midx_pack() checks if m->packs[pack_int_id] is NULL in order to determine if a pack has been opened and installed in the repository before. But a caller can construct two copies of the same MIDX by calling get_multi_pack_index() and load_multi_pack_index() since the former manipulates the object store directly but the latter is a lower-level routine which allocates a new MIDX for each call.

So if prepare_midx_pack() is called on multiple MIDXs with the same pack_int_id, then that pack will be installed twice in the object store's packed_git pointer.

This can lead to problems in, for e.g., the pack-bitmap code, which does something like the following (in pack-bitmap.c:open_pack_bitmap()):

    struct bitmap_index *bitmap_git = ...;
    for (p = get_all_packs(r); p; p = p->next) {
      if (open_pack_bitmap_1(bitmap_git, p) == 0)
        ret = 0;
    }

which is a problem if two copies of the same pack exist in the packed_git list because pack-bitmap.c:open_pack_bitmap_1() contains a conditional like the following:

    if (bitmap_git->pack || bitmap_git->midx) {
      /* ignore extra bitmap file; we can only handle one */
      warning("ignoring extra bitmap file: %s", packfile->pack_name);
      close(fd);
      return -1;
    }

Avoid this scenario by not letting write_midx_internal() open a MIDX that isn't also pointed at by the object store. So long as this is the case, other routines should prefer to open MIDXs with get_multi_pack_index() or reprepare_packed_git() instead of creating instances on their own. Because get_multi_pack_index() returns r->object_store->multi_pack_index if it is non-NULL, we'll only have one instance of a MIDX open at one time, avoiding these problems.

To encourage this, drop the struct multi_pack_index * parameter from write_midx_internal(), and rely instead on the object_dir to find (or initialize) the correct MIDX instance.

Likewise, replace the call to close_midx() with close_object_store(), since we're about to replace the MIDX with a new one and should invalidate the object store's memory of any MIDX that might have existed beforehand.

Note that this now forbids passing object directories that don't belong to alternate repositories over --object-dir, since before we would have happily opened a MIDX in any directory, but now restrict ourselves to only those reachable by r->objects->multi_pack_index (and alternate MIDXs that we can see by walking the next pointer).

As far as I can tell, supporting arbitrary directories with --object-dir was a historical accident, since even the documentation says <alt> when referring to the value passed to this option.

---
 Documentation/git-multi-pack-index.txt |  2 ++
 builtin/commit-graph.c                 | 22 -------------------
 midx.c                                 | 29 ++++++++++++++++----------
 object-file.c                          | 21 +++++++++++++++++++
 object-store.h                         |  1 +
 t/t5319-multi-pack-index.sh            | 10 ++++++++-
 6 files changed, 51 insertions(+), 34 deletions(-)
Documentation/git-multi-pack-index.txt +2 -0
index c9b063d31e..0af6beb2dd 100644
@@ -23,6 +23,8 @@ OPTIONS
 	Use given directory for the location of Git objects. We check
 	`<dir>/packs/multi-pack-index` for the current MIDX file, and
 	`<dir>/packs` for the pack-files to index.
++
+`<dir>` must be an alternate of the current repository.
 
 --[no-]progress::
 	Turn progress on/off explicitly. If neither is specified, progress is
builtin/commit-graph.c +0 -22
index cd86315221..003eaaac5c 100644
@@ -43,28 +43,6 @@ static struct opts_commit_graph {
 	int enable_changed_paths;
 } opts;
 
-static struct object_directory *find_odb(struct repository *r,
-					 const char *obj_dir)
-{
-	struct object_directory *odb;
-	char *obj_dir_real = real_pathdup(obj_dir, 1);
-	struct strbuf odb_path_real = STRBUF_INIT;
-
-	prepare_alt_odb(r);
-	for (odb = r->objects->odb; odb; odb = odb->next) {
-		strbuf_realpath(&odb_path_real, odb->path, 1);
-		if (!strcmp(obj_dir_real, odb_path_real.buf))
-			break;
-	}
-
-	free(obj_dir_real);
-	strbuf_release(&odb_path_real);
-
-	if (!odb)
-		die(_("could not find object directory matching %s"), obj_dir);
-	return odb;
-}
-
 static int graph_verify(int argc, const char **argv)
 {
 	struct commit_graph *graph = NULL;
midx.c +18 -11
index e83f22b5ee..25906044ff 100644
@@ -893,7 +893,7 @@ static int midx_checksum_valid(struct multi_pack_index *m)
 	return hashfile_checksum_valid(m->data, m->data_len);
 }
 
-static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
+static int write_midx_internal(const char *object_dir,
 			       struct string_list *packs_to_drop,
 			       const char *preferred_pack_name,
 			       unsigned flags)
@@ -904,20 +904,26 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	struct hashfile *f = NULL;
 	struct lock_file lk;
 	struct write_midx_context ctx = { 0 };
+	struct multi_pack_index *cur;
 	int pack_name_concat_len = 0;
 	int dropped_packs = 0;
 	int result = 0;
 	struct chunkfile *cf;
 
+	/* Ensure the given object_dir is local, or a known alternate. */
+	find_odb(the_repository, object_dir);
+
 	midx_name = get_midx_filename(object_dir);
 	if (safe_create_leading_directories(midx_name))
 		die_errno(_("unable to create leading directories of %s"),
 			  midx_name);
 
-	if (m)
-		ctx.m = m;
-	else
-		ctx.m = load_multi_pack_index(object_dir, 1);
+	for (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {
+		if (!strcmp(object_dir, cur->object_dir)) {
+			ctx.m = cur;
+			break;
+		}
+	}
 
 	if (ctx.m && !midx_checksum_valid(ctx.m)) {
 		warning(_("ignoring existing multi-pack-index; checksum mismatch"));
@@ -1119,7 +1125,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 	f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
 
 	if (ctx.m)
-		close_midx(ctx.m);
+		close_object_store(the_repository->objects);
 
 	if (ctx.nr - dropped_packs == 0) {
 		error(_("no pack files to index."));
@@ -1182,8 +1188,7 @@ int write_midx_file(const char *object_dir,
 		    const char *preferred_pack_name,
 		    unsigned flags)
 {
-	return write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,
-				   flags);
+	return write_midx_internal(object_dir, NULL, preferred_pack_name, flags);
 }
 
 struct clear_midx_data {
@@ -1461,8 +1466,10 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla
 
 	free(count);
 
-	if (packs_to_drop.nr)
-		result = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);
+	if (packs_to_drop.nr) {
+		result = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);
+		m = NULL;
+	}
 
 	string_list_clear(&packs_to_drop, 0);
 	return result;
@@ -1651,7 +1658,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,
 		goto cleanup;
 	}
 
-	result = write_midx_internal(object_dir, m, NULL, NULL, flags);
+	result = write_midx_internal(object_dir, NULL, NULL, flags);
 	m = NULL;
 
 cleanup:
object-file.c +21 -0
index a8be899481..a4d720b4f5 100644
@@ -820,6 +820,27 @@ char *compute_alternate_path(const char *path, struct strbuf *err)
 	return ref_git;
 }
 
+struct object_directory *find_odb(struct repository *r, const char *obj_dir)
+{
+	struct object_directory *odb;
+	char *obj_dir_real = real_pathdup(obj_dir, 1);
+	struct strbuf odb_path_real = STRBUF_INIT;
+
+	prepare_alt_odb(r);
+	for (odb = r->objects->odb; odb; odb = odb->next) {
+		strbuf_realpath(&odb_path_real, odb->path, 1);
+		if (!strcmp(obj_dir_real, odb_path_real.buf))
+			break;
+	}
+
+	free(obj_dir_real);
+	strbuf_release(&odb_path_real);
+
+	if (!odb)
+		die(_("could not find object directory matching %s"), obj_dir);
+	return odb;
+}
+
 static void fill_alternate_refs_command(struct child_process *cmd,
 					const char *repo_path)
 {
object-store.h +1 -0
index d24915ced1..250aa5f33c 100644
@@ -38,6 +38,7 @@ KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 
 void prepare_alt_odb(struct repository *r);
 char *compute_alternate_path(const char *path, struct strbuf *err);
+struct object_directory *find_odb(struct repository *r, const char *obj_dir);
 typedef int alt_odb_fn(struct object_directory *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
 typedef void alternate_ref_fn(const struct object_id *oid, void *);
t/t5319-multi-pack-index.sh +9 -1
index d7e4988f2b..bd09c3194b 100755
@@ -582,7 +582,15 @@ test_expect_success 'force some 64-bit offsets with pack-objects' '
 	idx64=objects64/pack/test-64-$pack64.idx &&
 	chmod u+w $idx64 &&
 	corrupt_data $idx64 $(test_oid idxoff) "\02" &&
-	midx64=$(git multi-pack-index --object-dir=objects64 write) &&
+	# objects64 is not a real repository, but can serve as an alternate
+	# anyway so we can write a MIDX into it
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+		( cd ../objects64 && pwd ) >.git/objects/info/alternates &&
+		midx64=$(git multi-pack-index --object-dir=../objects64 write)
+	) &&
 	midx_read_expect 1 63 5 objects64 " large-offsets"
 '
 

--
2.33.0.96.g73915697e6

#268 lore raw
Quoted 2 lines · Taylor Blau

Sure, I don't mind getting more strict here in this series. If you want, the below could be queued instead of the original 11/27:

That may make the documentation and the code more consistent.

Quoted 3 lines

As far as I can tell, supporting arbitrary directories with --object-dir was a historical accident, since even the documentation says <alt> when referring to the value passed to this option.

The synopsis has [--object-dir=<dir>], which wants to be cleaned up for consistency (or <alt> updated to <dir>, but I tend to agree with you that unifying to <alt> may make our intention more clear).

It is unfortunate that "git multi-pack-index -h" says <file>, which is probably doubly wrong. It seems this is the only instance that abuses OPT_FILENAME() for a non-file, so perhaps it is not too bad to fix it using the lower-level OPTION_FILENAME (instead of adding a one-off OPT_DIRECTORY_NAME() helper).

Neither is something that would block this step, of course.

#269 lore raw
Quoted 15 lines · Junio C Hamano
Quoted 2 lines · Taylor Blau

Sure, I don't mind getting more strict here in this series. If you want, the below could be queued instead of the original 11/27:

That may make the documentation and the code more consistent.

Quoted 3 lines

As far as I can tell, supporting arbitrary directories with --object-dir was a historical accident, since even the documentation says <alt> when referring to the value passed to this option.

The synopsis has [--object-dir=<dir>], which wants to be cleaned up for consistency (or <alt> updated to <dir>, but I tend to agree with you that unifying to <alt> may make our intention more clear).

Quoted 7 lines

It is unfortunate that "git multi-pack-index -h" says <file>, which is probably doubly wrong. It seems this is the only instance that abuses OPT_FILENAME() for a non-file, so perhaps it is not too bad to fix it using the lower-level OPTION_FILENAME (instead of adding a one-off OPT_DIRECTORY_NAME() helper).

Neither is something that would block this step, of course.

I think there is definitely plenty of opportunity to clean all of this up even more. But I don't think this already-long series is the place to do it necessarily, since we don't want to let these last-minute (mostly) cosmetic issues get in the way of this series as a whole.

Hopefully this v5 is at a point where we could start merging it down to 'next' and then address things like the helptext, s/dir/alt and so on.

Thanks,
Taylor

#270 lore raw
In reply to Junio C Hamano
Quoted 8 lines · Junio C Hamano
Quoted 3 lines

As far as I can tell, supporting arbitrary directories with --object-dir was a historical accident, since even the documentation says <alt> when referring to the value passed to this option.

The synopsis has [--object-dir=<dir>], which wants to be cleaned up for consistency (or <alt> updated to <dir>, but I tend to agree with you that unifying to <alt> may make our intention more clear).

Quoted 5 lines

It is unfortunate that "git multi-pack-index -h" says <file>, which is probably doubly wrong. It seems this is the only instance that abuses OPT_FILENAME() for a non-file, so perhaps it is not too bad to fix it using the lower-level OPTION_FILENAME (instead of adding a one-off OPT_DIRECTORY_NAME() helper).

That made me wonder what "git commit-graph -h" says. It says "<dir>" (even though it already must be an alternate), because it uses OPT_STRING().

I think using OPTION_FILENAME or similar is better there, too, though, because it reinterprets the name after the repo-setup chdir() step. But now you'd have to for an OPT_DIRNAME() helper. :)

Quoted 1 line

Neither is something that would block this step, of course.

Yeah, very much agreed that this can come later on top.

-Peff

#271 lore raw
In reply to Taylor Blau
Quoted 10 lines · Taylor Blau
Quoted 9 lines

Oh, no, don't get me wrong. I am comfortable with the documented limitation, as that is what the area experts have agreed that is reasonable given the expected use case.

I however am much less comfortable with a documented limitation that we make no attempt to enforce, and that is why the first thing I looked for after seeing the documentation update was new code to make sure we reject a random directory that is not our alternate object store.

Quoted 6 lines

Sure, I don't mind getting more strict here in this series. If you want, the below could be queued instead of the original 11/27:

--- 8< ---

Subject: [PATCH] midx: avoid opening multiple MIDXs when writing

I think this is worth doing here, as part of this series.

Two observations (neither of which would lead to changing the patch):

Documentation/git-multi-pack-index.txt OPTIONS · line 27

Quoted 10 lines
Documentation/git-multi-pack-index.txt +2 -0
index c9b063d31e..0af6beb2dd 100644
@@ -23,6 +23,8 @@ OPTIONS
 	Use given directory for the location of Git objects. We check
 	`<dir>/packs/multi-pack-index` for the current MIDX file, and
 	`<dir>/packs` for the pack-files to index.
++
+`<dir>` must be an alternate of the current repository.

I wondered if this needed to say "must be the main object directory of or an alternate of the current repository". But if you are intending to operate in the main object directory, you would simply omit --object-dir entirely. It is good that it will still work if you specified it explicitly, but I don't think we need to clutter the documentation with it.

Quoted 18 lines

index cd86315221..003eaaac5c 100644

builtin/commit-graph.c +0 -11
@@ -43,28 +43,6 @@ static struct opts_commit_graph {
 	int enable_changed_paths;
 } opts;
 
-static struct object_directory *find_odb(struct repository *r,
-					 const char *obj_dir)
-{
-	struct object_directory *odb;
-	char *obj_dir_real = real_pathdup(obj_dir, 1);
-	struct strbuf odb_path_real = STRBUF_INIT;
-
-	prepare_alt_odb(r);
-	for (odb = r->objects->odb; odb; odb = odb->next) {
-		strbuf_realpath(&odb_path_real, odb->path, 1);
-		if (!strcmp(obj_dir_real, odb_path_real.buf))

builtin/commit-graph.c static struct opts_commit_graph { · line 66

Quoted 10 lines
builtin/commit-graph.c +0 -10
-			break;
-	}
-
-	free(obj_dir_real);
-	strbuf_release(&odb_path_real);
-
-	if (!odb)
-		die(_("could not find object directory matching %s"), obj_dir);
-	return odb;
-}

Ah, right, commit-graph faces this same conundrum we've been discussing. And it behaves in the way that we concluded:

  $ git init one
  $ git commit-graph write --object-dir $PWD/one/.git/objects
  fatal: not a git repository (or any of the parent directories): .git
  $ git init two
  $ git -C two commit-graph write --object-dir $PWD/one/.git/objects
  fatal: could not find object directory matching /home/peff/tmp/one/.git/objects

That gives me more confidence in the direction we decided on.

(Apologies if this was obvious to others, but I didn't see any mention of commit-graph's similar option in the recent discussion).

-Peff

#272 lore raw
In reply to Taylor Blau
Quoted 9 lines · Taylor Blau

This series resolves that discussion by leaving everything as-is, and only changing the following:

- git multi-pack-index will not run when outside of a Git repository.

- The --object-dir argument will only recognize object directories belonging to an alternate of the current repository.

Quoted 6 lines

- Using --object-dir to point to a repository which uses a different hash than the repository in the current working directory will continue to not work (as was the case before this series).

And because this incorporates [1], we will also not accidentally clean .rev files from the wrong object directory.

Thanks. I read over the new patches, and all looks good to me (using the revised patch 11 you already sent, and which I commented on separately).

-Peff