Skip to contentSkip to conversation

cruft packs

PATCH Superseded Taylor Blau 11 participants 201 messages Nov 29, 2021 – Jun 1, 2023 Page 2 of 2

Subsystems (2)
lore
Share & tools
lore
#146 lore raw

RFC PATCH 1/2 packfile API: add and use a pack_name_to_ext() utility function

In reply to Ævar Arnfjörð Bjarmason

Add and use a pack_name_to_ext() utility function for the copy/pasted cases of creating a FOO.ext file given a string like FOO.pack.

---
 pack-bitmap.c   |  6 +-----
 pack-revindex.c |  5 +----
 packfile.c      | 14 ++++++++++----
 packfile.h      |  9 +++++++++
 4 files changed, 21 insertions(+), 13 deletions(-)
pack-bitmap.c +1 -5
index 97909d48da3..0c3770d038d 100644
@@ -302,11 +302,7 @@ char *midx_bitmap_filename(struct multi_pack_index *midx)
 
 char *pack_bitmap_filename(struct packed_git *p)
 {
-	size_t len;
-
-	if (!strip_suffix(p->pack_name, ".pack", &len))
-		BUG("pack_name does not end in .pack");
-	return xstrfmt("%.*s.bitmap", (int)len, p->pack_name);
+	return pack_name_to_ext(p->pack_name, "bitmap");
 }
 
 static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
pack-revindex.c +1 -4
index 08dc1601679..69dc5688796 100644
@@ -179,10 +179,7 @@ static int create_pack_revindex_in_memory(struct packed_git *p)
 
 static char *pack_revindex_filename(struct packed_git *p)
 {
-	size_t len;
-	if (!strip_suffix(p->pack_name, ".pack", &len))
-		BUG("pack_name does not end in .pack");
-	return xstrfmt("%.*s.rev", (int)len, p->pack_name);
+	return pack_name_to_ext(p->pack_name, "rev");
 }
 
 #define RIDX_HEADER_SIZE (12)
packfile.c +10 -4
index 835b2d27164..bd6ad441bf5 100644
@@ -191,15 +191,12 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
 int open_pack_index(struct packed_git *p)
 {
 	char *idx_name;
-	size_t len;
 	int ret;
 
 	if (p->index_data)
 		return 0;
 
-	if (!strip_suffix(p->pack_name, ".pack", &len))
-		BUG("pack_name does not end in .pack");
-	idx_name = xstrfmt("%.*s.idx", (int)len, p->pack_name);
+	idx_name = pack_name_to_ext(p->pack_name, "idx");
 	ret = check_packed_git_idx(idx_name, p);
 	free(idx_name);
 	return ret;
@@ -2266,3 +2263,12 @@ int is_promisor_object(const struct object_id *oid)
 	}
 	return oidset_contains(&promisor_objects, oid);
 }
+
+char *pack_name_to_ext(const char *pack_name, const char *ext)
+{
+	size_t len;
+
+	if (!strip_suffix(pack_name, ".pack", &len))
+		BUG("pack_name does not end in .pack");
+	return xstrfmt("%.*s.%s", (int)len, pack_name, ext);
+}
packfile.h +9 -0
index a3f6723857b..6890c57ebdb 100644
@@ -195,4 +195,13 @@ int is_promisor_object(const struct object_id *oid);
 int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
 	     size_t idx_size, struct packed_git *p);
 
+/**
+ * Given a string like "foo.pack" and "ext" returns an xstrdup()'d
+ * "foo.ext" string. Used for creating e.g. PACK.{bitmap,rev,...}
+ * filenames from PACK.pack.
+ *
+ * Will BUG() if the expected string can't be created from the
+ * "pack_name" argument.
+ */
+char *pack_name_to_ext(const char *pack_name, const char *ext);
 #endif
Signature · 1 line
2.36.1.952.g6652f7f0e6b
#147 lore raw

Re: [RFC PATCH 1/2] packfile API: add and use a pack_name_to_ext() utility function

Quoted 2 lines · Ævar Arnfjörð Bjarmason

Add and use a pack_name_to_ext() utility function for the copy/pasted cases of creating a FOO.ext file given a string like FOO.pack.

I agree that "remove .pack extension and replace it with .foo extention" is a common thing to do and it would be a welcome simplification.

But pack_name_to_ext() sounds like taking pack-123456...9876.pack and returning its extension (i.e. ".pack") that will become useful when we introduce a drastically different naming convention and start calling packfiles in newer format with different extensions like ".pac4".

I wonder if this has easier-to-understand name that would be equally (or slightly more) useful?

	char *replace_ext(const char *name, const char *src, const char *dst)
	{
		size_t len;

		if (!strip_suffix(name, src, &len))
			BUG("name '%s' does not end in suffix '%s'", name, src);
		return xstrfmt("%.*s.%s", (int)len, name, dst);
	}
Quoted 93 lines
---
 pack-bitmap.c   |  6 +-----
 pack-revindex.c |  5 +----
 packfile.c      | 14 ++++++++++----
 packfile.h      |  9 +++++++++
 4 files changed, 21 insertions(+), 13 deletions(-)
pack-bitmap.c +1 -5
index 97909d48da3..0c3770d038d 100644
@@ -302,11 +302,7 @@ char *midx_bitmap_filename(struct multi_pack_index *midx)
 
 char *pack_bitmap_filename(struct packed_git *p)
 {
-	size_t len;
-
-	if (!strip_suffix(p->pack_name, ".pack", &len))
-		BUG("pack_name does not end in .pack");
-	return xstrfmt("%.*s.bitmap", (int)len, p->pack_name);
+	return pack_name_to_ext(p->pack_name, "bitmap");
 }
 
 static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
pack-revindex.c +1 -4
index 08dc1601679..69dc5688796 100644
@@ -179,10 +179,7 @@ static int create_pack_revindex_in_memory(struct packed_git *p)
 
 static char *pack_revindex_filename(struct packed_git *p)
 {
-	size_t len;
-	if (!strip_suffix(p->pack_name, ".pack", &len))
-		BUG("pack_name does not end in .pack");
-	return xstrfmt("%.*s.rev", (int)len, p->pack_name);
+	return pack_name_to_ext(p->pack_name, "rev");
 }
 
 #define RIDX_HEADER_SIZE (12)
packfile.c +10 -4
index 835b2d27164..bd6ad441bf5 100644
@@ -191,15 +191,12 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
 int open_pack_index(struct packed_git *p)
 {
 	char *idx_name;
-	size_t len;
 	int ret;
 
 	if (p->index_data)
 		return 0;
 
-	if (!strip_suffix(p->pack_name, ".pack", &len))
-		BUG("pack_name does not end in .pack");
-	idx_name = xstrfmt("%.*s.idx", (int)len, p->pack_name);
+	idx_name = pack_name_to_ext(p->pack_name, "idx");
 	ret = check_packed_git_idx(idx_name, p);
 	free(idx_name);
 	return ret;
@@ -2266,3 +2263,12 @@ int is_promisor_object(const struct object_id *oid)
 	}
 	return oidset_contains(&promisor_objects, oid);
 }
+
+char *pack_name_to_ext(const char *pack_name, const char *ext)
+{
+	size_t len;
+
+	if (!strip_suffix(pack_name, ".pack", &len))
+		BUG("pack_name does not end in .pack");
+	return xstrfmt("%.*s.%s", (int)len, pack_name, ext);
+}
packfile.h +9 -0
index a3f6723857b..6890c57ebdb 100644
@@ -195,4 +195,13 @@ int is_promisor_object(const struct object_id *oid);
 int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
 	     size_t idx_size, struct packed_git *p);
 
+/**
+ * Given a string like "foo.pack" and "ext" returns an xstrdup()'d
+ * "foo.ext" string. Used for creating e.g. PACK.{bitmap,rev,...}
+ * filenames from PACK.pack.
+ *
+ * Will BUG() if the expected string can't be created from the
+ * "pack_name" argument.
+ */
+char *pack_name_to_ext(const char *pack_name, const char *ext);
 #endif
#148 lore raw

RFC PATCH 2/2 hash API: add and use a hash_short_id_by_algo() function

In reply to Ævar Arnfjörð Bjarmason

Add and use a hash_short_id_by_algo() function. As noted in the comment being modified here (added in [1]) the intention wasn't to have these end up in on-disk formats, but since [2], [3] and [3] that's been the case, and there's an outstanding patch to add another format that uses these[5].

So let's expose this functionality as a documented utility function, instead of copy/pasting this code in various places.

Replacing the die() in the existing functions with a BUG() might be overzelous, it's correct for the case of
e.g. write_commit_graph_file() and write_midx_header(), but we also use this for parsing on-disk files, e.g. in parse_commit_graph().

We could add a "gently" version of this, but for now I think that worrying about the distinction would be worrying too much. If we ever end up parsing such files that'll almost certainly be a bug in our own writing code, so the distinction would be rather academic, even though such files could theoretically occur without a bug of ours.

1. f50e766b7b3 (Add structure representing hash algorithm, 2017-11-12)
2. 665d70ad033 (commit-graph: use the "hash version" byte, 2020-08-17)
3. d96075428a9 (multi-pack-index: use hash version byte, 2020-08-17)
4. 8ef50d9958f (pack-write.c: prepare to write 'pack-*.rev' files, 2021-01-25)
5. https://lore.kernel.org/git/1d775f9850f00b0c3d1e9133669a6365c8d7bbba.1652915424.git.me@ttaylorr.com/

---
 commit-graph.c | 18 +++---------------
 hash.h         | 26 ++++++++++++++++++++++++--
 midx.c         | 18 +++---------------
 pack-write.c   | 12 +-----------
 4 files changed, 31 insertions(+), 43 deletions(-)
commit-graph.c +3 -15
index 06107beedcb..157de4dd717 100644
@@ -193,18 +193,6 @@ char *get_commit_graph_chain_filename(struct object_directory *odb)
 	return xstrfmt("%s/info/commit-graphs/commit-graph-chain", odb->path);
 }
 
-static uint8_t oid_version(void)
-{
-	switch (hash_algo_by_ptr(the_hash_algo)) {
-	case GIT_HASH_SHA1:
-		return 1;
-	case GIT_HASH_SHA256:
-		return 2;
-	default:
-		die(_("invalid hash version"));
-	}
-}
-
 static struct commit_graph *alloc_commit_graph(void)
 {
 	struct commit_graph *g = xcalloc(1, sizeof(*g));
@@ -365,9 +353,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,
 	}
 
 	hash_version = *(unsigned char*)(data + 5);
-	if (hash_version != oid_version()) {
+	if (hash_version != hash_short_id_by_algo()) {
 		error(_("commit-graph hash version %X does not match version %X"),
-		      hash_version, oid_version());
+		      hash_version, hash_short_id_by_algo());
 		return NULL;
 	}
 
@@ -1924,7 +1912,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
 	hashwrite_be32(f, GRAPH_SIGNATURE);
 
 	hashwrite_u8(f, GRAPH_VERSION);
-	hashwrite_u8(f, oid_version());
+	hashwrite_u8(f, hash_short_id_by_algo());
 	hashwrite_u8(f, get_num_chunks(cf));
 	hashwrite_u8(f, ctx->num_commit_graphs_after - 1);
 
hash.h +24 -2
index 5d40368f18a..31293401809 100644
@@ -80,8 +80,7 @@ static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *s
 
 /*
  * Note that these constants are suitable for indexing the hash_algos array and
- * comparing against each other, but are otherwise arbitrary, so they should not
- * be exposed to the user or serialized to disk.  To know whether a
+ * comparing against each other, but are otherwise arbitrary. To know whether a
  * git_hash_algo struct points to some usable hash function, test the format_id
  * field for being non-zero.  Use the name field for user-visible situations and
  * the format_id field for fixed-length fields on disk.
@@ -337,4 +336,27 @@ static inline void oid_set_algo(struct object_id *oid, const struct git_hash_alg
 const char *empty_tree_oid_hex(void);
 const char *empty_blob_oid_hex(void);
 
+/**
+ * Convert GIT_HASH_SHA1 to 1, GIT_HASH_SHA256 to 2 etc.
+ *
+ * It's preferable to use GIT_{SHA1,SHA256}_FORMAT_ID instead for file
+ * formats. The original intention was not to make these short
+ * constants part of any file format.
+ * 
+ * But since that ship has sailed for various on-disk formats this
+ * utility function allows us to do that consistently in one place.
+ */
+static inline int hash_short_id_by_algo(void)
+{
+	int hash_algo = hash_algo_by_ptr(the_hash_algo);
+
+	switch (hash_algo) {
+	case GIT_HASH_SHA1:
+	case GIT_HASH_SHA256:
+		return hash_algo;
+	default:
+		BUG("invalid hash version");
+	}
+}
+
 #endif
midx.c +3 -15
index 3db0e47735f..2e42afa5f00 100644
@@ -41,18 +41,6 @@
 
 #define PACK_EXPIRED UINT_MAX
 
-static uint8_t oid_version(void)
-{
-	switch (hash_algo_by_ptr(the_hash_algo)) {
-	case GIT_HASH_SHA1:
-		return 1;
-	case GIT_HASH_SHA256:
-		return 2;
-	default:
-		die(_("invalid hash version"));
-	}
-}
-
 const unsigned char *get_midx_checksum(struct multi_pack_index *m)
 {
 	return m->data + m->data_len - the_hash_algo->rawsz;
@@ -134,9 +122,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local
 		      m->version);
 
 	hash_version = m->data[MIDX_BYTE_HASH_VERSION];
-	if (hash_version != oid_version()) {
+	if (hash_version != hash_short_id_by_algo()) {
 		error(_("multi-pack-index hash version %u does not match version %u"),
-		      hash_version, oid_version());
+		      hash_version, hash_short_id_by_algo());
 		goto cleanup_fail;
 	}
 	m->hash_len = the_hash_algo->rawsz;
@@ -420,7 +408,7 @@ static size_t write_midx_header(struct hashfile *f,
 {
 	hashwrite_be32(f, MIDX_SIGNATURE);
 	hashwrite_u8(f, MIDX_VERSION);
-	hashwrite_u8(f, oid_version());
+	hashwrite_u8(f, hash_short_id_by_algo());
 	hashwrite_u8(f, num_chunks);
 	hashwrite_u8(f, 0); /* unused */
 	hashwrite_be32(f, num_packs);
pack-write.c +1 -11
index 51812cb1299..c1ce8f6df8f 100644
@@ -181,17 +181,7 @@ static int pack_order_cmp(const void *va, const void *vb, void *ctx)
 
 static void write_rev_header(struct hashfile *f)
 {
-	uint32_t oid_version;
-	switch (hash_algo_by_ptr(the_hash_algo)) {
-	case GIT_HASH_SHA1:
-		oid_version = 1;
-		break;
-	case GIT_HASH_SHA256:
-		oid_version = 2;
-		break;
-	default:
-		die("write_rev_header: unknown hash version");
-	}
+	uint32_t oid_version = hash_short_id_by_algo();
 
 	hashwrite_be32(f, RIDX_SIGNATURE);
 	hashwrite_be32(f, RIDX_VERSION);
Signature · 1 line
2.36.1.952.g6652f7f0e6b
#149 lore raw

Re: [RFC PATCH 2/2] hash API: add and use a hash_short_id_by_algo() function

Quoted 3 lines · Ævar Arnfjörð Bjarmason

Add and use a hash_short_id_by_algo() function. As noted in the comment being modified here (added in [1]) the intention wasn't to have these end up in on-disk formats, but since [2], [3] and [3]

double [3]?

Quoted 10 lines

that's been the case, and there's an outstanding patch to add another format that uses these[5].

So let's expose this functionality as a documented utility function, instead of copy/pasting this code in various places.

Replacing the die() in the existing functions with a BUG() might be overzelous, it's correct for the case of
e.g. write_commit_graph_file() and write_midx_header(), but we also use this for parsing on-disk files, e.g. in parse_commit_graph().

If we know the offending data can come from outside, not from literals in the code, then there is no "might be", such a use of BUG() is simply wrong.

Quoted 4 lines

We could add a "gently" version of this, but for now I think that worrying about the distinction would be worrying too much. If we ever end up parsing such files that'll almost certainly be a bug in our own writing code, so the distinction would be rather academic, even though

It would be a file written by an ancient buggy version of our code, or a buggy third-party reimplementation of Git. It could be that a new version of Git is using a yet-to-be-invented algorithm this version of Git does not know about.

The distinction matters in that "The version of Git I downloaded and built last week out of the latest release tag said BUG" should mean only one thing: that version that reports BUG() is the culprit, not some random other thing we do not even know where it came from that left a corrupt data on disk.

#150 lore raw

Re: [RFC PATCH 2/2] hash API: add and use a hash_short_id_by_algo() function

Quoted 7 lines · Junio C Hamano
Quoted 3 lines · Ævar Arnfjörð Bjarmason

Add and use a hash_short_id_by_algo() function. As noted in the comment being modified here (added in [1]) the intention wasn't to have these end up in on-disk formats, but since [2], [3] and [3]

double [3]?

*nod*, sorry.

Quoted 6 lines
Quoted 6 lines

that's been the case, and there's an outstanding patch to add another format that uses these[5].

So let's expose this functionality as a documented utility function, instead of copy/pasting this code in various places.

Quoted 8 lines
Quoted 4 lines

Replacing the die() in the existing functions with a BUG() might be overzelous, it's correct for the case of
e.g. write_commit_graph_file() and write_midx_header(), but we also use this for parsing on-disk files, e.g. in parse_commit_graph().

If we know the offending data can come from outside, not from literals in the code, then there is no "might be", such a use of BUG() is simply wrong.

Fair enough, we could change it to have 1/2 of this (the writing) use BUG(), and die() for the other part, or just leave it as die() for both.

Quoted 10 lines
Quoted 4 lines

We could add a "gently" version of this, but for now I think that worrying about the distinction would be worrying too much. If we ever end up parsing such files that'll almost certainly be a bug in our own writing code, so the distinction would be rather academic, even though

It would be a file written by an ancient buggy version of our code, or a buggy third-party reimplementation of Git. It could be that a new version of Git is using a yet-to-be-invented algorithm this version of Git does not know about.

Quoted 5 lines

The distinction matters in that "The version of Git I downloaded and built last week out of the latest release tag said BUG" should mean only one thing: that version that reports BUG() is the culprit, not some random other thing we do not even know where it came from that left a corrupt data on disk.

Makes sense.

#151 lore raw

Re: [RFC PATCH 0/2] Utility functions for duplicated pack(write) code

In reply to Ævar Arnfjörð Bjarmason
Quoted 4 lines · Ævar Arnfjörð Bjarmason

Minor cleanups thath would semantically & textually conflict with Taylor's
https://lore.kernel.org/git/[email protected]/; but which I noted while reading through it.

Very much appreciated that you marked this as RFC.

It is natural and easy to notice problems in the code that is in flux, because it is inevitable for anybody working on the codebase to see the changes in-flight and the original code as they review, or as they make trial merges of their own work and see conflicts.

But making patches to address them immediately out of spinal reflex would not help anybody. Marking them as RFC and calling attention by those involved in the "other topic" while the code being cleaned up is still fresh in their mind makes it efficient to review the clean-up while letting the "other topic" to either proceed without clean-up or with it rolled in.

#152 lore raw
In reply to Taylor Blau
Quoted 11 lines · Taylor Blau

Here is another reroll of my series to implement "cruft packs", which is based on the v2.36 tree, and incorporates feedback from the discussion we had about mixed-version GCs with cruft packs in [1].

The changes here are limited to:

- a cautionary note in Documentation/technical/cruft-packs.txt describing the potential interaction between pruning GCs across pre- and post-cruft pack versions of Git, as discussed towards the bottom of [2]

Quoted 7 lines

- updating the finalize_hashfile() calls for writing .mtimes files to indicate that they are FSYNC_COMPONENT_PACK_METADATA, since the original version of this series predates the fine-grained fsync configuration in 2.36.

As always, a range-diff is below. Thanks in advance for taking another look!

I left some minor & nit-y comments on this v4, but overall I think this looks really good with not much to add.

#153 lore raw

v5 cover cruft packs

In reply to Taylor Blau

Here is another reroll of my series to implement "cruft packs". This is really more like v4.1, since it has only cosmetic changes incorporated from the review on v4 of this topic.

Since last time:

- The new section in pack-format.txt (describing the ".mtimes" format) now says at the top "all 4-byte numbers are in network byte order", and avoids repeating "network [byte] order" throughout that section to reduce confusion.

- A sub-shell in t5329 which incorrectly masked over the exit code of a "git" process was removed.

- An overly-long line in builtin/repack.c::collect_pack_filenames() was eliminated, and matching braces are added.

...and that's pretty much it. In any case, a range-diff is included below. Thanks again for all of the thoughtful feedback on this series.

Taylor Blau (17):
Documentation/technical: add cruft-packs.txt
pack-mtimes: support reading .mtimes files
pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles' chunk-format.h: extract oid_version()
pack-mtimes: support writing pack .mtimes files t/helper: add 'pack-mtimes' test-tool builtin/pack-objects.c: return from create_object_entry() builtin/pack-objects.c: --cruft without expiration
reachable: add options to add_unseen_recent_objects_to_traversal
reachable: report precise timestamps from objects in cruft packs builtin/pack-objects.c: --cruft with expiration builtin/repack.c: support generating a cruft pack builtin/repack.c: allow configuring cruft pack generation builtin/repack.c: use named flags for existing_packs builtin/repack.c: add cruft packs to MIDX during geometric repack builtin/gc.c: conditionally avoid pruning objects via loose sha1-file.c: don't freshen cruft packs

 Documentation/Makefile                  |   1 +
 Documentation/config/gc.txt             |  21 +-
 Documentation/config/repack.txt         |   9 +
 Documentation/git-gc.txt                |   5 +
 Documentation/git-pack-objects.txt      |  30 +
 Documentation/git-repack.txt            |  11 +
 Documentation/technical/cruft-packs.txt | 123 ++++
 Documentation/technical/pack-format.txt |  19 +
 Makefile                                |   2 +
 builtin/gc.c                            |  10 +-
 builtin/pack-objects.c                  | 304 +++++++++-
 builtin/repack.c                        | 185 +++++-
 bulk-checkin.c                          |   2 +-
 chunk-format.c                          |  12 +
 chunk-format.h                          |   3 +
 commit-graph.c                          |  18 +-
 midx.c                                  |  18 +-
 object-file.c                           |   4 +-
 object-store.h                          |   7 +-
 pack-mtimes.c                           | 126 ++++
 pack-mtimes.h                           |  15 +
 pack-objects.c                          |   6 +
 pack-objects.h                          |  25 +
 pack-write.c                            |  93 ++-
 pack.h                                  |   4 +
 packfile.c                              |  19 +-
 reachable.c                             |  58 +-
 reachable.h                             |   9 +-
 t/helper/test-pack-mtimes.c             |  56 ++
 t/helper/test-tool.c                    |   1 +
 t/helper/test-tool.h                    |   1 +
 t/t5329-pack-objects-cruft.sh           | 739 ++++++++++++++++++++++++
 32 files changed, 1834 insertions(+), 102 deletions(-)
 create mode 100644 Documentation/technical/cruft-packs.txt
 create mode 100644 pack-mtimes.c
 create mode 100644 pack-mtimes.h
 create mode 100644 t/helper/test-pack-mtimes.c
 create mode 100755 t/t5329-pack-objects-cruft.sh

Range-diff against v4:

 -:  ---------- >  1:  f494ef7377 Documentation/technical: add cruft-packs.txt
 1:  8f9fd21be9 !  2:  91a9d21b0b pack-mtimes: support reading .mtimes files

@@ Documentation/technical/pack-format.txt: Pack file entry: <+

+== pack-*.mtimes files have the format:
+
++All 4-byte numbers are in network byte order.
++
+ - A 4-byte magic number '0x4d544d45' ('MTME').
+
+ - A 4-byte version identifier (= 1).
+
+ - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256). +
-+ - A table of 4-byte unsigned integers in network order. The ith
-+ value is the modification time (mtime) of the ith object in the
-+ corresponding pack by lexicographic (index) order. The mtimes
-+ count standard epoch seconds.
++ - A table of 4-byte unsigned integers. The ith value is the
++ modification time (mtime) of the ith object in the corresponding
++ pack by lexicographic (index) order. The mtimes count standard
++ epoch seconds.
+
+ - A trailer, containing a checksum of the corresponding packfile,
+ and a checksum of all of the above (each having length according
+ to the specified hash function).
-+
-+All 4-byte numbers are in network order.
+
== multi-pack-index (MIDX) files have the following format:

 2:  cdb21236e1 =  3:  67c4e7209d pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles'
 3:  1d775f9850 =  4:  fc86506881 chunk-format.h: extract oid_version()
 4:  6172861bd9 =  5:  788d1f96f2 pack-mtimes: support writing pack .mtimes files
 5:  5f9a9a5b7b =  6:  2a6cfb00bf t/helper: add 'pack-mtimes' test-tool
 6:  b8a38fe2e4 =  7:  edb6fcd5ec builtin/pack-objects.c: return from create_object_entry()
 7:  94fe03cc65 =  8:  e3185741f2 builtin/pack-objects.c: --cruft without expiration
 8:  da7273f41f =  9:  1cf00d462c reachable: add options to add_unseen_recent_objects_to_traversal
 9:  58fecd1747 = 10:  d66be44d9a reachable: report precise timestamps from objects in cruft packs
10:  1740b8ef01 = 11:  1434e37623 builtin/pack-objects.c: --cruft with expiration
11:  5992a72cbf ! 12:  0d3555d595 builtin/repack.c: support generating a cruft pack
    @@ t/t5329-pack-objects-cruft.sh: test_expect_success 'expired objects are pruned'
     +		git repack &&
     +
     +		tip="$(git rev-parse cruft)" &&
    -+		path="$objdir/$(test_oid_to_path "$(git rev-parse cruft)")" &&
    ++		path="$objdir/$(test_oid_to_path "$tip")" &&
     +		test-tool chmtime --get +1000 "$path" >expect &&
     +
     +		git checkout main &&
12:  1b241f8f91 = 13:  4b721d3ee9 builtin/repack.c: allow configuring cruft pack generation
13:  ffae78852c = 14:  f9e3ab56b1 builtin/repack.c: use named flags for existing_packs
14:  0743e373ba ! 15:  e9f46e7b5e builtin/repack.c: add cruft packs to MIDX during geometric repack
    @@ builtin/repack.c
      static int pack_everything;
      static int delta_base_offset = 1;
     @@ builtin/repack.c: static void collect_pack_filenames(struct string_list *fname_nonkept_list,
    + 		fname = xmemdupz(e->d_name, len);
    + 
      		if ((extra_keep->nr > 0 && i < extra_keep->nr) ||
    - 		    (file_exists(mkpath("%s/%s.keep", packdir, fname))))
    +-		    (file_exists(mkpath("%s/%s.keep", packdir, fname))))
    ++		    (file_exists(mkpath("%s/%s.keep", packdir, fname)))) {
      			string_list_append_nodup(fname_kept_list, fname);
     -		else
     -			string_list_append_nodup(fname_nonkept_list, fname);
    -+		else {
    -+			struct string_list_item *item = string_list_append_nodup(fname_nonkept_list, fname);
    ++		} else {
    ++			struct string_list_item *item;
    ++			item = string_list_append_nodup(fname_nonkept_list,
    ++							fname);
     +			if (file_exists(mkpath("%s/%s.mtimes", packdir, fname)))
     +				item->util = (void*)(uintptr_t)CRUFT_PACK;
     +		}
15:  9f7e0acac6 = 16:  43c14eec07 builtin/gc.c: conditionally avoid pruning objects via loose
16:  07fa9d4b47 = 17:  1e313b89e8 sha1-file.c: don't freshen cruft packs
Signature · 1 line
2.36.1.94.gb0d54bedca
#154 lore raw

PATCH v5 1/17 Documentation/technical: add cruft-packs.txt

Create a technical document to explain cruft packs. It contains a brief overview of the problem, some background, details on the implementation, and a couple of alternative approaches not considered here.

---
 Documentation/Makefile                  |   1 +
 Documentation/technical/cruft-packs.txt | 123 ++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
 create mode 100644 Documentation/technical/cruft-packs.txt
Documentation/Makefile +1 -0
index adb2f1b50a..2faffb52ab 100644
@@ -94,6 +94,7 @@ TECH_DOCS += MyFirstContribution
 TECH_DOCS += MyFirstObjectWalk
 TECH_DOCS += SubmittingPatches
 TECH_DOCS += technical/bundle-format
+TECH_DOCS += technical/cruft-packs
 TECH_DOCS += technical/hash-function-transition
 TECH_DOCS += technical/http-protocol
 TECH_DOCS += technical/index-format
Documentation/technical/cruft-packs.txt new file +123 -0

123 lines not on this page · raw message

Signature · 1 line
2.36.1.94.gb0d54bedca
#155 lore raw

PATCH v5 2/17 pack-mtimes: support reading .mtimes files

In reply to Taylor Blau

To store the individual mtimes of objects in a cruft pack, introduce a new .mtimes format that can optionally accompany a single pack in the repository.

The format is defined in Documentation/technical/pack-format.txt, and stores a 4-byte network order timestamp for each object in name (index) order.

This patch prepares for cruft packs by defining the .mtimes format, and introducing a basic API that callers can use to read out individual mtimes.

---
 Documentation/technical/pack-format.txt |  19 ++++
 Makefile                                |   1 +
 builtin/repack.c                        |   1 +
 object-store.h                          |   5 +-
 pack-mtimes.c                           | 126 ++++++++++++++++++++++++
 pack-mtimes.h                           |  15 +++
 packfile.c                              |  19 +++-
 7 files changed, 183 insertions(+), 3 deletions(-)
 create mode 100644 pack-mtimes.c
 create mode 100644 pack-mtimes.h
Documentation/technical/pack-format.txt +19 -0
index 6d3efb7d16..b520aa9c45 100644
@@ -294,6 +294,25 @@ Pack file entry: <+
 
 All 4-byte numbers are in network order.
 
+== pack-*.mtimes files have the format:
+
+All 4-byte numbers are in network byte order.
+
+  - A 4-byte magic number '0x4d544d45' ('MTME').
+
+  - A 4-byte version identifier (= 1).
+
+  - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).
+
+  - A table of 4-byte unsigned integers. The ith value is the
+    modification time (mtime) of the ith object in the corresponding
+    pack by lexicographic (index) order. The mtimes count standard
+    epoch seconds.
+
+  - A trailer, containing a checksum of the corresponding packfile,
+    and a checksum of all of the above (each having length according
+    to the specified hash function).
+
 == multi-pack-index (MIDX) files have the following format:
 
 The multi-pack-index files refer to multiple pack-files and loose objects.
Makefile +1 -0
index 61aadf3ce8..a299580b7c 100644
@@ -993,6 +993,7 @@ LIB_OBJS += oidtree.o
 LIB_OBJS += pack-bitmap-write.o
 LIB_OBJS += pack-bitmap.o
 LIB_OBJS += pack-check.o
+LIB_OBJS += pack-mtimes.o
 LIB_OBJS += pack-objects.o
 LIB_OBJS += pack-revindex.o
 LIB_OBJS += pack-write.o
builtin/repack.c +1 -0
index d1a563d5b6..e7a3920c6d 100644
@@ -217,6 +217,7 @@ static struct {
 } exts[] = {
 	{".pack"},
 	{".rev", 1},
+	{".mtimes", 1},
 	{".bitmap", 1},
 	{".promisor", 1},
 	{".idx"},
object-store.h +4 -1
index 53996018c1..2c4671ed7a 100644
@@ -115,12 +115,15 @@ struct packed_git {
 		 freshened:1,
 		 do_not_close:1,
 		 pack_promisor:1,
-		 multi_pack_index:1;
+		 multi_pack_index:1,
+		 is_cruft:1;
 	unsigned char hash[GIT_MAX_RAWSZ];
 	struct revindex_entry *revindex;
 	const uint32_t *revindex_data;
 	const uint32_t *revindex_map;
 	size_t revindex_size;
+	const uint32_t *mtimes_map;
+	size_t mtimes_size;
 	/* something like ".git/objects/pack/xxxxx.pack" */
 	char pack_name[FLEX_ARRAY]; /* more */
 };
pack-mtimes.c new file +126 -0

126 lines not on this page · raw message

pack-mtimes.h new file +15 -0
new file mode 100644
index 0000000000..38ddb9f893
@@ -0,0 +1,15 @@
+#ifndef PACK_MTIMES_H
+#define PACK_MTIMES_H
+
+#include "git-compat-util.h"
+
+#define MTIMES_SIGNATURE 0x4d544d45 /* "MTME" */
+#define MTIMES_VERSION 1
+
+struct packed_git;
+
+int load_pack_mtimes(struct packed_git *p);
+
+uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos);
+
+#endif
packfile.c +17 -2
index 835b2d2716..fc0245fbab 100644
@@ -334,12 +334,22 @@ static void close_pack_revindex(struct packed_git *p)
 	p->revindex_data = NULL;
 }
 
+static void close_pack_mtimes(struct packed_git *p)
+{
+	if (!p->mtimes_map)
+		return;
+
+	munmap((void *)p->mtimes_map, p->mtimes_size);
+	p->mtimes_map = NULL;
+}
+
 void close_pack(struct packed_git *p)
 {
 	close_pack_windows(p);
 	close_pack_fd(p);
 	close_pack_index(p);
 	close_pack_revindex(p);
+	close_pack_mtimes(p);
 	oidset_clear(&p->bad_objects);
 }
 
@@ -363,7 +373,7 @@ void close_object_store(struct raw_object_store *o)
 
 void unlink_pack_path(const char *pack_name, int force_delete)
 {
-	static const char *exts[] = {".pack", ".idx", ".rev", ".keep", ".bitmap", ".promisor"};
+	static const char *exts[] = {".pack", ".idx", ".rev", ".keep", ".bitmap", ".promisor", ".mtimes"};
 	int i;
 	struct strbuf buf = STRBUF_INIT;
 	size_t plen;
@@ -718,6 +728,10 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 	if (!access(p->pack_name, F_OK))
 		p->pack_promisor = 1;
 
+	xsnprintf(p->pack_name + path_len, alloc - path_len, ".mtimes");
+	if (!access(p->pack_name, F_OK))
+		p->is_cruft = 1;
+
 	xsnprintf(p->pack_name + path_len, alloc - path_len, ".pack");
 	if (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {
 		free(p);
@@ -869,7 +883,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 	    ends_with(file_name, ".pack") ||
 	    ends_with(file_name, ".bitmap") ||
 	    ends_with(file_name, ".keep") ||
-	    ends_with(file_name, ".promisor"))
+	    ends_with(file_name, ".promisor") ||
+	    ends_with(file_name, ".mtimes"))
 		string_list_append(data->garbage, full_name);
 	else
 		report_garbage(PACKDIR_FILE_GARBAGE, full_name);
Signature · 1 line
2.36.1.94.gb0d54bedca
#156 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Hi,

Quoted 3 lines · Taylor Blau

This patch prepares for cruft packs by defining the .mtimes format, and introducing a basic API that callers can use to read out individual mtimes.

Makes sense. Does this intend to produce any functional change? I'm guessing not (and the lack of tests agrees), but the commit message doesn't say so.

By the way, is this something we could cover in tests, e.g. using a test helper that exercises the new code?

[...]

Quoted 15 lines
Documentation/technical/pack-format.txt +9 -0
@@ -294,6 +294,25 @@ Pack file entry: <+
 
 All 4-byte numbers are in network order.
 
+== pack-*.mtimes files have the format:
+
+All 4-byte numbers are in network byte order.
+
+  - A 4-byte magic number '0x4d544d45' ('MTME').
+
+  - A 4-byte version identifier (= 1).
+
+  - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).

Documentation/technical/pack-format.txt Pack file entry: <+ · line 315

Quoted 10 lines
Documentation/technical/pack-format.txt +10 -0
+
+  - A table of 4-byte unsigned integers. The ith value is the
+    modification time (mtime) of the ith object in the corresponding
+    pack by lexicographic (index) order. The mtimes count standard
+    epoch seconds.
+
+  - A trailer, containing a checksum of the corresponding packfile,
+    and a checksum of all of the above (each having length according
+    to the specified hash function).
+

This describes the "syntax" but not the "semantics" of the file. Should I look to a separate piece of documentation for the semantics? If so, can this one include a mention of that piece of documentation to make it easier to find?

[...]

Quoted 6 lines
object-store.h +0 -0
@@ -115,12 +115,15 @@ struct packed_git {
 		 freshened:1,
 		 do_not_close:1,
 		 pack_promisor:1,

object-store.h struct packed_git { · line 126

Quoted 10 lines
object-store.h +4 -1
-		 multi_pack_index:1;
+		 multi_pack_index:1,
+		 is_cruft:1;
 	unsigned char hash[GIT_MAX_RAWSZ];
 	struct revindex_entry *revindex;
 	const uint32_t *revindex_data;
 	const uint32_t *revindex_map;
 	size_t revindex_size;
+	const uint32_t *mtimes_map;
+	size_t mtimes_size;

What does mtimes_map contain? A comment would help.

pack-mtimes.c line 3

Quoted 6 lines
pack-mtimes.c new file +3 -0
@@ -0,0 +1,126 @@
+#include "pack-mtimes.h"
+#include "object-store.h"
+#include "packfile.h"

Missing #include of git-compat-util.h.

pack-mtimes.c line 12

Quoted 9 lines
pack-mtimes.c new file +9 -0
+
+static char *pack_mtimes_filename(struct packed_git *p)
+{
+	size_t len;
+	if (!strip_suffix(p->pack_name, ".pack", &len))
+		BUG("pack_name does not end in .pack");
+	/* NEEDSWORK: this could reuse code from pack-revindex.c. */
+	return xstrfmt("%.*s.mtimes", (int)len, p->pack_name);
+}

This seems simple enough that it's not obvious we need more code sharing. Do you agree? If so, I'd suggest just removing the NEEDSWORK comment.

pack-mtimes.c line 15

Quoted 3 lines
pack-mtimes.c new file +3 -0
+
+#define MTIMES_HEADER_SIZE (12)
+#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 * the_hash_algo->rawsz))

Hm, the all-caps name makes this feel like a compile-time constant but it contains a reference to the_hash_algo. Could it be an inline function instead?

pack-mtimes.c line 25

Quoted 10 lines
pack-mtimes.c new file +10 -0
+
+struct mtimes_header {
+	uint32_t signature;
+	uint32_t version;
+	uint32_t hash_id;
+};
+
+static int load_pack_mtimes_file(char *mtimes_file,
+				 uint32_t num_objects,
+				 const uint32_t **data_p, size_t *len_p)

What does this function do? A comment would help.

pack-mtimes.c line 36

Quoted 11 lines
pack-mtimes.c new file +11 -0
+{
+	int fd, ret = 0;
+	struct stat st;
+	void *data = NULL;
+	size_t mtimes_size;
+	struct mtimes_header header;
+	uint32_t *hdr;
+
+	fd = git_open(mtimes_file);
+
+	if (fd < 0) {

nit: this would be more readable without the blank line between setting and checking fd (likewise for the other examples below).

Quoted 3 lines
pack-mtimes.c new file +3 -0
+		ret = -1;
+		goto cleanup;
+	}

[...]

pack-mtimes.c line 52

Quoted 1 line
pack-mtimes.c new file +1 -0
+	if (mtimes_size - MTIMES_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {

This presupposes that the hash_id matches the_hash_algo. Maybe worth a NEEDSWORK comment.

[...]

pack-mtimes.c line 86

Quoted 7 lines
pack-mtimes.c new file +7 -0
+cleanup:
+	if (ret) {
+		if (data)
+			munmap(data, mtimes_size);
+	} else {
+		*len_p = mtimes_size;
+		*data_p = (const uint32_t *)data;

Do we know that 'data' is uint32_t aligned? Casting earlier in the function could make that more obvious.

[...]

Quoted 1 line

+int load_pack_mtimes(struct packed_git *p)

This could use a doc comment in the header file. For example, what requirements do we have on what the caller passes as 'p'?

[...]

Quoted 1 line

+uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos)

Likewise.

[...]

Quoted 2 lines
packfile.c +0 -0

[...]

packfile.c void close_object_store(struct raw_object_store *o) · line 376

Quoted 6 lines
packfile.c +1 -1
@@ -363,7 +373,7 @@ void close_object_store(struct raw_object_store *o)
 
 void unlink_pack_path(const char *pack_name, int force_delete)
 {
-	static const char *exts[] = {".pack", ".idx", ".rev", ".keep", ".bitmap", ".promisor"};
+	static const char *exts[] = {".pack", ".idx", ".rev", ".keep", ".bitmap", ".promisor", ".mtimes"};

Are these in any particular order? Should they be?

[...]

Quoted 12 lines
packfile.c +4 -0
@@ -718,6 +728,10 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 	if (!access(p->pack_name, F_OK))
 		p->pack_promisor = 1;
 
+	xsnprintf(p->pack_name + path_len, alloc - path_len, ".mtimes");
+	if (!access(p->pack_name, F_OK))
+		p->is_cruft = 1;
+
 	xsnprintf(p->pack_name + path_len, alloc - path_len, ".pack");
 	if (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {
 		free(p);

packfile.c static void prepare_pack(const char *full_name, size_t full_name_len, · line 887

Quoted 6 lines
packfile.c +2 -1
@@ -869,7 +883,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 	    ends_with(file_name, ".pack") ||
 	    ends_with(file_name, ".bitmap") ||
 	    ends_with(file_name, ".keep") ||
-	    ends_with(file_name, ".promisor"))
+	    ends_with(file_name, ".promisor") ||
+	    ends_with(file_name, ".mtimes"))

likewise

Thanks,
Jonathan

#157 lore raw

RE: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Quoted 7 lines · Taylor Blau
Quoted 3 lines · Taylor Blau

This patch prepares for cruft packs by defining the .mtimes format, and introducing a basic API that callers can use to read out individual mtimes.

Makes sense. Does this intend to produce any functional change? I'm

guessing

Quoted 3 lines

not (and the lack of tests agrees), but the commit message doesn't say so.

By the way, is this something we could cover in tests, e.g. using a test

helper that

Quoted 24 lines

exercises the new code?

[...]

Quoted 21 lines
Documentation/technical/pack-format.txt +15 -0
@@ -294,6 +294,25 @@ Pack file entry: <+
 
 All 4-byte numbers are in network order.
 
+== pack-*.mtimes files have the format:
+
+All 4-byte numbers are in network byte order.
+
+  - A 4-byte magic number '0x4d544d45' ('MTME').
+
+  - A 4-byte version identifier (= 1).
+
+  - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).
+
+  - A table of 4-byte unsigned integers. The ith value is the
+    modification time (mtime) of the ith object in the corresponding
+    pack by lexicographic (index) order. The mtimes count standard
+    epoch seconds.
+
Quoted 8 lines

Documentation/technical/pack-format.txt Pack file entry: <+ · line 315

Quoted 4 lines
Documentation/technical/pack-format.txt +4 -0
+  - A trailer, containing a checksum of the corresponding packfile,
+    and a checksum of all of the above (each having length according
+    to the specified hash function).
+

This describes the "syntax" but not the "semantics" of the file. Should I look to a separate piece of documentation for the semantics? If so, can this one include a mention of that piece of documentation to

make it

Quoted 32 lines

easier to find?

[...]

Quoted 6 lines
object-store.h +0 -0
@@ -115,12 +115,15 @@ struct packed_git {
 		 freshened:1,
 		 do_not_close:1,
 		 pack_promisor:1,

object-store.h struct packed_git { · line 126

Quoted 10 lines
object-store.h +4 -1
-		 multi_pack_index:1;
+		 multi_pack_index:1,
+		 is_cruft:1;
 	unsigned char hash[GIT_MAX_RAWSZ];
 	struct revindex_entry *revindex;
 	const uint32_t *revindex_data;
 	const uint32_t *revindex_map;
 	size_t revindex_size;
+	const uint32_t *mtimes_map;
+	size_t mtimes_size;

What does mtimes_map contain? A comment would help.

pack-mtimes.c line 3

Quoted 6 lines
pack-mtimes.c new file +3 -0
@@ -0,0 +1,126 @@
+#include "pack-mtimes.h"
+#include "object-store.h"
+#include "packfile.h"

Missing #include of git-compat-util.h.

Quoted 9 lines

pack-mtimes.c

Quoted 7 lines
pack-mtimes.c new file +7 -0
+
+static char *pack_mtimes_filename(struct packed_git *p) {
+	size_t len;
+	if (!strip_suffix(p->pack_name, ".pack", &len))
+		BUG("pack_name does not end in .pack");
+	/* NEEDSWORK: this could reuse code from pack-revindex.c. */
+	return xstrfmt("%.*s.mtimes", (int)len, p->pack_name); }

This seems simple enough that it's not obvious we need more code sharing.

Do

Quoted 8 lines

you agree? If so, I'd suggest just removing the NEEDSWORK comment.

pack-mtimes.c

Quoted 4 lines
pack-mtimes.c new file +4 -0
+
+#define MTIMES_HEADER_SIZE (12)
+#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 *
+the_hash_algo->rawsz))

Hm, the all-caps name makes this feel like a compile-time constant but it

contains

Quoted 41 lines

a reference to the_hash_algo. Could it be an inline function instead?

pack-mtimes.c line 25

Quoted 10 lines
pack-mtimes.c new file +10 -0
+
+struct mtimes_header {
+	uint32_t signature;
+	uint32_t version;
+	uint32_t hash_id;
+};
+
+static int load_pack_mtimes_file(char *mtimes_file,
+				 uint32_t num_objects,
+				 const uint32_t **data_p, size_t *len_p)

What does this function do? A comment would help.

pack-mtimes.c line 36

Quoted 11 lines
pack-mtimes.c new file +11 -0
+{
+	int fd, ret = 0;
+	struct stat st;
+	void *data = NULL;
+	size_t mtimes_size;
+	struct mtimes_header header;
+	uint32_t *hdr;
+
+	fd = git_open(mtimes_file);
+
+	if (fd < 0) {

nit: this would be more readable without the blank line between setting and checking fd (likewise for the other examples below).

Quoted 3 lines
pack-mtimes.c new file +3 -0
+		ret = -1;
+		goto cleanup;
+	}

[...]

pack-mtimes.c

Quoted 2 lines
pack-mtimes.c new file +2 -0
+	if (mtimes_size - MTIMES_MIN_SIZE != st_mult(sizeof(uint32_t),
+num_objects)) {

This presupposes that the hash_id matches the_hash_algo. Maybe worth a NEEDSWORK comment.

[...]

Quoted 9 lines

pack-mtimes.c line 86

Quoted 7 lines
pack-mtimes.c new file +7 -0
+cleanup:
+	if (ret) {
+		if (data)
+			munmap(data, mtimes_size);
+	} else {
+		*len_p = mtimes_size;
+		*data_p = (const uint32_t *)data;

Do we know that 'data' is uint32_t aligned? Casting earlier in the

function could

Quoted 6 lines

make that more obvious.

[...]

Quoted 1 line

+int load_pack_mtimes(struct packed_git *p)

This could use a doc comment in the header file. For example, what

requirements

Quoted 8 lines

do we have on what the caller passes as 'p'?

[...]

Quoted 1 line

+uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos)

Likewise.

[...]

Quoted 8 lines
Quoted 2 lines
packfile.c +0 -0

[...]

packfile.c void close_object_store(struct raw_object_store · line 366

Quoted 5 lines
packfile.c +0 -1
@@ -363,7 +373,7 @@ void close_object_store(struct raw_object_store
 *o)
 
 void unlink_pack_path(const char *pack_name, int force_delete)  {
-	static const char *exts[] = {".pack", ".idx", ".rev", ".keep",

".bitmap",

Quoted 12 lines

".promisor"};

packfile.c void close_object_store(struct raw_object_store · line 377

Quoted 2 lines
packfile.c +2 -0
+	static const char *exts[] = {".pack", ".idx", ".rev", ".keep",
+".bitmap", ".promisor", ".mtimes"};

Are these in any particular order? Should they be?

[...]

packfile.c struct packed_git *add_packed_git(const char *path,

Quoted 1 line
packfile.c +0 -0
@@ -718,6 +728,10 @@ struct packed_git *add_packed_git(const char *path,

size_t path_len, int local)

Quoted 3 lines
packfile.c +0 -0
 	if (!access(p->pack_name, F_OK))
 		p->pack_promisor = 1;
 
Quoted 8 lines

packfile.c static void prepare_pack(const char *full_name,

Quoted 8 lines
packfile.c +4 -0
+	xsnprintf(p->pack_name + path_len, alloc - path_len, ".mtimes");
+	if (!access(p->pack_name, F_OK))
+		p->is_cruft = 1;
+
 	xsnprintf(p->pack_name + path_len, alloc - path_len, ".pack");
 	if (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {
 		free(p);
@@ -869,7 +883,8 @@ static void prepare_pack(const char *full_name,

size_t

Quoted 9 lines

full_name_len,

packfile.c static void prepare_pack(const char *full_name, size_t full_name_len, · line 887

Quoted 6 lines
packfile.c +2 -1
 	    ends_with(file_name, ".pack") ||
 	    ends_with(file_name, ".bitmap") ||
 	    ends_with(file_name, ".keep") ||
-	    ends_with(file_name, ".promisor"))
+	    ends_with(file_name, ".promisor") ||
+	    ends_with(file_name, ".mtimes"))

likewise

I am again concerned about 32-bit time_t assumptions. time_t is 32-bit on some platforms, signed/unsigned, and sometimes 64-bit. We are talking about potentially long-persistent files, as I understand this series, so we should not be limiting times to end at 2038. That's only 16 years off and I would wager that many clones that exist today will exist then. --Randall

#158 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Quoted 5 lines · On Tue, May 24, 2022 at 03:44:00PM -0400, [email protected] wrote:

I am again concerned about 32-bit time_t assumptions. time_t is 32-bit on some platforms, signed/unsigned, and sometimes 64-bit. We are talking about potentially long-persistent files, as I understand this series, so we should not be limiting times to end at 2038. That's only 16 years off and I would wager that many clones that exist today will exist then.

Note that we're using unsigned fields here, so we have until 2106 (see my earlier response on this in
https://lore.kernel.org/git/[email protected]/).

Thanks,
Taylor

#159 lore raw

RE: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

On May 24, 2022 6:25 PM ,Taylor Blau write:

Quoted 9 lines · On Tue, May 24, 2022 at 03:44:00PM -0400, [email protected] wrote:
Quoted 5 lines

I am again concerned about 32-bit time_t assumptions. time_t is 32-bit on some platforms, signed/unsigned, and sometimes 64-bit. We are talking about potentially long-persistent files, as I understand this series, so we should not be limiting times to end at 2038. That's only 16 years off and I would wager that many clones that exist today will exist then.

Note that we're using unsigned fields here, so we have until 2106 (see my earlier response on this in https://lore.kernel.org/git/[email protected]/).

I appreciate that, but 32-bit time_t is still signed on many platforms, so when cast, it still might, at some point in another series, cause issues. Please be cautious. I expect that this is the particular hill on which I will die. 😉 --Randall

#160 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Quoted 8 lines · On Tue, May 24, 2022 at 07:24:14PM -0400, [email protected] wrote:

On May 24, 2022 6:25 PM ,Taylor Blau write:

Quoted 7 lines · On Tue, May 24, 2022 at 03:44:00PM -0400, [email protected] wrote:
Quoted 5 lines

I am again concerned about 32-bit time_t assumptions. time_t is 32-bit on some platforms, signed/unsigned, and sometimes 64-bit. We are talking about potentially long-persistent files, as I understand this series, so we should not be limiting times to end at 2038. That's only 16 years off and I would wager that many clones that exist today will exist then.

Quoted 8 lines
Quoted 2 lines

Note that we're using unsigned fields here, so we have until 2106 (see my earlier response on this in https://lore.kernel.org/git/[email protected]/).

I appreciate that, but 32-bit time_t is still signed on many platforms, so when cast, it still might, at some point in another series, cause issues. Please be cautious. I expect that this is the particular hill on which I will die. 😉
--Randall

Yes, definitely. There is only one spot that we turn the result of nth_packed_mtime() into a time_t, and that's in add_object_in_unpacked_pack(). The code there is something like:

    time_t mtime;
    if (pack->is_cruft)
      mtime = nth_packed_mtime(pack, object_pos);
    else
      mtime = pack->mtime;

    ...

    add_cruft_object_entry(oid, ..., mtime);

...and the reason mtime is a time_t is because that's the type of pack->mtime.

And we quickly convert that back to a uint32_t in add_cruft_object_entry(). If time_t is signed, then we'll truncate any values beyond 2106, and pre-epoch values will become large positive values. That means our error is one-sided in the favorable direction, i.e., that we'll keep objects around for longer instead of pruning something that we shouldn't have.

Thanks,
Taylor

#161 lore raw

RE: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Quoted 35 lines · Taylor Blau
Quoted 8 lines · On Tue, May 24, 2022 at 07:24:14PM -0400, [email protected] wrote:

On May 24, 2022 6:25 PM ,Taylor Blau write:

Quoted 7 lines · On Tue, May 24, 2022 at 03:44:00PM -0400, [email protected] wrote:
Quoted 6 lines

I am again concerned about 32-bit time_t assumptions. time_t is 32-bit on some platforms, signed/unsigned, and sometimes 64-bit. We are talking about potentially long-persistent files, as I understand this series, so we should not be limiting times to end at 2038. That's only
16 years off and I would wager that many clones that exist today will exist

then.

Quoted 3 lines
Quoted 3 lines

Note that we're using unsigned fields here, so we have until 2106 (see my earlier response on this in

https://lore.kernel.org/git/[email protected]/).

Quoted 6 lines

I appreciate that, but 32-bit time_t is still signed on many platforms, so when cast, it still might, at some point in another series, cause issues. Please be cautious. I expect that this is the particular hill on which I will die. 😉
--Randall

Yes, definitely. There is only one spot that we turn the result of nth_packed_mtime() into a time_t, and that's in add_object_in_unpacked_pack(). The code there is something like:

   time_t mtime;
   if (pack->is_cruft)
     mtime = nth_packed_mtime(pack, object_pos);
   else
     mtime = pack->mtime;
   ...

   add_cruft_object_entry(oid, ..., mtime);
Quoted 8 lines

...and the reason mtime is a time_t is because that's the type of pack->mtime.

And we quickly convert that back to a uint32_t in add_cruft_object_entry(). If time_t is signed, then we'll truncate any values beyond 2106, and pre-epoch values will become large positive values. That means our error is one-sided in the favorable direction, i.e., that we'll keep objects around for longer instead of pruning something that we shouldn't have.

I can only hope. I am working with the platform compiler team on time_t issues. Hoping we can get to 64-bit builds within two years, but out of my control. That would make time_t an int64_t, which puts failure well outside my own lifespan and anyone else in my company. Provisioning for signed 64-bit time values would be prudent even if unsupported in a specific build. We are almost 1/4 through 20xx. --Randall

#162 lore raw

adding new 32-bit on-disk (unsigned) timestamp formats (was: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files)

In reply to Taylor Blau
Quoted 35 lines · Taylor Blau
Quoted 8 lines · On Tue, May 24, 2022 at 07:24:14PM -0400, [email protected] wrote:

On May 24, 2022 6:25 PM ,Taylor Blau write:

Quoted 7 lines · On Tue, May 24, 2022 at 03:44:00PM -0400, [email protected] wrote:
Quoted 5 lines

I am again concerned about 32-bit time_t assumptions. time_t is 32-bit on some platforms, signed/unsigned, and sometimes 64-bit. We are talking about potentially long-persistent files, as I understand this series, so we should not be limiting times to end at 2038. That's only 16 years off and I would wager that many clones that exist today will exist then.

Quoted 8 lines
Quoted 2 lines

Note that we're using unsigned fields here, so we have until 2106 (see my earlier response on this in https://lore.kernel.org/git/[email protected]/).

I appreciate that, but 32-bit time_t is still signed on many platforms, so when cast, it still might, at some point in another series, cause issues. Please be cautious. I expect that this is the particular hill on which I will die. 😉
--Randall

Yes, definitely. There is only one spot that we turn the result of nth_packed_mtime() into a time_t, and that's in add_object_in_unpacked_pack(). The code there is something like:

    time_t mtime;
    if (pack->is_cruft)
      mtime = nth_packed_mtime(pack, object_pos);
    else
      mtime = pack->mtime;

    ...

    add_cruft_object_entry(oid, ..., mtime);

...and the reason mtime is a time_t is because that's the type of pack->mtime.

Quoted 6 lines

And we quickly convert that back to a uint32_t in add_cruft_object_entry(). If time_t is signed, then we'll truncate any values beyond 2106, and pre-epoch values will become large positive values. That means our error is one-sided in the favorable direction, i.e., that we'll keep objects around for longer instead of pruning something that we shouldn't have.

I must say that I really don't like this part of the format. Is it really necessary to optimize the storage space here in a way that leaves open questions about future time_t compatibility, and having to introduce the first use of unsigned 32 bit timestamps to git's codebase?

Yes, this is its own self-contained format, so we don't *need* time_t here, but it's also really handy if we can eventually consistently use 64 time_t everywhere and not worry about any compatibility issues, or unsigned v.s. signed, or to create our own little ext4-like signed 32 bit timestamp format.

Once we hit 2038 (or near that date) this would be the only part of our codebase & on-disk formats that I'm aware of that would differ from time_t's signedness, but perhaps there's some I've missed.

If there isn't a demonstrable reason (as in some real numbers, or accompanying benchmark etc.) to special-snowflake this I really think we should just go for signed 64 bit here, i.e. matching time_t on 64 bit systems.

If we really are trying to micro-optimize storage space here I'm willing to bet that this is still a bad/premature optimization. There's much better ways to store this sort of data in a compact way if that's the concern. E.g. you'd store a 64 bit "base" timestamp in the header for the first entry, and have smaller (signed) "delta" timestamps storing offsets from that "base" timestamp.

This would take advantage of the fact that when we find loose objects we're vanishingly unlikely to have them splayed over more than a days/weeks/months or in the worst case small number of years from the "base" (and if we ever do we could simply shrug and leave such objects out of the pack entirely).

We could thus keep the 32 bit second-resolution timestamps you have here, they'd just be signed deltas to the 64 bit signed "base" in a header.

Even better (again, if micro-optimizing this is really needed) would be to store a 64 bit signed base and a table of 16 bit signed offsets.

We'd simply declare that for our expiry times we'd "snap" any such values to the next day. Our current GC config exposes down-to-the-second expiry times, but in practice nobody needs that. A 16 bit signed "day offset" would give you 2^15/365 = 89 years +/- of day-resolution expiry for objects. To avoid thundering herds we could even fake up an exact down-to-the-second expiry on the computed day by combining the expiry time & the first few bits of the OID.

== BREAK

Aside about time_t being signed v.s. unsigned. This is edited from an older off-list E-Mail of mine (from git-security): For time_t itself no standard says that time_t must be signed, but in practice it's ubiquitous

This thread is informative
http://mm.icann.org/pipermail/tz/2004-July/012503.html it continues the month after: http://mm.icann.org/pipermail/tz/2004-August/thread.html

Summary: Yeah it can be unsigned in theory, but it seems like nobody's been crazy enough to try it, so it's de-facto standardized to signed. Everyone has a Y2038 problem, nobody has a Y2106 problem. Well, with time_t, e.g. Linux filesystems tend to use unsigned 32 bit epochs:
https://kernelnewbies.org/y2038/vfs

#163 lore raw

Re: adding new 32-bit on-disk (unsigned) timestamp formats (was: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files)

Quoted 4 lines · Ævar Arnfjörð Bjarmason

I must say that I really don't like this part of the format. Is it really necessary to optimize the storage space here in a way that leaves open questions about future time_t compatibility, and having to introduce the first use of unsigned 32 bit timestamps to git's codebase?

The commit-graph file format uses unsigned 34-bit timestamps (packed with 30-bit topological levels in the CDAT chunk), so this "not-64-bit signed timestamps" thing is something we've done before.

Quoted 5 lines

Yes, this is its own self-contained format, so we don't *need* time_t here, but it's also really handy if we can eventually consistently use 64 time_t everywhere and not worry about any compatibility issues, or unsigned v.s. signed, or to create our own little ext4-like signed 32 bit timestamp format.

We can also use a new file format version when it is necessary. We have a lot of time to add that detail without overly complicating the format right now.

Quoted 6 lines

If we really are trying to micro-optimize storage space here I'm willing to bet that this is still a bad/premature optimization. There's much better ways to store this sort of data in a compact way if that's the concern. E.g. you'd store a 64 bit "base" timestamp in the header for the first entry, and have smaller (signed) "delta" timestamps storing offsets from that "base" timestamp.

This is a good idea for a v2 format when that is necessary.

Thanks,
-Stolee

#164 lore raw

Re: adding new 32-bit on-disk (unsigned) timestamp formats (was: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files)

Quoted 20 lines · Derrick Stolee
Quoted 4 lines · Ævar Arnfjörð Bjarmason

I must say that I really don't like this part of the format. Is it really necessary to optimize the storage space here in a way that leaves open questions about future time_t compatibility, and having to introduce the first use of unsigned 32 bit timestamps to git's codebase?

The commit-graph file format uses unsigned 34-bit timestamps (packed with 30-bit topological levels in the CDAT chunk), so this "not-64-bit signed timestamps" thing is something we've done before.

Quoted 5 lines

Yes, this is its own self-contained format, so we don't *need* time_t here, but it's also really handy if we can eventually consistently use 64 time_t everywhere and not worry about any compatibility issues, or unsigned v.s. signed, or to create our own little ext4-like signed 32 bit timestamp format.

We can also use a new file format version when it is necessary. We have a lot of time to add that detail without overly complicating the format right now.

Quoted 8 lines
Quoted 6 lines

If we really are trying to micro-optimize storage space here I'm willing to bet that this is still a bad/premature optimization. There's much better ways to store this sort of data in a compact way if that's the concern. E.g. you'd store a 64 bit "base" timestamp in the header for the first entry, and have smaller (signed) "delta" timestamps storing offsets from that "base" timestamp.

This is a good idea for a v2 format when that is necessary.

I agree here.

I'm not opposed to such a change (or even being the one to work on it!), but I would encourage us to pursue that change outside of this series, since it can easily be done on top.

Of course, if we ever did decide to implement 64-bit mtimes, we would have to maintain support for reading both the 32-bit and 64-bit values. But I think the code is well-equipped to do that, and it could be done on top without significant additional complexity.

Thanks,
Taylor

#165 lore raw

Re: adding new 32-bit on-disk (unsigned) timestamp formats (was: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files)

Quoted 32 lines · Taylor Blau
Quoted 20 lines · Derrick Stolee
Quoted 4 lines · Ævar Arnfjörð Bjarmason

I must say that I really don't like this part of the format. Is it really necessary to optimize the storage space here in a way that leaves open questions about future time_t compatibility, and having to introduce the first use of unsigned 32 bit timestamps to git's codebase?

The commit-graph file format uses unsigned 34-bit timestamps (packed with 30-bit topological levels in the CDAT chunk), so this "not-64-bit signed timestamps" thing is something we've done before.

Quoted 5 lines

Yes, this is its own self-contained format, so we don't *need* time_t here, but it's also really handy if we can eventually consistently use 64 time_t everywhere and not worry about any compatibility issues, or unsigned v.s. signed, or to create our own little ext4-like signed 32 bit timestamp format.

We can also use a new file format version when it is necessary. We have a lot of time to add that detail without overly complicating the format right now.

Quoted 8 lines
Quoted 6 lines

If we really are trying to micro-optimize storage space here I'm willing to bet that this is still a bad/premature optimization. There's much better ways to store this sort of data in a compact way if that's the concern. E.g. you'd store a 64 bit "base" timestamp in the header for the first entry, and have smaller (signed) "delta" timestamps storing offsets from that "base" timestamp.

This is a good idea for a v2 format when that is necessary.

I agree here.

Quoted 8 lines

I'm not opposed to such a change (or even being the one to work on it!), but I would encourage us to pursue that change outside of this series, since it can easily be done on top.

Of course, if we ever did decide to implement 64-bit mtimes, we would have to maintain support for reading both the 32-bit and 64-bit values. But I think the code is well-equipped to do that, and it could be done on top without significant additional complexity.

Do you mean "on top" in the sense that we'd expect that before the next release, so that we wouldn't need to deal with bumping the format, and have some phase-out period for the older version etc.

Or that we would need to treat what's landing here as something we'll need to support going forward?

I think if a format change is worthwhile doing at all that it's worth just doing it now if it's going to be the latter of those, as changing file formats before they're in the wild is easy, but after that it's at best a bit tedious. E.g. we'll need testing to see how we deal with mixed new/old format files etc. etc.

#166 lore raw

Re: adding new 32-bit on-disk (unsigned) timestamp formats (was: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files)

Quoted 20 lines · Ævar Arnfjörð Bjarmason
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines

If we really are trying to micro-optimize storage space here I'm willing to bet that this is still a bad/premature optimization. There's much better ways to store this sort of data in a compact way if that's the concern. E.g. you'd store a 64 bit "base" timestamp in the header for the first entry, and have smaller (signed) "delta" timestamps storing offsets from that "base" timestamp.

This is a good idea for a v2 format when that is necessary.

I agree here.

Quoted 8 lines

I'm not opposed to such a change (or even being the one to work on it!), but I would encourage us to pursue that change outside of this series, since it can easily be done on top.

Of course, if we ever did decide to implement 64-bit mtimes, we would have to maintain support for reading both the 32-bit and 64-bit values. But I think the code is well-equipped to do that, and it could be done on top without significant additional complexity.

Quoted 6 lines

Do you mean "on top" in the sense that we'd expect that before the next release, so that we wouldn't need to deal with bumping the format, and have some phase-out period for the older version etc.

Or that we would need to treat what's landing here as something we'll need to support going forward?

My plan is to treat what will hopefully land here as something we're going to support.

I meant "on top" in the sense that the format implemented here does not restrict us against making changes (like adding support for wider records) in the future. IOW, I did not mean to suggest that we should expect more patches from me in this cycle to deprecate parts of the v1 format.

In other words (again ;-)), I would like to see us ship this format with the existing 32-bit records.

Quoted 5 lines

I think if a format change is worthwhile doing at all that it's worth just doing it now if it's going to be the latter of those, as changing file formats before they're in the wild is easy, but after that it's at best a bit tedious. E.g. we'll need testing to see how we deal with mixed new/old format files etc. etc.

I can understand where you're coming from, though as I noted earlier in the thread, I don't think changing the format in the manner you suggest would be that difficult in practice.

But in the meantime, the existing format is useful and works, and I don't think we should go back to the drawing board for something that we can do later if we decide to.

Thanks,
Taylor

#167 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

In reply to Jonathan Nieder
Quoted 8 lines · Jonathan Nieder

Hi,

Quoted 3 lines · Taylor Blau

This patch prepares for cruft packs by defining the .mtimes format, and introducing a basic API that callers can use to read out individual mtimes.

Quoted 6 lines

Makes sense. Does this intend to produce any functional change? I'm guessing not (and the lack of tests agrees), but the commit message doesn't say so.

By the way, is this something we could cover in tests, e.g. using a test helper that exercises the new code?

This does not produce a functional change, no. This commit in isolation adds a bunch of dead code that will be used (and tested) in the following patches.

There is a test helper that is added (and then used extensively further on in the series) four patches later, c.f., "t/helper: add 'pack-mtimes' test-tool".

Quoted 43 lines

[...]

Quoted 15 lines
Documentation/technical/pack-format.txt +9 -0
@@ -294,6 +294,25 @@ Pack file entry: <+
 
 All 4-byte numbers are in network order.
 
+== pack-*.mtimes files have the format:
+
+All 4-byte numbers are in network byte order.
+
+  - A 4-byte magic number '0x4d544d45' ('MTME').
+
+  - A 4-byte version identifier (= 1).
+
+  - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).

Documentation/technical/pack-format.txt Pack file entry: <+ · line 315

Quoted 10 lines
Documentation/technical/pack-format.txt +10 -0
+
+  - A table of 4-byte unsigned integers. The ith value is the
+    modification time (mtime) of the ith object in the corresponding
+    pack by lexicographic (index) order. The mtimes count standard
+    epoch seconds.
+
+  - A trailer, containing a checksum of the corresponding packfile,
+    and a checksum of all of the above (each having length according
+    to the specified hash function).
+

This describes the "syntax" but not the "semantics" of the file. Should I look to a separate piece of documentation for the semantics? If so, can this one include a mention of that piece of documentation to make it easier to find?

[...]

Quoted 10 lines
object-store.h +2 -1
@@ -115,12 +115,15 @@ struct packed_git {
 		 freshened:1,
 		 do_not_close:1,
 		 pack_promisor:1,
-		 multi_pack_index:1;
+		 multi_pack_index:1,
+		 is_cruft:1;
 	unsigned char hash[GIT_MAX_RAWSZ];
Quoted 8 lines

object-store.h struct packed_git { · line 126

Quoted 6 lines
object-store.h +2 -0
 	struct revindex_entry *revindex;
 	const uint32_t *revindex_data;
 	const uint32_t *revindex_map;
 	size_t revindex_size;
+	const uint32_t *mtimes_map;
+	size_t mtimes_size;

What does mtimes_map contain? A comment would help.

It contains a pointer at the beginning of the mmapped region of the .mtimes file, similar to revindex_map above it.

Quoted 9 lines

pack-mtimes.c line 3

Quoted 7 lines
pack-mtimes.c new file +3 -0
@@ -0,0 +1,126 @@
+#include "pack-mtimes.h"
+#include "object-store.h"
+#include "packfile.h"

Missing #include of git-compat-util.h.

Ah, good eyes: thanks.

Junio: would you like a replacement patch / a whole new copy of the series / or can you amend this locally when queuing? Whatever is lowest effort for you works for me.

Quoted 5 lines
Quoted 5 lines
pack-mtimes.c new file +5 -0
+
+static char *pack_mtimes_filename(struct packed_git *p)
+{
+	size_t len;
+	if (!strip_suffix(p->pack_name, ".pack", &len))
Quoted 8 lines

pack-mtimes.c line 12

Quoted 4 lines
pack-mtimes.c new file +4 -0
+		BUG("pack_name does not end in .pack");
+	/* NEEDSWORK: this could reuse code from pack-revindex.c. */
+	return xstrfmt("%.*s.mtimes", (int)len, p->pack_name);
+}

This seems simple enough that it's not obvious we need more code sharing. Do you agree? If so, I'd suggest just removing the NEEDSWORK comment.

Yeah, it is conceptually simple, though it feels like the sort of thing that could benefit from not having to be written once for each extension (hence the comment).

Quoted 7 lines

pack-mtimes.c line 15

Quoted 3 lines
pack-mtimes.c new file +3 -0
+
+#define MTIMES_HEADER_SIZE (12)
+#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 * the_hash_algo->rawsz))

Hm, the all-caps name makes this feel like a compile-time constant but it contains a reference to the_hash_algo. Could it be an inline function instead?

Yes, it could be an inline function, but I don't think there is necessarily anything wrong with it being a #define'd macro. There are some other examples, e.g., RIDX_MIN_SIZE, MIDX_MIN_SIZE, GRAPH_DATA_WIDTH, and PACK_SIZE_THRESHOLD (to name a few) which also use the_hash_algo on the right-hand side of a #define.

Quoted 4 lines
Quoted 4 lines
pack-mtimes.c new file +4 -0
+
+struct mtimes_header {
+	uint32_t signature;
+	uint32_t version;
Quoted 8 lines

pack-mtimes.c line 25

Quoted 6 lines
pack-mtimes.c new file +6 -0
+	uint32_t hash_id;
+};
+
+static int load_pack_mtimes_file(char *mtimes_file,
+				 uint32_t num_objects,
+				 const uint32_t **data_p, size_t *len_p)

What does this function do? A comment would help.

I know that I'm biased as the author of this code, but I think the signature is clear here. At least, I'm not sure what information a comment would add that the function name and its arguments don't already convey.

Quoted 4 lines

pack-mtimes.c line 52

Quoted 1 line
pack-mtimes.c new file +1 -0
+	if (mtimes_size - MTIMES_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {

This presupposes that the hash_id matches the_hash_algo. Maybe worth a NEEDSWORK comment.

Good catch.

Quoted 11 lines

[...]

pack-mtimes.c line 86

Quoted 7 lines
pack-mtimes.c new file +7 -0
+cleanup:
+	if (ret) {
+		if (data)
+			munmap(data, mtimes_size);
+	} else {
+		*len_p = mtimes_size;
+		*data_p = (const uint32_t *)data;

Do we know that 'data' is uint32_t aligned? Casting earlier in the function could make that more obvious.

data is definitely uint32_t aligned, but this is a tradeoff, since if we wrote:

    uint32_t *data = xmmap(...);

then I think we would have to change the case where ret is non-zero to be:

    if (data)
        munmap((void*)data, ...);

and likewise, data_p is const.

Quoted 9 lines
Quoted 1 line

+int load_pack_mtimes(struct packed_git *p)

This could use a doc comment in the header file. For example, what requirements do we have on what the caller passes as 'p'?

[...]

Quoted 1 line

+uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos)

Likewise.

Sure. I wonder when we should do that, though. I'm not trying to be impatient to get this merged, but iterating on the documentation feels like it could be done on top without having to re-send the substantive parts of this series over and over.

Quoted 6 lines

[...]

Quoted 2 lines
packfile.c +0 -0

[...]

Quoted 2 lines
packfile.c +0 -0
@@ -363,7 +373,7 @@ void close_object_store(struct raw_object_store *o)
 
Quoted 8 lines

packfile.c void close_object_store(struct raw_object_store *o) · line 376

Quoted 4 lines
packfile.c +1 -1
 void unlink_pack_path(const char *pack_name, int force_delete)
 {
-	static const char *exts[] = {".pack", ".idx", ".rev", ".keep", ".bitmap", ".promisor"};
+	static const char *exts[] = {".pack", ".idx", ".rev", ".keep", ".bitmap", ".promisor", ".mtimes"};

Are these in any particular order? Should they be?

[...]

These aren't technically in any particular order (nor should they be), though .idx should be first. I'm leaving it alone here (semi-intentionally, since the race it opens up isn't related to this series, and it's on my list to deal with after this code has settled).

Quoted 4 lines
Quoted 4 lines
packfile.c +0 -0
@@ -718,6 +728,10 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 	if (!access(p->pack_name, F_OK))
 		p->pack_promisor = 1;
 
Quoted 16 lines
Quoted 8 lines
packfile.c +4 -0
+	xsnprintf(p->pack_name + path_len, alloc - path_len, ".mtimes");
+	if (!access(p->pack_name, F_OK))
+		p->is_cruft = 1;
+
 	xsnprintf(p->pack_name + path_len, alloc - path_len, ".pack");
 	if (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {
 		free(p);

packfile.c static void prepare_pack(const char *full_name, size_t full_name_len, · line 887

Quoted 6 lines
packfile.c +2 -1
@@ -869,7 +883,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 	    ends_with(file_name, ".pack") ||
 	    ends_with(file_name, ".bitmap") ||
 	    ends_with(file_name, ".keep") ||
-	    ends_with(file_name, ".promisor"))
+	    ends_with(file_name, ".promisor") ||
+	    ends_with(file_name, ".mtimes"))

likewise

No specific order here (since these are all OR'd together).

Thanks,
Taylor

#168 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Hi,

Quoted 5 lines · Taylor Blau
Quoted 3 lines · Jonathan Nieder

Makes sense. Does this intend to produce any functional change? I'm guessing not (and the lack of tests agrees), but the commit message doesn't say so.

[...]

Quoted 3 lines

This does not produce a functional change, no. This commit in isolation adds a bunch of dead code that will be used (and tested) in the following patches.

[...]

Quoted 4 lines
Quoted 1 line

What does mtimes_map contain? A comment would help.

It contains a pointer at the beginning of the mmapped region of the .mtimes file, similar to revindex_map above it.

To be clear, in cases like this by "comment" I mean "in-code comment". I.e., my interest is not that _I_ find out the answer but that the code becomes more maintainable via the answer becoming easier to find.

[...]

Quoted 7 lines
Quoted 3 lines

This seems simple enough that it's not obvious we need more code sharing. Do you agree? If so, I'd suggest just removing the NEEDSWORK comment.

Yeah, it is conceptually simple, though it feels like the sort of thing that could benefit from not having to be written once for each extension (hence the comment).

The reason I asked is that the NEEDSWORK here actually got in the way of comprehension for me --- it made me wonder "is there some complexity here I'm missing?"

That's why I'd suggest one of
- removing the NEEDSWORK comment
- going ahead and implementing the code sharing you mean, or
- fleshing out the NEEDSWORK comment so the reader can wonder less

Quoted 8 lines
Quoted 7 lines

pack-mtimes.c line 15

Quoted 3 lines
pack-mtimes.c new file +3 -0
+
+#define MTIMES_HEADER_SIZE (12)
+#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 * the_hash_algo->rawsz))

Hm, the all-caps name makes this feel like a compile-time constant but it contains a reference to the_hash_algo. Could it be an inline function instead?

Quoted 5 lines

Yes, it could be an inline function, but I don't think there is necessarily anything wrong with it being a #define'd macro. There are some other examples, e.g., RIDX_MIN_SIZE, MIDX_MIN_SIZE, GRAPH_DATA_WIDTH, and PACK_SIZE_THRESHOLD (to name a few) which also use the_hash_algo on the right-hand side of a #define.

Those are due to an incomplete migration from use of the true constant GIT_SHA1_RAWSZ to use of the dynamic value the_hash_algo->rawsz, no? In other words, "other examples do it wrong" doesn't feel like a great justification for making it worse in new code.

[...]

Quoted 10 lines
Quoted 5 lines

pack-mtimes.c line 25

Quoted 3 lines
pack-mtimes.c new file +3 -0
+static int load_pack_mtimes_file(char *mtimes_file,
+				 uint32_t num_objects,
+				 const uint32_t **data_p, size_t *len_p)

What does this function do? A comment would help.

I know that I'm biased as the author of this code, but I think the signature is clear here. At least, I'm not sure what information a comment would add that the function name and its arguments don't already convey.

Ah, thanks for this point of clarification. What isn't clear from the signature is
- when should I call this function?
- what does its return value represent?
- how does it handle errors?

I agree that the parameters are self-explanatory.

Quoted 14 lines
Quoted 10 lines

pack-mtimes.c line 86

Quoted 7 lines
pack-mtimes.c new file +7 -0
+cleanup:
+	if (ret) {
+		if (data)
+			munmap(data, mtimes_size);
+	} else {
+		*len_p = mtimes_size;
+		*data_p = (const uint32_t *)data;

Do we know that 'data' is uint32_t aligned? Casting earlier in the function could make that more obvious.

data is definitely uint32_t aligned, but this is a tradeoff, since if we wrote:

Quoted 8 lines
    uint32_t *data = xmmap(...);

then I think we would have to change the case where ret is non-zero to be:

    if (data)
        munmap((void*)data, ...);

and likewise, data_p is const.

Doing it that way sounds great to me. That way, the type contains the information we need up-front and the safety of the cast is obvious in the place where the cast is needed.

(Although my understanding is also that in C it's fine to pass a uint32_t* to a function expecting a void*, so the second cast would also not be needed.)

[...]

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

+int load_pack_mtimes(struct packed_git *p)

This could use a doc comment in the header file. For example, what requirements do we have on what the caller passes as 'p'?

[...]

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

+uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos)

Likewise.

Sure. I wonder when we should do that, though. I'm not trying to be impatient to get this merged, but iterating on the documentation feels like it could be done on top without having to re-send the substantive parts of this series over and over.

In terms of re-sending patches, sending a "fixup!" patch with the minor changes you want to make doesn't seem too problematic to me. In general a major benefit of code review is getting others' eyes on new code from the standpoint of readability and maintainability; including comments like this up front doesn't seem like a huge amount to ask (versus getting those comments to be perfect, which would be unreasonable to expect since it's not hard to update them over time).

Quoted 2 lines

Thanks,
Taylor

Thanks for looking it through.

Sincerely,
Jonathan

#169 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Quoted 8 lines · Jonathan Nieder
Quoted 4 lines
Quoted 1 line

What does mtimes_map contain? A comment would help.

It contains a pointer at the beginning of the mmapped region of the .mtimes file, similar to revindex_map above it.

To be clear, in cases like this by "comment" I mean "in-code comment". I.e., my interest is not that _I_ find out the answer but that the code becomes more maintainable via the answer becoming easier to find.

OK. I'll add a comment in the fixup! patch which I'm about to send.

Quoted 9 lines

[...]

Quoted 7 lines
Quoted 3 lines

This seems simple enough that it's not obvious we need more code sharing. Do you agree? If so, I'd suggest just removing the NEEDSWORK comment.

Yeah, it is conceptually simple, though it feels like the sort of thing that could benefit from not having to be written once for each extension (hence the comment).

Quoted 8 lines

The reason I asked is that the NEEDSWORK here actually got in the way of comprehension for me --- it made me wonder "is there some complexity here I'm missing?"

That's why I'd suggest one of
- removing the NEEDSWORK comment
- going ahead and implementing the code sharing you mean, or
- fleshing out the NEEDSWORK comment so the reader can wonder less

I am a little sad to remove it, since I thought it was useful as-is. But I can just as easily remember to come back to this myself in the future, so if it is distracting to you in the meantime, then I don't mind holding onto it in my own head.

Quoted 14 lines
Quoted 8 lines
Quoted 7 lines

pack-mtimes.c line 15

Quoted 3 lines
pack-mtimes.c new file +3 -0
+
+#define MTIMES_HEADER_SIZE (12)
+#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 * the_hash_algo->rawsz))

Hm, the all-caps name makes this feel like a compile-time constant but it contains a reference to the_hash_algo. Could it be an inline function instead?

Quoted 5 lines

Yes, it could be an inline function, but I don't think there is necessarily anything wrong with it being a #define'd macro. There are some other examples, e.g., RIDX_MIN_SIZE, MIDX_MIN_SIZE, GRAPH_DATA_WIDTH, and PACK_SIZE_THRESHOLD (to name a few) which also use the_hash_algo on the right-hand side of a #define.

Quoted 4 lines

Those are due to an incomplete migration from use of the true constant GIT_SHA1_RAWSZ to use of the dynamic value the_hash_algo->rawsz, no? In other words, "other examples do it wrong" doesn't feel like a great justification for making it worse in new code.

Fair point. I can imagine reasons for the existing pattern, but updating it to handle the variable rawsz is easy to do (and it probably should have been that way since the beginning).

Quoted 12 lines

[...]

Quoted 10 lines
Quoted 5 lines

pack-mtimes.c line 25

Quoted 3 lines
pack-mtimes.c new file +3 -0
+static int load_pack_mtimes_file(char *mtimes_file,
+				 uint32_t num_objects,
+				 const uint32_t **data_p, size_t *len_p)

What does this function do? A comment would help.

I know that I'm biased as the author of this code, but I think the signature is clear here. At least, I'm not sure what information a comment would add that the function name and its arguments don't already convey.

Quoted 7 lines

Ah, thanks for this point of clarification. What isn't clear from the signature is
- when should I call this function?
- what does its return value represent?
- how does it handle errors?

I agree that the parameters are self-explanatory.

I'm hesitant to over-document a static function with a single caller, but when looking at this, I think there is an opportunity to document _its_ caller (load_pack_mtimes()) which isn't static, but was also missing documentation.

Quoted 27 lines
Quoted 14 lines
Quoted 10 lines

pack-mtimes.c line 86

Quoted 7 lines
pack-mtimes.c new file +7 -0
+cleanup:
+	if (ret) {
+		if (data)
+			munmap(data, mtimes_size);
+	} else {
+		*len_p = mtimes_size;
+		*data_p = (const uint32_t *)data;

Do we know that 'data' is uint32_t aligned? Casting earlier in the function could make that more obvious.

data is definitely uint32_t aligned, but this is a tradeoff, since if we wrote:

Quoted 8 lines
    uint32_t *data = xmmap(...);

then I think we would have to change the case where ret is non-zero to be:

    if (data)
        munmap((void*)data, ...);

and likewise, data_p is const.

Doing it that way sounds great to me. That way, the type contains the information we need up-front and the safety of the cast is obvious in the place where the cast is needed.

Quoted 5 lines

(Although my understanding is also that in C it's fine to pass a uint32_t* to a function expecting a void*, so the second cast would also not be needed.)

[...]

Done, thanks for the suggestion.

Thanks,
Taylor

#170 lore raw

RE: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Quoted 77 lines · Taylor Blau
Quoted 8 lines · Jonathan Nieder
Quoted 4 lines
Quoted 1 line

What does mtimes_map contain? A comment would help.

It contains a pointer at the beginning of the mmapped region of the .mtimes file, similar to revindex_map above it.

To be clear, in cases like this by "comment" I mean "in-code comment". I.e., my interest is not that _I_ find out the answer but that the code becomes more maintainable via the answer becoming easier to find.

OK. I'll add a comment in the fixup! patch which I'm about to send.

Quoted 9 lines

[...]

Quoted 7 lines
Quoted 3 lines

This seems simple enough that it's not obvious we need more code sharing. Do you agree? If so, I'd suggest just removing the NEEDSWORK comment.

Yeah, it is conceptually simple, though it feels like the sort of thing that could benefit from not having to be written once for each extension (hence the comment).

Quoted 8 lines

The reason I asked is that the NEEDSWORK here actually got in the way of comprehension for me --- it made me wonder "is there some complexity here I'm missing?"

That's why I'd suggest one of
- removing the NEEDSWORK comment
- going ahead and implementing the code sharing you mean, or
- fleshing out the NEEDSWORK comment so the reader can wonder less

I am a little sad to remove it, since I thought it was useful as-is. But I can just as easily remember to come back to this myself in the future, so if it is distracting to you in the meantime, then I don't mind holding onto it in my own head.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

pack-mtimes.c

Quoted 4 lines
pack-mtimes.c new file +4 -0
+
+#define MTIMES_HEADER_SIZE (12)
+#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 *
+the_hash_algo->rawsz))

Hm, the all-caps name makes this feel like a compile-time constant but it contains a reference to the_hash_algo. Could it be an inline function instead?

Quoted 4 lines
Quoted 4 lines

Yes, it could be an inline function, but I don't think there is necessarily anything wrong with it being a #define'd macro. There are some other examples, e.g., RIDX_MIN_SIZE, MIDX_MIN_SIZE, GRAPH_DATA_WIDTH, and PACK_SIZE_THRESHOLD (to name a few) which

also

Quoted 6 lines
Quoted 1 line

use the_hash_algo on the right-hand side of a #define.

Those are due to an incomplete migration from use of the true constant GIT_SHA1_RAWSZ to use of the dynamic value the_hash_algo->rawsz, no? In other words, "other examples do it wrong" doesn't feel like a great justification for making it worse in new code.

Fair point. I can imagine reasons for the existing pattern, but updating it to handle the variable rawsz is easy to do (and it probably should have been that way since the beginning).

Quoted 18 lines

[...]

Quoted 10 lines
Quoted 5 lines

pack-mtimes.c line 25

Quoted 3 lines
pack-mtimes.c new file +3 -0
+static int load_pack_mtimes_file(char *mtimes_file,
+				 uint32_t num_objects,
+				 const uint32_t **data_p, size_t *len_p)

What does this function do? A comment would help.

I know that I'm biased as the author of this code, but I think the signature is clear here. At least, I'm not sure what information a comment would add that the function name and its arguments don't already convey.

Ah, thanks for this point of clarification. What isn't clear from the signature is
- when should I call this function?
- what does its return value represent?
- how does it handle errors?

Quoted 36 lines
Quoted 1 line

I agree that the parameters are self-explanatory.

I'm hesitant to over-document a static function with a single caller, but when looking at this, I think there is an opportunity to document _its_ caller (load_pack_mtimes()) which isn't static, but was also missing documentation.

Quoted 30 lines
Quoted 14 lines
Quoted 10 lines

pack-mtimes.c line 86

Quoted 7 lines
pack-mtimes.c new file +7 -0
+cleanup:
+	if (ret) {
+		if (data)
+			munmap(data, mtimes_size);
+	} else {
+		*len_p = mtimes_size;
+		*data_p = (const uint32_t *)data;

Do we know that 'data' is uint32_t aligned? Casting earlier in the function could make that more obvious.

data is definitely uint32_t aligned, but this is a tradeoff, since if we wrote:

Quoted 8 lines
    uint32_t *data = xmmap(...);

then I think we would have to change the case where ret is non-zero to be:

    if (data)
        munmap((void*)data, ...);

and likewise, data_p is const.

Doing it that way sounds great to me. That way, the type contains the information we need up-front and the safety of the cast is obvious in the place where the cast is needed.

(Although my understanding is also that in C it's fine to pass a uint32_t* to a function expecting a void*, so the second cast would also not be needed.)

I do not think c99 allows this in 100% of cases - specifically if there a const void * involved. gcc does not care. I do not think c89 cares either. I will watch out for it when this is merged. --Randall

#171 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Quoted 16 lines · On Wed, May 25, 2022 at 05:58:49PM -0400, [email protected] wrote:
Quoted 16 lines
Quoted 16 lines
Quoted 11 lines

data is definitely uint32_t aligned, but this is a tradeoff, since if we wrote:

    uint32_t *data = xmmap(...);

then I think we would have to change the case where ret is non-zero to be:

    if (data)
        munmap((void*)data, ...);

and likewise, data_p is const.

Doing it that way sounds great to me. That way, the type contains the information we need up-front and the safety of the cast is obvious in the place where the cast is needed.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

(Although my understanding is also that in C it's fine to pass a uint32_t* to a function expecting a void*, so the second cast would also not be needed.)

I do not think c99 allows this in 100% of cases - specifically if there a const void * involved. gcc does not care. I do not think c89 cares either. I will watch out for it when this is merged.

Thanks for the heads up. I looked through the results of "git grep '= xmmap'" to see if we had contemporary examples of either assigning to a non-'void *', or passing a non-'void *' variable to munmap.

Luckily, we have both, so this shouldn't cause a problem. fixup! patch incoming shortly...

Thanks,
Taylor

#172 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

In reply to Taylor Blau

Junio,

Quoted 3 lines · Taylor Blau

To store the individual mtimes of objects in a cruft pack, introduce a new .mtimes format that can optionally accompany a single pack in the repository.

Like I mentioned in this sub-thread, here is a small fixup! to apply on top of this patch when queueing. I'm hoping this will be easier than reapplying the dozen+ or so patches in this series (the rest of which are unchanged). But if it isn't, please let me know and I can send you a reroll of the whole thing.

In the meantime, here's the fixup...

--- 8< ---
Subject: [PATCH] fixup! pack-mtimes: support reading .mtimes files

---
 object-store.h |  5 +++++
 pack-mtimes.c  | 35 +++++++++++++++++++----------------
 pack-mtimes.h  | 11 +++++++++++
 3 files changed, 35 insertions(+), 16 deletions(-)
object-store.h +5 -0
index 2c4671ed7a..05cc9a33ed 100644
@@ -122,6 +122,11 @@ struct packed_git {
 	const uint32_t *revindex_data;
 	const uint32_t *revindex_map;
 	size_t revindex_size;
+	/*
+	 * mtimes_map points at the beginning of the memory mapped region of
+	 * this pack's corresponding .mtimes file, and mtimes_size is the size
+	 * of that .mtimes file
+	 */
 	const uint32_t *mtimes_map;
 	size_t mtimes_size;
 	/* something like ".git/objects/pack/xxxxx.pack" */
pack-mtimes.c +19 -16
index 46ad584af1..0e0aafdcb0 100644
@@ -1,3 +1,4 @@
+#include "git-compat-util.h"
 #include "pack-mtimes.h"
 #include "object-store.h"
 #include "packfile.h"
@@ -7,12 +8,10 @@ static char *pack_mtimes_filename(struct packed_git *p)
 	size_t len;
 	if (!strip_suffix(p->pack_name, ".pack", &len))
 		BUG("pack_name does not end in .pack");
-	/* NEEDSWORK: this could reuse code from pack-revindex.c. */
 	return xstrfmt("%.*s.mtimes", (int)len, p->pack_name);
 }
 
 #define MTIMES_HEADER_SIZE (12)
-#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 * the_hash_algo->rawsz))
 
 struct mtimes_header {
 	uint32_t signature;
@@ -26,10 +25,9 @@ static int load_pack_mtimes_file(char *mtimes_file,
 {
 	int fd, ret = 0;
 	struct stat st;
-	void *data = NULL;
-	size_t mtimes_size;
+	uint32_t *data = NULL;
+	size_t mtimes_size, expected_size;
 	struct mtimes_header header;
-	uint32_t *hdr;
 
 	fd = git_open(mtimes_file);
 
@@ -44,21 +42,16 @@ static int load_pack_mtimes_file(char *mtimes_file,
 
 	mtimes_size = xsize_t(st.st_size);
 
-	if (mtimes_size < MTIMES_MIN_SIZE) {
+	if (mtimes_size < MTIMES_HEADER_SIZE) {
 		ret = error(_("mtimes file %s is too small"), mtimes_file);
 		goto cleanup;
 	}
 
-	if (mtimes_size - MTIMES_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {
-		ret = error(_("mtimes file %s is corrupt"), mtimes_file);
-		goto cleanup;
-	}
+	data = xmmap(NULL, mtimes_size, PROT_READ, MAP_PRIVATE, fd, 0);
 
-	data = hdr = xmmap(NULL, mtimes_size, PROT_READ, MAP_PRIVATE, fd, 0);
-
-	header.signature = ntohl(hdr[0]);
-	header.version = ntohl(hdr[1]);
-	header.hash_id = ntohl(hdr[2]);
+	header.signature = ntohl(data[0]);
+	header.version = ntohl(data[1]);
+	header.hash_id = ntohl(data[2]);
 
 	if (header.signature != MTIMES_SIGNATURE) {
 		ret = error(_("mtimes file %s has unknown signature"), mtimes_file);
@@ -77,13 +70,23 @@ static int load_pack_mtimes_file(char *mtimes_file,
 		goto cleanup;
 	}
 
+
+	expected_size = MTIMES_HEADER_SIZE;
+	expected_size = st_add(expected_size, st_mult(sizeof(uint32_t), num_objects));
+	expected_size = st_add(expected_size, 2 * (header.hash_id == 1 ? GIT_SHA1_RAWSZ : GIT_SHA256_RAWSZ));
+
+	if (mtimes_size != expected_size) {
+		ret = error(_("mtimes file %s is corrupt"), mtimes_file);
+		goto cleanup;
+	}
+
 cleanup:
 	if (ret) {
 		if (data)
 			munmap(data, mtimes_size);
 	} else {
 		*len_p = mtimes_size;
-		*data_p = (const uint32_t *)data;
+		*data_p = data;
 	}
 
 	close(fd);
pack-mtimes.h +11 -0
index 38ddb9f893..cc957b3e85 100644
@@ -8,8 +8,19 @@
 
 struct packed_git;
 
+/*
+ * Loads the .mtimes file corresponding to "p", if any, returning zero
+ * on success.
+ */
 int load_pack_mtimes(struct packed_git *p);
 
+/* Returns the mtime associated with the object at position "pos" (in
+ * lexicographic/index order) in pack "p".
+ *
+ * Note that it is a BUG() to call this function if either (a) "p" does
+ * not have a corresponding .mtimes file, or (b) it does, but it hasn't
+ * been loaded
+ */
 uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos);
 
 #endif

--
2.36.1.94.gb0d54bedca

--- >8 ---

#173 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

Quoted 49 lines · Taylor Blau
pack-mtimes.c +4 -10
index 46ad584af1..0e0aafdcb0 100644
@@ -1,3 +1,4 @@
+#include "git-compat-util.h"
 #include "pack-mtimes.h"
 #include "object-store.h"
 #include "packfile.h"
@@ -7,12 +8,10 @@ static char *pack_mtimes_filename(struct packed_git *p)
 	size_t len;
 	if (!strip_suffix(p->pack_name, ".pack", &len))
 		BUG("pack_name does not end in .pack");
-	/* NEEDSWORK: this could reuse code from pack-revindex.c. */
 	return xstrfmt("%.*s.mtimes", (int)len, p->pack_name);
 }
 
 #define MTIMES_HEADER_SIZE (12)
-#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 * the_hash_algo->rawsz))
 
 struct mtimes_header {
 	uint32_t signature;
@@ -26,10 +25,9 @@ static int load_pack_mtimes_file(char *mtimes_file,
 {
 	int fd, ret = 0;
 	struct stat st;
-	void *data = NULL;
-	size_t mtimes_size;
+	uint32_t *data = NULL;
+	size_t mtimes_size, expected_size;
 	struct mtimes_header header;
-	uint32_t *hdr;
 
 	fd = git_open(mtimes_file);
 
@@ -44,21 +42,16 @@ static int load_pack_mtimes_file(char *mtimes_file,
 
 	mtimes_size = xsize_t(st.st_size);
 
-	if (mtimes_size < MTIMES_MIN_SIZE) {
+	if (mtimes_size < MTIMES_HEADER_SIZE) {
 		ret = error(_("mtimes file %s is too small"), mtimes_file);
 		goto cleanup;
 	}
 
-	if (mtimes_size - MTIMES_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {
-		ret = error(_("mtimes file %s is corrupt"), mtimes_file);
-		goto cleanup;
-	}

pack-mtimes.c static int load_pack_mtimes_file(char *mtimes_file, · line 54

Quoted 10 lines
pack-mtimes.c +4 -5
+	data = xmmap(NULL, mtimes_size, PROT_READ, MAP_PRIVATE, fd, 0);
 
-	data = hdr = xmmap(NULL, mtimes_size, PROT_READ, MAP_PRIVATE, fd, 0);
-
-	header.signature = ntohl(hdr[0]);
-	header.version = ntohl(hdr[1]);
-	header.hash_id = ntohl(hdr[2]);
+	header.signature = ntohl(data[0]);
+	header.version = ntohl(data[1]);
+	header.hash_id = ntohl(data[2]);

So, instead of assuming that the size of the file is cast in stone to match the size the current implementation happens to give and reject a file from a future version, we check the header first to give a more readable error when we see a version of the file that we do not understand.

Makes sense.

At least, "here is a small fixup!" should have been accompanied by a brief explanation to say something like that, i.e. why a fixup is needed, what shortcoming in the original it is meant to address, etc.

Will queue between 2/17 and 3/17 without squashing (yet).

Thanks.

Quoted 55 lines
pack-mtimes.c +11 -1
 	if (header.signature != MTIMES_SIGNATURE) {
 		ret = error(_("mtimes file %s has unknown signature"), mtimes_file);
@@ -77,13 +70,23 @@ static int load_pack_mtimes_file(char *mtimes_file,
 		goto cleanup;
 	}
 
+
+	expected_size = MTIMES_HEADER_SIZE;
+	expected_size = st_add(expected_size, st_mult(sizeof(uint32_t), num_objects));
+	expected_size = st_add(expected_size, 2 * (header.hash_id == 1 ? GIT_SHA1_RAWSZ : GIT_SHA256_RAWSZ));
+
+	if (mtimes_size != expected_size) {
+		ret = error(_("mtimes file %s is corrupt"), mtimes_file);
+		goto cleanup;
+	}
+
 cleanup:
 	if (ret) {
 		if (data)
 			munmap(data, mtimes_size);
 	} else {
 		*len_p = mtimes_size;
-		*data_p = (const uint32_t *)data;
+		*data_p = data;
 	}
 
 	close(fd);
pack-mtimes.h +11 -1
index 38ddb9f893..cc957b3e85 100644
@@ -8,8 +8,19 @@
 
 struct packed_git;
 
+/*
+ * Loads the .mtimes file corresponding to "p", if any, returning zero
+ * on success.
+ */
 int load_pack_mtimes(struct packed_git *p);
 
+/* Returns the mtime associated with the object at position "pos" (in
+ * lexicographic/index order) in pack "p".
+ *
+ * Note that it is a BUG() to call this function if either (a) "p" does
+ * not have a corresponding .mtimes file, or (b) it does, but it hasn't
+ * been loaded
+ */
 uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos);
 
 #endif
--

2.36.1.94.gb0d54bedca

--- >8 ---

#174 lore raw

Re: [PATCH v5 02/17] pack-mtimes: support reading .mtimes files

In reply to Taylor Blau

Documentation/technical/pack-format.txt Pack file entry: <+ · line 301

Quoted 13 lines · Taylor Blau
Documentation/technical/pack-format.txt +5 -0
index 6d3efb7d16..b520aa9c45 100644
@@ -294,6 +294,25 @@ Pack file entry: <+
 
 All 4-byte numbers are in network order.
 
+== pack-*.mtimes files have the format:
+
+All 4-byte numbers are in network byte order.
+
+  - A 4-byte magic number '0x4d544d45' ('MTME').

This is identified by file(1) as "Multitracker Version 4.05". ;-)

Signature · 3 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."
#175 lore raw

PATCH v5 3/17 pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles'

In reply to Taylor Blau

This structure will be used to communicate the per-object mtimes when writing a cruft pack. Here, we need the full packing_data structure because the mtime information is stored in an array there, not on the individual object_entry's themselves (to avoid paying the overhead in structure width for operations which do not generate a cruft pack).

We haven't passed this information down before because one of the two callers (in bulk-checkin.c) does not have a packing_data structure at all. In that case (where no cruft pack will be generated), NULL is passed instead.

---
 builtin/pack-objects.c | 3 ++-
 bulk-checkin.c         | 2 +-
 pack-write.c           | 1 +
 pack.h                 | 3 +++
 4 files changed, 7 insertions(+), 2 deletions(-)
builtin/pack-objects.c +2 -1
index 014dcd4bc9..6ac927047c 100644
@@ -1262,7 +1262,8 @@ static void write_pack_file(void)
 
 			stage_tmp_packfiles(&tmpname, pack_tmp_name,
 					    written_list, nr_written,
-					    &pack_idx_opts, hash, &idx_tmp_name);
+					    &to_pack, &pack_idx_opts, hash,
+					    &idx_tmp_name);
 
 			if (write_bitmap_index) {
 				size_t tmpname_len = tmpname.len;
bulk-checkin.c +1 -1
index 6d6c37171c..e988a388b6 100644
@@ -33,7 +33,7 @@ static void finish_tmp_packfile(struct strbuf *basename,
 	char *idx_tmp_name = NULL;
 
 	stage_tmp_packfiles(basename, pack_tmp_name, written_list, nr_written,
-			    pack_idx_opts, hash, &idx_tmp_name);
+			    NULL, pack_idx_opts, hash, &idx_tmp_name);
 	rename_tmp_packfile_idx(basename, &idx_tmp_name);
 
 	free(idx_tmp_name);
pack-write.c +1 -0
index 51812cb129..a2adc565f4 100644
@@ -484,6 +484,7 @@ void stage_tmp_packfiles(struct strbuf *name_buffer,
 			 const char *pack_tmp_name,
 			 struct pack_idx_entry **written_list,
 			 uint32_t nr_written,
+			 struct packing_data *to_pack,
 			 struct pack_idx_option *pack_idx_opts,
 			 unsigned char hash[],
 			 char **idx_tmp_name)
pack.h +3 -0
index b22bfc4a18..fd27cfdfd7 100644
@@ -109,11 +109,14 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,
 #define PH_ERROR_PROTOCOL	(-3)
 int read_pack_header(int fd, struct pack_header *);
 
+struct packing_data;
+
 struct hashfile *create_tmp_packfile(char **pack_tmp_name);
 void stage_tmp_packfiles(struct strbuf *name_buffer,
 			 const char *pack_tmp_name,
 			 struct pack_idx_entry **written_list,
 			 uint32_t nr_written,
+			 struct packing_data *to_pack,
 			 struct pack_idx_option *pack_idx_opts,
 			 unsigned char hash[],
 			 char **idx_tmp_name);
Signature · 1 line
2.36.1.94.gb0d54bedca
#176 lore raw

PATCH v5 4/17 chunk-format.h: extract oid_version()

In reply to Taylor Blau

There are three definitions of an identical function which converts the_hash_algo into either 1 (for SHA-1) or 2 (for SHA-256). There is a copy of this function for writing both the commit-graph and multi-pack-index file, and another inline definition used to write the .rev header.

Consolidate these into a single definition in chunk-format.h. It's not clear that this is the best header to define this function in, but it should do for now.

(Worth noting, the .rev caller expects a 4-byte unsigned, but the other two callers work with a single unsigned byte. The consolidated version uses the latter type, and lets the compiler widen it when required).

Another caller will be added in a subsequent patch.

---
 chunk-format.c | 12 ++++++++++++
 chunk-format.h |  3 +++
 commit-graph.c | 18 +++---------------
 midx.c         | 18 +++---------------
 pack-write.c   | 15 ++-------------
 5 files changed, 23 insertions(+), 43 deletions(-)
chunk-format.c +12 -0
index 1c3dca62e2..0275b74a89 100644
@@ -181,3 +181,15 @@ int read_chunk(struct chunkfile *cf,
 
 	return CHUNK_NOT_FOUND;
 }
+
+uint8_t oid_version(const struct git_hash_algo *algop)
+{
+	switch (hash_algo_by_ptr(algop)) {
+	case GIT_HASH_SHA1:
+		return 1;
+	case GIT_HASH_SHA256:
+		return 2;
+	default:
+		die(_("invalid hash version"));
+	}
+}
chunk-format.h +3 -0
index 9ccbe00377..7885aa0848 100644
@@ -2,6 +2,7 @@
 #define CHUNK_FORMAT_H
 
 #include "git-compat-util.h"
+#include "hash.h"
 
 struct hashfile;
 struct chunkfile;
@@ -65,4 +66,6 @@ int read_chunk(struct chunkfile *cf,
 	       chunk_read_fn fn,
 	       void *data);
 
+uint8_t oid_version(const struct git_hash_algo *algop);
+
 #endif
commit-graph.c +3 -15
index 06107beedc..066d82ed6a 100644
@@ -193,18 +193,6 @@ char *get_commit_graph_chain_filename(struct object_directory *odb)
 	return xstrfmt("%s/info/commit-graphs/commit-graph-chain", odb->path);
 }
 
-static uint8_t oid_version(void)
-{
-	switch (hash_algo_by_ptr(the_hash_algo)) {
-	case GIT_HASH_SHA1:
-		return 1;
-	case GIT_HASH_SHA256:
-		return 2;
-	default:
-		die(_("invalid hash version"));
-	}
-}
-
 static struct commit_graph *alloc_commit_graph(void)
 {
 	struct commit_graph *g = xcalloc(1, sizeof(*g));
@@ -365,9 +353,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,
 	}
 
 	hash_version = *(unsigned char*)(data + 5);
-	if (hash_version != oid_version()) {
+	if (hash_version != oid_version(the_hash_algo)) {
 		error(_("commit-graph hash version %X does not match version %X"),
-		      hash_version, oid_version());
+		      hash_version, oid_version(the_hash_algo));
 		return NULL;
 	}
 
@@ -1924,7 +1912,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
 	hashwrite_be32(f, GRAPH_SIGNATURE);
 
 	hashwrite_u8(f, GRAPH_VERSION);
-	hashwrite_u8(f, oid_version());
+	hashwrite_u8(f, oid_version(the_hash_algo));
 	hashwrite_u8(f, get_num_chunks(cf));
 	hashwrite_u8(f, ctx->num_commit_graphs_after - 1);
 
midx.c +3 -15
index 3db0e47735..c617c51cd0 100644
@@ -41,18 +41,6 @@
 
 #define PACK_EXPIRED UINT_MAX
 
-static uint8_t oid_version(void)
-{
-	switch (hash_algo_by_ptr(the_hash_algo)) {
-	case GIT_HASH_SHA1:
-		return 1;
-	case GIT_HASH_SHA256:
-		return 2;
-	default:
-		die(_("invalid hash version"));
-	}
-}
-
 const unsigned char *get_midx_checksum(struct multi_pack_index *m)
 {
 	return m->data + m->data_len - the_hash_algo->rawsz;
@@ -134,9 +122,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local
 		      m->version);
 
 	hash_version = m->data[MIDX_BYTE_HASH_VERSION];
-	if (hash_version != oid_version()) {
+	if (hash_version != oid_version(the_hash_algo)) {
 		error(_("multi-pack-index hash version %u does not match version %u"),
-		      hash_version, oid_version());
+		      hash_version, oid_version(the_hash_algo));
 		goto cleanup_fail;
 	}
 	m->hash_len = the_hash_algo->rawsz;
@@ -420,7 +408,7 @@ static size_t write_midx_header(struct hashfile *f,
 {
 	hashwrite_be32(f, MIDX_SIGNATURE);
 	hashwrite_u8(f, MIDX_VERSION);
-	hashwrite_u8(f, oid_version());
+	hashwrite_u8(f, oid_version(the_hash_algo));
 	hashwrite_u8(f, num_chunks);
 	hashwrite_u8(f, 0); /* unused */
 	hashwrite_be32(f, num_packs);
pack-write.c +2 -13
index a2adc565f4..27b171e440 100644
@@ -2,6 +2,7 @@
 #include "pack.h"
 #include "csum-file.h"
 #include "remote.h"
+#include "chunk-format.h"
 
 void reset_pack_idx_option(struct pack_idx_option *opts)
 {
@@ -181,21 +182,9 @@ static int pack_order_cmp(const void *va, const void *vb, void *ctx)
 
 static void write_rev_header(struct hashfile *f)
 {
-	uint32_t oid_version;
-	switch (hash_algo_by_ptr(the_hash_algo)) {
-	case GIT_HASH_SHA1:
-		oid_version = 1;
-		break;
-	case GIT_HASH_SHA256:
-		oid_version = 2;
-		break;
-	default:
-		die("write_rev_header: unknown hash version");
-	}
-
 	hashwrite_be32(f, RIDX_SIGNATURE);
 	hashwrite_be32(f, RIDX_VERSION);
-	hashwrite_be32(f, oid_version);
+	hashwrite_be32(f, oid_version(the_hash_algo));
 }
 
 static void write_rev_index_positions(struct hashfile *f,
Signature · 1 line
2.36.1.94.gb0d54bedca
#177 lore raw

PATCH v5 5/17 pack-mtimes: support writing pack .mtimes files

In reply to Taylor Blau

Now that the .mtimes format is defined, supplement the pack-write API to be able to conditionally write an .mtimes file along with a pack by setting an additional flag and passing an oidmap that contains the timestamps corresponding to each object in the pack.

---
 pack-objects.c |  6 ++++
 pack-objects.h | 25 ++++++++++++++++
 pack-write.c   | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++
 pack.h         |  1 +
 4 files changed, 109 insertions(+)
pack-objects.c +6 -0
index fe2a4eace9..272e8d4517 100644
@@ -170,6 +170,9 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,
 
 		if (pdata->layer)
 			REALLOC_ARRAY(pdata->layer, pdata->nr_alloc);
+
+		if (pdata->cruft_mtime)
+			REALLOC_ARRAY(pdata->cruft_mtime, pdata->nr_alloc);
 	}
 
 	new_entry = pdata->objects + pdata->nr_objects++;
@@ -198,6 +201,9 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,
 	if (pdata->layer)
 		pdata->layer[pdata->nr_objects - 1] = 0;
 
+	if (pdata->cruft_mtime)
+		pdata->cruft_mtime[pdata->nr_objects - 1] = 0;
+
 	return new_entry;
 }
 
pack-objects.h +25 -0
index dca2351ef9..393b9db546 100644
@@ -168,6 +168,14 @@ struct packing_data {
 	/* delta islands */
 	unsigned int *tree_depth;
 	unsigned char *layer;
+
+	/*
+	 * Used when writing cruft packs.
+	 *
+	 * Object mtimes are stored in pack order when writing, but
+	 * written out in lexicographic (index) order.
+	 */
+	uint32_t *cruft_mtime;
 };
 
 void prepare_packing_data(struct repository *r, struct packing_data *pdata);
@@ -289,4 +297,21 @@ static inline void oe_set_layer(struct packing_data *pack,
 	pack->layer[e - pack->objects] = layer;
 }
 
+static inline uint32_t oe_cruft_mtime(struct packing_data *pack,
+				      struct object_entry *e)
+{
+	if (!pack->cruft_mtime)
+		return 0;
+	return pack->cruft_mtime[e - pack->objects];
+}
+
+static inline void oe_set_cruft_mtime(struct packing_data *pack,
+				      struct object_entry *e,
+				      uint32_t mtime)
+{
+	if (!pack->cruft_mtime)
+		CALLOC_ARRAY(pack->cruft_mtime, pack->nr_alloc);
+	pack->cruft_mtime[e - pack->objects] = mtime;
+}
+
 #endif
pack-write.c +77 -0

104 lines not on this page · raw message

pack.h +1 -0
index fd27cfdfd7..01d385903a 100644
@@ -44,6 +44,7 @@ struct pack_idx_option {
 #define WRITE_IDX_STRICT 02
 #define WRITE_REV 04
 #define WRITE_REV_VERIFY 010
+#define WRITE_MTIMES 020
 
 	uint32_t version;
 	uint32_t off32_limit;
Signature · 1 line
2.36.1.94.gb0d54bedca
#178 lore raw

PATCH v5 6/17 t/helper: add 'pack-mtimes' test-tool

In reply to Taylor Blau

In the next patch, we will implement and test support for writing a cruft pack via a special mode of git pack-objects. To make sure that objects are written with the correct timestamps, and a new test-tool that can dump the object names and corresponding timestamps from a given .mtimes file.

---
 Makefile                    |  1 +
 t/helper/test-pack-mtimes.c | 56 +++++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c        |  1 +
 t/helper/test-tool.h        |  1 +
 4 files changed, 59 insertions(+)
 create mode 100644 t/helper/test-pack-mtimes.c
Makefile +1 -0
index a299580b7c..0b6eab0453 100644
@@ -738,6 +738,7 @@ TEST_BUILTINS_OBJS += test-oid-array.o
 TEST_BUILTINS_OBJS += test-oidmap.o
 TEST_BUILTINS_OBJS += test-oidtree.o
 TEST_BUILTINS_OBJS += test-online-cpus.o
+TEST_BUILTINS_OBJS += test-pack-mtimes.o
 TEST_BUILTINS_OBJS += test-parse-options.o
 TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
 TEST_BUILTINS_OBJS += test-partial-clone.o
t/helper/test-pack-mtimes.c new file +56 -0

56 lines not on this page · raw message

t/helper/test-tool.c +1 -0
index 0424f7adf5..d2eacd302d 100644
@@ -48,6 +48,7 @@ static struct test_cmd cmds[] = {
 	{ "oidmap", cmd__oidmap },
 	{ "oidtree", cmd__oidtree },
 	{ "online-cpus", cmd__online_cpus },
+	{ "pack-mtimes", cmd__pack_mtimes },
 	{ "parse-options", cmd__parse_options },
 	{ "parse-pathspec-file", cmd__parse_pathspec_file },
 	{ "partial-clone", cmd__partial_clone },
t/helper/test-tool.h +1 -0
index c876e8246f..960cc27ef7 100644
@@ -38,6 +38,7 @@ int cmd__mktemp(int argc, const char **argv);
 int cmd__oidmap(int argc, const char **argv);
 int cmd__oidtree(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
+int cmd__pack_mtimes(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
 int cmd__partial_clone(int argc, const char **argv);
Signature · 1 line
2.36.1.94.gb0d54bedca
#179 lore raw

PATCH v5 7/17 builtin/pack-objects.c: return from create_object_entry()

In reply to Taylor Blau

A new caller in the next commit will want to immediately modify the object_entry structure created by create_object_entry(). Instead of forcing that caller to wastefully look-up the entry we just created, return it from create_object_entry() instead.

---
 builtin/pack-objects.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
builtin/pack-objects.c +9 -7
index 6ac927047c..c6d16872ee 100644
@@ -1516,13 +1516,13 @@ static int want_object_in_pack(const struct object_id *oid,
 	return 1;
 }
 
-static void create_object_entry(const struct object_id *oid,
-				enum object_type type,
-				uint32_t hash,
-				int exclude,
-				int no_try_delta,
-				struct packed_git *found_pack,
-				off_t found_offset)
+static struct object_entry *create_object_entry(const struct object_id *oid,
+						enum object_type type,
+						uint32_t hash,
+						int exclude,
+						int no_try_delta,
+						struct packed_git *found_pack,
+						off_t found_offset)
 {
 	struct object_entry *entry;
 
@@ -1539,6 +1539,8 @@ static void create_object_entry(const struct object_id *oid,
 	}
 
 	entry->no_try_delta = no_try_delta;
+
+	return entry;
 }
 
 static const char no_closure_warning[] = N_(
Signature · 1 line
2.36.1.94.gb0d54bedca
#180 lore raw

PATCH v5 8/17 builtin/pack-objects.c: --cruft without expiration

In reply to Taylor Blau

Teach pack-objects how to generate a cruft pack when no objects are dropped (i.e., --cruft-expiration=never). Later patches will teach pack-objects how to generate a cruft pack that prunes objects.

When generating a cruft pack which does not prune objects, we want to collect all unreachable objects into a single pack (noting and updating their mtimes as we accumulate them). Ordinary use will pass the result of a git repack -A as a kept pack, so when this patch says "kept pack", readers should think "reachable objects".

Generating a non-expiring cruft packs works as follows:

- Callers provide a list of every pack they know about, and indicate which packs are about to be removed.

- All packs which are going to be removed (we'll call these the redundant ones) are marked as kept in-core.

Any packs the caller did not mention (but are known to the pack-objects process) are also marked as kept in-core. Packs not mentioned by the caller are assumed to be unknown to them, i.e., they entered the repository after the caller decided which packs should be kept and which should be discarded.

Since we do not want to include objects in these "unknown" packs (because we don't know which of their objects are or aren't reachable), these are also marked as kept in-core.

- Then, we enumerate all objects in the repository, and add them to our packing list if they do not appear in an in-core kept pack.

This results in a new cruft pack which contains all known objects that aren't included in the kept packs. When the kept pack is the result of git repack -A, the resulting pack contains all unreachable objects.

---
 Documentation/git-pack-objects.txt |  30 ++++
 builtin/pack-objects.c             | 201 +++++++++++++++++++++++++-
 object-file.c                      |   2 +-
 object-store.h                     |   2 +
 t/t5329-pack-objects-cruft.sh      | 218 +++++++++++++++++++++++++++++
 5 files changed, 448 insertions(+), 5 deletions(-)
 create mode 100755 t/t5329-pack-objects-cruft.sh
Documentation/git-pack-objects.txt +30 -0
index f8344e1e5b..a9995a932c 100644
@@ -13,6 +13,7 @@ SYNOPSIS
 	[--no-reuse-delta] [--delta-base-offset] [--non-empty]
 	[--local] [--incremental] [--window=<n>] [--depth=<n>]
 	[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
+	[--cruft] [--cruft-expiration=<time>]
 	[--stdout [--filter=<filter-spec>] | <base-name>]
 	[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>
 
@@ -95,6 +96,35 @@ base-name::
 Incompatible with `--revs`, or options that imply `--revs` (such as
 `--all`), with the exception of `--unpacked`, which is compatible.
 
+--cruft::
+	Packs unreachable objects into a separate "cruft" pack, denoted
+	by the existence of a `.mtimes` file. Typically used by `git
+	repack --cruft`. Callers provide a list of pack names and
+	indicate which packs will remain in the repository, along with
+	which packs will be deleted (indicated by the `-` prefix). The
+	contents of the cruft pack are all objects not contained in the
+	surviving packs which have not exceeded the grace period (see
+	`--cruft-expiration` below), or which have exceeded the grace
+	period, but are reachable from an other object which hasn't.
++
+When the input lists a pack containing all reachable objects (and lists
+all other packs as pending deletion), the corresponding cruft pack will
+contain all unreachable objects (with mtime newer than the
+`--cruft-expiration`) along with any unreachable objects whose mtime is
+older than the `--cruft-expiration`, but are reachable from an
+unreachable object whose mtime is newer than the `--cruft-expiration`).
++
+Incompatible with `--unpack-unreachable`, `--keep-unreachable`,
+`--pack-loose-unreachable`, `--stdin-packs`, as well as any other
+options which imply `--revs`. Also incompatible with `--max-pack-size`;
+when this option is set, the maximum pack size is not inferred from
+`pack.packSizeLimit`.
+
+--cruft-expiration=<approxidate>::
+	If specified, objects are eliminated from the cruft pack if they
+	have an mtime older than `<approxidate>`. If unspecified (and
+	given `--cruft`), then no objects are eliminated.
+
 --window=<n>::
 --depth=<n>::
 	These two options affect how the objects contained in
builtin/pack-objects.c +197 -4

273 lines not on this page · raw message

object-file.c +1 -1
index 5ffbf3d4fd..ff0cffe68e 100644
@@ -997,7 +997,7 @@ int has_loose_object_nonlocal(const struct object_id *oid)
 	return check_and_freshen_nonlocal(oid, 0);
 }
 
-static int has_loose_object(const struct object_id *oid)
+int has_loose_object(const struct object_id *oid)
 {
 	return check_and_freshen(oid, 0);
 }
object-store.h +2 -0
index 2c4671ed7a..c41609e8db 100644
@@ -330,6 +330,8 @@ int repo_has_object_file_with_flags(struct repository *r,
  */
 int has_loose_object_nonlocal(const struct object_id *);
 
+int has_loose_object(const struct object_id *);
+
 /**
  * format_object_header() is a thin wrapper around s xsnprintf() that
  * writes the initial "<type> <obj-len>" part of the loose object
t/t5329-pack-objects-cruft.sh new file +218 -0

218 lines not on this page · raw message

Signature · 1 line
2.36.1.94.gb0d54bedca
#181 lore raw

PATCH v5 9/17 reachable: add options to add_unseen_recent_objects_to_traversal

In reply to Taylor Blau

This function behaves very similarly to what we will need in pack-objects in order to implement cruft packs with expiration. But it is lacking a couple of things. Namely, it needs:

- a mechanism to communicate the timestamps of individual recent objects to some external caller

- and, in the case of packed objects, our future caller will also want to know the originating pack, as well as the offset within that pack at which the object can be found

- finally, it needs a way to skip over packs which are marked as kept in-core.

To address the first two, add a callback interface in this patch which reports the time of each recent object, as well as a (packed_git, off_t) pair for packed objects.

Likewise, add a new option to the packed object iterators to skip over packs which are marked as kept in core. This option will become implicitly tested in a future patch.

---
 builtin/pack-objects.c |  2 +-
 reachable.c            | 51 +++++++++++++++++++++++++++++++++++-------
 reachable.h            |  9 +++++++-
 3 files changed, 52 insertions(+), 10 deletions(-)
builtin/pack-objects.c +1 -1
index 9cf89be673..3b8bf6a3dd 100644
@@ -3957,7 +3957,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)
 	if (unpack_unreachable_expiration) {
 		revs->ignore_missing_links = 1;
 		if (add_unseen_recent_objects_to_traversal(revs,
-				unpack_unreachable_expiration))
+				unpack_unreachable_expiration, NULL, 0))
 			die(_("unable to add recent objects"));
 		if (prepare_revision_walk(revs))
 			die(_("revision walk setup failed"));
reachable.c +43 -8

108 lines not on this page · raw message

reachable.h +8 -1
index 5df932ad8f..b776761baa 100644
@@ -1,11 +1,18 @@
 #ifndef REACHEABLE_H
 #define REACHEABLE_H
 
+#include "object.h"
+
 struct progress;
 struct rev_info;
 
+typedef void report_recent_object_fn(const struct object *, struct packed_git *,
+				     off_t, time_t);
+
 int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
-					   timestamp_t timestamp);
+					   timestamp_t timestamp,
+					   report_recent_object_fn cb,
+					   int ignore_in_core_kept_packs);
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    timestamp_t mark_recent, struct progress *);
 
Signature · 1 line
2.36.1.94.gb0d54bedca
#182 lore raw

PATCH v5 10/17 reachable: report precise timestamps from objects in cruft packs

In reply to Taylor Blau

When generating a cruft pack, the caller within pack-objects will want to know the precise timestamps of cruft objects (i.e., their corresponding values in the .mtimes table) rather than the mtime of the cruft pack itself.

Teach add_recent_packed() to lookup each object's precise mtime from the .mtimes file if one exists (indicated by the is_cruft bit on the packed_git structure).

A couple of small things worth noting here:

- load_pack_mtimes() needs to be called before asking for nth_packed_mtime(), and that call is done lazily here. That function exits early if the .mtimes file has already been opened and parsed, so only the first call is slow.

- Checking the is_cruft bit can be done without any extra work on the caller's behalf, since it is set up for us automatically as a side-effect of calling add_packed_git() (just like the 'pack_keep' and 'pack_promisor' bits).

---
 reachable.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
reachable.c +8 -1
index d4507c4270..aba63ebeb3 100644
@@ -13,6 +13,7 @@
 #include "worktree.h"
 #include "object-store.h"
 #include "pack-bitmap.h"
+#include "pack-mtimes.h"
 
 struct connectivity_progress {
 	struct progress *progress;
@@ -155,6 +156,7 @@ static int add_recent_packed(const struct object_id *oid,
 			     void *data)
 {
 	struct object *obj;
+	timestamp_t mtime = p->mtime;
 
 	if (!want_recent_object(data, oid))
 		return 0;
@@ -163,7 +165,12 @@ static int add_recent_packed(const struct object_id *oid,
 
 	if (obj && obj->flags & SEEN)
 		return 0;
-	add_recent_object(oid, p, nth_packed_object_offset(p, pos), p->mtime, data);
+	if (p->is_cruft) {
+		if (load_pack_mtimes(p) < 0)
+			die(_("could not load cruft pack .mtimes"));
+		mtime = nth_packed_mtime(p, pos);
+	}
+	add_recent_object(oid, p, nth_packed_object_offset(p, pos), mtime, data);
 	return 0;
 }
 
Signature · 1 line
2.36.1.94.gb0d54bedca
#183 lore raw

PATCH v5 11/17 builtin/pack-objects.c: --cruft with expiration

In reply to Taylor Blau

In a previous patch, pack-objects learned how to generate a cruft pack so long as no objects are dropped.

This patch teaches pack-objects to handle the case where a non-never --cruft-expiration value is passed. This case is slightly more complicated than before, because we want pack-objects to save unreachable objects which would have been pruned when there is another recent (i.e., non-prunable) unreachable object which reaches the other. We'll call these objects "unreachable but reachable-from-recent".

Here is how pack-objects handles --cruft-expiration:

- Instead of adding all objects outside of the kept pack(s) into the packing list, only handle the ones whose mtime is within the grace period.

- Construct a reachability traversal whose tips are the unreachable-but-recent objects.

- Then, walk along that traversal, stopping if we reach an object in the kept pack. At each step along the traversal, we add the object we are visiting to the packing list.

In the majority of these cases, any object we visit in this traversal will already be in our packing list. But we will sometimes encounter reachable-from-recent cruft objects, which we want to retain even if they aged out of the grace period.

The most subtle point of this process is that we actually don't need to bother to update the rescued object's mtime. Even though we will write an .mtimes file with a value that is older than the expiration window, it will continue to survive cruft repacks so long as any objects which reach it haven't aged out.

That is, a future repack will also exclude that object from the initial packing list, only to discover it later on when doing the reachability traversal.

Finally, stopping early once an object is found in a kept pack is safe to do because the kept packs ordinarily represent which packs will survive after repacking. Assuming that it _isn't_ safe to halt a traversal early would mean that there is some ancestor object which is missing, which implies repository corruption (i.e., the complete set of reachable objects isn't present).

---
 builtin/pack-objects.c        |  84 +++++++++++++++++++-
 reachable.h                   |   4 +-
 t/t5329-pack-objects-cruft.sh | 143 ++++++++++++++++++++++++++++++++++
 3 files changed, 228 insertions(+), 3 deletions(-)
builtin/pack-objects.c +83 -1
index 3b8bf6a3dd..8decc9dc0c 100644
@@ -3447,6 +3447,44 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type
 	return;
 }
 
+static void show_cruft_object(struct object *obj, const char *name, void *data)
+{
+	/*
+	 * if we did not record it earlier, it's at least as old as our
+	 * expiration value. Rather than find it exactly, just use that
+	 * value.  This may bump it forward from its real mtime, but it
+	 * will still be "too old" next time we run with the same
+	 * expiration.
+	 *
+	 * if obj does appear in the packing list, this call is a noop (or may
+	 * set the namehash).
+	 */
+	add_cruft_object_entry(&obj->oid, obj->type, NULL, 0, name, cruft_expiration);
+}
+
+static void show_cruft_commit(struct commit *commit, void *data)
+{
+	show_cruft_object((struct object*)commit, NULL, data);
+}
+
+static int cruft_include_check_obj(struct object *obj, void *data)
+{
+	return !has_object_kept_pack(&obj->oid, IN_CORE_KEEP_PACKS);
+}
+
+static int cruft_include_check(struct commit *commit, void *data)
+{
+	return cruft_include_check_obj((struct object*)commit, data);
+}
+
+static void set_cruft_mtime(const struct object *object,
+			    struct packed_git *pack,
+			    off_t offset, time_t mtime)
+{
+	add_cruft_object_entry(&object->oid, object->type, pack, offset, NULL,
+			       mtime);
+}
+
 static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)
 {
 	struct string_list_item *item = NULL;
@@ -3472,6 +3510,50 @@ static void enumerate_cruft_objects(void)
 	stop_progress(&progress_state);
 }
 
+static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs)
+{
+	struct packed_git *p;
+	struct rev_info revs;
+	int ret;
+
+	repo_init_revisions(the_repository, &revs, NULL);
+
+	revs.tag_objects = 1;
+	revs.tree_objects = 1;
+	revs.blob_objects = 1;
+
+	revs.include_check = cruft_include_check;
+	revs.include_check_obj = cruft_include_check_obj;
+
+	revs.ignore_missing_links = 1;
+
+	if (progress)
+		progress_state = start_progress(_("Enumerating cruft objects"), 0);
+	ret = add_unseen_recent_objects_to_traversal(&revs, cruft_expiration,
+						     set_cruft_mtime, 1);
+	stop_progress(&progress_state);
+
+	if (ret)
+		die(_("unable to add cruft objects"));
+
+	/*
+	 * Re-mark only the fresh packs as kept so that objects in
+	 * unknown packs do not halt the reachability traversal early.
+	 */
+	for (p = get_all_packs(the_repository); p; p = p->next)
+		p->pack_keep_in_core = 0;
+	mark_pack_kept_in_core(fresh_packs, 1);
+
+	if (prepare_revision_walk(&revs))
+		die(_("revision walk setup failed"));
+	if (progress)
+		progress_state = start_progress(_("Traversing cruft objects"), 0);
+	nr_seen = 0;
+	traverse_commit_list(&revs, show_cruft_commit, show_cruft_object, NULL);
+
+	stop_progress(&progress_state);
+}
+
 static void read_cruft_objects(void)
 {
 	struct strbuf buf = STRBUF_INIT;
@@ -3523,7 +3605,7 @@ static void read_cruft_objects(void)
 	mark_pack_kept_in_core(&discard_packs, 0);
 
 	if (cruft_expiration)
-		die("--cruft-expiration not yet implemented");
+		enumerate_and_traverse_cruft_objects(&fresh_packs);
 	else
 		enumerate_cruft_objects();
 
reachable.h +2 -2
index b776761baa..020a887b99 100644
@@ -1,10 +1,10 @@
 #ifndef REACHEABLE_H
 #define REACHEABLE_H
 
-#include "object.h"
-
 struct progress;
 struct rev_info;
+struct object;
+struct packed_git;
 
 typedef void report_recent_object_fn(const struct object *, struct packed_git *,
 				     off_t, time_t);
t/t5329-pack-objects-cruft.sh +143 -0

148 lines not on this page · raw message

Signature · 1 line
2.36.1.94.gb0d54bedca
#184 lore raw

PATCH v5 12/17 builtin/repack.c: support generating a cruft pack

In reply to Taylor Blau

Expose a way to split the contents of a repository into a main and cruft pack when doing an all-into-one repack with git repack --cruft -d, and a complementary configuration variable.

---
 Documentation/git-repack.txt            |  11 ++
 Documentation/technical/cruft-packs.txt |   2 +-
 builtin/repack.c                        | 105 +++++++++++-
 t/t5329-pack-objects-cruft.sh           | 207 ++++++++++++++++++++++++
 4 files changed, 319 insertions(+), 6 deletions(-)
Documentation/git-repack.txt +11 -0
index ee30edc178..0bf13893d8 100644
@@ -63,6 +63,17 @@ to the new separate pack will be written.
 	Also run  'git prune-packed' to remove redundant
 	loose object files.
 
+--cruft::
+	Same as `-a`, unless `-d` is used. Then any unreachable objects
+	are packed into a separate cruft pack. Unreachable objects can
+	be pruned using the normal expiry rules with the next `git gc`
+	invocation (see linkgit:git-gc[1]). Incompatible with `-k`.
+
+--cruft-expiration=<approxidate>::
+	Expire unreachable objects older than `<approxidate>`
+	immediately instead of waiting for the next `git gc` invocation.
+	Only useful with `--cruft -d`.
+
 -l::
 	Pass the `--local` option to 'git pack-objects'. See
 	linkgit:git-pack-objects[1].
Documentation/technical/cruft-packs.txt +1 -1
index c0f583cd48..d81f3a8982 100644
@@ -17,7 +17,7 @@ pruned according to normal expiry rules with the next 'git gc' invocation.
 
 Unreachable objects aren't removed immediately, since doing so could race with
 an incoming push which may reference an object which is about to be deleted.
-Instead, those unreachable objects are stored as loose object and stay that way
+Instead, those unreachable objects are stored as loose objects and stay that way
 until they are older than the expiration window, at which point they are removed
 by linkgit:git-prune[1].
 
builtin/repack.c +100 -5

165 lines not on this page · raw message

t/t5329-pack-objects-cruft.sh +207 -0

211 lines not on this page · raw message

Signature · 1 line
2.36.1.94.gb0d54bedca
#185 lore raw

PATCH v5 13/17 builtin/repack.c: allow configuring cruft pack generation

In reply to Taylor Blau

In servers which set the pack.window configuration to a large value, we can wind up spending quite a lot of time finding new bases when breaking delta chains between reachable and unreachable objects while generating a cruft pack.

Introduce a handful of repack.cruft* configuration variables to control the parameters used by pack-objects when generating a cruft pack.

---
 Documentation/config/repack.txt |  9 ++++
 builtin/repack.c                | 49 +++++++++++++------
 t/t5329-pack-objects-cruft.sh   | 83 +++++++++++++++++++++++++++++++++
 3 files changed, 127 insertions(+), 14 deletions(-)
Documentation/config/repack.txt +9 -0
index 41ac6953c8..c79af6d7b8 100644
@@ -30,3 +30,12 @@ repack.updateServerInfo::
 	If set to false, linkgit:git-repack[1] will not run
 	linkgit:git-update-server-info[1]. Defaults to true. Can be overridden
 	when true by the `-n` option of linkgit:git-repack[1].
+
+repack.cruftWindow::
+repack.cruftWindowMemory::
+repack.cruftDepth::
+repack.cruftThreads::
+	Parameters used by linkgit:git-pack-objects[1] when generating
+	a cruft pack and the respective parameters are not given over
+	the command line. See similarly named `pack.*` configuration
+	variables for defaults and meaning.
builtin/repack.c +35 -14

88 lines not on this page · raw message

t/t5329-pack-objects-cruft.sh +83 -0

87 lines not on this page · raw message

Signature · 1 line
2.36.1.94.gb0d54bedca
#186 lore raw

PATCH v5 14/17 builtin/repack.c: use named flags for existing_packs

In reply to Taylor Blau

We use the util pointer for items in the existing_packs string list to indicate which packs are going to be deleted. Since that has so far been the only use of that util pointer, we just set it to 0 or 1.

But we're going to add an additional state to this field in the next patch, so prepare for that by adding a #define for the first bit so we can more expressively inspect the flags state.

---
 builtin/repack.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
builtin/repack.c +6 -3
index b85483a148..36d1f03671 100644
@@ -22,6 +22,8 @@
 #define LOOSEN_UNREACHABLE 2
 #define PACK_CRUFT 4
 
+#define DELETE_PACK 1
+
 static int pack_everything;
 static int delta_base_offset = 1;
 static int pack_kept_objects = -1;
@@ -564,7 +566,7 @@ static void midx_included_packs(struct string_list *include,
 		}
 	} else {
 		for_each_string_list_item(item, existing_nonkept_packs) {
-			if (item->util)
+			if ((uintptr_t)item->util & DELETE_PACK)
 				continue;
 			string_list_insert(include, xstrfmt("%s.idx", item->string));
 		}
@@ -1002,7 +1004,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 			 * was given) and that we will actually delete this pack
 			 * (if `-d` was given).
 			 */
-			item->util = (void*)(intptr_t)!string_list_has_string(&names, sha1);
+			if (!string_list_has_string(&names, sha1))
+				item->util = (void*)(uintptr_t)((size_t)item->util | DELETE_PACK);
 		}
 	}
 
@@ -1026,7 +1029,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 	if (delete_redundant) {
 		int opts = 0;
 		for_each_string_list_item(item, &existing_nonkept_packs) {
-			if (!item->util)
+			if (!((uintptr_t)item->util & DELETE_PACK))
 				continue;
 			remove_redundant_pack(packdir, item->string);
 		}
Signature · 1 line
2.36.1.94.gb0d54bedca
#187 lore raw

PATCH v5 15/17 builtin/repack.c: add cruft packs to MIDX during geometric repack

In reply to Taylor Blau

When using cruft packs, the following race can occur when a geometric repack that writes a MIDX bitmap takes place afterwords:

- First, create an unreachable object and do an all-into-one cruft repack which stores that object in the repository's cruft pack.
- Then make that object reachable.
- Finally, do a geometric repack and write a MIDX bitmap.

Assuming that we are sufficiently unlucky as to select a commit from the MIDX which reaches that object for bitmapping, then the git multi-pack-index process will complain that that object is missing.

The reason is because we don't include cruft packs in the MIDX when doing a geometric repack. Since the "make that object reachable" doesn't necessarily mean that we'll create a new copy of that object in one of the packs that will get rolled up as part of a geometric repack, it's possible that the MIDX won't see any copies of that now-reachable object.

Of course, it's desirable to avoid including cruft packs in the MIDX because it causes the MIDX to store a bunch of objects which are likely to get thrown away. But excluding that pack does open us up to the above race.

This patch demonstrates the bug, and resolves it by including cruft packs in the MIDX even when doing a geometric repack.

---
 builtin/repack.c              | 23 ++++++++++++++++++++---
 t/t5329-pack-objects-cruft.sh | 26 ++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 3 deletions(-)
builtin/repack.c +20 -3
index 36d1f03671..15071fadbe 100644
@@ -23,6 +23,7 @@
 #define PACK_CRUFT 4
 
 #define DELETE_PACK 1
+#define CRUFT_PACK 2
 
 static int pack_everything;
 static int delta_base_offset = 1;
@@ -159,10 +160,15 @@ static void collect_pack_filenames(struct string_list *fname_nonkept_list,
 		fname = xmemdupz(e->d_name, len);
 
 		if ((extra_keep->nr > 0 && i < extra_keep->nr) ||
-		    (file_exists(mkpath("%s/%s.keep", packdir, fname))))
+		    (file_exists(mkpath("%s/%s.keep", packdir, fname)))) {
 			string_list_append_nodup(fname_kept_list, fname);
-		else
-			string_list_append_nodup(fname_nonkept_list, fname);
+		} else {
+			struct string_list_item *item;
+			item = string_list_append_nodup(fname_nonkept_list,
+							fname);
+			if (file_exists(mkpath("%s/%s.mtimes", packdir, fname)))
+				item->util = (void*)(uintptr_t)CRUFT_PACK;
+		}
 	}
 	closedir(dir);
 }
@@ -564,6 +570,17 @@ static void midx_included_packs(struct string_list *include,
 
 			string_list_insert(include, strbuf_detach(&buf, NULL));
 		}
+
+		for_each_string_list_item(item, existing_nonkept_packs) {
+			if (!((uintptr_t)item->util & CRUFT_PACK)) {
+				/*
+				 * no need to check DELETE_PACK, since we're not
+				 * doing an ALL_INTO_ONE repack
+				 */
+				continue;
+			}
+			string_list_insert(include, xstrfmt("%s.idx", item->string));
+		}
 	} else {
 		for_each_string_list_item(item, existing_nonkept_packs) {
 			if ((uintptr_t)item->util & DELETE_PACK)
t/t5329-pack-objects-cruft.sh +26 -0
index c82f973b41..8de87afce2 100755
@@ -648,4 +648,30 @@ test_expect_success 'cruft --local drops unreachable objects' '
 	)
 '
 
+test_expect_success 'MIDX bitmaps tolerate reachable cruft objects' '
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit reachable &&
+		test_commit cruft &&
+		unreachable="$(git rev-parse cruft)" &&
+
+		git reset --hard $unreachable^ &&
+		git tag -d cruft &&
+		git reflog expire --all --expire=all &&
+
+		git repack --cruft -d &&
+
+		# resurrect the unreachable object via a new commit. the
+		# new commit will get selected for a bitmap, but be
+		# missing one of its parents from the selected packs.
+		git reset --hard $unreachable &&
+		test_commit resurrect &&
+
+		git repack --write-midx --write-bitmap-index --geometric=2 -d
+	)
+'
+
 test_done
Signature · 1 line
2.36.1.94.gb0d54bedca
#188 lore raw

PATCH v5 16/17 builtin/gc.c: conditionally avoid pruning objects via loose

In reply to Taylor Blau

Expose the new git repack --cruft mode from git gc via a new opt-in flag. When invoked like git gc --cruft, git gc will avoid exploding unreachable objects as loose ones, and instead create a cruft pack and .mtimes file.

---
 Documentation/config/gc.txt   | 21 +++++++++++++-------
 Documentation/git-gc.txt      |  5 +++++
 builtin/gc.c                  | 10 +++++++++-
 t/t5329-pack-objects-cruft.sh | 37 +++++++++++++++++++++++++++++++++++
 4 files changed, 65 insertions(+), 8 deletions(-)
Documentation/config/gc.txt +14 -7
index c834e07991..38fea076a2 100644
@@ -81,14 +81,21 @@ gc.packRefs::
 	to enable it within all non-bare repos or it can be set to a
 	boolean value.  The default is `true`.
 
+gc.cruftPacks::
+	Store unreachable objects in a cruft pack (see
+	linkgit:git-repack[1]) instead of as loose objects. The default
+	is `false`.
+
 gc.pruneExpire::
-	When 'git gc' is run, it will call 'prune --expire 2.weeks.ago'.
-	Override the grace period with this config variable.  The value
-	"now" may be used to disable this grace period and always prune
-	unreachable objects immediately, or "never" may be used to
-	suppress pruning.  This feature helps prevent corruption when
-	'git gc' runs concurrently with another process writing to the
-	repository; see the "NOTES" section of linkgit:git-gc[1].
+	When 'git gc' is run, it will call 'prune --expire 2.weeks.ago'
+	(and 'repack --cruft --cruft-expiration 2.weeks.ago' if using
+	cruft packs via `gc.cruftPacks` or `--cruft`).  Override the
+	grace period with this config variable.  The value "now" may be
+	used to disable this grace period and always prune unreachable
+	objects immediately, or "never" may be used to suppress pruning.
+	This feature helps prevent corruption when 'git gc' runs
+	concurrently with another process writing to the repository; see
+	the "NOTES" section of linkgit:git-gc[1].
 
 gc.worktreePruneExpire::
 	When 'git gc' is run, it calls
Documentation/git-gc.txt +5 -0
index 853967dea0..ba4e67700e 100644
@@ -54,6 +54,11 @@ other housekeeping tasks (e.g. rerere, working trees, reflog...) will
 be performed as well.
 
 
+--cruft::
+	When expiring unreachable objects, pack them separately into a
+	cruft pack instead of storing the loose objects as loose
+	objects.
+
 --prune=<date>::
 	Prune loose objects older than date (default is 2 weeks ago,
 	overridable by the config variable `gc.pruneExpire`).
builtin/gc.c +9 -1
index b335cffa33..4d995e85e9 100644
@@ -42,6 +42,7 @@ static const char * const builtin_gc_usage[] = {
 
 static int pack_refs = 1;
 static int prune_reflogs = 1;
+static int cruft_packs = 0;
 static int aggressive_depth = 50;
 static int aggressive_window = 250;
 static int gc_auto_threshold = 6700;
@@ -152,6 +153,7 @@ static void gc_config(void)
 	git_config_get_int("gc.auto", &gc_auto_threshold);
 	git_config_get_int("gc.autopacklimit", &gc_auto_pack_limit);
 	git_config_get_bool("gc.autodetach", &detach_auto);
+	git_config_get_bool("gc.cruftpacks", &cruft_packs);
 	git_config_get_expiry("gc.pruneexpire", &prune_expire);
 	git_config_get_expiry("gc.worktreepruneexpire", &prune_worktrees_expire);
 	git_config_get_expiry("gc.logexpiry", &gc_log_expire);
@@ -331,7 +333,11 @@ static void add_repack_all_option(struct string_list *keep_pack)
 {
 	if (prune_expire && !strcmp(prune_expire, "now"))
 		strvec_push(&repack, "-a");
-	else {
+	else if (cruft_packs) {
+		strvec_push(&repack, "--cruft");
+		if (prune_expire)
+			strvec_pushf(&repack, "--cruft-expiration=%s", prune_expire);
+	} else {
 		strvec_push(&repack, "-A");
 		if (prune_expire)
 			strvec_pushf(&repack, "--unpack-unreachable=%s", prune_expire);
@@ -551,6 +557,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 		{ OPTION_STRING, 0, "prune", &prune_expire, N_("date"),
 			N_("prune unreferenced objects"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
+		OPT_BOOL(0, "cruft", &cruft_packs, N_("pack unreferenced objects separately")),
 		OPT_BOOL(0, "aggressive", &aggressive, N_("be more thorough (increased runtime)")),
 		OPT_BOOL_F(0, "auto", &auto_gc, N_("enable auto-gc mode"),
 			   PARSE_OPT_NOCOMPLETE),
@@ -670,6 +677,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 			die(FAILED_RUN, repack.v[0]);
 
 		if (prune_expire) {
+			/* run `git prune` even if using cruft packs */
 			strvec_push(&prune, prune_expire);
 			if (quiet)
 				strvec_push(&prune, "--no-progress");
t/t5329-pack-objects-cruft.sh +37 -0
index 8de87afce2..70a6a9553c 100755
@@ -429,6 +429,43 @@ test_expect_success 'loose objects mtimes upsert others' '
 	)
 '
 
+test_expect_success 'expiring cruft objects with git gc' '
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit reachable &&
+		git branch -M main &&
+		git checkout --orphan other &&
+		test_commit unreachable &&
+
+		git checkout main &&
+		git branch -D other &&
+		git tag -d unreachable &&
+		# objects are not cruft if they are contained in the reflogs
+		git reflog expire --all --expire=all &&
+
+		git rev-list --objects --all --no-object-names >reachable.raw &&
+		git cat-file --batch-all-objects --batch-check="%(objectname)" >objects &&
+		sort <reachable.raw >reachable &&
+		comm -13 reachable objects >unreachable &&
+
+		git repack --cruft -d &&
+
+		mtimes=$(ls .git/objects/pack/pack-*.mtimes) &&
+		test_path_is_file $mtimes &&
+
+		git gc --cruft --prune=now &&
+
+		git cat-file --batch-all-objects --batch-check="%(objectname)" >objects &&
+
+		comm -23 unreachable objects >removed &&
+		test_cmp unreachable removed &&
+		test_path_is_missing $mtimes
+	)
+'
+
 test_expect_success 'cruft packs are not included in geometric repack' '
 	git init repo &&
 	test_when_finished "rm -fr repo" &&
Signature · 1 line
2.36.1.94.gb0d54bedca
#189 lore raw

Re: [PATCH v5 16/17] builtin/gc.c: conditionally avoid pruning objects via loose

Am 21.05.22 um 01:18 schrieb Taylor Blau:

Documentation/git-gc.txt other housekeeping tasks (e.g. rerere, working trees, reflog...) will · line 60

Quoted 12 lines
Documentation/git-gc.txt +4 -0
index 853967dea0..ba4e67700e 100644
@@ -54,6 +54,11 @@ other housekeeping tasks (e.g. rerere, working trees, reflog...) will
 be performed as well.
 
 
+--cruft::
+	When expiring unreachable objects, pack them separately into a
+	cruft pack instead of storing the loose objects as loose
+	objects.

The last part looks tautological. How about:

--- >8 ---
Subject: [PATCH] gc: simplify --cruft description

Remove duplicate "loose objects".

---
 Documentation/git-gc.txt | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
Documentation/git-gc.txt +1 -2
index ba4e67700e..0af7540a0c 100644
@@ -56,8 +56,7 @@ be performed as well.
 
 --cruft::
 	When expiring unreachable objects, pack them separately into a
-	cruft pack instead of storing the loose objects as loose
-	objects.
+	cruft pack instead of storing them as loose objects.
 
 --prune=<date>::
 	Prune loose objects older than date (default is 2 weeks ago,

--
2.36.1

#190 lore raw

Re: [PATCH v5 16/17] builtin/gc.c: conditionally avoid pruning objects via loose

Quoted 16 lines · René Scharfe

Am 21.05.22 um 01:18 schrieb Taylor Blau:

Documentation/git-gc.txt other housekeeping tasks (e.g. rerere, working trees, reflog...) will · line 60

Quoted 12 lines
Documentation/git-gc.txt +4 -0
index 853967dea0..ba4e67700e 100644
@@ -54,6 +54,11 @@ other housekeeping tasks (e.g. rerere, working trees, reflog...) will
 be performed as well.
 
 
+--cruft::
+	When expiring unreachable objects, pack them separately into a
+	cruft pack instead of storing the loose objects as loose
+	objects.

The last part looks tautological. How about:

Quoted 7 lines

--- >8 ---
Subject: [PATCH] gc: simplify --cruft description

Remove duplicate "loose objects".

---

Sounds good. Will apply.

Thanks.

Quoted 19 lines
 Documentation/git-gc.txt | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
Documentation/git-gc.txt +1 -3
index ba4e67700e..0af7540a0c 100644
@@ -56,8 +56,7 @@ be performed as well.
 
 --cruft::
 	When expiring unreachable objects, pack them separately into a
-	cruft pack instead of storing the loose objects as loose
-	objects.
+	cruft pack instead of storing them as loose objects.
 
 --prune=<date>::
 	Prune loose objects older than date (default is 2 weeks ago,
--

2.36.1

#191 lore raw

PATCH v5 17/17 sha1-file.c: don't freshen cruft packs

In reply to Taylor Blau

We don't bother to freshen objects stored in a cruft pack individually by updating the .mtimes file. This is because we can't portably mmap and write into the middle of a file (i.e., to update the mtime of just one object). Instead, we would have to rewrite the entire .mtimes file which may incur some wasted effort especially if there a lot of cruft objects and they are freshened infrequently.

Instead, force the freshening code to avoid an optimizing write by writing out the object loose and letting it pick up a current mtime.

This works because we prefer the mtime of the loose copy of an object when both a loose and packed one exist (whether or not the packed copy comes from a cruft pack or not).

This could certainly do with a test and/or be included earlier in this series/PR, but I want to wait until after I have a chance to clean up the overly-repetitive nature of the cruft pack tests in general.

---
 object-file.c                 |  2 ++
 t/t5329-pack-objects-cruft.sh | 25 +++++++++++++++++++++++++
 2 files changed, 27 insertions(+)
object-file.c +2 -0
index ff0cffe68e..495a359200 100644
@@ -2035,6 +2035,8 @@ static int freshen_packed_object(const struct object_id *oid)
 	struct pack_entry e;
 	if (!find_pack_entry(the_repository, oid, &e))
 		return 0;
+	if (e.p->is_cruft)
+		return 0;
 	if (e.p->freshened)
 		return 1;
 	if (!freshen_file(e.p->pack_name))
t/t5329-pack-objects-cruft.sh +25 -0
index 70a6a9553c..b481224b93 100755
@@ -711,4 +711,29 @@ test_expect_success 'MIDX bitmaps tolerate reachable cruft objects' '
 	)
 '
 
+test_expect_success 'cruft objects are freshend via loose' '
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		echo "cruft" >contents &&
+		blob="$(git hash-object -w -t blob contents)" &&
+		loose="$objdir/$(test_oid_to_path $blob)" &&
+
+		test_commit base &&
+
+		git repack --cruft -d &&
+
+		test_path_is_missing "$loose" &&
+		test-tool pack-mtimes "$(basename "$(ls $packdir/pack-*.mtimes)")" >cruft &&
+		grep "$blob" cruft &&
+
+		# write the same object again
+		git hash-object -w -t blob contents &&
+
+		test_path_is_file "$loose"
+	)
+'
+
 test_done
Signature · 1 line
2.36.1.94.gb0d54bedca
#192 lore raw
In reply to Taylor Blau
Quoted 4 lines · Taylor Blau

- The new section in pack-format.txt (describing the ".mtimes" format) now says at the top "all 4-byte numbers are in network byte order", and avoids repeating "network [byte] order" throughout that section to reduce confusion.

Suggestion (outside this series) since that's fixed perhaps a small stand-alone patch to fix the existing "network order" occurances in the same file?

Quoted 2 lines

...and that's pretty much it. In any case, a range-diff is included below. Thanks again for all of the thoughtful feedback on this series.

It seems this didn't make it on-list, but for the last round (well, it seems I replied to v1 by accident) A sent this (at
https://lore.kernel.org/git/[email protected]/
if it eventually shows up):

	Return-Path: <[email protected]>
	Received: from gmgdl (dhcp-077-248-183-071.chello.nl. [77.248.183.71])
	        by smtp.gmail.com with ESMTPSA id en21-20020a17090728d500b006fa9820b4a2sm1979775ejc.165.2022.05.19.04.54.26
	        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
	        Thu, 19 May 2022 04:54:26 -0700 (PDT)
	Received: from avar by gmgdl with local (Exim 4.95)
		(envelope-from <[email protected]>)
		id 1nrejZ-0026V8-AE;
		Thu, 19 May 2022 13:54:25 +0200
	From: =?utf-8?B?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason <[email protected]>
	To: Taylor Blau <[email protected]>
	Cc: [email protected], [email protected], [email protected],
	 [email protected], [email protected], brian m. carlson <[email protected]>
	Subject: Re: [PATCH 05/17] pack-mtimes: support writing pack .mtimes files
	Date: Thu, 19 May 2022 13:48:42 +0200
	References: <[email protected]>
	 <deece9eb70e9750bb8350946679b521e59139fe2.1638224692.git.me@ttaylorr.com>
	User-agent: Debian GNU/Linux bookworm/sid; Emacs 27.1; mu4e 1.7.12
	In-reply-to: <deece9eb70e9750bb8350946679b521e59139fe2.1638224692.git.me@ttaylorr.com>
	Message-ID: <[email protected]>
	MIME-Version: 1.0
	Content-Type: text/plain
	X-TUID: ta0yc5HgmrrD
	
	
	On Mon, Nov 29 2021, Taylor Blau wrote:
	
	> +static void write_mtimes_header(struct hashfile *f)
	> +{
	> +	hashwrite_be32(f, MTIMES_SIGNATURE);
	> +	hashwrite_be32(f, MTIMES_VERSION);
	> +	hashwrite_be32(f, oid_version(the_hash_algo));
	> +}
	
	Given the history noted in
	https://lore.kernel.org/git/[email protected]/
	maybe we can say this ship has just sailed at this point.
	
	But since this is a new format I think it's worth considering not using
	the 1 or 2 you get from oid_version(), but the "format_id",
	i.e. GIT_SHA1_FORMAT_ID or GIT_SHA256_FORMAT_ID.
	
	You'll use the same space in the format for it, but we'll end up with
	something more obvious (as the integer encodes the sha1 or sha256 name).
	
	AFAICT this code is just copied from your earlier work on *.rev, which
	in turn seems copied from earlier work on midx & commit-graph, which
	seems to have used this way of referring to the hash version more as an
	accident than anything explicitly indended...
	
	Then again we could just say that both are equally valid at this point,
	especially given the use in adjacent formats.

I.e. do we think continuing to use 1 v.s. 2 in new formats over instead of 0x73686131 and 0x73323536 is the right choice?

Other than that the only question I have (I think) on this series is if Jonathan Nieder is happy with it. I looked back in my logs and there was an extensive on-IRC discussion about it at the end of March, which ended in you sending: https://lore.kernel.org/git/[email protected]/

But it seems Jonathan didn't chime in since then, and he had some major issues with the approach here. I think those should have been addressed by that discussion, but it would be nice to get a confirmation.

Quoted 136 lines

Taylor Blau (17):
Documentation/technical: add cruft-packs.txt
pack-mtimes: support reading .mtimes files
pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles' chunk-format.h: extract oid_version()
pack-mtimes: support writing pack .mtimes files t/helper: add 'pack-mtimes' test-tool builtin/pack-objects.c: return from create_object_entry() builtin/pack-objects.c: --cruft without expiration
reachable: add options to add_unseen_recent_objects_to_traversal
reachable: report precise timestamps from objects in cruft packs builtin/pack-objects.c: --cruft with expiration builtin/repack.c: support generating a cruft pack builtin/repack.c: allow configuring cruft pack generation builtin/repack.c: use named flags for existing_packs builtin/repack.c: add cruft packs to MIDX during geometric repack builtin/gc.c: conditionally avoid pruning objects via loose sha1-file.c: don't freshen cruft packs

 Documentation/Makefile                  |   1 +
 Documentation/config/gc.txt             |  21 +-
 Documentation/config/repack.txt         |   9 +
 Documentation/git-gc.txt                |   5 +
 Documentation/git-pack-objects.txt      |  30 +
 Documentation/git-repack.txt            |  11 +
 Documentation/technical/cruft-packs.txt | 123 ++++
 Documentation/technical/pack-format.txt |  19 +
 Makefile                                |   2 +
 builtin/gc.c                            |  10 +-
 builtin/pack-objects.c                  | 304 +++++++++-
 builtin/repack.c                        | 185 +++++-
 bulk-checkin.c                          |   2 +-
 chunk-format.c                          |  12 +
 chunk-format.h                          |   3 +
 commit-graph.c                          |  18 +-
 midx.c                                  |  18 +-
 object-file.c                           |   4 +-
 object-store.h                          |   7 +-
 pack-mtimes.c                           | 126 ++++
 pack-mtimes.h                           |  15 +
 pack-objects.c                          |   6 +
 pack-objects.h                          |  25 +
 pack-write.c                            |  93 ++-
 pack.h                                  |   4 +
 packfile.c                              |  19 +-
 reachable.c                             |  58 +-
 reachable.h                             |   9 +-
 t/helper/test-pack-mtimes.c             |  56 ++
 t/helper/test-tool.c                    |   1 +
 t/helper/test-tool.h                    |   1 +
 t/t5329-pack-objects-cruft.sh           | 739 ++++++++++++++++++++++++
 32 files changed, 1834 insertions(+), 102 deletions(-)
 create mode 100644 Documentation/technical/cruft-packs.txt
 create mode 100644 pack-mtimes.c
 create mode 100644 pack-mtimes.h
 create mode 100644 t/helper/test-pack-mtimes.c
 create mode 100755 t/t5329-pack-objects-cruft.sh

Range-diff against v4:

 -:  ---------- >  1:  f494ef7377 Documentation/technical: add cruft-packs.txt
 1:  8f9fd21be9 !  2:  91a9d21b0b pack-mtimes: support reading .mtimes files

@@ Documentation/technical/pack-format.txt: Pack file entry: <+

+== pack-*.mtimes files have the format:
+
++All 4-byte numbers are in network byte order.
++
+ - A 4-byte magic number '0x4d544d45' ('MTME').
+
+ - A 4-byte version identifier (= 1).
+
+ - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256). +
-+ - A table of 4-byte unsigned integers in network order. The ith
-+ value is the modification time (mtime) of the ith object in the
-+ corresponding pack by lexicographic (index) order. The mtimes
-+ count standard epoch seconds.
++ - A table of 4-byte unsigned integers. The ith value is the
++ modification time (mtime) of the ith object in the corresponding
++ pack by lexicographic (index) order. The mtimes count standard
++ epoch seconds.
+
+ - A trailer, containing a checksum of the corresponding packfile,
+ and a checksum of all of the above (each having length according
+ to the specified hash function).
-+
-+All 4-byte numbers are in network order.
+
== multi-pack-index (MIDX) files have the following format:

 2:  cdb21236e1 =  3:  67c4e7209d pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles'
 3:  1d775f9850 =  4:  fc86506881 chunk-format.h: extract oid_version()
 4:  6172861bd9 =  5:  788d1f96f2 pack-mtimes: support writing pack .mtimes files
 5:  5f9a9a5b7b =  6:  2a6cfb00bf t/helper: add 'pack-mtimes' test-tool
 6:  b8a38fe2e4 =  7:  edb6fcd5ec builtin/pack-objects.c: return from create_object_entry()
 7:  94fe03cc65 =  8:  e3185741f2 builtin/pack-objects.c: --cruft without expiration
 8:  da7273f41f =  9:  1cf00d462c reachable: add options to add_unseen_recent_objects_to_traversal
 9:  58fecd1747 = 10:  d66be44d9a reachable: report precise timestamps from objects in cruft packs
10:  1740b8ef01 = 11:  1434e37623 builtin/pack-objects.c: --cruft with expiration
11:  5992a72cbf ! 12:  0d3555d595 builtin/repack.c: support generating a cruft pack
    @@ t/t5329-pack-objects-cruft.sh: test_expect_success 'expired objects are pruned'
     +		git repack &&
     +
     +		tip="$(git rev-parse cruft)" &&
    -+		path="$objdir/$(test_oid_to_path "$(git rev-parse cruft)")" &&
    ++		path="$objdir/$(test_oid_to_path "$tip")" &&
     +		test-tool chmtime --get +1000 "$path" >expect &&
     +
     +		git checkout main &&
12:  1b241f8f91 = 13:  4b721d3ee9 builtin/repack.c: allow configuring cruft pack generation
13:  ffae78852c = 14:  f9e3ab56b1 builtin/repack.c: use named flags for existing_packs
14:  0743e373ba ! 15:  e9f46e7b5e builtin/repack.c: add cruft packs to MIDX during geometric repack
    @@ builtin/repack.c
      static int pack_everything;
      static int delta_base_offset = 1;
     @@ builtin/repack.c: static void collect_pack_filenames(struct string_list *fname_nonkept_list,
    + 		fname = xmemdupz(e->d_name, len);
    + 
      		if ((extra_keep->nr > 0 && i < extra_keep->nr) ||
    - 		    (file_exists(mkpath("%s/%s.keep", packdir, fname))))
    +-		    (file_exists(mkpath("%s/%s.keep", packdir, fname))))
    ++		    (file_exists(mkpath("%s/%s.keep", packdir, fname)))) {
      			string_list_append_nodup(fname_kept_list, fname);
     -		else
     -			string_list_append_nodup(fname_nonkept_list, fname);
    -+		else {
    -+			struct string_list_item *item = string_list_append_nodup(fname_nonkept_list, fname);
    ++		} else {
    ++			struct string_list_item *item;
    ++			item = string_list_append_nodup(fname_nonkept_list,
    ++							fname);
     +			if (file_exists(mkpath("%s/%s.mtimes", packdir, fname)))
     +				item->util = (void*)(uintptr_t)CRUFT_PACK;
     +		}
15:  9f7e0acac6 = 16:  43c14eec07 builtin/gc.c: conditionally avoid pruning objects via loose
16:  07fa9d4b47 = 17:  1e313b89e8 sha1-file.c: don't freshen cruft packs
#193 lore raw

Hi,

Quoted 9 lines · Ævar Arnfjörð Bjarmason

On Fri, May 20 2022, Taylor Blau wrote:

	On Mon, Nov 29 2021, Taylor Blau wrote:

	> +static void write_mtimes_header(struct hashfile *f)
	> +{
	> +	hashwrite_be32(f, MTIMES_SIGNATURE);
	> +	hashwrite_be32(f, MTIMES_VERSION);
	> +	hashwrite_be32(f, oid_version(the_hash_algo));
	> +}

[...]

Quoted 6 lines

But since this is a new format I think it's worth considering not using the 1 or 2 you get from oid_version(), but the "format_id", i.e. GIT_SHA1_FORMAT_ID or GIT_SHA256_FORMAT_ID.

You'll use the same space in the format for it, but we'll end up with something more obvious (as the integer encodes the sha1 or sha256 name).

Agreed.

[...]

Quoted 8 lines

Other than that the only question I have (I think) on this series is if Jonathan Nieder is happy with it. I looked back in my logs and there was an extensive on-IRC discussion about it at the end of March, which ended in you sending: https://lore.kernel.org/git/[email protected]/

But it seems Jonathan didn't chime in since then, and he had some major issues with the approach here. I think those should have been addressed by that discussion, but it would be nice to get a confirmation.

I would still prefer if this used a repository format extension, but that preference is not strong enough that I'd say "this must not go in without one". What I think would help would be some information in the user-facing documentation for commands that create and work with cruft packs. In other words, if our take on people sharing repositories between implementations that understand and don't understand cruft packs and get objects moving back and forth between packed and loose objects is "you should have known you were doing something strange", the least we can do is to warn them.

I don't see a config to enable PACK_CRUFT by default yet in this series. I'd like one, so that people can turn it on and get the good new behavior. :)

Thanks,
Jonathan

#194 lore raw
Quoted 11 lines · Jonathan Nieder
Quoted 9 lines · Ævar Arnfjörð Bjarmason

On Fri, May 20 2022, Taylor Blau wrote:

	On Mon, Nov 29 2021, Taylor Blau wrote:

	> +static void write_mtimes_header(struct hashfile *f)
	> +{
	> +	hashwrite_be32(f, MTIMES_SIGNATURE);
	> +	hashwrite_be32(f, MTIMES_VERSION);
	> +	hashwrite_be32(f, oid_version(the_hash_algo));
	> +}

[...]

Quoted 8 lines
Quoted 6 lines

But since this is a new format I think it's worth considering not using the 1 or 2 you get from oid_version(), but the "format_id", i.e. GIT_SHA1_FORMAT_ID or GIT_SHA256_FORMAT_ID.

You'll use the same space in the format for it, but we'll end up with something more obvious (as the integer encodes the sha1 or sha256 name).

Agreed.

I know we recommend using the format_id for on-disk formats, but I think there is enough existing uses of "1" or "2" that either are acceptable in practice.

E.g., grepping around for "hashwrite.*oid_version", there are three existing formats that use "1" or "2" instead of the format_id. They are:

- the commit-graph format
- the midx format
- the .rev format

Moreover, I can't seem to find any formats that _don't_ use that convention. So I have a vague preference towards using the values "1" and "2" as we currently do in these patches. (TBH, I don't find "sha1" significantly more interpretable than just "1", so I would be just as happy leaving it as-is).

Quoted 10 lines

[...]

Quoted 8 lines

Other than that the only question I have (I think) on this series is if Jonathan Nieder is happy with it. I looked back in my logs and there was an extensive on-IRC discussion about it at the end of March, which ended in you sending: https://lore.kernel.org/git/[email protected]/

But it seems Jonathan didn't chime in since then, and he had some major issues with the approach here. I think those should have been addressed by that discussion, but it would be nice to get a confirmation.

Quoted 9 lines

I would still prefer if this used a repository format extension, but that preference is not strong enough that I'd say "this must not go in without one". What I think would help would be some information in the user-facing documentation for commands that create and work with cruft packs. In other words, if our take on people sharing repositories between implementations that understand and don't understand cruft packs and get objects moving back and forth between packed and loose objects is "you should have known you were doing something strange", the least we can do is to warn them.

I think that's a good suggestion. We already have some documentation in Documentation/technical/cruft-packs.txt, but I think it could be helpful to add user-facing documentation, too.

Would you be opposed to doing that outside of this series? ISTM that the technical discussion has mostly settled, so I'd rather wordsmith the user-facing documentation separately.

Quoted 3 lines

I don't see a config to enable PACK_CRUFT by default yet in this series. I'd like one, so that people can turn it on and get the good new behavior. :)

git gc has support for this (c.f., "gc.cruftPacks"). git repack requires you to pass --cruft; IIRC I originally had a similar configuration in git repack which would change the behavior of -A / -a when set, but I found it too confusing and scrapped it.

Thanks,
Taylor

#195 lore raw
Quoted 28 lines · Taylor Blau
Quoted 11 lines · Jonathan Nieder
Quoted 9 lines · Ævar Arnfjörð Bjarmason

On Fri, May 20 2022, Taylor Blau wrote:

	On Mon, Nov 29 2021, Taylor Blau wrote:

	> +static void write_mtimes_header(struct hashfile *f)
	> +{
	> +	hashwrite_be32(f, MTIMES_SIGNATURE);
	> +	hashwrite_be32(f, MTIMES_VERSION);
	> +	hashwrite_be32(f, oid_version(the_hash_algo));
	> +}

[...]

Quoted 8 lines
Quoted 6 lines

But since this is a new format I think it's worth considering not using the 1 or 2 you get from oid_version(), but the "format_id", i.e. GIT_SHA1_FORMAT_ID or GIT_SHA256_FORMAT_ID.

You'll use the same space in the format for it, but we'll end up with something more obvious (as the integer encodes the sha1 or sha256 name).

Agreed.

I know we recommend using the format_id for on-disk formats, but I think there is enough existing uses of "1" or "2" that either are acceptable in practice.

E.g., grepping around for "hashwrite.*oid_version", there are three existing formats that use "1" or "2" instead of the format_id. They are:

Quoted 6 lines

- the commit-graph format
- the midx format
- the .rev format

Moreover, I can't seem to find any formats that _don't_ use that convention.

It's used in the reftable format.

Quoted 2 lines

So I have a vague preference towards using the values "1" and "2" as we currently do in these patches.

I suspect that's less "vague" and more "c'mon, I'm using it in production already" :)

Anyway, I'm fine with leaving it be as you have it currently. I'd first encountered this magic with reftable I think, and only recently found that we use 1 and 2 in these other more recent places.

Quoted 3 lines

(TBH, I don't find "sha1"
significantly more interpretable than just "1", so I would be just as happy leaving it as-is).

Hrm, I'd think having it sha1 or s256 in big-endian would be a bit more self-explanatory. I.e. SHA-256 is 2, not 256, and our 3 (if that ever arrives) is likely not to be SHA-3 (but probably some successor).

#196 lore raw
Quoted 31 lines · Ævar Arnfjörð Bjarmason
Quoted 28 lines · Taylor Blau
Quoted 11 lines · Jonathan Nieder
Quoted 9 lines · Ævar Arnfjörð Bjarmason

On Fri, May 20 2022, Taylor Blau wrote:

	On Mon, Nov 29 2021, Taylor Blau wrote:

	> +static void write_mtimes_header(struct hashfile *f)
	> +{
	> +	hashwrite_be32(f, MTIMES_SIGNATURE);
	> +	hashwrite_be32(f, MTIMES_VERSION);
	> +	hashwrite_be32(f, oid_version(the_hash_algo));
	> +}

[...]

Quoted 8 lines
Quoted 6 lines

But since this is a new format I think it's worth considering not using the 1 or 2 you get from oid_version(), but the "format_id", i.e. GIT_SHA1_FORMAT_ID or GIT_SHA256_FORMAT_ID.

You'll use the same space in the format for it, but we'll end up with something more obvious (as the integer encodes the sha1 or sha256 name).

Agreed.

I know we recommend using the format_id for on-disk formats, but I think there is enough existing uses of "1" or "2" that either are acceptable in practice.

E.g., grepping around for "hashwrite.*oid_version", there are three existing formats that use "1" or "2" instead of the format_id. They are:

Quoted 8 lines
Quoted 6 lines

- the commit-graph format
- the midx format
- the .rev format

Moreover, I can't seem to find any formats that _don't_ use that convention.

It's used in the reftable format.

Ah, thanks for pointing it out. Still, I think there's enough uses of "1" and "2" over format_id that I'm not convinced here.

Quoted 5 lines
Quoted 2 lines

So I have a vague preference towards using the values "1" and "2" as we currently do in these patches.

I suspect that's less "vague" and more "c'mon, I'm using it in production already" :)

No, this wasn't a veil over anything. Yes, GitHub is using this in production already, but that isn't why I'm opposed here. I'm opposed for the reasons I explained in the quoted bits (and would happily carry a small amount of custom code in GitHub's fork to continue to recognize the "1" or "2" values if this ever changed to use format_id).

Quoted 3 lines

Anyway, I'm fine with leaving it be as you have it currently. I'd first encountered this magic with reftable I think, and only recently found that we use 1 and 2 in these other more recent places.

Sounds good. Unless others have a very strong opinion, let's leave it as is.

Thanks,
Taylor

#197 lore raw

Hi,

Quoted 6 lines · Taylor Blau
Quoted 4 lines · Ævar Arnfjörð Bjarmason
Quoted 2 lines

Moreover, I can't seem to find any formats that _don't_ use that convention.

It's used in the reftable format.

It's also used in the formats described in
Documentation/technical/hash-function-transition.

[...]

Quoted 2 lines

Sounds good. Unless others have a very strong opinion, let's leave it as is.

File formats are one of those things where a little time early can save a lot of work later. If there were a strong reason to use "1" and "2" here then I'd be okay with living with it --- I'm a pragmatic person. But in general, using the magic numbers instead of a sequential value is really helpful both in making the file formats more self-explanatory and in making it possible to experiment with multiple new hash_algos at the same time.

The main argument I'm hearing for using "1" and "2" is "because some other formats got that wrong". That reason is the opposite of compelling to me: it makes me suspect that as a project we should more eagerly break the old bad habits and form new ones. I guess this qualifies as a very strong opinion.

Thanks,
Jonathan

#198 lore raw
Quoted 7 lines · Jonathan Nieder
Quoted 6 lines · Taylor Blau
Quoted 4 lines · Ævar Arnfjörð Bjarmason
Quoted 2 lines

Moreover, I can't seem to find any formats that _don't_ use that convention.

It's used in the reftable format.

The use in reftable is the only one I can find and that implementation is not idiomatic. Specifically, the way the four-byte header was implemented is not easy to extract and share in other formats.

This series does the good work of extracting oid_version() as a common method across these formats so it is easier to share.

Quoted 2 lines

It's also used in the formats described in
Documentation/technical/hash-function-transition.

It documents things that have not been implemented, such as the v3 pack-index format:

Pack index (.idx) files use a new v3 format that supports multiple hash functions. They have the following format (all integers are in network byte order):
(...)
* 4-byte number of object formats in this pack index: 2
* For each object format:
** 4-byte format identifier (e.g., 'sha1' for SHA-1) ** 4-byte length in bytes of shortened object names. This is the
shortest possible length needed to make names in the shortened object name table unambiguous.
** 4-byte integer, recording where tables relating to this format
are stored in this index file, as an offset from the beginning.

This was added in your 752414ae431 (technical doc: add a design doc for hash function transition, 2017-09-27), but has not been acted upon yet.

Quoted 12 lines

[...]

Quoted 2 lines

Sounds good. Unless others have a very strong opinion, let's leave it as is.

File formats are one of those things where a little time early can save a lot of work later. If there were a strong reason to use "1" and "2" here then I'd be okay with living with it --- I'm a pragmatic person. But in general, using the magic numbers instead of a sequential value is really helpful both in making the file formats more self-explanatory and in making it possible to experiment with multiple new hash_algos at the same time.

Quoted 5 lines

The main argument I'm hearing for using "1" and "2" is "because some other formats got that wrong". That reason is the opposite of compelling to me: it makes me suspect that as a project we should more eagerly break the old bad habits and form new ones. I guess this qualifies as a very strong opinion.

Either way, these are magic numbers. One happens to somewhat spell out something when looking at the file in a hex editor with ASCII previews, but that doesn't change the fact that it is most important that the hash function is correctly indicated by the file format and parsed by the Git executable (not a human).

I'd much rather have a consistent and proven way of specifying the hash value (using the oid_version() helper) than to try and make a new mechanism.

Thanks,
-Stolee

#199 lore raw
Quoted 3 lines · Derrick Stolee

I'd much rather have a consistent and proven way of specifying the hash value (using the oid_version() helper) than to try and make a new mechanism.

To be clear, I absolutely don't think any of us should have the attitude of repeating past bad decisions for the sake of consistency.

As best I can tell, our (Jonathan and I's) disagreement is on whether using "1" and "2" to identify which hash function is used by the .mtimes file is OK or not. I happen to think that it is acceptable, so the choice to continue to adopt this pattern was motivated by being consistent with a pattern that is good and works.

Thanks,
Taylor

#200 lore raw
Quoted 5 lines · Taylor Blau
Quoted 3 lines · Derrick Stolee

I'd much rather have a consistent and proven way of specifying the hash value (using the oid_version() helper) than to try and make a new mechanism.

Quoted 8 lines

To be clear, I absolutely don't think any of us should have the attitude of repeating past bad decisions for the sake of consistency.

As best I can tell, our (Jonathan and I's) disagreement is on whether using "1" and "2" to identify which hash function is used by the .mtimes file is OK or not. I happen to think that it is acceptable, so the choice to continue to adopt this pattern was motivated by being consistent with a pattern that is good and works.

I don't have a strong opinion on whether we "bless" that or not, and say that we should just use 1, 2 etc. going forward or not.

But I do think that us doing so initially wasn't intentional, and has been in opposition to a strongly worded claim in a comment in hash.h (which I modified in my earlier related RFC series).

So maybe not part of this series, but it seems prudent if you feel strongly about using this for new formats over what hash.h is currently recommending that we have some patch sooner than later to update it accordingly.