Skip to contentSkip to conversation

rev-parse: compute object names in another hash algorithm

PATCH New Dimitri John Ledkov 1 participant 1 message Aug 14

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rev-parse: compute object names in another hash algorithm

"git rev-parse --output-object-format=sha256 HEAD^{tree}" could not answer in a SHA-1 repository. And vice-versa. Without extensions.compatObjectFormat the option was rejected outright, and with it the answer was wrong: the extension records a mapping only for objects written while it is enabled, so for the objects a repository already contained the lookup missed, repo_oid_to_algop() left the object ID untouched, and rev-parse printed the SHA-1 name with a zero exit code.

The machinery to convert object contents between algorithms is already here. convert_tree_object() and friends rewrite the object IDs a tree, commit or tag refers to, calling repo_oid_to_algop() for each one, so the recursion is written; what is missing is a base case. Give repo_oid_to_algop() one: on a lookup miss, read the object, convert its contents, and hash the result. A blob needs no conversion at all, as its contents are the same either way, only a rehash.

Names computed this way are remembered during the process execution, so that a tree that reaches the same subtree by several paths pays for it once. These are not stored permamently. In the future a side-car cache could be added for these results, or eventual dual-format v3 could be used to lazy compute/convert and store these.

Commits are refused rather than computed. A commit names its parents, so converting one converts the entire history behind it; that is a repository conversion, not an answer about a single object, and recursing over it here would be bounded only by the length of the history. A tree holding a submodule is refused for the same reason, and now says which entry is at fault instead of reporting an unreadable object.

Since names can now be computed, accept any algorithm git knows rather than only the configured compatibility one, and check what repo_oid_to_algop() returns so that a name we cannot produce is an error instead of the storage name.

On git.git's tree at 9936c1b52a (3089 files, 26MB) this names the tree in 0.26s, against 0.25s for reading and hashing that content by hand, so close to the floor for the work involved. On linux.git tree of 1.6GiB it takes 8s to compute.

Initially, I have implemented contrib git-sha256-tree.sh script using a temporary git repo and perform fast-export/fast-import of a single commit and its tree. It is a lot slower due to needless work of fast-import that is discarded. However, a contrib script carries less maintainance and is more portable to existing installations. If there is interest, I can publish that implementation as well. The goal is to help with interop, and have the ability to record tree names in either format today; to check again later if and when a given project switches to sha256 format.

Assisted-by Claude Opus 5 xHigh
---
 Documentation/git-rev-parse.adoc |  13 +++
 builtin/rev-parse.c              |  23 ++++-
 object-file-convert.c            | 145 +++++++++++++++++++++++++++++--
 object-file-convert.h            |   6 ++
 repository.c                     |   3 +
 repository.h                     |   8 ++
 t/meson.build                    |   1 +
 t/t1018-output-object-format.sh  | 118 +++++++++++++++++++++++++
 8 files changed, 308 insertions(+), 9 deletions(-)
 create mode 100755 t/t1018-output-object-format.sh
Documentation/git-rev-parse.adoc +13 -0
index 5398691f3f..c18ae9ca81 100644
@@ -181,6 +181,19 @@ Specifying "sha256" translates if necessary and returns a sha256 oid.
 +
 Specifying "storage" translates if necessary and returns an oid in
 encoded in the storage hash algorithm.
++
+A translation that `extensions.compatObjectFormat` did not record, which is
+every object a repository already contained when that extension was turned
+on, is computed on demand.  Since an object's name in another algorithm is
+defined over the names of the objects it refers to, this reads everything
+reachable from the object, and so costs time proportional to the total size
+of that content.
++
+Commits cannot be named this way.  A commit refers to its parents, so naming
+one would mean converting the whole history behind it, which is a repository
+conversion rather than a question about a single object.  For the same
+reason a tree containing a submodule cannot be named, as its gitlink refers
+to a commit in another repository.
 
 Options for Objects
 ~~~~~~~~~~~~~~~~~~~
builtin/rev-parse.c +19 -4
index 43693454d5..7dd06f324e 100644
@@ -876,8 +876,21 @@ int cmd_rev_parse(int argc,
 					flags |= GET_OID_HASH_ANY;
 					output_algo = compat;
 					continue;
+				} else {
+					/*
+					 * Names in another algorithm can be
+					 * computed on demand, so accept any
+					 * algorithm we know rather than only
+					 * the configured compatibility one.
+					 */
+					uint32_t algo = hash_algo_by_name(arg);
+
+					if (algo == GIT_HASH_UNKNOWN)
+						die(_("unsupported object format: %s"), arg);
+					flags |= GET_OID_HASH_ANY;
+					output_algo = &hash_algos[algo];
+					continue;
 				}
-				else die(_("unsupported object format: %s"), arg);
 			}
 			if (opt_with_value(arg, "--short", &arg)) {
 				filter &= ~(DO_FLAGS|DO_NOREV);
@@ -1162,9 +1175,11 @@ int cmd_rev_parse(int argc,
 		}
 		if (!repo_get_oid_with_flags(the_repository, name, &oid,
 					     flags)) {
-			if (output_algo)
-				repo_oid_to_algop(the_repository, &oid,
-						  output_algo, &oid);
+			if (output_algo &&
+			    repo_oid_to_algop(the_repository, &oid,
+					      output_algo, &oid))
+				die(_("cannot express %s as a %s object name"),
+				    name, output_algo->name);
 			if (verify)
 				revs_count++;
 			else
object-file-convert.c +140 -5

177 lines not on this page · raw message

object-file-convert.h +6 -0
index 9b3cc5e533..af384c362b 100644
@@ -10,6 +10,12 @@ struct strbuf;
 int repo_oid_to_algop(struct repository *repo, const struct object_id *src,
 		      const struct git_hash_algo *to, struct object_id *dest);
 
+/*
+ * Release the object names repo_oid_to_algop() computed on demand.  Called
+ * from repo_clear().
+ */
+void repo_clear_compat_oid_cache(struct repository *repo);
+
 /*
  * Convert an object file from one hash algorithm to another algorithm.
  * Return -1 on failure, 0 on success.
repository.c +3 -0
index 651b0f6933..16201ec5ea 100644
@@ -7,6 +7,7 @@
 #include "config.h"
 #include "gettext.h"
 #include "object.h"
+#include "object-file-convert.h"
 #include "lockfile.h"
 #include "path.h"
 #include "read-cache-ll.h"
@@ -386,6 +387,8 @@ void repo_clear(struct repository *repo)
 	odb_free(repo->objects);
 	repo->objects = NULL;
 
+	repo_clear_compat_oid_cache(repo);
+
 	parsed_object_pool_clear(repo->parsed_objects);
 	FREE_AND_NULL(repo->parsed_objects);
 
repository.h +8 -0
index 3b467a2513..1a89559cd5 100644
@@ -12,6 +12,7 @@ struct index_state;
 struct lock_file;
 struct pathspec;
 struct object_database;
+struct oidmap;
 struct submodule_cache;
 struct promisor_remote_config;
 struct remote_state;
@@ -167,6 +168,13 @@ struct repository {
 	/* Repository's compatibility hash algorithm. */
 	const struct git_hash_algo *compat_hash_algo;
 
+	/*
+	 * Object names computed in another hash algorithm on demand, keyed by
+	 * the name the object has in this repository.  Owned and populated by
+	 * repo_oid_to_algop().
+	 */
+	struct oidmap *compat_oid_cache;
+
 	/* Repository-specific configuration values. */
 	struct repo_config_values config_values_private_;
 
t/meson.build +1 -0
index a25f37d2f5..bc3e644709 100644
@@ -172,6 +172,7 @@ integration_tests = [
   't1015-read-index-unmerged.sh',
   't1016-compatObjectFormat.sh',
   't1017-cat-file-remote-object-info.sh',
+  't1018-output-object-format.sh',
   't1020-subdirectory.sh',
   't1022-read-tree-partial-clone.sh',
   't1050-large.sh',
t/t1018-output-object-format.sh new file +118 -0

118 lines not on this page · raw message

Signature · 1 line
2.53.0