Skip to contentSkip to conversation

First steps towards partial clone submodules

PATCH Superseded Jonathan Tan 6 participants 77 messages Jun 1, 2021 – Jun 19, 2021 Kbuild

lore
Share & tools
lore
#0 lore raw

cover First steps towards partial clone submodules

This is a preliminary step towards supporting partial clone submodules (e.g., by cloning with --recurse-submodules and having the given filter propagate to submodules). Even with this patch set, we won't be there yet (notably, some code in Git access objects in submodules by adding them as alternates - so lazy-fetching missing objects in submodules wouldn't work here), but at least this is a first step.

This patch set would also be useful if Git needed to operate on other repositories (other than in the submodule case), but I can't think of such a situation right now.

As mentioned, there is still more work that needs to be done. Any help is appreciated, and as for me, I hope to get back to this in the 3rd quarter of the year.

Jonathan Tan (4):
promisor-remote: read partialClone config here
promisor-remote: support per-repository config
run-command: move envvar-resetting function
promisor-remote: teach lazy-fetch in any repo

 Makefile                      |   1 +
 cache.h                       |   1 -
 object-file.c                 |   7 +-
 promisor-remote.c             | 119 +++++++++++++++++++---------------
 promisor-remote.h             |  26 +++++---
 repository.h                  |   4 ++
 run-command.c                 |  10 +++
 run-command.h                 |   7 ++
 setup.c                       |  10 ++-
 submodule.c                   |  14 +---
 t/helper/test-partial-clone.c |  34 ++++++++++
 t/helper/test-tool.c          |   1 +
 t/helper/test-tool.h          |   1 +
 t/t0410-partial-clone.sh      |  24 +++++++
 14 files changed, 177 insertions(+), 82 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c
Signature · 1 line
2.32.0.rc0.204.g9fa02ecfa5-goog
#1 lore raw

PATCH 1/4 promisor-remote: read partialClone config here

Currently, the reading of config related to promisor remotes is done in two places: once in setup.c (which sets the global variable repository_format_partial_clone, to be read by the code in promisor-remote.c), and once in promisor-remote.c. This means that care must be taken to ensure that repository_format_partial_clone is set before any code in promisor-remote.c accesses it.

To simplify the code, move all such config reading to promisor-remote.c. By doing this, it will be easier to see when repository_format_partial_clone is written and, thus, to reason about the code. This will be especially helpful in a subsequent commit, which modifies this code.

---
 cache.h           |  1 -
 promisor-remote.c | 10 +++++-----
 promisor-remote.h |  6 ------
 setup.c           | 10 +++++++---
 4 files changed, 12 insertions(+), 15 deletions(-)
cache.h +0 -1
index ba04ff8bd3..dbdcec8601 100644
@@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;
 struct repository_format {
 	int version;
 	int precious_objects;
-	char *partial_clone; /* value of extensions.partialclone */
 	int worktree_config;
 	int is_bare;
 	int hash_algo;
promisor-remote.c +5 -5
index da3f2ca261..bfe8eee5f2 100644
@@ -7,11 +7,6 @@
 
 static char *repository_format_partial_clone;
 
-void set_repository_format_partial_clone(char *partial_clone)
-{
-	repository_format_partial_clone = xstrdup_or_null(partial_clone);
-}
-
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
@@ -99,6 +94,11 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	size_t namelen;
 	const char *subkey;
 
+	if (!strcmp(var, "extensions.partialclone")) {
+		repository_format_partial_clone = xstrdup(value);
+		return 0;
+	}
+
 	if (parse_config_key(var, "remote", &name, &namelen, &subkey) < 0)
 		return 0;
 
promisor-remote.h +0 -6
index c7a14063c5..687210ab87 100644
@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,
 			       const struct object_id *oids,
 			       int oid_nr);
 
-/*
- * This should be used only once from setup.c to set the value we got
- * from the extensions.partialclone config option.
- */
-void set_repository_format_partial_clone(char *partial_clone);
-
 #endif /* PROMISOR_REMOTE_H */
setup.c +7 -3
index 59e2facd9d..d60b6bc554 100644
@@ -470,7 +470,13 @@ static enum extension_result handle_extension_v0(const char *var,
 		} else if (!strcmp(ext, "partialclone")) {
 			if (!value)
 				return config_error_nonbool(var);
-			data->partial_clone = xstrdup(value);
+			/*
+			 * This config variable will be read together with the
+			 * other relevant config variables in
+			 * promisor_remote_config() in promisor_remote.c, so we
+			 * do not need to read it here. Just report that this
+			 * extension is known.
+			 */
 			return EXTENSION_OK;
 		} else if (!strcmp(ext, "worktreeconfig")) {
 			data->worktree_config = git_config_bool(var, value);
@@ -566,7 +572,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_
 	}
 
 	repository_format_precious_objects = candidate->precious_objects;
-	set_repository_format_partial_clone(candidate->partial_clone);
 	repository_format_worktree_config = candidate->worktree_config;
 	string_list_clear(&candidate->unknown_extensions, 0);
 	string_list_clear(&candidate->v1_only_extensions, 0);
@@ -650,7 +655,6 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->unknown_extensions, 0);
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
-	free(format->partial_clone);
 	init_repository_format(format);
 }
 
Signature · 1 line
2.32.0.rc0.204.g9fa02ecfa5-goog
#5 lore raw

PATCH 2/4 promisor-remote: support per-repository config

In reply to Jonathan Tan

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation.

---
 promisor-remote.c | 101 +++++++++++++++++++++++++---------------------
 promisor-remote.h |  20 +++++++--
 repository.h      |   4 ++
 3 files changed, 77 insertions(+), 48 deletions(-)
promisor-remote.c +56 -45
index bfe8eee5f2..5819d2cf28 100644
@@ -5,7 +5,11 @@
 #include "transport.h"
 #include "strvec.h"
 
-static char *repository_format_partial_clone;
+struct promisor_remote_config {
+	char *repository_format_partial_clone;
+	struct promisor_remote *promisors;
+	struct promisor_remote **promisors_tail;
+};
 
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
@@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name,
 	return finish_command(&child) ? -1 : 0;
 }
 
-static struct promisor_remote *promisors;
-static struct promisor_remote **promisors_tail = &promisors;
-
-static struct promisor_remote *promisor_remote_new(const char *remote_name)
+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,
+						   const char *remote_name)
 {
 	struct promisor_remote *r;
 
@@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)
 
 	FLEX_ALLOC_STR(r, name, remote_name);
 
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 
 	return r;
 }
 
-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,
+						      const char *remote_name,
 						      struct promisor_remote **previous)
 {
 	struct promisor_remote *r, *p;
 
-	for (p = NULL, r = promisors; r; p = r, r = r->next)
+	for (p = NULL, r = config->promisors; r; p = r, r = r->next)
 		if (!strcmp(r->name, remote_name)) {
 			if (previous)
 				*previous = p;
@@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
 	return NULL;
 }
 
-static void promisor_remote_move_to_tail(struct promisor_remote *r,
+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,
+					 struct promisor_remote *r,
 					 struct promisor_remote *previous)
 {
 	if (r->next == NULL)
@@ -82,20 +86,21 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,
 	if (previous)
 		previous->next = r->next;
 	else
-		promisors = r->next ? r->next : r;
+		config->promisors = r->next ? r->next : r;
 	r->next = NULL;
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 }
 
 static int promisor_remote_config(const char *var, const char *value, void *data)
 {
+	struct promisor_remote_config *config = data;
 	const char *name;
 	size_t namelen;
 	const char *subkey;
 
 	if (!strcmp(var, "extensions.partialclone")) {
-		repository_format_partial_clone = xstrdup(value);
+		config->repository_format_partial_clone = xstrdup(value);
 		return 0;
 	}
 
@@ -110,8 +115,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 
 		remote_name = xmemdupz(name, namelen);
 
-		if (!promisor_remote_lookup(remote_name, NULL))
-			promisor_remote_new(remote_name);
+		if (!promisor_remote_lookup(config, remote_name, NULL))
+			promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 		return 0;
@@ -120,9 +125,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 		struct promisor_remote *r;
 		char *remote_name = xmemdupz(name, namelen);
 
-		r = promisor_remote_lookup(remote_name, NULL);
+		r = promisor_remote_lookup(config, remote_name, NULL);
 		if (!r)
-			r = promisor_remote_new(remote_name);
+			r = promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 
@@ -135,59 +140,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	return 0;
 }
 
-static int initialized;
-
-static void promisor_remote_init(void)
+static void promisor_remote_init(struct repository *r)
 {
-	if (initialized)
+	struct promisor_remote_config *config;
+
+	if (r->promisor_remote_config)
 		return;
-	initialized = 1;
+	config = r->promisor_remote_config =
+		xcalloc(sizeof(*r->promisor_remote_config), 1);
+	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, NULL);
+	git_config(promisor_remote_config, config);
 
-	if (repository_format_partial_clone) {
+	if (config->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
 
-		o = promisor_remote_lookup(repository_format_partial_clone,
+		o = promisor_remote_lookup(config,
+					   config->repository_format_partial_clone,
 					   &previous);
 		if (o)
-			promisor_remote_move_to_tail(o, previous);
+			promisor_remote_move_to_tail(config, o, previous);
 		else
-			promisor_remote_new(repository_format_partial_clone);
+			promisor_remote_new(config, config->repository_format_partial_clone);
 	}
 }
 
-static void promisor_remote_clear(void)
+static void promisor_remote_clear(struct promisor_remote_config *config)
 {
-	while (promisors) {
-		struct promisor_remote *r = promisors;
-		promisors = promisors->next;
+	while (config->promisors) {
+		struct promisor_remote *r = config->promisors;
+		config->promisors = config->promisors->next;
 		free(r);
 	}
 
-	promisors_tail = &promisors;
+	config->promisors_tail = &config->promisors;
 }
 
-void promisor_remote_reinit(void)
+void repo_promisor_remote_reinit(struct repository *r)
 {
-	initialized = 0;
-	promisor_remote_clear();
-	promisor_remote_init();
+	promisor_remote_clear(r->promisor_remote_config);
+	FREE_AND_NULL(r->promisor_remote_config);
+	promisor_remote_init(r);
 }
 
-struct promisor_remote *promisor_remote_find(const char *remote_name)
+struct promisor_remote *repo_promisor_remote_find(struct repository *r,
+						  const char *remote_name)
 {
-	promisor_remote_init();
+	promisor_remote_init(r);
 
 	if (!remote_name)
-		return promisors;
+		return r->promisor_remote_config->promisors;
 
-	return promisor_remote_lookup(remote_name, NULL);
+	return promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);
 }
 
-int has_promisor_remote(void)
+int repo_has_promisor_remote(struct repository *r)
 {
-	return !!promisor_remote_find(NULL);
+	return !!repo_promisor_remote_find(r, NULL);
 }
 
 static int remove_fetched_oids(struct repository *repo,
@@ -235,9 +244,11 @@ int promisor_remote_get_direct(struct repository *repo,
 	if (oid_nr == 0)
 		return 0;
 
-	promisor_remote_init();
+	promisor_remote_init(repo);
 
-	for (r = promisors; r; r = r->next) {
+	if (repo != the_repository)
+		BUG("only the_repository is supported for now");
+	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
 		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
promisor-remote.h +17 -3
index 687210ab87..5390d3e7bf 100644
@@ -17,9 +17,23 @@ struct promisor_remote {
 	const char name[FLEX_ARRAY];
 };
 
-void promisor_remote_reinit(void);
-struct promisor_remote *promisor_remote_find(const char *remote_name);
-int has_promisor_remote(void);
+void repo_promisor_remote_reinit(struct repository *r);
+static inline void promisor_remote_reinit(void)
+{
+	repo_promisor_remote_reinit(the_repository);
+}
+
+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);
+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)
+{
+	return repo_promisor_remote_find(the_repository, remote_name);
+}
+
+int repo_has_promisor_remote(struct repository *r);
+static inline int has_promisor_remote(void)
+{
+	return repo_has_promisor_remote(the_repository);
+}
 
 /*
  * Fetches all requested objects from all promisor remotes, trying them one at
repository.h +4 -0
index a45f7520fd..fc06c154e2 100644
@@ -10,6 +10,7 @@ struct lock_file;
 struct pathspec;
 struct raw_object_store;
 struct submodule_cache;
+struct promisor_remote_config;
 
 enum untracked_cache_setting {
 	UNTRACKED_CACHE_UNSET = -1,
@@ -139,6 +140,9 @@ struct repository {
 	/* True if commit-graph has been disabled within this process. */
 	int commit_graph_disabled;
 
+	/* Configurations related to promisor remotes. */
+	struct promisor_remote_config *promisor_remote_config;
+
 	/* Configurations */
 
 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
Signature · 1 line
2.32.0.rc0.204.g9fa02ecfa5-goog
#11 lore raw

PATCH 3/4 run-command: move envvar-resetting function

In reply to Jonathan Tan

There is a function that resets environment variables, used when invoking a sub-process in a submodule. The lazy-fetching code (used in partial clones) will need this function in a subsequent commit, so move it to a more central location.

---
 run-command.c | 10 ++++++++++
 run-command.h |  7 +++++++
 submodule.c   | 14 ++------------
 3 files changed, 19 insertions(+), 12 deletions(-)
run-command.c +10 -0
index be6bc128cd..a6c458119c 100644
@@ -1892,3 +1892,13 @@ int run_auto_maintenance(int quiet)
 
 	return run_command(&maint);
 }
+
+void prepare_other_repo_env(struct strvec *env_array)
+{
+	const char * const *var;
+
+	for (var = local_repo_env; *var; var++) {
+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+			strvec_push(env_array, *var);
+	}
+}
run-command.h +7 -0
index d08414a92e..6f61ec7703 100644
@@ -483,4 +483,11 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
 			       task_finished_fn, void *pp_cb,
 			       const char *tr2_category, const char *tr2_label);
 
+/**
+ * Convenience function that adds entries to env_array that resets all
+ * repo-specific environment variables except for CONFIG_DATA_ENVIRONMENT. See
+ * local_repo_env in cache.h for more information.
+ */
+void prepare_other_repo_env(struct strvec *env_array);
+
 #endif
submodule.c +2 -12
index 0b1d9c1dde..a30216db52 100644
@@ -484,26 +484,16 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *
 	strbuf_release(&sb);
 }
 
-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
-{
-	const char * const *var;
-
-	for (var = local_repo_env; *var; var++) {
-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
-			strvec_push(out, *var);
-	}
-}
-
 void prepare_submodule_repo_env(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
+	prepare_other_repo_env(out);
 	strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
 		     DEFAULT_GIT_DIR_ENVIRONMENT);
 }
 
 static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
+	prepare_other_repo_env(out);
 	strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
 }
 
Signature · 1 line
2.32.0.rc0.204.g9fa02ecfa5-goog
#15 lore raw

PATCH 4/4 promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

---
 Makefile                      |  1 +
 object-file.c                 |  7 ++-----
 promisor-remote.c             | 14 +++++++++-----
 t/helper/test-partial-clone.c | 34 ++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c          |  1 +
 t/helper/test-tool.h          |  1 +
 t/t0410-partial-clone.sh      | 24 ++++++++++++++++++++++++
 7 files changed, 72 insertions(+), 10 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c
Makefile +1 -0
index c3565fc0f8..f6653bcd5e 100644
@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o
 TEST_BUILTINS_OBJS += test-online-cpus.o
 TEST_BUILTINS_OBJS += test-parse-options.o
 TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
+TEST_BUILTINS_OBJS += test-partial-clone.o
 TEST_BUILTINS_OBJS += test-path-utils.o
 TEST_BUILTINS_OBJS += test-pcre2-config.o
 TEST_BUILTINS_OBJS += test-pkt-line.o
object-file.c +2 -5
index f233b440b2..ebf273e9e7 100644
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
 		}
 
 		/* Check if it is a missing object */
-		if (fetch_if_missing && has_promisor_remote() &&
-		    !already_retried && r == the_repository &&
+		if (fetch_if_missing && repo_has_promisor_remote(r) &&
+		    !already_retried &&
 		    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
 			/*
 			 * TODO Investigate checking promisor_remote_get_direct()
 			 * TODO return value and stopping on error here.
-			 * TODO Pass a repository struct through
-			 * promisor_remote_get_direct(), such that arbitrary
-			 * repositories work.
 			 */
 			promisor_remote_get_direct(r, real, 1);
 			already_retried = 1;
promisor-remote.c +9 -5
index 5819d2cf28..1601f05d79 100644
@@ -11,7 +11,8 @@ struct promisor_remote_config {
 	struct promisor_remote **promisors_tail;
 };
 
-static int fetch_objects(const char *remote_name,
+static int fetch_objects(struct repository *repo,
+			 const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
 {
@@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,
 
 	child.git_cmd = 1;
 	child.in = -1;
+	if (repo != the_repository) {
+		prepare_other_repo_env(&child.env_array);
+		strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT,
+			     repo->gitdir);
+	}
 	strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop",
 		     "fetch", remote_name, "--no-tags",
 		     "--no-write-fetch-head", "--recurse-submodules=no",
@@ -150,7 +156,7 @@ static void promisor_remote_init(struct repository *r)
 		xcalloc(sizeof(*r->promisor_remote_config), 1);
 	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, config);
+	repo_config(r, promisor_remote_config, config);
 
 	if (config->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
@@ -246,10 +252,8 @@ int promisor_remote_get_direct(struct repository *repo,
 
 	promisor_remote_init(repo);
 
-	if (repo != the_repository)
-		BUG("only the_repository is supported for now");
 	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
-		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
+		if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
 			remaining_nr = remove_fetched_oids(repo, &remaining_oids,
t/helper/test-partial-clone.c new file +34 -0
new file mode 100644
index 0000000000..e7bc7eb21f
@@ -0,0 +1,34 @@
+#include "cache.h"
+#include "test-tool.h"
+#include "repository.h"
+#include "object-store.h"
+
+static void object_info(const char *gitdir, const char *oid_hex)
+{
+	struct repository r;
+	struct object_id oid;
+	unsigned long size;
+	struct object_info oi = {.sizep = &size};
+	const char *p;
+
+	if (repo_init(&r, gitdir, NULL))
+		die("could not init repo");
+	if (parse_oid_hex(oid_hex, &oid, &p))
+		die("could not parse oid");
+	if (oid_object_info_extended(&r, &oid, &oi, 0))
+		die("could not obtain object info");
+	printf("%d\n", (int) size);
+}
+
+int cmd__partial_clone(int argc, const char **argv)
+{
+	if (argc < 4)
+		die("too few arguments");
+
+	if (!strcmp(argv[1], "object-info"))
+		object_info(argv[2], argv[3]);
+	else
+		die("invalid argument '%s'", argv[1]);
+
+	return 0;
+}
t/helper/test-tool.c +1 -0
index c5bd0c6d4c..b21e8f1519 100644
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
 	{ "online-cpus", cmd__online_cpus },
 	{ "parse-options", cmd__parse_options },
 	{ "parse-pathspec-file", cmd__parse_pathspec_file },
+	{ "partial-clone", cmd__partial_clone },
 	{ "path-utils", cmd__path_utils },
 	{ "pcre2-config", cmd__pcre2_config },
 	{ "pkt-line", cmd__pkt_line },
t/helper/test-tool.h +1 -0
index e8069a3b22..f845ced4b3 100644
@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
+int cmd__partial_clone(int argc, const char **argv);
 int cmd__path_utils(int argc, const char **argv);
 int cmd__pcre2_config(int argc, const char **argv);
 int cmd__pkt_line(int argc, const char **argv);
t/t0410-partial-clone.sh +24 -0
index 584a039b85..e804d267e6 100755
@@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
 	git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+	rm -rf full partial.git &&
+	test_create_repo full &&
+	printf 12345 >full/file.txt &&
+	git -C full add file.txt &&
+	git -C full commit -m "first commit" &&
+
+	test_config -C full uploadpack.allowfilter 1 &&
+	test_config -C full uploadpack.allowanysha1inwant 1 &&
+	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+	FILE_HASH=$(git hash-object --stdin <full/file.txt) &&
+
+	# Sanity check that the file is missing
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	grep "[?]$FILE_HASH" out &&
+
+	OUT=$(test-tool partial-clone object-info partial.git "$FILE_HASH") &&
+	test "$OUT" -eq 5 &&
+
+	# Sanity check that the file is now present
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	! grep "[?]$FILE_HASH" out
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
Signature · 1 line
2.32.0.rc0.204.g9fa02ecfa5-goog
#2 lore raw

Re: [PATCH 1/4] promisor-remote: read partialClone config here

In reply to Jonathan Tan

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 98

Quoted 6 lines · Jonathan Tan
promisor-remote.c +2 -0
@@ -99,6 +94,11 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	size_t namelen;
 	const char *subkey;
 
+	if (!strcmp(var, "extensions.partialclone")) {
+		repository_format_partial_clone = xstrdup(value);

Can value ever be NULL here? I think the answer is "no, because we check earlier in setup.c:handle_extension_v0()", but there is an implicit conversion from xstrdup_or_null() to just xstrdup(), which would fault if value were to be NULL.

Looking deeper, this path is a little confusing to me, since (in the pre-image), handle_extension_v0() makes a copy of value and binds it to data->partial_clone. But then check_repository_format_gently() makes another copy of canidate->partial_clone (which is the same location as data->partial_clone).

So, the extra copy is a little strange to me, because even though the copy in handle_extension_v0() is definitely necessary, I'm not certain that the one in set_repository_format_partial_clone() is. And this patch removes the latter one, which I think is good. But we never free repository_format_partial_clone.

Maybe that is added in a later patch, let's see...

Thanks,
Taylor

#6 lore raw

Re: [PATCH 2/4] promisor-remote: support per-repository config

In reply to Jonathan Tan
Quoted 9 lines · Jonathan Tan

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation.

Makes sense to me. I found my answer to the question that I raised during my review of the previous patch, and I think it would make sense to address in an amended version of this patch.

Other than that, the translation all looked very faithful to me.

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 181

Quoted 7 lines
promisor-remote.c +2 -4
-void promisor_remote_reinit(void)
+void repo_promisor_remote_reinit(struct repository *r)
 {
-	initialized = 0;
-	promisor_remote_clear();
-	promisor_remote_init();
+	promisor_remote_clear(r->promisor_remote_config);

Ah, this is probably where I would have expected to see r->promisor_remote_config->repository_format_partial_clone freed as well.

I wondered whether or not that should have been freed, since on first read it seemed that this function was mostly concerned with the list of promisor remotes rather than the structure containing them. But on a closer look, we are re-initializing the whole structure with promisor_remote_init(), which runs the whole promisor_remote_config callback again.

So I do think we want to free that part of the structure, too, before reinitializing it. I would probably do it in promisor_remote_clear().

promisor-remote.c int promisor_remote_get_direct(struct repository *repo, · line 250

Quoted 10 lines
promisor-remote.c +3 -2
@@ -235,9 +244,11 @@ int promisor_remote_get_direct(struct repository *repo,
 	if (oid_nr == 0)
 		return 0;
 
-	promisor_remote_init();
+	promisor_remote_init(repo);
 
-	for (r = promisors; r; r = r->next) {
+	if (repo != the_repository)
+		BUG("only the_repository is supported for now");

I could go either way on whether this is worthy of a BUG() or not, but I don't really have much of a strong feeling about it.

Thanks,
Taylor

#12 lore raw

Re: [PATCH 3/4] run-command: move envvar-resetting function

In reply to Jonathan Tan
Quoted 6 lines · Jonathan Tan

There is a function that resets environment variables, used when invoking a sub-process in a submodule. The lazy-fetching code (used in partial clones) will need this function in a subsequent commit, so move it to a more central location.

All seems pretty normal to me. I did have one question, though:

Quoted 2 lines

+/**
+ * Convenience function that adds entries to env_array that resets all

Hmm. Why "resets"? IIUC local_repo_env is the array of environment variables that change behavior. With that understanding in mind, I probably would have written something more like:

Convenience function which adds all GIT_* environment variables to env_array with the exception of GIT_CONFIG_PARAMETERS. See local_repo_env in cache.h for more information.

(Confusingly, cache.h calls this variable CONFIG_DATA_ENVIRONMENT, but binds it to GIT_CONFIG_PARAMETERS. I think it probably makes more sense to use the environment variable's name rather than our #define, since we're saying "all GIT_* variables, except this one", so it would be weird for "this one" not to start with "GIT_".

Otherwise the movement looks fine to me.

Thanks,
Taylor

#8 lore raw

Re: [PATCH 2/4] promisor-remote: support per-repository config

In reply to Jonathan Tan
Quoted 252 lines · Jonathan Tan

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation.

---
 promisor-remote.c | 101 +++++++++++++++++++++++++---------------------
 promisor-remote.h |  20 +++++++--
 repository.h      |   4 ++
 3 files changed, 77 insertions(+), 48 deletions(-)
promisor-remote.c +56 -45
index bfe8eee5f2..5819d2cf28 100644
@@ -5,7 +5,11 @@
 #include "transport.h"
 #include "strvec.h"
 
-static char *repository_format_partial_clone;
+struct promisor_remote_config {
+       char *repository_format_partial_clone;
+       struct promisor_remote *promisors;
+       struct promisor_remote **promisors_tail;
+};
 
 static int fetch_objects(const char *remote_name,
                         const struct object_id *oids,
@@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name,
        return finish_command(&child) ? -1 : 0;
 }
 
-static struct promisor_remote *promisors;
-static struct promisor_remote **promisors_tail = &promisors;
-
-static struct promisor_remote *promisor_remote_new(const char *remote_name)
+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,
+                                                  const char *remote_name)
 {
        struct promisor_remote *r;
 
@@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)
 
        FLEX_ALLOC_STR(r, name, remote_name);
 
-       *promisors_tail = r;
-       promisors_tail = &r->next;
+       *config->promisors_tail = r;
+       config->promisors_tail = &r->next;
 
        return r;
 }
 
-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,
+                                                     const char *remote_name,
                                                      struct promisor_remote **previous)
 {
        struct promisor_remote *r, *p;
 
-       for (p = NULL, r = promisors; r; p = r, r = r->next)
+       for (p = NULL, r = config->promisors; r; p = r, r = r->next)
                if (!strcmp(r->name, remote_name)) {
                        if (previous)
                                *previous = p;
@@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
        return NULL;
 }
 
-static void promisor_remote_move_to_tail(struct promisor_remote *r,
+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,
+                                        struct promisor_remote *r,
                                         struct promisor_remote *previous)
 {
        if (r->next == NULL)
@@ -82,20 +86,21 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,
        if (previous)
                previous->next = r->next;
        else
-               promisors = r->next ? r->next : r;
+               config->promisors = r->next ? r->next : r;
        r->next = NULL;
-       *promisors_tail = r;
-       promisors_tail = &r->next;
+       *config->promisors_tail = r;
+       config->promisors_tail = &r->next;
 }
 
 static int promisor_remote_config(const char *var, const char *value, void *data)
 {
+       struct promisor_remote_config *config = data;
        const char *name;
        size_t namelen;
        const char *subkey;
 
        if (!strcmp(var, "extensions.partialclone")) {
-               repository_format_partial_clone = xstrdup(value);
+               config->repository_format_partial_clone = xstrdup(value);
                return 0;
        }
 
@@ -110,8 +115,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 
                remote_name = xmemdupz(name, namelen);
 
-               if (!promisor_remote_lookup(remote_name, NULL))
-                       promisor_remote_new(remote_name);
+               if (!promisor_remote_lookup(config, remote_name, NULL))
+                       promisor_remote_new(config, remote_name);
 
                free(remote_name);
                return 0;
@@ -120,9 +125,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data
                struct promisor_remote *r;
                char *remote_name = xmemdupz(name, namelen);
 
-               r = promisor_remote_lookup(remote_name, NULL);
+               r = promisor_remote_lookup(config, remote_name, NULL);
                if (!r)
-                       r = promisor_remote_new(remote_name);
+                       r = promisor_remote_new(config, remote_name);
 
                free(remote_name);
 
@@ -135,59 +140,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data
        return 0;
 }
 
-static int initialized;
-
-static void promisor_remote_init(void)
+static void promisor_remote_init(struct repository *r)
 {
-       if (initialized)
+       struct promisor_remote_config *config;
+
+       if (r->promisor_remote_config)
                return;
-       initialized = 1;
+       config = r->promisor_remote_config =
+               xcalloc(sizeof(*r->promisor_remote_config), 1);
+       config->promisors_tail = &config->promisors;
 
-       git_config(promisor_remote_config, NULL);
+       git_config(promisor_remote_config, config);
 
-       if (repository_format_partial_clone) {
+       if (config->repository_format_partial_clone) {
                struct promisor_remote *o, *previous;
 
-               o = promisor_remote_lookup(repository_format_partial_clone,
+               o = promisor_remote_lookup(config,
+                                          config->repository_format_partial_clone,
                                           &previous);
                if (o)
-                       promisor_remote_move_to_tail(o, previous);
+                       promisor_remote_move_to_tail(config, o, previous);
                else
-                       promisor_remote_new(repository_format_partial_clone);
+                       promisor_remote_new(config, config->repository_format_partial_clone);
        }
 }
 
-static void promisor_remote_clear(void)
+static void promisor_remote_clear(struct promisor_remote_config *config)
 {
-       while (promisors) {
-               struct promisor_remote *r = promisors;
-               promisors = promisors->next;
+       while (config->promisors) {
+               struct promisor_remote *r = config->promisors;
+               config->promisors = config->promisors->next;
                free(r);
        }
 
-       promisors_tail = &promisors;
+       config->promisors_tail = &config->promisors;
 }
 
-void promisor_remote_reinit(void)
+void repo_promisor_remote_reinit(struct repository *r)
 {
-       initialized = 0;
-       promisor_remote_clear();
-       promisor_remote_init();
+       promisor_remote_clear(r->promisor_remote_config);
+       FREE_AND_NULL(r->promisor_remote_config);
+       promisor_remote_init(r);
 }
 
-struct promisor_remote *promisor_remote_find(const char *remote_name)
+struct promisor_remote *repo_promisor_remote_find(struct repository *r,
+                                                 const char *remote_name)
 {
-       promisor_remote_init();
+       promisor_remote_init(r);
 
        if (!remote_name)
-               return promisors;
+               return r->promisor_remote_config->promisors;
 
-       return promisor_remote_lookup(remote_name, NULL);
+       return promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);
 }
 
-int has_promisor_remote(void)
+int repo_has_promisor_remote(struct repository *r)
 {
-       return !!promisor_remote_find(NULL);
+       return !!repo_promisor_remote_find(r, NULL);
 }
 
 static int remove_fetched_oids(struct repository *repo,
@@ -235,9 +244,11 @@ int promisor_remote_get_direct(struct repository *repo,
        if (oid_nr == 0)
                return 0;
 
-       promisor_remote_init();
+       promisor_remote_init(repo);
 
-       for (r = promisors; r; r = r->next) {
+       if (repo != the_repository)
+               BUG("only the_repository is supported for now");
+       for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
                if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
                        if (remaining_nr == 1)
                                continue;
promisor-remote.h +7 -3
index 687210ab87..5390d3e7bf 100644
@@ -17,9 +17,23 @@ struct promisor_remote {
        const char name[FLEX_ARRAY];
 };
 
-void promisor_remote_reinit(void);
-struct promisor_remote *promisor_remote_find(const char *remote_name);
-int has_promisor_remote(void);
+void repo_promisor_remote_reinit(struct repository *r);
+static inline void promisor_remote_reinit(void)
+{
+       repo_promisor_remote_reinit(the_repository);
+}
+
+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);

promisor-remote.h struct promisor_remote { · line 36

Quoted 10 lines
promisor-remote.h +10 -0
+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)
+{
+       return repo_promisor_remote_find(the_repository, remote_name);
+}
+
+int repo_has_promisor_remote(struct repository *r);
+static inline int has_promisor_remote(void)
+{
+       return repo_has_promisor_remote(the_repository);
+}

Is part of the plan for supporting partial clones within submodules to audit the code for use of these inline wrappers and convert them over to the repo_* variants? I'm particularly interested in the has_promisor_remote() function, since there are calls in diffcore-rename at least that protect that call with a check against r == the_repository.

Quoted 26 lines
promisor-remote.h +0 -0
 /*
  * Fetches all requested objects from all promisor remotes, trying them one at
repository.h +4 -1
index a45f7520fd..fc06c154e2 100644
@@ -10,6 +10,7 @@ struct lock_file;
 struct pathspec;
 struct raw_object_store;
 struct submodule_cache;
+struct promisor_remote_config;
 
 enum untracked_cache_setting {
        UNTRACKED_CACHE_UNSET = -1,
@@ -139,6 +140,9 @@ struct repository {
        /* True if commit-graph has been disabled within this process. */
        int commit_graph_disabled;
 
+       /* Configurations related to promisor remotes. */
+       struct promisor_remote_config *promisor_remote_config;
+
        /* Configurations */
 
        /* Indicate if a repository has a different 'commondir' from 'gitdir' */
--
Quoted 1 line

2.32.0.rc0.204.g9fa02ecfa5-goog

Looks like a reasonable step in moving away from globals and have repository-specific variants of these functions; I didn't spot any problems, just one question about additional plans.

#16 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan
Quoted 9 lines · Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

OK. Everything you wrote seemed reasonable to me, but I did have a couple of questions on the test you added:

Quoted 17 lines
t/helper/test-partial-clone.c new file +11 -0
new file mode 100644
index 0000000000..e7bc7eb21f
@@ -0,0 +1,34 @@
+#include "cache.h"
+#include "test-tool.h"
+#include "repository.h"
+#include "object-store.h"
+
+static void object_info(const char *gitdir, const char *oid_hex)
+{
+	struct repository r;
+	struct object_id oid;
+	unsigned long size;
+	struct object_info oi = {.sizep = &size};

t/helper/test-partial-clone.c line 21

Quoted 10 lines
t/helper/test-partial-clone.c new file +10 -0
+	const char *p;
+
+	if (repo_init(&r, gitdir, NULL))
+		die("could not init repo");
+	if (parse_oid_hex(oid_hex, &oid, &p))
+		die("could not parse oid");
+	if (oid_object_info_extended(&r, &oid, &oi, 0))
+		die("could not obtain object info");
+	printf("%d\n", (int) size);
+}

Hmm. Is there a reason that the same couldn't be implemented by calling "git cat-file -s" from the partial clone?

Quoted 5 lines
t/t0410-partial-clone.sh +0 -0
index 584a039b85..e804d267e6 100755

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 612

Quoted 9 lines
t/t0410-partial-clone.sh +6 -0
@@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
 	git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+	rm -rf full partial.git &&
+	test_create_repo full &&
+	printf 12345 >full/file.txt &&
+	git -C full add file.txt &&
+	git -C full commit -m "first commit" &&

This is a stylistic nit, but I think using test_commit is better here for a non-superficial reason. My guess is that you wanted to avoid specifying a message and file (which are required positional arguments to test_commit before you can specify the contents). But I think there are two good reasons to use test_commit here:

- It saves three lines of test script here.
- You don't have to make the expected size a magic number (i.e., because you knew ahead of time that the contents was "12345").

I probably would have expected this test to end with:

git -C full cat-file -s $FILE_HASH >expect && git -C partial.git cat-file -s $FILE_HASH >actual && test_cmp expect actual

which reads more clearly to me (although I think the much more important test is that $FILE_HASH doesn't show up in the output of the rev-list --missing=print that is run in the partial clone).

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 617

Quoted 5 lines
t/t0410-partial-clone.sh +5 -0
+
+	test_config -C full uploadpack.allowfilter 1 &&
+	test_config -C full uploadpack.allowanysha1inwant 1 &&
+	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+	FILE_HASH=$(git hash-object --stdin <full/file.txt) &&

This works for me, although I wouldn't have been sad to see the sub-shell contain "git -C full rev-parse HEAD:file.txt" instead.

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 623

Quoted 5 lines
t/t0410-partial-clone.sh +5 -0
+	# Sanity check that the file is missing
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	grep "[?]$FILE_HASH" out &&
+
+	OUT=$(test-tool partial-clone object-info partial.git "$FILE_HASH") &&

Coming back to my point about the utility of the partial-clone helper, could this be replaced by saying just OUT="$(git -C partial.git cat-file -s "$FILE_HASH")" instead?

Thanks,
Taylor

#18 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan
Quoted 40 lines · Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

---
 Makefile                      |  1 +
 object-file.c                 |  7 ++-----
 promisor-remote.c             | 14 +++++++++-----
 t/helper/test-partial-clone.c | 34 ++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c          |  1 +
 t/helper/test-tool.h          |  1 +
 t/t0410-partial-clone.sh      | 24 ++++++++++++++++++++++++
 7 files changed, 72 insertions(+), 10 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c
Makefile +1 -0
index c3565fc0f8..f6653bcd5e 100644
@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o
 TEST_BUILTINS_OBJS += test-online-cpus.o
 TEST_BUILTINS_OBJS += test-parse-options.o
 TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
+TEST_BUILTINS_OBJS += test-partial-clone.o
 TEST_BUILTINS_OBJS += test-path-utils.o
 TEST_BUILTINS_OBJS += test-pcre2-config.o
 TEST_BUILTINS_OBJS += test-pkt-line.o
object-file.c +0 -0
index f233b440b2..ebf273e9e7 100644

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1574

Quoted 7 lines
object-file.c +2 -2
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
                }
 
                /* Check if it is a missing object */
-               if (fetch_if_missing && has_promisor_remote() &&
-                   !already_retried && r == the_repository &&
+               if (fetch_if_missing && repo_has_promisor_remote(r) &&
+                   !already_retried &&

So here you removed the special check against the_repository while looking for promisor_remotes. There are other such special checks in the code; I also see:

diff.c: if (options->repo == the_repository && has_promisor_remote() &&
diffcore-break.c:       if (r == the_repository && has_promisor_remote()) {
diffcore-rename.c:      if (r == the_repository && has_promisor_remote()) {

and a series I'm planning to submit soon will add another to merge.ort.c.

Do these need to all be fixed as part of the partial clone submodule support as well? Do I need to change anything about my series? I guess since I'm asking that, I should probably submit it first so you can actually see it and answer my question. (And the timing may be good since the area is fresh in your memory...)

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1581

Quoted 7 lines
object-file.c +0 -3
                    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
                        /*
                         * TODO Investigate checking promisor_remote_get_direct()
                         * TODO return value and stopping on error here.
-                        * TODO Pass a repository struct through
-                        * promisor_remote_get_direct(), such that arbitrary
-                        * repositories work.

Odd, it appears that when this comment was added (in commit b14ed5adaf ("Use promisor_remote_get_direct() and has_promisor_remote()", 2019-06-25)), a repository was passed to promisor_remote_get_direct(). Sure, it was just a transliteration of the comment that was there before when fetch_objects() was the function being called, but since the code was being changed and the comment being updated, it seems the TODO should have been removed back then.

Oh, well, good to update it now at least.

Quoted 150 lines
object-file.c +0 -0
                         */
                        promisor_remote_get_direct(r, real, 1);
                        already_retried = 1;
promisor-remote.c +9 -5
index 5819d2cf28..1601f05d79 100644
@@ -11,7 +11,8 @@ struct promisor_remote_config {
        struct promisor_remote **promisors_tail;
 };
 
-static int fetch_objects(const char *remote_name,
+static int fetch_objects(struct repository *repo,
+                        const char *remote_name,
                         const struct object_id *oids,
                         int oid_nr)
 {
@@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,
 
        child.git_cmd = 1;
        child.in = -1;
+       if (repo != the_repository) {
+               prepare_other_repo_env(&child.env_array);
+               strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT,
+                            repo->gitdir);
+       }
        strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop",
                     "fetch", remote_name, "--no-tags",
                     "--no-write-fetch-head", "--recurse-submodules=no",
@@ -150,7 +156,7 @@ static void promisor_remote_init(struct repository *r)
                xcalloc(sizeof(*r->promisor_remote_config), 1);
        config->promisors_tail = &config->promisors;
 
-       git_config(promisor_remote_config, config);
+       repo_config(r, promisor_remote_config, config);
 
        if (config->repository_format_partial_clone) {
                struct promisor_remote *o, *previous;
@@ -246,10 +252,8 @@ int promisor_remote_get_direct(struct repository *repo,
 
        promisor_remote_init(repo);
 
-       if (repo != the_repository)
-               BUG("only the_repository is supported for now");
        for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
-               if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
+               if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {
                        if (remaining_nr == 1)
                                continue;
                        remaining_nr = remove_fetched_oids(repo, &remaining_oids,
t/helper/test-partial-clone.c new file +34 -0
new file mode 100644
index 0000000000..e7bc7eb21f
@@ -0,0 +1,34 @@
+#include "cache.h"
+#include "test-tool.h"
+#include "repository.h"
+#include "object-store.h"
+
+static void object_info(const char *gitdir, const char *oid_hex)
+{
+       struct repository r;
+       struct object_id oid;
+       unsigned long size;
+       struct object_info oi = {.sizep = &size};
+       const char *p;
+
+       if (repo_init(&r, gitdir, NULL))
+               die("could not init repo");
+       if (parse_oid_hex(oid_hex, &oid, &p))
+               die("could not parse oid");
+       if (oid_object_info_extended(&r, &oid, &oi, 0))
+               die("could not obtain object info");
+       printf("%d\n", (int) size);
+}
+
+int cmd__partial_clone(int argc, const char **argv)
+{
+       if (argc < 4)
+               die("too few arguments");
+
+       if (!strcmp(argv[1], "object-info"))
+               object_info(argv[2], argv[3]);
+       else
+               die("invalid argument '%s'", argv[1]);
+
+       return 0;
+}
t/helper/test-tool.c +1 -0
index c5bd0c6d4c..b21e8f1519 100644
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
        { "online-cpus", cmd__online_cpus },
        { "parse-options", cmd__parse_options },
        { "parse-pathspec-file", cmd__parse_pathspec_file },
+       { "partial-clone", cmd__partial_clone },
        { "path-utils", cmd__path_utils },
        { "pcre2-config", cmd__pcre2_config },
        { "pkt-line", cmd__pkt_line },
t/helper/test-tool.h +1 -0
index e8069a3b22..f845ced4b3 100644
@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
+int cmd__partial_clone(int argc, const char **argv);
 int cmd__path_utils(int argc, const char **argv);
 int cmd__pcre2_config(int argc, const char **argv);
 int cmd__pkt_line(int argc, const char **argv);
t/t0410-partial-clone.sh +24 -1
index 584a039b85..e804d267e6 100755
@@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
        git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+       rm -rf full partial.git &&
+       test_create_repo full &&
+       printf 12345 >full/file.txt &&
+       git -C full add file.txt &&
+       git -C full commit -m "first commit" &&
+
+       test_config -C full uploadpack.allowfilter 1 &&
+       test_config -C full uploadpack.allowanysha1inwant 1 &&
+       git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+       FILE_HASH=$(git hash-object --stdin <full/file.txt) &&
+
+       # Sanity check that the file is missing
+       git -C partial.git rev-list --objects --missing=print HEAD >out &&
+       grep "[?]$FILE_HASH" out &&
+
+       OUT=$(test-tool partial-clone object-info partial.git "$FILE_HASH") &&
+       test "$OUT" -eq 5 &&
+
+       # Sanity check that the file is now present
+       git -C partial.git rev-list --objects --missing=print HEAD >out &&
+       ! grep "[?]$FILE_HASH" out
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
--
Quoted 1 line

2.32.0.rc0.204.g9fa02ecfa5-goog

Looks good to me.

#21 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan

Hi,

Quoted 11 lines · Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

...

Quoted 86 lines
t/helper/test-partial-clone.c new file +34 -0
new file mode 100644
index 0000000000..e7bc7eb21f
@@ -0,0 +1,34 @@
+#include "cache.h"
+#include "test-tool.h"
+#include "repository.h"
+#include "object-store.h"
+
+static void object_info(const char *gitdir, const char *oid_hex)
+{
+       struct repository r;
+       struct object_id oid;
+       unsigned long size;
+       struct object_info oi = {.sizep = &size};
+       const char *p;
+
+       if (repo_init(&r, gitdir, NULL))
+               die("could not init repo");
+       if (parse_oid_hex(oid_hex, &oid, &p))
+               die("could not parse oid");
+       if (oid_object_info_extended(&r, &oid, &oi, 0))
+               die("could not obtain object info");
+       printf("%d\n", (int) size);
+}
+
+int cmd__partial_clone(int argc, const char **argv)
+{
+       if (argc < 4)
+               die("too few arguments");
+
+       if (!strcmp(argv[1], "object-info"))
+               object_info(argv[2], argv[3]);
+       else
+               die("invalid argument '%s'", argv[1]);
+
+       return 0;
+}
t/helper/test-tool.c +1 -0
index c5bd0c6d4c..b21e8f1519 100644
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
        { "online-cpus", cmd__online_cpus },
        { "parse-options", cmd__parse_options },
        { "parse-pathspec-file", cmd__parse_pathspec_file },
+       { "partial-clone", cmd__partial_clone },
        { "path-utils", cmd__path_utils },
        { "pcre2-config", cmd__pcre2_config },
        { "pkt-line", cmd__pkt_line },
t/helper/test-tool.h +1 -0
index e8069a3b22..f845ced4b3 100644
@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
+int cmd__partial_clone(int argc, const char **argv);
 int cmd__path_utils(int argc, const char **argv);
 int cmd__pcre2_config(int argc, const char **argv);
 int cmd__pkt_line(int argc, const char **argv);
t/t0410-partial-clone.sh +14 -0
index 584a039b85..e804d267e6 100755
@@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
        git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+       rm -rf full partial.git &&
+       test_create_repo full &&
+       printf 12345 >full/file.txt &&
+       git -C full add file.txt &&
+       git -C full commit -m "first commit" &&
+
+       test_config -C full uploadpack.allowfilter 1 &&
+       test_config -C full uploadpack.allowanysha1inwant 1 &&
+       git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+       FILE_HASH=$(git hash-object --stdin <full/file.txt) &&
+
+       # Sanity check that the file is missing
+       git -C partial.git rev-list --objects --missing=print HEAD >out &&

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 630

Quoted 10 lines
t/t0410-partial-clone.sh +10 -0
+       grep "[?]$FILE_HASH" out &&
+
+       OUT=$(test-tool partial-clone object-info partial.git "$FILE_HASH") &&
+       test "$OUT" -eq 5 &&
+
+       # Sanity check that the file is now present
+       git -C partial.git rev-list --objects --missing=print HEAD >out &&
+       ! grep "[?]$FILE_HASH" out
+'
+

Turns out that this test fails under GIT_TEST_DEFAULT_HASH=sha256; output:

error: wrong index v2 file size in /home/newren/floss/git/t/trash directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx
error: wrong index v2 file size in /home/newren/floss/git/t/trash directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx
fatal: couldn't find remote ref 74242c6e4a0d89f454d89d3496a1f7cb3f1f39f0
error: wrong index v2 file size in /home/newren/floss/git/t/trash directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx
error: wrong index v2 file size in /home/newren/floss/git/t/trash directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx
fatal: could not obtain object info

#3 lore raw

Re: [PATCH 1/4] promisor-remote: read partialClone config here

In reply to Taylor Blau
Quoted 5 lines · Jonathan Tan
Quoted 4 lines
promisor-remote.c +0 -0
@@ -99,6 +94,11 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	size_t namelen;
 	const char *subkey;
 
Quoted 7 lines

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 98

Quoted 2 lines
promisor-remote.c +2 -0
+	if (!strcmp(var, "extensions.partialclone")) {
+		repository_format_partial_clone = xstrdup(value);

Can value ever be NULL here? I think the answer is "no, because we check earlier in setup.c:handle_extension_v0()", but there is an implicit conversion from xstrdup_or_null() to just xstrdup(), which would fault if value were to be NULL.

Ah, good catch. I think you're right, but it's better to be defensive here. I'll add that check and also say that this will be caught by setup.c.

Quoted 11 lines

Looking deeper, this path is a little confusing to me, since (in the pre-image), handle_extension_v0() makes a copy of value and binds it to data->partial_clone. But then check_repository_format_gently() makes another copy of canidate->partial_clone (which is the same location as data->partial_clone).

So, the extra copy is a little strange to me, because even though the copy in handle_extension_v0() is definitely necessary, I'm not certain that the one in set_repository_format_partial_clone() is. And this patch removes the latter one, which I think is good. But we never free repository_format_partial_clone.

Yes, it's also a simplification that we go straight from config to variable here, instead of going through the candidate struct. As for freeing repository_format_partial_clone, this is run once and never again (guarded by "initialized" check) so there is no prior value to free, and I don't think this variable needs to be freed.

Quoted 1 line

Maybe that is added in a later patch, let's see...

So I think things are fine here, but in a later patch, repository_format_partial_clone is moved to the struct repository, which can be cleared with repo_clear(). I'll need to add a free there :-)

Quoted 2 lines

Thanks,
Taylor

Thanks for looking at the patch set too.

#7 lore raw

Re: [PATCH 2/4] promisor-remote: support per-repository config

In reply to Taylor Blau
Quoted 6 lines · Jonathan Tan
Quoted 5 lines

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

Quoted 8 lines
Quoted 4 lines

The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation.

Makes sense to me. I found my answer to the question that I raised during my review of the previous patch, and I think it would make sense to address in an amended version of this patch.

Just to clarify, what is the question and what is the answer?

Quoted 21 lines

Other than that, the translation all looked very faithful to me.

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 181

Quoted 7 lines
promisor-remote.c +2 -4
-void promisor_remote_reinit(void)
+void repo_promisor_remote_reinit(struct repository *r)
 {
-	initialized = 0;
-	promisor_remote_clear();
-	promisor_remote_init();
+	promisor_remote_clear(r->promisor_remote_config);

Ah, this is probably where I would have expected to see r->promisor_remote_config->repository_format_partial_clone freed as well.

I wondered whether or not that should have been freed, since on first read it seemed that this function was mostly concerned with the list of promisor remotes rather than the structure containing them. But on a closer look, we are re-initializing the whole structure with promisor_remote_init(), which runs the whole promisor_remote_config callback again.

Quoted 2 lines

So I do think we want to free that part of the structure, too, before reinitializing it. I would probably do it in promisor_remote_clear().

I'll add that in the next version.

Quoted 7 lines
Quoted 7 lines
promisor-remote.c +1 -1
@@ -235,9 +244,11 @@ int promisor_remote_get_direct(struct repository *repo,
 	if (oid_nr == 0)
 		return 0;
 
-	promisor_remote_init();
+	promisor_remote_init(repo);
 
Quoted 6 lines

promisor-remote.c int promisor_remote_get_direct(struct repository *repo, · line 250

Quoted 3 lines
promisor-remote.c +2 -1
-	for (r = promisors; r; r = r->next) {
+	if (repo != the_repository)
+		BUG("only the_repository is supported for now");

I could go either way on whether this is worthy of a BUG() or not, but I don't really have much of a strong feeling about it.

This BUG() will be removed in patch 4. I'm OK either way (adding BUG() here and removing it in patch 4, or never adding BUG() in the first place).

#9 lore raw

Re: [PATCH 2/4] promisor-remote: support per-repository config

In reply to Elijah Newren
Quoted 4 lines
Quoted 4 lines
promisor-remote.h +4 -0
+int repo_has_promisor_remote(struct repository *r);
+static inline int has_promisor_remote(void)
+{
+       return repo_has_promisor_remote(the_repository);
Quoted 8 lines
Quoted 1 line

+}

Is part of the plan for supporting partial clones within submodules to audit the code for use of these inline wrappers and convert them over to the repo_* variants? I'm particularly interested in the has_promisor_remote() function, since there are calls in diffcore-rename at least that protect that call with a check against r == the_repository.

Good point. Yes, that should be part of the plan - at least, we should see what those invocations are for.

Quoted 9 lines
Quoted 9 lines
repository.h +3 -0
@@ -139,6 +140,9 @@ struct repository {
        /* True if commit-graph has been disabled within this process. */
        int commit_graph_disabled;
 
+       /* Configurations related to promisor remotes. */
+       struct promisor_remote_config *promisor_remote_config;
+
        /* Configurations */
 
Quoted 7 lines
Quoted 3 lines
(quoted diff) +0 -1
        /* Indicate if a repository has a different 'commondir' from 'gitdir' */
--

2.32.0.rc0.204.g9fa02ecfa5-goog

Looks like a reasonable step in moving away from globals and have repository-specific variants of these functions; I didn't spot any problems, just one question about additional plans.

Thanks for taking a look.

#13 lore raw

Re: [PATCH 3/4] run-command: move envvar-resetting function

In reply to Taylor Blau
Quoted 13 lines · Jonathan Tan
Quoted 6 lines

There is a function that resets environment variables, used when invoking a sub-process in a submodule. The lazy-fetching code (used in partial clones) will need this function in a subsequent commit, so move it to a more central location.

All seems pretty normal to me. I did have one question, though:

Quoted 2 lines

+/**
+ * Convenience function that adds entries to env_array that resets all

Quoted 7 lines

Hmm. Why "resets"? IIUC local_repo_env is the array of environment variables that change behavior. With that understanding in mind, I probably would have written something more like:

Convenience function which adds all GIT_* environment variables to env_array with the exception of GIT_CONFIG_PARAMETERS. See local_repo_env in cache.h for more information.

I mentioned "reset" because the effect of adding the name without any value makes the environment variable of that name unset in the subprocess. I'll word it as you say, and add "When used as the env_array of a subprocess, these entries cause the corresponding environment variables to be unset in the subprocess." after the first sentence.

Quoted 5 lines

(Confusingly, cache.h calls this variable CONFIG_DATA_ENVIRONMENT, but binds it to GIT_CONFIG_PARAMETERS. I think it probably makes more sense to use the environment variable's name rather than our #define, since we're saying "all GIT_* variables, except this one", so it would be weird for "this one" not to start with "GIT_".

OK, that makes sense.

Quoted 4 lines

Otherwise the movement looks fine to me.

Thanks,
Taylor

Thanks.

#17 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Taylor Blau
Quoted 11 lines
Quoted 11 lines
t/helper/test-partial-clone.c new file +11 -0
+static void object_info(const char *gitdir, const char *oid_hex)
+{
+	struct repository r;
+	struct object_id oid;
+	unsigned long size;
+	struct object_info oi = {.sizep = &size};
+	const char *p;
+
+	if (repo_init(&r, gitdir, NULL))
+		die("could not init repo");
+	if (parse_oid_hex(oid_hex, &oid, &p))
Quoted 8 lines

t/helper/test-partial-clone.c line 21

Quoted 5 lines
t/helper/test-partial-clone.c new file +5 -0
+		die("could not parse oid");
+	if (oid_object_info_extended(&r, &oid, &oi, 0))
+		die("could not obtain object info");
+	printf("%d\n", (int) size);
+}

Hmm. Is there a reason that the same couldn't be implemented by calling "git cat-file -s" from the partial clone?

I don't think "git cat-file" (when run in the superproject) by itself can access an object from a submodule. "git -C name_of_submodule cat-file $HASH" would access that object, but I specifically want to test oid_object_info_extended() on a repo that is not the_repository (which would not work with -C, because the_repository would then be the submodule).

Quoted 19 lines

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 612

Quoted 6 lines
t/t0410-partial-clone.sh +6 -0
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+	rm -rf full partial.git &&
+	test_create_repo full &&
+	printf 12345 >full/file.txt &&
+	git -C full add file.txt &&
+	git -C full commit -m "first commit" &&

This is a stylistic nit, but I think using test_commit is better here for a non-superficial reason. My guess is that you wanted to avoid specifying a message and file (which are required positional arguments to test_commit before you can specify the contents). But I think there are two good reasons to use test_commit here:

- It saves three lines of test script here.
- You don't have to make the expected size a magic number (i.e., because you knew ahead of time that the contents was "12345").

I probably would have expected this test to end with:

Quoted 7 lines

git -C full cat-file -s $FILE_HASH >expect && git -C partial.git cat-file -s $FILE_HASH >actual && test_cmp expect actual

which reads more clearly to me (although I think the much more important test is that $FILE_HASH doesn't show up in the output of the rev-list --missing=print that is run in the partial clone).

That makes sense.

Quoted 7 lines

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 617

Quoted 4 lines
t/t0410-partial-clone.sh +4 -0
+	test_config -C full uploadpack.allowfilter 1 &&
+	test_config -C full uploadpack.allowanysha1inwant 1 &&
+	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+	FILE_HASH=$(git hash-object --stdin <full/file.txt) &&

This works for me, although I wouldn't have been sad to see the sub-shell contain "git -C full rev-parse HEAD:file.txt" instead.

I'll do this too.

Quoted 10 lines

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 623

Quoted 5 lines
t/t0410-partial-clone.sh +5 -0
+	# Sanity check that the file is missing
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	grep "[?]$FILE_HASH" out &&
+
+	OUT=$(test-tool partial-clone object-info partial.git "$FILE_HASH") &&

Coming back to my point about the utility of the partial-clone helper, could this be replaced by saying just OUT="$(git -C partial.git cat-file -s "$FILE_HASH")" instead?

Quoted 2 lines

Thanks,
Taylor

Same answer as above - I specifically want to test accessing (and thereby lazy-fetching) an object when the object is not in the_repository. I'll add some documentation to explain what it does and that it's equivalent to using "git -C repo cat-file -s", except that this specifically tests another code path.

#19 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Elijah Newren
Quoted 21 lines

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1574

Quoted 12 lines
object-file.c +2 -2
index f233b440b2..ebf273e9e7 100644
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
                }
 
                /* Check if it is a missing object */
-               if (fetch_if_missing && has_promisor_remote() &&
-                   !already_retried && r == the_repository &&
+               if (fetch_if_missing && repo_has_promisor_remote(r) &&
+                   !already_retried &&

So here you removed the special check against the_repository while looking for promisor_remotes. There are other such special checks in the code; I also see:

diff.c: if (options->repo == the_repository && has_promisor_remote() &&
diffcore-break.c:       if (r == the_repository && has_promisor_remote()) {
diffcore-rename.c:      if (r == the_repository && has_promisor_remote()) {
Quoted 7 lines

and a series I'm planning to submit soon will add another to merge.ort.c.

Do these need to all be fixed as part of the partial clone submodule support as well? Do I need to change anything about my series? I guess since I'm asking that, I should probably submit it first so you can actually see it and answer my question. (And the timing may be good since the area is fresh in your memory...)

Thanks for raising this. Looking at the 3 you listed, they all have to do with prefetching. This is fine both now and later. Now, since partial clones in submodules still don't work, any fetching of any sort (pre- or not) will not work. Later, since this prefetching is just an optimization. (Of course, we should come back and add prefetching for submodule partial clones, but that is an optimization, not a correctness issue.)

Quoted 16 lines

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1581

Quoted 7 lines
object-file.c +0 -3
                    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
                        /*
                         * TODO Investigate checking promisor_remote_get_direct()
                         * TODO return value and stopping on error here.
-                        * TODO Pass a repository struct through
-                        * promisor_remote_get_direct(), such that arbitrary
-                        * repositories work.

Odd, it appears that when this comment was added (in commit b14ed5adaf ("Use promisor_remote_get_direct() and has_promisor_remote()", 2019-06-25)), a repository was passed to promisor_remote_get_direct(). Sure, it was just a transliteration of the comment that was there before when fetch_objects() was the function being called, but since the code was being changed and the comment being updated, it seems the TODO should have been removed back then.

Quoted 1 line

Oh, well, good to update it now at least.

Yes - perhaps the comment was emphasizing the "such that arbitrary repositories work" part. But anyway, yes, it is now removed.

[snip]

Quoted 1 line

Looks good to me.

Thanks for taking a look.

#22 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Elijah Newren
Quoted 12 lines

Turns out that this test fails under GIT_TEST_DEFAULT_HASH=sha256; output:

error: wrong index v2 file size in /home/newren/floss/git/t/trash directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx
error: wrong index v2 file size in /home/newren/floss/git/t/trash directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx
fatal: couldn't find remote ref 74242c6e4a0d89f454d89d3496a1f7cb3f1f39f0
error: wrong index v2 file size in /home/newren/floss/git/t/trash directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx
error: wrong index v2 file size in /home/newren/floss/git/t/trash directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx
fatal: could not obtain object info

Thanks for noticing this. I'll take a look.

#20 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Elijah Newren

A quick update...

Quoted 3 lines · Elijah Newren
Quoted 1 line · Jonathan Tan

...

Quoted 21 lines

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1574

Quoted 12 lines
object-file.c +2 -2
index f233b440b2..ebf273e9e7 100644
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
                }
 
                /* Check if it is a missing object */
-               if (fetch_if_missing && has_promisor_remote() &&
-                   !already_retried && r == the_repository &&
+               if (fetch_if_missing && repo_has_promisor_remote(r) &&
+                   !already_retried &&

So here you removed the special check against the_repository while looking for promisor_remotes. There are other such special checks in the code; I also see:

diff.c: if (options->repo == the_repository && has_promisor_remote() &&
diffcore-break.c:       if (r == the_repository && has_promisor_remote()) {
diffcore-rename.c:      if (r == the_repository && has_promisor_remote()) {
Quoted 7 lines

and a series I'm planning to submit soon will add another to merge.ort.c.

Do these need to all be fixed as part of the partial clone submodule support as well? Do I need to change anything about my series? I guess since I'm asking that, I should probably submit it first so you can actually see it and answer my question. (And the timing may be good since the area is fresh in your memory...)

That other topic is now over here:
https://lore.kernel.org/git/[email protected]/T/#t

Reviewed-by on v1 1/4
#4 lore raw

Re: [PATCH 1/4] promisor-remote: read partialClone config here

In reply to Jonathan Tan
Quoted 8 lines · Jonathan Tan

Currently, the reading of config related to promisor remotes is done in two places: once in setup.c (which sets the global variable repository_format_partial_clone, to be read by the code in promisor-remote.c), and once in promisor-remote.c. This means that care must be taken to ensure that repository_format_partial_clone is set before any code in promisor-remote.c accesses it.

Quoted 5 lines

To simplify the code, move all such config reading to promisor-remote.c. By doing this, it will be easier to see when repository_format_partial_clone is written and, thus, to reason about the code. This will be especially helpful in a subsequent commit, which modifies this code.

Do we reliably call promisor-remote.c:promisor_remote_config()? It's called only during promisor_remote_init(), which happens if we call something like promisor_remote_get_direct(), and I guess we call that one unconditionally (e.g. there's no "if (partial_clone) promisor_remote_get_direct();" that I saw in a brief glance) then it's OK.

cache.h extern int repository_format_worktree_config; · line 1064

Quoted 5 lines
cache.h +0 -1
@@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;
 struct repository_format {
 	int version;
 	int precious_objects;
-	char *partial_clone; /* value of extensions.partialclone */

I also don't see that this repository_format.partial_clone value gets checked anywhere anyways - I only see where it's set and freed in a brief grep - so this seems fine to me.

I saw Taylor's comment about NULL-ness and other than that, this patch looks good to me.

#25 lore raw

v2 cover First steps towards partial clone submodules

In reply to Jonathan Tan

Thanks everyone for your reviews. I believe I've addressed all review comments, including the one from Elijah about the test failing with sha256 (which turns out to be because I didn't add a call to setup_git_directory(), which the other test helpers do).

Jonathan Tan (4):
promisor-remote: read partialClone config here
promisor-remote: support per-repository config
run-command: move envvar-resetting function
promisor-remote: teach lazy-fetch in any repo

 Makefile                      |   1 +
 cache.h                       |   1 -
 object-file.c                 |   7 +-
 promisor-remote.c             | 125 ++++++++++++++++++++--------------
 promisor-remote.h             |  28 +++++---
 repository.c                  |   6 ++
 repository.h                  |   4 ++
 run-command.c                 |  10 +++
 run-command.h                 |   9 +++
 setup.c                       |  10 ++-
 submodule.c                   |  14 +---
 t/helper/test-partial-clone.c |  43 ++++++++++++
 t/helper/test-tool.c          |   1 +
 t/helper/test-tool.h          |   1 +
 t/t0410-partial-clone.sh      |  23 +++++++
 15 files changed, 201 insertions(+), 82 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c

Range-diff against v1:
1: 4a7ad9ffeb ! 1: 07290cba86 promisor-remote: read partialClone config here

    @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char
      	const char *subkey;
      
     +	if (!strcmp(var, "extensions.partialclone")) {
    -+		repository_format_partial_clone = xstrdup(value);
    ++		/*
    ++		 * NULL value is handled in handle_extension_v0 in setup.c.
    ++		 */
    ++		if (value)
    ++			repository_format_partial_clone = xstrdup(value);
     +		return 0;
     +	}
     +
2:  d8f5fa9b9f ! 2:  c462927ff2 promisor-remote: support per-repository config
    @@ promisor-remote.c: static void promisor_remote_move_to_tail(struct promisor_remo
      	const char *name;
      	size_t namelen;
      	const char *subkey;
    - 
    - 	if (!strcmp(var, "extensions.partialclone")) {
    --		repository_format_partial_clone = xstrdup(value);
    -+		config->repository_format_partial_clone = xstrdup(value);
    +@@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data
    + 		 * NULL value is handled in handle_extension_v0 in setup.c.
    + 		 */
    + 		if (value)
    +-			repository_format_partial_clone = xstrdup(value);
    ++			config->repository_format_partial_clone = xstrdup(value);
      		return 0;
      	}
      
    @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char
      }
      
     -static void promisor_remote_clear(void)
    -+static void promisor_remote_clear(struct promisor_remote_config *config)
    ++void promisor_remote_clear(struct promisor_remote_config *config)
      {
     -	while (promisors) {
     -		struct promisor_remote *r = promisors;
     -		promisors = promisors->next;
    ++	FREE_AND_NULL(config->repository_format_partial_clone);
    ++
     +	while (config->promisors) {
     +		struct promisor_remote *r = config->promisors;
     +		config->promisors = config->promisors->next;
    @@ promisor-remote.h: struct promisor_remote {
     +	repo_promisor_remote_reinit(the_repository);
     +}
     +
    ++void promisor_remote_clear(struct promisor_remote_config *config);
    ++
     +struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);
     +static inline struct promisor_remote *promisor_remote_find(const char *remote_name)
     +{
    @@ promisor-remote.h: struct promisor_remote {
      /*
       * Fetches all requested objects from all promisor remotes, trying them one at
     
    + ## repository.c ##
    +@@
    + #include "lockfile.h"
    + #include "submodule-config.h"
    + #include "sparse-index.h"
    ++#include "promisor-remote.h"
    + 
    + /* The main repository */
    + static struct repository the_repo;
    +@@ repository.c: void repo_clear(struct repository *repo)
    + 		if (repo->index != &the_index)
    + 			FREE_AND_NULL(repo->index);
    + 	}
    ++
    ++	if (repo->promisor_remote_config) {
    ++		promisor_remote_clear(repo->promisor_remote_config);
    ++		FREE_AND_NULL(repo->promisor_remote_config);
    ++	}
    + }
    + 
    + int repo_read_index(struct repository *repo)
    +
      ## repository.h ##
     @@ repository.h: struct lock_file;
      struct pathspec;

3: c5307a9f02 ! 3: 9cbdf60981 run-command: move envvar-resetting function
@@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai
const char *tr2_category, const char *tr2_label);

+/**
-+ * Convenience function that adds entries to env_array that resets all -+ * repo-specific environment variables except for CONFIG_DATA_ENVIRONMENT. See -+ * local_repo_env in cache.h for more information.
++ * Convenience function which adds all GIT_* environment variables to env_array ++ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a ++ * subprocess, these entries cause the corresponding environment variables to ++ * be unset in the subprocess. See local_repo_env in cache.h for more ++ * information.
+ */
+void prepare_other_repo_env(struct strvec *env_array);
+
4: b70a00b9b0 ! 4: 5b41569ace promisor-remote: teach lazy-fetch in any repo

    @@ t/helper/test-partial-clone.c (new)
     +#include "repository.h"
     +#include "object-store.h"
     +
    ++/*
    ++ * Prints the size of the object corresponding to the given hash in a specific
    ++ * gitdir. This is similar to "git -C gitdir cat-file -s", except that this
    ++ * exercises the code that accesses the object of an arbitrary repository that
    ++ * is not the_repository. ("git -C gitdir" makes it so that the_repository is
    ++ * the one in gitdir.)
    ++ */
     +static void object_info(const char *gitdir, const char *oid_hex)
     +{
     +	struct repository r;
    @@ t/helper/test-partial-clone.c (new)
     +
     +int cmd__partial_clone(int argc, const char **argv)
     +{
    ++	setup_git_directory();
    ++
     +	if (argc < 4)
     +		die("too few arguments");
     +
    @@ t/t0410-partial-clone.sh: test_expect_success 'do not fetch when checking existe
     +test_expect_success 'lazy-fetch when accessing object not in the_repository' '
     +	rm -rf full partial.git &&
     +	test_create_repo full &&
    -+	printf 12345 >full/file.txt &&
    -+	git -C full add file.txt &&
    -+	git -C full commit -m "first commit" &&
    ++	test_commit -C full create-a-file file.txt &&
     +
     +	test_config -C full uploadpack.allowfilter 1 &&
     +	test_config -C full uploadpack.allowanysha1inwant 1 &&
     +	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
    -+	FILE_HASH=$(git hash-object --stdin <full/file.txt) &&
    ++	FILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&
     +
     +	# Sanity check that the file is missing
     +	git -C partial.git rev-list --objects --missing=print HEAD >out &&
     +	grep "[?]$FILE_HASH" out &&
     +
    -+	OUT=$(test-tool partial-clone object-info partial.git "$FILE_HASH") &&
    -+	test "$OUT" -eq 5 &&
    ++	git -C full cat-file -s "$FILE_HASH" >expect &&
    ++	test-tool partial-clone object-info partial.git "$FILE_HASH" >actual &&
    ++	test_cmp expect actual &&
     +
     +	# Sanity check that the file is now present
     +	git -C partial.git rev-list --objects --missing=print HEAD >out &&
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#35 lore raw

PATCH v2 2/4 promisor-remote: support per-repository config

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation.

---
 promisor-remote.c | 103 ++++++++++++++++++++++++++--------------------
 promisor-remote.h |  22 ++++++++--
 repository.c      |   6 +++
 repository.h      |   4 ++
 4 files changed, 87 insertions(+), 48 deletions(-)
promisor-remote.c +58 -45
index c0e5061dfe..e1e1f7e93a 100644
@@ -5,7 +5,11 @@
 #include "transport.h"
 #include "strvec.h"
 
-static char *repository_format_partial_clone;
+struct promisor_remote_config {
+	char *repository_format_partial_clone;
+	struct promisor_remote *promisors;
+	struct promisor_remote **promisors_tail;
+};
 
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
@@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name,
 	return finish_command(&child) ? -1 : 0;
 }
 
-static struct promisor_remote *promisors;
-static struct promisor_remote **promisors_tail = &promisors;
-
-static struct promisor_remote *promisor_remote_new(const char *remote_name)
+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,
+						   const char *remote_name)
 {
 	struct promisor_remote *r;
 
@@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)
 
 	FLEX_ALLOC_STR(r, name, remote_name);
 
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 
 	return r;
 }
 
-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,
+						      const char *remote_name,
 						      struct promisor_remote **previous)
 {
 	struct promisor_remote *r, *p;
 
-	for (p = NULL, r = promisors; r; p = r, r = r->next)
+	for (p = NULL, r = config->promisors; r; p = r, r = r->next)
 		if (!strcmp(r->name, remote_name)) {
 			if (previous)
 				*previous = p;
@@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
 	return NULL;
 }
 
-static void promisor_remote_move_to_tail(struct promisor_remote *r,
+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,
+					 struct promisor_remote *r,
 					 struct promisor_remote *previous)
 {
 	if (r->next == NULL)
@@ -82,14 +86,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,
 	if (previous)
 		previous->next = r->next;
 	else
-		promisors = r->next ? r->next : r;
+		config->promisors = r->next ? r->next : r;
 	r->next = NULL;
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 }
 
 static int promisor_remote_config(const char *var, const char *value, void *data)
 {
+	struct promisor_remote_config *config = data;
 	const char *name;
 	size_t namelen;
 	const char *subkey;
@@ -99,7 +104,7 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 		 * NULL value is handled in handle_extension_v0 in setup.c.
 		 */
 		if (value)
-			repository_format_partial_clone = xstrdup(value);
+			config->repository_format_partial_clone = xstrdup(value);
 		return 0;
 	}
 
@@ -114,8 +119,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 
 		remote_name = xmemdupz(name, namelen);
 
-		if (!promisor_remote_lookup(remote_name, NULL))
-			promisor_remote_new(remote_name);
+		if (!promisor_remote_lookup(config, remote_name, NULL))
+			promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 		return 0;
@@ -124,9 +129,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 		struct promisor_remote *r;
 		char *remote_name = xmemdupz(name, namelen);
 
-		r = promisor_remote_lookup(remote_name, NULL);
+		r = promisor_remote_lookup(config, remote_name, NULL);
 		if (!r)
-			r = promisor_remote_new(remote_name);
+			r = promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 
@@ -139,59 +144,65 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	return 0;
 }
 
-static int initialized;
-
-static void promisor_remote_init(void)
+static void promisor_remote_init(struct repository *r)
 {
-	if (initialized)
+	struct promisor_remote_config *config;
+
+	if (r->promisor_remote_config)
 		return;
-	initialized = 1;
+	config = r->promisor_remote_config =
+		xcalloc(sizeof(*r->promisor_remote_config), 1);
+	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, NULL);
+	git_config(promisor_remote_config, config);
 
-	if (repository_format_partial_clone) {
+	if (config->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
 
-		o = promisor_remote_lookup(repository_format_partial_clone,
+		o = promisor_remote_lookup(config,
+					   config->repository_format_partial_clone,
 					   &previous);
 		if (o)
-			promisor_remote_move_to_tail(o, previous);
+			promisor_remote_move_to_tail(config, o, previous);
 		else
-			promisor_remote_new(repository_format_partial_clone);
+			promisor_remote_new(config, config->repository_format_partial_clone);
 	}
 }
 
-static void promisor_remote_clear(void)
+void promisor_remote_clear(struct promisor_remote_config *config)
 {
-	while (promisors) {
-		struct promisor_remote *r = promisors;
-		promisors = promisors->next;
+	FREE_AND_NULL(config->repository_format_partial_clone);
+
+	while (config->promisors) {
+		struct promisor_remote *r = config->promisors;
+		config->promisors = config->promisors->next;
 		free(r);
 	}
 
-	promisors_tail = &promisors;
+	config->promisors_tail = &config->promisors;
 }
 
-void promisor_remote_reinit(void)
+void repo_promisor_remote_reinit(struct repository *r)
 {
-	initialized = 0;
-	promisor_remote_clear();
-	promisor_remote_init();
+	promisor_remote_clear(r->promisor_remote_config);
+	FREE_AND_NULL(r->promisor_remote_config);
+	promisor_remote_init(r);
 }
 
-struct promisor_remote *promisor_remote_find(const char *remote_name)
+struct promisor_remote *repo_promisor_remote_find(struct repository *r,
+						  const char *remote_name)
 {
-	promisor_remote_init();
+	promisor_remote_init(r);
 
 	if (!remote_name)
-		return promisors;
+		return r->promisor_remote_config->promisors;
 
-	return promisor_remote_lookup(remote_name, NULL);
+	return promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);
 }
 
-int has_promisor_remote(void)
+int repo_has_promisor_remote(struct repository *r)
 {
-	return !!promisor_remote_find(NULL);
+	return !!repo_promisor_remote_find(r, NULL);
 }
 
 static int remove_fetched_oids(struct repository *repo,
@@ -239,9 +250,11 @@ int promisor_remote_get_direct(struct repository *repo,
 	if (oid_nr == 0)
 		return 0;
 
-	promisor_remote_init();
+	promisor_remote_init(repo);
 
-	for (r = promisors; r; r = r->next) {
+	if (repo != the_repository)
+		BUG("only the_repository is supported for now");
+	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
 		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
promisor-remote.h +19 -3
index 687210ab87..edc45ab0f5 100644
@@ -17,9 +17,25 @@ struct promisor_remote {
 	const char name[FLEX_ARRAY];
 };
 
-void promisor_remote_reinit(void);
-struct promisor_remote *promisor_remote_find(const char *remote_name);
-int has_promisor_remote(void);
+void repo_promisor_remote_reinit(struct repository *r);
+static inline void promisor_remote_reinit(void)
+{
+	repo_promisor_remote_reinit(the_repository);
+}
+
+void promisor_remote_clear(struct promisor_remote_config *config);
+
+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);
+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)
+{
+	return repo_promisor_remote_find(the_repository, remote_name);
+}
+
+int repo_has_promisor_remote(struct repository *r);
+static inline int has_promisor_remote(void)
+{
+	return repo_has_promisor_remote(the_repository);
+}
 
 /*
  * Fetches all requested objects from all promisor remotes, trying them one at
repository.c +6 -0
index 448cd557d4..dca0a11ab6 100644
@@ -11,6 +11,7 @@
 #include "lockfile.h"
 #include "submodule-config.h"
 #include "sparse-index.h"
+#include "promisor-remote.h"
 
 /* The main repository */
 static struct repository the_repo;
@@ -258,6 +259,11 @@ void repo_clear(struct repository *repo)
 		if (repo->index != &the_index)
 			FREE_AND_NULL(repo->index);
 	}
+
+	if (repo->promisor_remote_config) {
+		promisor_remote_clear(repo->promisor_remote_config);
+		FREE_AND_NULL(repo->promisor_remote_config);
+	}
 }
 
 int repo_read_index(struct repository *repo)
repository.h +4 -0
index a45f7520fd..fc06c154e2 100644
@@ -10,6 +10,7 @@ struct lock_file;
 struct pathspec;
 struct raw_object_store;
 struct submodule_cache;
+struct promisor_remote_config;
 
 enum untracked_cache_setting {
 	UNTRACKED_CACHE_UNSET = -1,
@@ -139,6 +140,9 @@ struct repository {
 	/* True if commit-graph has been disabled within this process. */
 	int commit_graph_disabled;
 
+	/* Configurations related to promisor remotes. */
+	struct promisor_remote_config *promisor_remote_config;
+
 	/* Configurations */
 
 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#38 lore raw

PATCH v2 3/4 run-command: move envvar-resetting function

In reply to Jonathan Tan

There is a function that resets environment variables, used when invoking a sub-process in a submodule. The lazy-fetching code (used in partial clones) will need this function in a subsequent commit, so move it to a more central location.

---
 run-command.c | 10 ++++++++++
 run-command.h |  9 +++++++++
 submodule.c   | 14 ++------------
 3 files changed, 21 insertions(+), 12 deletions(-)
run-command.c +10 -0
index be6bc128cd..a6c458119c 100644
@@ -1892,3 +1892,13 @@ int run_auto_maintenance(int quiet)
 
 	return run_command(&maint);
 }
+
+void prepare_other_repo_env(struct strvec *env_array)
+{
+	const char * const *var;
+
+	for (var = local_repo_env; *var; var++) {
+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+			strvec_push(env_array, *var);
+	}
+}
run-command.h +9 -0
index d08414a92e..a1d9107f5b 100644
@@ -483,4 +483,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
 			       task_finished_fn, void *pp_cb,
 			       const char *tr2_category, const char *tr2_label);
 
+/**
+ * Convenience function which adds all GIT_* environment variables to env_array
+ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a
+ * subprocess, these entries cause the corresponding environment variables to
+ * be unset in the subprocess. See local_repo_env in cache.h for more
+ * information.
+ */
+void prepare_other_repo_env(struct strvec *env_array);
+
 #endif
submodule.c +2 -12
index 0b1d9c1dde..a30216db52 100644
@@ -484,26 +484,16 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *
 	strbuf_release(&sb);
 }
 
-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
-{
-	const char * const *var;
-
-	for (var = local_repo_env; *var; var++) {
-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
-			strvec_push(out, *var);
-	}
-}
-
 void prepare_submodule_repo_env(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
+	prepare_other_repo_env(out);
 	strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
 		     DEFAULT_GIT_DIR_ENVIRONMENT);
 }
 
 static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
+	prepare_other_repo_env(out);
 	strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
 }
 
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#26 lore raw

PATCH v2 1/4 promisor-remote: read partialClone config here

In reply to Jonathan Tan

Currently, the reading of config related to promisor remotes is done in two places: once in setup.c (which sets the global variable repository_format_partial_clone, to be read by the code in promisor-remote.c), and once in promisor-remote.c. This means that care must be taken to ensure that repository_format_partial_clone is set before any code in promisor-remote.c accesses it.

To simplify the code, move all such config reading to promisor-remote.c. By doing this, it will be easier to see when repository_format_partial_clone is written and, thus, to reason about the code. This will be especially helpful in a subsequent commit, which modifies this code.

---
 cache.h           |  1 -
 promisor-remote.c | 14 +++++++++-----
 promisor-remote.h |  6 ------
 setup.c           | 10 +++++++---
 4 files changed, 16 insertions(+), 15 deletions(-)
cache.h +0 -1
index ba04ff8bd3..dbdcec8601 100644
@@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;
 struct repository_format {
 	int version;
 	int precious_objects;
-	char *partial_clone; /* value of extensions.partialclone */
 	int worktree_config;
 	int is_bare;
 	int hash_algo;
promisor-remote.c +9 -5
index da3f2ca261..c0e5061dfe 100644
@@ -7,11 +7,6 @@
 
 static char *repository_format_partial_clone;
 
-void set_repository_format_partial_clone(char *partial_clone)
-{
-	repository_format_partial_clone = xstrdup_or_null(partial_clone);
-}
-
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
@@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	size_t namelen;
 	const char *subkey;
 
+	if (!strcmp(var, "extensions.partialclone")) {
+		/*
+		 * NULL value is handled in handle_extension_v0 in setup.c.
+		 */
+		if (value)
+			repository_format_partial_clone = xstrdup(value);
+		return 0;
+	}
+
 	if (parse_config_key(var, "remote", &name, &namelen, &subkey) < 0)
 		return 0;
 
promisor-remote.h +0 -6
index c7a14063c5..687210ab87 100644
@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,
 			       const struct object_id *oids,
 			       int oid_nr);
 
-/*
- * This should be used only once from setup.c to set the value we got
- * from the extensions.partialclone config option.
- */
-void set_repository_format_partial_clone(char *partial_clone);
-
 #endif /* PROMISOR_REMOTE_H */
setup.c +7 -3
index 59e2facd9d..d60b6bc554 100644
@@ -470,7 +470,13 @@ static enum extension_result handle_extension_v0(const char *var,
 		} else if (!strcmp(ext, "partialclone")) {
 			if (!value)
 				return config_error_nonbool(var);
-			data->partial_clone = xstrdup(value);
+			/*
+			 * This config variable will be read together with the
+			 * other relevant config variables in
+			 * promisor_remote_config() in promisor_remote.c, so we
+			 * do not need to read it here. Just report that this
+			 * extension is known.
+			 */
 			return EXTENSION_OK;
 		} else if (!strcmp(ext, "worktreeconfig")) {
 			data->worktree_config = git_config_bool(var, value);
@@ -566,7 +572,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_
 	}
 
 	repository_format_precious_objects = candidate->precious_objects;
-	set_repository_format_partial_clone(candidate->partial_clone);
 	repository_format_worktree_config = candidate->worktree_config;
 	string_list_clear(&candidate->unknown_extensions, 0);
 	string_list_clear(&candidate->v1_only_extensions, 0);
@@ -650,7 +655,6 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->unknown_extensions, 0);
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
-	free(format->partial_clone);
 	init_repository_format(format);
 }
 
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#43 lore raw

PATCH v2 4/4 promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

---
 Makefile                      |  1 +
 object-file.c                 |  7 ++----
 promisor-remote.c             | 14 ++++++++----
 t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c          |  1 +
 t/helper/test-tool.h          |  1 +
 t/t0410-partial-clone.sh      | 23 +++++++++++++++++++
 7 files changed, 80 insertions(+), 10 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c
Makefile +1 -0
index c3565fc0f8..f6653bcd5e 100644
@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o
 TEST_BUILTINS_OBJS += test-online-cpus.o
 TEST_BUILTINS_OBJS += test-parse-options.o
 TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
+TEST_BUILTINS_OBJS += test-partial-clone.o
 TEST_BUILTINS_OBJS += test-path-utils.o
 TEST_BUILTINS_OBJS += test-pcre2-config.o
 TEST_BUILTINS_OBJS += test-pkt-line.o
object-file.c +2 -5
index f233b440b2..ebf273e9e7 100644
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
 		}
 
 		/* Check if it is a missing object */
-		if (fetch_if_missing && has_promisor_remote() &&
-		    !already_retried && r == the_repository &&
+		if (fetch_if_missing && repo_has_promisor_remote(r) &&
+		    !already_retried &&
 		    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
 			/*
 			 * TODO Investigate checking promisor_remote_get_direct()
 			 * TODO return value and stopping on error here.
-			 * TODO Pass a repository struct through
-			 * promisor_remote_get_direct(), such that arbitrary
-			 * repositories work.
 			 */
 			promisor_remote_get_direct(r, real, 1);
 			already_retried = 1;
promisor-remote.c +9 -5
index e1e1f7e93a..1491374d65 100644
@@ -11,7 +11,8 @@ struct promisor_remote_config {
 	struct promisor_remote **promisors_tail;
 };
 
-static int fetch_objects(const char *remote_name,
+static int fetch_objects(struct repository *repo,
+			 const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
 {
@@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,
 
 	child.git_cmd = 1;
 	child.in = -1;
+	if (repo != the_repository) {
+		prepare_other_repo_env(&child.env_array);
+		strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT,
+			     repo->gitdir);
+	}
 	strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop",
 		     "fetch", remote_name, "--no-tags",
 		     "--no-write-fetch-head", "--recurse-submodules=no",
@@ -154,7 +160,7 @@ static void promisor_remote_init(struct repository *r)
 		xcalloc(sizeof(*r->promisor_remote_config), 1);
 	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, config);
+	repo_config(r, promisor_remote_config, config);
 
 	if (config->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
@@ -252,10 +258,8 @@ int promisor_remote_get_direct(struct repository *repo,
 
 	promisor_remote_init(repo);
 
-	if (repo != the_repository)
-		BUG("only the_repository is supported for now");
 	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
-		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
+		if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
 			remaining_nr = remove_fetched_oids(repo, &remaining_oids,
t/helper/test-partial-clone.c new file +43 -0

43 lines not on this page · raw message

t/helper/test-tool.c +1 -0
index c5bd0c6d4c..b21e8f1519 100644
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
 	{ "online-cpus", cmd__online_cpus },
 	{ "parse-options", cmd__parse_options },
 	{ "parse-pathspec-file", cmd__parse_pathspec_file },
+	{ "partial-clone", cmd__partial_clone },
 	{ "path-utils", cmd__path_utils },
 	{ "pcre2-config", cmd__pcre2_config },
 	{ "pkt-line", cmd__pkt_line },
t/helper/test-tool.h +1 -0
index e8069a3b22..f845ced4b3 100644
@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
+int cmd__partial_clone(int argc, const char **argv);
 int cmd__path_utils(int argc, const char **argv);
 int cmd__pcre2_config(int argc, const char **argv);
 int cmd__pkt_line(int argc, const char **argv);
t/t0410-partial-clone.sh +23 -0
index 584a039b85..a211a66c67 100755
@@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
 	git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+	rm -rf full partial.git &&
+	test_create_repo full &&
+	test_commit -C full create-a-file file.txt &&
+
+	test_config -C full uploadpack.allowfilter 1 &&
+	test_config -C full uploadpack.allowanysha1inwant 1 &&
+	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+	FILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&
+
+	# Sanity check that the file is missing
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	grep "[?]$FILE_HASH" out &&
+
+	git -C full cat-file -s "$FILE_HASH" >expect &&
+	test-tool partial-clone object-info partial.git "$FILE_HASH" >actual &&
+	test_cmp expect actual &&
+
+	# Sanity check that the file is now present
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	! grep "[?]$FILE_HASH" out
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#10 lore raw

Re: [PATCH 2/4] promisor-remote: support per-repository config

In reply to Jonathan Tan
Quoted 5 lines · Jonathan Tan

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

Nice!

Quoted 14 lines
promisor-remote.c +5 -1
index bfe8eee5f2..5819d2cf28 100644
@@ -5,7 +5,11 @@
 #include "transport.h"
 #include "strvec.h"
 
-static char *repository_format_partial_clone;
+struct promisor_remote_config {
+	char *repository_format_partial_clone;
+	struct promisor_remote *promisors;
+	struct promisor_remote **promisors_tail;
+};

[snip]

promisor-remote.c static int fetch_objects(const char *remote_name, · line 41

Quoted 2 lines
promisor-remote.c +0 -2
-static struct promisor_remote *promisors;
-static struct promisor_remote **promisors_tail = &promisors;

Nice, so all the old globals are contained in a data struct now...

promisor-remote.c static int fetch_objects(const char *remote_name, · line 45

Quoted 3 lines
promisor-remote.c +2 -1
-static struct promisor_remote *promisor_remote_new(const char *remote_name)
+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,
+						   const char *remote_name)

...which we use during promisor_remote creation now.

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 138

Quoted 5 lines
promisor-remote.c +0 -1
@@ -135,59 +140,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	return 0;
 }
 
-static int initialized;

Very happy to see this pattern vanquished. ;)

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 144

Quoted 9 lines
promisor-remote.c +4 -3
-static void promisor_remote_init(void)
+static void promisor_remote_init(struct repository *r)
 {
-	if (initialized)
+	struct promisor_remote_config *config;
+
+	if (r->promisor_remote_config)
 		return;
-	initialized = 1;

So it's not a "call once, set global state" anymore, but we are also careful not to trample any existing state on the 'struct repository'. Nice.

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 150

Quoted 2 lines
promisor-remote.c +2 -0
+	config = r->promisor_remote_config =
+		xcalloc(sizeof(*r->promisor_remote_config), 1);

Hm. Am I the only one who doesn't like assigning from the result of an assignment like this? ...Based on 'git grep "=[^=]\+=[^=]"' yes, I am the only one :)

Overall this patch looks OK to me, but I see that there are some changes suggested for the next version, so I'll hold off on a reviewed-by so I can have a look at those too.

- Emily

Reviewed-by on v1 3/4
#14 lore raw

Re: [PATCH 3/4] run-command: move envvar-resetting function

In reply to Jonathan Tan
Quoted 17 lines · Jonathan Tan

There is a function that resets environment variables, used when invoking a sub-process in a submodule. The lazy-fetching code (used in partial clones) will need this function in a subsequent commit, so move it to a more central location.

---
 run-command.c | 10 ++++++++++
 run-command.h |  7 +++++++
 submodule.c   | 14 ++------------
 3 files changed, 19 insertions(+), 12 deletions(-)
run-command.h +0 -0
index d08414a92e..6f61ec7703 100644

run-command.h int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn, · line 491

Quoted 9 lines
run-command.h +6 -0
@@ -483,4 +483,11 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
 			       task_finished_fn, void *pp_cb,
 			       const char *tr2_category, const char *tr2_label);
 
+/**
+ * Convenience function that adds entries to env_array that resets all
+ * repo-specific environment variables except for CONFIG_DATA_ENVIRONMENT. See
+ * local_repo_env in cache.h for more information.
+ */
+void prepare_other_repo_env(struct strvec *env_array);

I like the name change on the function as well.

Quoted 4 lines
run-command.h +1 -1
 void prepare_submodule_repo_env(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
+	prepare_other_repo_env(out);

run-command.h int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,

Quoted 10 lines
run-command.h +1 -1
 	strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
 		     DEFAULT_GIT_DIR_ENVIRONMENT);
 }
 
 static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
+	prepare_other_repo_env(out);
 	strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
 }

This call was used in less places than I thought (I guess that's part of why you're making it more public/central), so my worry about having some large scale change was for nothing.

As for Taylor's comment about the CONFIG_DATA_ENVIRONMENT variable, it was named like that before you got here, so I am not too worried whether or not you change it.

#23 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan
Quoted 40 lines · Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

---
 Makefile                      |  1 +
 object-file.c                 |  7 ++-----
 promisor-remote.c             | 14 +++++++++-----
 t/helper/test-partial-clone.c | 34 ++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c          |  1 +
 t/helper/test-tool.h          |  1 +
 t/t0410-partial-clone.sh      | 24 ++++++++++++++++++++++++
 7 files changed, 72 insertions(+), 10 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c
Makefile +1 -0
index c3565fc0f8..f6653bcd5e 100644
@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o
 TEST_BUILTINS_OBJS += test-online-cpus.o
 TEST_BUILTINS_OBJS += test-parse-options.o
 TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
+TEST_BUILTINS_OBJS += test-partial-clone.o
 TEST_BUILTINS_OBJS += test-path-utils.o
 TEST_BUILTINS_OBJS += test-pcre2-config.o
 TEST_BUILTINS_OBJS += test-pkt-line.o
object-file.c +0 -0
index f233b440b2..ebf273e9e7 100644

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1574

Quoted 7 lines
object-file.c +2 -2
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
 		}
 
 		/* Check if it is a missing object */
-		if (fetch_if_missing && has_promisor_remote() &&
-		    !already_retried && r == the_repository &&
+		if (fetch_if_missing && repo_has_promisor_remote(r) &&
+		    !already_retried &&

So we remove the explicit "if we've got promisors and are operating on the repo we launched in" and instead ask "if the repo we're operating on has promisors" - definitely a step towards in-process submodule happiness :)

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1580

Quoted 9 lines
object-file.c +0 -3
 		    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
 			/*
 			 * TODO Investigate checking promisor_remote_get_direct()
 			 * TODO return value and stopping on error here.
-			 * TODO Pass a repository struct through
-			 * promisor_remote_get_direct(), such that arbitrary
-			 * repositories work.
 			 */
 			promisor_remote_get_direct(r, real, 1);

And this seems like a stale comment, since I see we were already passing 'r' here. But arbitrary repositories still don't just work, right? Or, I guess your point was "partial clone + submodules don't just work, because of the alternates thing" - so maybe this part is OK?

object-file.c static void promisor_remote_init(struct repository *r) · line 159

Quoted 6 lines
object-file.c +1 -1
@@ -150,7 +156,7 @@ static void promisor_remote_init(struct repository *r)
 		xcalloc(sizeof(*r->promisor_remote_config), 1);
 	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, config);
+	repo_config(r, promisor_remote_config, config);

Should this change have happened when we added 'r' to promisor_remote_init? If r==the_repository then there's no difference between these two calls, right?

Quoted 69 lines
t/helper/test-partial-clone.c new file +34 -0
new file mode 100644
index 0000000000..e7bc7eb21f
@@ -0,0 +1,34 @@
+#include "cache.h"
+#include "test-tool.h"
+#include "repository.h"
+#include "object-store.h"
+
+static void object_info(const char *gitdir, const char *oid_hex)
+{
+	struct repository r;
+	struct object_id oid;
+	unsigned long size;
+	struct object_info oi = {.sizep = &size};
+	const char *p;
+
+	if (repo_init(&r, gitdir, NULL))
+		die("could not init repo");
+	if (parse_oid_hex(oid_hex, &oid, &p))
+		die("could not parse oid");
+	if (oid_object_info_extended(&r, &oid, &oi, 0))
+		die("could not obtain object info");
+	printf("%d\n", (int) size);
+}
+
+int cmd__partial_clone(int argc, const char **argv)
+{
+	if (argc < 4)
+		die("too few arguments");
+
+	if (!strcmp(argv[1], "object-info"))
+		object_info(argv[2], argv[3]);
+	else
+		die("invalid argument '%s'", argv[1]);
+
+	return 0;
+}
t/helper/test-tool.c +1 -0
index c5bd0c6d4c..b21e8f1519 100644
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
 	{ "online-cpus", cmd__online_cpus },
 	{ "parse-options", cmd__parse_options },
 	{ "parse-pathspec-file", cmd__parse_pathspec_file },
+	{ "partial-clone", cmd__partial_clone },
 	{ "path-utils", cmd__path_utils },
 	{ "pcre2-config", cmd__pcre2_config },
 	{ "pkt-line", cmd__pkt_line },
t/helper/test-tool.h +1 -0
index e8069a3b22..f845ced4b3 100644
@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
+int cmd__partial_clone(int argc, const char **argv);
 int cmd__path_utils(int argc, const char **argv);
 int cmd__pcre2_config(int argc, const char **argv);
 int cmd__pkt_line(int argc, const char **argv);
t/t0410-partial-clone.sh +0 -0
index 584a039b85..e804d267e6 100755

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 612

Quoted 9 lines
t/t0410-partial-clone.sh +6 -0
@@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
 	git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+	rm -rf full partial.git &&
+	test_create_repo full &&
+	printf 12345 >full/file.txt &&
+	git -C full add file.txt &&
+	git -C full commit -m "first commit" &&

I think there is some test_commit or similar function here that's more commonly used, right?

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 615

Quoted 3 lines
t/t0410-partial-clone.sh +3 -0
+
+	test_config -C full uploadpack.allowfilter 1 &&
+	test_config -C full uploadpack.allowanysha1inwant 1 &&

I wasn't sure what these configs are for, but it looks like .allowfilter is to allow 'full' to serve as a remote to a partial clone. But what do you need .allowAnySha1InWant for here? Are we expecting to ask for SHAs that 'full' doesn't have?

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 624

Quoted 9 lines
t/t0410-partial-clone.sh +9 -0
+	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+	FILE_HASH=$(git hash-object --stdin <full/file.txt) &&
+
+	# Sanity check that the file is missing
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	grep "[?]$FILE_HASH" out &&
+
+	OUT=$(test-tool partial-clone object-info partial.git "$FILE_HASH") &&
+	test "$OUT" -eq 5 &&

Hm. I guess I am confused about why this fetches the desired object into partial.git. Maybe the test-helper needs a comment (and maybe here too) on the line where fetch will be triggered?

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 628

Quoted 4 lines
t/t0410-partial-clone.sh +4 -0
+
+	# Sanity check that the file is now present
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	! grep "[?]$FILE_HASH" out

- Emily

#56 lore raw
In reply to Jonathan Tan
Quoted 8 lines · Jonathan Tan

This is a preliminary step towards supporting partial clone submodules (e.g., by cloning with --recurse-submodules and having the given filter propagate to submodules). Even with this patch set, we won't be there yet (notably, some code in Git access objects in submodules by adding them as alternates - so lazy-fetching missing objects in submodules wouldn't work here), but at least this is a first step.

Quoted 8 lines

This patch set would also be useful if Git needed to operate on other repositories (other than in the submodule case), but I can't think of such a situation right now.

As mentioned, there is still more work that needs to be done. Any help is appreciated, and as for me, I hope to get back to this in the 3rd quarter of the year.

I see there's a v2 that came while I was still reviewing, oops. But overall I like this series:

- It's small
- It does reasonable code cleanup which benefits the codebase on its own
- It paves the way for a series later on without being part of that series, meaning that the later series will be slightly smaller because of it (a lesson I should learn for myself)

Thanks. I'll try and review v2 later in the week.

- Emily

#27 lore raw

Re: [PATCH v2 1/4] promisor-remote: read partialClone config here

In reply to Jonathan Tan
Quoted 6 lines · Jonathan Tan

Currently, the reading of config related to promisor remotes is done in two places: once in setup.c (which sets the global variable repository_format_partial_clone, to be read by the code in promisor-remote.c), and once in promisor-remote.c. This means that care must be taken to ensure that repository_format_partial_clone is set before any code in promisor-remote.c accesses it.

The above is very true, but I am puzzled by the chosen direction of the code movement.

Given that the value in the field repository_format.partial_clone comes from an extension, and an extension that is not understood by the version of Git that is running MUST abort the execution of Git, wouldn't it be guaranteed that, in a correctly written program, the .partial_clone field must already be set up correctly before anything else, including those in promissor-remote.c, accesses it?

Quoted 5 lines

To simplify the code, move all such config reading to promisor-remote.c. By doing this, it will be easier to see when repository_format_partial_clone is written and, thus, to reason about the code. This will be especially helpful in a subsequent commit, which modifies this code.

So, I am not sure if this simplifies the code the way we want to read our code. Doing a thing in one place is indeed simpler than doing it in two places, but it looks like promisor-remote code should be using the repository-format data more, not the other way around, at least to me.

Perhaps I am missing some other motivation, though.

Thanks.

Quoted 102 lines
---
 cache.h           |  1 -
 promisor-remote.c | 14 +++++++++-----
 promisor-remote.h |  6 ------
 setup.c           | 10 +++++++---
 4 files changed, 16 insertions(+), 15 deletions(-)
cache.h +0 -1
index ba04ff8bd3..dbdcec8601 100644
@@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;
 struct repository_format {
 	int version;
 	int precious_objects;
-	char *partial_clone; /* value of extensions.partialclone */
 	int worktree_config;
 	int is_bare;
 	int hash_algo;
promisor-remote.c +9 -5
index da3f2ca261..c0e5061dfe 100644
@@ -7,11 +7,6 @@
 
 static char *repository_format_partial_clone;
 
-void set_repository_format_partial_clone(char *partial_clone)
-{
-	repository_format_partial_clone = xstrdup_or_null(partial_clone);
-}
-
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
@@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	size_t namelen;
 	const char *subkey;
 
+	if (!strcmp(var, "extensions.partialclone")) {
+		/*
+		 * NULL value is handled in handle_extension_v0 in setup.c.
+		 */
+		if (value)
+			repository_format_partial_clone = xstrdup(value);
+		return 0;
+	}
+
 	if (parse_config_key(var, "remote", &name, &namelen, &subkey) < 0)
 		return 0;
 
promisor-remote.h +0 -6
index c7a14063c5..687210ab87 100644
@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,
 			       const struct object_id *oids,
 			       int oid_nr);
 
-/*
- * This should be used only once from setup.c to set the value we got
- * from the extensions.partialclone config option.
- */
-void set_repository_format_partial_clone(char *partial_clone);
-
 #endif /* PROMISOR_REMOTE_H */
setup.c +7 -3
index 59e2facd9d..d60b6bc554 100644
@@ -470,7 +470,13 @@ static enum extension_result handle_extension_v0(const char *var,
 		} else if (!strcmp(ext, "partialclone")) {
 			if (!value)
 				return config_error_nonbool(var);
-			data->partial_clone = xstrdup(value);
+			/*
+			 * This config variable will be read together with the
+			 * other relevant config variables in
+			 * promisor_remote_config() in promisor_remote.c, so we
+			 * do not need to read it here. Just report that this
+			 * extension is known.
+			 */
 			return EXTENSION_OK;
 		} else if (!strcmp(ext, "worktreeconfig")) {
 			data->worktree_config = git_config_bool(var, value);
@@ -566,7 +572,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_
 	}
 
 	repository_format_precious_objects = candidate->precious_objects;
-	set_repository_format_partial_clone(candidate->partial_clone);
 	repository_format_worktree_config = candidate->worktree_config;
 	string_list_clear(&candidate->unknown_extensions, 0);
 	string_list_clear(&candidate->v1_only_extensions, 0);
@@ -650,7 +655,6 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->unknown_extensions, 0);
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
-	free(format->partial_clone);
 	init_repository_format(format);
 }
#36 lore raw

Re: [PATCH v2 2/4] promisor-remote: support per-repository config

In reply to Jonathan Tan
Quoted 4 lines · Jonathan Tan

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

This does make sense. In general, repository extensions are per repository, so anything read from "extensions.*" should be stored per in-core repository structure.

But doesn't that mean the thing that should be fixed is on the setup.c side, where not just extensions.partialClone but other data is read into "struct repository_format *format"? Shouldn't we have a pointer to that struct in the in-core repository object?

Special casing the "partialClone" field alone feels somewhat strange to me.

Thanks.

Quoted 312 lines

The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation.

---
 promisor-remote.c | 103 ++++++++++++++++++++++++++--------------------
 promisor-remote.h |  22 ++++++++--
 repository.c      |   6 +++
 repository.h      |   4 ++
 4 files changed, 87 insertions(+), 48 deletions(-)
promisor-remote.c +58 -45
index c0e5061dfe..e1e1f7e93a 100644
@@ -5,7 +5,11 @@
 #include "transport.h"
 #include "strvec.h"
 
-static char *repository_format_partial_clone;
+struct promisor_remote_config {
+	char *repository_format_partial_clone;
+	struct promisor_remote *promisors;
+	struct promisor_remote **promisors_tail;
+};
 
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
@@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name,
 	return finish_command(&child) ? -1 : 0;
 }
 
-static struct promisor_remote *promisors;
-static struct promisor_remote **promisors_tail = &promisors;
-
-static struct promisor_remote *promisor_remote_new(const char *remote_name)
+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,
+						   const char *remote_name)
 {
 	struct promisor_remote *r;
 
@@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)
 
 	FLEX_ALLOC_STR(r, name, remote_name);
 
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 
 	return r;
 }
 
-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,
+						      const char *remote_name,
 						      struct promisor_remote **previous)
 {
 	struct promisor_remote *r, *p;
 
-	for (p = NULL, r = promisors; r; p = r, r = r->next)
+	for (p = NULL, r = config->promisors; r; p = r, r = r->next)
 		if (!strcmp(r->name, remote_name)) {
 			if (previous)
 				*previous = p;
@@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
 	return NULL;
 }
 
-static void promisor_remote_move_to_tail(struct promisor_remote *r,
+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,
+					 struct promisor_remote *r,
 					 struct promisor_remote *previous)
 {
 	if (r->next == NULL)
@@ -82,14 +86,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,
 	if (previous)
 		previous->next = r->next;
 	else
-		promisors = r->next ? r->next : r;
+		config->promisors = r->next ? r->next : r;
 	r->next = NULL;
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 }
 
 static int promisor_remote_config(const char *var, const char *value, void *data)
 {
+	struct promisor_remote_config *config = data;
 	const char *name;
 	size_t namelen;
 	const char *subkey;
@@ -99,7 +104,7 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 		 * NULL value is handled in handle_extension_v0 in setup.c.
 		 */
 		if (value)
-			repository_format_partial_clone = xstrdup(value);
+			config->repository_format_partial_clone = xstrdup(value);
 		return 0;
 	}
 
@@ -114,8 +119,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 
 		remote_name = xmemdupz(name, namelen);
 
-		if (!promisor_remote_lookup(remote_name, NULL))
-			promisor_remote_new(remote_name);
+		if (!promisor_remote_lookup(config, remote_name, NULL))
+			promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 		return 0;
@@ -124,9 +129,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 		struct promisor_remote *r;
 		char *remote_name = xmemdupz(name, namelen);
 
-		r = promisor_remote_lookup(remote_name, NULL);
+		r = promisor_remote_lookup(config, remote_name, NULL);
 		if (!r)
-			r = promisor_remote_new(remote_name);
+			r = promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 
@@ -139,59 +144,65 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	return 0;
 }
 
-static int initialized;
-
-static void promisor_remote_init(void)
+static void promisor_remote_init(struct repository *r)
 {
-	if (initialized)
+	struct promisor_remote_config *config;
+
+	if (r->promisor_remote_config)
 		return;
-	initialized = 1;
+	config = r->promisor_remote_config =
+		xcalloc(sizeof(*r->promisor_remote_config), 1);
+	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, NULL);
+	git_config(promisor_remote_config, config);
 
-	if (repository_format_partial_clone) {
+	if (config->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
 
-		o = promisor_remote_lookup(repository_format_partial_clone,
+		o = promisor_remote_lookup(config,
+					   config->repository_format_partial_clone,
 					   &previous);
 		if (o)
-			promisor_remote_move_to_tail(o, previous);
+			promisor_remote_move_to_tail(config, o, previous);
 		else
-			promisor_remote_new(repository_format_partial_clone);
+			promisor_remote_new(config, config->repository_format_partial_clone);
 	}
 }
 
-static void promisor_remote_clear(void)
+void promisor_remote_clear(struct promisor_remote_config *config)
 {
-	while (promisors) {
-		struct promisor_remote *r = promisors;
-		promisors = promisors->next;
+	FREE_AND_NULL(config->repository_format_partial_clone);
+
+	while (config->promisors) {
+		struct promisor_remote *r = config->promisors;
+		config->promisors = config->promisors->next;
 		free(r);
 	}
 
-	promisors_tail = &promisors;
+	config->promisors_tail = &config->promisors;
 }
 
-void promisor_remote_reinit(void)
+void repo_promisor_remote_reinit(struct repository *r)
 {
-	initialized = 0;
-	promisor_remote_clear();
-	promisor_remote_init();
+	promisor_remote_clear(r->promisor_remote_config);
+	FREE_AND_NULL(r->promisor_remote_config);
+	promisor_remote_init(r);
 }
 
-struct promisor_remote *promisor_remote_find(const char *remote_name)
+struct promisor_remote *repo_promisor_remote_find(struct repository *r,
+						  const char *remote_name)
 {
-	promisor_remote_init();
+	promisor_remote_init(r);
 
 	if (!remote_name)
-		return promisors;
+		return r->promisor_remote_config->promisors;
 
-	return promisor_remote_lookup(remote_name, NULL);
+	return promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);
 }
 
-int has_promisor_remote(void)
+int repo_has_promisor_remote(struct repository *r)
 {
-	return !!promisor_remote_find(NULL);
+	return !!repo_promisor_remote_find(r, NULL);
 }
 
 static int remove_fetched_oids(struct repository *repo,
@@ -239,9 +250,11 @@ int promisor_remote_get_direct(struct repository *repo,
 	if (oid_nr == 0)
 		return 0;
 
-	promisor_remote_init();
+	promisor_remote_init(repo);
 
-	for (r = promisors; r; r = r->next) {
+	if (repo != the_repository)
+		BUG("only the_repository is supported for now");
+	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
 		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
promisor-remote.h +19 -3
index 687210ab87..edc45ab0f5 100644
@@ -17,9 +17,25 @@ struct promisor_remote {
 	const char name[FLEX_ARRAY];
 };
 
-void promisor_remote_reinit(void);
-struct promisor_remote *promisor_remote_find(const char *remote_name);
-int has_promisor_remote(void);
+void repo_promisor_remote_reinit(struct repository *r);
+static inline void promisor_remote_reinit(void)
+{
+	repo_promisor_remote_reinit(the_repository);
+}
+
+void promisor_remote_clear(struct promisor_remote_config *config);
+
+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);
+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)
+{
+	return repo_promisor_remote_find(the_repository, remote_name);
+}
+
+int repo_has_promisor_remote(struct repository *r);
+static inline int has_promisor_remote(void)
+{
+	return repo_has_promisor_remote(the_repository);
+}
 
 /*
  * Fetches all requested objects from all promisor remotes, trying them one at
repository.c +6 -0
index 448cd557d4..dca0a11ab6 100644
@@ -11,6 +11,7 @@
 #include "lockfile.h"
 #include "submodule-config.h"
 #include "sparse-index.h"
+#include "promisor-remote.h"
 
 /* The main repository */
 static struct repository the_repo;
@@ -258,6 +259,11 @@ void repo_clear(struct repository *repo)
 		if (repo->index != &the_index)
 			FREE_AND_NULL(repo->index);
 	}
+
+	if (repo->promisor_remote_config) {
+		promisor_remote_clear(repo->promisor_remote_config);
+		FREE_AND_NULL(repo->promisor_remote_config);
+	}
 }
 
 int repo_read_index(struct repository *repo)
repository.h +4 -0
index a45f7520fd..fc06c154e2 100644
@@ -10,6 +10,7 @@ struct lock_file;
 struct pathspec;
 struct raw_object_store;
 struct submodule_cache;
+struct promisor_remote_config;
 
 enum untracked_cache_setting {
 	UNTRACKED_CACHE_UNSET = -1,
@@ -139,6 +140,9 @@ struct repository {
 	/* True if commit-graph has been disabled within this process. */
 	int commit_graph_disabled;
 
+	/* Configurations related to promisor remotes. */
+	struct promisor_remote_config *promisor_remote_config;
+
 	/* Configurations */
 
 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
#39 lore raw

Re: [PATCH v2 3/4] run-command: move envvar-resetting function

In reply to Jonathan Tan
Quoted 20 lines · Jonathan Tan

There is a function that resets environment variables, used when invoking a sub-process in a submodule. The lazy-fetching code (used in partial clones) will need this function in a subsequent commit, so move it to a more central location.

---
 run-command.c | 10 ++++++++++
 run-command.h |  9 +++++++++
 submodule.c   | 14 ++------------
 3 files changed, 21 insertions(+), 12 deletions(-)
run-command.c +0 -0
index be6bc128cd..a6c458119c 100644
@@ -1892,3 +1892,13 @@ int run_auto_maintenance(int quiet)
 
 	return run_command(&maint);
 }

run-command.c int run_auto_maintenance(int quiet) · line 1904

Quoted 10 lines
run-command.c +10 -0
+
+void prepare_other_repo_env(struct strvec *env_array)
+{
+	const char * const *var;
+
+	for (var = local_repo_env; *var; var++) {
+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+			strvec_push(env_array, *var);
+	}
+}

It does make sense to move this to run-command.c from submodule.c and the function name is already suitable for being global. I however cannot help wondering if we should also pay attention to the GIT_CONFIG_KEY_$n and GIT_CONFIG_VALUE_$n pairs (which is not a new problem in this patch).

This helper may sit better next to prep_childenv(), instead of just saying "the location does not matter, just append randomly at the end", though.

Otherwise looking good.

Thanks.

Quoted 50 lines
run-command.h +9 -0
index d08414a92e..a1d9107f5b 100644
@@ -483,4 +483,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
 			       task_finished_fn, void *pp_cb,
 			       const char *tr2_category, const char *tr2_label);
 
+/**
+ * Convenience function which adds all GIT_* environment variables to env_array
+ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a
+ * subprocess, these entries cause the corresponding environment variables to
+ * be unset in the subprocess. See local_repo_env in cache.h for more
+ * information.
+ */
+void prepare_other_repo_env(struct strvec *env_array);
+
 #endif
submodule.c +2 -12
index 0b1d9c1dde..a30216db52 100644
@@ -484,26 +484,16 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *
 	strbuf_release(&sb);
 }
 
-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
-{
-	const char * const *var;
-
-	for (var = local_repo_env; *var; var++) {
-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
-			strvec_push(out, *var);
-	}
-}
-
 void prepare_submodule_repo_env(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
+	prepare_other_repo_env(out);
 	strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
 		     DEFAULT_GIT_DIR_ENVIRONMENT);
 }
 
 static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
+	prepare_other_repo_env(out);
 	strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
 }
#44 lore raw

Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1574

Quoted 5 lines · Jonathan Tan
object-file.c +2 -2
 		/* Check if it is a missing object */
-		if (fetch_if_missing && has_promisor_remote() &&
-		    !already_retried && r == the_repository &&
+		if (fetch_if_missing && repo_has_promisor_remote(r) &&
+		    !already_retried &&

Turning has_promisor_remote() into repo_has_promisor_remote(r) does make tons of sense. Is this part of the code ready to lose "'r' must be the_repository because has_promisor_remote() only works on the primary in-core repository" we had before?

promisor-remote.c static int fetch_objects(const char *remote_name, · line 29

Quoted 9 lines
promisor-remote.c +5 -0
@@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,
 
 	child.git_cmd = 1;
 	child.in = -1;
+	if (repo != the_repository) {
+		prepare_other_repo_env(&child.env_array);
+		strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT,
+			     repo->gitdir);
+	}

This is what prepare_submodule_repo_env_in_gitdir() does; it makes me wonder if it (i.e. set up environment for that other repository, including the GIT_DIR and possibly other per-repository environment variable override) should be the primary API callers would want, instead of a more limited prepare_other_repo_env() that does not even take 'repo' parameter. Doesn't it feel somewhat strange for a function that is supposed to help preparing a part of child process by filling appropriate environ[] array to be run in a repository that is different from ours (which is "other repo" part of its name) not to want to even know which repository the "other" repo is?

Quoted 8 lines
t/helper/test-partial-clone.c new file +2 -0
new file mode 100644
index 0000000000..3f102cfddd
@@ -0,0 +1,43 @@
+#include "cache.h"
+#include "test-tool.h"

t/helper/test-partial-clone.c line 12

Quoted 10 lines
t/helper/test-partial-clone.c new file +10 -0
+#include "repository.h"
+#include "object-store.h"
+
+/*
+ * Prints the size of the object corresponding to the given hash in a specific
+ * gitdir. This is similar to "git -C gitdir cat-file -s", except that this
+ * exercises the code that accesses the object of an arbitrary repository that
+ * is not the_repository. ("git -C gitdir" makes it so that the_repository is
+ * the one in gitdir.)
+ */

The reason why this only gives size is because it will eventually become unnecessary once the main code starts running things in a submodule repository properly (i.e. without doing the alternate odb thing), and a more elaborate check is not worth your engineering effort? Object type and object sizes are something that you can safely express in plain text, would be handy for testing, and would not require too much extra code, I'd imagine.

Quoted 5 lines
t/helper/test-partial-clone.c new file +5 -0
+static void object_info(const char *gitdir, const char *oid_hex)
+{
+	struct repository r;
+	struct object_id oid;
+	unsigned long size;

t/helper/test-partial-clone.c line 27

Quoted 10 lines
t/helper/test-partial-clone.c new file +10 -0
+	struct object_info oi = {.sizep = &size};
+	const char *p;
+
+	if (repo_init(&r, gitdir, NULL))
+		die("could not init repo");
+	if (parse_oid_hex(oid_hex, &oid, &p))
+		die("could not parse oid");
+	if (oid_object_info_extended(&r, &oid, &oi, 0))
+		die("could not obtain object info");
+	printf("%d\n", (int) size);

Mimicking what builtin/cat-file.c::cat_one_file() does, for example, and using

	printf("%"PRIuMAX"\n", (uintmax_t)size);

might be better (I was wondering if we can extract reusable helpers, but I do not think that is worth doing, if this is meant to be temporary stop-gap measure).

Thanks.

#31 lore raw

Re: [PATCH v2 1/4] promisor-remote: read partialClone config here

In reply to Jonathan Tan
Quoted 52 lines · Jonathan Tan

Currently, the reading of config related to promisor remotes is done in two places: once in setup.c (which sets the global variable repository_format_partial_clone, to be read by the code in promisor-remote.c), and once in promisor-remote.c. This means that care must be taken to ensure that repository_format_partial_clone is set before any code in promisor-remote.c accesses it.

To simplify the code, move all such config reading to promisor-remote.c. By doing this, it will be easier to see when repository_format_partial_clone is written and, thus, to reason about the code. This will be especially helpful in a subsequent commit, which modifies this code.

---
 cache.h           |  1 -
 promisor-remote.c | 14 +++++++++-----
 promisor-remote.h |  6 ------
 setup.c           | 10 +++++++---
 4 files changed, 16 insertions(+), 15 deletions(-)
cache.h +0 -1
index ba04ff8bd3..dbdcec8601 100644
@@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;
 struct repository_format {
        int version;
        int precious_objects;
-       char *partial_clone; /* value of extensions.partialclone */
        int worktree_config;
        int is_bare;
        int hash_algo;
promisor-remote.c +0 -5
index da3f2ca261..c0e5061dfe 100644
@@ -7,11 +7,6 @@
 
 static char *repository_format_partial_clone;
 
-void set_repository_format_partial_clone(char *partial_clone)
-{
-       repository_format_partial_clone = xstrdup_or_null(partial_clone);
-}
-
 static int fetch_objects(const char *remote_name,
                         const struct object_id *oids,
                         int oid_nr)
@@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data
        size_t namelen;

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 104

Quoted 10 lines
promisor-remote.c +8 -0
        const char *subkey;
 
+       if (!strcmp(var, "extensions.partialclone")) {
+               /*
+                * NULL value is handled in handle_extension_v0 in setup.c.
+                */
+               if (value)
+                       repository_format_partial_clone = xstrdup(value);
+               return 0;
+       }

This is actually slightly hard to parse out. I was trying to figure out where repository_format_partial_clone was initialized, and it's not handled when value is NULL in handle_extension_v0; it's the fact that repository_format_partial_clone is declared a static global variable.

But in the next patch you make it a member of struct promisor_remote_config, and instead rely on the xcalloc call in promisor_remote_init().

That means everything is properly initialized and you haven't made any mistakes here, but the logic is a bit hard to follow. Perhaps it'd be nicer to just write this as

+ if (!strcmp(var, "extensions.partialclone")) {
+ repository_format_partial_clone = xstrdup_or_null(value);
+ return 0;
+ }

which makes the code shorter and easier to follow, at least for me.

#49 lore raw

Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan
Quoted 10 lines · Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

I wonder if this commit message is a good place to call out that we also want to eventually audit codepaths using the old has_promisor_remote() wrapper function (particularly the ones protected by a repo == the_repository check) as well.

Quoted 204 lines
---
 Makefile                      |  1 +
 object-file.c                 |  7 ++----
 promisor-remote.c             | 14 ++++++++----
 t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c          |  1 +
 t/helper/test-tool.h          |  1 +
 t/t0410-partial-clone.sh      | 23 +++++++++++++++++++
 7 files changed, 80 insertions(+), 10 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c
Makefile +1 -0
index c3565fc0f8..f6653bcd5e 100644
@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o
 TEST_BUILTINS_OBJS += test-online-cpus.o
 TEST_BUILTINS_OBJS += test-parse-options.o
 TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
+TEST_BUILTINS_OBJS += test-partial-clone.o
 TEST_BUILTINS_OBJS += test-path-utils.o
 TEST_BUILTINS_OBJS += test-pcre2-config.o
 TEST_BUILTINS_OBJS += test-pkt-line.o
object-file.c +2 -5
index f233b440b2..ebf273e9e7 100644
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
                }
 
                /* Check if it is a missing object */
-               if (fetch_if_missing && has_promisor_remote() &&
-                   !already_retried && r == the_repository &&
+               if (fetch_if_missing && repo_has_promisor_remote(r) &&
+                   !already_retried &&
                    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
                        /*
                         * TODO Investigate checking promisor_remote_get_direct()
                         * TODO return value and stopping on error here.
-                        * TODO Pass a repository struct through
-                        * promisor_remote_get_direct(), such that arbitrary
-                        * repositories work.
                         */
                        promisor_remote_get_direct(r, real, 1);
                        already_retried = 1;
promisor-remote.c +9 -5
index e1e1f7e93a..1491374d65 100644
@@ -11,7 +11,8 @@ struct promisor_remote_config {
        struct promisor_remote **promisors_tail;
 };
 
-static int fetch_objects(const char *remote_name,
+static int fetch_objects(struct repository *repo,
+                        const char *remote_name,
                         const struct object_id *oids,
                         int oid_nr)
 {
@@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,
 
        child.git_cmd = 1;
        child.in = -1;
+       if (repo != the_repository) {
+               prepare_other_repo_env(&child.env_array);
+               strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT,
+                            repo->gitdir);
+       }
        strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop",
                     "fetch", remote_name, "--no-tags",
                     "--no-write-fetch-head", "--recurse-submodules=no",
@@ -154,7 +160,7 @@ static void promisor_remote_init(struct repository *r)
                xcalloc(sizeof(*r->promisor_remote_config), 1);
        config->promisors_tail = &config->promisors;
 
-       git_config(promisor_remote_config, config);
+       repo_config(r, promisor_remote_config, config);
 
        if (config->repository_format_partial_clone) {
                struct promisor_remote *o, *previous;
@@ -252,10 +258,8 @@ int promisor_remote_get_direct(struct repository *repo,
 
        promisor_remote_init(repo);
 
-       if (repo != the_repository)
-               BUG("only the_repository is supported for now");
        for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
-               if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
+               if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {
                        if (remaining_nr == 1)
                                continue;
                        remaining_nr = remove_fetched_oids(repo, &remaining_oids,
t/helper/test-partial-clone.c new file +43 -0
new file mode 100644
index 0000000000..3f102cfddd
@@ -0,0 +1,43 @@
+#include "cache.h"
+#include "test-tool.h"
+#include "repository.h"
+#include "object-store.h"
+
+/*
+ * Prints the size of the object corresponding to the given hash in a specific
+ * gitdir. This is similar to "git -C gitdir cat-file -s", except that this
+ * exercises the code that accesses the object of an arbitrary repository that
+ * is not the_repository. ("git -C gitdir" makes it so that the_repository is
+ * the one in gitdir.)
+ */
+static void object_info(const char *gitdir, const char *oid_hex)
+{
+       struct repository r;
+       struct object_id oid;
+       unsigned long size;
+       struct object_info oi = {.sizep = &size};
+       const char *p;
+
+       if (repo_init(&r, gitdir, NULL))
+               die("could not init repo");
+       if (parse_oid_hex(oid_hex, &oid, &p))
+               die("could not parse oid");
+       if (oid_object_info_extended(&r, &oid, &oi, 0))
+               die("could not obtain object info");
+       printf("%d\n", (int) size);
+}
+
+int cmd__partial_clone(int argc, const char **argv)
+{
+       setup_git_directory();
+
+       if (argc < 4)
+               die("too few arguments");
+
+       if (!strcmp(argv[1], "object-info"))
+               object_info(argv[2], argv[3]);
+       else
+               die("invalid argument '%s'", argv[1]);
+
+       return 0;
+}
t/helper/test-tool.c +1 -0
index c5bd0c6d4c..b21e8f1519 100644
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
        { "online-cpus", cmd__online_cpus },
        { "parse-options", cmd__parse_options },
        { "parse-pathspec-file", cmd__parse_pathspec_file },
+       { "partial-clone", cmd__partial_clone },
        { "path-utils", cmd__path_utils },
        { "pcre2-config", cmd__pcre2_config },
        { "pkt-line", cmd__pkt_line },
t/helper/test-tool.h +1 -0
index e8069a3b22..f845ced4b3 100644
@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
+int cmd__partial_clone(int argc, const char **argv);
 int cmd__path_utils(int argc, const char **argv);
 int cmd__pcre2_config(int argc, const char **argv);
 int cmd__pkt_line(int argc, const char **argv);
t/t0410-partial-clone.sh +23 -1
index 584a039b85..a211a66c67 100755
@@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
        git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+       rm -rf full partial.git &&
+       test_create_repo full &&
+       test_commit -C full create-a-file file.txt &&
+
+       test_config -C full uploadpack.allowfilter 1 &&
+       test_config -C full uploadpack.allowanysha1inwant 1 &&
+       git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+       FILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&
+
+       # Sanity check that the file is missing
+       git -C partial.git rev-list --objects --missing=print HEAD >out &&
+       grep "[?]$FILE_HASH" out &&
+
+       git -C full cat-file -s "$FILE_HASH" >expect &&
+       test-tool partial-clone object-info partial.git "$FILE_HASH" >actual &&
+       test_cmp expect actual &&
+
+       # Sanity check that the file is now present
+       git -C partial.git rev-list --objects --missing=print HEAD >out &&
+       ! grep "[?]$FILE_HASH" out
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
--

2.32.0.rc1.229.g3e70b5a671-goog

#51 lore raw
In reply to Jonathan Tan
Quoted 5 lines · Jonathan Tan

Thanks everyone for your reviews. I believe I've addressed all review comments, including the one from Elijah about the test failing with sha256 (which turns out to be because I didn't add a call to setup_git_directory(), which the other test helpers do).

Thanks for fixing those up. I spotted some minor nits/questions, but nothing big.

Looks like Junio did spot some bigger items...which raises a question for me. I have a series
(https://lore.kernel.org/git/[email protected]/)
that also touches partial clones. Our series are semantically independent, but we both add a repository parameter to fetch_objects(). So we both make the same change, but you also make additional nearby changes, resulting in two trivial conflicts. So, should I rebase my series on yours, should you rebase on mine, or should we just let both proceed independently and double-check Junio resolves the trivial conflicts in favor of your side?

Thoughts?

#52 lore raw
Quoted 7 lines · Elijah Newren
Quoted 5 lines · Jonathan Tan

Thanks everyone for your reviews. I believe I've addressed all review comments, including the one from Elijah about the test failing with sha256 (which turns out to be because I didn't add a call to setup_git_directory(), which the other test helpers do).

Quoted 5 lines

Thanks for fixing those up. I spotted some minor nits/questions, but nothing big.

Looks like Junio did spot some bigger items...which raises a question for me. I have a series ...

Do you mean, by "bigger items", that we may want to turn it around to have repo extension data to the in-core repository structure?

Quoted 5 lines

(https://lore.kernel.org/git/[email protected]/)
that also touches partial clones. Our series are semantically independent, but we both add a repository parameter to fetch_objects(). So we both make the same change, but you also make additional nearby changes, resulting in two trivial conflicts. ...

I can sort of see how the above plan would work if we are not going to fix the "keep only the partialclone related extension thing, instead of solving the larger structural problem that the current arrangement ignores that repository extensions are per repository". But wouldn't that leave us with two series with technical debt? Also, if Jonathan's series fixes the "bigger item", would the above "proceed more or less independently or rebase one on top of the other" plan work well without making the same fix in yours?

I guess a better first step would be to stop, think and decide what to do with the "bigger" thing---if only to dismiss it with a firm declaration that we would never do such a fix and move extension data piecemeal to relevant subsystems, so that we'd reduce conflicts in the future, as I am reasonably sure that the "bigger item" will be tempting to fix even after the two series lands, and doing so at that time would be twice larger surgery.

#53 lore raw
Quoted 10 lines · Junio C Hamano
Quoted 7 lines · Elijah Newren
Quoted 5 lines · Jonathan Tan

Thanks everyone for your reviews. I believe I've addressed all review comments, including the one from Elijah about the test failing with sha256 (which turns out to be because I didn't add a call to setup_git_directory(), which the other test helpers do).

Quoted 8 lines
Quoted 5 lines

Thanks for fixing those up. I spotted some minor nits/questions, but nothing big.

Looks like Junio did spot some bigger items...which raises a question for me. I have a series ...

Do you mean, by "bigger items", that we may want to turn it around to have repo extension data to the in-core repository structure?

Yes.

Quoted 6 lines
Quoted 5 lines

(https://lore.kernel.org/git/[email protected]/)
that also touches partial clones. Our series are semantically independent, but we both add a repository parameter to fetch_objects(). So we both make the same change, but you also make additional nearby changes, resulting in two trivial conflicts. ...

Quoted 8 lines

I can sort of see how the above plan would work if we are not going to fix the "keep only the partialclone related extension thing, instead of solving the larger structural problem that the current arrangement ignores that repository extensions are per repository". But wouldn't that leave us with two series with technical debt? Also, if Jonathan's series fixes the "bigger item", would the above "proceed more or less independently or rebase one on top of the other" plan work well without making the same fix in yours?

My series is completely independent of the partialclone extension stuff.

My series merely adds the recording of a single statistic (number of fetched objects) to the partial clone stuff; everything else is higher level diffcore-rename and merge-ort stuff.

Quoted 7 lines

I guess a better first step would be to stop, think and decide what to do with the "bigger" thing---if only to dismiss it with a firm declaration that we would never do such a fix and move extension data piecemeal to relevant subsystems, so that we'd reduce conflicts in the future, as I am reasonably sure that the "bigger item" will be tempting to fix even after the two series lands, and doing so at that time would be twice larger surgery.

I don't understand how you think the partialclone extension stuff is relevant to my series at all. My changes to promisor-remote.c are just a couple lines, and if he expands or rearranges his work, the amount of conflicts can't really get any bigger because there's only a few lines on my side for it to conflict with.

#54 lore raw
Quoted 1 line · Elijah Newren

My series is completely independent of the partialclone extension stuff.

OK, that makes it simpler.

#28 lore raw

Re: [PATCH v2 1/4] promisor-remote: read partialClone config here

In reply to Junio C Hamano
Quoted 31 lines
Quoted 6 lines · Jonathan Tan

Currently, the reading of config related to promisor remotes is done in two places: once in setup.c (which sets the global variable repository_format_partial_clone, to be read by the code in promisor-remote.c), and once in promisor-remote.c. This means that care must be taken to ensure that repository_format_partial_clone is set before any code in promisor-remote.c accesses it.

The above is very true, but I am puzzled by the chosen direction of the code movement.

Given that the value in the field repository_format.partial_clone comes from an extension, and an extension that is not understood by the version of Git that is running MUST abort the execution of Git, wouldn't it be guaranteed that, in a correctly written program, the .partial_clone field must already be set up correctly before anything else, including those in promissor-remote.c, accesses it?

Quoted 5 lines

To simplify the code, move all such config reading to promisor-remote.c. By doing this, it will be easier to see when repository_format_partial_clone is written and, thus, to reason about the code. This will be especially helpful in a subsequent commit, which modifies this code.

So, I am not sure if this simplifies the code the way we want to read our code. Doing a thing in one place is indeed simpler than doing it in two places, but it looks like promisor-remote code should be using the repository-format data more, not the other way around, at least to me.

Quoted 3 lines

Perhaps I am missing some other motivation, though.

Thanks.

I'm reluctant to add more fields to struct repository_format. Right now, the way it is used is to hold any information we gathered (e.g. hash type) while determining if a repo is one that we can handle. Any information we still need is copied somewhere else, and the struct itself is immediately freed.

If we were to use it for promisor remote config, we would have to read config into struct repository_format's fields and copy those fields into struct repository in setup.c, and then access the same fields in promisor-remote.c. It seems more straightforward to just do everything in promisor-remote.c - for example, if we needed to change the type of one of those fields, we would just need to change it in one file instead of two.

I acknowledge that there still remains the duplication that setup.c needs to know that extensions.partialClone is a valid extension, and that promsior-remote.c needs to interpret extensions.partialClone.

Having said that, I don't feel very strongly about keeping everything in promisor-remote.c, so I can move it into setup.c if that's the reviewer consensus.

#37 lore raw

Re: [PATCH v2 2/4] promisor-remote: support per-repository config

In reply to Junio C Hamano
Quoted 16 lines
Quoted 4 lines · Jonathan Tan

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

This does make sense. In general, repository extensions are per repository, so anything read from "extensions.*" should be stored per in-core repository structure.

But doesn't that mean the thing that should be fixed is on the setup.c side, where not just extensions.partialClone but other data is read into "struct repository_format *format"? Shouldn't we have a pointer to that struct in the in-core repository object?

Quoted 4 lines

Special casing the "partialClone" field alone feels somewhat strange to me.

Thanks.

My reply is the same as what I replied to the query in patch 1 [1].

[1] https://lore.kernel.org/git/[email protected]/

#40 lore raw

Re: [PATCH v2 3/4] run-command: move envvar-resetting function

In reply to Junio C Hamano
Quoted 5 lines

It does make sense to move this to run-command.c from submodule.c and the function name is already suitable for being global. I however cannot help wondering if we should also pay attention to the GIT_CONFIG_KEY_$n and GIT_CONFIG_VALUE_$n pairs (which is not a new problem in this patch).

Note that I changed the function name (the previous one was too submodule-specific). As for the config pairs, they are currently being passed through - do you have a situation in mind in which they should not be passed through?

Quoted 7 lines

This helper may sit better next to prep_childenv(), instead of just saying "the location does not matter, just append randomly at the end", though.

Otherwise looking good.

Thanks.

Thanks - I'll move it.

#45 lore raw

Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Junio C Hamano
Quoted 12 lines

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1574

Quoted 5 lines · Jonathan Tan
object-file.c +2 -2
 		/* Check if it is a missing object */
-		if (fetch_if_missing && has_promisor_remote() &&
-		    !already_retried && r == the_repository &&
+		if (fetch_if_missing && repo_has_promisor_remote(r) &&
+		    !already_retried &&

Turning has_promisor_remote() into repo_has_promisor_remote(r) does make tons of sense. Is this part of the code ready to lose "'r' must be the_repository because has_promisor_remote() only works on the primary in-core repository" we had before?

Yes - that is precisely what I test with the test helper (running this code path with "r" that is not the_repository).

Quoted 12 lines

promisor-remote.c static int fetch_objects(const char *remote_name, · line 29

Quoted 9 lines
promisor-remote.c +5 -0
@@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,
 
 	child.git_cmd = 1;
 	child.in = -1;
+	if (repo != the_repository) {
+		prepare_other_repo_env(&child.env_array);
+		strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT,
+			     repo->gitdir);
+	}

This is what prepare_submodule_repo_env_in_gitdir() does; it makes me wonder if it (i.e. set up environment for that other repository,

Quoted 8 lines

including the GIT_DIR and possibly other per-repository environment variable override) should be the primary API callers would want, instead of a more limited prepare_other_repo_env() that does not even take 'repo' parameter. Doesn't it feel somewhat strange for a function that is supposed to help preparing a part of child process by filling appropriate environ[] array to be run in a repository that is different from ours (which is "other repo" part of its name) not to want to even know which repository the "other" repo is?

Good point. I'll update prepare_other_repo_env() to have a gitdir parameter.

Quoted 15 lines
Quoted 15 lines
t/helper/test-partial-clone.c new file +9 -0
new file mode 100644
index 0000000000..3f102cfddd
@@ -0,0 +1,43 @@
+#include "cache.h"
+#include "test-tool.h"
+#include "repository.h"
+#include "object-store.h"
+
+/*
+ * Prints the size of the object corresponding to the given hash in a specific
+ * gitdir. This is similar to "git -C gitdir cat-file -s", except that this
+ * exercises the code that accesses the object of an arbitrary repository that
Quoted 8 lines

t/helper/test-partial-clone.c line 12

Quoted 3 lines
t/helper/test-partial-clone.c new file +3 -0
+ * is not the_repository. ("git -C gitdir" makes it so that the_repository is
+ * the one in gitdir.)
+ */

The reason why this only gives size is because it will eventually become unnecessary once the main code starts running things in a submodule repository properly (i.e. without doing the alternate odb thing),

If you mean that this code path can be tested through user-visible commands (e.g. git grep with submodule recursion) once the main code starts avoiding doing the alternate odb thing, so this helper will eventually be unnecessary, then the answer is yes.

Quoted 2 lines

and a more elaborate check is not worth your engineering effort?

Even now, I don't think a more elaborate check is worth the engineering effort - I just want to check that the file indeed was lazy-fetched, so any minor datum would suffice.

Quoted 3 lines

Object type and object sizes are something that you can safely express in plain text, would be handy for testing, and would not require too much extra code, I'd imagine.

It would, but we can already use "git cat-file -s" (or -t) for that. The helper is meant to test a specific code path wherein we access a submodule object during a process running in the superproject.

Quoted 5 lines

t/helper/test-partial-clone.c line 27

Quoted 1 line
t/helper/test-partial-clone.c new file +1 -0
+	printf("%d\n", (int) size);

Mimicking what builtin/cat-file.c::cat_one_file() does, for example, and using

Quoted 7 lines
	printf("%"PRIuMAX"\n", (uintmax_t)size);

might be better (I was wondering if we can extract reusable helpers, but I do not think that is worth doing, if this is meant to be temporary stop-gap measure).

Thanks.

Sounds good - I'll change this.

#32 lore raw

Re: [PATCH v2 1/4] promisor-remote: read partialClone config here

In reply to Elijah Newren
Quoted 32 lines

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 104

Quoted 12 lines
promisor-remote.c +8 -0
@@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data
        size_t namelen;
        const char *subkey;
 
+       if (!strcmp(var, "extensions.partialclone")) {
+               /*
+                * NULL value is handled in handle_extension_v0 in setup.c.
+                */
+               if (value)
+                       repository_format_partial_clone = xstrdup(value);
+               return 0;
+       }

This is actually slightly hard to parse out. I was trying to figure out where repository_format_partial_clone was initialized, and it's not handled when value is NULL in handle_extension_v0; it's the fact that repository_format_partial_clone is declared a static global variable.

But in the next patch you make it a member of struct promisor_remote_config, and instead rely on the xcalloc call in promisor_remote_init().

That means everything is properly initialized and you haven't made any mistakes here, but the logic is a bit hard to follow. Perhaps it'd be nicer to just write this as

promisor-remote.c +4 -0
+       if (!strcmp(var, "extensions.partialclone")) {
+               repository_format_partial_clone = xstrdup_or_null(value);
+               return 0;
+       }
Quoted 1 line

which makes the code shorter and easier to follow, at least for me.

Hmm...is your concern about the case in which repository_format_partial_clone is uninitialized, or about ignoring a potential NULL value? If the former, I don't see how your suggestion fixes things, since extensions.partialclone may never have been in the config in the first place (and would thus leave repository_format_partial_clone uninitialized, if it weren't for the fact that it is in static storage and thus initialized to 0). If the latter, I guess I should be more detailed about how it's being handled in setup.c (or maybe just leave out the comment altogether - the code here can handle a NULL repository_format_partial_clone for some reason).

#50 lore raw

Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Elijah Newren
Quoted 12 lines · Jonathan Tan
Quoted 10 lines

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

Quoted 4 lines

I wonder if this commit message is a good place to call out that we also want to eventually audit codepaths using the old has_promisor_remote() wrapper function (particularly the ones protected by a repo == the_repository check) as well.

Sounds good. I think we will need to check uses of all wrappers.

#24 lore raw

Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Emily Shaffer
Quoted 6 lines
Quoted 6 lines
object-file.c +0 -2
 		    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
 			/*
 			 * TODO Investigate checking promisor_remote_get_direct()
 			 * TODO return value and stopping on error here.
-			 * TODO Pass a repository struct through
-			 * promisor_remote_get_direct(), such that arbitrary
Quoted 8 lines

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1580

Quoted 3 lines
object-file.c +0 -1
-			 * repositories work.
 			 */
 			promisor_remote_get_direct(r, real, 1);

And this seems like a stale comment, since I see we were already passing 'r' here. But arbitrary repositories still don't just work, right? Or, I guess your point was "partial clone + submodules don't just work, because of the alternates thing" - so maybe this part is OK?

This part is OK (arbitrary repositories work here), yes.

Quoted 10 lines

promisor-remote.c static void promisor_remote_init(struct repository *r) · line 159

Quoted 6 lines
promisor-remote.c +1 -1
@@ -150,7 +156,7 @@ static void promisor_remote_init(struct repository *r)
 		xcalloc(sizeof(*r->promisor_remote_config), 1);
 	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, config);
+	repo_config(r, promisor_remote_config, config);

Should this change have happened when we added 'r' to promisor_remote_init? If r==the_repository then there's no difference between these two calls, right?

Good point - yes, it should have. I'll change it.

Quoted 8 lines

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 612

Quoted 6 lines
t/t0410-partial-clone.sh +6 -0
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+	rm -rf full partial.git &&
+	test_create_repo full &&
+	printf 12345 >full/file.txt &&
+	git -C full add file.txt &&
+	git -C full commit -m "first commit" &&

I think there is some test_commit or similar function here that's more commonly used, right?

Taylor Blau suggested a similar thing, and I have changed it in v2.

Quoted 8 lines

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 615

Quoted 4 lines
t/t0410-partial-clone.sh +3 -0
+
+	test_config -C full uploadpack.allowfilter 1 &&
+	test_config -C full uploadpack.allowanysha1inwant 1 &&

I wasn't sure what these configs are for, but it looks like .allowfilter is to allow 'full' to serve as a remote to a partial clone. But what do you need .allowAnySha1InWant for here? Are we expecting to ask for SHAs that 'full' doesn't have?

We are expecting to ask for SHAs that 'full' doesn't *advertise*, yes (namely, the hash of a certain blob).

Quoted 5 lines
Quoted 5 lines
t/t0410-partial-clone.sh +5 -0
+	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+	FILE_HASH=$(git hash-object --stdin <full/file.txt) &&
+
+	# Sanity check that the file is missing
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
Quoted 8 lines

t/t0410-partial-clone.sh test_expect_success 'do not fetch when checking existence of tree we construct o · line 624

Quoted 4 lines
t/t0410-partial-clone.sh +4 -0
+	grep "[?]$FILE_HASH" out &&
+
+	OUT=$(test-tool partial-clone object-info partial.git "$FILE_HASH") &&
+	test "$OUT" -eq 5 &&

Hm. I guess I am confused about why this fetches the desired object into partial.git. Maybe the test-helper needs a comment (and maybe here too) on the line where fetch will be triggered?

I've added a comment to the test-helper code in v2 - could you take a look and see if that clarifies things? But in any case, the answer is that this test-tool invocation attempts to read an object in the submodule while running as a process in the superproject. The read attempt is a read of a missing object, so that object is lazily fetched.

#55 lore raw
In reply to Elijah Newren
Quoted 11 lines

Looks like Junio did spot some bigger items...which raises a question for me. I have a series
(https://lore.kernel.org/git/[email protected]/)
that also touches partial clones. Our series are semantically independent, but we both add a repository parameter to fetch_objects(). So we both make the same change, but you also make additional nearby changes, resulting in two trivial conflicts. So, should I rebase my series on yours, should you rebase on mine, or should we just let both proceed independently and double-check Junio resolves the trivial conflicts in favor of your side?

Quoted 1 line

Thoughts?

From [1], looks like this is already resolved, but in any case I think we can just let both proceed independently since the conflicts are relatively trivial. If it turns out to be not so trivial, I think Junio can just let one of us know on-list and whoever it is can rebase on the other's.

[1] https://lore.kernel.org/git/[email protected]/

#41 lore raw

Re: [PATCH v2 3/4] run-command: move envvar-resetting function

In reply to Jonathan Tan
Quoted 8 lines · Jonathan Tan
Quoted 5 lines

It does make sense to move this to run-command.c from submodule.c and the function name is already suitable for being global. I however cannot help wondering if we should also pay attention to the GIT_CONFIG_KEY_$n and GIT_CONFIG_VALUE_$n pairs (which is not a new problem in this patch).

Note that I changed the function name (the previous one was too submodule-specific).

Ah, sorry for a stale mention of that thing---in any case, the name is suitable for a global.

Quoted 3 lines

As for the config pairs, they are currently being
passed through - do you have a situation in mind in which they should not be passed through?

Wasn't the GIT_CONFIG_KEY/VALUE meant as a moral equivalent of the GIT_CONFIG_PARAMETERS for those scripts that do not want to bother following the quoting rules of the single parameter approach?

I do not see why we should filter configuration variables passed via one mechanism and let variables passed via the other machanism through. That feels inconsistent (I suspect that there may already be inconsistencies introduced when GIT_CONFIG_KEY/VALUE mechanism was added, though).

#46 lore raw

Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan
Quoted 6 lines · Jonathan Tan
Quoted 3 lines

by filling appropriate environ[] array to be run in a repository that is different from ours (which is "other repo" part of its name) not to want to even know which repository the "other" repo is?

Good point. I'll update prepare_other_repo_env() to have a gitdir parameter.

I actually meant that the function should take an in-core "repo" structure.

Quoted 7 lines
Quoted 3 lines

Object type and object sizes are something that you can safely express in plain text, would be handy for testing, and would not require too much extra code, I'd imagine.

It would, but we can already use "git cat-file -s" (or -t) for that. The helper is meant to test a specific code path wherein we access a submodule object during a process running in the superproject.

I know, but can you use "git cat-file -s" to check the codepath you care about? I do not think so. Hence the suggestion.

#33 lore raw

Re: [PATCH v2 1/4] promisor-remote: read partialClone config here

In reply to Jonathan Tan
Quoted 35 lines · Jonathan Tan
Quoted 33 lines

promisor-remote.c static int promisor_remote_config(const char *var, const char *value, void *data · line 104

Quoted 13 lines
promisor-remote.c +8 -0
@@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data
        size_t namelen;
        const char *subkey;
 
+       if (!strcmp(var, "extensions.partialclone")) {
+               /*
+                * NULL value is handled in handle_extension_v0 in setup.c.
+                */
+               if (value)
+                       repository_format_partial_clone = xstrdup(value);
+               return 0;
+       }

This is actually slightly hard to parse out. I was trying to figure out where repository_format_partial_clone was initialized, and it's not handled when value is NULL in handle_extension_v0; it's the fact that repository_format_partial_clone is declared a static global variable.

But in the next patch you make it a member of struct promisor_remote_config, and instead rely on the xcalloc call in promisor_remote_init().

That means everything is properly initialized and you haven't made any mistakes here, but the logic is a bit hard to follow. Perhaps it'd be nicer to just write this as

promisor-remote.c +4 -0
+       if (!strcmp(var, "extensions.partialclone")) {
+               repository_format_partial_clone = xstrdup_or_null(value);
+               return 0;
+       }
Quoted 1 line

which makes the code shorter and easier to follow, at least for me.

Quoted 10 lines

Hmm...is your concern about the case in which repository_format_partial_clone is uninitialized, or about ignoring a potential NULL value? If the former, I don't see how your suggestion fixes things, since extensions.partialclone may never have been in the config in the first place (and would thus leave repository_format_partial_clone uninitialized, if it weren't for the fact that it is in static storage and thus initialized to 0). If the latter, I guess I should be more detailed about how it's being handled in setup.c (or maybe just leave out the comment altogether - the code here can handle a NULL repository_format_partial_clone for some reason).

My comment was about the latter; I was trying to understand what the comment meant relative to that case, and how and where that case would be handled in the code. With that frame of reference, the comment seemed misleading to me...though perhaps the comment was intended to answer some other question entirely.

#29 lore raw

Re: [PATCH v2 1/4] promisor-remote: read partialClone config here

In reply to Jonathan Tan
Quoted 1 line · Jonathan Tan

I'm reluctant to add more fields to struct repository_format.

I am only suggesting to add one new member to either struct repository or struct repo_settings, so that it becomes crystal clear that struct repository_format is about each single repository, not the global the_repository. Other things that partial repository support needs to keep *and* do not directly come from extensions would not belong to repository_format and should not be added there, but what we read from extensions.* for each repository belongs to each instance of in-core repository structure and should be discoverable starting from "struct repository", no?

#30 lore raw

Re: [PATCH v2 1/4] promisor-remote: read partialClone config here

Quoted 4 lines
Quoted 1 line · Jonathan Tan

I'm reluctant to add more fields to struct repository_format.

Quoted 9 lines

I am only suggesting to add one new member to either struct repository or struct repo_settings, so that it becomes crystal clear that struct repository_format is about each single repository, not the global the_repository. Other things that partial repository support needs to keep *and* do not directly come from extensions would not belong to repository_format and should not be added there, but what we read from extensions.* for each repository belongs to each instance of in-core repository structure and should be discoverable starting from "struct repository", no?

Ah, I see. I'll try this.

#42 lore raw

Re: [PATCH v2 3/4] run-command: move envvar-resetting function

In reply to Junio C Hamano
Quoted 8 lines
Quoted 3 lines

As for the config pairs, they are currently being
passed through - do you have a situation in mind in which they should not be passed through?

Wasn't the GIT_CONFIG_KEY/VALUE meant as a moral equivalent of the GIT_CONFIG_PARAMETERS for those scripts that do not want to bother following the quoting rules of the single parameter approach?

Quoted 5 lines

I do not see why we should filter configuration variables passed via one mechanism and let variables passed via the other machanism through. That feels inconsistent (I suspect that there may already be inconsistencies introduced when GIT_CONFIG_KEY/VALUE mechanism was added, though).

Yes, I think you're right. The filter was added in 14111fc492 ("git:
submodule honor -c credential.* from command line", 2016-03-01) (letting a sanitized version of GIT_CONFIG_PARAMETERS through) and took its present form in 89044baa8b ("submodule: stop sanitizing config options", 2016-05-06) (removing the sanitization). GIT_CONFIG_KEY/VALUE was only added in d8d77153ea ("config: allow specifying config entries via envvar pairs", 2021-01-15).

Reading the commit messages of these commits, I agree that GIT_CONFIG_KEY/VALUE should be treated the same way as GIT_CONFIG_PARAMETERS. In particular, 14111fc492 mentioned passing credentials as the reason for the feature, and in d8d77153ea, the ability to pass secrets without exposing them to a user running "ps" was one of the motivating reasons. I'll add a commit changing this before this commit.

#47 lore raw

Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo

In reply to Junio C Hamano
Quoted 11 lines
Quoted 6 lines · Jonathan Tan
Quoted 3 lines

by filling appropriate environ[] array to be run in a repository that is different from ours (which is "other repo" part of its name) not to want to even know which repository the "other" repo is?

Good point. I'll update prepare_other_repo_env() to have a gitdir parameter.

I actually meant that the function should take an in-core "repo" structure.

But that seems like we're passing much more than we need - we only need the git_dir. Also, there is a function that wants to pass a literal "." as the gitdir; if we do this, I'll have to check if there is still a struct repository that we can pass that will result in the same gitdir.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

Object type and object sizes are something that you can safely express in plain text, would be handy for testing, and would not require too much extra code, I'd imagine.

It would, but we can already use "git cat-file -s" (or -t) for that. The helper is meant to test a specific code path wherein we access a submodule object during a process running in the superproject.

I know, but can you use "git cat-file -s" to check the codepath you care about? I do not think so. Hence the suggestion.

I'm still not convinced that we'll need it in the future, but you're right that it is not too much trouble. I'll add it in.

#48 lore raw

Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo

Quoted 9 lines · Jonathan Tan
Quoted 9 lines
Quoted 6 lines · Jonathan Tan
Quoted 3 lines

by filling appropriate environ[] array to be run in a repository that is different from ours (which is "other repo" part of its name) not to want to even know which repository the "other" repo is?

Good point. I'll update prepare_other_repo_env() to have a gitdir parameter.

Quoted 7 lines
Quoted 2 lines

I actually meant that the function should take an in-core "repo" structure.

But that seems like we're passing much more than we need - we only need the git_dir. Also, there is a function that wants to pass a literal "." as the gitdir; if we do this, I'll have to check if there is still a struct repository that we can pass that will result in the same gitdir.

OK. If the caller at the point does not have anything but git-dir, there may not be much point in instantiating a full in-core repo structure to pass to prepare_other_repo_env() to it. If the helper needs to learn more about that repository, it can go from the git-dir and do things itself.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Object type and object sizes are something that you can safely express in plain text, would be handy for testing, and would not require too much extra code, I'd imagine.

It would, but we can already use "git cat-file -s" (or -t) for that. The helper is meant to test a specific code path wherein we access a submodule object during a process running in the superproject.

Quoted 5 lines
Quoted 2 lines

I know, but can you use "git cat-file -s" to check the codepath you care about? I do not think so. Hence the suggestion.

I'm still not convinced that we'll need it in the future, but you're right that it is not too much trouble. I'll add it in.

As your answer to my initial question was that this is purely a stop-gap testing measure until we get the support fully plumbed in so that the real-world codepath can be tested end-to-end, I do not think it matters all that much. If it is easy to add, I suspect that it would help to catch more bugs, but I wouldn't lose sleep if it doesn't get added.

Thanks.

#34 lore raw

Re: [PATCH v2 1/4] promisor-remote: read partialClone config here

In reply to Elijah Newren
Quoted 11 lines
Quoted 10 lines

Hmm...is your concern about the case in which repository_format_partial_clone is uninitialized, or about ignoring a potential NULL value? If the former, I don't see how your suggestion fixes things, since extensions.partialclone may never have been in the config in the first place (and would thus leave repository_format_partial_clone uninitialized, if it weren't for the fact that it is in static storage and thus initialized to 0). If the latter, I guess I should be more detailed about how it's being handled in setup.c (or maybe just leave out the comment altogether - the code here can handle a NULL repository_format_partial_clone for some reason).

Quoted 5 lines

My comment was about the latter; I was trying to understand what the comment meant relative to that case, and how and where that case would be handled in the code. With that frame of reference, the comment seemed misleading to me...though perhaps the comment was intended to answer some other question entirely.

Junio suggested [1] that repository_format_partial_clone be handled when the repo format is validated, so this part of the code can just make use of the repository_format_partial_clone value in struct repository and not read the config itself. So I believe that this part is now obsolete (but you can take a look at patches 1 and 2 to verify, if you want).

[1] https://lore.kernel.org/git/[email protected]/

#60 lore raw

PATCH v3 2/5 promisor-remote: support per-repository config

In reply to Jonathan Tan

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation.

---
 promisor-remote.c | 98 ++++++++++++++++++++++++++---------------------
 promisor-remote.h | 22 +++++++++--
 repository.c      |  6 +++
 repository.h      |  2 +
 4 files changed, 82 insertions(+), 46 deletions(-)
promisor-remote.c +55 -43
index d24081dc21..1e00e16b0f 100644
@@ -5,6 +5,11 @@
 #include "transport.h"
 #include "strvec.h"
 
+struct promisor_remote_config {
+	struct promisor_remote *promisors;
+	struct promisor_remote **promisors_tail;
+};
+
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
@@ -35,10 +40,8 @@ static int fetch_objects(const char *remote_name,
 	return finish_command(&child) ? -1 : 0;
 }
 
-static struct promisor_remote *promisors;
-static struct promisor_remote **promisors_tail = &promisors;
-
-static struct promisor_remote *promisor_remote_new(const char *remote_name)
+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,
+						   const char *remote_name)
 {
 	struct promisor_remote *r;
 
@@ -50,18 +53,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)
 
 	FLEX_ALLOC_STR(r, name, remote_name);
 
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 
 	return r;
 }
 
-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,
+						      const char *remote_name,
 						      struct promisor_remote **previous)
 {
 	struct promisor_remote *r, *p;
 
-	for (p = NULL, r = promisors; r; p = r, r = r->next)
+	for (p = NULL, r = config->promisors; r; p = r, r = r->next)
 		if (!strcmp(r->name, remote_name)) {
 			if (previous)
 				*previous = p;
@@ -71,7 +75,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
 	return NULL;
 }
 
-static void promisor_remote_move_to_tail(struct promisor_remote *r,
+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,
+					 struct promisor_remote *r,
 					 struct promisor_remote *previous)
 {
 	if (r->next == NULL)
@@ -80,14 +85,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,
 	if (previous)
 		previous->next = r->next;
 	else
-		promisors = r->next ? r->next : r;
+		config->promisors = r->next ? r->next : r;
 	r->next = NULL;
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 }
 
 static int promisor_remote_config(const char *var, const char *value, void *data)
 {
+	struct promisor_remote_config *config = data;
 	const char *name;
 	size_t namelen;
 	const char *subkey;
@@ -103,8 +109,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 
 		remote_name = xmemdupz(name, namelen);
 
-		if (!promisor_remote_lookup(remote_name, NULL))
-			promisor_remote_new(remote_name);
+		if (!promisor_remote_lookup(config, remote_name, NULL))
+			promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 		return 0;
@@ -113,9 +119,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 		struct promisor_remote *r;
 		char *remote_name = xmemdupz(name, namelen);
 
-		r = promisor_remote_lookup(remote_name, NULL);
+		r = promisor_remote_lookup(config, remote_name, NULL);
 		if (!r)
-			r = promisor_remote_new(remote_name);
+			r = promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 
@@ -128,59 +134,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	return 0;
 }
 
-static int initialized;
-
-static void promisor_remote_init(void)
+static void promisor_remote_init(struct repository *r)
 {
-	if (initialized)
+	struct promisor_remote_config *config;
+
+	if (r->promisor_remote_config)
 		return;
-	initialized = 1;
+	config = r->promisor_remote_config =
+		xcalloc(sizeof(*r->promisor_remote_config), 1);
+	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, NULL);
+	repo_config(r, promisor_remote_config, config);
 
-	if (the_repository->repository_format_partial_clone) {
+	if (r->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
 
-		o = promisor_remote_lookup(the_repository->repository_format_partial_clone,
+		o = promisor_remote_lookup(config,
+					   r->repository_format_partial_clone,
 					   &previous);
 		if (o)
-			promisor_remote_move_to_tail(o, previous);
+			promisor_remote_move_to_tail(config, o, previous);
 		else
-			promisor_remote_new(the_repository->repository_format_partial_clone);
+			promisor_remote_new(config, r->repository_format_partial_clone);
 	}
 }
 
-static void promisor_remote_clear(void)
+void promisor_remote_clear(struct promisor_remote_config *config)
 {
-	while (promisors) {
-		struct promisor_remote *r = promisors;
-		promisors = promisors->next;
+	while (config->promisors) {
+		struct promisor_remote *r = config->promisors;
+		config->promisors = config->promisors->next;
 		free(r);
 	}
 
-	promisors_tail = &promisors;
+	config->promisors_tail = &config->promisors;
 }
 
-void promisor_remote_reinit(void)
+void repo_promisor_remote_reinit(struct repository *r)
 {
-	initialized = 0;
-	promisor_remote_clear();
-	promisor_remote_init();
+	promisor_remote_clear(r->promisor_remote_config);
+	FREE_AND_NULL(r->promisor_remote_config);
+	promisor_remote_init(r);
 }
 
-struct promisor_remote *promisor_remote_find(const char *remote_name)
+struct promisor_remote *repo_promisor_remote_find(struct repository *r,
+						  const char *remote_name)
 {
-	promisor_remote_init();
+	promisor_remote_init(r);
 
 	if (!remote_name)
-		return promisors;
+		return r->promisor_remote_config->promisors;
 
-	return promisor_remote_lookup(remote_name, NULL);
+	return promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);
 }
 
-int has_promisor_remote(void)
+int repo_has_promisor_remote(struct repository *r)
 {
-	return !!promisor_remote_find(NULL);
+	return !!repo_promisor_remote_find(r, NULL);
 }
 
 static int remove_fetched_oids(struct repository *repo,
@@ -228,9 +238,11 @@ int promisor_remote_get_direct(struct repository *repo,
 	if (oid_nr == 0)
 		return 0;
 
-	promisor_remote_init();
+	promisor_remote_init(repo);
 
-	for (r = promisors; r; r = r->next) {
+	if (repo != the_repository)
+		BUG("only the_repository is supported for now");
+	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
 		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
promisor-remote.h +19 -3
index 687210ab87..edc45ab0f5 100644
@@ -17,9 +17,25 @@ struct promisor_remote {
 	const char name[FLEX_ARRAY];
 };
 
-void promisor_remote_reinit(void);
-struct promisor_remote *promisor_remote_find(const char *remote_name);
-int has_promisor_remote(void);
+void repo_promisor_remote_reinit(struct repository *r);
+static inline void promisor_remote_reinit(void)
+{
+	repo_promisor_remote_reinit(the_repository);
+}
+
+void promisor_remote_clear(struct promisor_remote_config *config);
+
+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);
+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)
+{
+	return repo_promisor_remote_find(the_repository, remote_name);
+}
+
+int repo_has_promisor_remote(struct repository *r);
+static inline int has_promisor_remote(void)
+{
+	return repo_has_promisor_remote(the_repository);
+}
 
 /*
  * Fetches all requested objects from all promisor remotes, trying them one at
repository.c +6 -0
index 4878c297d9..a14074d964 100644
@@ -11,6 +11,7 @@
 #include "lockfile.h"
 #include "submodule-config.h"
 #include "sparse-index.h"
+#include "promisor-remote.h"
 
 /* The main repository */
 static struct repository the_repo;
@@ -261,6 +262,11 @@ void repo_clear(struct repository *repo)
 		if (repo->index != &the_index)
 			FREE_AND_NULL(repo->index);
 	}
+
+	if (repo->promisor_remote_config) {
+		promisor_remote_clear(repo->promisor_remote_config);
+		FREE_AND_NULL(repo->promisor_remote_config);
+	}
 }
 
 int repo_read_index(struct repository *repo)
repository.h +2 -0
index 6fb16ed336..3740c93bc0 100644
@@ -10,6 +10,7 @@ struct lock_file;
 struct pathspec;
 struct raw_object_store;
 struct submodule_cache;
+struct promisor_remote_config;
 
 enum untracked_cache_setting {
 	UNTRACKED_CACHE_UNSET = -1,
@@ -141,6 +142,7 @@ struct repository {
 
 	/* Configurations related to promisor remotes. */
 	char *repository_format_partial_clone;
+	struct promisor_remote_config *promisor_remote_config;
 
 	/* Configurations */
 
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#67 lore raw

PATCH v3 5/5 promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

Besides that, there is some code that uses the wrapper functions like has_promisor_remote(). Those will need to be checked to see if they could support the non-wrapper functions instead (and thus support any repository, not just the_repository).

---
 Makefile                      |  1 +
 object-file.c                 |  7 ++----
 promisor-remote.c             |  9 ++++----
 t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c          |  1 +
 t/helper/test-tool.h          |  1 +
 t/t0410-partial-clone.sh      | 23 +++++++++++++++++++
 7 files changed, 76 insertions(+), 9 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c
Makefile +1 -0
index c3565fc0f8..f6653bcd5e 100644
@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o
 TEST_BUILTINS_OBJS += test-online-cpus.o
 TEST_BUILTINS_OBJS += test-parse-options.o
 TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
+TEST_BUILTINS_OBJS += test-partial-clone.o
 TEST_BUILTINS_OBJS += test-path-utils.o
 TEST_BUILTINS_OBJS += test-pcre2-config.o
 TEST_BUILTINS_OBJS += test-pkt-line.o
object-file.c +2 -5
index f233b440b2..ebf273e9e7 100644
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
 		}
 
 		/* Check if it is a missing object */
-		if (fetch_if_missing && has_promisor_remote() &&
-		    !already_retried && r == the_repository &&
+		if (fetch_if_missing && repo_has_promisor_remote(r) &&
+		    !already_retried &&
 		    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
 			/*
 			 * TODO Investigate checking promisor_remote_get_direct()
 			 * TODO return value and stopping on error here.
-			 * TODO Pass a repository struct through
-			 * promisor_remote_get_direct(), such that arbitrary
-			 * repositories work.
 			 */
 			promisor_remote_get_direct(r, real, 1);
 			already_retried = 1;
promisor-remote.c +5 -4
index 1e00e16b0f..c088dcbff3 100644
@@ -10,7 +10,8 @@ struct promisor_remote_config {
 	struct promisor_remote **promisors_tail;
 };
 
-static int fetch_objects(const char *remote_name,
+static int fetch_objects(struct repository *repo,
+			 const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
 {
@@ -20,6 +21,8 @@ static int fetch_objects(const char *remote_name,
 
 	child.git_cmd = 1;
 	child.in = -1;
+	if (repo != the_repository)
+		prepare_other_repo_env(&child.env_array, repo->gitdir);
 	strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop",
 		     "fetch", remote_name, "--no-tags",
 		     "--no-write-fetch-head", "--recurse-submodules=no",
@@ -240,10 +243,8 @@ int promisor_remote_get_direct(struct repository *repo,
 
 	promisor_remote_init(repo);
 
-	if (repo != the_repository)
-		BUG("only the_repository is supported for now");
 	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
-		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
+		if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
 			remaining_nr = remove_fetched_oids(repo, &remaining_oids,
t/helper/test-partial-clone.c new file +43 -0

43 lines not on this page · raw message

t/helper/test-tool.c +1 -0
index c5bd0c6d4c..b21e8f1519 100644
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
 	{ "online-cpus", cmd__online_cpus },
 	{ "parse-options", cmd__parse_options },
 	{ "parse-pathspec-file", cmd__parse_pathspec_file },
+	{ "partial-clone", cmd__partial_clone },
 	{ "path-utils", cmd__path_utils },
 	{ "pcre2-config", cmd__pcre2_config },
 	{ "pkt-line", cmd__pkt_line },
t/helper/test-tool.h +1 -0
index e8069a3b22..f845ced4b3 100644
@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
+int cmd__partial_clone(int argc, const char **argv);
 int cmd__path_utils(int argc, const char **argv);
 int cmd__pcre2_config(int argc, const char **argv);
 int cmd__pkt_line(int argc, const char **argv);
t/t0410-partial-clone.sh +23 -0
index 584a039b85..a211a66c67 100755
@@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
 	git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+	rm -rf full partial.git &&
+	test_create_repo full &&
+	test_commit -C full create-a-file file.txt &&
+
+	test_config -C full uploadpack.allowfilter 1 &&
+	test_config -C full uploadpack.allowanysha1inwant 1 &&
+	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+	FILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&
+
+	# Sanity check that the file is missing
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	grep "[?]$FILE_HASH" out &&
+
+	git -C full cat-file -s "$FILE_HASH" >expect &&
+	test-tool partial-clone object-info partial.git "$FILE_HASH" >actual &&
+	test_cmp expect actual &&
+
+	# Sanity check that the file is now present
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	! grep "[?]$FILE_HASH" out
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#57 lore raw

v3 cover First steps towards partial clone submodules

In reply to Jonathan Tan

I think I've addressed all review comments. As for Junio's suggestion about also printing the type in the former patch 4 (now patch 5) [1], I decided to just leave the code as-is and not also print the type.

The main changes are that patch 1 is somewhat rewritten - we still remove the global variable, but we no longer read the extensions.partialClone config directly from promisor-remote.c. Instead, we store it in struct repository when the format of a repository is being verified, and promisor-remote.c merely reads it from there. Patch 3 is a new patch that updates the environment variable preparation before it is moved in patch 4 (formerly patch 3).

[1] https://lore.kernel.org/git/[email protected]/

Jonathan Tan (5):
repository: move global r_f_p_c to repo struct
promisor-remote: support per-repository config
submodule: refrain from filtering GIT_CONFIG_COUNT
run-command: refactor subprocess env preparation
promisor-remote: teach lazy-fetch in any repo

 Makefile                      |   1 +
 object-file.c                 |   7 +--
 promisor-remote.c             | 108 ++++++++++++++++++----------------
 promisor-remote.h             |  28 ++++++---
 repository.c                  |   9 +++
 repository.h                  |   5 ++
 run-command.c                 |  12 ++++
 run-command.h                 |  10 ++++
 setup.c                       |  16 +++--
 submodule.c                   |  17 +-----
 t/helper/test-partial-clone.c |  43 ++++++++++++++
 t/helper/test-tool.c          |   1 +
 t/helper/test-tool.h          |   1 +
 t/t0410-partial-clone.sh      |  23 ++++++++
 14 files changed, 196 insertions(+), 85 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c

Range-diff against v2:

1:  d99598ca50 < -:  ---------- promisor-remote: read partialClone config here
-:  ---------- > 1:  255d112256 repository: move global r_f_p_c to repo struct
2:  5a1ccae335 ! 2:  a52448cff2 promisor-remote: support per-repository config
    @@ promisor-remote.c
      #include "transport.h"
      #include "strvec.h"
      
    --static char *repository_format_partial_clone;
     +struct promisor_remote_config {
    -+	char *repository_format_partial_clone;
     +	struct promisor_remote *promisors;
     +	struct promisor_remote **promisors_tail;
     +};
    - 
    ++
      static int fetch_objects(const char *remote_name,
      			 const struct object_id *oids,
    + 			 int oid_nr)
     @@ promisor-remote.c: static int fetch_objects(const char *remote_name,
      	return finish_command(&child) ? -1 : 0;
      }
    @@ promisor-remote.c: static void promisor_remote_move_to_tail(struct promisor_remo
      	const char *name;
      	size_t namelen;
      	const char *subkey;
    -@@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data
    - 		 * NULL value is handled in handle_extension_v0 in setup.c.
    - 		 */
    - 		if (value)
    --			repository_format_partial_clone = xstrdup(value);
    -+			config->repository_format_partial_clone = xstrdup(value);
    - 		return 0;
    - 	}
    - 
     @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data
      
      		remote_name = xmemdupz(name, namelen);
    @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char
     +	config->promisors_tail = &config->promisors;
      
     -	git_config(promisor_remote_config, NULL);
    -+	git_config(promisor_remote_config, config);
    ++	repo_config(r, promisor_remote_config, config);
      
    --	if (repository_format_partial_clone) {
    -+	if (config->repository_format_partial_clone) {
    +-	if (the_repository->repository_format_partial_clone) {
    ++	if (r->repository_format_partial_clone) {
      		struct promisor_remote *o, *previous;
      
    --		o = promisor_remote_lookup(repository_format_partial_clone,
    +-		o = promisor_remote_lookup(the_repository->repository_format_partial_clone,
     +		o = promisor_remote_lookup(config,
    -+					   config->repository_format_partial_clone,
    ++					   r->repository_format_partial_clone,
      					   &previous);
      		if (o)
     -			promisor_remote_move_to_tail(o, previous);
     +			promisor_remote_move_to_tail(config, o, previous);
      		else
    --			promisor_remote_new(repository_format_partial_clone);
    -+			promisor_remote_new(config, config->repository_format_partial_clone);
    +-			promisor_remote_new(the_repository->repository_format_partial_clone);
    ++			promisor_remote_new(config, r->repository_format_partial_clone);
      	}
      }
      
    @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char
     -	while (promisors) {
     -		struct promisor_remote *r = promisors;
     -		promisors = promisors->next;
    -+	FREE_AND_NULL(config->repository_format_partial_clone);
    -+
     +	while (config->promisors) {
     +		struct promisor_remote *r = config->promisors;
     +		config->promisors = config->promisors->next;
    @@ repository.h: struct lock_file;
      enum untracked_cache_setting {
      	UNTRACKED_CACHE_UNSET = -1,
     @@ repository.h: struct repository {
    - 	/* True if commit-graph has been disabled within this process. */
    - 	int commit_graph_disabled;
      
    -+	/* Configurations related to promisor remotes. */
    + 	/* Configurations related to promisor remotes. */
    + 	char *repository_format_partial_clone;
     +	struct promisor_remote_config *promisor_remote_config;
    -+
    + 
      	/* Configurations */
      
    - 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
-:  ---------- > 3:  e1a40108f4 submodule: refrain from filtering GIT_CONFIG_COUNT
3:  3f7c4e6e67 ! 4:  fd6907822c run-command: move envvar-resetting function
    @@ Metadata
     Author: Jonathan Tan <[email protected]>
     
      ## Commit message ##
    -    run-command: move envvar-resetting function
    +    run-command: refactor subprocess env preparation
     
    -    There is a function that resets environment variables, used when
    -    invoking a sub-process in a submodule. The lazy-fetching code (used in
    -    partial clones) will need this function in a subsequent commit, so move
    -    it to a more central location.
    +    submodule.c has functionality that prepares the environment for running
    +    a subprocess in a new repo. The lazy-fetching code (used in partial
    +    clones) will need this in a subsequent commit, so move it to a more
    +    central location.
     
         Signed-off-by: Jonathan Tan <[email protected]>
     
    @@ run-command.c: int run_auto_maintenance(int quiet)
      	return run_command(&maint);
      }
     +
    -+void prepare_other_repo_env(struct strvec *env_array)
    ++void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)
     +{
     +	const char * const *var;
     +
     +	for (var = local_repo_env; *var; var++) {
    -+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
    ++		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
    ++		    strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
     +			strvec_push(env_array, *var);
     +	}
    ++	strvec_pushf(env_array, "%s=%s", GIT_DIR_ENVIRONMENT, new_git_dir);
     +}
     
      ## run-command.h ##
    @@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai
      			       const char *tr2_category, const char *tr2_label);
      
     +/**
    -+ * Convenience function which adds all GIT_* environment variables to env_array
    -+ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a
    -+ * subprocess, these entries cause the corresponding environment variables to
    -+ * be unset in the subprocess. See local_repo_env in cache.h for more
    ++ * Convenience function which prepares env_array for a command to be run in a
    ++ * new repo. This adds all GIT_* environment variables to env_array with the
    ++ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding
    ++ * environment variables to be unset in the subprocess) and adds an environment
    ++ * variable pointing to new_git_dir. See local_repo_env in cache.h for more
     + * information.
     + */
    -+void prepare_other_repo_env(struct strvec *env_array);
    ++void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);
     +
      #endif
     
    @@ submodule.c: static void print_submodule_diff_summary(struct repository *r, stru
     -	const char * const *var;
     -
     -	for (var = local_repo_env; *var; var++) {
    --		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
    +-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
    +-		    strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
     -			strvec_push(out, *var);
     -	}
     -}
    @@ submodule.c: static void print_submodule_diff_summary(struct repository *r, stru
      void prepare_submodule_repo_env(struct strvec *out)
      {
     -	prepare_submodule_repo_env_no_git_dir(out);
    -+	prepare_other_repo_env(out);
    - 	strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
    - 		     DEFAULT_GIT_DIR_ENVIRONMENT);
    +-	strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
    +-		     DEFAULT_GIT_DIR_ENVIRONMENT);
    ++	prepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);
      }
      
      static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
      {
     -	prepare_submodule_repo_env_no_git_dir(out);
    -+	prepare_other_repo_env(out);
    - 	strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
    +-	strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
    ++	prepare_other_repo_env(out, ".");
      }
      
    + /*
4:  655607d575 ! 5:  a6d73662b1 promisor-remote: teach lazy-fetch in any repo
    @@ Commit message
         prevents testing of the functionality in this patch by user-facing
         commands. So for now, test this mechanism using a test helper.
     
    +    Besides that, there is some code that uses the wrapper functions
    +    like has_promisor_remote(). Those will need to be checked to see if they
    +    could support the non-wrapper functions instead (and thus support any
    +    repository, not just the_repository).
    +
         Signed-off-by: Jonathan Tan <[email protected]>
     
      ## Makefile ##
    @@ promisor-remote.c: static int fetch_objects(const char *remote_name,
      
      	child.git_cmd = 1;
      	child.in = -1;
    -+	if (repo != the_repository) {
    -+		prepare_other_repo_env(&child.env_array);
    -+		strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT,
    -+			     repo->gitdir);
    -+	}
    ++	if (repo != the_repository)
    ++		prepare_other_repo_env(&child.env_array, repo->gitdir);
      	strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop",
      		     "fetch", remote_name, "--no-tags",
      		     "--no-write-fetch-head", "--recurse-submodules=no",
    -@@ promisor-remote.c: static void promisor_remote_init(struct repository *r)
    - 		xcalloc(sizeof(*r->promisor_remote_config), 1);
    - 	config->promisors_tail = &config->promisors;
    - 
    --	git_config(promisor_remote_config, config);
    -+	repo_config(r, promisor_remote_config, config);
    - 
    - 	if (config->repository_format_partial_clone) {
    - 		struct promisor_remote *o, *previous;
     @@ promisor-remote.c: int promisor_remote_get_direct(struct repository *repo,
      
      	promisor_remote_init(repo);
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#58 lore raw

PATCH v3 1/5 repository: move global r_f_p_c to repo struct

Move repository_format_partial_clone, which is currently a global variable, into struct repository. (Full support for per-repository partial clone config will be done in a subsequent commit - this is split into its own commit because of the extent of the changes needed.)

The new repo-specific variable cannot be set in check_repository_format_gently() (as is currently), because that function does not know which repo it is operating on (or even whether the value is important); therefore this responsibility is delegated to the outermost caller that knows. Of all the outermost callers that know (found by looking at all functions that call clear_repository_format()), I looked at those that either read from the main Git directory or write into a struct repository. These callers have been modified accordingly (write to the_repository in the former case and write to the given struct repository in the latter case).

---
 promisor-remote.c | 13 +++----------
 promisor-remote.h |  6 ------
 repository.c      |  3 +++
 repository.h      |  3 +++
 setup.c           | 16 +++++++++++-----
 5 files changed, 20 insertions(+), 21 deletions(-)
promisor-remote.c +3 -10
index da3f2ca261..d24081dc21 100644
@@ -5,13 +5,6 @@
 #include "transport.h"
 #include "strvec.h"
 
-static char *repository_format_partial_clone;
-
-void set_repository_format_partial_clone(char *partial_clone)
-{
-	repository_format_partial_clone = xstrdup_or_null(partial_clone);
-}
-
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
@@ -145,15 +138,15 @@ static void promisor_remote_init(void)
 
 	git_config(promisor_remote_config, NULL);
 
-	if (repository_format_partial_clone) {
+	if (the_repository->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
 
-		o = promisor_remote_lookup(repository_format_partial_clone,
+		o = promisor_remote_lookup(the_repository->repository_format_partial_clone,
 					   &previous);
 		if (o)
 			promisor_remote_move_to_tail(o, previous);
 		else
-			promisor_remote_new(repository_format_partial_clone);
+			promisor_remote_new(the_repository->repository_format_partial_clone);
 	}
 }
 
promisor-remote.h +0 -6
index c7a14063c5..687210ab87 100644
@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,
 			       const struct object_id *oids,
 			       int oid_nr);
 
-/*
- * This should be used only once from setup.c to set the value we got
- * from the extensions.partialclone config option.
- */
-void set_repository_format_partial_clone(char *partial_clone);
-
 #endif /* PROMISOR_REMOTE_H */
repository.c +3 -0
index 448cd557d4..4878c297d9 100644
@@ -172,6 +172,9 @@ int repo_init(struct repository *repo,
 
 	repo_set_hash_algo(repo, format.hash_algo);
 
+	repo->repository_format_partial_clone = format.partial_clone;
+	format.partial_clone = NULL;
+
 	if (worktree)
 		repo_set_worktree(repo, worktree);
 
repository.h +3 -0
index a45f7520fd..6fb16ed336 100644
@@ -139,6 +139,9 @@ struct repository {
 	/* True if commit-graph has been disabled within this process. */
 	int commit_graph_disabled;
 
+	/* Configurations related to promisor remotes. */
+	char *repository_format_partial_clone;
+
 	/* Configurations */
 
 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
setup.c +11 -5
index 59e2facd9d..fbedfe8e03 100644
@@ -468,8 +468,6 @@ static enum extension_result handle_extension_v0(const char *var,
 			data->precious_objects = git_config_bool(var, value);
 			return EXTENSION_OK;
 		} else if (!strcmp(ext, "partialclone")) {
-			if (!value)
-				return config_error_nonbool(var);
 			data->partial_clone = xstrdup(value);
 			return EXTENSION_OK;
 		} else if (!strcmp(ext, "worktreeconfig")) {
@@ -566,7 +564,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_
 	}
 
 	repository_format_precious_objects = candidate->precious_objects;
-	set_repository_format_partial_clone(candidate->partial_clone);
 	repository_format_worktree_config = candidate->worktree_config;
 	string_list_clear(&candidate->unknown_extensions, 0);
 	string_list_clear(&candidate->v1_only_extensions, 0);
@@ -1193,6 +1190,10 @@ int discover_git_directory(struct strbuf *commondir,
 		return -1;
 	}
 
+	the_repository->repository_format_partial_clone =
+		candidate.partial_clone;
+	candidate.partial_clone = NULL;
+
 	clear_repository_format(&candidate);
 	return 0;
 }
@@ -1300,8 +1301,12 @@ const char *setup_git_directory_gently(int *nongit_ok)
 				gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
 			setup_git_env(gitdir);
 		}
-		if (startup_info->have_repository)
+		if (startup_info->have_repository) {
 			repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
+			the_repository->repository_format_partial_clone =
+				repo_fmt.partial_clone;
+			repo_fmt.partial_clone = NULL;
+		}
 	}
 	/*
 	 * Since precompose_string_if_needed() needs to look at
@@ -1319,7 +1324,6 @@ const char *setup_git_directory_gently(int *nongit_ok)
 		setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
 	}
 
-
 	strbuf_release(&dir);
 	strbuf_release(&gitdir);
 	clear_repository_format(&repo_fmt);
@@ -1386,6 +1390,8 @@ void check_repository_format(struct repository_format *fmt)
 	check_repository_format_gently(get_git_dir(), fmt, NULL);
 	startup_info->have_repository = 1;
 	repo_set_hash_algo(the_repository, fmt->hash_algo);
+	the_repository->repository_format_partial_clone =
+		xstrdup_or_null(fmt->partial_clone);
 	clear_repository_format(&repo_fmt);
 }
 
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#61 lore raw

PATCH v3 3/5 submodule: refrain from filtering GIT_CONFIG_COUNT

In reply to Jonathan Tan

14111fc492 ("git: submodule honor -c credential.* from command line", 2016-03-01) taught Git to pass through the GIT_CONFIG_PARAMETERS environment variable when invoking a subprocess on behalf of a submodule. But when d8d77153ea ("config: allow specifying config entries via envvar pairs", 2021-01-15) introduced support for GIT_CONFIG_COUNT (and its associated GIT_CONFIG_KEY_? and GIT_CONFIG_VALUE_?), the subprocess mechanism wasn't updated to also pass through these variables.

Since they are conceptually the same (d8d77153ea was written to address a shortcoming of GIT_CONFIG_PARAMETERS), update the submodule subprocess mechanism to also pass through GIT_CONFIG_COUNT.

---
 submodule.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
submodule.c +2 -1
index 0b1d9c1dde..f09031e397 100644
@@ -489,7 +489,8 @@ static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
 	const char * const *var;
 
 	for (var = local_repo_env; *var; var++) {
-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
+		    strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
 			strvec_push(out, *var);
 	}
 }
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#65 lore raw

PATCH v3 4/5 run-command: refactor subprocess env preparation

In reply to Jonathan Tan

submodule.c has functionality that prepares the environment for running a subprocess in a new repo. The lazy-fetching code (used in partial clones) will need this in a subsequent commit, so move it to a more central location.

---
 run-command.c | 12 ++++++++++++
 run-command.h | 10 ++++++++++
 submodule.c   | 18 ++----------------
 3 files changed, 24 insertions(+), 16 deletions(-)
run-command.c +12 -0
index be6bc128cd..549a94a6a4 100644
@@ -1892,3 +1892,15 @@ int run_auto_maintenance(int quiet)
 
 	return run_command(&maint);
 }
+
+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)
+{
+	const char * const *var;
+
+	for (var = local_repo_env; *var; var++) {
+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
+		    strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
+			strvec_push(env_array, *var);
+	}
+	strvec_pushf(env_array, "%s=%s", GIT_DIR_ENVIRONMENT, new_git_dir);
+}
run-command.h +10 -0
index d08414a92e..92f1c00b11 100644
@@ -483,4 +483,14 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
 			       task_finished_fn, void *pp_cb,
 			       const char *tr2_category, const char *tr2_label);
 
+/**
+ * Convenience function which prepares env_array for a command to be run in a
+ * new repo. This adds all GIT_* environment variables to env_array with the
+ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding
+ * environment variables to be unset in the subprocess) and adds an environment
+ * variable pointing to new_git_dir. See local_repo_env in cache.h for more
+ * information.
+ */
+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);
+
 #endif
submodule.c +2 -16
index f09031e397..8e611fe1db 100644
@@ -484,28 +484,14 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *
 	strbuf_release(&sb);
 }
 
-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
-{
-	const char * const *var;
-
-	for (var = local_repo_env; *var; var++) {
-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
-		    strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
-			strvec_push(out, *var);
-	}
-}
-
 void prepare_submodule_repo_env(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
-	strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
-		     DEFAULT_GIT_DIR_ENVIRONMENT);
+	prepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);
 }
 
 static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
-	strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
+	prepare_other_repo_env(out, ".");
 }
 
 /*
Signature · 1 line
2.32.0.rc1.229.g3e70b5a671-goog
#59 lore raw

Re: [PATCH v3 1/5] repository: move global r_f_p_c to repo struct

In reply to Jonathan Tan
Quoted 83 lines · Jonathan Tan

Move repository_format_partial_clone, which is currently a global variable, into struct repository. (Full support for per-repository partial clone config will be done in a subsequent commit - this is split into its own commit because of the extent of the changes needed.)

The new repo-specific variable cannot be set in check_repository_format_gently() (as is currently), because that function does not know which repo it is operating on (or even whether the value is important); therefore this responsibility is delegated to the outermost caller that knows. Of all the outermost callers that know (found by looking at all functions that call clear_repository_format()), I looked at those that either read from the main Git directory or write into a struct repository. These callers have been modified accordingly (write to the_repository in the former case and write to the given struct repository in the latter case).

---
 promisor-remote.c | 13 +++----------
 promisor-remote.h |  6 ------
 repository.c      |  3 +++
 repository.h      |  3 +++
 setup.c           | 16 +++++++++++-----
 5 files changed, 20 insertions(+), 21 deletions(-)
promisor-remote.c +3 -10
index da3f2ca261..d24081dc21 100644
@@ -5,13 +5,6 @@
 #include "transport.h"
 #include "strvec.h"
 
-static char *repository_format_partial_clone;
-
-void set_repository_format_partial_clone(char *partial_clone)
-{
-       repository_format_partial_clone = xstrdup_or_null(partial_clone);
-}
-
 static int fetch_objects(const char *remote_name,
                         const struct object_id *oids,
                         int oid_nr)
@@ -145,15 +138,15 @@ static void promisor_remote_init(void)
 
        git_config(promisor_remote_config, NULL);
 
-       if (repository_format_partial_clone) {
+       if (the_repository->repository_format_partial_clone) {
                struct promisor_remote *o, *previous;
 
-               o = promisor_remote_lookup(repository_format_partial_clone,
+               o = promisor_remote_lookup(the_repository->repository_format_partial_clone,
                                           &previous);
                if (o)
                        promisor_remote_move_to_tail(o, previous);
                else
-                       promisor_remote_new(repository_format_partial_clone);
+                       promisor_remote_new(the_repository->repository_format_partial_clone);
        }
 }
 
promisor-remote.h +0 -6
index c7a14063c5..687210ab87 100644
@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,
                               const struct object_id *oids,
                               int oid_nr);
 
-/*
- * This should be used only once from setup.c to set the value we got
- * from the extensions.partialclone config option.
- */
-void set_repository_format_partial_clone(char *partial_clone);
-
 #endif /* PROMISOR_REMOTE_H */
repository.c +0 -0
index 448cd557d4..4878c297d9 100644

repository.c int repo_init(struct repository *repo, · line 176

Quoted 5 lines
repository.c +2 -0
@@ -172,6 +172,9 @@ int repo_init(struct repository *repo,
 
        repo_set_hash_algo(repo, format.hash_algo);
 
+       repo->repository_format_partial_clone = format.partial_clone;
+       format.partial_clone = NULL;

This was surprising to me, and I tried to dig around to find out why you set it to NULL. AFAICT, you're trying to avoid the need to do a xstrdup(), so you take over ownership in the first line and set to NULL in the second to avoid a double-free. So, it makes sense, but given how surprising it was and it took me a while to figure it out, perhaps it's worth adding a comment that this is what you're doing here? The same comment would also apply in a few other places in this patch...

Quoted 40 lines
repository.c +1 -0
+
        if (worktree)
                repo_set_worktree(repo, worktree);
 
repository.h +3 -0
index a45f7520fd..6fb16ed336 100644
@@ -139,6 +139,9 @@ struct repository {
        /* True if commit-graph has been disabled within this process. */
        int commit_graph_disabled;
 
+       /* Configurations related to promisor remotes. */
+       char *repository_format_partial_clone;
+
        /* Configurations */
 
        /* Indicate if a repository has a different 'commondir' from 'gitdir' */
setup.c +0 -3
index 59e2facd9d..fbedfe8e03 100644
@@ -468,8 +468,6 @@ static enum extension_result handle_extension_v0(const char *var,
                        data->precious_objects = git_config_bool(var, value);
                        return EXTENSION_OK;
                } else if (!strcmp(ext, "partialclone")) {
-                       if (!value)
-                               return config_error_nonbool(var);
                        data->partial_clone = xstrdup(value);
                        return EXTENSION_OK;
                } else if (!strcmp(ext, "worktreeconfig")) {
@@ -566,7 +564,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_
        }
 
        repository_format_precious_objects = candidate->precious_objects;
-       set_repository_format_partial_clone(candidate->partial_clone);
        repository_format_worktree_config = candidate->worktree_config;
        string_list_clear(&candidate->unknown_extensions, 0);
        string_list_clear(&candidate->v1_only_extensions, 0);

setup.c int discover_git_directory(struct strbuf *commondir, · line 1196

Quoted 7 lines
setup.c +4 -0
@@ -1193,6 +1190,10 @@ int discover_git_directory(struct strbuf *commondir,
                return -1;
        }
 
+       the_repository->repository_format_partial_clone =
+               candidate.partial_clone;
+       candidate.partial_clone = NULL;
+

comment would also be nice here

setup.c const char *setup_git_directory_gently(int *nongit_ok) · line 1308

Quoted 13 lines
setup.c +4 -1
        clear_repository_format(&candidate);
        return 0;
 }
@@ -1300,8 +1301,12 @@ const char *setup_git_directory_gently(int *nongit_ok)
                                gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
                        setup_git_env(gitdir);
                }
-               if (startup_info->have_repository)
+               if (startup_info->have_repository) {
                        repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
+                       the_repository->repository_format_partial_clone =
+                               repo_fmt.partial_clone;
+                       repo_fmt.partial_clone = NULL;

and here

setup.c const char *setup_git_directory_gently(int *nongit_ok) · line 1322

Quoted 9 lines
setup.c +1 -1
+               }
        }
        /*
         * Since precompose_string_if_needed() needs to look at
@@ -1319,7 +1324,6 @@ const char *setup_git_directory_gently(int *nongit_ok)
                setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
        }
 
-

why the stray whitespace change?

Quoted 13 lines
setup.c +2 -1
        strbuf_release(&dir);
        strbuf_release(&gitdir);
        clear_repository_format(&repo_fmt);
@@ -1386,6 +1390,8 @@ void check_repository_format(struct repository_format *fmt)
        check_repository_format_gently(get_git_dir(), fmt, NULL);
        startup_info->have_repository = 1;
        repo_set_hash_algo(the_repository, fmt->hash_algo);
+       the_repository->repository_format_partial_clone =
+               xstrdup_or_null(fmt->partial_clone);
        clear_repository_format(&repo_fmt);
 }
 
--
Quoted 1 line

2.32.0.rc1.229.g3e70b5a671-goog

Other than the minor nits above, this looks good.

#62 lore raw

Re: [PATCH v3 3/5] submodule: refrain from filtering GIT_CONFIG_COUNT

In reply to Jonathan Tan

Adding Peff to cc as per comments about 89044baa8b ("submodule: stop sanitizing config options", 2016-05-04) below.

Quoted 34 lines · Jonathan Tan

14111fc492 ("git: submodule honor -c credential.* from command line", 2016-03-01) taught Git to pass through the GIT_CONFIG_PARAMETERS environment variable when invoking a subprocess on behalf of a submodule. But when d8d77153ea ("config: allow specifying config entries via envvar pairs", 2021-01-15) introduced support for GIT_CONFIG_COUNT (and its associated GIT_CONFIG_KEY_? and GIT_CONFIG_VALUE_?), the subprocess mechanism wasn't updated to also pass through these variables.

Since they are conceptually the same (d8d77153ea was written to address a shortcoming of GIT_CONFIG_PARAMETERS), update the submodule subprocess mechanism to also pass through GIT_CONFIG_COUNT.

---
 submodule.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
submodule.c +2 -2
index 0b1d9c1dde..f09031e397 100644
@@ -489,7 +489,8 @@ static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
        const char * const *var;
 
        for (var = local_repo_env; *var; var++) {
-               if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+               if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
+                   strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
                        strvec_push(out, *var);
        }
 }
--
Quoted 1 line

2.32.0.rc1.229.g3e70b5a671-goog

I'm super confused. It appears that prepare_submodule_repo_env_no_git_dir() is filtering out "GIT_CONFIG_PARAMETERS" (CONFIG_DATA_ENVIRONMENT) and "GIT_CONFIG_COUNT" (CONFIG_COUNT_ENVIRONMENT), using all environment variables other than these ones. But the commit message talks about adding an extra environment variable, rather than filtering another out. I must be mis-reading something somewhere, but I'm struggling to figure it out.

Digging around for a while led me to commit 89044baa8b ("submodule:
stop sanitizing config options", 2016-05-04), which suggests that the passing of GIT_CONFIG_PARAMETERS is not done here but in git-submodule.sh. It still didn't make it clear to me why it's stripped out here, but something makes me thing that git-submodule.sh should be affected by your change as well.

Also, from looking at the other commit messages you reference, it appears GIT_CONFIG_PARAMETERS was just one big environment variable, whereas GIT_CONFIG_COUNT is closely associated with 2*N other environment variables...so shouldn't your loop (and perhaps also git-submodule.sh) also be checking GIT_CONFIG_KEY_\d+ and GIT_CONFIG_VALUE_\d+ ?

I've been looking at this patch longer than I care to admit and I still feel like I don't have a clue what's going on.

#66 lore raw

Re: [PATCH v3 4/5] run-command: refactor subprocess env preparation

In reply to Jonathan Tan
Quoted 38 lines · Jonathan Tan

submodule.c has functionality that prepares the environment for running a subprocess in a new repo. The lazy-fetching code (used in partial clones) will need this in a subsequent commit, so move it to a more central location.

---
 run-command.c | 12 ++++++++++++
 run-command.h | 10 ++++++++++
 submodule.c   | 18 ++----------------
 3 files changed, 24 insertions(+), 16 deletions(-)
run-command.c +12 -0
index be6bc128cd..549a94a6a4 100644
@@ -1892,3 +1892,15 @@ int run_auto_maintenance(int quiet)
 
        return run_command(&maint);
 }
+
+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)
+{
+       const char * const *var;
+
+       for (var = local_repo_env; *var; var++) {
+               if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
+                   strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
+                       strvec_push(env_array, *var);
+       }
+       strvec_pushf(env_array, "%s=%s", GIT_DIR_ENVIRONMENT, new_git_dir);
+}
run-command.h +0 -0
index d08414a92e..92f1c00b11 100644

run-command.h int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn, · line 492

Quoted 10 lines
run-command.h +7 -0
@@ -483,4 +483,14 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
                               task_finished_fn, void *pp_cb,
                               const char *tr2_category, const char *tr2_label);
 
+/**
+ * Convenience function which prepares env_array for a command to be run in a
+ * new repo. This adds all GIT_* environment variables to env_array with the
+ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding
+ * environment variables to be unset in the subprocess) and adds an environment
+ * variable pointing to new_git_dir. See local_repo_env in cache.h for more
+ * information.

This comment is out-of-date as of your previous patch. There's (at least) one more variable that is also excluded.

Quoted 41 lines
run-command.h +3 -0
+ */
+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);
+
 #endif
submodule.c +2 -17
index f09031e397..8e611fe1db 100644
@@ -484,28 +484,14 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *
        strbuf_release(&sb);
 }
 
-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
-{
-       const char * const *var;
-
-       for (var = local_repo_env; *var; var++) {
-               if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
-                   strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
-                       strvec_push(out, *var);
-       }
-}
-
 void prepare_submodule_repo_env(struct strvec *out)
 {
-       prepare_submodule_repo_env_no_git_dir(out);
-       strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
-                    DEFAULT_GIT_DIR_ENVIRONMENT);
+       prepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);
 }
 
 static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
 {
-       prepare_submodule_repo_env_no_git_dir(out);
-       strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
+       prepare_other_repo_env(out, ".");
 }
 
 /*
--

2.32.0.rc1.229.g3e70b5a671-goog

#68 lore raw
In reply to Jonathan Tan
Quoted 5 lines · Jonathan Tan

I think I've addressed all review comments. As for Junio's suggestion about also printing the type in the former patch 4 (now patch 5) [1], I decided to just leave the code as-is and not also print the type.

Quoted 7 lines

The main changes are that patch 1 is somewhat rewritten - we still remove the global variable, but we no longer read the extensions.partialClone config directly from promisor-remote.c. Instead, we store it in struct repository when the format of a repository is being verified, and promisor-remote.c merely reads it from there. Patch 3 is a new patch that updates the environment variable preparation before it is moved in patch 4 (formerly patch 3).

I've read through all the patches. 2 & 5 look good to me, I had small nitpicks on 1 & 4, and I'm totally lost on patch 3. Patch 3 is just a one-liner and it might be fine, but for some reason I can't figure out the code before or after the patch even after digging around into other commits and other files to try to get my bearings. Hopefully someone else can comment on that one.

Quoted 253 lines

[1] https://lore.kernel.org/git/[email protected]/

Jonathan Tan (5):
repository: move global r_f_p_c to repo struct
promisor-remote: support per-repository config
submodule: refrain from filtering GIT_CONFIG_COUNT
run-command: refactor subprocess env preparation
promisor-remote: teach lazy-fetch in any repo

 Makefile                      |   1 +
 object-file.c                 |   7 +--
 promisor-remote.c             | 108 ++++++++++++++++++----------------
 promisor-remote.h             |  28 ++++++---
 repository.c                  |   9 +++
 repository.h                  |   5 ++
 run-command.c                 |  12 ++++
 run-command.h                 |  10 ++++
 setup.c                       |  16 +++--
 submodule.c                   |  17 +-----
 t/helper/test-partial-clone.c |  43 ++++++++++++++
 t/helper/test-tool.c          |   1 +
 t/helper/test-tool.h          |   1 +
 t/t0410-partial-clone.sh      |  23 ++++++++
 14 files changed, 196 insertions(+), 85 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c

Range-diff against v2:

1:  d99598ca50 < -:  ---------- promisor-remote: read partialClone config here
-:  ---------- > 1:  255d112256 repository: move global r_f_p_c to repo struct
2:  5a1ccae335 ! 2:  a52448cff2 promisor-remote: support per-repository config
    @@ promisor-remote.c
      #include "transport.h"
      #include "strvec.h"

    --static char *repository_format_partial_clone;
     +struct promisor_remote_config {
    -+  char *repository_format_partial_clone;
     +  struct promisor_remote *promisors;
     +  struct promisor_remote **promisors_tail;
     +};
    -
    ++
      static int fetch_objects(const char *remote_name,
                         const struct object_id *oids,
    +                    int oid_nr)
     @@ promisor-remote.c: static int fetch_objects(const char *remote_name,
        return finish_command(&child) ? -1 : 0;
      }
    @@ promisor-remote.c: static void promisor_remote_move_to_tail(struct promisor_remo
        const char *name;
        size_t namelen;
        const char *subkey;
    -@@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data
    -            * NULL value is handled in handle_extension_v0 in setup.c.
    -            */
    -           if (value)
    --                  repository_format_partial_clone = xstrdup(value);
    -+                  config->repository_format_partial_clone = xstrdup(value);
    -           return 0;
    -   }
    -
     @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data

                remote_name = xmemdupz(name, namelen);
    @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char
     +  config->promisors_tail = &config->promisors;

     -  git_config(promisor_remote_config, NULL);
    -+  git_config(promisor_remote_config, config);
    ++  repo_config(r, promisor_remote_config, config);

    --  if (repository_format_partial_clone) {
    -+  if (config->repository_format_partial_clone) {
    +-  if (the_repository->repository_format_partial_clone) {
    ++  if (r->repository_format_partial_clone) {
                struct promisor_remote *o, *previous;

    --          o = promisor_remote_lookup(repository_format_partial_clone,
    +-          o = promisor_remote_lookup(the_repository->repository_format_partial_clone,
     +          o = promisor_remote_lookup(config,
    -+                                     config->repository_format_partial_clone,
    ++                                     r->repository_format_partial_clone,
                                           &previous);
                if (o)
     -                  promisor_remote_move_to_tail(o, previous);
     +                  promisor_remote_move_to_tail(config, o, previous);
                else
    --                  promisor_remote_new(repository_format_partial_clone);
    -+                  promisor_remote_new(config, config->repository_format_partial_clone);
    +-                  promisor_remote_new(the_repository->repository_format_partial_clone);
    ++                  promisor_remote_new(config, r->repository_format_partial_clone);
        }
      }

    @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char
     -  while (promisors) {
     -          struct promisor_remote *r = promisors;
     -          promisors = promisors->next;
    -+  FREE_AND_NULL(config->repository_format_partial_clone);
    -+
     +  while (config->promisors) {
     +          struct promisor_remote *r = config->promisors;
     +          config->promisors = config->promisors->next;
    @@ repository.h: struct lock_file;
      enum untracked_cache_setting {
        UNTRACKED_CACHE_UNSET = -1,
     @@ repository.h: struct repository {
    -   /* True if commit-graph has been disabled within this process. */
    -   int commit_graph_disabled;

    -+  /* Configurations related to promisor remotes. */
    +   /* Configurations related to promisor remotes. */
    +   char *repository_format_partial_clone;
     +  struct promisor_remote_config *promisor_remote_config;
    -+
    +
        /* Configurations */

    -   /* Indicate if a repository has a different 'commondir' from 'gitdir' */
-:  ---------- > 3:  e1a40108f4 submodule: refrain from filtering GIT_CONFIG_COUNT
3:  3f7c4e6e67 ! 4:  fd6907822c run-command: move envvar-resetting function

@@ Metadata
Author: Jonathan Tan <[email protected]>

## Commit message ##
- run-command: move envvar-resetting function
+ run-command: refactor subprocess env preparation

- There is a function that resets environment variables, used when
- invoking a sub-process in a submodule. The lazy-fetching code (used in
- partial clones) will need this function in a subsequent commit, so move
- it to a more central location.
+ submodule.c has functionality that prepares the environment for running
+ a subprocess in a new repo. The lazy-fetching code (used in partial
+ clones) will need this in a subsequent commit, so move it to a more
+ central location.

@@ run-command.c: int run_auto_maintenance(int quiet)
return run_command(&maint);
}
+
-+void prepare_other_repo_env(struct strvec *env_array)
++void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)
+{
+ const char * const *var;
+
+ for (var = local_repo_env; *var; var++) {
-+ if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
++ if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
++ strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
+ strvec_push(env_array, *var);
+ }
++ strvec_pushf(env_array, "%s=%s", GIT_DIR_ENVIRONMENT, new_git_dir);
+}

## run-command.h ##
@@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai
const char *tr2_category, const char *tr2_label);

+/**
-+ * Convenience function which adds all GIT_* environment variables to env_array -+ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a -+ * subprocess, these entries cause the corresponding environment variables to -+ * be unset in the subprocess. See local_repo_env in cache.h for more ++ * Convenience function which prepares env_array for a command to be run in a ++ * new repo. This adds all GIT_* environment variables to env_array with the ++ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding ++ * environment variables to be unset in the subprocess) and adds an environment ++ * variable pointing to new_git_dir. See local_repo_env in cache.h for more
+ * information.
+ */
-+void prepare_other_repo_env(struct strvec *env_array);
++void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);
+
#endif

@@ submodule.c: static void print_submodule_diff_summary(struct repository *r, stru
- const char * const *var;
-
- for (var = local_repo_env; *var; var++) {
-- if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+- if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
+- strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
- strvec_push(out, *var);
- }
-}
@@ submodule.c: static void print_submodule_diff_summary(struct repository *r, stru
void prepare_submodule_repo_env(struct strvec *out)
{
- prepare_submodule_repo_env_no_git_dir(out);
-+ prepare_other_repo_env(out);
- strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
- DEFAULT_GIT_DIR_ENVIRONMENT);
+- strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
+- DEFAULT_GIT_DIR_ENVIRONMENT);
++ prepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);
}

static void prepare_submodule_repo_env_in_gitdir(struct strvec *out) {
- prepare_submodule_repo_env_no_git_dir(out);
-+ prepare_other_repo_env(out);
- strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
+- strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
++ prepare_other_repo_env(out, ".");
}

+ /*
4: 655607d575 ! 5: a6d73662b1 promisor-remote: teach lazy-fetch in any repo
@@ Commit message
prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

+ Besides that, there is some code that uses the wrapper functions
+ like has_promisor_remote(). Those will need to be checked to see if they
+ could support the non-wrapper functions instead (and thus support any
+ repository, not just the_repository).
+

## Makefile ##
@@ promisor-remote.c: static int fetch_objects(const char *remote_name,

child.git_cmd = 1;
child.in = -1;
-+ if (repo != the_repository) {
-+ prepare_other_repo_env(&child.env_array);
-+ strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT,
-+ repo->gitdir);
-+ }
++ if (repo != the_repository)
++ prepare_other_repo_env(&child.env_array, repo->gitdir);
strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop",
"fetch", remote_name, "--no-tags", "--no-write-fetch-head", "--recurse-submodules=no",
-@@ promisor-remote.c: static void promisor_remote_init(struct repository *r)
- xcalloc(sizeof(*r->promisor_remote_config), 1);
- config->promisors_tail = &config->promisors;
-
-- git_config(promisor_remote_config, config);
-+ repo_config(r, promisor_remote_config, config);
-
- if (config->repository_format_partial_clone) {
- struct promisor_remote *o, *previous;
@@ promisor-remote.c: int promisor_remote_get_direct(struct repository *repo,

promisor_remote_init(repo);
--
2.32.0.rc1.229.g3e70b5a671-goog

#63 lore raw

Re: [PATCH v3 3/5] submodule: refrain from filtering GIT_CONFIG_COUNT

In reply to Elijah Newren
Quoted 17 lines · Elijah Newren
Quoted 15 lines
submodule.c +2 -2
index 0b1d9c1dde..f09031e397 100644
@@ -489,7 +489,8 @@ static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
        const char * const *var;
 
        for (var = local_repo_env; *var; var++) {
-               if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+               if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
+                   strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
                        strvec_push(out, *var);
        }
 }
--
Quoted 1 line

2.32.0.rc1.229.g3e70b5a671-goog

Quoted 8 lines

I'm super confused. It appears that prepare_submodule_repo_env_no_git_dir() is filtering out "GIT_CONFIG_PARAMETERS" (CONFIG_DATA_ENVIRONMENT) and "GIT_CONFIG_COUNT" (CONFIG_COUNT_ENVIRONMENT), using all environment variables other than these ones. But the commit message talks about adding an extra environment variable, rather than filtering another out. I must be mis-reading something somewhere, but I'm struggling to figure it out.

I think there might be a double (triple?) negative here:

- we want to pass through the config parameters variable, but not

    other local repo env variables;

- so we _don't_ want the config variable to appear in the "out" strvec, because its presence would cause it to be cleared from the child process environment;

- so we go through the list adding everything _except_ that variable;

- and we match using strcmp(), so a true value means "did not match", so we should add it to the list

Quoted 6 lines

Also, from looking at the other commit messages you reference, it appears GIT_CONFIG_PARAMETERS was just one big environment variable, whereas GIT_CONFIG_COUNT is closely associated with 2*N other environment variables...so shouldn't your loop (and perhaps also git-submodule.sh) also be checking GIT_CONFIG_KEY_\d+ and GIT_CONFIG_VALUE_\d+ ?

We definitely could clean out those GIT_CONFIG_KEY_* values. But the COUNT serves as a master parameter. Anybody who sets COUNT would then also set the individual key/value parameters, too (and even it only sets it to "5", and there is a crufty GIT_CONFIG_KEY_6 in the environment, that is not wrong).

-Peff

#64 lore raw

Re: [PATCH v3 3/5] submodule: refrain from filtering GIT_CONFIG_COUNT

Quoted 30 lines
Quoted 8 lines

I'm super confused. It appears that prepare_submodule_repo_env_no_git_dir() is filtering out "GIT_CONFIG_PARAMETERS" (CONFIG_DATA_ENVIRONMENT) and "GIT_CONFIG_COUNT" (CONFIG_COUNT_ENVIRONMENT), using all environment variables other than these ones. But the commit message talks about adding an extra environment variable, rather than filtering another out. I must be mis-reading something somewhere, but I'm struggling to figure it out.

I think there might be a double (triple?) negative here:

- we want to pass through the config parameters variable, but not

    other local repo env variables;

- so we _don't_ want the config variable to appear in the "out" strvec, because its presence would cause it to be cleared from the child process environment;

- so we go through the list adding everything _except_ that variable;

- and we match using strcmp(), so a true value means "did not match", so we should add it to the list

Quoted 6 lines

Also, from looking at the other commit messages you reference, it appears GIT_CONFIG_PARAMETERS was just one big environment variable, whereas GIT_CONFIG_COUNT is closely associated with 2*N other environment variables...so shouldn't your loop (and perhaps also git-submodule.sh) also be checking GIT_CONFIG_KEY_\d+ and GIT_CONFIG_VALUE_\d+ ?

Quoted 7 lines

We definitely could clean out those GIT_CONFIG_KEY_* values. But the COUNT serves as a master parameter. Anybody who sets COUNT would then also set the individual key/value parameters, too (and even it only sets it to "5", and there is a crufty GIT_CONFIG_KEY_6 in the environment, that is not wrong).

-Peff

As Peff describes, if an envvar is present in the list, it becomes unset. (Perhaps confusingly, if an string of the form "ENVVAR=VALUE" (note the "=") is present in the list, it becomes set to the given value.) So in order to *not* filter out the envvar from the subprocess, we need to filter out the envvar from env_array.

If you can think of a better way to document this, please let me know. One way I thought of that might reduce confusion is for this function to take the struct child_process directly. I don't like taking the whole struct when we're just modifying env_array, but I think that this becomes easier to document (just say that we're unsetting all these envvars from the child process, and in the function body, say that to unset a variable, we need to make it appear without a "=" in env_array).

#69 lore raw
In reply to Elijah Newren

Saw this series mentioned in "What's cooking" and remembered I didn't give an update.

Quoted 15 lines · Elijah Newren
Quoted 5 lines · Jonathan Tan

I think I've addressed all review comments. As for Junio's suggestion about also printing the type in the former patch 4 (now patch 5) [1], I decided to just leave the code as-is and not also print the type.

Quoted 7 lines

The main changes are that patch 1 is somewhat rewritten - we still remove the global variable, but we no longer read the extensions.partialClone config directly from promisor-remote.c. Instead, we store it in struct repository when the format of a repository is being verified, and promisor-remote.c merely reads it from there. Patch 3 is a new patch that updates the environment variable preparation before it is moved in patch 4 (formerly patch 3).

Quoted 6 lines

I've read through all the patches. 2 & 5 look good to me, I had small nitpicks on 1 & 4, and I'm totally lost on patch 3. Patch 3 is just a one-liner and it might be fine, but for some reason I can't figure out the code before or after the patch even after digging around into other commits and other files to try to get my bearings. Hopefully someone else can comment on that one.

I'm happy with Jonathan and Peff's responses on patch 3; as I mentioned above I just didn't understand the original code before Jonathan's changes. (Perhaps some comments could be added to clarify that code area, but again that's clarifying the code that existed before Jonathan's patch so it doesn't need to be part of his series.) So that only leaves my nitpicks on patches 1 & 4; otherwise the series looks good to me.

#70 lore raw

v4 cover First steps towards partial clone submodules

In reply to Jonathan Tan

Quoting from [1]:

Quoted 7 lines

I'm happy with Jonathan and Peff's responses on patch 3; as I mentioned above I just didn't understand the original code before Jonathan's changes. (Perhaps some comments could be added to clarify that code area, but again that's clarifying the code that existed before Jonathan's patch so it doesn't need to be part of his series.) So that only leaves my nitpicks on patches 1 & 4; otherwise the series looks good to me.

I've addressed Elijah's comments on patches 1 and 4.

[1] https://lore.kernel.org/git/CABPp-BFD5=98C0+WnfK=+s7twZ960ORiZzUSP94GD2A4bXJ69Q@mail.gmail.com/

Jonathan Tan (5):
repository: move global r_f_p_c to repo struct
promisor-remote: support per-repository config
submodule: refrain from filtering GIT_CONFIG_COUNT
run-command: refactor subprocess env preparation
promisor-remote: teach lazy-fetch in any repo

 Makefile                      |   1 +
 object-file.c                 |   7 +--
 promisor-remote.c             | 108 ++++++++++++++++++----------------
 promisor-remote.h             |  28 ++++++---
 repository.c                  |  10 ++++
 repository.h                  |   5 ++
 run-command.c                 |  12 ++++
 run-command.h                 |  10 ++++
 setup.c                       |  17 ++++--
 submodule.c                   |  17 +-----
 t/helper/test-partial-clone.c |  43 ++++++++++++++
 t/helper/test-tool.c          |   1 +
 t/helper/test-tool.h          |   1 +
 t/t0410-partial-clone.sh      |  23 ++++++++
 14 files changed, 199 insertions(+), 84 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c

Range-diff against v3:
1: e8e6a95951 ! 1: 0bd009597d repository: move global r_f_p_c to repo struct

    @@ repository.c: int repo_init(struct repository *repo,
      
      	repo_set_hash_algo(repo, format.hash_algo);
      
    ++	/* take ownership of format.partial_clone */
     +	repo->repository_format_partial_clone = format.partial_clone;
     +	format.partial_clone = NULL;
     +
    @@ setup.c: int discover_git_directory(struct strbuf *commondir,
      		return -1;
      	}
      
    ++	/* take ownership of candidate.partial_clone */
     +	the_repository->repository_format_partial_clone =
     +		candidate.partial_clone;
     +	candidate.partial_clone = NULL;
    @@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)
     -		if (startup_info->have_repository)
     +		if (startup_info->have_repository) {
      			repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
    ++			/* take ownership of repo_fmt.partial_clone */
     +			the_repository->repository_format_partial_clone =
     +				repo_fmt.partial_clone;
     +			repo_fmt.partial_clone = NULL;
    @@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)
      	}
      	/*
      	 * Since precompose_string_if_needed() needs to look at
    -@@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)
    - 		setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
    - 	}
    - 
    --
    - 	strbuf_release(&dir);
    - 	strbuf_release(&gitdir);
    - 	clear_repository_format(&repo_fmt);
     @@ setup.c: void check_repository_format(struct repository_format *fmt)
      	check_repository_format_gently(get_git_dir(), fmt, NULL);
      	startup_info->have_repository = 1;
2:  07eb0a0f39 = 2:  8a478b46bf promisor-remote: support per-repository config
3:  004ac92e9b = 3:  78b4108ae1 submodule: refrain from filtering GIT_CONFIG_COUNT
4:  ce0454f442 ! 4:  1778cbf878 run-command: refactor subprocess env preparation

@@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai
+/**
+ * Convenience function which prepares env_array for a command to be run in a
+ * new repo. This adds all GIT_* environment variables to env_array with the
-+ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding -+ * environment variables to be unset in the subprocess) and adds an environment -+ * variable pointing to new_git_dir. See local_repo_env in cache.h for more -+ * information.
++ * exception of GIT_CONFIG_PARAMETERS and GIT_CONFIG_COUNT (which cause the ++ * corresponding environment variables to be unset in the subprocess) and adds ++ * an environment variable pointing to new_git_dir. See local_repo_env in ++ * cache.h for more information.
+ */
+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir); +
5: a3278d61f0 = 5: dbba426b6a promisor-remote: teach lazy-fetch in any repo

Signature · 1 line
2.32.0.272.g935e593368-goog
#71 lore raw

PATCH v4 1/5 repository: move global r_f_p_c to repo struct

Move repository_format_partial_clone, which is currently a global variable, into struct repository. (Full support for per-repository partial clone config will be done in a subsequent commit - this is split into its own commit because of the extent of the changes needed.)

The new repo-specific variable cannot be set in check_repository_format_gently() (as is currently), because that function does not know which repo it is operating on (or even whether the value is important); therefore this responsibility is delegated to the outermost caller that knows. Of all the outermost callers that know (found by looking at all functions that call clear_repository_format()), I looked at those that either read from the main Git directory or write into a struct repository. These callers have been modified accordingly (write to the_repository in the former case and write to the given struct repository in the latter case).

---
 promisor-remote.c | 13 +++----------
 promisor-remote.h |  6 ------
 repository.c      |  4 ++++
 repository.h      |  3 +++
 setup.c           | 17 +++++++++++++----
 5 files changed, 23 insertions(+), 20 deletions(-)
promisor-remote.c +3 -10
index da3f2ca261..d24081dc21 100644
@@ -5,13 +5,6 @@
 #include "transport.h"
 #include "strvec.h"
 
-static char *repository_format_partial_clone;
-
-void set_repository_format_partial_clone(char *partial_clone)
-{
-	repository_format_partial_clone = xstrdup_or_null(partial_clone);
-}
-
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
@@ -145,15 +138,15 @@ static void promisor_remote_init(void)
 
 	git_config(promisor_remote_config, NULL);
 
-	if (repository_format_partial_clone) {
+	if (the_repository->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
 
-		o = promisor_remote_lookup(repository_format_partial_clone,
+		o = promisor_remote_lookup(the_repository->repository_format_partial_clone,
 					   &previous);
 		if (o)
 			promisor_remote_move_to_tail(o, previous);
 		else
-			promisor_remote_new(repository_format_partial_clone);
+			promisor_remote_new(the_repository->repository_format_partial_clone);
 	}
 }
 
promisor-remote.h +0 -6
index c7a14063c5..687210ab87 100644
@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,
 			       const struct object_id *oids,
 			       int oid_nr);
 
-/*
- * This should be used only once from setup.c to set the value we got
- * from the extensions.partialclone config option.
- */
-void set_repository_format_partial_clone(char *partial_clone);
-
 #endif /* PROMISOR_REMOTE_H */
repository.c +4 -0
index 448cd557d4..057a4748a0 100644
@@ -172,6 +172,10 @@ int repo_init(struct repository *repo,
 
 	repo_set_hash_algo(repo, format.hash_algo);
 
+	/* take ownership of format.partial_clone */
+	repo->repository_format_partial_clone = format.partial_clone;
+	format.partial_clone = NULL;
+
 	if (worktree)
 		repo_set_worktree(repo, worktree);
 
repository.h +3 -0
index a45f7520fd..6fb16ed336 100644
@@ -139,6 +139,9 @@ struct repository {
 	/* True if commit-graph has been disabled within this process. */
 	int commit_graph_disabled;
 
+	/* Configurations related to promisor remotes. */
+	char *repository_format_partial_clone;
+
 	/* Configurations */
 
 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
setup.c +13 -4
index ead2f80cd8..eb9367ca5c 100644
@@ -468,8 +468,6 @@ static enum extension_result handle_extension_v0(const char *var,
 			data->precious_objects = git_config_bool(var, value);
 			return EXTENSION_OK;
 		} else if (!strcmp(ext, "partialclone")) {
-			if (!value)
-				return config_error_nonbool(var);
 			data->partial_clone = xstrdup(value);
 			return EXTENSION_OK;
 		} else if (!strcmp(ext, "worktreeconfig")) {
@@ -566,7 +564,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_
 	}
 
 	repository_format_precious_objects = candidate->precious_objects;
-	set_repository_format_partial_clone(candidate->partial_clone);
 	repository_format_worktree_config = candidate->worktree_config;
 	string_list_clear(&candidate->unknown_extensions, 0);
 	string_list_clear(&candidate->v1_only_extensions, 0);
@@ -1197,6 +1194,11 @@ int discover_git_directory(struct strbuf *commondir,
 		return -1;
 	}
 
+	/* take ownership of candidate.partial_clone */
+	the_repository->repository_format_partial_clone =
+		candidate.partial_clone;
+	candidate.partial_clone = NULL;
+
 	clear_repository_format(&candidate);
 	return 0;
 }
@@ -1304,8 +1306,13 @@ const char *setup_git_directory_gently(int *nongit_ok)
 				gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
 			setup_git_env(gitdir);
 		}
-		if (startup_info->have_repository)
+		if (startup_info->have_repository) {
 			repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
+			/* take ownership of repo_fmt.partial_clone */
+			the_repository->repository_format_partial_clone =
+				repo_fmt.partial_clone;
+			repo_fmt.partial_clone = NULL;
+		}
 	}
 	/*
 	 * Since precompose_string_if_needed() needs to look at
@@ -1390,6 +1397,8 @@ void check_repository_format(struct repository_format *fmt)
 	check_repository_format_gently(get_git_dir(), fmt, NULL);
 	startup_info->have_repository = 1;
 	repo_set_hash_algo(the_repository, fmt->hash_algo);
+	the_repository->repository_format_partial_clone =
+		xstrdup_or_null(fmt->partial_clone);
 	clear_repository_format(&repo_fmt);
 }
 
Signature · 1 line
2.32.0.272.g935e593368-goog
#72 lore raw

PATCH v4 2/5 promisor-remote: support per-repository config

In reply to Jonathan Tan

Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository.

The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation.

---
 promisor-remote.c | 98 ++++++++++++++++++++++++++---------------------
 promisor-remote.h | 22 +++++++++--
 repository.c      |  6 +++
 repository.h      |  2 +
 4 files changed, 82 insertions(+), 46 deletions(-)
promisor-remote.c +55 -43
index d24081dc21..1e00e16b0f 100644
@@ -5,6 +5,11 @@
 #include "transport.h"
 #include "strvec.h"
 
+struct promisor_remote_config {
+	struct promisor_remote *promisors;
+	struct promisor_remote **promisors_tail;
+};
+
 static int fetch_objects(const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
@@ -35,10 +40,8 @@ static int fetch_objects(const char *remote_name,
 	return finish_command(&child) ? -1 : 0;
 }
 
-static struct promisor_remote *promisors;
-static struct promisor_remote **promisors_tail = &promisors;
-
-static struct promisor_remote *promisor_remote_new(const char *remote_name)
+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,
+						   const char *remote_name)
 {
 	struct promisor_remote *r;
 
@@ -50,18 +53,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)
 
 	FLEX_ALLOC_STR(r, name, remote_name);
 
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 
 	return r;
 }
 
-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,
+						      const char *remote_name,
 						      struct promisor_remote **previous)
 {
 	struct promisor_remote *r, *p;
 
-	for (p = NULL, r = promisors; r; p = r, r = r->next)
+	for (p = NULL, r = config->promisors; r; p = r, r = r->next)
 		if (!strcmp(r->name, remote_name)) {
 			if (previous)
 				*previous = p;
@@ -71,7 +75,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,
 	return NULL;
 }
 
-static void promisor_remote_move_to_tail(struct promisor_remote *r,
+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,
+					 struct promisor_remote *r,
 					 struct promisor_remote *previous)
 {
 	if (r->next == NULL)
@@ -80,14 +85,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,
 	if (previous)
 		previous->next = r->next;
 	else
-		promisors = r->next ? r->next : r;
+		config->promisors = r->next ? r->next : r;
 	r->next = NULL;
-	*promisors_tail = r;
-	promisors_tail = &r->next;
+	*config->promisors_tail = r;
+	config->promisors_tail = &r->next;
 }
 
 static int promisor_remote_config(const char *var, const char *value, void *data)
 {
+	struct promisor_remote_config *config = data;
 	const char *name;
 	size_t namelen;
 	const char *subkey;
@@ -103,8 +109,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 
 		remote_name = xmemdupz(name, namelen);
 
-		if (!promisor_remote_lookup(remote_name, NULL))
-			promisor_remote_new(remote_name);
+		if (!promisor_remote_lookup(config, remote_name, NULL))
+			promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 		return 0;
@@ -113,9 +119,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 		struct promisor_remote *r;
 		char *remote_name = xmemdupz(name, namelen);
 
-		r = promisor_remote_lookup(remote_name, NULL);
+		r = promisor_remote_lookup(config, remote_name, NULL);
 		if (!r)
-			r = promisor_remote_new(remote_name);
+			r = promisor_remote_new(config, remote_name);
 
 		free(remote_name);
 
@@ -128,59 +134,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data
 	return 0;
 }
 
-static int initialized;
-
-static void promisor_remote_init(void)
+static void promisor_remote_init(struct repository *r)
 {
-	if (initialized)
+	struct promisor_remote_config *config;
+
+	if (r->promisor_remote_config)
 		return;
-	initialized = 1;
+	config = r->promisor_remote_config =
+		xcalloc(sizeof(*r->promisor_remote_config), 1);
+	config->promisors_tail = &config->promisors;
 
-	git_config(promisor_remote_config, NULL);
+	repo_config(r, promisor_remote_config, config);
 
-	if (the_repository->repository_format_partial_clone) {
+	if (r->repository_format_partial_clone) {
 		struct promisor_remote *o, *previous;
 
-		o = promisor_remote_lookup(the_repository->repository_format_partial_clone,
+		o = promisor_remote_lookup(config,
+					   r->repository_format_partial_clone,
 					   &previous);
 		if (o)
-			promisor_remote_move_to_tail(o, previous);
+			promisor_remote_move_to_tail(config, o, previous);
 		else
-			promisor_remote_new(the_repository->repository_format_partial_clone);
+			promisor_remote_new(config, r->repository_format_partial_clone);
 	}
 }
 
-static void promisor_remote_clear(void)
+void promisor_remote_clear(struct promisor_remote_config *config)
 {
-	while (promisors) {
-		struct promisor_remote *r = promisors;
-		promisors = promisors->next;
+	while (config->promisors) {
+		struct promisor_remote *r = config->promisors;
+		config->promisors = config->promisors->next;
 		free(r);
 	}
 
-	promisors_tail = &promisors;
+	config->promisors_tail = &config->promisors;
 }
 
-void promisor_remote_reinit(void)
+void repo_promisor_remote_reinit(struct repository *r)
 {
-	initialized = 0;
-	promisor_remote_clear();
-	promisor_remote_init();
+	promisor_remote_clear(r->promisor_remote_config);
+	FREE_AND_NULL(r->promisor_remote_config);
+	promisor_remote_init(r);
 }
 
-struct promisor_remote *promisor_remote_find(const char *remote_name)
+struct promisor_remote *repo_promisor_remote_find(struct repository *r,
+						  const char *remote_name)
 {
-	promisor_remote_init();
+	promisor_remote_init(r);
 
 	if (!remote_name)
-		return promisors;
+		return r->promisor_remote_config->promisors;
 
-	return promisor_remote_lookup(remote_name, NULL);
+	return promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);
 }
 
-int has_promisor_remote(void)
+int repo_has_promisor_remote(struct repository *r)
 {
-	return !!promisor_remote_find(NULL);
+	return !!repo_promisor_remote_find(r, NULL);
 }
 
 static int remove_fetched_oids(struct repository *repo,
@@ -228,9 +238,11 @@ int promisor_remote_get_direct(struct repository *repo,
 	if (oid_nr == 0)
 		return 0;
 
-	promisor_remote_init();
+	promisor_remote_init(repo);
 
-	for (r = promisors; r; r = r->next) {
+	if (repo != the_repository)
+		BUG("only the_repository is supported for now");
+	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
 		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
promisor-remote.h +19 -3
index 687210ab87..edc45ab0f5 100644
@@ -17,9 +17,25 @@ struct promisor_remote {
 	const char name[FLEX_ARRAY];
 };
 
-void promisor_remote_reinit(void);
-struct promisor_remote *promisor_remote_find(const char *remote_name);
-int has_promisor_remote(void);
+void repo_promisor_remote_reinit(struct repository *r);
+static inline void promisor_remote_reinit(void)
+{
+	repo_promisor_remote_reinit(the_repository);
+}
+
+void promisor_remote_clear(struct promisor_remote_config *config);
+
+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);
+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)
+{
+	return repo_promisor_remote_find(the_repository, remote_name);
+}
+
+int repo_has_promisor_remote(struct repository *r);
+static inline int has_promisor_remote(void)
+{
+	return repo_has_promisor_remote(the_repository);
+}
 
 /*
  * Fetches all requested objects from all promisor remotes, trying them one at
repository.c +6 -0
index 057a4748a0..b2bf44c6fa 100644
@@ -11,6 +11,7 @@
 #include "lockfile.h"
 #include "submodule-config.h"
 #include "sparse-index.h"
+#include "promisor-remote.h"
 
 /* The main repository */
 static struct repository the_repo;
@@ -262,6 +263,11 @@ void repo_clear(struct repository *repo)
 		if (repo->index != &the_index)
 			FREE_AND_NULL(repo->index);
 	}
+
+	if (repo->promisor_remote_config) {
+		promisor_remote_clear(repo->promisor_remote_config);
+		FREE_AND_NULL(repo->promisor_remote_config);
+	}
 }
 
 int repo_read_index(struct repository *repo)
repository.h +2 -0
index 6fb16ed336..3740c93bc0 100644
@@ -10,6 +10,7 @@ struct lock_file;
 struct pathspec;
 struct raw_object_store;
 struct submodule_cache;
+struct promisor_remote_config;
 
 enum untracked_cache_setting {
 	UNTRACKED_CACHE_UNSET = -1,
@@ -141,6 +142,7 @@ struct repository {
 
 	/* Configurations related to promisor remotes. */
 	char *repository_format_partial_clone;
+	struct promisor_remote_config *promisor_remote_config;
 
 	/* Configurations */
 
Signature · 1 line
2.32.0.272.g935e593368-goog
#73 lore raw

PATCH v4 3/5 submodule: refrain from filtering GIT_CONFIG_COUNT

In reply to Jonathan Tan

14111fc492 ("git: submodule honor -c credential.* from command line", 2016-03-01) taught Git to pass through the GIT_CONFIG_PARAMETERS environment variable when invoking a subprocess on behalf of a submodule. But when d8d77153ea ("config: allow specifying config entries via envvar pairs", 2021-01-15) introduced support for GIT_CONFIG_COUNT (and its associated GIT_CONFIG_KEY_? and GIT_CONFIG_VALUE_?), the subprocess mechanism wasn't updated to also pass through these variables.

Since they are conceptually the same (d8d77153ea was written to address a shortcoming of GIT_CONFIG_PARAMETERS), update the submodule subprocess mechanism to also pass through GIT_CONFIG_COUNT.

---
 submodule.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
submodule.c +2 -1
index 0b1d9c1dde..f09031e397 100644
@@ -489,7 +489,8 @@ static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
 	const char * const *var;
 
 	for (var = local_repo_env; *var; var++) {
-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
+		    strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
 			strvec_push(out, *var);
 	}
 }
Signature · 1 line
2.32.0.272.g935e593368-goog
#74 lore raw

PATCH v4 4/5 run-command: refactor subprocess env preparation

In reply to Jonathan Tan

submodule.c has functionality that prepares the environment for running a subprocess in a new repo. The lazy-fetching code (used in partial clones) will need this in a subsequent commit, so move it to a more central location.

---
 run-command.c | 12 ++++++++++++
 run-command.h | 10 ++++++++++
 submodule.c   | 18 ++----------------
 3 files changed, 24 insertions(+), 16 deletions(-)
run-command.c +12 -0
index be6bc128cd..549a94a6a4 100644
@@ -1892,3 +1892,15 @@ int run_auto_maintenance(int quiet)
 
 	return run_command(&maint);
 }
+
+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)
+{
+	const char * const *var;
+
+	for (var = local_repo_env; *var; var++) {
+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
+		    strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
+			strvec_push(env_array, *var);
+	}
+	strvec_pushf(env_array, "%s=%s", GIT_DIR_ENVIRONMENT, new_git_dir);
+}
run-command.h +10 -0
index d08414a92e..22132536a9 100644
@@ -483,4 +483,14 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
 			       task_finished_fn, void *pp_cb,
 			       const char *tr2_category, const char *tr2_label);
 
+/**
+ * Convenience function which prepares env_array for a command to be run in a
+ * new repo. This adds all GIT_* environment variables to env_array with the
+ * exception of GIT_CONFIG_PARAMETERS and GIT_CONFIG_COUNT (which cause the
+ * corresponding environment variables to be unset in the subprocess) and adds
+ * an environment variable pointing to new_git_dir. See local_repo_env in
+ * cache.h for more information.
+ */
+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);
+
 #endif
submodule.c +2 -16
index f09031e397..8e611fe1db 100644
@@ -484,28 +484,14 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *
 	strbuf_release(&sb);
 }
 
-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)
-{
-	const char * const *var;
-
-	for (var = local_repo_env; *var; var++) {
-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&
-		    strcmp(*var, CONFIG_COUNT_ENVIRONMENT))
-			strvec_push(out, *var);
-	}
-}
-
 void prepare_submodule_repo_env(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
-	strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT,
-		     DEFAULT_GIT_DIR_ENVIRONMENT);
+	prepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);
 }
 
 static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
 {
-	prepare_submodule_repo_env_no_git_dir(out);
-	strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT);
+	prepare_other_repo_env(out, ".");
 }
 
 /*
Signature · 1 line
2.32.0.272.g935e593368-goog
#75 lore raw

PATCH v4 5/5 promisor-remote: teach lazy-fetch in any repo

In reply to Jonathan Tan

This is one step towards supporting partial clone submodules.

Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper.

Besides that, there is some code that uses the wrapper functions like has_promisor_remote(). Those will need to be checked to see if they could support the non-wrapper functions instead (and thus support any repository, not just the_repository).

---
 Makefile                      |  1 +
 object-file.c                 |  7 ++----
 promisor-remote.c             |  9 ++++----
 t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c          |  1 +
 t/helper/test-tool.h          |  1 +
 t/t0410-partial-clone.sh      | 23 +++++++++++++++++++
 7 files changed, 76 insertions(+), 9 deletions(-)
 create mode 100644 t/helper/test-partial-clone.c
Makefile +1 -0
index c3565fc0f8..f6653bcd5e 100644
@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o
 TEST_BUILTINS_OBJS += test-online-cpus.o
 TEST_BUILTINS_OBJS += test-parse-options.o
 TEST_BUILTINS_OBJS += test-parse-pathspec-file.o
+TEST_BUILTINS_OBJS += test-partial-clone.o
 TEST_BUILTINS_OBJS += test-path-utils.o
 TEST_BUILTINS_OBJS += test-pcre2-config.o
 TEST_BUILTINS_OBJS += test-pkt-line.o
object-file.c +2 -5
index f233b440b2..ebf273e9e7 100644
@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,
 		}
 
 		/* Check if it is a missing object */
-		if (fetch_if_missing && has_promisor_remote() &&
-		    !already_retried && r == the_repository &&
+		if (fetch_if_missing && repo_has_promisor_remote(r) &&
+		    !already_retried &&
 		    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
 			/*
 			 * TODO Investigate checking promisor_remote_get_direct()
 			 * TODO return value and stopping on error here.
-			 * TODO Pass a repository struct through
-			 * promisor_remote_get_direct(), such that arbitrary
-			 * repositories work.
 			 */
 			promisor_remote_get_direct(r, real, 1);
 			already_retried = 1;
promisor-remote.c +5 -4
index 1e00e16b0f..c088dcbff3 100644
@@ -10,7 +10,8 @@ struct promisor_remote_config {
 	struct promisor_remote **promisors_tail;
 };
 
-static int fetch_objects(const char *remote_name,
+static int fetch_objects(struct repository *repo,
+			 const char *remote_name,
 			 const struct object_id *oids,
 			 int oid_nr)
 {
@@ -20,6 +21,8 @@ static int fetch_objects(const char *remote_name,
 
 	child.git_cmd = 1;
 	child.in = -1;
+	if (repo != the_repository)
+		prepare_other_repo_env(&child.env_array, repo->gitdir);
 	strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop",
 		     "fetch", remote_name, "--no-tags",
 		     "--no-write-fetch-head", "--recurse-submodules=no",
@@ -240,10 +243,8 @@ int promisor_remote_get_direct(struct repository *repo,
 
 	promisor_remote_init(repo);
 
-	if (repo != the_repository)
-		BUG("only the_repository is supported for now");
 	for (r = repo->promisor_remote_config->promisors; r; r = r->next) {
-		if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {
+		if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {
 			if (remaining_nr == 1)
 				continue;
 			remaining_nr = remove_fetched_oids(repo, &remaining_oids,
t/helper/test-partial-clone.c new file +43 -0

43 lines not on this page · raw message

t/helper/test-tool.c +1 -0
index c5bd0c6d4c..b21e8f1519 100644
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
 	{ "online-cpus", cmd__online_cpus },
 	{ "parse-options", cmd__parse_options },
 	{ "parse-pathspec-file", cmd__parse_pathspec_file },
+	{ "partial-clone", cmd__partial_clone },
 	{ "path-utils", cmd__path_utils },
 	{ "pcre2-config", cmd__pcre2_config },
 	{ "pkt-line", cmd__pkt_line },
t/helper/test-tool.h +1 -0
index e8069a3b22..f845ced4b3 100644
@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);
 int cmd__online_cpus(int argc, const char **argv);
 int cmd__parse_options(int argc, const char **argv);
 int cmd__parse_pathspec_file(int argc, const char** argv);
+int cmd__partial_clone(int argc, const char **argv);
 int cmd__path_utils(int argc, const char **argv);
 int cmd__pcre2_config(int argc, const char **argv);
 int cmd__pkt_line(int argc, const char **argv);
t/t0410-partial-clone.sh +23 -0
index 584a039b85..a211a66c67 100755
@@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o
 	git -C repo cherry-pick side1
 '
 
+test_expect_success 'lazy-fetch when accessing object not in the_repository' '
+	rm -rf full partial.git &&
+	test_create_repo full &&
+	test_commit -C full create-a-file file.txt &&
+
+	test_config -C full uploadpack.allowfilter 1 &&
+	test_config -C full uploadpack.allowanysha1inwant 1 &&
+	git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git &&
+	FILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&
+
+	# Sanity check that the file is missing
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	grep "[?]$FILE_HASH" out &&
+
+	git -C full cat-file -s "$FILE_HASH" >expect &&
+	test-tool partial-clone object-info partial.git "$FILE_HASH" >actual &&
+	test_cmp expect actual &&
+
+	# Sanity check that the file is now present
+	git -C partial.git rev-list --objects --missing=print HEAD >out &&
+	! grep "[?]$FILE_HASH" out
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
Signature · 1 line
2.32.0.272.g935e593368-goog
#76 lore raw
In reply to Jonathan Tan
Quoted 11 lines · Jonathan Tan

Quoting from [1]:

Quoted 7 lines

I'm happy with Jonathan and Peff's responses on patch 3; as I mentioned above I just didn't understand the original code before Jonathan's changes. (Perhaps some comments could be added to clarify that code area, but again that's clarifying the code that existed before Jonathan's patch so it doesn't need to be part of his series.) So that only leaves my nitpicks on patches 1 & 4; otherwise the series looks good to me.

Quoted 1 line

I've addressed Elijah's comments on patches 1 and 4.

Yep, patches 1, 2, 4, and 5 are Reviewed-by me. While I looked over Patch 3, I made Peff explain it to me, so he's the one who reviewed that one. ;-)