Skip to contentSkip to conversation

More add_submodule_odb() cleanup in merge code

PATCH Superseded Jonathan Tan 3 participants 14 messages Sep 8, 2021 – Sep 14, 2021

lore
Share & tools
lore
#0 lore raw

cover More add_submodule_odb() cleanup in merge code

(CC-ing Elijah in case he has insight into the merge-ort part.)

While continuing the effort into removing add_submodule_odb() (as part of the submodule partial clone effort) I came across this part of the merge code that specifies the repository being operated on in two ways:
one as a struct repository pointer and the other as a path. This patch set unifies the two.

I normally would not send refactoring patches unless I also have a feature patch that uses the results of said refactoring, but in this case, I think that these patches are worth having since they clarify a potentially unclear part of the API.

Note that these patches mean that the merging code no longer supports submodules that have their .git dirs in the worktree, but from what I can tell, this seems to be the direction we're going in (repo_submodule_init() does not support such submodules).

Patch 3 is included to show how I'm verifying some things. Including something like that in the master branch would probably require conditional compilation (to exclude the additional field in struct object used for checking, among other things), so I'm just including it here for informational purposes.

All these patches work under GIT_TEST_MERGE_ALGORITHM=recursive and GIT_TEST_MERGE_ALGORITHM=ort (and when that envvar is unset, for good measure).

Jonathan Tan (3):
t6437: run absorbgitdirs on repos
revision: remove "submodule" from opt struct DO NOT SUBMIT commit-reach,revision: verify non-mixing

 alloc.c                    |  2 +
 commit-reach.c             | 60 +++++++++++++++++-------
 merge-ort.c                | 55 +++++++++++++++-------
 merge-recursive.c          | 51 +++++++++++++-------
 object.h                   |  1 +
 revision.c                 | 96 ++++++++++++++++++++++----------------
 revision.h                 |  1 -
 t/t6437-submodule-merge.sh |  9 ++--
 8 files changed, 179 insertions(+), 96 deletions(-)
Signature · 1 line
2.33.0.309.g3052b89438-goog
#1 lore raw

PATCH 1/3 t6437: run absorbgitdirs on repos

The submodule merge code is being transitioned away from add_submodule_odb() to repo_submodule_init(), and the latter does not support submodules that have their .git directories in the worktree (instead of in .git/modules). Migrate the test code by calling absorbgitdirs wherever necessary to place the .git directories of submodules in .git/modules of the superproject.

---
 t/t6437-submodule-merge.sh | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
t/t6437-submodule-merge.sh +6 -3
index e5e89c2045..8efce86b42 100755
@@ -27,7 +27,8 @@ test_expect_success setup '
 	 git add file &&
 	 test_tick &&
 	 git commit -m sub-root) &&
-	git add sub &&
+	git submodule add ./sub sub &&
+	git submodule absorbgitdirs &&
 	test_tick &&
 	git commit -m root &&
 
@@ -82,7 +83,8 @@ test_expect_success 'setup for merge search' '
 	 git branch sub-a) &&
 	git commit --allow-empty -m init &&
 	git branch init &&
-	git add sub &&
+	git submodule add ./sub sub &&
+	git submodule absorbgitdirs &&
 	git commit -m "a" &&
 	git branch a &&
 
@@ -112,7 +114,8 @@ test_expect_success 'setup for merge search' '
 	git checkout -b g init &&
 	(cd sub &&
 	 git checkout -b sub-g sub-c) &&
-	git add sub &&
+	git submodule add ./sub sub &&
+	git submodule absorbgitdirs &&
 	git commit -a -m "g")
 '
 
Signature · 1 line
2.33.0.309.g3052b89438-goog
#5 lore raw

PATCH 2/3 revision: remove "submodule" from opt struct

In reply to Jonathan Tan

Clean up a TODO in revision.h by removing the "submodule" field from struct setup_revision_opt. This field is only used to specify the ref store to use, so use rev_info->repo to determine the ref store instead.

The only users of this field are merge-ort.c and merge-recursive.c. However, both these files specify the superproject as rev_info->repo and the submodule as setup_revision_opt->submodule. In order to be able to pass the submodule as rev_info->repo, all commits must be parsed with the submodule explicitly specified; this patch does that as well. (An incremental solution in which only some commits are parsed with explicit submodule will not work, because if the same commit is parsed twice in different repositories, there will be 2 heap-allocated object structs corresponding to that commit, and any flag set by the revision walking mechanism on one of them will not be reflected onto the other.)

---
 merge-ort.c       | 55 +++++++++++++++++++++++++++++++----------------
 merge-recursive.c | 51 ++++++++++++++++++++++++++++---------------
 revision.c        | 16 +++++---------
 revision.h        |  1 -
 4 files changed, 76 insertions(+), 47 deletions(-)
merge-ort.c +37 -18
index 515dc39b7f..419ac0f396 100644
@@ -32,6 +32,7 @@
 #include "promisor-remote.h"
 #include "revision.h"
 #include "strmap.h"
+#include "submodule-config.h"
 #include "submodule.h"
 #include "tree.h"
 #include "unpack-trees.h"
@@ -1511,7 +1512,6 @@ static int find_first_merges(struct repository *repo,
 	xsnprintf(merged_revision, sizeof(merged_revision), "^%s",
 		  oid_to_hex(&a->object.oid));
 	repo_init_revisions(repo, &revs, NULL);
-	rev_opts.submodule = path;
 	/* FIXME: can't handle linked worktrees in submodules yet */
 	revs.single_worktree = path != NULL;
 	setup_revisions(ARRAY_SIZE(rev_args)-1, rev_args, &revs, &rev_opts);
@@ -1521,7 +1521,7 @@ static int find_first_merges(struct repository *repo,
 		die("revision walk setup failed");
 	while ((commit = get_revision(&revs)) != NULL) {
 		struct object *o = &(commit->object);
-		if (in_merge_bases(b, commit))
+		if (repo_in_merge_bases(repo, b, commit))
 			add_object_array(o, NULL, &merges);
 	}
 	reset_revision_walk();
@@ -1536,7 +1536,7 @@ static int find_first_merges(struct repository *repo,
 		contains_another = 0;
 		for (j = 0; j < merges.nr; j++) {
 			struct commit *m2 = (struct commit *) merges.objects[j].item;
-			if (i != j && in_merge_bases(m2, m1)) {
+			if (i != j && repo_in_merge_bases(repo, m2, m1)) {
 				contains_another = 1;
 				break;
 			}
@@ -1557,10 +1557,13 @@ static int merge_submodule(struct merge_options *opt,
 			   const struct object_id *b,
 			   struct object_id *result)
 {
+	const struct submodule *submodule;
+	struct repository subrepo;
+	struct strbuf sb = STRBUF_INIT;
+	int ret = 0;
 	struct commit *commit_o, *commit_a, *commit_b;
 	int parent_count;
 	struct object_array merges;
-	struct strbuf sb = STRBUF_INIT;
 
 	int i;
 	int search = !opt->priv->call_depth;
@@ -1576,6 +1579,10 @@ static int merge_submodule(struct merge_options *opt,
 	if (is_null_oid(b))
 		return 0;
 
+	/*
+	 * NEEDSWORK: Remove this when all submodule object accesses are
+	 * through explicitly specified repositores.
+	 */
 	if (add_submodule_odb(path)) {
 		path_msg(opt, path, 0,
 			 _("Failed to merge submodule %s (not checked out)"),
@@ -1583,39 +1590,49 @@ static int merge_submodule(struct merge_options *opt,
 		return 0;
 	}
 
-	if (!(commit_o = lookup_commit_reference(opt->repo, o)) ||
-	    !(commit_a = lookup_commit_reference(opt->repo, a)) ||
-	    !(commit_b = lookup_commit_reference(opt->repo, b))) {
+	submodule = submodule_from_path(opt->repo, null_oid(), path);
+	if (repo_submodule_init(&subrepo, opt->repo, submodule)) {
+		path_msg(opt, path, 0,
+				_("Failed to merge submodule %s (not checked out)"),
+				path);
+		return 0;
+	}
+
+	if (!(commit_o = lookup_commit_reference(&subrepo, o)) ||
+	    !(commit_a = lookup_commit_reference(&subrepo, a)) ||
+	    !(commit_b = lookup_commit_reference(&subrepo, b))) {
 		path_msg(opt, path, 0,
 			 _("Failed to merge submodule %s (commits not present)"),
 			 path);
-		return 0;
+		goto cleanup;
 	}
 
 	/* check whether both changes are forward */
-	if (!in_merge_bases(commit_o, commit_a) ||
-	    !in_merge_bases(commit_o, commit_b)) {
+	if (!repo_in_merge_bases(&subrepo, commit_o, commit_a) ||
+	    !repo_in_merge_bases(&subrepo, commit_o, commit_b)) {
 		path_msg(opt, path, 0,
 			 _("Failed to merge submodule %s "
 			   "(commits don't follow merge-base)"),
 			 path);
-		return 0;
+		goto cleanup;
 	}
 
 	/* Case #1: a is contained in b or vice versa */
-	if (in_merge_bases(commit_a, commit_b)) {
+	if (repo_in_merge_bases(&subrepo, commit_a, commit_b)) {
 		oidcpy(result, b);
 		path_msg(opt, path, 1,
 			 _("Note: Fast-forwarding submodule %s to %s"),
 			 path, oid_to_hex(b));
-		return 1;
+		ret = 1;
+		goto cleanup;
 	}
-	if (in_merge_bases(commit_b, commit_a)) {
+	if (repo_in_merge_bases(&subrepo, commit_b, commit_a)) {
 		oidcpy(result, a);
 		path_msg(opt, path, 1,
 			 _("Note: Fast-forwarding submodule %s to %s"),
 			 path, oid_to_hex(a));
-		return 1;
+		ret = 1;
+		goto cleanup;
 	}
 
 	/*
@@ -1627,10 +1644,10 @@ static int merge_submodule(struct merge_options *opt,
 
 	/* Skip the search if makes no sense to the calling context.  */
 	if (!search)
-		return 0;
+		goto cleanup;
 
 	/* find commit which merges them */
-	parent_count = find_first_merges(opt->repo, path, commit_a, commit_b,
+	parent_count = find_first_merges(&subrepo, path, commit_a, commit_b,
 					 &merges);
 	switch (parent_count) {
 	case 0:
@@ -1664,7 +1681,9 @@ static int merge_submodule(struct merge_options *opt,
 	}
 
 	object_array_clear(&merges);
-	return 0;
+cleanup:
+	repo_clear(&subrepo);
+	return ret;
 }
 
 static void initialize_attr_index(struct merge_options *opt)
merge-recursive.c +34 -17

127 lines not on this page · raw message

revision.c +5 -11
index 0dabb5a0bc..7e7c41fedd 100644
@@ -2563,8 +2563,7 @@ static int for_each_good_bisect_ref(struct ref_store *refs, each_ref_fn fn, void
 	return for_each_bisect_ref(refs, fn, cb_data, term_good);
 }
 
-static int handle_revision_pseudo_opt(const char *submodule,
-				      struct rev_info *revs,
+static int handle_revision_pseudo_opt(struct rev_info *revs,
 				      const char **argv, int *flags)
 {
 	const char *arg = argv[0];
@@ -2572,7 +2571,7 @@ static int handle_revision_pseudo_opt(const char *submodule,
 	struct ref_store *refs;
 	int argcount;
 
-	if (submodule) {
+	if (revs->repo != the_repository) {
 		/*
 		 * We need some something like get_submodule_worktrees()
 		 * before we can go through all worktrees of a submodule,
@@ -2581,9 +2580,8 @@ static int handle_revision_pseudo_opt(const char *submodule,
 		 */
 		if (!revs->single_worktree)
 			BUG("--single-worktree cannot be used together with submodule");
-		refs = get_submodule_ref_store(submodule);
-	} else
-		refs = get_main_ref_store(revs->repo);
+	}
+	refs = get_main_ref_store(revs->repo);
 
 	/*
 	 * NOTE!
@@ -2707,12 +2705,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s
 {
 	int i, flags, left, seen_dashdash, revarg_opt;
 	struct strvec prune_data = STRVEC_INIT;
-	const char *submodule = NULL;
 	int seen_end_of_options = 0;
 
-	if (opt)
-		submodule = opt->submodule;
-
 	/* First, search for "--" */
 	if (opt && opt->assume_dashdash) {
 		seen_dashdash = 1;
@@ -2741,7 +2735,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s
 		if (!seen_end_of_options && *arg == '-') {
 			int opts;
 
-			opts = handle_revision_pseudo_opt(submodule,
+			opts = handle_revision_pseudo_opt(
 						revs, argv + i,
 						&flags);
 			if (opts > 0) {
revision.h +0 -1
index 0c65a760ee..5578bb4720 100644
@@ -336,7 +336,6 @@ extern volatile show_early_output_fn_t show_early_output;
 struct setup_revision_opt {
 	const char *def;
 	void (*tweak)(struct rev_info *, struct setup_revision_opt *);
-	const char *submodule;	/* TODO: drop this and use rev_info->repo */
 	unsigned int	assume_dashdash:1,
 			allow_exclude_promisor_objects:1;
 	unsigned revarg_opt;
Signature · 1 line
2.33.0.309.g3052b89438-goog
#6 lore raw

PATCH 3/3 DO NOT SUBMIT commit-reach,revision: verify non-mixing

In reply to Jonathan Tan

This patch is provided to show reviewers how I'm verifying that there is no mixing of object structs in the revision walking mechanisms. I've intercepted any creation of commit nodes in alloc.c (whether through alloc_commit_node() directly or through alloc_object_node()) to add the repository in which that node is allocated, and checked all additions of commits to data structures (as far as I can tell) in commit-reach.c and revision.c to make sure that the repository in the added objects is what we expect. This ensures that any flags set are set on the correct nodes (for example, it won't be set on an object allocated in the_repository and then checked from an object allocated in a submodule).

I'll leave out this patch in subsequent revisions of this patch set.

---
 alloc.c        |  2 ++
 commit-reach.c | 60 +++++++++++++++++++++++++------------
 object.h       |  1 +
 revision.c     | 80 ++++++++++++++++++++++++++++++++------------------
 4 files changed, 97 insertions(+), 46 deletions(-)
alloc.c +2 -0
index 957a0af362..98b8c06834 100644
@@ -96,6 +96,7 @@ void *alloc_object_node(struct repository *r)
 {
 	struct object *obj = alloc_node(r->parsed_objects->object_state, sizeof(union any_object));
 	obj->type = OBJ_NONE;
+	obj->repo = r;
 	return obj;
 }
 
@@ -120,6 +121,7 @@ void *alloc_commit_node(struct repository *r)
 {
 	struct commit *c = alloc_node(r->parsed_objects->commit_state, sizeof(struct commit));
 	init_commit_node(c);
+	c->object.repo = r;
 	return c;
 }
 
commit-reach.c +42 -18

171 lines not on this page · raw message

object.h +1 -0
index 3b38c9cc98..22e9d2a52a 100644
@@ -90,6 +90,7 @@ struct object {
 	unsigned type : TYPE_BITS;
 	unsigned flags : FLAG_BITS;
 	struct object_id oid;
+	struct repository *repo;
 };
 
 const char *type_name(unsigned int type);
revision.c +52 -28

235 lines not on this page · raw message

Signature · 1 line
2.33.0.309.g3052b89438-goog
#2 lore raw

Re: [PATCH 1/3] t6437: run absorbgitdirs on repos

In reply to Jonathan Tan
Quoted 6 lines

The submodule merge code is being transitioned away from add_submodule_odb() to repo_submodule_init(), and the latter does not support submodules that have their .git directories in the worktree (instead of in .git/modules). Migrate the test code by calling absorbgitdirs wherever necessary to place the .git directories of submodules in .git/modules of the superproject.

Upon further work, there seems to be quite a lot of test code to migrate, and it might be easier instead to add support for unabsorbed submodules to repo_submodule_init() (say, following what open_submodule() in submodule.c does). Feel free to comment on this or other aspects of the patch set, and in the meantime, I'll investigate that.

#3 lore raw

Re: [PATCH 1/3] t6437: run absorbgitdirs on repos

In reply to Jonathan Tan
Quoted 6 lines · Jonathan Tan

The submodule merge code is being transitioned away from add_submodule_odb() to repo_submodule_init(), and the latter does not support submodules that have their .git directories in the worktree (instead of in .git/modules). Migrate the test code by calling absorbgitdirs wherever necessary to place the .git directories of submodules in .git/modules of the superproject.

It is wonderful that we can work with the "absorbed" form of the submodule repositories, and having all the submodules in "absorbed" form may be ideal in the longer term, but I fail to convince myself that this particular change to test script is a good idea.

After all, if you are not cloning from somebody else but creating a new submodule in a superproject you have (whether that was locally created or cloned from elsewhere), wouldn't it more natural to have a submodule that is not "absorbed" there? Is it easy to create an "absorbed" submodule locally?

The change to tests in this patch may make things work with the repo_submodule_init() that is incapable of work with submodules with in-worktree .git repository. In theory, the users can do the same, i.e. have an extra "git submodule absorbgitdirs" call, when adding a new submodule, to make things work with repo_submodule_init(), too.

But that sounds like a workaround, not a feature.

Quite honestly, if repo_submodule_init() can only work with the "absorbed" form, isn't it simply a bug?

Two independent (i.e. we can do either or both) things may improve the situation:

- allow repo_submodule_init() to work on both forms (in place---no cheating by calling "absorb" behind user's back).

- teach "git submodule add" to transition an old-style submodule into the "absorbed" form (either with an option, or do so by default with an escape hatch to disable).

The latter indirectly attacks the "repo_submodule_init() can only work with absorbed form" issue by making it harder to create a non-absorbed submodule to begin with.

Hmm?

#7 lore raw
In reply to Jonathan Tan
Quoted 2 lines · Jonathan Tan

(CC-ing Elijah in case he has insight into the merge-ort part.)

All the submodule merging related functions were lifted from merge-recursive.c and minimally modified to fit the new structure. The only substantive change I made was to fix the merge result for virtual merge bases, but that's like one or two lines of code. In particular, everything relating to submodule objects was totally untouched...and I think that's reflected in the fact that your PATCH 2 basically is the same patch twice, once for merge-recursive and once for merge-ort.

I read over PATCH 2 and I didn't find anything that looked problematic, but I'm not up-to-speed on the add_submodule_odb and repo handling bits of the codebase so I'm not sure I would catch anything. But I am encouraged by the fact that it looks like you did the same stuff to merge-recursive and merge-ort; I'd be worried you missed something if that weren't the case.

As a sidenote, though...

This does remind me that I noticed that the following functions from object-store.h do not take an explicit repository:

write_object_file()
hash_object_file()
hash_object_file_literally()
force_object_loose()

I have a patch sitting around somewhere (possibly only still referenced in my 'temp' branch) to make repo_*() variants of the above functions, and turn the above into simple wrappers of the repo_*() variants which just pass the_repository (much like someone else did with read_object_file() and repo_read_object_file()). It also updates merge-ort to use the new repo_*() functions. However, I ended up excluding it from my merge-ort submissions since it wasn't necessary. Would this be of interest in your submodule work, though? I guess it'd only matter if we started doing real merges of submodules as part of a parent repo merge. (As opposed to the fast-forward-only merges that merge-recursive/merge-ort do right now for submodules.)

Quoted 44 lines

While continuing the effort into removing add_submodule_odb() (as part of the submodule partial clone effort) I came across this part of the merge code that specifies the repository being operated on in two ways:
one as a struct repository pointer and the other as a path. This patch set unifies the two.

I normally would not send refactoring patches unless I also have a feature patch that uses the results of said refactoring, but in this case, I think that these patches are worth having since they clarify a potentially unclear part of the API.

Note that these patches mean that the merging code no longer supports submodules that have their .git dirs in the worktree, but from what I can tell, this seems to be the direction we're going in (repo_submodule_init() does not support such submodules).

Patch 3 is included to show how I'm verifying some things. Including something like that in the master branch would probably require conditional compilation (to exclude the additional field in struct object used for checking, among other things), so I'm just including it here for informational purposes.

All these patches work under GIT_TEST_MERGE_ALGORITHM=recursive and GIT_TEST_MERGE_ALGORITHM=ort (and when that envvar is unset, for good measure).

Jonathan Tan (3):
t6437: run absorbgitdirs on repos
revision: remove "submodule" from opt struct DO NOT SUBMIT commit-reach,revision: verify non-mixing

 alloc.c                    |  2 +
 commit-reach.c             | 60 +++++++++++++++++-------
 merge-ort.c                | 55 +++++++++++++++-------
 merge-recursive.c          | 51 +++++++++++++-------
 object.h                   |  1 +
 revision.c                 | 96 ++++++++++++++++++++++----------------
 revision.h                 |  1 -
 t/t6437-submodule-merge.sh |  9 ++--
 8 files changed, 179 insertions(+), 96 deletions(-)

--
2.33.0.309.g3052b89438-goog

#4 lore raw

Re: [PATCH 1/3] t6437: run absorbgitdirs on repos

In reply to Junio C Hamano
Quoted 9 lines

Quite honestly, if repo_submodule_init() can only work with the "absorbed" form, isn't it simply a bug?

Two independent (i.e. we can do either or both) things may improve the situation:

- allow repo_submodule_init() to work on both forms (in place---no cheating by calling "absorb" behind user's back).

Quoted 7 lines

- teach "git submodule add" to transition an old-style submodule into the "absorbed" form (either with an option, or do so by default with an escape hatch to disable).

The latter indirectly attacks the "repo_submodule_init() can only work with absorbed form" issue by making it harder to create a non-absorbed submodule to begin with.

Thanks for taking a look.

I took a further look and it doesn't look too difficult to teach repo_submodule_init() to work on both forms (without calling "absorbgitdirs") - I'll send an updated patch set soon. As for "git submodule add" transitioning it into the absorbed form, I'll leave that for someone else to do - although if repo_submodule_init() supports the unabsorbed form now, that is not so urgent to do, I think.

#8 lore raw
In reply to Elijah Newren
Quoted 13 lines · Jonathan Tan
Quoted 2 lines

(CC-ing Elijah in case he has insight into the merge-ort part.)

All the submodule merging related functions were lifted from merge-recursive.c and minimally modified to fit the new structure. The only substantive change I made was to fix the merge result for virtual merge bases, but that's like one or two lines of code. In particular, everything relating to submodule objects was totally untouched...and I think that's reflected in the fact that your PATCH 2 basically is the same patch twice, once for merge-recursive and once for merge-ort.

Quoted 6 lines

I read over PATCH 2 and I didn't find anything that looked problematic, but I'm not up-to-speed on the add_submodule_odb and repo handling bits of the codebase so I'm not sure I would catch anything. But I am encouraged by the fact that it looks like you did the same stuff to merge-recursive and merge-ort; I'd be worried you missed something if that weren't the case.

Thanks for taking a look.

Quoted 10 lines

As a sidenote, though...

This does remind me that I noticed that the following functions from object-store.h do not take an explicit repository:

write_object_file()
hash_object_file()
hash_object_file_literally()
force_object_loose()

Quoted 11 lines

I have a patch sitting around somewhere (possibly only still referenced in my 'temp' branch) to make repo_*() variants of the above functions, and turn the above into simple wrappers of the repo_*() variants which just pass the_repository (much like someone else did with read_object_file() and repo_read_object_file()). It also updates merge-ort to use the new repo_*() functions. However, I ended up excluding it from my merge-ort submissions since it wasn't necessary. Would this be of interest in your submodule work, though? I guess it'd only matter if we started doing real merges of submodules as part of a parent repo merge. (As opposed to the fast-forward-only merges that merge-recursive/merge-ort do right now for submodules.)

I think that these functions would be useful if we were to write into submodules (for example, in a real merge of submodules, as you said). Right now my submodule work is just making it support partial clones, so I'm not planning to introduce any new submodule writes. (Well, besides the fact that a lazy-fetch in a submodule would end up with new objects, but that's already handled because we do the fetch in a separate process in the submodule's gitdir.)

#9 lore raw

v2 cover More add_submodule_odb() cleanup in merge code

In reply to Jonathan Tan

This is on jt/grep-wo-submodule-odb-as-alternate (unlike version 1) because that branch introduces another usage of repo_submodule_init(), and I need to update it.

The main change from version 1 is that I have introduced a patch (patch
2) that teaches repo_submodule_init() to support submodules with unabsorbed gitdirs, which means that I no longer need the original patch 1 that updated t6437 to absorb all git dirs.

Once again, all these patches work under GIT_TEST_MERGE_ALGORITHM=recursive and GIT_TEST_MERGE_ALGORITHM=ort (and when that envvar is unset, for good measure).

I have also tested it with the "DO NOT SUBMIT" patch 3 from version 1.

Jonathan Tan (3):
submodule: remove unnecessary unabsorbed fallback
repository: support unabsorbed in repo_submodule_init
revision: remove "submodule" from opt struct

 builtin/grep.c                               |  5 +-
 builtin/ls-files.c                           |  4 +-
 builtin/submodule--helper.c                  |  7 +--
 merge-ort.c                                  | 53 +++++++++++++-------
 merge-recursive.c                            | 49 +++++++++++-------
 repository.c                                 | 21 ++++----
 repository.h                                 | 15 +++---
 revision.c                                   | 16 ++----
 revision.h                                   |  1 -
 submodule.c                                  | 24 ++-------
 t/helper/test-submodule-nested-repo-config.c |  4 +-
 11 files changed, 102 insertions(+), 97 deletions(-)
Signature · 1 line
2.33.0.309.g3052b89438-goog
#10 lore raw

PATCH v2 1/3 submodule: remove unnecessary unabsorbed fallback

In get_submodule_repo_for(), there is a fallback code path for the case in which a submodule has an unabsorbed gitdir. (See the documentation for "git submodule absorbgitdirs" for more information about absorbed and unabsorbed gitdirs.) However, this code path is unnecessary, because such submodules are already handled: when the fetch_task is created in fetch_task_create(), it will create its own struct submodule with a path and name, and repo_submodule_init() can handle such a struct.

This fallback was introduced in 26f80ccfc1 ("submodule: migrate get_next_submodule to use repository structs", 2018-12-05). It was unnecessary even then, but perhaps it escaped notice because its parent commit d5498e0871 ("repository: repo_submodule_init to take a submodule struct", 2018-12-05) was the one that taught repo_submodule_init() to handle such created structs. Before, it took a path and always checked .gitmodules, so it truly would have failed if there were no entry in .gitmodules.

(Note to reviewers: in 26f80ccfc1, the "own struct submodule" I mentioned is in get_next_submodule(), not fetch_task_create().)

---
 submodule.c | 15 ++-------------
 1 file changed, 2 insertions(+), 13 deletions(-)
submodule.c +2 -13
index 8de1aecaeb..3af3da5b5e 100644
@@ -1409,19 +1409,8 @@ static struct repository *get_submodule_repo_for(struct repository *r,
 	struct repository *ret = xmalloc(sizeof(*ret));
 
 	if (repo_submodule_init(ret, r, sub)) {
-		/*
-		 * No entry in .gitmodules? Technically not a submodule,
-		 * but historically we supported repositories that happen to be
-		 * in-place where a gitlink is. Keep supporting them.
-		 */
-		struct strbuf gitdir = STRBUF_INIT;
-		strbuf_repo_worktree_path(&gitdir, r, "%s/.git", sub->path);
-		if (repo_init(ret, gitdir.buf, NULL)) {
-			strbuf_release(&gitdir);
-			free(ret);
-			return NULL;
-		}
-		strbuf_release(&gitdir);
+		free(ret);
+		return NULL;
 	}
 
 	return ret;
Signature · 1 line
2.33.0.309.g3052b89438-goog
#11 lore raw

PATCH v2 2/3 repository: support unabsorbed in repo_submodule_init

In reply to Jonathan Tan

In preparation for a subsequent commit that migrates code using add_submodule_odb() to repo_submodule_init(), teach repo_submodule_init() to support submodules with unabsorbed gitdirs. (See the documentation for "git submodule absorbgitdirs" for more information about absorbed and unabsorbed gitdirs.)

---
 builtin/grep.c                               |  5 +----
 builtin/ls-files.c                           |  4 +---
 builtin/submodule--helper.c                  |  7 +------
 repository.c                                 | 21 +++++++++++---------
 repository.h                                 | 15 ++++++++------
 submodule.c                                  |  9 +++------
 t/helper/test-submodule-nested-repo-config.c |  4 +---
 7 files changed, 28 insertions(+), 37 deletions(-)
builtin/grep.c +1 -4
index 51278b01fa..8af5249a7b 100644
@@ -433,17 +433,14 @@ static int grep_submodule(struct grep_opt *opt,
 {
 	struct repository *subrepo;
 	struct repository *superproject = opt->repo;
-	const struct submodule *sub;
 	struct grep_opt subopt;
 	int hit = 0;
 
-	sub = submodule_from_path(superproject, null_oid(), path);
-
 	if (!is_submodule_active(superproject, path))
 		return 0;
 
 	subrepo = xmalloc(sizeof(*subrepo));
-	if (repo_submodule_init(subrepo, superproject, sub)) {
+	if (repo_submodule_init(subrepo, superproject, path, null_oid())) {
 		free(subrepo);
 		return 0;
 	}
builtin/ls-files.c +1 -3
index 29a26ad8ae..ec19bf54b2 100644
@@ -209,10 +209,8 @@ static void show_submodule(struct repository *superproject,
 			   struct dir_struct *dir, const char *path)
 {
 	struct repository subrepo;
-	const struct submodule *sub = submodule_from_path(superproject,
-							  null_oid(), path);
 
-	if (repo_submodule_init(&subrepo, superproject, sub))
+	if (repo_submodule_init(&subrepo, superproject, path, null_oid()))
 		return;
 
 	if (repo_read_index(&subrepo) < 0)
builtin/submodule--helper.c +1 -6
index ef2776a9e4..6718f202db 100644
@@ -2540,7 +2540,6 @@ static int push_check(int argc, const char **argv, const char *prefix)
 
 static int ensure_core_worktree(int argc, const char **argv, const char *prefix)
 {
-	const struct submodule *sub;
 	const char *path;
 	const char *cw;
 	struct repository subrepo;
@@ -2550,11 +2549,7 @@ static int ensure_core_worktree(int argc, const char **argv, const char *prefix)
 
 	path = argv[1];
 
-	sub = submodule_from_path(the_repository, null_oid(), path);
-	if (!sub)
-		BUG("We could get the submodule handle before?");
-
-	if (repo_submodule_init(&subrepo, the_repository, sub))
+	if (repo_submodule_init(&subrepo, the_repository, path, null_oid()))
 		die(_("could not get a repository handle for submodule '%s'"), path);
 
 	if (!repo_config_get_string_tmp(&subrepo, "core.worktree", &cw)) {
repository.c +12 -9
index b2bf44c6fa..e4a1afb0ac 100644
@@ -190,19 +190,15 @@ int repo_init(struct repository *repo,
 
 int repo_submodule_init(struct repository *subrepo,
 			struct repository *superproject,
-			const struct submodule *sub)
+			const char *path,
+			const struct object_id *treeish_name)
 {
 	struct strbuf gitdir = STRBUF_INIT;
 	struct strbuf worktree = STRBUF_INIT;
 	int ret = 0;
 
-	if (!sub) {
-		ret = -1;
-		goto out;
-	}
-
-	strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", sub->path);
-	strbuf_repo_worktree_path(&worktree, superproject, "%s", sub->path);
+	strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", path);
+	strbuf_repo_worktree_path(&worktree, superproject, "%s", path);
 
 	if (repo_init(subrepo, gitdir.buf, worktree.buf)) {
 		/*
@@ -212,6 +208,13 @@ int repo_submodule_init(struct repository *subrepo,
 		 * in the superproject's 'modules' directory.  In this case the
 		 * submodule would not have a worktree.
 		 */
+		const struct submodule *sub =
+			submodule_from_path(superproject, treeish_name, path);
+		if (!sub) {
+			ret = -1;
+			goto out;
+		}
+
 		strbuf_reset(&gitdir);
 		strbuf_repo_git_path(&gitdir, superproject,
 				     "modules/%s", sub->name);
@@ -225,7 +228,7 @@ int repo_submodule_init(struct repository *subrepo,
 	subrepo->submodule_prefix = xstrfmt("%s%s/",
 					    superproject->submodule_prefix ?
 					    superproject->submodule_prefix :
-					    "", sub->path);
+					    "", path);
 
 out:
 	strbuf_release(&gitdir);
repository.h +9 -6
index 3740c93bc0..c24e177c7e 100644
@@ -172,15 +172,18 @@ void initialize_the_repository(void);
 int repo_init(struct repository *r, const char *gitdir, const char *worktree);
 
 /*
- * Initialize the repository 'subrepo' as the submodule given by the
- * struct submodule 'sub' in parent repository 'superproject'.
- * Return 0 upon success and a non-zero value upon failure, which may happen
- * if the submodule is not found, or 'sub' is NULL.
+ * Initialize the repository 'subrepo' as the submodule at the given path. If
+ * the submodule's gitdir cannot be found at <path>/.git, this function calls
+ * submodule_from_path() to try to find it. treeish_name is only used if
+ * submodule_from_path() needs to be called; see its documentation for more
+ * information.
+ * Return 0 upon success and a non-zero value upon failure.
  */
-struct submodule;
+struct object_id;
 int repo_submodule_init(struct repository *subrepo,
 			struct repository *superproject,
-			const struct submodule *sub);
+			const char *path,
+			const struct object_id *treeish_name);
 void repo_clear(struct repository *repo);
 
 /*
submodule.c +3 -6
index 3af3da5b5e..ecda0229af 100644
@@ -520,9 +520,6 @@ static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)
 /*
  * Initialize a repository struct for a submodule based on the provided 'path'.
  *
- * Unlike repo_submodule_init, this tolerates submodules not present
- * in .gitmodules. This function exists only to preserve historical behavior,
- *
  * Returns the repository struct on success,
  * NULL when the submodule is not present.
  */
@@ -1404,11 +1401,11 @@ static void fetch_task_release(struct fetch_task *p)
 }
 
 static struct repository *get_submodule_repo_for(struct repository *r,
-						 const struct submodule *sub)
+						 const char *path)
 {
 	struct repository *ret = xmalloc(sizeof(*ret));
 
-	if (repo_submodule_init(ret, r, sub)) {
+	if (repo_submodule_init(ret, r, path, null_oid())) {
 		free(ret);
 		return NULL;
 	}
@@ -1452,7 +1449,7 @@ static int get_next_submodule(struct child_process *cp,
 			continue;
 		}
 
-		task->repo = get_submodule_repo_for(spf->r, task->sub);
+		task->repo = get_submodule_repo_for(spf->r, task->sub->path);
 		if (task->repo) {
 			struct strbuf submodule_prefix = STRBUF_INIT;
 			child_process_init(cp);
t/helper/test-submodule-nested-repo-config.c +1 -3
index e3f11ff5a7..dc1c14bde3 100644
@@ -11,15 +11,13 @@ static void die_usage(const char **argv, const char *msg)
 int cmd__submodule_nested_repo_config(int argc, const char **argv)
 {
 	struct repository subrepo;
-	const struct submodule *sub;
 
 	if (argc < 3)
 		die_usage(argv, "Wrong number of arguments.");
 
 	setup_git_directory();
 
-	sub = submodule_from_path(the_repository, null_oid(), argv[1]);
-	if (repo_submodule_init(&subrepo, the_repository, sub)) {
+	if (repo_submodule_init(&subrepo, the_repository, argv[1], null_oid())) {
 		die_usage(argv, "Submodule not found.");
 	}
 
Signature · 1 line
2.33.0.309.g3052b89438-goog
#12 lore raw

PATCH v2 3/3 revision: remove "submodule" from opt struct

In reply to Jonathan Tan

Clean up a TODO in revision.h by removing the "submodule" field from struct setup_revision_opt. This field is only used to specify the ref store to use, so use rev_info->repo to determine the ref store instead.

The only users of this field are merge-ort.c and merge-recursive.c. However, both these files specify the superproject as rev_info->repo and the submodule as setup_revision_opt->submodule. In order to be able to pass the submodule as rev_info->repo, all commits must be parsed with the submodule explicitly specified; this patch does that as well. (An incremental solution in which only some commits are parsed with explicit submodule will not work, because if the same commit is parsed twice in different repositories, there will be 2 heap-allocated object structs corresponding to that commit, and any flag set by the revision walking mechanism on one of them will not be reflected onto the other.)

---
 merge-ort.c       | 53 +++++++++++++++++++++++++++++++----------------
 merge-recursive.c | 49 ++++++++++++++++++++++++++++---------------
 revision.c        | 16 +++++---------
 revision.h        |  1 -
 4 files changed, 72 insertions(+), 47 deletions(-)
merge-ort.c +35 -18
index 6eb910d6f0..b8efaee8e0 100644
@@ -32,6 +32,7 @@
 #include "promisor-remote.h"
 #include "revision.h"
 #include "strmap.h"
+#include "submodule-config.h"
 #include "submodule.h"
 #include "tree.h"
 #include "unpack-trees.h"
@@ -1499,7 +1500,6 @@ static int find_first_merges(struct repository *repo,
 	xsnprintf(merged_revision, sizeof(merged_revision), "^%s",
 		  oid_to_hex(&a->object.oid));
 	repo_init_revisions(repo, &revs, NULL);
-	rev_opts.submodule = path;
 	/* FIXME: can't handle linked worktrees in submodules yet */
 	revs.single_worktree = path != NULL;
 	setup_revisions(ARRAY_SIZE(rev_args)-1, rev_args, &revs, &rev_opts);
@@ -1509,7 +1509,7 @@ static int find_first_merges(struct repository *repo,
 		die("revision walk setup failed");
 	while ((commit = get_revision(&revs)) != NULL) {
 		struct object *o = &(commit->object);
-		if (in_merge_bases(b, commit))
+		if (repo_in_merge_bases(repo, b, commit))
 			add_object_array(o, NULL, &merges);
 	}
 	reset_revision_walk();
@@ -1524,7 +1524,7 @@ static int find_first_merges(struct repository *repo,
 		contains_another = 0;
 		for (j = 0; j < merges.nr; j++) {
 			struct commit *m2 = (struct commit *) merges.objects[j].item;
-			if (i != j && in_merge_bases(m2, m1)) {
+			if (i != j && repo_in_merge_bases(repo, m2, m1)) {
 				contains_another = 1;
 				break;
 			}
@@ -1545,10 +1545,12 @@ static int merge_submodule(struct merge_options *opt,
 			   const struct object_id *b,
 			   struct object_id *result)
 {
+	struct repository subrepo;
+	struct strbuf sb = STRBUF_INIT;
+	int ret = 0;
 	struct commit *commit_o, *commit_a, *commit_b;
 	int parent_count;
 	struct object_array merges;
-	struct strbuf sb = STRBUF_INIT;
 
 	int i;
 	int search = !opt->priv->call_depth;
@@ -1564,6 +1566,10 @@ static int merge_submodule(struct merge_options *opt,
 	if (is_null_oid(b))
 		return 0;
 
+	/*
+	 * NEEDSWORK: Remove this when all submodule object accesses are
+	 * through explicitly specified repositores.
+	 */
 	if (add_submodule_odb(path)) {
 		path_msg(opt, path, 0,
 			 _("Failed to merge submodule %s (not checked out)"),
@@ -1571,39 +1577,48 @@ static int merge_submodule(struct merge_options *opt,
 		return 0;
 	}
 
-	if (!(commit_o = lookup_commit_reference(opt->repo, o)) ||
-	    !(commit_a = lookup_commit_reference(opt->repo, a)) ||
-	    !(commit_b = lookup_commit_reference(opt->repo, b))) {
+	if (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {
+		path_msg(opt, path, 0,
+				_("Failed to merge submodule %s (not checked out)"),
+				path);
+		return 0;
+	}
+
+	if (!(commit_o = lookup_commit_reference(&subrepo, o)) ||
+	    !(commit_a = lookup_commit_reference(&subrepo, a)) ||
+	    !(commit_b = lookup_commit_reference(&subrepo, b))) {
 		path_msg(opt, path, 0,
 			 _("Failed to merge submodule %s (commits not present)"),
 			 path);
-		return 0;
+		goto cleanup;
 	}
 
 	/* check whether both changes are forward */
-	if (!in_merge_bases(commit_o, commit_a) ||
-	    !in_merge_bases(commit_o, commit_b)) {
+	if (!repo_in_merge_bases(&subrepo, commit_o, commit_a) ||
+	    !repo_in_merge_bases(&subrepo, commit_o, commit_b)) {
 		path_msg(opt, path, 0,
 			 _("Failed to merge submodule %s "
 			   "(commits don't follow merge-base)"),
 			 path);
-		return 0;
+		goto cleanup;
 	}
 
 	/* Case #1: a is contained in b or vice versa */
-	if (in_merge_bases(commit_a, commit_b)) {
+	if (repo_in_merge_bases(&subrepo, commit_a, commit_b)) {
 		oidcpy(result, b);
 		path_msg(opt, path, 1,
 			 _("Note: Fast-forwarding submodule %s to %s"),
 			 path, oid_to_hex(b));
-		return 1;
+		ret = 1;
+		goto cleanup;
 	}
-	if (in_merge_bases(commit_b, commit_a)) {
+	if (repo_in_merge_bases(&subrepo, commit_b, commit_a)) {
 		oidcpy(result, a);
 		path_msg(opt, path, 1,
 			 _("Note: Fast-forwarding submodule %s to %s"),
 			 path, oid_to_hex(a));
-		return 1;
+		ret = 1;
+		goto cleanup;
 	}
 
 	/*
@@ -1615,10 +1630,10 @@ static int merge_submodule(struct merge_options *opt,
 
 	/* Skip the search if makes no sense to the calling context.  */
 	if (!search)
-		return 0;
+		goto cleanup;
 
 	/* find commit which merges them */
-	parent_count = find_first_merges(opt->repo, path, commit_a, commit_b,
+	parent_count = find_first_merges(&subrepo, path, commit_a, commit_b,
 					 &merges);
 	switch (parent_count) {
 	case 0:
@@ -1652,7 +1667,9 @@ static int merge_submodule(struct merge_options *opt,
 	}
 
 	object_array_clear(&merges);
-	return 0;
+cleanup:
+	repo_clear(&subrepo);
+	return ret;
 }
 
 static void initialize_attr_index(struct merge_options *opt)
merge-recursive.c +32 -17

125 lines not on this page · raw message

revision.c +5 -11
index cddd0542a6..31fc1884d2 100644
@@ -2561,8 +2561,7 @@ static int for_each_good_bisect_ref(struct ref_store *refs, each_ref_fn fn, void
 	return for_each_bisect_ref(refs, fn, cb_data, term_good);
 }
 
-static int handle_revision_pseudo_opt(const char *submodule,
-				      struct rev_info *revs,
+static int handle_revision_pseudo_opt(struct rev_info *revs,
 				      const char **argv, int *flags)
 {
 	const char *arg = argv[0];
@@ -2570,7 +2569,7 @@ static int handle_revision_pseudo_opt(const char *submodule,
 	struct ref_store *refs;
 	int argcount;
 
-	if (submodule) {
+	if (revs->repo != the_repository) {
 		/*
 		 * We need some something like get_submodule_worktrees()
 		 * before we can go through all worktrees of a submodule,
@@ -2579,9 +2578,8 @@ static int handle_revision_pseudo_opt(const char *submodule,
 		 */
 		if (!revs->single_worktree)
 			BUG("--single-worktree cannot be used together with submodule");
-		refs = get_submodule_ref_store(submodule);
-	} else
-		refs = get_main_ref_store(revs->repo);
+	}
+	refs = get_main_ref_store(revs->repo);
 
 	/*
 	 * NOTE!
@@ -2699,12 +2697,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s
 {
 	int i, flags, left, seen_dashdash, revarg_opt;
 	struct strvec prune_data = STRVEC_INIT;
-	const char *submodule = NULL;
 	int seen_end_of_options = 0;
 
-	if (opt)
-		submodule = opt->submodule;
-
 	/* First, search for "--" */
 	if (opt && opt->assume_dashdash) {
 		seen_dashdash = 1;
@@ -2733,7 +2727,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s
 		if (!seen_end_of_options && *arg == '-') {
 			int opts;
 
-			opts = handle_revision_pseudo_opt(submodule,
+			opts = handle_revision_pseudo_opt(
 						revs, argv + i,
 						&flags);
 			if (opts > 0) {
revision.h +0 -1
index fbb068da9f..d8e0e93026 100644
@@ -339,7 +339,6 @@ extern volatile show_early_output_fn_t show_early_output;
 struct setup_revision_opt {
 	const char *def;
 	void (*tweak)(struct rev_info *, struct setup_revision_opt *);
-	const char *submodule;	/* TODO: drop this and use rev_info->repo */
 	unsigned int	assume_dashdash:1,
 			allow_exclude_promisor_objects:1;
 	unsigned revarg_opt;
Signature · 1 line
2.33.0.309.g3052b89438-goog
#13 lore raw
In reply to Jonathan Tan
Quoted 10 lines · Jonathan Tan

This is on jt/grep-wo-submodule-odb-as-alternate (unlike version 1) because that branch introduces another usage of repo_submodule_init(), and I need to update it.

The main change from version 1 is that I have introduced a patch (patch
2) that teaches repo_submodule_init() to support submodules with unabsorbed gitdirs, which means that I no longer need the original patch 1 that updated t6437 to absorb all git dirs.

Quoted 5 lines

Once again, all these patches work under GIT_TEST_MERGE_ALGORITHM=recursive and GIT_TEST_MERGE_ALGORITHM=ort (and when that envvar is unset, for good measure).

I have also tested it with the "DO NOT SUBMIT" patch 3 from version 1.

While I don't yet know the details of repositories and submodules, I know the merge-recursive and merge-ort side of things and those changes all look reasonable to me. So here's an Ack for that side of patch 3.

Despite not being familiar with the repository and submodule storage parts of the code, I took a look over the remainder of the 3 patches to look for anything that might stand out and look amiss. I didn't spot anything; it all looks reasonable to me.