fetch --recurse-submodules: fetch unpopulated submodules
cover fetch --recurse-submodules: fetch unpopulated submodules
= Background
When fetching submodule commits, "git fetch --recurse-submodules" only considers populated submodules, and not all of the submodules in $GIT_DIR/modules as one might expect. As a result, "git fetch --recurse-submodules" behaves differently based on which commit is checked out.
This can be a problem, for instance, if the user has a branch with submodules and a branch without:
# the submodules were initialized at some point in history.. git checkout -b branch-with-submodules origin/branch-with-submodules git submodule update --init
# later down the road.. git checkout --recurse-submodules branch-without-submodules # no submodules are fetched! git fetch --recurse-submodules # if origin/branch-with-submodules has new submodule commits, this # checkout will fail because we never fetched the submodule git checkout --recurse-submodules branch-with-submodules
This series makes "git fetch" fetch the right submodules regardless of which commit is checked out, as long as the submodule has already been cloned. In particular, "git fetch" learns to:
1. read submodules from the relevant superproject commit instead of the file system
2. fetch all changed submodules, even if they are not populated
= Patch organization
- Patches 1-3 teach "git fetch" to read .gitmodules from the superproject commit.
- Patches 4-5 are quality-of-life improvements to the test suite that make it easier to write the tests in patch 7.
- Patch 6 separates the steps of "finding which submodules to fetch" and "fetching the submodules", making it easier to tell "git fetch" to fetch unpopulated submodules.
- Patch 7 teaches "git fetch" to fetch changed, unpopulated submodules in addition to populated submodules.
- Patch 8 is an optional bugfix + cleanup of the "git fetch" code that removes the last caller of the deprecated "add_submodule_odb()".
= Future work
Even with this series, there is no guarantee that "git fetch" will fetch every necessary submodule commit because a superproject commit can introduce new submodules, and since those submodules are not cloned, "git fetch" cannot fetch those commits yet. This series should get us closer to that goal because "git fetch" can read submodules from the superproject commit, which is a necessary precursor to figuring out whether to clone submodules from superproject commits.
Glen Choo (8):
submodule: inline submodule_commits() into caller
submodule: store new submodule commits oid_array in a struct
submodule: make static functions read submodules from commits
t5526: introduce test helper to assert on fetches
t5526: use grep to assert on fetches
submodule: extract get_fetch_task()
fetch: fetch unpopulated, changed submodules
submodule: fix bug and remove add_submodule_odb()
Documentation/fetch-options.txt | 26 ++- Documentation/git-fetch.txt | 10 +- submodule.c | 316 ++++++++++++++++---------- submodule.h | 9 +- t/t5526-fetch-submodules.sh | 386 ++++++++++++++++++++++++-------- 5 files changed, 524 insertions(+), 223 deletions(-)
base-commit: 679e3693aba0c17af60c031f7eef68f2296b8dad
Signature · 1 line
2.33.GIT
PATCH 1/8 submodule: inline submodule_commits() into caller
When collecting the string_list of changed submodule names, the new submodules commits are stored in the string_list_item.util as an oid_array. A subsequent commit will replace the oid_array with a struct that has more information.
Prepare for this change by inlining submodule_commits() (which inserts into the string_list and initializes the string_list_item.util) into its only caller. This simplifies the code and makes it easier for the caller to add information to the string_list_item.util.
--- submodule.c | 22 ++++++---------------- 1 file changed, 6 insertions(+), 16 deletions(-)
submodule.c +6 -16
@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)return submodule_from_path (the_repository,null_oid (), ce->name); } -static struct oid_array *submodule_commits (struct string_list *submodules, -const char *name) -{ -struct string_list_item *item; - - item =string_list_insert (submodules, name); -if (item->util) -return (struct oid_array *) item->util; - -/* NEEDSWORK: should we have oid_array_init()? */ - item->util =xcalloc (1 ,sizeof (struct oid_array )); -return (struct oid_array *) item->util; -}struct collect_changed_submodules_cb_data {struct repository *repo;
@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,for (i =0 ; i < q->nr; i++) {struct diff_filepair *p = q->queue[i]; -struct oid_array *commits;const struct submodule *submodule;const char *name; +struct string_list_item *item;if (!S_ISGITLINK (p->two->mode))continue ;
@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,if (!name)continue ; - commits =submodule_commits (changed, name); -oid_array_append (commits, &p->two->oid); + item =string_list_insert (changed, name); +if (!item->util) +/* NEEDSWORK: should we have oid_array_init()? */ + item->util =xcalloc (1 ,sizeof (struct oid_array )); +oid_array_append (item->util, &p->two->oid); } }
Signature · 1 line
2.33.GIT
PATCH 2/8 submodule: store new submodule commits oid_array in a struct
In reply to Glen ChooThis commit prepares for a future commit that will teach git fetch --recurse-submodules how to fetch submodules that are present in <gitdir>/modules, but are not populated. To do this, we need to store more information about the changed submodule so that we can read the submodule configuration from the superproject commit instead of the filesystem.
Refactor the changed submodules string_list.util to hold a struct instead of an oid_array. This struct only holds the new_commits oid_array for now; more information will be added later.
--- submodule.c | 60 ++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 19 deletions(-)
submodule.c +41 -19
@@ -806,6 +806,21 @@ static const char *default_name_or_path(const char *path_or_name)return path_or_name; } +/* +* Holds relevant information for a changed submodule. Used as the .util +* member of the changed submodule string_list_item. +*/ +struct changed_submodule_data { +/* The submodule commits that have changed in the rev walk. */ +struct oid_array *new_commits; +}; + +static void changed_submodule_data_clear (struct changed_submodule_data *cs_data) +{ +oid_array_clear (cs_data->new_commits); +free (cs_data->new_commits); +} +static void collect_changed_submodules_cb (struct diff_queue_struct *q,struct diff_options *options,void *data)
@@ -820,6 +835,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,const struct submodule *submodule;const char *name;struct string_list_item *item; +struct changed_submodule_data *cs_data;if (!S_ISGITLINK (p->two->mode))continue ;
@@ -847,10 +863,15 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,continue ; item =string_list_insert (changed, name); -if (!item->util) +if (item->util) + cs_data = item->util; +else { + cs_data =xcalloc (1 ,sizeof (struct changed_submodule_data ));/* NEEDSWORK: should we have oid_array_init()? */ - item->util =xcalloc (1 ,sizeof (struct oid_array )); -oid_array_append (item->util, &p->two->oid); + cs_data->new_commits =xcalloc (1 ,sizeof (struct oid_array )); + item->util = cs_data; + } +oid_array_append (cs_data->new_commits, &p->two->oid); } }
@@ -897,11 +918,12 @@ static void collect_changed_submodules(struct repository *r,reset_revision_walk (); } -static void free_submodules_oids (struct string_list *submodules) +static void free_submodules_data (struct string_list *submodules) {struct string_list_item *item; -for_each_string_list_item (item, submodules) -oid_array_clear ((struct oid_array *) item->util); +for_each_string_list_item (item, submodules) { +changed_submodule_data_clear (item->util); + }string_list_clear (submodules,1 ); }
@@ -1067,7 +1089,7 @@ int find_unpushed_submodules(struct repository *r,collect_changed_submodules (r, &submodules, &argv);for_each_string_list_item (name, &submodules) { -struct oid_array *commits = name->util; +struct changed_submodule_data *cs_data = name->util;const struct submodule *submodule;const char *path =NULL ;
@@ -1080,11 +1102,11 @@ int find_unpushed_submodules(struct repository *r,if (!path)continue ; -if (submodule_needs_pushing (r, path, commits)) +if (submodule_needs_pushing (r, path, cs_data->new_commits))string_list_insert (needs_pushing, path); } -free_submodules_oids (&submodules); +free_submodules_data (&submodules);strvec_clear (&argv);return needs_pushing->nr;
@@ -1254,7 +1276,7 @@ static void calculate_changed_submodule_paths(struct repository *r,collect_changed_submodules (r, changed_submodule_names, &argv);for_each_string_list_item (name, changed_submodule_names) { -struct oid_array *commits = name->util; +struct changed_submodule_data *cs_data = name->util;const struct submodule *submodule;const char *path =NULL ;
@@ -1267,8 +1289,8 @@ static void calculate_changed_submodule_paths(struct repository *r,if (!path)continue ; -if (submodule_has_commits (r, path, commits)) { -oid_array_clear (commits); +if (submodule_has_commits (r, path, cs_data->new_commits)) { +oid_array_clear (cs_data->new_commits); *name->string ='\0' ; } }
@@ -1305,7 +1327,7 @@ int submodule_touches_in_range(struct repository *r,strvec_clear (&args); -free_submodules_oids (&subs); +free_submodules_data (&subs);return ret; }
@@ -1587,7 +1609,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,struct fetch_task *task = task_cb;struct string_list_item *it; -struct oid_array *commits; +struct changed_submodule_data *cs_data;if (!task || !task->sub)BUG ("callback cookie bogus" );
@@ -1615,14 +1637,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,/* Could be an unchanged submodule, not contained in the list */ goto out; - commits = it->util; -oid_array_filter (commits, + cs_data = it->util; +oid_array_filter (cs_data->new_commits, commit_missing_in_sub, task->repo);/* Are there commits we want, but do not exist? */ -if (commits->nr) { - task->commits = commits; +if (cs_data->new_commits->nr) { + task->commits = cs_data->new_commits;ALLOC_GROW (spf->oid_fetch_tasks, spf->oid_fetch_tasks_nr +1 , spf->oid_fetch_tasks_alloc);
@@ -1680,7 +1702,7 @@ int fetch_populated_submodules(struct repository *r,strvec_clear (&spf.args); out: -free_submodules_oids (&spf.changed_submodule_names); +free_submodules_data (&spf.changed_submodule_names);return spf.result; }
Signature · 1 line
2.33.GIT
Re: [PATCH 2/8] submodule: store new submodule commits oid_array in a struct
submodule.c static const char *default_name_or_path(const char *path_or_name) · line 816
Quoted 8 lines · Glen Choo
submodule.c +8
-0+/* +* Holds relevant information for a changed submodule. Used as the .util +* member of the changed submodule string_list_item. +*/ +struct changed_submodule_data { +/* The submodule commits that have changed in the rev walk. */ +struct oid_array *new_commits; +};
Overall this change is straightforward and looks good, except that I think that the struct oid_array can be embedded directly instead of through a pointer.
Re: [PATCH 2/8] submodule: store new submodule commits oid_array in a struct
Quoted 13 lines · Jonathan Tan
submodule.c static const char *default_name_or_path(const char *path_or_name) · line 816
Quoted 8 lines · Glen Choo
submodule.c +8
-0+/* +* Holds relevant information for a changed submodule. Used as the .util +* member of the changed submodule string_list_item. +*/ +struct changed_submodule_data { +/* The submodule commits that have changed in the rev walk. */ +struct oid_array *new_commits; +};Overall this change is straightforward and looks good, except that I think that the struct
oid_arraycan be embedded directly instead of through a pointer.
True.
I am a bit behind and haven't seen the simplicity 1/8 promised to bring to us, but hopefully we'll see soon enough why 1/8 is a good idea.
PATCH 3/8 submodule: make static functions read submodules from commits
In reply to Glen ChooA future commit will teach "fetch --recurse-submodules" to fetch unpopulated submodules. Prepare for this by teaching the necessary static functions to read submodules from superproject commits instead of the index and filesystem. Then, store the necessary fields (path and super_oid), and use them in "fetch --recurse-submodules" where possible.
As a result, "git fetch" now reads changed submodules using the .gitmodules and path from super_oid's tree (which is where "git fetch" actually noticed the changed submodule) instead of the filesystem.
--- submodule.c | 39 ++++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 11 deletions(-)
submodule.c +28 -11
@@ -811,6 +811,16 @@ static const char *default_name_or_path(const char *path_or_name)* member of the changed submodule string_list_item. */ struct changed_submodule_data { +/* +* The first superproject commit in the rev walk that points to the +* submodule. +*/ +const struct object_id *super_oid; +/* +* Path to the submodule in the superproject commit referenced +* by 'super_oid'. +*/ +char *path;/* The submodule commits that have changed in the rev walk. */ struct oid_array *new_commits; };
@@ -819,6 +829,7 @@ static void changed_submodule_data_clear(struct changed_submodule_data *cs_data) {oid_array_clear (cs_data->new_commits);free (cs_data->new_commits); +free (cs_data->path); } static void collect_changed_submodules_cb(struct diff_queue_struct *q,
@@ -869,6 +880,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q, cs_data =xcalloc (1 ,sizeof (struct changed_submodule_data ));/* NEEDSWORK: should we have oid_array_init()? */ cs_data->new_commits =xcalloc (1 ,sizeof (struct oid_array )); + cs_data->super_oid = commit_oid; + cs_data->path =xstrdup (p->two->path); item->util = cs_data; }oid_array_append (cs_data->new_commits, &p->two->oid);
@@ -944,6 +957,7 @@ struct has_commit_data {struct repository *repo;int result;const char *path; +const struct object_id *super_oid; };static int check_has_commit (const struct object_id *oid,void *data)
@@ -952,7 +966,7 @@ static int check_has_commit(const struct object_id *oid, void *data)struct repository subrepo;enum object_type type; -if (repo_submodule_init (&subrepo, cb->repo, cb->path,null_oid ())) { +if (repo_submodule_init (&subrepo, cb->repo, cb->path, cb->super_oid)) { cb->result =0 ; goto cleanup; }
@@ -980,9 +994,10 @@ static int check_has_commit(const struct object_id *oid, void *data)static int submodule_has_commits (struct repository *r,const char *path, +const struct object_id *super_oid,struct oid_array *commits) { -struct has_commit_data has_commit = { r,1 , path }; +struct has_commit_data has_commit = { r,1 , path, super_oid };/* * Perform a cheap, but incorrect check for the existence of 'commits'.
@@ -1029,7 +1044,7 @@ static int submodule_needs_pushing(struct repository *r,const char *path, struct oid_array *commits) { - if (!submodule_has_commits(r, path, commits)) + if (!submodule_has_commits(r, path, null_oid(), commits))/* * NOTE: We do consider it safe to return "no" here. The * correct answer would be "We do not know" instead of
@@ -1280,7 +1295,7 @@ static void calculate_changed_submodule_paths(struct repository *r,const struct submodule *submodule;const char *path =NULL ; - submodule =submodule_from_name (r,null_oid (), name->string); + submodule =submodule_from_name (r, cs_data->super_oid, name->string);if (submodule) path = submodule->path;else
@@ -1289,7 +1304,7 @@ static void calculate_changed_submodule_paths(struct repository *r,if (!path)continue ; -if (submodule_has_commits (r, path, cs_data->new_commits)) { +if (submodule_has_commits (r, path, cs_data->super_oid, cs_data->new_commits)) {oid_array_clear (cs_data->new_commits); *name->string ='\0' ; }
@@ -1414,12 +1429,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path) }static struct fetch_task *fetch_task_create (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct fetch_task *task =xmalloc (sizeof (*task));memset (task,0 ,sizeof (*task)); - task->sub =submodule_from_path (r,null_oid (), path); + task->sub =submodule_from_path (r, treeish_name, path);if (!task->sub) {/* * No entry in .gitmodules? Technically not a submodule,
@@ -1451,11 +1467,12 @@ static void fetch_task_release(struct fetch_task *p) }static struct repository *get_submodule_repo_for (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct repository *ret =xmalloc (sizeof (*ret)); -if (repo_submodule_init (ret, r, path,null_oid ())) { +if (repo_submodule_init (ret, r, path, treeish_name)) {free (ret);return NULL ; }
@@ -1476,7 +1493,7 @@ static int get_next_submodule(struct child_process *cp,if (!S_ISGITLINK (ce->ce_mode))continue ; - task =fetch_task_create (spf->r, ce->name); + task =fetch_task_create (spf->r, ce->name,null_oid ());if (!task)continue ;
@@ -1499,7 +1516,7 @@ static int get_next_submodule(struct child_process *cp,continue ; } - task->repo =get_submodule_repo_for (spf->r, task->sub->path); + task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());if (task->repo) {struct strbuf submodule_prefix =STRBUF_INIT ;child_process_init (cp);
Signature · 1 line
2.33.GIT
Re: [PATCH 3/8] submodule: make static functions read submodules from commits
Quoted 3 lines · Glen Choo
As a result, "git fetch" now reads changed submodules using the
.gitmodulesand path from super_oid's tree (which is where "git fetch" actually noticed the changed submodule) instead of the filesystem.
Could we have a test showing what has changed?
submodule.c static int submodule_needs_pushing(struct repository *r, · line 1047
Quoted 6 lines
submodule.c +1
-1@@ -1029,7 +1044,7 @@ static int submodule_needs_pushing(struct repository *r,const char *path, struct oid_array *commits) { - if (!submodule_has_commits(r, path, commits)) + if (!submodule_has_commits(r, path, null_oid(), commits))
This confused me at first, but I see that this code is not for fetching, but for pushing. This patch set concerns itself with fetching, so passing null_oid() to preserve existing behavior is good.
submodule.c static const struct submodule *get_non_gitmodules_submodule(const char *path) · line 1438
Quoted 13 lines
submodule.c +3
-2@@ -1414,12 +1429,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path) }static struct fetch_task *fetch_task_create (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct fetch_task *task =xmalloc (sizeof (*task));memset (task,0 ,sizeof (*task)); - task->sub =submodule_from_path (r,null_oid (), path); + task->sub =submodule_from_path (r, treeish_name, path);
If there is not a good reason to have "path" before "treeish_name", probably best to maintain the same parameter order as submodule_from_path().
submodule.c static int get_next_submodule(struct child_process *cp, · line 1496
Quoted 6 lines
submodule.c +1
-1@@ -1476,7 +1493,7 @@ static int get_next_submodule(struct child_process *cp,if (!S_ISGITLINK (ce->ce_mode))continue ; - task =fetch_task_create (spf->r, ce->name); + task =fetch_task_create (spf->r, ce->name,null_oid ());
Hmm...is the plumbing incomplete? This code is about fetching, but we're not passing any superproject commit OID here. If this will be fixed in a future commit, maybe the distribution of what goes into each commit needs to be revised.
submodule.c static int get_next_submodule(struct child_process *cp, · line 1519
Quoted 6 lines
submodule.c +1
-1@@ -1499,7 +1516,7 @@ static int get_next_submodule(struct child_process *cp,continue ; } - task->repo =get_submodule_repo_for (spf->r, task->sub->path); + task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());
Same comment here.
Re: [PATCH 3/8] submodule: make static functions read submodules from commits
Hm, after reading your feedback, I'm starting to question whether this patch makes sense in its current form.
Quoted 6 lines · Jonathan Tan
Quoted 3 lines · Glen Choo
As a result, "git fetch" now reads changed submodules using the
.gitmodulesand path from super_oid's tree (which is where "git fetch" actually noticed the changed submodule) instead of the filesystem.Could we have a test showing what has changed?
Looking at this closer, I don't think a test is feasible, but even more importantly, I don't think this behavior change is even desirable at all..
I was confused when I wrote the commit message. Prior to this patch, "git fetch" already records the names of changed submodules by correctly reading .gitmodules from the given commit. From collect_changed_submodules_cb():
submodule = submodule_from_path(me->repo,
commit_oid, p->two->path);
[...]
item = string_list_insert(changed, name);
if (!item->util)
item->util = xcalloc(1, sizeof(struct changed_submodule_data));
cs_data = item->util;
oid_array_append(&cs_data->new_commits, &p->two->oid);The only behavior that actually _does_ change is that we plumb super_oid through submodule_has_commits(). "git fetch" invokes this function to figure out if it already has all of the needed commits, and if so, skip the fetch.
Before plumbing super_oid, we could not check for commits in an unpopulated submodule, but now we do. We will need this when we fetch in unpopulated submodules, but as of this patch, we never fetch in unpopulated submodules anyway, so this check is just wasted effort.
And because we never fetch anyway, we can't test any meaningful behavior. We could check whether or not we check the submodule odb, but that's a lot of effort to spend on something we don't need.
So we probably don't want this behavior change. I can preserve the existing behavior by passing null_oid() instead, and pass super_oid in the actual "fetch unpopulated submodules" patch.
Quoted 12 lines
submodule.c static int get_next_submodule(struct child_process *cp, · line 1496
Quoted 6 lines
submodule.c +1
-1@@ -1476,7 +1493,7 @@ static int get_next_submodule(struct child_process *cp,if (!S_ISGITLINK (ce->ce_mode))continue ; - task =fetch_task_create (spf->r, ce->name); + task =fetch_task_create (spf->r, ce->name,null_oid ());Hmm...is the plumbing incomplete? This code is about fetching, but we're not passing any superproject commit OID here. If this will be fixed in a future commit, maybe the distribution of what goes into each commit needs to be revised.
Quoted 8 lines
submodule.c static int get_next_submodule(struct child_process *cp, · line 1519
Quoted 6 lines
submodule.c +1
-1@@ -1499,7 +1516,7 @@ static int get_next_submodule(struct child_process *cp,continue ; } - task->repo =get_submodule_repo_for (spf->r, task->sub->path); + task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());Same comment here.
This is intentional (but I admit that I also got confused rereading this). This should be null_oid() (i.e. read from the filesystem) because we are iterating through the index to get submodule paths. So we should pass null_oid() so that we read .gitmodules from the filesystem, not a possibly out-of-sync superproject commit.
Re: [PATCH 3/8] submodule: make static functions read submodules from commits
In reply to Jonathan TanHm, after reading your feedback, I'm starting to question whether this patch makes sense in its current form.
Quoted 6 lines · Jonathan Tan
Quoted 3 lines · Glen Choo
As a result, "git fetch" now reads changed submodules using the
.gitmodulesand path from super_oid's tree (which is where "git fetch" actually noticed the changed submodule) instead of the filesystem.Could we have a test showing what has changed?
Looking at this closer, I don't think a test is feasible, but even more importantly, I don't think this behavior change is even desirable at all..
I was confused when I wrote the commit message. Prior to this patch, "git fetch" already records the names of changed submodules by correctly reading .gitmodules from the given commit. From collect_changed_submodules_cb():
submodule = submodule_from_path(me->repo,
commit_oid, p->two->path);
[...]
item = string_list_insert(changed, name);
if (!item->util)
item->util = xcalloc(1, sizeof(struct changed_submodule_data));
cs_data = item->util;
oid_array_append(&cs_data->new_commits, &p->two->oid);The only behavior that actually _does_ change is that we plumb super_oid through submodule_has_commits(). "git fetch" invokes this function to figure out if it already has all of the needed commits, and if so, skip the fetch.
Before plumbing super_oid, we could not check for commits in an unpopulated submodule, but now we do. We will need this when we fetch in unpopulated submodules, but as of this patch, we never fetch in unpopulated submodules anyway, so this check is just wasted effort.
And because we never fetch anyway, we can't test any meaningful behavior. We could check whether or not we check the submodule odb, but that's a lot of effort to spend on something we don't need.
So we probably don't want this behavior change. I can preserve the existing behavior by passing null_oid() instead, and pass super_oid in the actual "fetch unpopulated submodules" patch.
Quoted 12 lines
submodule.c static int get_next_submodule(struct child_process *cp, · line 1496
Quoted 6 lines
submodule.c +1
-1@@ -1476,7 +1493,7 @@ static int get_next_submodule(struct child_process *cp,if (!S_ISGITLINK (ce->ce_mode))continue ; - task =fetch_task_create (spf->r, ce->name); + task =fetch_task_create (spf->r, ce->name,null_oid ());Hmm...is the plumbing incomplete? This code is about fetching, but we're not passing any superproject commit OID here. If this will be fixed in a future commit, maybe the distribution of what goes into each commit needs to be revised.
Quoted 8 lines
submodule.c static int get_next_submodule(struct child_process *cp, · line 1519
Quoted 6 lines
submodule.c +1
-1@@ -1499,7 +1516,7 @@ static int get_next_submodule(struct child_process *cp,continue ; } - task->repo =get_submodule_repo_for (spf->r, task->sub->path); + task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());Same comment here.
This is intentional (but I admit that I also got confused rereading this). This should be null_oid() (i.e. read from the filesystem) because we are iterating through the index to get submodule paths. So we should pass null_oid() so that we read .gitmodules from the filesystem, not a possibly out-of-sync superproject commit.
PATCH 4/8 t5526: introduce test helper to assert on fetches
In reply to Glen ChooA future commit will change the stderr of "git fetch --recurse-submodules" and add new tests to t/t5526-fetch-submodules.sh. This poses two challenges:
* The tests use test_cmp to assert on the stderr, which will fail on the future test because the stderr changes slightly, even though it still contains the information we expect.
* The expect.err file is constructed by the add_upstream_commit() helper as input into test_cmp, but most tests fetch a different combination of repos from expect.err. This results in noisy tests that modify parts of that expect.err to generate the expected output.
To address both of these issues, introduce a verify_fetch_result() helper to t/t5526-fetch-submodules.sh that asserts on the output of "git fetch --recurse-submodules" and handles the ordering of expect.err.
As a result, the tests no longer construct expect.err manually. test_cmp is still invoked by verify_fetch_result(), but that will be replaced in a later commit.
--- t/t5526-fetch-submodules.sh | 136 +++++++++++++++++++++--------------- 1 file changed, 81 insertions(+), 55 deletions(-)
t/t5526-fetch-submodules.sh +81 -55
@@ -13,6 +13,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +# For each submodule in the test setup, this creates a commit and writes +# a file that contains the expected err if that new commit were fetched. +# These output files get concatenated in the right order by +# verify_fetch_result(). add_upstream_commit() { ( cd submodule &&
@@ -22,9 +26,9 @@ add_upstream_commit() { git add subfile && git commit -m new subfile && head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err && - echo "From $pwd/submodule" >> ../expect.err && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err + echo "Fetching submodule submodule" > ../expect.err.sub && + echo "From $pwd/submodule" >> ../expect.err.sub && + echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub ) && ( cd deepsubmodule &&
@@ -34,12 +38,33 @@ add_upstream_commit() { git add deepsubfile && git commit -m new deepsubfile && head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule/subdir/deepsubmodule" >> ../expect.err - echo "From $pwd/deepsubmodule" >> ../expect.err && - echo " $head1..$head2 deep -> origin/deep" >> ../expect.err + echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep + echo "From $pwd/deepsubmodule" >> ../expect.err.deep && + echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep ) } +# Verifies that the expected repositories were fetched. This is done by +# concatenating the files expect.err.[super|sub|deep] in the correct +# order and comparing it to the actual stderr. +# +# If a repo should not be fetched in the test, its corresponding +# expect.err file should be rm-ed. +verify_fetch_result() { + ACTUAL_ERR=$1 && + rm -f expect.err.combined && + if [ -f expect.err.super ]; then + cat expect.err.super >>expect.err.combined + fi && + if [ -f expect.err.sub ]; then + cat expect.err.sub >>expect.err.combined + fi && + if [ -f expect.err.deep ]; then + cat expect.err.deep >>expect.err.combined + fi && + test_cmp expect.err.combined $ACTUAL_ERR +} + test_expect_success setup ' mkdir deepsubmodule && (
@@ -77,7 +102,7 @@ test_expect_success "fetch --recurse-submodules recurses into submodules" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "submodule.recurse option triggers recursive fetch" '
@@ -87,7 +112,7 @@ test_expect_success "submodule.recurse option triggers recursive fetch" ' git -c submodule.recurse fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "fetch --recurse-submodules -j2 has the same output behaviour" '
@@ -97,7 +122,7 @@ test_expect_success "fetch --recurse-submodules -j2 has the same output behaviou GIT_TRACE="$TRASH_DIRECTORY/trace.out" git fetch --recurse-submodules -j2 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err && + verify_fetch_result actual.err && grep "2 tasks" trace.out '
@@ -127,7 +152,7 @@ test_expect_success "using fetchRecurseSubmodules=true in .gitmodules recurses i git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--no-recurse-submodules overrides .gitmodules config" '
@@ -158,7 +183,7 @@ test_expect_success "--recurse-submodules overrides fetchRecurseSubmodules setti git config --unset submodule.submodule.fetchRecurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--quiet propagates to submodules" '
@@ -186,7 +211,7 @@ test_expect_success "--dry-run propagates to submodules" ' git fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "Without --dry-run propagates to submodules" '
@@ -195,7 +220,7 @@ test_expect_success "Without --dry-run propagates to submodules" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "recurseSubmodules=true propagates into submodules" '
@@ -206,7 +231,7 @@ test_expect_success "recurseSubmodules=true propagates into submodules" ' git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--recurse-submodules overrides config in submodule" '
@@ -220,7 +245,7 @@ test_expect_success "--recurse-submodules overrides config in submodule" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--no-recurse-submodules overrides config setting" '
@@ -253,14 +278,14 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.sub && - echo " $head1..$head2 super -> origin/super" >>expect.err.sub && - head -3 expect.err >> expect.err.sub && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && - test_cmp expect.err.sub actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -271,14 +296,16 @@ test_expect_success "Recursion doesn't happen when new superproject commits don' git add file && git commit -m "new file" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.file && - echo " $head1..$head2 super -> origin/super" >> expect.err.file && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + rm expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err.file actual.err + verify_fetch_result actual.err ' test_expect_success "Recursion picks up config in submodule" '
@@ -295,9 +322,8 @@ test_expect_success "Recursion picks up config in submodule" ' git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.sub && - echo " $head1..$head2 super -> origin/super" >> expect.err.sub && - cat expect.err >> expect.err.sub && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && ( cd downstream && git fetch >../actual.out 2>../actual.err &&
@@ -306,7 +332,7 @@ test_expect_success "Recursion picks up config in submodule" ' git config --unset fetch.recurseSubmodules ) ) && - test_cmp expect.err.sub actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -331,15 +357,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >> expect.err.2 && - cat expect.err.sub >> expect.err.2 && - tail -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && - test_cmp expect.err.2 actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -375,11 +399,8 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - tail -3 expect.err > expect.err.deepsub && - echo "From $pwd/." > expect.err && - echo " $head1..$head2 super -> origin/super" >>expect.err && - cat expect.err.sub >> expect.err && - cat expect.err.deepsub >> expect.err && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -395,7 +416,7 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess ) ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" '
@@ -405,14 +426,16 @@ test_expect_success "'--recurse-submodules=on-demand' stops when no new submodul git add file && git commit -m "new file" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.file && - echo " $head1..$head2 super -> origin/super" >> expect.err.file && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + rm expect.err.deep && ( cd downstream && git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err.file actual.err + verify_fetch_result actual.err ' test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config" '
@@ -426,9 +449,9 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git config fetch.recurseSubmodules on-demand &&
@@ -440,7 +463,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config git config --unset fetch.recurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err + verify_fetch_result actual.err ' test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules" '
@@ -454,9 +477,9 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git config submodule.submodule.fetchRecurseSubmodules on-demand &&
@@ -468,7 +491,7 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override git config --unset submodule.submodule.fetchRecurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err + verify_fetch_result actual.err ' test_expect_success "don't fetch submodule when newly recorded commits are already present" '
@@ -480,14 +503,17 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea git add submodule && git commit -m "submodule rewound" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err && - echo " $head1..$head2 super -> origin/super" >> expect.err && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + # This file does not exist, but rm -f for readability + rm -f expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err && + verify_fetch_result actual.err && ( cd submodule && git checkout -q sub
@@ -505,9 +531,9 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git rm .gitmodules && git commit -m "new submodule without .gitmodules" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." >expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >>expect.err.2 && + echo "From $pwd/." >expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && rm .gitmodules &&
@@ -523,7 +549,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git reset --hard ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err && + verify_fetch_result actual.err && git checkout HEAD^ -- .gitmodules && git add .gitmodules && git commit -m "new submodule restored .gitmodules"
Signature · 1 line
2.33.GIT
PATCH 5/8 t5526: use grep to assert on fetches
In reply to Glen ChooIn a previous commit, we replaced test_cmp invocations with verify_fetch_result(). Finish the process of removing test_cmp by using grep in verify_fetch_result() instead.
This makes the tests less sensitive to changes because, instead of checking the whole stderr, we only grep for the lines of the form
* "<old-head>..<new-head>\s+branch\s+-> origin/branch"
* "Fetching submodule <submodule-path>" (if fetching a submodule)
when we expect the repo to have fetched. If we expect the repo to not have fetched, grep to make sure the lines are absent. Also, simplify the assertions by using grep patterns that match only the relevant pieces of information, e.g. <old-head> is irrelevant because we only want to know if the fetch was performed, so we don't need to know where the branch was before the fetch.
--- t/t5526-fetch-submodules.sh | 131 +++++++++++++----------------------- 1 file changed, 48 insertions(+), 83 deletions(-)
t/t5526-fetch-submodules.sh +48 -83
@@ -20,49 +20,52 @@ pwd=$(pwd) add_upstream_commit() { ( cd submodule && - head1=$(git rev-parse --short HEAD) && echo new >> subfile && test_tick && git add subfile && git commit -m new subfile && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && ( cd deepsubmodule && - head1=$(git rev-parse --short HEAD) && echo new >> deepsubfile && test_tick && git add deepsubfile && git commit -m new deepsubfile && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep - echo "From $pwd/deepsubmodule" >> ../expect.err.deep && - echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep + git rev-parse --short HEAD >../deephead ) } # Verifies that the expected repositories were fetched. This is done by -# concatenating the files expect.err.[super|sub|deep] in the correct -# order and comparing it to the actual stderr. +# checking that the branches of [super|sub|deep] were updated to +# [super|sub|deep]head if the corresponding file exists. # -# If a repo should not be fetched in the test, its corresponding -# expect.err file should be rm-ed. +# If the [super|sub|deep] head file does not exist, this verifies that +# the corresponding repo was not fetched. Thus, if a repo should not be +# fetched in the test, its corresponding head file should be +# rm-ed. verify_fetch_result() { ACTUAL_ERR=$1 && - rm -f expect.err.combined && - if [ -f expect.err.super ]; then - cat expect.err.super >>expect.err.combined + # Each grep pattern is guaranteed to match the correct repo + # because each repo uses a different name for their branch i.e. + # "super", "sub" and "deep". + if [ -f superhead ]; then + grep -E "\.\.$(cat superhead)\s+super\s+-> origin/super" $ACTUAL_ERR + else + ! grep "super" $ACTUAL_ERR fi && - if [ -f expect.err.sub ]; then - cat expect.err.sub >>expect.err.combined + if [ -f subhead ]; then + grep "Fetching submodule submodule" $ACTUAL_ERR && + grep -E "\.\.$(cat subhead)\s+sub\s+-> origin/sub" $ACTUAL_ERR + else + ! grep "Fetching submodule submodule" $ACTUAL_ERR fi && - if [ -f expect.err.deep ]; then - cat expect.err.deep >>expect.err.combined - fi && - test_cmp expect.err.combined $ACTUAL_ERR + if [ -f deephead ]; then + grep "Fetching submodule submodule/subdir/deepsubmodule" $ACTUAL_ERR && + grep -E "\.\.$(cat deephead)\s+deep\s+-> origin/deep" $ACTUAL_ERR + else + ! grep "Fetching submodule submodule/subdir/deepsubmodule" $ACTUAL_ERR + fi } test_expect_success setup '
@@ -274,13 +277,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in ' test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm deephead && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -291,15 +291,12 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" test_expect_success "Recursion doesn't happen when new superproject commits don't change any submodules" ' add_upstream_commit && - head1=$(git rev-parse --short HEAD) && echo a > file && git add file && git commit -m "new file" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && - rm expect.err.sub && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm subhead && + rm deephead && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -318,12 +315,9 @@ test_expect_success "Recursion picks up config in submodule" ' ) ) && add_upstream_commit && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git fetch >../actual.out 2>../actual.err &&
@@ -345,20 +339,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -376,13 +363,9 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo Fetching submodule submodule > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && ( cd downstream &&
@@ -395,12 +378,9 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne ' test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -421,15 +401,12 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" ' add_upstream_commit && - head1=$(git rev-parse --short HEAD) && echo a >> file && git add file && git commit -m "new file" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && - rm expect.err.sub && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm subhead && + rm deephead && ( cd downstream && git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err
@@ -445,13 +422,10 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config ) && add_upstream_commit && git config --global fetch.recurseSubmodules false && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm deephead && ( cd downstream && git config fetch.recurseSubmodules on-demand &&
@@ -473,13 +447,10 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override ) && add_upstream_commit && git config fetch.recurseSubmodules false && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm deephead && ( cd downstream && git config submodule.submodule.fetchRecurseSubmodules on-demand &&
@@ -499,15 +470,12 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea cd submodule && git checkout -q HEAD^^ ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "submodule rewound" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && - rm expect.err.sub && + git rev-parse --short HEAD >superhead && + rm subhead && # This file does not exist, but rm -f for readability - rm -f expect.err.deep && + rm -f deephead && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -526,14 +494,11 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git fetch --recurse-submodules ) && add_upstream_commit && - head1=$(git rev-parse --short HEAD) && git add submodule && git rm .gitmodules && git commit -m "new submodule without .gitmodules" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." >expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm deephead && ( cd downstream && rm .gitmodules &&
Signature · 1 line
2.33.GIT
Re: [PATCH 5/8] t5526: use grep to assert on fetches
Quoted 3 lines · Glen Choo
In a previous commit, we replaced
test_cmpinvocations withverify_fetch_result(). Finish the process of removingtest_cmpby using grep inverify_fetch_result()instead.
Thanks; this and the previous patch look like a better scheme for testing.
PATCH 6/8 submodule: extract get_fetch_task()
In reply to Glen Chooget_next_submodule() configures the parallel submodule fetch by performing two functions:
* iterate the index to find submodules
* configure the child processes to fetch the submodules found in the previous step
Extract the index iterating code into an iterator function, get_fetch_task(), so that get_next_submodule() is agnostic of how to find submodules. This prepares for a subsequent commit will teach the fetch machinery to also iterate through the list of changed submodules (in addition to the index).
--- submodule.c | 75 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 43 insertions(+), 32 deletions(-)
submodule.c +43 -32
@@ -1480,14 +1480,12 @@ static struct repository *get_submodule_repo_for(struct repository *r,return ret; } -static int get_next_submodule (struct child_process *cp, -struct strbuf *err,void *data,void **task_cb) +static struct fetch_task * +get_fetch_task (struct submodule_parallel_fetch *spf, +const char **default_argv,struct strbuf *err) { -struct submodule_parallel_fetch *spf = data; -for (; spf->count < spf->r->index->cache_nr; spf->count++) {const struct cache_entry *ce = spf->r->index->cache[spf->count]; -const char *default_argv;struct fetch_task *task;if (!S_ISGITLINK (ce->ce_mode))
@@ -1507,41 +1505,17 @@ static int get_next_submodule(struct child_process *cp, &spf->changed_submodule_names, task->sub->name)) continue; - default_argv ="on-demand" ; + *default_argv ="on-demand" ;break ;case RECURSE_SUBMODULES_ON : - default_argv ="yes" ; + *default_argv ="yes" ;break ;case RECURSE_SUBMODULES_OFF :continue ; } task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ()); -if (task->repo) { -struct strbuf submodule_prefix =STRBUF_INIT ; -child_process_init (cp); - cp->dir = task->repo->gitdir; -prepare_submodule_repo_env_in_gitdir (&cp->env_array); - cp->git_cmd =1 ; -if (!spf->quiet) -strbuf_addf (err,_ ("Fetching submodule %s%s\n" ), - spf->prefix, ce->name); -strvec_init (&cp->args); -strvec_pushv (&cp->args, spf->args.v); -strvec_push (&cp->args, default_argv); -strvec_push (&cp->args,"--submodule-prefix" ); - -strbuf_addf (&submodule_prefix,"%s%s/" , - spf->prefix, - task->sub->path); -strvec_push (&cp->args, submodule_prefix.buf); - - spf->count++; - *task_cb = task; - -strbuf_release (&submodule_prefix); -return 1 ; - }else { +if (!task->repo) {struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);
@@ -1562,7 +1536,44 @@ static int get_next_submodule(struct child_process *cp, ce->name); }strbuf_release (&empty_submodule_path); +continue ; } +if (!spf->quiet) +strbuf_addf (err,_ ("Fetching submodule %s%s\n" ), + spf->prefix, ce->name); + + spf->count++; +return task; + } +return NULL ; +} + +static int get_next_submodule (struct child_process *cp,struct strbuf *err, +void *data,void **task_cb) +{ +struct submodule_parallel_fetch *spf = data; +const char *default_argv =NULL ; +struct fetch_task *task =get_fetch_task (spf, &default_argv, err); + +if (task) { +struct strbuf submodule_prefix =STRBUF_INIT ; + +child_process_init (cp); + cp->dir = task->repo->gitdir; +prepare_submodule_repo_env_in_gitdir (&cp->env_array); + cp->git_cmd =1 ; +strvec_init (&cp->args); +strvec_pushv (&cp->args, spf->args.v); +strvec_push (&cp->args, default_argv); +strvec_push (&cp->args,"--submodule-prefix" ); + +strbuf_addf (&submodule_prefix,"%s%s/" , spf->prefix, + task->sub->path); +strvec_push (&cp->args, submodule_prefix.buf); + *task_cb = task; + +strbuf_release (&submodule_prefix); +return 1 ; }if (spf->oid_fetch_tasks_nr) {
Signature · 1 line
2.33.GIT
Re: [PATCH 6/8] submodule: extract get_fetch_task()
Quoted 5 lines · Glen Choo
Extract the index iterating code into an iterator function,
get_fetch_task(), so thatget_next_submodule()is agnostic of how to find submodules. This prepares for a subsequent commit will teach the fetch machinery to also iterate through the list of changed submodules (in addition to the index).
The transformation looks correct, but there are several things that would have made it much easier to review.
Quoted 1 line
(quoted diff) +0
-0@@ -1507,41 +1505,17 @@ static int get_next_submodule(struct child_process *cp,
[snip]
Quoted 34 lines
submodule.c +2
-25-if (task->repo) { -struct strbuf submodule_prefix =STRBUF_INIT ; -child_process_init (cp); - cp->dir = task->repo->gitdir; -prepare_submodule_repo_env_in_gitdir (&cp->env_array); - cp->git_cmd =1 ; -if (!spf->quiet) -strbuf_addf (err,_ ("Fetching submodule %s%s\n" ), - spf->prefix, ce->name); -strvec_init (&cp->args); -strvec_pushv (&cp->args, spf->args.v); -strvec_push (&cp->args, default_argv); -strvec_push (&cp->args,"--submodule-prefix" ); - -strbuf_addf (&submodule_prefix,"%s%s/" , - spf->prefix, - task->sub->path); -strvec_push (&cp->args, submodule_prefix.buf); - - spf->count++; - *task_cb = task; - -strbuf_release (&submodule_prefix); -return 1 ; - }else { +if (!task->repo) {struct strbuf empty_submodule_path =STRBUF_INIT ;fetch_task_release (task);@@ -1562,7 +1536,44 @@ static int get_next_submodule(struct child_process *cp, ce->name); }strbuf_release (&empty_submodule_path); +continue ;
submodule.c static int get_next_submodule(struct child_process *cp, · line 1549
Quoted 10 lines
submodule.c +9
-0} +if (!spf->quiet) +strbuf_addf (err,_ ("Fetching submodule %s%s\n" ), + spf->prefix, ce->name); + + spf->count++; +return task; + } +return NULL ; +}
You could have retained the "if (task->repo) { } else { }" structure instead of adding a "continue;".
Also, the "if (!spf->quiet)" could be moved into get_next_submodule(), but I see why it's there (it needs ce->name, which we otherwise don't need), so leaving it where it is in this patch is fine too.
submodule.c static int get_next_submodule(struct child_process *cp, · line 1571
Quoted 2 lines
submodule.c +2
-0+strbuf_addf (&submodule_prefix,"%s%s/" , spf->prefix, + task->sub->path);
It would have been clearer if this line wasn't rewrapped.
As a reviewer, sometimes it's hard to make the tradeoff between asking the author to make formatting changes versus leaving it alone because the reviewer has already inspected the changes and decided that any errors are only in formatting, not in logic. In this case, though, because there is only one more patch in the series and the formatting change I'm suggesting here won't really affect it that much, I think it's better if you make the formatting change for the benefit of other reviewers who are currently reviewing this patch set, and anyone looking at this commit in the future.
PATCH 7/8 fetch: fetch unpopulated, changed submodules
In reply to Glen Choo"git fetch --recurse-submodules" only considers populated submodules (i.e. submodules that can be found by iterating the index), which makes "git fetch" behave differently based on which commit is checked out. As a result, even if the user has initialized all submodules correctly, they may not fetch the necessary submodule commits, and commands like "git checkout --recurse-submodules" might fail.
Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated (this is in addition to the current behavior of fetching populated submodules).
Since a submodule may be encountered multiple times (via the list of populated submodules or via the list of changed submodules), maintain a list of seen submodules to avoid fetching a submodule more than once.
---submodule.c has a seemingly-unrelated change that teaches the "find changed submodules" rev walk to call is_repository_shallow(). This fixes what I believe is a legitimate bug - the rev walk would fail on a shallow repo.
Our test suite did not catch this prior to this commit because we skip the rev walk if .gitmodules is not found, and thus the test suite did not attempt the rev walk on a shallow clone. After this commit, we always attempt to find changed submodules (regardless of whether there is a .gitmodules file), and the test suite noticed the bug.
Documentation/fetch-options.txt | 26 ++-- Documentation/git-fetch.txt | 10 +- submodule.c | 101 +++++++++++++-- t/t5526-fetch-submodules.sh | 217 ++++++++++++++++++++++++++++++++ 4 files changed, 328 insertions(+), 26 deletions(-)
Documentation/fetch-options.txt +17 -9
@@ -185,15 +185,23 @@ endif::git-pull[] ifndef::git-pull[] --recurse-submodules[=yes|on-demand|no]:: This option controls if and under what conditions new commits of - populated submodules should be fetched too. It can be used as a - boolean option to completely disable recursion when set to 'no' or to - unconditionally recurse into all populated submodules when set to - 'yes', which is the default when this option is used without any - value. Use 'on-demand' to only recurse into a populated submodule - when the superproject retrieves a commit that updates the submodule's - reference to a commit that isn't already in the local submodule - clone. By default, 'on-demand' is used, unless - `fetch.recurseSubmodules` is set (see linkgit:git-config[1]). + submodules should be fetched too. When recursing through submodules, + `git fetch` always attempts to fetch "changed" submodules, that is, a + submodule that has commits that are referenced by a newly fetched + superproject commit but are missing in the local submodule clone. A + changed submodule can be fetched as long as it is present locally e.g. + in `$GIT_DIR/modules/` (see linkgit:gitsubmodules[7]); if the upstream + adds a new submodule, that submodule cannot be fetched until it is + cloned e.g. by `git submodule update`. ++ +When set to 'on-demand', only changed submodules are fetched. When set +to 'yes', all populated submodules are fetched and submodules that are +both unpopulated and changed are fetched. When set to 'no', submodules +are never fetched. ++ +When unspecified, this uses the value of `fetch.recurseSubmodules` if it +is set (see linkgit:git-config[1]), defaulting to 'on-demand' if unset. +When this option is used without any value, it defaults to 'yes'. endif::git-pull[] -j::
Documentation/git-fetch.txt +4 -6
@@ -287,12 +287,10 @@ include::transfer-data-leaks.txt[] BUGS ---- -Using --recurse-submodules can only fetch new commits in already checked -out submodules right now. When e.g. upstream added a new submodule in the -just fetched commits of the superproject the submodule itself cannot be -fetched, making it impossible to check out that submodule later without -having to do a fetch again. This is expected to be fixed in a future Git -version. +Using --recurse-submodules can only fetch new commits in submodules that are +present locally e.g. in `$GIT_DIR/modules/`. If the upstream adds a new +submodule, that submodule cannot be fetched until it is cloned e.g. by `git +submodule update`. This is expected to be fixed in a future Git version. SEE ALSO --------
submodule.c +90 -11
168 lines not on this page · raw message
t/t5526-fetch-submodules.sh +217 -0
223 lines not on this page · raw message
Signature · 1 line
2.33.GIT
Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules
submodule.c static void calculate_changed_submodule_paths(struct repository *r, · line 1279
Quoted 8 lines · Glen Choo
submodule.c +0
-4@@ -1273,10 +1277,6 @@ static void calculate_changed_submodule_paths(struct repository *r,struct strvec argv =STRVEC_INIT ;struct string_list_item *name; -/* No need to check if there are no submodules configured */ -if (!submodule_from_path (r,NULL ,NULL )) -return ; -
It looks to me that this hunk reverts 18322bad (fetch: skip on-demand checking when no submodules are configured, 2011-09-09), which tried to avoid high cost computation when we know there is no submodule. Intended? Perhaps it should be replaced with an equivalent check that (1) still says "we do care about submodules" even if the current checkout has no submodules (i.e. ls-files shows no gitlinks), but (2) says "no, there is nothing interesting" when $GIT_COMMON_DIR/modules/ is empty or some other cheap check we can use?
Quoted 28 lines
submodule.c +14
-2+get_fetch_task_from_index(struct submodule_parallel_fetch *spf, + const char **default_argv, struct strbuf *err) { -for (; spf->count < spf->r->index->cache_nr; spf->count++) { -const struct cache_entry *ce = spf->r->index->cache[spf->count]; + for (; spf->index_count < spf->r->index->cache_nr; spf->index_count++) { +const struct cache_entry *ce = + spf->r->index->cache[spf->index_count];struct fetch_task *task;if (!S_ISGITLINK (ce->ce_mode))@@ -1495,6 +1499,15 @@ get_fetch_task(struct submodule_parallel_fetch *spf,if (!task)continue ; +/* +* We might have already considered this submodule +* because we saw it when iterating the changed +* submodule names. +*/ +if (string_list_lookup (&spf->seen_submodule_names, + task->sub->name)) +continue ; +switch (get_fetch_recurse_config (task->sub, spf)) {default :
submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, · line 1562
Quoted 9 lines
submodule.c +5
-1@@ -1542,7 +1555,69 @@ get_fetch_task(struct submodule_parallel_fetch *spf,strbuf_addf (err,_ ("Fetching submodule %s%s\n" ), spf->prefix, ce->name); - spf->count++; + spf->index_count++; +return task; + } +return NULL ; +}
Sorry, but I am confused. If we are gathering which submodules to fetch from the changes to gitlinks in the range of superproject changes, why do we even need to scan the index (i.e. the current checkout in the superproject) to begin with? If it was changed, we'd know get_fetch_task_from_changed() would take care of it, and if there was no change to the submodule between the superproject's commits before and after the fetch, there is nothing gained from fetching in the submodules, no?
Quoted 6 lines
submodule.c +4
-0+static struct fetch_task * +get_fetch_task_from_changed (struct submodule_parallel_fetch *spf, +const char **default_argv,struct strbuf *err) +{
submodule.c static int get_next_submodule(struct child_process *cp, struct strbuf *err, · line 1634
Quoted 8 lines
submodule.c +4
-1@@ -1553,7 +1628,10 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err, {struct submodule_parallel_fetch *spf = data;const char *default_argv =NULL ; -struct fetch_task *task =get_fetch_task (spf, &default_argv, err); +struct fetch_task *task = +get_fetch_task_from_index (spf, &default_argv, err); +if (!task) + task =get_fetch_task_from_changed (spf, &default_argv, err);
Hmph, intersting. So if "from index" grabbed some submodules already, then the "from the changes in the superproject, we know these submodules need refreshing" is not happen at all? I am afraid that I am still not following this...
Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules
Quoted 5 lines · Junio C Hamano
Quoted 4 lines
submodule.c +4
-0+static struct fetch_task * +get_fetch_task_from_changed (struct submodule_parallel_fetch *spf, +const char **default_argv,struct strbuf *err) +{
Quoted 14 lines
submodule.c static int get_next_submodule(struct child_process *cp, struct strbuf *err, · line 1634
Quoted 9 lines
submodule.c +4
-1@@ -1553,7 +1628,10 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err, {struct submodule_parallel_fetch *spf = data;const char *default_argv =NULL ; -struct fetch_task *task =get_fetch_task (spf, &default_argv, err); +struct fetch_task *task = +get_fetch_task_from_index (spf, &default_argv, err); +if (!task) + task =get_fetch_task_from_changed (spf, &default_argv, err);Hmph, intersting. So if "from index" grabbed some submodules already, then the "from the changes in the superproject, we know these submodules need refreshing" is not happen at all? I am afraid that I am still not following this...
Hm, perhaps the following will help:
- get_next_submodule() is an iterator, specifically, it is a get_next_task_fn passed to run_processes_parallel_tr2(). It gets called until it is exhausted.
- Since get_next_submodule() is an iterator, I've implemented get_fetch_task_from_index() and get_fetch_task_from_changed() as iterators (they return NULL when they are exhausted).
So in practice:
- We repeatedly call get_next_submodule(), which tries to get a fetch task by calling the get_fetch_task_* functions.
- If get_fetch_task_from_index() returns non-NULL, get_next_submodule() uses that fetch task.
- Eventually, we will have considered every submodule in the index. At that point, get_fetch_task_from_index() is exhausted and always returns NULL.
- Since get_fetch_task_from_index() returns NULL, get_next_submodule() now gets its fetch tasks from get_fetch_task_from_changed().
- Eventually, we will also have considered every changed submodule, and get_fetch_task_from_changed() is exhausted.
- get_next_submodule() has now been exhausted and we are done.
As for the other questions, I'll dig a bit deeper before getting back to you with answers.
Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules
Quoted 22 lines · Glen Choo
Quoted 5 lines · Junio C Hamano
Quoted 4 lines
submodule.c +4
-0+static struct fetch_task * +get_fetch_task_from_changed (struct submodule_parallel_fetch *spf, +const char **default_argv,struct strbuf *err) +{Quoted 14 lines
submodule.c static int get_next_submodule(struct child_process *cp, struct strbuf *err, · line 1634
Quoted 9 lines
submodule.c +4
-1@@ -1553,7 +1628,10 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err, {struct submodule_parallel_fetch *spf = data;const char *default_argv =NULL ; -struct fetch_task *task =get_fetch_task (spf, &default_argv, err); +struct fetch_task *task = +get_fetch_task_from_index (spf, &default_argv, err); +if (!task) + task =get_fetch_task_from_changed (spf, &default_argv, err);Hmph, intersting. So if "from index" grabbed some submodules already, then the "from the changes in the superproject, we know these submodules need refreshing" is not happen at all? I am afraid that I am still not following this...
Quoted 5 lines
Hm, perhaps the following will help:
-
get_next_submodule()is an iterator, specifically, it is aget_next_task_fnpassed torun_processes_parallel_tr2(). It gets called until it is exhausted.
Ahh, yeah, I totally forgot how we designed these things to work.
Even though these functions have a loop, (1) they start iterating at the point where they left off in the last call, and (2) they return as soon as they find the first item in the loop, which should have stood out as a typical generator pattern, but somehow I missed these signs.
Quoted 3 lines
So in practice:
...
-get_next_submodule()has now been exhausted and we are done.
But my original question (based on my misunderstanding that a single call to these would grab all submodules that needs fetching by inspecting either the index or the history) still stands, doesn't it?
Presumably the "history scan" part is because we assume that we already had all the necessary submodule commits to check out any superproject commits before this recursive fetch started. That is the reason why we do not scan the history behind the "old tips". We inspect only the history newer than them, leading to the "new tips", and try to grab all submodule commits that newly appear, to ensure that we can check out all the superproject commits we just obtained and have no missing submodule commits necessary to do so. Combined with the assumption on the state before this fetch that we had all necessary submodule commits to check out the superproject commits up to "old tips", we maintain the invariant that we can check out any superproject commits recursively, no?
If we are doing so, especially with this series where we do the "history scan" to complete submodule commits necessary for all new commits in the superproject, regardless of the branch being checked out in the superproject, why do we still need to scan the index to ensure that the current checkout can recurse down the submodule without hitting a missing commit?
The only case the "index scan" may make a difference is when the assumption, the invariant that we can check out any superproject commits recursively, did not hold before we started the fetch, no?
Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules
In reply to Glen ChooQuoted 10 lines · Glen Choo
submodule.chas a seemingly-unrelated change that teaches the "find changed submodules" rev walk to callis_repository_shallow(). This fixes what I believe is a legitimate bug - the rev walk would fail on a shallow repo.Our test suite did not catch this prior to this commit because we skip the rev walk if .gitmodules is not found, and thus the test suite did not attempt the rev walk on a shallow clone. After this commit, we always attempt to find changed submodules (regardless of whether there is a .gitmodules file), and the test suite noticed the bug.
Is this bug present without the other code introduced in this patch? If yes, it's better to put the bugfix in a separate patch with a test that would have failed but now passes.
Some more high-level comments:
submodule.c static void calculate_changed_submodule_paths(struct repository *r, · line 1278
Quoted 7 lines
submodule.c +0
-3@@ -1273,10 +1277,6 @@ static void calculate_changed_submodule_paths(struct repository *r,struct strvec argv =STRVEC_INIT ;struct string_list_item *name; -/* No need to check if there are no submodules configured */ -if (!submodule_from_path (r,NULL ,NULL )) -return ;
I think this is removed because "no submodules configured" here actually means "no submodules configured in the index", but submodules may be configured in the superproject commits we're fetching.
I wonder if this should be mentioned in the commit message, but I'm OK either way.
submodule.c int submodule_touches_in_range(struct repository *r, · line 1351
Quoted 4 lines
submodule.c +2
-1struct submodule_parallel_fetch { -int count; +int index_count; +int changed_count;
Here (and elsewhere) we're checking both the index and the superproject commits for .gitmodules. Do we still need to check the index?
Quoted 12 lines
submodule.c +8
-0@@ -1495,6 +1499,15 @@ get_fetch_task(struct submodule_parallel_fetch *spf,if (!task)continue ; +/* +* We might have already considered this submodule +* because we saw it when iterating the changed +* submodule names. +*/ +if (string_list_lookup (&spf->seen_submodule_names, + task->sub->name)) +continue ;
[snip]
submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, · line 1580
Quoted 6 lines
submodule.c +6
-0+/* +* We might have already considered this submodule +* because we saw it in the index. +*/ +if (string_list_lookup (&spf->seen_submodule_names, item.string)) +continue ;
Hmm...it's odd that the checks happen in both places, when theoretically we would do one after the other, so this check would only need to be in one place. Maybe this is because of how we had to implement it (looping over everything every time when we get the next fetch task) but if it's easy to avoid, that would be great.
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 408
Quoted 7 lines
t/t5526-fetch-submodules.sh +7
-0+# Cleans up after tests that checkout branches other than the main ones +# in the tests. +checkout_main_branches() { + git -C downstream checkout --recurse-submodules super && + git -C downstream/submodule checkout --recurse-submodules sub && + git -C downstream/submodule/subdir/deepsubmodule checkout --recurse-submodules deep +}
If we need to clean up in this way, I think it's better if we store a pristine copy somewhere (e.g. pristine-downstream), delete downstream, and copy it over when we need to.
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 422
Quoted 13 lines
t/t5526-fetch-submodules.sh +13
-0+# Test that we can fetch submodules in other branches by running fetch +# in a branch that has no submodules. +test_expect_success 'setup downstream branch without submodules' ' + ( + cd downstream && + git checkout --recurse-submodules -b no-submodules && + rm .gitmodules && + git rm submodule && + git add .gitmodules && + git commit -m "no submodules" && + git checkout --recurse-submodules super + ) +'
The tip of the branch indeed doesn't have any submodules, but when fetching this branch, we might end up fetching some of the tip's ancestors (depending on the repo we're fetching into), which do have submodules. If we need a branch without submodules, I think that all ancestors should not have submodules too.
That might be an argument for creating our own downstream and upstream repos instead of reusing the existing ones.
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 433
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+test_expect_success "'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules" ' + test_when_finished "checkout_main_branches" && + git -C downstream fetch --recurse-submodules && + # Create new superproject commit with updated submodules + add_upstream_commit && + ( + cd submodule && + ( + cd subdir/deepsubmodule && + git fetch &&
Hmm...I thought submodule/subdir/deepsubmodule is upstream. Why is it fetching?
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 447
Quoted 6 lines
t/t5526-fetch-submodules.sh +6
-0+ # Fetch the new superproject commit + ( + cd downstream && + git switch --recurse-submodules no-submodules && + git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err && + git checkout --recurse-submodules origin/super 2>../actual-checkout.err
This patch set is about fetching, so the checkout here seems odd. To verify that the fetch happened successfully, I think that we should obtain the hashes of the commits that we expect to be fetched from upstream, and then verify that they are present downstream.
Review
Quoted 2 lines
(quoted diff) +2
-0+ # Assert that we can checkout the superproject commit with --recurse-submodules + ! grep -E "error: Submodule .+ could not be updated" actual-checkout.err
Negative greps are error-prone, since they will also appear to work if the message was just misspelled. We should probably check that the expected commit is present instead.
Quoted 16 lines
t/t5526-fetch-submodules.sh +16
-0+# Test that we properly fetch the submodules in the index as well as +# submodules in other branches. +test_expect_success 'setup downstream branch with other submodule' ' + mkdir submodule2 && + ( + cd submodule2 && + git init && + echo sub2content >sub2file && + git add sub2file && + git commit -a -m new && + git branch -M sub2 + ) && + git checkout -b super-sub2-only && + git submodule add "$pwd/submodule2" submodule2 && + git commit -m "add sub2" && + git checkout super &&
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 561
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ ( + cd downstream && + git fetch --recurse-submodules origin && + git checkout super-sub2-only && + # Explicitly run "git submodule update" because sub2 is new + # and has not been cloned. + git submodule update --init && + git checkout --recurse-submodules super + ) +'
I couldn't see the submodule in the index to be fetched; maybe it's there somewhere but it's not obvious to me. Also, why do we need to run "git submodule update"? This patch set concerns itself with fetching existing submodules, not cloning new ones.
Quoted 1 line
+test_expect_success "'--recurse-submodules' should fetch submodule commits in changed submodules and the index" '
Same comment about where in the index is the submodule to be fetched.
Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules
Quoted 14 lines · Jonathan Tan
Quoted 12 lines
submodule.c +8
-0@@ -1495,6 +1499,15 @@ get_fetch_task(struct submodule_parallel_fetch *spf,if (!task)continue ; +/* +* We might have already considered this submodule +* because we saw it when iterating the changed +* submodule names. +*/ +if (string_list_lookup (&spf->seen_submodule_names, + task->sub->name)) +continue ;[snip]
Quoted 12 lines
submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, · line 1580
Quoted 6 lines
submodule.c +6
-0+/* +* We might have already considered this submodule +* because we saw it in the index. +*/ +if (string_list_lookup (&spf->seen_submodule_names, item.string)) +continue ;Hmm...it's odd that the checks happen in both places, when theoretically we would do one after the other, so this check would only need to be in one place. Maybe this is because of how we had to implement it (looping over everything every time when we get the next fetch task) but if it's easy to avoid, that would be great.
Yes, in order for the code to be correct, we only need this check once, but I chose to check twice for defensiveness. That is, we avoid creating implicit dependencies between the functions like "function A does not consider whether a submodule has already been fetched, so it must always be called before function B".
Perhaps there is another concern that overrides this? e.g. performance.
Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules
In reply to Jonathan TanQuoted 12 lines · Jonathan Tan
Quoted 10 lines · Glen Choo
submodule.chas a seemingly-unrelated change that teaches the "find changed submodules" rev walk to callis_repository_shallow(). This fixes what I believe is a legitimate bug - the rev walk would fail on a shallow repo.Our test suite did not catch this prior to this commit because we skip the rev walk if .gitmodules is not found, and thus the test suite did not attempt the rev walk on a shallow clone. After this commit, we always attempt to find changed submodules (regardless of whether there is a .gitmodules file), and the test suite noticed the bug.
Quoted 3 lines
Is this bug present without the other code introduced in this patch? If yes, it's better to put the bugfix in a separate patch with a test that would have failed but now passes.
Makes sense, I'll do so.
Quoted 12 lines
submodule.c static void calculate_changed_submodule_paths(struct repository *r, · line 1278
Quoted 7 lines
submodule.c +0
-3@@ -1273,10 +1277,6 @@ static void calculate_changed_submodule_paths(struct repository *r,struct strvec argv =STRVEC_INIT ;struct string_list_item *name; -/* No need to check if there are no submodules configured */ -if (!submodule_from_path (r,NULL ,NULL )) -return ;I think this is removed because "no submodules configured" here actually means "no submodules configured in the index", but submodules may be configured in the superproject commits we're fetching.
Quoted 2 lines
I wonder if this should be mentioned in the commit message, but I'm OK either way.
Yes, your interpretation is correct. Though, as Junio mentioned in <[email protected]>, I think we'd prefer to have _some kind_ of check, even though this one no longer makes sense.
Quoted 8 lines
submodule.c int submodule_touches_in_range(struct repository *r, · line 1351
Quoted 5 lines
submodule.c +2
-1struct submodule_parallel_fetch { -int count; +int index_count; +int changed_count;Here (and elsewhere) we're checking both the index and the superproject commits for .gitmodules. Do we still need to check the index?
Since this is a frequently asked question, I answered this elsewhere, namely <[email protected]>.
Quoted 11 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 408
Quoted 7 lines
t/t5526-fetch-submodules.sh +7
-0+# Cleans up after tests that checkout branches other than the main ones +# in the tests. +checkout_main_branches() { + git -C downstream checkout --recurse-submodules super && + git -C downstream/submodule checkout --recurse-submodules sub && + git -C downstream/submodule/subdir/deepsubmodule checkout --recurse-submodules deep +}If we need to clean up in this way, I think it's better if we store a pristine copy somewhere (e.g. pristine-downstream), delete downstream, and copy it over when we need to.
The need for cleanup isn't that big; this just checks out the right branches after we've checked out _other_ branches. If remove the checkout, we won't need this any more, and...
Quoted 11 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 447
Quoted 6 lines
t/t5526-fetch-submodules.sh +6
-0+ # Fetch the new superproject commit + ( + cd downstream && + git switch --recurse-submodules no-submodules && + git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err && + git checkout --recurse-submodules origin/super 2>../actual-checkout.errThis patch set is about fetching, so the checkout here seems odd. To verify that the fetch happened successfully, I think that we should obtain the hashes of the commits that we expect to be fetched from upstream, and then verify that they are present downstream.
IIUC this feedback correctly, the checkout is just an indirect way of checking if we have the commit, so it makes more sense to just check if we have the commit.
But explicitly checking for the commit (with "git cat-file -e" I assume?) is probably overkill - verify_fetch_result() already checks for this by grep-ing the output of "git fetch".
So I think it's ok to drop the checkout and not check for the commit (beyond verify_fetch_result()).
Quoted 20 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 422
Quoted 13 lines
t/t5526-fetch-submodules.sh +13
-0+# Test that we can fetch submodules in other branches by running fetch +# in a branch that has no submodules. +test_expect_success 'setup downstream branch without submodules' ' + ( + cd downstream && + git checkout --recurse-submodules -b no-submodules && + rm .gitmodules && + git rm submodule && + git add .gitmodules && + git commit -m "no submodules" && + git checkout --recurse-submodules super + ) +'The tip of the branch indeed doesn't have any submodules, but when fetching this branch, we might end up fetching some of the tip's ancestors (depending on the repo we're fetching into), which do have submodules. If we need a branch without submodules, I think that all ancestors should not have submodules too.
Quoted 2 lines
That might be an argument for creating our own downstream and upstream repos instead of reusing the existing ones.
I think I just made a silly wording error, I meant a "commit" or "working tree state" without submodules, not a branch. The behavior I wanted to test is whether or not changed submodules are fetched in the absence of submodules and .gitmodules in the index/working tree.
Quoted 5 lines
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+test_expect_success "'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules" ' + test_when_finished "checkout_main_branches" && + git -C downstream fetch --recurse-submodules && + # Create new superproject commit with updated submodules + add_upstream_commit &&
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 433
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+ ( + cd submodule && + ( + cd subdir/deepsubmodule && + git fetch &&Hmm...I thought submodule/subdir/deepsubmodule is upstream. Why is it fetching?
Ah, deepsubmodule is a submodule in the "submodule/" repo, whose remote is in "deepsubmodule/":
test_expect_success setup '
mkdir deepsubmodule &&
(
cd deepsubmodule &&
git init &&
echo deepsubcontent > deepsubfile &&
git add deepsubfile &&
git commit -m new deepsubfile &&
git branch -M deep
) &&
mkdir submodule &&
(
cd submodule &&
git init &&
echo subcontent > subfile &&
git add subfile &&
git submodule add "$pwd/deepsubmodule" subdir/deepsubmodule &&
git commit -a -m new &&
git branch -M sub
)So we fetch in "submodule/subdir/deepsubmodule" to get a new deepsubmodule and (non-deep) submodule commit. Both of these commits are then used to construct a new superproject commit.
If this is too confusing, maybe I should try to make the test simpler.
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ # Assert that we can checkout the superproject commit with --recurse-submodules + ! grep -E "error: Submodule .+ could not be updated" actual-checkout.errNegative greps are error-prone, since they will also appear to work if the message was just misspelled. We should probably check that the expected commit is present instead.
That's a good point, I hadn't considered that.
Quoted 21 lines
Quoted 21 lines
t/t5526-fetch-submodules.sh +21
-0+# Test that we properly fetch the submodules in the index as well as +# submodules in other branches. +test_expect_success 'setup downstream branch with other submodule' ' + mkdir submodule2 && + ( + cd submodule2 && + git init && + echo sub2content >sub2file && + git add sub2file && + git commit -a -m new && + git branch -M sub2 + ) && + git checkout -b super-sub2-only && + git submodule add "$pwd/submodule2" submodule2 && + git commit -m "add sub2" && + git checkout super && + ( + cd downstream && + git fetch --recurse-submodules origin && + git checkout super-sub2-only && + # Explicitly run "git submodule update" because sub2 is new
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 561
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+ # and has not been cloned. + git submodule update --init && + git checkout --recurse-submodules super + ) +'I couldn't see the submodule in the index to be fetched; maybe it's there somewhere but it's not obvious to me.
If it helps, I've updated the description to:
# In downstream, init "submodule2", but do not check it out while
# fetching. This lets us assert that unpopulated submodules can be # fetched.
The 'submodules in the index' are the existing submodules ("submodule" and "submodule/subdir/deepsubmodule"), and the changed, unpopulated submodule is "submodule2".
Quoted 3 lines
Also, why do we need to run "git submodule update"? This patch set concerns itself with fetching existing submodules, not cloning new ones.
In this setup step, 'downstream' clones 'submodule2' using "git submodule update". So from the perspective of the following tests, 'submodule2' is an existing submodule. We could have cloned 'submodule2' in an earlier setup step, but it wouldn't have been needed until these tests.
Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules
In reply to Glen ChooQuoted 3 lines · Glen Choo
Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated (this is in addition to the current behavior of fetching populated submodules).
Reviewers (and myself) have rightfully asked why "git fetch" should continue to bother looking for submodules in the index if it already fetches all of the changed submodules. The reasons for this are twofold:
1. The primary reason is that --recurse-submodules, aka --recurse-submodules=yes does an unconditional fetch in each of the submodules regardless of whether they have been changed by a superproject commit. This is the behavior of e.g. from t/t5526-fetch-submodules.sh:101:
test_expect_success "fetch --recurse-submodules recurses into submodules" '
# Creates commits in the submodules but NOT the superproject
add_upstream_commit &&
(
cd downstream &&
git fetch --recurse-submodules >../actual.out 2>../actual.err
) &&
test_must_be_empty actual.out &&
# Assert that the new submodule commits have been fetched and that
# no superproject commit was fetched
verify_fetch_result actual.err
'Thus, we continue to check the index to implement this unconditional fetching behavior.
2. In the --recurse-submodule=on-demand case, it can be correct to ignore the index because "on-demand" only requires us to fetch changed submodules. But in the event that a submodule is both changed and populated, we may prefer to read the index instead of the superproject commit, because the contents of the index are more obvious and more actionable to the user.
For example, we print the path of the submodule when attempting to fetch a submodule for debugging purposes:
- For a populated submodule, we print "Fetching submodule <path>"
- For an unpopulated submodule, we print "Fetching submodule <path> at commit foo"
Presumably, the user would prefer to see the "populated submodule" message because it's easier to work with, e.g. "git -C <path> <fix-the-problem>" instead of "git checkout --recurse-submodules <commit-with-submodule> && git <fix-the-problem>".
The latter is not a sufficient reason to read the index and then the changed submodule list (because we could try to read the changed submodule configuration from index), but since we need to support --recurse-submodules=yes, this implementation is convenient for achieving both goals.
PATCH 8/8 submodule: fix bug and remove add_submodule_odb()
In reply to Glen Chooadd_submodule_odb() is a hack - it adds a submodule's odb as an alternate, allowing the submodule's objects to be read via the_repository. Its last caller is submodule_has_commits(), which calls add_submodule_odb() to prepare for check_has_commit(). This used to be necessary because check_has_commit() used the_repository's odb, but this is longer true as of 13a2f620b2 (submodule: pass repo to check_has_commit(), 2021-10-08).
Removing add_submodule_odb() reveals a bug in check_has_commit(), where check_has_commit() will segfault if the submodule is missing (e.g. the user has not init-ed the submodule). This happens because the submodule's struct repository cannot be initialized, but check_has_commit() tries to cleanup the uninitialized struct anyway. This was masked by add_submodule_odb(), because add_submodule_odb() fails when the submodule is missing, causing the caller to return early and avoid calling check_has_commit().
Fix the bug and remove the call to add_submodule_odb(). Since add_submodule_odb() has no more callers, remove it too.
Note that submodule odbs can still by added as alternates via add_submodule_odb_by_path().
---
This bug only exists because we can't call repo_clear() twice on the same struct repository. So instead of just fixing this site, an alternative (and maybe better) fix would be to fix repo_clear(). If others think that's a good idea, I'll do that instead.
submodule.c | 35 ++--------------------------------- submodule.h | 9 ++++----- 2 files changed, 6 insertions(+), 38 deletions(-)
submodule.c +2 -33
@@ -168,26 +168,6 @@ void stage_updated_gitmodules(struct index_state *istate)static struct string_list added_submodule_odb_paths =STRING_LIST_INIT_NODUP ; -/* TODO: remove this function, use repo_submodule_init instead. */ -int add_submodule_odb (const char *path) -{ -struct strbuf objects_directory =STRBUF_INIT ; -int ret =0 ; - - ret =strbuf_git_path_submodule (&objects_directory, path,"objects/" ); -if (ret) - goto done; -if (!is_directory (objects_directory.buf)) { - ret =-1 ; - goto done; - } -string_list_insert (&added_submodule_odb_paths, -strbuf_detach (&objects_directory,NULL )); -done: -strbuf_release (&objects_directory); -return ret; -} -void add_submodule_odb_by_path (const char *path) {string_list_insert (&added_submodule_odb_paths,xstrdup (path));
@@ -972,7 +952,8 @@ static int check_has_commit(const struct object_id *oid, void *data)if (repo_submodule_init (&subrepo, cb->repo, cb->path, cb->super_oid)) { cb->result =0 ; - goto cleanup; +/* subrepo failed to init, so don't clean it up. */ +return 0 ; } type =oid_object_info (&subrepo, oid,NULL );
@@ -1003,18 +984,6 @@ static int submodule_has_commits(struct repository *r, {struct has_commit_data has_commit = { r,1 , path, super_oid }; -/* -* Perform a cheap, but incorrect check for the existence of 'commits'. -* This is done by adding the submodule's object store to the in-core -* object store, and then querying for each commit's existence. If we -* do not have the commit object anywhere, there is no chance we have -* it in the object store of the correct submodule and have it -* reachable from a ref, so we can fail early without spawning rev-list -* which is expensive. -*/ -if (add_submodule_odb (path)) -return 0 ; -oid_array_for_each_unique (commits, check_has_commit, &has_commit);if (has_commit.result) {
submodule.h +4 -5
@@ -103,12 +103,11 @@ int submodule_uses_gitfile(const char *path);int bad_to_remove_submodule (const char *path,unsigned flags);/* -* Call add_submodule_odb() to add the submodule at the given path to a list. -* When register_all_submodule_odb_as_alternates() is called, the object stores -* of all submodules in that list will be added as alternates in -* the_repository. +* Call add_submodule_odb_by_path() to add the submodule at the given +* path to a list. When register_all_submodule_odb_as_alternates() is +* called, the object stores of all submodules in that list will be +* added as alternates in the_repository. */ -int add_submodule_odb (const char *path);void add_submodule_odb_by_path (const char *path);int register_all_submodule_odb_as_alternates (void );
Signature · 1 line
2.33.GIT
Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()
Quoted 7 lines · Glen Choo
add_submodule_odb()is a hack - it adds a submodule's odb as an alternate, allowing the submodule's objects to be read viathe_repository. Its last caller issubmodule_has_commits(), which callsadd_submodule_odb()to prepare forcheck_has_commit(). This used to be necessary becausecheck_has_commit()used the_repository's odb, but this is longer true as of13a2f620b2(submodule: pass repo tocheck_has_commit(), 2021-10-08).
Yes! I wonder if we can do this much earlier in the series (or even an independent clean-up that the rest of the series depends on) and have it graduate earlier?
Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()
Quoted 10 lines · Junio C Hamano
Quoted 7 lines · Glen Choo
add_submodule_odb()is a hack - it adds a submodule's odb as an alternate, allowing the submodule's objects to be read viathe_repository. Its last caller issubmodule_has_commits(), which callsadd_submodule_odb()to prepare forcheck_has_commit(). This used to be necessary becausecheck_has_commit()used the_repository's odb, but this is longer true as of13a2f620b2(submodule: pass repo tocheck_has_commit(), 2021-10-08).
Quoted 3 lines
Yes! I wonder if we can do this much earlier in the series (or even an independent clean-up that the rest of the series depends on) and have it graduate earlier?
This patch is totally conflict-free and dependency-free with regard to the rest of the series, so there's almost no overhead to sending this as an independent clean up.
And since you seem interested in seeing this graduate early, I'll send it out independently :)
Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()
In reply to Glen ChooQuoted 23 lines · Glen Choo
add_submodule_odb()is a hack - it adds a submodule's odb as an alternate, allowing the submodule's objects to be read viathe_repository. Its last caller issubmodule_has_commits(), which callsadd_submodule_odb()to prepare forcheck_has_commit(). This used to be necessary becausecheck_has_commit()used the_repository's odb, but this is longer true as of13a2f620b2(submodule: pass repo tocheck_has_commit(), 2021-10-08).Removing
add_submodule_odb()reveals a bug incheck_has_commit(), wherecheck_has_commit()will segfault if the submodule is missing (e.g. the user has not init-ed the submodule). This happens because the submodule's struct repository cannot be initialized, butcheck_has_commit()tries to cleanup the uninitialized struct anyway. This was masked byadd_submodule_odb(), becauseadd_submodule_odb()fails when the submodule is missing, causing the caller to return early and avoid callingcheck_has_commit().Fix the bug and remove the call to
add_submodule_odb(). Sinceadd_submodule_odb()has no more callers, remove it too.Note that submodule odbs can still by added as alternates via
add_submodule_odb_by_path().
Quoted 6 lines
Signed-off-by Glen Choo <[email protected]>---
This bug only exists because we can't callrepo_clear()twice on the same struct repository. So instead of just fixing this site, an alternative (and maybe better) fix would be to fixrepo_clear(). If others think that's a good idea, I'll do that instead.
Reading the first paragraph of the commit message, I'm given the impression that this is the last site in which we attempt to add a submodule ODB as an alternate, but that is not true. This is indeed the last usage of add_submodule_odb(), but add_submodule_odb_by_path() still exists.
I think the primary point of this commit is to fix a latent bug in check_has_commit(), and add_submodule_odb()'s role here is just hiding it. Its hacky behavior does not play a role.
I would write the commit message like this:
submodule: fix latent check_has_commit() bug
check_has_commit() will attempt to clear a non-initialized struct repository if initialization fails. This bug is masked by its only caller, submodule_has_commits(), first calling add_submodule_odb(). The latter fails if the repo does not exist, making submodule_has_commits() exit early and not invoke check_has_commit() in a situation in which initialization would fail.
Fix this bug, and because calling add_submodule_odb() is no longer necessary as of 13a2f620b2 (submodule: pass repo to check_has_commit(), 2021-10-08), remove that call too.
This is the last caller of add_submodule_odb(), so remove that function. (Adding submodule ODBs as alternates is still present in the form of add_submodule_odb_by_path().)
Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()
Quoted 52 lines · Jonathan Tan
Quoted 23 lines · Glen Choo
add_submodule_odb()is a hack - it adds a submodule's odb as an alternate, allowing the submodule's objects to be read viathe_repository. Its last caller issubmodule_has_commits(), which callsadd_submodule_odb()to prepare forcheck_has_commit(). This used to be necessary becausecheck_has_commit()used the_repository's odb, but this is longer true as of13a2f620b2(submodule: pass repo tocheck_has_commit(), 2021-10-08).Removing
add_submodule_odb()reveals a bug incheck_has_commit(), wherecheck_has_commit()will segfault if the submodule is missing (e.g. the user has not init-ed the submodule). This happens because the submodule's struct repository cannot be initialized, butcheck_has_commit()tries to cleanup the uninitialized struct anyway. This was masked byadd_submodule_odb(), becauseadd_submodule_odb()fails when the submodule is missing, causing the caller to return early and avoid callingcheck_has_commit().Fix the bug and remove the call to
add_submodule_odb(). Sinceadd_submodule_odb()has no more callers, remove it too.Note that submodule odbs can still by added as alternates via
add_submodule_odb_by_path().Quoted 6 lines
Signed-off-by Glen Choo <[email protected]>---
This bug only exists because we can't callrepo_clear()twice on the same struct repository. So instead of just fixing this site, an alternative (and maybe better) fix would be to fixrepo_clear(). If others think that's a good idea, I'll do that instead.Reading the first paragraph of the commit message, I'm given the impression that this is the last site in which we attempt to add a submodule ODB as an alternate, but that is not true. This is indeed the last usage of
add_submodule_odb(), butadd_submodule_odb_by_path()still exists.I think the primary point of this commit is to fix a latent bug in
check_has_commit(), and add_submodule_odb()'s role here is just hiding it. Its hacky behavior does not play a role.I would write the commit message like this:
submodule: fix latent
check_has_commit()bug
check_has_commit()will attempt to clear a non-initialized struct repository if initialization fails. This bug is masked by its only caller,submodule_has_commits(), first callingadd_submodule_odb(). The latter fails if the repo does not exist, makingsubmodule_has_commits()exit early and not invokecheck_has_commit()in a situation in which initialization would fail.
Quoted 7 lines
Fix this bug, and because calling
add_submodule_odb()is no longer necessary as of13a2f620b2(submodule: pass repo tocheck_has_commit(), 2021-10-08), remove that call too.This is the last caller of
add_submodule_odb(), so remove that function. (Adding submodule ODBs as alternates is still present in the form ofadd_submodule_odb_by_path().)
Hm.. that is a good point, the commit message seems to promise more than what it actually delivers. I'll take your suggestion, thanks!
Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()
In reply to Jonathan TanQuoted 11 lines · Jonathan Tan
I would write the commit message like this:
submodule: fix latent
check_has_commit()bug
check_has_commit()will attempt to clear a non-initialized struct repository if initialization fails. This bug is masked by its only caller,submodule_has_commits(), first callingadd_submodule_odb(). The latter fails if the repo does not exist, makingsubmodule_has_commits()exit early and not invokecheck_has_commit()in a situation in which initialization would fail.
Quoted 7 lines
Fix this bug, and because calling
add_submodule_odb()is no longer necessary as of13a2f620b2(submodule: pass repo tocheck_has_commit(), 2021-10-08), remove that call too.This is the last caller of
add_submodule_odb(), so remove that function. (Adding submodule ODBs as alternates is still present in the form ofadd_submodule_odb_by_path().)
Looks more clearly explained.
We still end up calling add_to_alternate_memory(), so I take the "let's have this early" back.
Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()
Quoted 17 lines · Junio C Hamano
Quoted 15 lines · Jonathan Tan
I would write the commit message like this:
submodule: fix latent
check_has_commit()bug
check_has_commit()will attempt to clear a non-initialized struct repository if initialization fails. This bug is masked by its only caller,submodule_has_commits(), first callingadd_submodule_odb(). The latter fails if the repo does not exist, makingsubmodule_has_commits()exit early and not invokecheck_has_commit()in a situation in which initialization would fail.Fix this bug, and because calling
add_submodule_odb()is no longer necessary as of13a2f620b2(submodule: pass repo tocheck_has_commit(), 2021-10-08), remove that call too.
Quoted 8 lines
Quoted 3 lines
This is the last caller of
add_submodule_odb(), so remove that function. (Adding submodule ODBs as alternates is still present in the form ofadd_submodule_odb_by_path().)Looks more clearly explained.
We still end up calling
add_to_alternate_memory(), so I take the "let's have this early" back.
Ok, so I won't send this patch separately. Thanks Jonathan for making things clearer :)
Quoted 7 lines · Glen Choo
= Background
When fetching submodule commits, "git fetch --recurse-submodules" only considers populated submodules, and not all of the submodules in $GIT_DIR/modules as one might expect. As a result, "git fetch --recurse-submodules" behaves differently based on which commit is checked out.
After getting 'init'ed, which is a sign that the user is interested in that submodule, when we happen to check out a superproject commit that lack the submodule in question, do we _lose_ the record that it was once of interest? I do not think so. The cloned copy in $GIT_DIR/modules/ would not go away, so we _could_ update it, even there is no checkout, when the superproject we currently have may not have the submodule.
But I am not sure if that is a problem. After all, the recursive fetch tries to make sure that the superproject commit that is checked out is reproduced as recorded by fetching the submodule commit recorded in the superproject commit, not a commit that happens to be at the tip of random branch in the submodule.
It is OK to allow fetching into submodule that is not currently have a checkout, but I think we should view it purely as prefetching. We do not even know, after doing such a fetch in the submodule, we have the commit necessary for the _next_ commit in superproject we will check out.
Quoted 7 lines
This can be a problem, for instance, if the user has a branch with submodules and a branch without:
# the submodules were initialized at some point in history.. git checkout -b branch-with-submodules origin/branch-with-submodules git submodule update --init
Quoted 7 lines
# later down the road.. git checkout --recurse-submodules branch-without-submodules # no submodules are fetched! git fetch --recurse-submodules # if origin/branch-with-submodules has new submodule commits, this # checkout will fail because we never fetched the submodule git checkout --recurse-submodules branch-with-submodules
That is expected, and UNLESS we fetched _everything_ offered by the remote repository to the submodule in the previous step, there is no guarantee that this "recurse-submodules" checkout would succeed.
Quoted 7 lines
This series makes "git fetch" fetch the right submodules regardless of which commit is checked out, as long as the submodule has already been cloned. In particular, "git fetch" learns to:
1. read submodules from the relevant superproject commit instead of the file system
2. fetch all changed submodules, even if they are not populated
The real question is not "in which submodules we fetch", but "what commits we fetch in these submodules". I do not think there is a good answer to the latter.
Of course, we we take this sequence instead:
git checkout branch-with-submodules git fetch --recurse-submodules git checkout --recurse-submodules branch-with-submodules
things should work correctly (I think we both are assuming that the other side allows to fetch _any_ object, not just ref), as "fetch" knows what superproject commit it is asked to complete, unlike the previous example you gave, where it does not have a clue on what superproject commit it is preparing submodules for, right?
So, I am not quite sure if we are solving the right problem, let alone with the right approach.
Also, if the strategy is to prefetch in all submodules that were 'init'ed, we do not have to read .gitmodules from the superproject commit at all, right? We can just go check .git/modules/ which is available locally. We need to see which submodules are of local interest by consulting .git/config and/or .git/modules/ anyway even if we read .gitmodules from the superproject commit to learn what modules are there.
Thanks.
Quoted 24 lines · Junio C Hamano
Quoted 7 lines · Glen Choo
= Background
When fetching submodule commits, "git fetch --recurse-submodules" only considers populated submodules, and not all of the submodules in $GIT_DIR/modules as one might expect. As a result, "git fetch --recurse-submodules" behaves differently based on which commit is checked out.
After getting 'init'ed, which is a sign that the user is interested in that submodule, when we happen to check out a superproject commit that lack the submodule in question, do we _lose_ the record that it was once of interest? I do not think so. The cloned copy in $GIT_DIR/modules/ would not go away, so we _could_ update it, even there is no checkout, when the superproject we currently have may not have the submodule.
But I am not sure if that is a problem. After all, the recursive fetch tries to make sure that the superproject commit that is checked out is reproduced as recorded by fetching the submodule commit recorded in the superproject commit, not a commit that happens to be at the tip of random branch in the submodule.
Quoted 5 lines
It is OK to allow fetching into submodule that is not currently have a checkout, but I think we should view it purely as prefetching. We do not even know, after doing such a fetch in the submodule, we have the commit necessary for the _next_ commit in superproject we will check out.
Hm, I may be misreading your message, but by "tip of random branch in the submodule", did you mean "tip of random branch in the _superproject_"?
If so, prior to this series, recursive fetch already fetches submodule commits that are recorded by superproject commits other than the one checked out. submodule.c:calculate_changed_submodule_paths() performs a rev walk starting from the newly fetched superproject branch tips to find missing submodule commits that are referenced by superproject commits. These missing submodule commits are explicitly fetched by the recursive fetch.
So we already do prefetching, but this series makes the prefetching smarter by also prefetching in submodules that aren't checked out.
(I think my cover letter could have been clearer; I should have explicitly called out the fact that we already do prefetching.)
Quoted 15 lines
Quoted 7 lines
This can be a problem, for instance, if the user has a branch with submodules and a branch without:
# the submodules were initialized at some point in history.. git checkout -b branch-with-submodules origin/branch-with-submodules git submodule update --initQuoted 7 lines
# later down the road.. git checkout --recurse-submodules branch-without-submodules # no submodules are fetched! git fetch --recurse-submodules # if origin/branch-with-submodules has new submodule commits, this # checkout will fail because we never fetched the submodule git checkout --recurse-submodules branch-with-submodules
Quoted 3 lines
That is expected, and UNLESS we fetched _everything_ offered by the remote repository to the submodule in the previous step, there is no guarantee that this "recurse-submodules" checkout would succeed.
Yes. With the current prefetching, I don't think we make any guarantee to the user that all submodule commits will be fetched (even if all of the subomdules are checked out).
But if I understand the "find changed submodules" rev walk correctly, we look for changed submodules in the ancestry chains of the branch tips (but I'm not sure how the rev walk decides to stop). So we might be _very close_ to fetching all the commits that we think users care about even though we don't guarantee that all commits will be fetched.
Quoted 18 lines
Quoted 7 lines
This series makes "git fetch" fetch the right submodules regardless of which commit is checked out, as long as the submodule has already been cloned. In particular, "git fetch" learns to:
1. read submodules from the relevant superproject commit instead of the file system
2. fetch all changed submodules, even if they are not populatedThe real question is not "in which submodules we fetch", but "what commits we fetch in these submodules". I do not think there is a good answer to the latter.
Of course, we we take this sequence instead:
git checkout branch-with-submodules git fetch --recurse-submodules git checkout --recurse-submodules branch-with-submodules
Quoted 5 lines
things should work correctly (I think we both are assuming that the other side allows to fetch _any_ object, not just ref), as "fetch" knows what superproject commit it is asked to complete, unlike the previous example you gave, where it does not have a clue on what superproject commit it is preparing submodules for, right?
So, given my prior description of recursive fetch, we actually _do_ know which superproject commits to prepare for and which submodule commits to fetch.
Quoted 7 lines
Also, if the strategy is to prefetch in all submodules that were 'init'ed, we do not have to read .gitmodules from the superproject commit at all, right? We can just go check .git/modules/ which is available locally. We need to see which submodules are of local interest by consulting .git/config and/or .git/modules/ anyway even if we read .gitmodules from the superproject commit to learn what modules are there.
Hm, good point. Finding submodules of interest in .git/modules or .git/config sounds like common sense (it's more obvious than trying to identify all submodules by doing a rev walk at least).
That said, just looking at what submodules we have doesn't tell us which submodule commits we need, which is why we have the "find changed submodules" rev walk. And since we already have the rev walk (which tells us which superproject commits we care about), it's not that much effort to fetch non-checked-out submodules.
So I think we'd eventually want to consult .git/modules and .git/config (we'll have to do this when we start teaching "git fetch" to clone new submodules, for example) but it's unnecessary for this series.
Quoted 9 lines · Glen Choo
Quoted 5 lines
It is OK to allow fetching into submodule that is not currently have a checkout, but I think we should view it purely as prefetching. We do not even know, after doing such a fetch in the submodule, we have the commit necessary for the _next_ commit in superproject we will check out.
Hm, I may be misreading your message, but by "tip of random branch in the submodule", did you mean "tip of random branch in the _superproject_"?
No, I meant something like "git submodule foreach 'git fetch --all'" (or without '--all' to fetch whatever the refspec there tells us), i.e. tips of branches in the submodule.
Quoted 16 lines
Quoted 10 lines
The real question is not "in which submodules we fetch", but "what commits we fetch in these submodules". I do not think there is a good answer to the latter.
Of course, we we take this sequence instead:
git checkout branch-with-submodules git fetch --recurse-submodules git checkout --recurse-submodules branch-with-submodulesQuoted 5 lines
things should work correctly (I think we both are assuming that the other side allows to fetch _any_ object, not just ref), as "fetch" knows what superproject commit it is asked to complete, unlike the previous example you gave, where it does not have a clue on what superproject commit it is preparing submodules for, right?
Quoted 3 lines
So, given my prior description of recursive fetch, we actually _do_ know which superproject commits to prepare for and which submodule commits to fetch.
Just to make sure I understand what is going on, let me rephrase.
* To find out which submodule commits we need to fetch, we find new commits in the superproject we just fetched, inspect the trees of these commits to see gitlinks that name commits we need to fetch into the submodule repositories.
* For that to work well, we need to know, from the path these commits appear in the trees of the superproject, to find out from which submodule to fetch these commits from. And to make the mapping from paths to submodule names, we need to read .gitmodules from the same superproject commit we found the submodule commit in (as during the history of the superproject, the submodule may have moved around).
If so, I understand why being able to read .gitmodules from superproject commits is essential. The flow would become like
(1) fetch in the superproject
(2) iterate over each new superproject commit:
- read its .gitmodules
- iterate over each gitlink found in the superproject commit:
- map the path we found gitlink at into module name
- find the submodule repository initialized for the module
- if the submodule is not of local interest, skip
- add the submodule commit pointed by gitlink to the set of commits that need to be fetched for the submodule [*]
(3) iterate over each submodule we found more than one commits that need to be fetched in, and fetch these commits (we do not have to go over the network to re-fetch commits that exist in the object store and are reachable from the refs, but "fetch" already knows how to optimize that).
Am I on the right track?
Thanks.
Quoted 49 lines · Junio C Hamano
Quoted 16 lines
Quoted 10 lines
The real question is not "in which submodules we fetch", but "what commits we fetch in these submodules". I do not think there is a good answer to the latter.
Of course, we we take this sequence instead:
git checkout branch-with-submodules git fetch --recurse-submodules git checkout --recurse-submodules branch-with-submodulesQuoted 5 lines
things should work correctly (I think we both are assuming that the other side allows to fetch _any_ object, not just ref), as "fetch" knows what superproject commit it is asked to complete, unlike the previous example you gave, where it does not have a clue on what superproject commit it is preparing submodules for, right?
Quoted 3 lines
So, given my prior description of recursive fetch, we actually _do_ know which superproject commits to prepare for and which submodule commits to fetch.
Just to make sure I understand what is going on, let me rephrase.
* To find out which submodule commits we need to fetch, we find new commits in the superproject we just fetched, inspect the trees of these commits to see gitlinks that name commits we need to fetch into the submodule repositories.
* For that to work well, we need to know, from the path these commits appear in the trees of the superproject, to find out from which submodule to fetch these commits from. And to make the mapping from paths to submodule names, we need to read .gitmodules from the same superproject commit we found the submodule commit in (as during the history of the superproject, the submodule may have moved around).
If so, I understand why being able to read .gitmodules from superproject commits is essential. The flow would become like
(1) fetch in the superproject
(2) iterate over each new superproject commit:
- read its .gitmodules
- iterate over each gitlink found in the superproject commit:
- map the path we found gitlink at into module name
- find the submodule repository initialized for the module
- if the submodule is not of local interest, skip
- add the submodule commit pointed by gitlink to the set of commits that need to be fetched for the submodule [*]
Quoted 7 lines
(3) iterate over each submodule we found more than one commits that need to be fetched in, and fetch these commits (we do not have to go over the network to re-fetch commits that exist in the object store and are reachable from the refs, but "fetch" already knows how to optimize that).
Am I on the right track?
Yup, I think that's quite an accurate description. In particular..
Quoted 2 lines
(2) iterate over each new superproject commit:
- read its .gitmodules
Prior to this series, .gitmodules is read from the filesystem even though we may notice the missing commit in a non-checked-out superproject commit.
Quoted 2 lines
(3) iterate over each submodule we found more than one commits that need to be fetched in, and fetch these commits
Yes, this describes the new "fetch changed submodules behavior" accurately. However, we also attempt to fetch checked out submodules, and this is where the two fetching strategies, "yes" and "on-demand" [1] matter:
"yes", aka "--recurse-submodules" tells "git fetch" to attempt to fetch _every_ checked out submodule regardless of whether Git thinks it has missing commits (if we do not find any missing commits, I believe it defaults to the "git fetch <remote-name>" form). [2]
"on-demand", aka "--recurse-submodules=on-demand", is the 'default' option. (It is 'default' as in the sense of not passing a "--recurse-submodules" arg, not default as in passing "--recurse-submodules" without an "="). With "on-demand", we _only_ attempt to fetch changed submodules. Vis-a-vis "yes", this strategy has no effect on fetching non-checked-out submodules because we only fetch changed, non-checked-out submodules anyway, but it lets us ignore unchanged, checked out submodules.
[1] From Documentation/fetch-options.txt:
--recurse-submodules[=yes|on-demand|no]::
This option controls if and under what conditions new commits of populated submodules should be fetched too. It can be used as a boolean option to completely disable recursion when set to 'no' or to unconditionally recurse into all populated submodules when set to 'yes', which is the default when this option is used without any value. Use 'on-demand' to only recurse into a populated submodule when the superproject retrieves a commit that updates the submodule's reference to a commit that isn't already in the local submodule clone. By default, 'on-demand' is used, unlessfetch.recurseSubmodules is set (see linkgit:git-config[1]).
[2] Sidenote on the "yes" option: I think the rationale for doing unconditional fetches is not clear to reviewers. Admittedly, beyond 'the test suite fails', I don't really remember why either. I'll dig into this as I respond to the reviewer feedback.
v2 cover fetch --recurse-submodules: fetch unpopulated submodules
In reply to Glen ChooOriginal cover letter: https://lore.kernel.org/git/[email protected]
This series is based on gc/branch-recurse-submodules.
Thanks for the kind feedback :) I believe this version addresses all of the feedback, though I suspect that I might not have given clear-enough answers to all of the questions. Let me know if more clarification is needed.
= Patch organization
The patches are organized somewhat differently vs v1. During review of v1, I realized that it didn't make sense to read submodules from the superproject commit until we actually fetch unpopulated submodules. So this version just has all of the C patches together, instead of putting the test suite patches after "read submodules from the superproject commit".
- Patches 1-2 are quality-of-life improvements to the test suite that make it easier to write the tests in patch 7.
- Patches 3-5 are preparation for "git fetch" to read .gitmodules from the superproject commit in patch 7.
- Patch 6 separates the steps of "finding which submodules to fetch" and "fetching the submodules", making it easier to tell "git fetch" to fetch unpopulated submodules.
- Patch 7 teaches "git fetch" to fetch changed, unpopulated submodules in addition to populated submodules.
- Patch 8 is an optional bugfix that fixes a bug with "git fetch --update-shallow" in a repository with submodules. This was discovered in v1 but is no longer necessary for tests to pass.
- Patch 9 is an optional bugfix + cleanup of the "git fetch" code that removes the last caller of the deprecated "add_submodule_odb()".
= Changes
Since v1:
- Numerous style fixes suggested by Jonathan (thanks!)
- In patch 3, don't prematurely read submodules from the superproject commit (see:
<[email protected]>).
- In patch 7, stop using "git checkout" and "! grep" in tests.
- In patch 7, stop doing the "find changed submodules" rev walk unconditionally. Instead, continue to check for .gitmodules, but also check for submodules in $GIT_DIR/modules.
- I'm not entirely happy with the helper function name, see "---" for details.
- Move "git fetch --update-shallow" bugfix to patch 8.
- Because the "find changed submodules" rev walk is no longer unconditional, this fix is no longer needed for tests to pass.
- Rename fetch_populated_submodules() to fetch_submodules().
Glen Choo (9):
t5526: introduce test helper to assert on fetches
t5526: use grep to assert on fetches
submodule: make static functions read submodules from commits
submodule: inline submodule_commits() into caller
submodule: store new submodule commits oid_array in a struct
submodule: extract get_fetch_task()
fetch: fetch unpopulated, changed submodules
submodule: read shallows when finding changed submodules
submodule: fix latent check_has_commit() bug
Documentation/fetch-options.txt | 26 ++- Documentation/git-fetch.txt | 10 +- builtin/fetch.c | 14 +- submodule.c | 321 +++++++++++++++++---------- submodule.h | 21 +- t/t5526-fetch-submodules.sh | 374 ++++++++++++++++++++++++-------- 6 files changed, 534 insertions(+), 232 deletions(-)
Range-diff against v1:
4: 432aa60296 = 1: a159cdaabb t5526: introduce test helper to assert on fetches 5: 9515d22804 = 2: 48894c6c43 t5526: use grep to assert on fetches 3: 3c28ea743c ! 3: 6cf5e76d62 submodule: make static functions read submodules from commits
@@ Commit message
submodule: make static functions read submodules from commits
A future commit will teach "fetch --recurse-submodules" to fetch
- unpopulated submodules. Prepare for this by teaching the necessary
- static functions to read submodules from superproject commits instead of
- the index and filesystem. Then, store the necessary fields (path and
- super_oid), and use them in "fetch --recurse-submodules" where possible.
+ unpopulated submodules. To prepare for this, teach the necessary static
+ functions how to read submodules from superproject commits using a
+ "treeish_name" argument (instead of always reading from the index and
+ filesystem) but do not actually change where submodules are read from.
+ Submodules will be read from commits when we fetch unpopulated
+ submodules.
- As a result, "git fetch" now reads changed submodules using the
- .gitmodules and path from super_oid's tree (which is where "git fetch"
- actually noticed the changed submodule) instead of the filesystem.
+ The changed function signatures follow repo_submodule_init()'s argument
+ order, i.e. "path" then "treeish_name". Where needed, reorder the
+ arguments of functions that already take "path" and "treeish_name" to be
+ consistent with this convention.
## submodule.c ##
-@@ submodule.c: static const char *default_name_or_path(const char *path_or_name)
- * member of the changed submodule string_list_item.
- */
- struct changed_submodule_data {
-+ /*
-+ * The first superproject commit in the rev walk that points to the
-+ * submodule.
-+ */
-+ const struct object_id *super_oid;
-+ /*
-+ * Path to the submodule in the superproject commit referenced
-+ * by 'super_oid'.
-+ */
-+ char *path;
- /* The submodule commits that have changed in the rev walk. */
- struct oid_array *new_commits;
- };
-@@ submodule.c: static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)
- {
- oid_array_clear(cs_data->new_commits);
- free(cs_data->new_commits);
-+ free(cs_data->path);
- }
-
- static void collect_changed_submodules_cb(struct diff_queue_struct *q,
-@@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct *q,
- cs_data = xcalloc(1, sizeof(struct changed_submodule_data));
- /* NEEDSWORK: should we have oid_array_init()? */
- cs_data->new_commits = xcalloc(1, sizeof(struct oid_array));
-+ cs_data->super_oid = commit_oid;
-+ cs_data->path = xstrdup(p->two->path);
- item->util = cs_data;
- }
- oid_array_append(cs_data->new_commits, &p->two->oid);
@@ submodule.c: struct has_commit_data {
struct repository *repo;
int result;
@@ submodule.c: static int submodule_needs_pushing(struct repository *r,
/*
* NOTE: We do consider it safe to return "no" here. The
* correct answer would be "We do not know" instead of
-@@ submodule.c: static void calculate_changed_submodule_paths(struct repository *r,
- const struct submodule *submodule;
- const char *path = NULL;
-
-- submodule = submodule_from_name(r, null_oid(), name->string);
-+ submodule = submodule_from_name(r, cs_data->super_oid, name->string);
- if (submodule)
- path = submodule->path;
- else
@@ submodule.c: static void calculate_changed_submodule_paths(struct repository *r,
if (!path)
continue;
-- if (submodule_has_commits(r, path, cs_data->new_commits)) {
-+ if (submodule_has_commits(r, path, cs_data->super_oid, cs_data->new_commits)) {
- oid_array_clear(cs_data->new_commits);
+- if (submodule_has_commits(r, path, commits)) {
++ if (submodule_has_commits(r, path, null_oid(), commits)) {
+ oid_array_clear(commits);
*name->string = '\0';
}
@@ submodule.c: static const struct submodule *get_non_gitmodules_submodule(const char *path)
1: a8ef64d16c ! 4: 07fd4ff0a9 submodule: inline submodule_commits() into caller
@@ Commit message
Prepare for this change by inlining submodule_commits() (which inserts into the string_list and initializes the string_list_item.util) into its
- only caller. This simplifies the code and makes it easier for the caller
- to add information to the string_list_item.util.
+ only caller so that the code is easier to refactor later.
2: 11e48fbd41 ! 5: f049cb231b submodule: store new submodule commits oid_array in a struct
@@ submodule.c: static const char *default_name_or_path(const char *path_or_name)
+ */
+struct changed_submodule_data {
+ /* The submodule commits that have changed in the rev walk. */
-+ struct oid_array *new_commits;
++ struct oid_array new_commits;
+};
+
+static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)
+{
-+ oid_array_clear(cs_data->new_commits);
-+ free(cs_data->new_commits);
++ oid_array_clear(&cs_data->new_commits);
+}
+
static void collect_changed_submodules_cb(struct diff_queue_struct *q,
@@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct
if (!S_ISGITLINK(p->two->mode))
continue;
@@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct *q,
- continue;
item = string_list_insert(changed, name);
-- if (!item->util)
-+ if (item->util)
-+ cs_data = item->util;
-+ else {
-+ cs_data = xcalloc(1, sizeof(struct changed_submodule_data));
- /* NEEDSWORK: should we have oid_array_init()? */
+ if (!item->util)
+- /* NEEDSWORK: should we have oid_array_init()? */
- item->util = xcalloc(1, sizeof(struct oid_array));
- oid_array_append(item->util, &p->two->oid);
-+ cs_data->new_commits = xcalloc(1, sizeof(struct oid_array));
-+ item->util = cs_data;
-+ }
-+ oid_array_append(cs_data->new_commits, &p->two->oid);
++ item->util = xcalloc(1, sizeof(struct changed_submodule_data));
++ cs_data = item->util;
++ oid_array_append(&cs_data->new_commits, &p->two->oid);
}
}
@@ submodule.c: int find_unpushed_submodules(struct repository *r,
continue;
- if (submodule_needs_pushing(r, path, commits))
-+ if (submodule_needs_pushing(r, path, cs_data->new_commits))
++ if (submodule_needs_pushing(r, path, &cs_data->new_commits))
string_list_insert(needs_pushing, path);
}
@@ submodule.c: static void calculate_changed_submodule_paths(struct repository *r,
if (!path)
continue;
-- if (submodule_has_commits(r, path, commits)) {
+- if (submodule_has_commits(r, path, null_oid(), commits)) {
- oid_array_clear(commits);
-+ if (submodule_has_commits(r, path, cs_data->new_commits)) {
-+ oid_array_clear(cs_data->new_commits);
++ if (submodule_has_commits(r, path, null_oid(), &cs_data->new_commits)) {
++ changed_submodule_data_clear(cs_data);
*name->string = '\0';
}
}
@@ submodule.c: static int fetch_finish(int retvalue, struct strbuf *err,
- commits = it->util;
- oid_array_filter(commits,
+ cs_data = it->util;
-+ oid_array_filter(cs_data->new_commits,
++ oid_array_filter(&cs_data->new_commits,
commit_missing_in_sub,
task->repo);
/* Are there commits we want, but do not exist? */
- if (commits->nr) {
- task->commits = commits;
-+ if (cs_data->new_commits->nr) {
-+ task->commits = cs_data->new_commits;
++ if (cs_data->new_commits.nr) {
++ task->commits = &cs_data->new_commits;
ALLOC_GROW(spf->oid_fetch_tasks,
spf->oid_fetch_tasks_nr + 1,
spf->oid_fetch_tasks_alloc);
6: db6de6617b ! 6: 814073eecc submodule: extract get_fetch_task()
@@ submodule.c: static int get_next_submodule(struct child_process *cp,
break;
case RECURSE_SUBMODULES_OFF:
continue;
- }
+@@ submodule.c: static int get_next_submodule(struct child_process *cp,
task->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());
-- if (task->repo) {
+ if (task->repo) {
- struct strbuf submodule_prefix = STRBUF_INIT;
- child_process_init(cp);
- cp->dir = task->repo->gitdir;
- prepare_submodule_repo_env_in_gitdir(&cp->env_array);
- cp->git_cmd = 1;
-- if (!spf->quiet)
-- strbuf_addf(err, _("Fetching submodule %s%s\n"),
-- spf->prefix, ce->name);
+ if (!spf->quiet)
+ strbuf_addf(err, _("Fetching submodule %s%s\n"),
+ spf->prefix, ce->name);
- strvec_init(&cp->args);
- strvec_pushv(&cp->args, spf->args.v);
- strvec_push(&cp->args, default_argv);
@@ submodule.c: static int get_next_submodule(struct child_process *cp,
- spf->prefix,
- task->sub->path);
- strvec_push(&cp->args, submodule_prefix.buf);
--
-- spf->count++;
+
+ spf->count++;
- *task_cb = task;
-
- strbuf_release(&submodule_prefix);
- return 1;
-- } else {
-+ if (!task->repo) {
++ return task;
+ } else {
struct strbuf empty_submodule_path = STRBUF_INIT;
- fetch_task_release(task);
@@ submodule.c: static int get_next_submodule(struct child_process *cp,
- ce->name);
- }
strbuf_release(&empty_submodule_path);
-+ continue;
}
-+ if (!spf->quiet)
-+ strbuf_addf(err, _("Fetching submodule %s%s\n"),
-+ spf->prefix, ce->name);
-+
-+ spf->count++;
-+ return task;
-+ }
+ }
+ return NULL;
+}
+
@@ submodule.c: static int get_next_submodule(struct child_process *cp,
+ strvec_push(&cp->args, default_argv);
+ strvec_push(&cp->args, "--submodule-prefix");
+
-+ strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix,
-+ task->sub->path);
++ strbuf_addf(&submodule_prefix, "%s%s/",
++ spf->prefix,
++ task->sub->path);
+ strvec_push(&cp->args, submodule_prefix.buf);
+ *task_cb = task;
+
+ strbuf_release(&submodule_prefix);
+ return 1;
- }
++ }
if (spf->oid_fetch_tasks_nr) {
+ struct fetch_task *task =
7: 1737338380 ! 7: 10fd5bf921 fetch: fetch unpopulated, changed submodules
@@ Documentation/git-fetch.txt: include::transfer-data-leaks.txt[]
SEE ALSO
--------
+ ## builtin/fetch.c ##
+@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
+ max_children = fetch_parallel_config;
+
+ add_options_to_argv(&options);
+- result = fetch_populated_submodules(the_repository,
+- &options,
+- submodule_prefix,
+- recurse_submodules,
+- recurse_submodules_default,
+- verbosity < 0,
+- max_children);
++ result = fetch_submodules(the_repository,
++ &options,
++ submodule_prefix,
++ recurse_submodules,
++ recurse_submodules_default,
++ verbosity < 0,
++ max_children);
+ strvec_clear(&options);
+ }
+
+
## submodule.c ##
-@@
- #include "parse-options.h"
- #include "object-store.h"
- #include "commit-reach.h"
-+#include "shallow.h"
+@@ submodule.c: static const char *default_name_or_path(const char *path_or_name)
+ * member of the changed submodule string_list_item.
+ */
+ struct changed_submodule_data {
++ /*
++ * The first superproject commit in the rev walk that points to the
++ * submodule.
++ */
++ const struct object_id *super_oid;
++ /*
++ * Path to the submodule in the superproject commit referenced
++ * by 'super_oid'.
++ */
++ char *path;
+ /* The submodule commits that have changed in the rev walk. */
+ struct oid_array new_commits;
+ };
+@@ submodule.c: struct changed_submodule_data {
+ static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)
+ {
+ oid_array_clear(&cs_data->new_commits);
++ free(cs_data->path);
+ }
- static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;
- static int initialized_fetch_ref_tips;
-@@ submodule.c: static void collect_changed_submodules(struct repository *r,
+ static void collect_changed_submodules_cb(struct diff_queue_struct *q,
+@@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct *q,
+ if (!item->util)
+ item->util = xcalloc(1, sizeof(struct changed_submodule_data));
+ cs_data = item->util;
++ cs_data->super_oid = commit_oid;
++ cs_data->path = xstrdup(p->two->path);
+ oid_array_append(&cs_data->new_commits, &p->two->oid);
+ }
+ }
+@@ submodule.c: void check_for_new_submodule_commits(struct object_id *oid)
+ oid_array_append(&ref_tips_after_fetch, oid);
+ }
- save_warning = warn_on_object_refname_ambiguity;
- warn_on_object_refname_ambiguity = 0;
-+ /* make sure shallows are read */
-+ is_repository_shallow(the_repository);
++/*
++ * Returns 1 if the repo has absorbed submodule gitdirs, and 0
++ * otherwise. Like submodule_name_to_gitdir(), this checks
++ * $GIT_DIR/modules, not $GIT_COMMON_DIR.
++ */
++static int repo_has_absorbed_submodules(struct repository *r)
++{
++ struct strbuf buf = STRBUF_INIT;
+
- repo_init_revisions(r, &rev, NULL);
- setup_revisions(argv->nr, argv->v, &rev, &s_r_opt);
- warn_on_object_refname_ambiguity = save_warning;
-@@ submodule.c: static void calculate_changed_submodule_paths(struct repository *r,
++ strbuf_repo_git_path(&buf, r, "modules/");
++ return file_exists(buf.buf) && !is_empty_dir(buf.buf);
++}
++
+ static void calculate_changed_submodule_paths(struct repository *r,
+ struct string_list *changed_submodule_names)
+ {
struct strvec argv = STRVEC_INIT;
struct string_list_item *name;
- /* No need to check if there are no submodules configured */
- if (!submodule_from_path(r, NULL, NULL))
-- return;
--
++ /* No need to check if no submodules would be fetched */
++ if (!submodule_from_path(r, NULL, NULL) &&
++ !repo_has_absorbed_submodules(r))
+ return;
+
strvec_push(&argv, "--"); /* argv[0] program name */
- oid_array_for_each_unique(&ref_tips_after_fetch,
- append_oid_to_argv, &argv);
@@ submodule.c: int submodule_touches_in_range(struct repository *r,
}
@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
{
default:
@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
- strbuf_addf(err, _("Fetching submodule %s%s\n"),
- spf->prefix, ce->name);
+ strbuf_addf(err, _("Fetching submodule %s%s\n"),
+ spf->prefix, ce->name);
+
+- spf->count++;
++ spf->index_count++;
+ return task;
+ } else {
+ struct strbuf empty_submodule_path = STRBUF_INIT;
+@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
+ return NULL;
+ }
-- spf->count++;
-+ spf->index_count++;
-+ return task;
-+ }
-+ return NULL;
-+}
-+
+static struct fetch_task *
+get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,
+ const char **default_argv, struct strbuf *err)
@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
+ find_unique_abbrev(cs_data->super_oid,
+ DEFAULT_ABBREV));
+ spf->changed_count++;
- return task;
- }
- return NULL;
-@@ submodule.c: static int get_next_submodule(struct child_process *cp, struct strbuf *err,
++ return task;
++ }
++ return NULL;
++}
++
+ static int get_next_submodule(struct child_process *cp, struct strbuf *err,
+ void *data, void **task_cb)
{
struct submodule_parallel_fetch *spf = data;
const char *default_argv = NULL;
@@ submodule.c: static int get_next_submodule(struct child_process *cp, struct strb
return 1;
}
+@@ submodule.c: static int fetch_finish(int retvalue, struct strbuf *err,
+ return 0;
+ }
+
+-int fetch_populated_submodules(struct repository *r,
+- const struct strvec *options,
+- const char *prefix, int command_line_option,
+- int default_option,
+- int quiet, int max_parallel_jobs)
++int fetch_submodules(struct repository *r,
++ const struct strvec *options,
++ const char *prefix, int command_line_option,
++ int default_option,
++ int quiet, int max_parallel_jobs)
+ {
+ int i;
+ struct submodule_parallel_fetch spf = SPF_INIT;
+
+ ## submodule.h ##
+@@ submodule.h: int should_update_submodules(void);
+ */
+ const struct submodule *submodule_from_ce(const struct cache_entry *ce);
+ void check_for_new_submodule_commits(struct object_id *oid);
+-int fetch_populated_submodules(struct repository *r,
+- const struct strvec *options,
+- const char *prefix,
+- int command_line_option,
+- int default_option,
+- int quiet, int max_parallel_jobs);
++int fetch_submodules(struct repository *r,
++ const struct strvec *options,
++ const char *prefix,
++ int command_line_option,
++ int default_option,
++ int quiet, int max_parallel_jobs);
+ unsigned is_submodule_modified(const char *path, int ignore_untracked);
+ int submodule_uses_gitfile(const char *path);
+
## t/t5526-fetch-submodules.sh ##
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess
verify_fetch_result actual.err
'
-+# Cleans up after tests that checkout branches other than the main ones
-+# in the tests.
-+checkout_main_branches() {
-+ git -C downstream checkout --recurse-submodules super &&
-+ git -C downstream/submodule checkout --recurse-submodules sub &&
-+ git -C downstream/submodule/subdir/deepsubmodule checkout --recurse-submodules deep
-+}
-+
+# Test that we can fetch submodules in other branches by running fetch
-+# in a branch that has no submodules.
++# in a commit that has no submodules.
+test_expect_success 'setup downstream branch without submodules' '
+ (
+ cd downstream &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+'
+
+test_expect_success "'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules" '
-+ test_when_finished "checkout_main_branches" &&
+ git -C downstream fetch --recurse-submodules &&
+ # Create new superproject commit with updated submodules
+ add_upstream_commit &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ (
+ cd downstream &&
+ git switch --recurse-submodules no-submodules &&
-+ git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err &&
-+ git checkout --recurse-submodules origin/super 2>../actual-checkout.err
++ git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err
+ ) &&
+ test_must_be_empty actual.out &&
+ git rev-parse --short HEAD >superhead &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+
+ # Assert that the fetch happened at the non-HEAD commits
+ grep "Fetching submodule submodule at commit $superhead" actual.err &&
-+ grep "Fetching submodule submodule/subdir/deepsubmodule at commit $subhead" actual.err &&
-+
-+ # Assert that we can checkout the superproject commit with --recurse-submodules
-+ ! grep -E "error: Submodule .+ could not be updated" actual-checkout.err
++ grep "Fetching submodule submodule/subdir/deepsubmodule at commit $subhead" actual.err
+'
+
+test_expect_success "'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules" '
-+ test_when_finished "checkout_main_branches" &&
+ # Fetch any leftover commits from other tests.
+ git -C downstream fetch --recurse-submodules &&
+ # Create new superproject commit with updated submodules
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ (
+ cd downstream &&
+ git switch --recurse-submodules no-submodules &&
-+ git fetch --recurse-submodules >../actual.out 2>../actual.err &&
-+ git checkout --recurse-submodules origin/super 2>../actual-checkout.err
++ git fetch --recurse-submodules >../actual.out 2>../actual.err
+ ) &&
+ test_must_be_empty actual.out &&
+ git rev-parse --short HEAD >superhead &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+
+ # Assert that the fetch happened at the non-HEAD commits
+ grep "Fetching submodule submodule at commit $superhead" actual.err &&
-+ grep "Fetching submodule submodule/subdir/deepsubmodule at commit $subhead" actual.err &&
-+
-+ # Assert that we can checkout the superproject commit with --recurse-submodules
-+ ! grep -E "error: Submodule .+ could not be updated" actual-checkout.err
++ grep "Fetching submodule submodule/subdir/deepsubmodule at commit $subhead" actual.err
+'
+
+test_expect_success "'--recurse-submodules' should ignore changed, inactive submodules" '
-+ test_when_finished "checkout_main_branches" &&
+ # Fetch any leftover commits from other tests.
+ git -C downstream fetch --recurse-submodules &&
+ # Create new superproject commit with updated submodules
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ verify_fetch_result actual.err
+'
+
-+# Test that we properly fetch the submodules in the index as well as
-+# submodules in other branches.
++# In downstream, init "submodule2", but do not check it out while
++# fetching. This lets us assert that unpopulated submodules can be
++# fetched.
+test_expect_success 'setup downstream branch with other submodule' '
+ mkdir submodule2 &&
+ (
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+'
+
+test_expect_success "'--recurse-submodules' should fetch submodule commits in changed submodules and the index" '
-+ test_when_finished "checkout_main_branches" &&
+ # Fetch any leftover commits from other tests.
+ git -C downstream fetch --recurse-submodules &&
+ # Create new commit in origin/super
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ git checkout super &&
+ (
+ cd downstream &&
-+ git fetch --recurse-submodules >../actual.out 2>../actual.err &&
-+ git checkout --recurse-submodules origin/super-sub2-only 2>../actual-checkout.err
++ git fetch --recurse-submodules >../actual.out 2>../actual.err
+ ) &&
+ test_must_be_empty actual.out &&
+
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ git -C submodule rev-parse --short HEAD >subhead &&
+ git -C deepsubmodule rev-parse --short HEAD >deephead &&
+ verify_fetch_result actual.err &&
-+ # Assert that submodule is read from the index, not from a commit
-+ ! grep "Fetching submodule submodule at commit" actual.err &&
++ # grep for the exact line to check that the submodule is read
++ # from the index, not from a commit
++ grep "^Fetching submodule submodule\$" actual.err &&
+
+ # Assert that super-sub2-only and submodule2 were fetched even
+ # though another branch is checked out
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ grep -E "\.\.${super_sub2_only_head}\s+super-sub2-only\s+-> origin/super-sub2-only" actual.err &&
+ grep "Fetching submodule submodule2 at commit $super_sub2_only_head" actual.err &&
+ sub2head=$(git -C submodule2 rev-parse --short HEAD) &&
-+ grep -E "\.\.${sub2head}\s+sub2\s+-> origin/sub2" actual.err &&
-+
-+ # Assert that we can checkout the superproject commit with --recurse-submodules
-+ ! grep -E "error: Submodule .+ could not be updated" actual-checkout.err
++ grep -E "\.\.${sub2head}\s+sub2\s+-> origin/sub2" actual.err
+'
+
test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" '
-: ---------- > 8: 8aa68111b0 submodule: read shallows when finding changed submodules
8: e44bb1560e ! 9: 05a8b93154 submodule: fix bug and remove add_submodule_odb() @@ Metadata
Author: Glen Choo <[email protected]>
## Commit message ##
- submodule: fix bug and remove add_submodule_odb()
+ submodule: fix latent check_has_commit() bug
- add_submodule_odb() is a hack - it adds a submodule's odb as an
- alternate, allowing the submodule's objects to be read via
- the_repository. Its last caller is submodule_has_commits(), which calls
- add_submodule_odb() to prepare for check_has_commit(). This used to be
- necessary because check_has_commit() used the_repository's odb, but this
- is longer true as of 13a2f620b2 (submodule: pass repo to
- check_has_commit(), 2021-10-08).
+ When check_has_commit() is called on a missing submodule, initialization
+ of the struct repository fails, but it attempts to clear the struct
+ anyway (which is a fatal error). This bug is masked by its only caller,
+ submodule_has_commits(), first calling add_submodule_odb(). The latter
+ fails if the submodule does not exist, making submodule_has_commits()
+ exit early and not invoke check_has_commit().
- Removing add_submodule_odb() reveals a bug in check_has_commit(), where
- check_has_commit() will segfault if the submodule is missing (e.g. the
- user has not init-ed the submodule). This happens because the submodule
- struct cannot be initialized, but check_has_commit() tries to cleanup
- the uninitialized struct anyway. This was masked by add_submodule_odb(),
- because add_submodule_odb() fails when the submodule is missing, causing
- the caller to return early and avoid calling check_has_commit().
+ Fix this bug, and because calling add_submodule_odb() is no longer
+ necessary as of 13a2f620b2 (submodule: pass repo to
+ check_has_commit(), 2021-10-08), remove that call too.
- Fix the bug and remove the call to add_submodule_odb(). Since
- add_submodule_odb() has no more callers, remove it too.
-
- Note that submodule odbs can still by added as alternates via
- add_submodule_odb_by_path().
+ This is the last caller of add_submodule_odb(), so remove that
+ function. (Submodule ODBs are still added as alternates via
+ add_submodule_odb_by_path().)
base-commit: 679e3693aba0c17af60c031f7eef68f2296b8dad
Signature · 1 line
2.33.GIT
PATCH v2 1/9 t5526: introduce test helper to assert on fetches
A future commit will change the stderr of "git fetch --recurse-submodules" and add new tests to t/t5526-fetch-submodules.sh. This poses two challenges:
* The tests use test_cmp to assert on the stderr, which will fail on the future test because the stderr changes slightly, even though it still contains the information we expect.
* The expect.err file is constructed by the add_upstream_commit() helper as input into test_cmp, but most tests fetch a different combination of repos from expect.err. This results in noisy tests that modify parts of that expect.err to generate the expected output.
To address both of these issues, introduce a verify_fetch_result() helper to t/t5526-fetch-submodules.sh that asserts on the output of "git fetch --recurse-submodules" and handles the ordering of expect.err.
As a result, the tests no longer construct expect.err manually. test_cmp is still invoked by verify_fetch_result(), but that will be replaced in a later commit.
--- t/t5526-fetch-submodules.sh | 136 +++++++++++++++++++++--------------- 1 file changed, 81 insertions(+), 55 deletions(-)
t/t5526-fetch-submodules.sh +81 -55
@@ -13,6 +13,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +# For each submodule in the test setup, this creates a commit and writes +# a file that contains the expected err if that new commit were fetched. +# These output files get concatenated in the right order by +# verify_fetch_result(). add_upstream_commit() { ( cd submodule &&
@@ -22,9 +26,9 @@ add_upstream_commit() { git add subfile && git commit -m new subfile && head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err && - echo "From $pwd/submodule" >> ../expect.err && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err + echo "Fetching submodule submodule" > ../expect.err.sub && + echo "From $pwd/submodule" >> ../expect.err.sub && + echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub ) && ( cd deepsubmodule &&
@@ -34,12 +38,33 @@ add_upstream_commit() { git add deepsubfile && git commit -m new deepsubfile && head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule/subdir/deepsubmodule" >> ../expect.err - echo "From $pwd/deepsubmodule" >> ../expect.err && - echo " $head1..$head2 deep -> origin/deep" >> ../expect.err + echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep + echo "From $pwd/deepsubmodule" >> ../expect.err.deep && + echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep ) } +# Verifies that the expected repositories were fetched. This is done by +# concatenating the files expect.err.[super|sub|deep] in the correct +# order and comparing it to the actual stderr. +# +# If a repo should not be fetched in the test, its corresponding +# expect.err file should be rm-ed. +verify_fetch_result() { + ACTUAL_ERR=$1 && + rm -f expect.err.combined && + if [ -f expect.err.super ]; then + cat expect.err.super >>expect.err.combined + fi && + if [ -f expect.err.sub ]; then + cat expect.err.sub >>expect.err.combined + fi && + if [ -f expect.err.deep ]; then + cat expect.err.deep >>expect.err.combined + fi && + test_cmp expect.err.combined $ACTUAL_ERR +} + test_expect_success setup ' mkdir deepsubmodule && (
@@ -77,7 +102,7 @@ test_expect_success "fetch --recurse-submodules recurses into submodules" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "submodule.recurse option triggers recursive fetch" '
@@ -87,7 +112,7 @@ test_expect_success "submodule.recurse option triggers recursive fetch" ' git -c submodule.recurse fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "fetch --recurse-submodules -j2 has the same output behaviour" '
@@ -97,7 +122,7 @@ test_expect_success "fetch --recurse-submodules -j2 has the same output behaviou GIT_TRACE="$TRASH_DIRECTORY/trace.out" git fetch --recurse-submodules -j2 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err && + verify_fetch_result actual.err && grep "2 tasks" trace.out '
@@ -127,7 +152,7 @@ test_expect_success "using fetchRecurseSubmodules=true in .gitmodules recurses i git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--no-recurse-submodules overrides .gitmodules config" '
@@ -158,7 +183,7 @@ test_expect_success "--recurse-submodules overrides fetchRecurseSubmodules setti git config --unset submodule.submodule.fetchRecurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--quiet propagates to submodules" '
@@ -186,7 +211,7 @@ test_expect_success "--dry-run propagates to submodules" ' git fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "Without --dry-run propagates to submodules" '
@@ -195,7 +220,7 @@ test_expect_success "Without --dry-run propagates to submodules" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "recurseSubmodules=true propagates into submodules" '
@@ -206,7 +231,7 @@ test_expect_success "recurseSubmodules=true propagates into submodules" ' git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--recurse-submodules overrides config in submodule" '
@@ -220,7 +245,7 @@ test_expect_success "--recurse-submodules overrides config in submodule" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--no-recurse-submodules overrides config setting" '
@@ -253,14 +278,14 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.sub && - echo " $head1..$head2 super -> origin/super" >>expect.err.sub && - head -3 expect.err >> expect.err.sub && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && - test_cmp expect.err.sub actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -271,14 +296,16 @@ test_expect_success "Recursion doesn't happen when new superproject commits don' git add file && git commit -m "new file" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.file && - echo " $head1..$head2 super -> origin/super" >> expect.err.file && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + rm expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err.file actual.err + verify_fetch_result actual.err ' test_expect_success "Recursion picks up config in submodule" '
@@ -295,9 +322,8 @@ test_expect_success "Recursion picks up config in submodule" ' git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.sub && - echo " $head1..$head2 super -> origin/super" >> expect.err.sub && - cat expect.err >> expect.err.sub && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && ( cd downstream && git fetch >../actual.out 2>../actual.err &&
@@ -306,7 +332,7 @@ test_expect_success "Recursion picks up config in submodule" ' git config --unset fetch.recurseSubmodules ) ) && - test_cmp expect.err.sub actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -331,15 +357,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >> expect.err.2 && - cat expect.err.sub >> expect.err.2 && - tail -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && - test_cmp expect.err.2 actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -375,11 +399,8 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - tail -3 expect.err > expect.err.deepsub && - echo "From $pwd/." > expect.err && - echo " $head1..$head2 super -> origin/super" >>expect.err && - cat expect.err.sub >> expect.err && - cat expect.err.deepsub >> expect.err && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -395,7 +416,7 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess ) ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" '
@@ -405,14 +426,16 @@ test_expect_success "'--recurse-submodules=on-demand' stops when no new submodul git add file && git commit -m "new file" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.file && - echo " $head1..$head2 super -> origin/super" >> expect.err.file && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + rm expect.err.deep && ( cd downstream && git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err.file actual.err + verify_fetch_result actual.err ' test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config" '
@@ -426,9 +449,9 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git config fetch.recurseSubmodules on-demand &&
@@ -440,7 +463,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config git config --unset fetch.recurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err + verify_fetch_result actual.err ' test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules" '
@@ -454,9 +477,9 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git config submodule.submodule.fetchRecurseSubmodules on-demand &&
@@ -468,7 +491,7 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override git config --unset submodule.submodule.fetchRecurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err + verify_fetch_result actual.err ' test_expect_success "don't fetch submodule when newly recorded commits are already present" '
@@ -480,14 +503,17 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea git add submodule && git commit -m "submodule rewound" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err && - echo " $head1..$head2 super -> origin/super" >> expect.err && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + # This file does not exist, but rm -f for readability + rm -f expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err && + verify_fetch_result actual.err && ( cd submodule && git checkout -q sub
@@ -505,9 +531,9 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git rm .gitmodules && git commit -m "new submodule without .gitmodules" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." >expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >>expect.err.2 && + echo "From $pwd/." >expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && rm .gitmodules &&
@@ -523,7 +549,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git reset --hard ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err && + verify_fetch_result actual.err && git checkout HEAD^ -- .gitmodules && git add .gitmodules && git commit -m "new submodule restored .gitmodules"
Signature · 1 line
2.33.GIT
Re: [PATCH v2 1/9] t5526: introduce test helper to assert on fetches
Quoted 4 lines · Glen Choo
(quoted diff) +4
-0+verify_fetch_result() { + ACTUAL_ERR=$1 && + rm -f expect.err.combined && + if [ -f expect.err.super ]; then
Review
Quoted 10 lines
(quoted diff) +10
-0+ cat expect.err.super >>expect.err.combined + fi && + if [ -f expect.err.sub ]; then + cat expect.err.sub >>expect.err.combined + fi && + if [ -f expect.err.deep ]; then + cat expect.err.deep >>expect.err.combined + fi && + test_cmp expect.err.combined $ACTUAL_ERR +}
I see this will get further modified in this series, but I wondered why not just something like:
t/t5526-fetch-submodules.sh +6 -10
@@ -49,16 +49,12 @@ add_upstream_commit() { # expect.err file should be rm-ed. verify_fetch_result() { ACTUAL_ERR=$1 && - rm -f expect.err.combined && - if [ -f expect.err.super ]; then - cat expect.err.super >>expect.err.combined - fi && - if [ -f expect.err.sub ]; then - cat expect.err.sub >>expect.err.combined - fi && - if [ -f expect.err.deep ]; then - cat expect.err.deep >>expect.err.combined - fi && + + { + cat expect.err.super + cat expect.err.sub + cat expect.err.deep + } >expect.err.combined test_cmp expect.err.combined $ACTUAL_ERR }
I.e. there's no law that we've got to avoid non-zero on every exit code. In this case we can avoid the existence checks and just "cat" them together, and if it fails we'll presumably fail on the test_cmp anyway...
PATCH v2 2/9 t5526: use grep to assert on fetches
In reply to Glen ChooIn a previous commit, we replaced test_cmp invocations with verify_fetch_result(). Finish the process of removing test_cmp by using grep in verify_fetch_result() instead.
This makes the tests less sensitive to changes because, instead of checking the whole stderr, we only grep for the lines of the form
* "<old-head>..<new-head>\s+branch\s+-> origin/branch"
* "Fetching submodule <submodule-path>" (if fetching a submodule)
when we expect the repo to have fetched. If we expect the repo to not have fetched, grep to make sure the lines are absent. Also, simplify the assertions by using grep patterns that match only the relevant pieces of information, e.g. <old-head> is irrelevant because we only want to know if the fetch was performed, so we don't need to know where the branch was before the fetch.
--- t/t5526-fetch-submodules.sh | 131 +++++++++++++----------------------- 1 file changed, 48 insertions(+), 83 deletions(-)
t/t5526-fetch-submodules.sh +48 -83
@@ -20,49 +20,52 @@ pwd=$(pwd) add_upstream_commit() { ( cd submodule && - head1=$(git rev-parse --short HEAD) && echo new >> subfile && test_tick && git add subfile && git commit -m new subfile && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && ( cd deepsubmodule && - head1=$(git rev-parse --short HEAD) && echo new >> deepsubfile && test_tick && git add deepsubfile && git commit -m new deepsubfile && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep - echo "From $pwd/deepsubmodule" >> ../expect.err.deep && - echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep + git rev-parse --short HEAD >../deephead ) } # Verifies that the expected repositories were fetched. This is done by -# concatenating the files expect.err.[super|sub|deep] in the correct -# order and comparing it to the actual stderr. +# checking that the branches of [super|sub|deep] were updated to +# [super|sub|deep]head if the corresponding file exists. # -# If a repo should not be fetched in the test, its corresponding -# expect.err file should be rm-ed. +# If the [super|sub|deep] head file does not exist, this verifies that +# the corresponding repo was not fetched. Thus, if a repo should not be +# fetched in the test, its corresponding head file should be +# rm-ed. verify_fetch_result() { ACTUAL_ERR=$1 && - rm -f expect.err.combined && - if [ -f expect.err.super ]; then - cat expect.err.super >>expect.err.combined + # Each grep pattern is guaranteed to match the correct repo + # because each repo uses a different name for their branch i.e. + # "super", "sub" and "deep". + if [ -f superhead ]; then + grep -E "\.\.$(cat superhead)\s+super\s+-> origin/super" $ACTUAL_ERR + else + ! grep "super" $ACTUAL_ERR fi && - if [ -f expect.err.sub ]; then - cat expect.err.sub >>expect.err.combined + if [ -f subhead ]; then + grep "Fetching submodule submodule" $ACTUAL_ERR && + grep -E "\.\.$(cat subhead)\s+sub\s+-> origin/sub" $ACTUAL_ERR + else + ! grep "Fetching submodule submodule" $ACTUAL_ERR fi && - if [ -f expect.err.deep ]; then - cat expect.err.deep >>expect.err.combined - fi && - test_cmp expect.err.combined $ACTUAL_ERR + if [ -f deephead ]; then + grep "Fetching submodule submodule/subdir/deepsubmodule" $ACTUAL_ERR && + grep -E "\.\.$(cat deephead)\s+deep\s+-> origin/deep" $ACTUAL_ERR + else + ! grep "Fetching submodule submodule/subdir/deepsubmodule" $ACTUAL_ERR + fi } test_expect_success setup '
@@ -274,13 +277,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in ' test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm deephead && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -291,15 +291,12 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" test_expect_success "Recursion doesn't happen when new superproject commits don't change any submodules" ' add_upstream_commit && - head1=$(git rev-parse --short HEAD) && echo a > file && git add file && git commit -m "new file" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && - rm expect.err.sub && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm subhead && + rm deephead && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -318,12 +315,9 @@ test_expect_success "Recursion picks up config in submodule" ' ) ) && add_upstream_commit && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git fetch >../actual.out 2>../actual.err &&
@@ -345,20 +339,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -376,13 +363,9 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo Fetching submodule submodule > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && ( cd downstream &&
@@ -395,12 +378,9 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne ' test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -421,15 +401,12 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" ' add_upstream_commit && - head1=$(git rev-parse --short HEAD) && echo a >> file && git add file && git commit -m "new file" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && - rm expect.err.sub && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm subhead && + rm deephead && ( cd downstream && git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err
@@ -445,13 +422,10 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config ) && add_upstream_commit && git config --global fetch.recurseSubmodules false && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm deephead && ( cd downstream && git config fetch.recurseSubmodules on-demand &&
@@ -473,13 +447,10 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override ) && add_upstream_commit && git config fetch.recurseSubmodules false && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm deephead && ( cd downstream && git config submodule.submodule.fetchRecurseSubmodules on-demand &&
@@ -499,15 +470,12 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea cd submodule && git checkout -q HEAD^^ ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "submodule rewound" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && - rm expect.err.sub && + git rev-parse --short HEAD >superhead && + rm subhead && # This file does not exist, but rm -f for readability - rm -f expect.err.deep && + rm -f deephead && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -526,14 +494,11 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git fetch --recurse-submodules ) && add_upstream_commit && - head1=$(git rev-parse --short HEAD) && git add submodule && git rm .gitmodules && git commit -m "new submodule without .gitmodules" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." >expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && - rm expect.err.deep && + git rev-parse --short HEAD >superhead && + rm deephead && ( cd downstream && rm .gitmodules &&
Signature · 1 line
2.33.GIT
Re: [PATCH v2 2/9] t5526: use grep to assert on fetches
Quoted 10 lines · Glen Choo
In a previous commit, we replaced
test_cmpinvocations withverify_fetch_result(). Finish the process of removingtest_cmpby using grep inverify_fetch_result()instead.This makes the tests less sensitive to changes because, instead of checking the whole stderr, we only grep for the lines of the form
* "<old-head>..<new-head>\s+branch\s+-> origin/branch"
* "Fetching submodule <submodule-path>" (if fetching a submodule)
Quoted 6 lines
when we expect the repo to have fetched. If we expect the repo to not have fetched, grep to make sure the lines are absent. Also, simplify the assertions by using grep patterns that match only the relevant pieces of information, e.g. <old-head> is irrelevant because we only want to know if the fetch was performed, so we don't need to know where the branch was before the fetch.
I tried ejecting 1/9 and 2/9 out of this series locally, and it passes all tests until the new tests you add in 7/9.
As ugly as some of the pre-image is, I wonder if dropping these first two and biting the bullet and just continuing with the test_cmp is better.
The test_cmp is going to catch issues that even the cleverest grep combinations won't, e.g. in the submodule-in-C series I discovered a bug where all of our testing & the existing series hadn't spotted that we were dropping a \n at the end in one of the messages.
Particularly as:
Quoted 34 lines
# Verifies that the expected repositories were fetched. This is done by
t/t5526-fetch-submodules.sh +18
-10-# concatenating the files expect.err.[super|sub|deep] in the correct -# order and comparing it to the actual stderr. +# checking that the branches of [super|sub|deep] were updated to +# [super|sub|deep]head if the corresponding file exists. # -# If a repo should not be fetched in the test, its corresponding -# expect.err file should be rm-ed. +# If the [super|sub|deep] head file does not exist, this verifies that +# the corresponding repo was not fetched. Thus, if a repo should not be +# fetched in the test, its corresponding head file should be +# rm-ed. verify_fetch_result() { ACTUAL_ERR=$1 && - rm -f expect.err.combined && - if [ -f expect.err.super ]; then - cat expect.err.super >>expect.err.combined + # Each grep pattern is guaranteed to match the correct repo + # because each repo uses a different name for their branch i.e. + # "super", "sub" and "deep". + if [ -f superhead ]; then + grep -E "\.\.$(cat superhead)\s+super\s+-> origin/super" $ACTUAL_ERR + else + ! grep "super" $ACTUAL_ERR fi && - if [ -f expect.err.sub ]; then - cat expect.err.sub >>expect.err.combined + if [ -f subhead ]; then + grep "Fetching submodule submodule" $ACTUAL_ERR && + grep -E "\.\.$(cat subhead)\s+sub\s+-> origin/sub" $ACTUAL_ERR + else + ! grep "Fetching submodule submodule" $ACTUAL_ERR fi && - if [ -f expect.err.deep ]; then
t/t5526-fetch-submodules.sh pwd=$(pwd) · line 69
Quoted 10 lines
t/t5526-fetch-submodules.sh +6
-3- cat expect.err.deep >>expect.err.combined - fi && - test_cmp expect.err.combined $ACTUAL_ERR + if [ -f deephead ]; then + grep "Fetching submodule submodule/subdir/deepsubmodule" $ACTUAL_ERR && + grep -E "\.\.$(cat deephead)\s+deep\s+-> origin/deep" $ACTUAL_ERR + else + ! grep "Fetching submodule submodule/subdir/deepsubmodule" $ACTUAL_ERR + fi }
This sort of thing is really hard to understand and review...
t/t5526-fetch-submodules.sh test_expect_success "Recursion doesn't happen when no new commits are fetched in · line 282
Quoted 11 lines
test_expect_successsetup 't/t5526-fetch-submodules.sh +0
-4@@ -274,13 +277,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in ' test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super &&
...as opposed to if we just rolled the generation of this into some utility printf function.
Re: [PATCH v2 2/9] t5526: use grep to assert on fetches
Quoted 22 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines · Glen Choo
In a previous commit, we replaced
test_cmpinvocations withverify_fetch_result(). Finish the process of removingtest_cmpby using grep inverify_fetch_result()instead.This makes the tests less sensitive to changes because, instead of checking the whole stderr, we only grep for the lines of the form
* "<old-head>..<new-head>\s+branch\s+-> origin/branch"
* "Fetching submodule <submodule-path>" (if fetching a submodule)Quoted 6 lines
when we expect the repo to have fetched. If we expect the repo to not have fetched, grep to make sure the lines are absent. Also, simplify the assertions by using grep patterns that match only the relevant pieces of information, e.g. <old-head> is irrelevant because we only want to know if the fetch was performed, so we don't need to know where the branch was before the fetch.
I tried ejecting 1/9 and 2/9 out of this series locally, and it passes all tests until the new tests you add in 7/9.
Quoted 8 lines
As ugly as some of the pre-image is, I wonder if dropping these first two and biting the bullet and just continuing with the
test_cmpis better.The
test_cmpis going to catch issues that even the cleverest grep combinations won't, e.g. in the submodule-in-C series I discovered a bug where all of our testing & the existing series hadn't spotted that we were dropping a \n at the end in one of the messages.
I think there are two schools of thought on how to test informational messages:
- assert an exact match on the exact output that we generate
- assert that the output contains the pieces of information we care about
These two approaches are virtually opposites on two axes - the former will catch unintentional changes (like you've noted) and the latter saves on maintenance effort and tends to be less noisy in tests.
Personally, I'm a bit torn between both approaches in general because I want tests to be maintainable (testing the exact output is a bit of an antipattern at Google), but I'm not very comfortable with the fact that unintended changes can sneak through.
So I don't think there's a correct answer in general. Maybe an acceptable rule of thumb is that test_cmp is good until it starts getting in the way of reading and writing understandable tests.
If we agree on that rule, then for this patch, I think replacing test_cmp is the way to go, primarily because it lets us ignore the 'old head' of the branch before the fetch, e.g. in the quoted example..
Quoted 4 lines
Quoted 4 lines
test_expect_successsetup '(quoted diff) +0
-0@@ -274,13 +277,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in '
Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success "Recursion doesn't happen when no new commits are fetched in · line 282
Quoted 7 lines
t/t5526-fetch-submodules.sh +0
-4test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super &&...as opposed to if we just rolled the generation of this into some utility printf function.
we'd still have to deal with $head1 if we use test_cmp. That's fine for this test, because it's pretty simple, but it gets pretty janky later on:
@@ -345,20 +339,13 @@ test_expect_success "Recursion picks up all submodules when necessary" '
git fetch &&
git checkout -q FETCH_HEAD
) &&
- head1=$(git rev-parse --short HEAD^) &&
git add subdir/deepsubmodule &&
git commit -m "new deepsubmodule" &&
- head2=$(git rev-parse --short HEAD) &&
- echo "Fetching submodule submodule" > ../expect.err.sub &&
- echo "From $pwd/submodule" >> ../expect.err.sub &&
- echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub
+ git rev-parse --short HEAD >../subhead
) &&
- head1=$(git rev-parse --short HEAD) &&
git add submodule &&
git commit -m "new submodule" &&
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >> expect.err.super &&
+ git rev-parse --short HEAD >superhead &&
(
cd downstream &&
git fetch >../actual.out 2>../actual.errIn this example, we have two $head1 variables in different subshells, one of which is HEAD, but the other is HEAD^. The reason why we want HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2 commits ahead because we add the deepsubmodule in a separate commit), in my opinion, and I got tripped up quite a few times trying to read and understand the test. That's a lot of effort to spend on irrelevant information - the test actually cares about what it fetched, not where the ref used to be.
So for that reason, I'd prefer to remove test_cmp for this test.
Re: [PATCH v2 2/9] t5526: use grep to assert on fetches
Quoted 36 lines · Glen Choo
Quoted 22 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines · Glen Choo
In a previous commit, we replaced
test_cmpinvocations withverify_fetch_result(). Finish the process of removingtest_cmpby using grep inverify_fetch_result()instead.This makes the tests less sensitive to changes because, instead of checking the whole stderr, we only grep for the lines of the form
* "<old-head>..<new-head>\s+branch\s+-> origin/branch"
* "Fetching submodule <submodule-path>" (if fetching a submodule)Quoted 6 lines
when we expect the repo to have fetched. If we expect the repo to not have fetched, grep to make sure the lines are absent. Also, simplify the assertions by using grep patterns that match only the relevant pieces of information, e.g. <old-head> is irrelevant because we only want to know if the fetch was performed, so we don't need to know where the branch was before the fetch.
I tried ejecting 1/9 and 2/9 out of this series locally, and it passes all tests until the new tests you add in 7/9.
Quoted 8 lines
As ugly as some of the pre-image is, I wonder if dropping these first two and biting the bullet and just continuing with the
test_cmpis better.The
test_cmpis going to catch issues that even the cleverest grep combinations won't, e.g. in the submodule-in-C series I discovered a bug where all of our testing & the existing series hadn't spotted that we were dropping a \n at the end in one of the messages.I think there are two schools of thought on how to test informational messages:
Quoted 6 lines
- assert an exact match on the exact output that we generate
- assert that the output contains the pieces of information we care aboutThese two approaches are virtually opposites on two axes - the former will catch unintentional changes (like you've noted)[...]
Yes, and to be fair I'm thoroughly in the "assert an exact match" camp, i.e. "let's just use test_cmp", and not everyone would agree with that.
I mean, I don't think we should test_cmp every single instance of a command, but for things that are *the tests* concerning themselves with what the output should be, yes we should do that.
Quoted 1 line
[...] and the latter saves on maintenance effort and tends to be less noisy in tests.
I also don't think you're right about the other approach "sav[ing] on [future] maintenance effort" in this case.
If I was needing to adjust some of this output I'd spend way longer on trying to carefully reason that some series of "grep" invocations were really doing the right thing, and probably end up doing the equivalent of a "test_cmp" for myself out of general paranoia, whereas adjusting the output.
Whereas adjusting the code, running the tests, and looking at the "diff -u" failures from "test_cmp" and adjusting the output is an easy matter of copy/pasting.
Then reviewers can just see what's a clear human-readable change, e.g. imagine reviewing a patch where we start trying to aligning something in the output where the patch has a pretty much 1=1 (test_cmp) mapping to the before/after, v.s. doing the same with whatever "grep" regex we wind up with.
Quoted 4 lines
Personally, I'm a bit torn between both approaches in general because I want tests to be maintainable (testing the exact output is a bit of an antipattern at Google), but I'm not very comfortable with the fact that unintended changes can sneak through.
Yes, anyway whatever one thinks in general what I meant to point out here with "biting the bullet" is that whatever one thinks in general about the right approch for new tests, this series in particular seems to be creating more work for itself than it needs by refactoring the test_cmp in existing tests just to add a few new ones.
I.e. even if you'd like to not use test_cmp-alike for the new tests, wouldn't it be simpler to just leave the old ones in place and use your new helper for your new tests?
Quoted 7 lines
So I don't think there's a correct answer in general. Maybe an acceptable rule of thumb is that
test_cmpis good until it starts getting in the way of reading and writing understandable tests.If we agree on that rule, then for this patch, I think replacing
test_cmpis the way to go, primarily because it lets us ignore the 'old head' of the branch before the fetch, e.g. in the quoted example..
[...]
Quoted 52 lines
Quoted 4 lines
Quoted 4 lines
test_expect_successsetup '(quoted diff) +0
-0@@ -274,13 +277,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in 'Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success "Recursion doesn't happen when no new commits are fetched in · line 282
Quoted 7 lines
t/t5526-fetch-submodules.sh +0
-4test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super &&...as opposed to if we just rolled the generation of this into some utility printf function.
we'd still have to deal with $head1 if we use
test_cmp. That's fine for this test, because it's pretty simple, but it gets pretty janky later on:@@ -345,20 +339,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git fetch >../actual.out 2>../actual.errIn this example, we have two $head1 variables in different subshells, one of which is HEAD, but the other is HEAD^. The reason why we want HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2 commits ahead because we add the deepsubmodule in a separate commit), in my opinion, and I got tripped up quite a few times trying to read and understand the test. That's a lot of effort to spend on irrelevant information - the test actually cares about what it fetched, not where the ref used to be.
Quoted 1 line
So for that reason, I'd prefer to remove
test_cmpfor this test.
I agree that it's pretty irrelevant, but I also think we'd be throwing the baby out with the bath water by entirely doing away with test_cmp here, there's an easier way to do this.
I.e. none of these tests surely need to test that we updated from $head1..$head2 again and again with the corresponding verbosity in test setup and shelling out to "git rev-parse --short HEAD" or whatever.
Instead let's just test once somewhere that when we run submodule fetching that submodules are indeed updated appropriately. Surely other submodule tests will break if the "update" code is made to NOOP, or update to the wrong HEAD>
Then for all these test_cmp tests we can just sed-away the $head1..$head2 with something like (untested):
sed -n -e 's/[^.]*\.\.[^.]*/OLD..NEW/g'
I.e. let's just skip this entire ceremony with asserting the old/new HEAD unless it's really needed (and then we can probably do it once outside a test_cmp).
If you grep through the test suite for "sed" adjacent to "test_cmp" you'll find a lot of such examples of munging the output before test_cmp-ing it.
That's perfectly fine here, since the actual point of the test_cmp is to check the formatting/order etc. of the output itself, not to continually re-assert that submodule updating still works, and that we get the right OIDs.
Re: [PATCH v2 2/9] t5526: use grep to assert on fetches
Quoted 45 lines · Ævar Arnfjörð Bjarmason
Quoted 36 lines · Glen Choo
Quoted 22 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines · Glen Choo
In a previous commit, we replaced
test_cmpinvocations withverify_fetch_result(). Finish the process of removingtest_cmpby using grep inverify_fetch_result()instead.This makes the tests less sensitive to changes because, instead of checking the whole stderr, we only grep for the lines of the form
* "<old-head>..<new-head>\s+branch\s+-> origin/branch"
* "Fetching submodule <submodule-path>" (if fetching a submodule)Quoted 6 lines
when we expect the repo to have fetched. If we expect the repo to not have fetched, grep to make sure the lines are absent. Also, simplify the assertions by using grep patterns that match only the relevant pieces of information, e.g. <old-head> is irrelevant because we only want to know if the fetch was performed, so we don't need to know where the branch was before the fetch.
I tried ejecting 1/9 and 2/9 out of this series locally, and it passes all tests until the new tests you add in 7/9.
Quoted 8 lines
As ugly as some of the pre-image is, I wonder if dropping these first two and biting the bullet and just continuing with the
test_cmpis better.The
test_cmpis going to catch issues that even the cleverest grep combinations won't, e.g. in the submodule-in-C series I discovered a bug where all of our testing & the existing series hadn't spotted that we were dropping a \n at the end in one of the messages.I think there are two schools of thought on how to test informational messages:
Quoted 6 lines
- assert an exact match on the exact output that we generate
- assert that the output contains the pieces of information we care aboutThese two approaches are virtually opposites on two axes - the former will catch unintentional changes (like you've noted)[...]
Quoted 6 lines
Yes, and to be fair I'm thoroughly in the "assert an exact match" camp, i.e. "let's just use
test_cmp", and not everyone would agree with that.I mean, I don't think we should
test_cmpevery single instance of a command, but for things that are *the tests* concerning themselves with what the output should be, yes we should do that.
That's a good point I hadn't considered, which is that if we want any hope of catching unintentional changes in our test suite, we'd want _some_ test to check the output. For "git fetch --recurse-submodules", it makes the most sense for that test to live in this file.
By eliminating all instances of test_cmp in this file in particular, we lose assurances that we don't introduce accidental changes. It makes sense to at least have some tests explicitly for output.
Quoted 11 lines
Quoted 2 lines
[...] and the latter saves on maintenance effort and tends to be less noisy in tests.
I also don't think you're right about the other approach "sav[ing] on [future] maintenance effort" in this case.
If I was needing to adjust some of this output I'd spend way longer on trying to carefully reason that some series of "grep" invocations were really doing the right thing, and probably end up doing the equivalent of a "
test_cmp" for myself out of general paranoia, whereas adjusting the output.
That's fair. I've optimized the tests for readability by putting complicated logic in the test helper. But any diligent test reader would need to read the test helper to convince themselves of its correctness. In this case, I agree that the helper is too complex.
Quoted 11 lines
Quoted 4 lines
Personally, I'm a bit torn between both approaches in general because I want tests to be maintainable (testing the exact output is a bit of an antipattern at Google), but I'm not very comfortable with the fact that unintended changes can sneak through.
Yes, anyway whatever one thinks in general what I meant to point out here with "biting the bullet" is that whatever one thinks in general about the right approch for new tests, this series in particular seems to be creating more work for itself than it needs by refactoring the
test_cmpin existing tests just to add a few new ones.
Quoted 3 lines
I.e. even if you'd like to not use test_cmp-alike for the new tests, wouldn't it be simpler to just leave the old ones in place and use your new helper for your new tests?
I'm not sure about this - avoiding changing old tests leads to fragmentation in the test suite and even the same file. I find it very challenging to read/modify files like this, because there is no longer a consistent style for the file, and I have to figure out which is the "good" way to write tests.
This suggestion makes sense if there's some qualitative difference between the new tests and old ones besides just 'being new'. This isn't true for this series, so I'd prefer to keep things consistent.
Quoted 67 lines
Quoted 7 lines
So I don't think there's a correct answer in general. Maybe an acceptable rule of thumb is that
test_cmpis good until it starts getting in the way of reading and writing understandable tests.If we agree on that rule, then for this patch, I think replacing
test_cmpis the way to go, primarily because it lets us ignore the 'old head' of the branch before the fetch, e.g. in the quoted example..[...]
Quoted 52 lines
Quoted 4 lines
Quoted 4 lines
test_expect_successsetup '(quoted diff) +0
-0@@ -274,13 +277,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in 'Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success "Recursion doesn't happen when no new commits are fetched in · line 282
Quoted 7 lines
t/t5526-fetch-submodules.sh +0
-4test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super &&...as opposed to if we just rolled the generation of this into some utility printf function.
we'd still have to deal with $head1 if we use
test_cmp. That's fine for this test, because it's pretty simple, but it gets pretty janky later on:@@ -345,20 +339,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git fetch >../actual.out 2>../actual.errIn this example, we have two $head1 variables in different subshells, one of which is HEAD, but the other is HEAD^. The reason why we want HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2 commits ahead because we add the deepsubmodule in a separate commit), in my opinion, and I got tripped up quite a few times trying to read and understand the test. That's a lot of effort to spend on irrelevant information - the test actually cares about what it fetched, not where the ref used to be.
Quoted 1 line
So for that reason, I'd prefer to remove
test_cmpfor this test.I agree that it's pretty irrelevant, but I also think we'd be throwing the baby out with the bath water by entirely doing away with
test_cmphere, there's an easier way to do this.
[...]
Quoted 10 lines
Instead let's just test once somewhere that when we run submodule fetching that submodules are indeed updated appropriately. Surely other submodule tests will break if the "update" code is made to NOOP, or update to the wrong HEAD>
Then for all these
test_cmptests we can just sed-away the $head1..$head2 with something like (untested):sed -n -e 's/[^.]*\.\.[^.]*/OLD..NEW/g'
Quoted 7 lines
I.e. let's just skip this entire ceremony with asserting the old/new HEAD unless it's really needed (and then we can probably do it once outside a
test_cmp).If you grep through the test suite for "sed" adjacent to "
test_cmp" you'll find a lot of such examples of munging the output before test_cmp-ing it.
Makes sense, I hadn't considered this (though I have seen the pattern in the test suite, oops). The most compelling argument in favor of this is that this could remove a lot of the complexity of verify_fetch_result(), which is impeding test readability.
Quoted 3 lines
I.e. none of these tests surely need to test that we updated from $head1..$head2 again and again with the corresponding verbosity in test setup and shelling out to "git rev-parse --short HEAD" or whatever.
I find the converse (we are testing the formatting over and over again) less convincing. In these tests, we really are checking for $head2 in the stderr to verify that the correct thing was fetched. I'm not convinced that we should be relying on _other_ submodule tests to tell us that submodule fetch is broken. Which brings me back to the original motivation of this patch..
Quoted 5 lines
That's perfectly fine here, since the actual point of the
test_cmpis to check the formatting/order etc. of the output itself, not to continually re-assert that submodule updating still works, and that we get the right OIDs.
which is that these tests actually are continually re-asserting the submodule updating works correctly in the different circumstances, and since we use the stderr to check this, test_cmp adds unwarranted noise.
But you are correct in that the point of test_cmp is to check formatting/order etc. There is value in using test_cmp for this purpose, and getting rid of it entirely creates a hole in our test coverage. (This wouldn't mean that we'd need to use test_cmp _everywhere_ though, only that we need to use test_cmp _somewhere_.)
As it stands:
+ test_cmp can improve the readability of the test helpers and
debuggability of tests vs grep
+/- test_cmp can catch formatting changes that are hard to catch
otherwise, though at the cost of being sensitive to _any_ formatting changes
- test_cmp needs some munging to eliminate unnecessary information
so on the whole, I think it's worth trying to use test_cmp in the test helper. We may not _need_ it everywhere, but if it would be nice to use it in as many places as possible.
Re: [PATCH v2 2/9] t5526: use grep to assert on fetches
Quoted 232 lines · Glen Choo
Quoted 45 lines · Ævar Arnfjörð Bjarmason
Quoted 36 lines · Glen Choo
Quoted 22 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines · Glen Choo
In a previous commit, we replaced
test_cmpinvocations withverify_fetch_result(). Finish the process of removingtest_cmpby using grep inverify_fetch_result()instead.This makes the tests less sensitive to changes because, instead of checking the whole stderr, we only grep for the lines of the form
* "<old-head>..<new-head>\s+branch\s+-> origin/branch"
* "Fetching submodule <submodule-path>" (if fetching a submodule)Quoted 6 lines
when we expect the repo to have fetched. If we expect the repo to not have fetched, grep to make sure the lines are absent. Also, simplify the assertions by using grep patterns that match only the relevant pieces of information, e.g. <old-head> is irrelevant because we only want to know if the fetch was performed, so we don't need to know where the branch was before the fetch.
I tried ejecting 1/9 and 2/9 out of this series locally, and it passes all tests until the new tests you add in 7/9.
Quoted 8 lines
As ugly as some of the pre-image is, I wonder if dropping these first two and biting the bullet and just continuing with the
test_cmpis better.The
test_cmpis going to catch issues that even the cleverest grep combinations won't, e.g. in the submodule-in-C series I discovered a bug where all of our testing & the existing series hadn't spotted that we were dropping a \n at the end in one of the messages.I think there are two schools of thought on how to test informational messages:
Quoted 6 lines
- assert an exact match on the exact output that we generate
- assert that the output contains the pieces of information we care aboutThese two approaches are virtually opposites on two axes - the former will catch unintentional changes (like you've noted)[...]
Quoted 6 lines
Yes, and to be fair I'm thoroughly in the "assert an exact match" camp, i.e. "let's just use
test_cmp", and not everyone would agree with that.I mean, I don't think we should
test_cmpevery single instance of a command, but for things that are *the tests* concerning themselves with what the output should be, yes we should do that.That's a good point I hadn't considered, which is that if we want any hope of catching unintentional changes in our test suite, we'd want _some_ test to check the output. For "git fetch --recurse-submodules", it makes the most sense for that test to live in this file.
By eliminating all instances of
test_cmpin this file in particular, we lose assurances that we don't introduce accidental changes. It makes sense to at least have some tests explicitly for output.Quoted 11 lines
Quoted 2 lines
[...] and the latter saves on maintenance effort and tends to be less noisy in tests.
I also don't think you're right about the other approach "sav[ing] on [future] maintenance effort" in this case.
If I was needing to adjust some of this output I'd spend way longer on trying to carefully reason that some series of "grep" invocations were really doing the right thing, and probably end up doing the equivalent of a "
test_cmp" for myself out of general paranoia, whereas adjusting the output.That's fair. I've optimized the tests for readability by putting complicated logic in the test helper. But any diligent test reader would need to read the test helper to convince themselves of its correctness. In this case, I agree that the helper is too complex.
Quoted 11 lines
Quoted 4 lines
Personally, I'm a bit torn between both approaches in general because I want tests to be maintainable (testing the exact output is a bit of an antipattern at Google), but I'm not very comfortable with the fact that unintended changes can sneak through.
Yes, anyway whatever one thinks in general what I meant to point out here with "biting the bullet" is that whatever one thinks in general about the right approch for new tests, this series in particular seems to be creating more work for itself than it needs by refactoring the
test_cmpin existing tests just to add a few new ones.Quoted 3 lines
I.e. even if you'd like to not use test_cmp-alike for the new tests, wouldn't it be simpler to just leave the old ones in place and use your new helper for your new tests?
I'm not sure about this - avoiding changing old tests leads to fragmentation in the test suite and even the same file. I find it very challenging to read/modify files like this, because there is no longer a consistent style for the file, and I have to figure out which is the "good" way to write tests.
This suggestion makes sense if there's some qualitative difference between the new tests and old ones besides just 'being new'. This isn't true for this series, so I'd prefer to keep things consistent.
Quoted 67 lines
Quoted 7 lines
So I don't think there's a correct answer in general. Maybe an acceptable rule of thumb is that
test_cmpis good until it starts getting in the way of reading and writing understandable tests.If we agree on that rule, then for this patch, I think replacing
test_cmpis the way to go, primarily because it lets us ignore the 'old head' of the branch before the fetch, e.g. in the quoted example..[...]
Quoted 52 lines
Quoted 4 lines
Quoted 4 lines
test_expect_successsetup '(quoted diff) +0
-0@@ -274,13 +277,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in 'Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success "Recursion doesn't happen when no new commits are fetched in · line 282
Quoted 7 lines
t/t5526-fetch-submodules.sh +0
-4test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super &&...as opposed to if we just rolled the generation of this into some utility printf function.
we'd still have to deal with $head1 if we use
test_cmp. That's fine for this test, because it's pretty simple, but it gets pretty janky later on:@@ -345,20 +339,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git fetch >../actual.out 2>../actual.errIn this example, we have two $head1 variables in different subshells, one of which is HEAD, but the other is HEAD^. The reason why we want HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2 commits ahead because we add the deepsubmodule in a separate commit), in my opinion, and I got tripped up quite a few times trying to read and understand the test. That's a lot of effort to spend on irrelevant information - the test actually cares about what it fetched, not where the ref used to be.
Quoted 1 line
So for that reason, I'd prefer to remove
test_cmpfor this test.I agree that it's pretty irrelevant, but I also think we'd be throwing the baby out with the bath water by entirely doing away with
test_cmphere, there's an easier way to do this.[...]
Quoted 10 lines
Instead let's just test once somewhere that when we run submodule fetching that submodules are indeed updated appropriately. Surely other submodule tests will break if the "update" code is made to NOOP, or update to the wrong HEAD>
Then for all these
test_cmptests we can just sed-away the $head1..$head2 with something like (untested):sed -n -e 's/[^.]*\.\.[^.]*/OLD..NEW/g'Quoted 7 lines
I.e. let's just skip this entire ceremony with asserting the old/new HEAD unless it's really needed (and then we can probably do it once outside a
test_cmp).If you grep through the test suite for "sed" adjacent to "
test_cmp" you'll find a lot of such examples of munging the output before test_cmp-ing it.Makes sense, I hadn't considered this (though I have seen the pattern in the test suite, oops). The most compelling argument in favor of this is that this could remove a lot of the complexity of
verify_fetch_result(), which is impeding test readability.Quoted 3 lines
I.e. none of these tests surely need to test that we updated from $head1..$head2 again and again with the corresponding verbosity in test setup and shelling out to "git rev-parse --short HEAD" or whatever.
I find the converse (we are testing the formatting over and over again) less convincing. In these tests, we really are checking for $head2 in the stderr to verify that the correct thing was fetched. I'm not convinced that we should be relying on _other_ submodule tests to tell us that submodule fetch is broken. Which brings me back to the original motivation of this patch..
Quoted 5 lines
That's perfectly fine here, since the actual point of the
test_cmpis to check the formatting/order etc. of the output itself, not to continually re-assert that submodule updating still works, and that we get the right OIDs.which is that these tests actually are continually re-asserting the submodule updating works correctly in the different circumstances, and since we use the stderr to check this,
test_cmpadds unwarranted noise.But you are correct in that the point of
test_cmpis to check formatting/order etc. There is value in usingtest_cmpfor this purpose, and getting rid of it entirely creates a hole in our test coverage. (This wouldn't mean that we'd need to usetest_cmp_everywhere_ though, only that we need to usetest_cmp_somewhere_.)As it stands:
(quoted diff) +2
-1+ test_cmp can improve the readability of the test helpers and debuggability of tests vs grep +/- test_cmp can catch formatting changes that are hard to catch otherwise, though at the cost of being sensitive to _any_ formatting changes - test_cmp needs some munging to eliminate unnecessary information
Quoted 3 lines
so on the whole, I think it's worth trying to use
test_cmpin the test helper. We may not _need_ it everywhere, but if it would be nice to use it in as many places as possible.
I think whatever you opt to go for here makes sense. I just wanted to provide the feedback in case it was helpful, i.e. when reading it I found these conversions a bit odd, wondered if they were strictly needed etc.
Thanks!
Re: [PATCH v2 2/9] t5526: use grep to assert on fetches
Quoted 238 lines · Ævar Arnfjörð Bjarmason
Quoted 232 lines · Glen Choo
Quoted 45 lines · Ævar Arnfjörð Bjarmason
Quoted 36 lines · Glen Choo
Quoted 22 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines · Glen Choo
In a previous commit, we replaced
test_cmpinvocations withverify_fetch_result(). Finish the process of removingtest_cmpby using grep inverify_fetch_result()instead.This makes the tests less sensitive to changes because, instead of checking the whole stderr, we only grep for the lines of the form
* "<old-head>..<new-head>\s+branch\s+-> origin/branch"
* "Fetching submodule <submodule-path>" (if fetching a submodule)Quoted 6 lines
when we expect the repo to have fetched. If we expect the repo to not have fetched, grep to make sure the lines are absent. Also, simplify the assertions by using grep patterns that match only the relevant pieces of information, e.g. <old-head> is irrelevant because we only want to know if the fetch was performed, so we don't need to know where the branch was before the fetch.
I tried ejecting 1/9 and 2/9 out of this series locally, and it passes all tests until the new tests you add in 7/9.
Quoted 8 lines
As ugly as some of the pre-image is, I wonder if dropping these first two and biting the bullet and just continuing with the
test_cmpis better.The
test_cmpis going to catch issues that even the cleverest grep combinations won't, e.g. in the submodule-in-C series I discovered a bug where all of our testing & the existing series hadn't spotted that we were dropping a \n at the end in one of the messages.I think there are two schools of thought on how to test informational messages:
Quoted 6 lines
- assert an exact match on the exact output that we generate
- assert that the output contains the pieces of information we care aboutThese two approaches are virtually opposites on two axes - the former will catch unintentional changes (like you've noted)[...]
Quoted 6 lines
Yes, and to be fair I'm thoroughly in the "assert an exact match" camp, i.e. "let's just use
test_cmp", and not everyone would agree with that.I mean, I don't think we should
test_cmpevery single instance of a command, but for things that are *the tests* concerning themselves with what the output should be, yes we should do that.That's a good point I hadn't considered, which is that if we want any hope of catching unintentional changes in our test suite, we'd want _some_ test to check the output. For "git fetch --recurse-submodules", it makes the most sense for that test to live in this file.
By eliminating all instances of
test_cmpin this file in particular, we lose assurances that we don't introduce accidental changes. It makes sense to at least have some tests explicitly for output.Quoted 11 lines
Quoted 2 lines
[...] and the latter saves on maintenance effort and tends to be less noisy in tests.
I also don't think you're right about the other approach "sav[ing] on [future] maintenance effort" in this case.
If I was needing to adjust some of this output I'd spend way longer on trying to carefully reason that some series of "grep" invocations were really doing the right thing, and probably end up doing the equivalent of a "
test_cmp" for myself out of general paranoia, whereas adjusting the output.That's fair. I've optimized the tests for readability by putting complicated logic in the test helper. But any diligent test reader would need to read the test helper to convince themselves of its correctness. In this case, I agree that the helper is too complex.
Quoted 11 lines
Quoted 4 lines
Personally, I'm a bit torn between both approaches in general because I want tests to be maintainable (testing the exact output is a bit of an antipattern at Google), but I'm not very comfortable with the fact that unintended changes can sneak through.
Yes, anyway whatever one thinks in general what I meant to point out here with "biting the bullet" is that whatever one thinks in general about the right approch for new tests, this series in particular seems to be creating more work for itself than it needs by refactoring the
test_cmpin existing tests just to add a few new ones.Quoted 3 lines
I.e. even if you'd like to not use test_cmp-alike for the new tests, wouldn't it be simpler to just leave the old ones in place and use your new helper for your new tests?
I'm not sure about this - avoiding changing old tests leads to fragmentation in the test suite and even the same file. I find it very challenging to read/modify files like this, because there is no longer a consistent style for the file, and I have to figure out which is the "good" way to write tests.
This suggestion makes sense if there's some qualitative difference between the new tests and old ones besides just 'being new'. This isn't true for this series, so I'd prefer to keep things consistent.
Quoted 67 lines
Quoted 7 lines
So I don't think there's a correct answer in general. Maybe an acceptable rule of thumb is that
test_cmpis good until it starts getting in the way of reading and writing understandable tests.If we agree on that rule, then for this patch, I think replacing
test_cmpis the way to go, primarily because it lets us ignore the 'old head' of the branch before the fetch, e.g. in the quoted example..[...]
Quoted 52 lines
Quoted 4 lines
Quoted 4 lines
test_expect_successsetup '(quoted diff) +0
-0@@ -274,13 +277,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in 'Quoted 10 lines
t/t5526-fetch-submodules.sh test_expect_success "Recursion doesn't happen when no new commits are fetched in · line 282
Quoted 7 lines
t/t5526-fetch-submodules.sh +0
-4test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super &&...as opposed to if we just rolled the generation of this into some utility printf function.
we'd still have to deal with $head1 if we use
test_cmp. That's fine for this test, because it's pretty simple, but it gets pretty janky later on:@@ -345,20 +339,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + git rev-parse --short HEAD >../subhead ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + git rev-parse --short HEAD >superhead && ( cd downstream && git fetch >../actual.out 2>../actual.errIn this example, we have two $head1 variables in different subshells, one of which is HEAD, but the other is HEAD^. The reason why we want HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2 commits ahead because we add the deepsubmodule in a separate commit), in my opinion, and I got tripped up quite a few times trying to read and understand the test. That's a lot of effort to spend on irrelevant information - the test actually cares about what it fetched, not where the ref used to be.
Quoted 1 line
So for that reason, I'd prefer to remove
test_cmpfor this test.I agree that it's pretty irrelevant, but I also think we'd be throwing the baby out with the bath water by entirely doing away with
test_cmphere, there's an easier way to do this.[...]
Quoted 10 lines
Instead let's just test once somewhere that when we run submodule fetching that submodules are indeed updated appropriately. Surely other submodule tests will break if the "update" code is made to NOOP, or update to the wrong HEAD>
Then for all these
test_cmptests we can just sed-away the $head1..$head2 with something like (untested):sed -n -e 's/[^.]*\.\.[^.]*/OLD..NEW/g'Quoted 7 lines
I.e. let's just skip this entire ceremony with asserting the old/new HEAD unless it's really needed (and then we can probably do it once outside a
test_cmp).If you grep through the test suite for "sed" adjacent to "
test_cmp" you'll find a lot of such examples of munging the output before test_cmp-ing it.Makes sense, I hadn't considered this (though I have seen the pattern in the test suite, oops). The most compelling argument in favor of this is that this could remove a lot of the complexity of
verify_fetch_result(), which is impeding test readability.Quoted 3 lines
I.e. none of these tests surely need to test that we updated from $head1..$head2 again and again with the corresponding verbosity in test setup and shelling out to "git rev-parse --short HEAD" or whatever.
I find the converse (we are testing the formatting over and over again) less convincing. In these tests, we really are checking for $head2 in the stderr to verify that the correct thing was fetched. I'm not convinced that we should be relying on _other_ submodule tests to tell us that submodule fetch is broken. Which brings me back to the original motivation of this patch..
Quoted 5 lines
That's perfectly fine here, since the actual point of the
test_cmpis to check the formatting/order etc. of the output itself, not to continually re-assert that submodule updating still works, and that we get the right OIDs.which is that these tests actually are continually re-asserting the submodule updating works correctly in the different circumstances, and since we use the stderr to check this,
test_cmpadds unwarranted noise.But you are correct in that the point of
test_cmpis to check formatting/order etc. There is value in usingtest_cmpfor this purpose, and getting rid of it entirely creates a hole in our test coverage. (This wouldn't mean that we'd need to usetest_cmp_everywhere_ though, only that we need to usetest_cmp_somewhere_.)As it stands:
(quoted diff) +2
-1+ test_cmp can improve the readability of the test helpers and debuggability of tests vs grep +/- test_cmp can catch formatting changes that are hard to catch otherwise, though at the cost of being sensitive to _any_ formatting changes - test_cmp needs some munging to eliminate unnecessary informationQuoted 3 lines
so on the whole, I think it's worth trying to use
test_cmpin the test helper. We may not _need_ it everywhere, but if it would be nice to use it in as many places as possible.
Quoted 6 lines
I think whatever you opt to go for here makes sense. I just wanted to provide the feedback in case it was helpful, i.e. when reading it I found these conversions a bit odd, wondered if they were strictly needed etc.
Thanks!
The additional perspective was indeed helpful, thanks!
PATCH v2 3/9 submodule: make static functions read submodules from commits
In reply to Glen ChooA future commit will teach "fetch --recurse-submodules" to fetch unpopulated submodules. To prepare for this, teach the necessary static functions how to read submodules from superproject commits using a "treeish_name" argument (instead of always reading from the index and filesystem) but do not actually change where submodules are read from. Submodules will be read from commits when we fetch unpopulated submodules.
The changed function signatures follow repo_submodule_init()'s argument order, i.e. "path" then "treeish_name". Where needed, reorder the arguments of functions that already take "path" and "treeish_name" to be consistent with this convention.
--- submodule.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-)
submodule.c +14 -10
@@ -932,6 +932,7 @@ struct has_commit_data {struct repository *repo;int result;const char *path; +const struct object_id *super_oid; };static int check_has_commit (const struct object_id *oid,void *data)
@@ -940,7 +941,7 @@ static int check_has_commit(const struct object_id *oid, void *data)struct repository subrepo;enum object_type type; -if (repo_submodule_init (&subrepo, cb->repo, cb->path,null_oid ())) { +if (repo_submodule_init (&subrepo, cb->repo, cb->path, cb->super_oid)) { cb->result =0 ; goto cleanup; }
@@ -968,9 +969,10 @@ static int check_has_commit(const struct object_id *oid, void *data)static int submodule_has_commits (struct repository *r,const char *path, +const struct object_id *super_oid,struct oid_array *commits) { -struct has_commit_data has_commit = { r,1 , path }; +struct has_commit_data has_commit = { r,1 , path, super_oid };/* * Perform a cheap, but incorrect check for the existence of 'commits'.
@@ -1017,7 +1019,7 @@ static int submodule_needs_pushing(struct repository *r,const char *path, struct oid_array *commits) { - if (!submodule_has_commits(r, path, commits)) + if (!submodule_has_commits(r, path, null_oid(), commits))/* * NOTE: We do consider it safe to return "no" here. The * correct answer would be "We do not know" instead of
@@ -1277,7 +1279,7 @@ static void calculate_changed_submodule_paths(struct repository *r,if (!path)continue ; -if (submodule_has_commits (r, path, commits)) { +if (submodule_has_commits (r, path,null_oid (), commits)) {oid_array_clear (commits); *name->string ='\0' ; }
@@ -1402,12 +1404,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path) }static struct fetch_task *fetch_task_create (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct fetch_task *task =xmalloc (sizeof (*task));memset (task,0 ,sizeof (*task)); - task->sub =submodule_from_path (r,null_oid (), path); + task->sub =submodule_from_path (r, treeish_name, path);if (!task->sub) {/* * No entry in .gitmodules? Technically not a submodule,
@@ -1439,11 +1442,12 @@ static void fetch_task_release(struct fetch_task *p) }static struct repository *get_submodule_repo_for (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct repository *ret =xmalloc (sizeof (*ret)); -if (repo_submodule_init (ret, r, path,null_oid ())) { +if (repo_submodule_init (ret, r, path, treeish_name)) {free (ret);return NULL ; }
@@ -1464,7 +1468,7 @@ static int get_next_submodule(struct child_process *cp,if (!S_ISGITLINK (ce->ce_mode))continue ; - task =fetch_task_create (spf->r, ce->name); + task =fetch_task_create (spf->r, ce->name,null_oid ());if (!task)continue ;
@@ -1487,7 +1491,7 @@ static int get_next_submodule(struct child_process *cp,continue ; } - task->repo =get_submodule_repo_for (spf->r, task->sub->path); + task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());if (task->repo) {struct strbuf submodule_prefix =STRBUF_INIT ;child_process_init (cp);
Signature · 1 line
2.33.GIT
Re: [PATCH v2 3/9] submodule: make static functions read submodules from commits
First of all, patches 1 and 2 look good since they are the same as in v1 and I have reviewed them. Moving on...
Quoted 4 lines · Glen Choo
The changed function signatures follow repo_submodule_init()'s argument order, i.e. "path" then "
treeish_name". Where needed, reorder the arguments of functions that already take "path" and "treeish_name" to be consistent with this convention.
This paragraph made me nervous, but looking at the diff, you didn't actually reorder any arguments. Probably best to delete this paragraph.
The fact that the additional functionality is not used also means that we can't tell for sure if all relevant functions are indeed changed, but perhaps we can determine this in a later patch.
Re: [PATCH v2 3/9] submodule: make static functions read submodules from commits
Quoted 8 lines · Jonathan Tan
Quoted 4 lines · Glen Choo
The changed function signatures follow repo_submodule_init()'s argument order, i.e. "path" then "
treeish_name". Where needed, reorder the arguments of functions that already take "path" and "treeish_name" to be consistent with this convention.This paragraph made me nervous, but looking at the diff, you didn't actually reorder any arguments. Probably best to delete this paragraph.
Oh you're right. I was sure that this paragraph used to be relevant, but I guess not. Thanks for the catch.
Re: [PATCH v2 3/9] submodule: make static functions read submodules from commits
In reply to Glen ChooQuoted 10 lines · Glen Choo
submodule.c +1
-0@@ -932,6 +932,7 @@ struct has_commit_data {struct repository *repo;int result;const char *path; +const struct object_id *super_oid; };
...
submodule.c static int check_has_commit(const struct object_id *oid, void *data) · line 975
Quoted 2 lines
submodule.c +1
-1-struct has_commit_data has_commit = { r,1 , path }; +struct has_commit_data has_commit = { r,1 , path, super_oid };
FWIW I wouldn't at all mind the tiny detour of just turning this into designated initializers while we're at it, instead of having to keep track of the positionals. I.e.:
[...] = {
.repo = r,
.result = 1,
.path = path,
,super_oid = super_oid
};Re: [PATCH v2 3/9] submodule: make static functions read submodules from commits
Quoted 22 lines · Ævar Arnfjörð Bjarmason
Quoted 10 lines · Glen Choo
submodule.c +1
-0@@ -932,6 +932,7 @@ struct has_commit_data {struct repository *repo;int result;const char *path; +const struct object_id *super_oid; };...
submodule.c static int check_has_commit(const struct object_id *oid, void *data) · line 975
Quoted 2 lines
submodule.c +1
-1-struct has_commit_data has_commit = { r,1 , path }; +struct has_commit_data has_commit = { r,1 , path, super_oid };FWIW I wouldn't at all mind the tiny detour of just turning this into designated initializers while we're at it, instead of having to keep track of the positionals. I.e.:
Quoted 6 lines
[...] = {
.repo = r,
.result = 1,
.path = path,
,super_oid = super_oid
};Since I'm touching the line anyway, this seems like a reasonable change.
PATCH v2 4/9 submodule: inline submodule_commits() into caller
In reply to Glen ChooWhen collecting the string_list of changed submodule names, the new submodules commits are stored in the string_list_item.util as an oid_array. A subsequent commit will replace the oid_array with a struct that has more information.
Prepare for this change by inlining submodule_commits() (which inserts into the string_list and initializes the string_list_item.util) into its only caller so that the code is easier to refactor later.
--- submodule.c | 22 ++++++---------------- 1 file changed, 6 insertions(+), 16 deletions(-)
submodule.c +6 -16
@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)return submodule_from_path (the_repository,null_oid (), ce->name); } -static struct oid_array *submodule_commits (struct string_list *submodules, -const char *name) -{ -struct string_list_item *item; - - item =string_list_insert (submodules, name); -if (item->util) -return (struct oid_array *) item->util; - -/* NEEDSWORK: should we have oid_array_init()? */ - item->util =xcalloc (1 ,sizeof (struct oid_array )); -return (struct oid_array *) item->util; -}struct collect_changed_submodules_cb_data {struct repository *repo;
@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,for (i =0 ; i < q->nr; i++) {struct diff_filepair *p = q->queue[i]; -struct oid_array *commits;const struct submodule *submodule;const char *name; +struct string_list_item *item;if (!S_ISGITLINK (p->two->mode))continue ;
@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,if (!name)continue ; - commits =submodule_commits (changed, name); -oid_array_append (commits, &p->two->oid); + item =string_list_insert (changed, name); +if (!item->util) +/* NEEDSWORK: should we have oid_array_init()? */ + item->util =xcalloc (1 ,sizeof (struct oid_array )); +oid_array_append (item->util, &p->two->oid); } }
Signature · 1 line
2.33.GIT
Re: [PATCH v2 4/9] submodule: inline submodule_commits() into caller
submodule.c static void collect_changed_submodules_cb(struct diff_queue_struct *q, · line 855
Quoted 9 lines · Glen Choo
submodule.c +5
-2- commits =submodule_commits (changed, name); -oid_array_append (commits, &p->two->oid); + item =string_list_insert (changed, name); +if (!item->util) +/* NEEDSWORK: should we have oid_array_init()? */ + item->util =xcalloc (1 ,sizeof (struct oid_array )); +oid_array_append (item->util, &p->two->oid); } }
Yes, just adding it while we're at it seems worthwhile, and if not defining this in terms of the macro would be better, as the two are guaranteed not to drift apart. I.e. the pattern seen in:
git grep -W 'memcpy.*&blank'
PATCH v2 5/9 submodule: store new submodule commits oid_array in a struct
In reply to Glen ChooThis commit prepares for a future commit that will teach git fetch --recurse-submodules how to fetch submodules that are present in <gitdir>/modules, but are not populated. To do this, we need to store more information about the changed submodule so that we can read the submodule configuration from the superproject commit instead of the filesystem.
Refactor the changed submodules string_list.util to hold a struct instead of an oid_array. This struct only holds the new_commits oid_array for now; more information will be added later.
--- submodule.c | 54 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 35 insertions(+), 19 deletions(-)
submodule.c +35 -19
@@ -806,6 +806,20 @@ static const char *default_name_or_path(const char *path_or_name)return path_or_name; } +/* +* Holds relevant information for a changed submodule. Used as the .util +* member of the changed submodule string_list_item. +*/ +struct changed_submodule_data { +/* The submodule commits that have changed in the rev walk. */ +struct oid_array new_commits; +}; + +static void changed_submodule_data_clear (struct changed_submodule_data *cs_data) +{ +oid_array_clear (&cs_data->new_commits); +} +static void collect_changed_submodules_cb (struct diff_queue_struct *q,struct diff_options *options,void *data)
@@ -820,6 +834,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,const struct submodule *submodule;const char *name;struct string_list_item *item; +struct changed_submodule_data *cs_data;if (!S_ISGITLINK (p->two->mode))continue ;
@@ -848,9 +863,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q, item =string_list_insert (changed, name);if (!item->util) -/* NEEDSWORK: should we have oid_array_init()? */ - item->util =xcalloc (1 ,sizeof (struct oid_array )); -oid_array_append (item->util, &p->two->oid); + item->util =xcalloc (1 ,sizeof (struct changed_submodule_data )); + cs_data = item->util; +oid_array_append (&cs_data->new_commits, &p->two->oid); } }
@@ -897,11 +912,12 @@ static void collect_changed_submodules(struct repository *r,reset_revision_walk (); } -static void free_submodules_oids (struct string_list *submodules) +static void free_submodules_data (struct string_list *submodules) {struct string_list_item *item; -for_each_string_list_item (item, submodules) -oid_array_clear ((struct oid_array *) item->util); +for_each_string_list_item (item, submodules) { +changed_submodule_data_clear (item->util); + }string_list_clear (submodules,1 ); }
@@ -1069,7 +1085,7 @@ int find_unpushed_submodules(struct repository *r,collect_changed_submodules (r, &submodules, &argv);for_each_string_list_item (name, &submodules) { -struct oid_array *commits = name->util; +struct changed_submodule_data *cs_data = name->util;const struct submodule *submodule;const char *path =NULL ;
@@ -1082,11 +1098,11 @@ int find_unpushed_submodules(struct repository *r,if (!path)continue ; -if (submodule_needs_pushing (r, path, commits)) +if (submodule_needs_pushing (r, path, &cs_data->new_commits))string_list_insert (needs_pushing, path); } -free_submodules_oids (&submodules); +free_submodules_data (&submodules);strvec_clear (&argv);return needs_pushing->nr;
@@ -1256,7 +1272,7 @@ static void calculate_changed_submodule_paths(struct repository *r,collect_changed_submodules (r, changed_submodule_names, &argv);for_each_string_list_item (name, changed_submodule_names) { -struct oid_array *commits = name->util; +struct changed_submodule_data *cs_data = name->util;const struct submodule *submodule;const char *path =NULL ;
@@ -1269,8 +1285,8 @@ static void calculate_changed_submodule_paths(struct repository *r,if (!path)continue ; -if (submodule_has_commits (r, path,null_oid (), commits)) { -oid_array_clear (commits); +if (submodule_has_commits (r, path,null_oid (), &cs_data->new_commits)) { +changed_submodule_data_clear (cs_data); *name->string ='\0' ; } }
@@ -1307,7 +1323,7 @@ int submodule_touches_in_range(struct repository *r,strvec_clear (&args); -free_submodules_oids (&subs); +free_submodules_data (&subs);return ret; }
@@ -1591,7 +1607,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,struct fetch_task *task = task_cb;struct string_list_item *it; -struct oid_array *commits; +struct changed_submodule_data *cs_data;if (!task || !task->sub)BUG ("callback cookie bogus" );
@@ -1619,14 +1635,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,/* Could be an unchanged submodule, not contained in the list */ goto out; - commits = it->util; -oid_array_filter (commits, + cs_data = it->util; +oid_array_filter (&cs_data->new_commits, commit_missing_in_sub, task->repo);/* Are there commits we want, but do not exist? */ -if (commits->nr) { - task->commits = commits; +if (cs_data->new_commits.nr) { + task->commits = &cs_data->new_commits;ALLOC_GROW (spf->oid_fetch_tasks, spf->oid_fetch_tasks_nr +1 , spf->oid_fetch_tasks_alloc);
@@ -1684,7 +1700,7 @@ int fetch_populated_submodules(struct repository *r,strvec_clear (&spf.args); out: -free_submodules_oids (&spf.changed_submodule_names); +free_submodules_data (&spf.changed_submodule_names);return spf.result; }
Signature · 1 line
2.33.GIT
Re: [PATCH v2 5/9] submodule: store new submodule commits oid_array in a struct
submodule.c static void collect_changed_submodules(struct repository *r, · line 920
Quoted 6 lines · Glen Choo
submodule.c +3
-2struct string_list_item *item; -for_each_string_list_item (item, submodules) -oid_array_clear ((struct oid_array *) item->util); +for_each_string_list_item (item, submodules) { +changed_submodule_data_clear (item->util); + }
Nit: These {} additions aren't needed here or in the end state of this series.
PATCH v2 6/9 submodule: extract get_fetch_task()
In reply to Glen Chooget_next_submodule() configures the parallel submodule fetch by performing two functions:
* iterate the index to find submodules
* configure the child processes to fetch the submodules found in the previous step
Extract the index iterating code into an iterator function, get_fetch_task(), so that get_next_submodule() is agnostic of how to find submodules. This prepares for a subsequent commit will teach the fetch machinery to also iterate through the list of changed submodules (in addition to the index).
---
Jonathan: I'm really happy with the formatting changes that you suggested because this diff is a lot easier to read, so thanks again! Going forward, I'd appreciate any and all formatting suggestions - if they seem possibly excessive, you can mark them as nits.
submodule.c | 62 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 37 insertions(+), 25 deletions(-)
submodule.c +37 -25
@@ -1461,14 +1461,12 @@ static struct repository *get_submodule_repo_for(struct repository *r,return ret; } -static int get_next_submodule (struct child_process *cp, -struct strbuf *err,void *data,void **task_cb) +static struct fetch_task * +get_fetch_task (struct submodule_parallel_fetch *spf, +const char **default_argv,struct strbuf *err) { -struct submodule_parallel_fetch *spf = data; -for (; spf->count < spf->r->index->cache_nr; spf->count++) {const struct cache_entry *ce = spf->r->index->cache[spf->count]; -const char *default_argv;struct fetch_task *task;if (!S_ISGITLINK (ce->ce_mode))
@@ -1488,10 +1486,10 @@ static int get_next_submodule(struct child_process *cp, &spf->changed_submodule_names, task->sub->name)) continue; - default_argv ="on-demand" ; + *default_argv ="on-demand" ;break ;case RECURSE_SUBMODULES_ON : - default_argv ="yes" ; + *default_argv ="yes" ;break ;case RECURSE_SUBMODULES_OFF :continue ;
@@ -1499,29 +1497,12 @@ static int get_next_submodule(struct child_process *cp, task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());if (task->repo) { -struct strbuf submodule_prefix =STRBUF_INIT ; -child_process_init (cp); - cp->dir = task->repo->gitdir; -prepare_submodule_repo_env_in_gitdir (&cp->env_array); - cp->git_cmd =1 ;if (!spf->quiet)strbuf_addf (err,_ ("Fetching submodule %s%s\n" ), spf->prefix, ce->name); -strvec_init (&cp->args); -strvec_pushv (&cp->args, spf->args.v); -strvec_push (&cp->args, default_argv); -strvec_push (&cp->args,"--submodule-prefix" ); - -strbuf_addf (&submodule_prefix,"%s%s/" , - spf->prefix, - task->sub->path); -strvec_push (&cp->args, submodule_prefix.buf); spf->count++; - *task_cb = task; - -strbuf_release (&submodule_prefix); -return 1 ; +return task; }else {struct strbuf empty_submodule_path =STRBUF_INIT ;
@@ -1545,6 +1526,37 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&empty_submodule_path); } } +return NULL ; +} + +static int get_next_submodule (struct child_process *cp,struct strbuf *err, +void *data,void **task_cb) +{ +struct submodule_parallel_fetch *spf = data; +const char *default_argv =NULL ; +struct fetch_task *task =get_fetch_task (spf, &default_argv, err); + +if (task) { +struct strbuf submodule_prefix =STRBUF_INIT ; + +child_process_init (cp); + cp->dir = task->repo->gitdir; +prepare_submodule_repo_env_in_gitdir (&cp->env_array); + cp->git_cmd =1 ; +strvec_init (&cp->args); +strvec_pushv (&cp->args, spf->args.v); +strvec_push (&cp->args, default_argv); +strvec_push (&cp->args,"--submodule-prefix" ); + +strbuf_addf (&submodule_prefix,"%s%s/" , + spf->prefix, + task->sub->path); +strvec_push (&cp->args, submodule_prefix.buf); + *task_cb = task; + +strbuf_release (&submodule_prefix); +return 1 ; + }if (spf->oid_fetch_tasks_nr) { struct fetch_task *task =
Signature · 1 line
2.33.GIT
PATCH v2 7/9 fetch: fetch unpopulated, changed submodules
In reply to Glen Choo"git fetch --recurse-submodules" only considers populated submodules (i.e. submodules that can be found by iterating the index), which makes "git fetch" behave differently based on which commit is checked out. As a result, even if the user has initialized all submodules correctly, they may not fetch the necessary submodule commits, and commands like "git checkout --recurse-submodules" might fail.
Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated (this is in addition to the current behavior of fetching populated submodules).
Since a submodule may be encountered multiple times (via the list of populated submodules or via the list of changed submodules), maintain a list of seen submodules to avoid fetching a submodule more than once.
---
As I mentioned in the cover letter, I'm not entirely happy with the name repo_has_absorbed_submodules() - it's not a standardized term AFAIK and it's a little clunky.
"absorbed submodule" is just a stand-in for "submodule in .git/modules", so if we have a better term for "submodule in .git/modules", let's use that instead.
Documentation/fetch-options.txt | 26 +++-- Documentation/git-fetch.txt | 10 +- builtin/fetch.c | 14 +-- submodule.c | 134 +++++++++++++++++++--- submodule.h | 12 +- t/t5526-fetch-submodules.sh | 195 ++++++++++++++++++++++++++++++++ 6 files changed, 349 insertions(+), 42 deletions(-)
Documentation/fetch-options.txt +17 -9
@@ -185,15 +185,23 @@ endif::git-pull[] ifndef::git-pull[] --recurse-submodules[=yes|on-demand|no]:: This option controls if and under what conditions new commits of - populated submodules should be fetched too. It can be used as a - boolean option to completely disable recursion when set to 'no' or to - unconditionally recurse into all populated submodules when set to - 'yes', which is the default when this option is used without any - value. Use 'on-demand' to only recurse into a populated submodule - when the superproject retrieves a commit that updates the submodule's - reference to a commit that isn't already in the local submodule - clone. By default, 'on-demand' is used, unless - `fetch.recurseSubmodules` is set (see linkgit:git-config[1]). + submodules should be fetched too. When recursing through submodules, + `git fetch` always attempts to fetch "changed" submodules, that is, a + submodule that has commits that are referenced by a newly fetched + superproject commit but are missing in the local submodule clone. A + changed submodule can be fetched as long as it is present locally e.g. + in `$GIT_DIR/modules/` (see linkgit:gitsubmodules[7]); if the upstream + adds a new submodule, that submodule cannot be fetched until it is + cloned e.g. by `git submodule update`. ++ +When set to 'on-demand', only changed submodules are fetched. When set +to 'yes', all populated submodules are fetched and submodules that are +both unpopulated and changed are fetched. When set to 'no', submodules +are never fetched. ++ +When unspecified, this uses the value of `fetch.recurseSubmodules` if it +is set (see linkgit:git-config[1]), defaulting to 'on-demand' if unset. +When this option is used without any value, it defaults to 'yes'. endif::git-pull[] -j::
Documentation/git-fetch.txt +4 -6
@@ -287,12 +287,10 @@ include::transfer-data-leaks.txt[] BUGS ---- -Using --recurse-submodules can only fetch new commits in already checked -out submodules right now. When e.g. upstream added a new submodule in the -just fetched commits of the superproject the submodule itself cannot be -fetched, making it impossible to check out that submodule later without -having to do a fetch again. This is expected to be fixed in a future Git -version. +Using --recurse-submodules can only fetch new commits in submodules that are +present locally e.g. in `$GIT_DIR/modules/`. If the upstream adds a new +submodule, that submodule cannot be fetched until it is cloned e.g. by `git +submodule update`. This is expected to be fixed in a future Git version. SEE ALSO --------
builtin/fetch.c +7 -7
@@ -2122,13 +2122,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) max_children = fetch_parallel_config;add_options_to_argv (&options); - result =fetch_populated_submodules (the_repository, - &options, - submodule_prefix, - recurse_submodules, - recurse_submodules_default, - verbosity <0 , - max_children); + result =fetch_submodules (the_repository, + &options, + submodule_prefix, + recurse_submodules, + recurse_submodules_default, + verbosity <0 , + max_children);strvec_clear (&options); }
submodule.c +120 -14
224 lines not on this page · raw message
submodule.h +6 -6
@@ -88,12 +88,12 @@ int should_update_submodules(void);*/ const struct submodule *submodule_from_ce (const struct cache_entry *ce);void check_for_new_submodule_commits (struct object_id *oid); -int fetch_populated_submodules (struct repository *r, -const struct strvec *options, -const char *prefix, -int command_line_option, -int default_option, -int quiet,int max_parallel_jobs); +int fetch_submodules (struct repository *r, +const struct strvec *options, +const char *prefix, +int command_line_option, +int default_option, +int quiet,int max_parallel_jobs);unsigned is_submodule_modified (const char *path,int ignore_untracked);int submodule_uses_gitfile (const char *path);
t/t5526-fetch-submodules.sh +195 -0
201 lines not on this page · raw message
Signature · 1 line
2.33.GIT
Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules
Patches 4-6 look good.
Quoted 3 lines · Glen Choo
Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated (this is in addition to the current behavior of fetching populated submodules).
I think add a note that the current behavior is regardless of what is being fetched. So, maybe something like:
Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated. This is in addition to the current behavior of fetching populated submodules (which happens regardless of what was fetched in the superproject, or even if nothing was fetched in the superproject).
Quoted 7 lines
As I mentioned in the cover letter, I'm not entirely happy with the name
repo_has_absorbed_submodules()- it's not a standardized term AFAIK and it's a little clunky."absorbed submodule" is just a stand-in for "submodule in .git/modules", so if we have a better term for "submodule in .git/modules", let's use that instead.
I think that this is OK if the doc comment is updated. I'll make the suggestion in the appropriate place below.
Quoted 6 lines
submodule.c +2
-0@@ -1248,14 +1261,28 @@ void check_for_new_submodule_commits(struct object_id *oid)oid_array_append (&ref_tips_after_fetch, oid); } +/* +* Returns 1 if the repo has absorbed submodule gitdirs, and 0
submodule.c void check_for_new_submodule_commits(struct object_id *oid) · line 1275
Quoted 10 lines
submodule.c +10
-0+* otherwise. Like submodule_name_to_gitdir(), this checks +* $GIT_DIR/modules, not $GIT_COMMON_DIR. +*/ +static int repo_has_absorbed_submodules (struct repository *r) +{ +struct strbuf buf =STRBUF_INIT ; + +strbuf_repo_git_path (&buf, r,"modules/" ); +return file_exists (buf.buf) && !is_empty_dir (buf.buf); +}
I think that if you replace the doc comment with something like:
Returns 1 if there is at least one submodule gitdir in $GIT_DIR/modules, and 0 otherwise. (End users can move submodule gitdirs into $GIT_DIR/modules by running "git submodule absorbgitdirs".) Like submodule_name_to_gitdir()...
then it would be fine.
submodule.c struct submodule_parallel_fetch { · line 1369
Quoted 5 lines
submodule.c +1
-0@@ -1338,6 +1366,7 @@ struct submodule_parallel_fetch {int result;struct string_list changed_submodule_names; +struct string_list seen_submodule_names;
Could we unify the 2 lists instead of having 2 separate ones?
Quoted 55 lines
submodule.c +51
-0@@ -1529,12 +1569,77 @@ get_fetch_task(struct submodule_parallel_fetch *spf,return NULL ; } +static struct fetch_task * +get_fetch_task_from_changed (struct submodule_parallel_fetch *spf, +const char **default_argv,struct strbuf *err) +{ +for (; spf->changed_count < spf->changed_submodule_names.nr; + spf->changed_count++) { +struct string_list_item item = + spf->changed_submodule_names.items[spf->changed_count]; +struct changed_submodule_data *cs_data = item.util; +struct fetch_task *task; + +/* +* We might have already considered this submodule +* because we saw it in the index. +*/ +if (string_list_lookup (&spf->seen_submodule_names, item.string)) +continue ; + + task =fetch_task_create (spf->r, cs_data->path, + cs_data->super_oid); +if (!task) +continue ; + +switch (get_fetch_recurse_config (task->sub, spf)) { +default : +case RECURSE_SUBMODULES_DEFAULT : +case RECURSE_SUBMODULES_ON_DEMAND : + *default_argv ="on-demand" ; +break ; +case RECURSE_SUBMODULES_ON : + *default_argv ="yes" ; +break ; +case RECURSE_SUBMODULES_OFF : +continue ; + } + + task->repo =get_submodule_repo_for (spf->r, task->sub->path, + cs_data->super_oid); +if (!task->repo) { +fetch_task_release (task); +free (task); + +strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), + cs_data->path); +continue ; + } +if (!is_tree_submodule_active (spf->r, cs_data->super_oid, + task->sub->path)) +continue ; + +if (!spf->quiet)
submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, · line 1632
Quoted 10 lines
submodule.c +10
-0+strbuf_addf (err, +_ ("Fetching submodule %s%s at commit %s\n" ), + spf->prefix, task->sub->path, +find_unique_abbrev (cs_data->super_oid, +DEFAULT_ABBREV )); + spf->changed_count++; +return task; + } +return NULL ; +}
This is very similar to get_fetch_task_from_index(). Both:
1. loop over something
2. exit early if the submodule name is seen
3. create the fetch task
4. set the "recurse config"
5. get the submodule repo
6. if success, increment a counter
7. if failure, check for some conditions and maybe append to err
Could a function be refactored that does 2-5?
Review get_fetch_task(struct submodule_parallel_fetch *spf, · line 1634
Quoted 2 lines
(quoted diff) +2
-0+test_expect_success "'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules" ' + git -C downstream fetch --recurse-submodules &&
First of all, thanks for updating the test - it is much easier to understand now.
About this line, we shouldn't use the code being tested to set up (we're testing "fetch --recurse-submodules", so we shouldn't use the same command to set up). Also, if we don't have confidence in the starting state, it may be a sign to write it out more explicitly instead of relying on a complicated command to do the right thing.
However, in this case, I don't think we need this. All we need is to see that the test contains a new superproject commit that points to a new submodule commit (and recursively). So we don't need this line.
Quoted 4 lines
t/t5526-fetch-submodules.sh +4
-0+ # Create new superproject commit with updated submodules + add_upstream_commit && + ( + cd submodule &&
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 431
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ ( + cd subdir/deepsubmodule && + git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule" + ) && + git add submodule && + git commit -m "new submodule" &&
I thought add_upstream_commit() would do this, but apparently it just adds commits to the submodules (which works for the earlier tests that just tested that the submodules were fetched, but not for this one). I think that all this should go into its own function.
Also, I understand that "git fetch" is there to pick up the commit we made in add_upstream_commit, but this indirection is unnecessary in a test, I think. Can we not use add_upstream_commit and just create our own in subdir/deepsubmodule? (This might conflict with subsequent tests that use the old scheme, but I think that it should be fine.)
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 442
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ # Fetch the new superproject commit + ( + cd downstream && + git switch --recurse-submodules no-submodules && + git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err + ) && + test_must_be_empty actual.out && + git rev-parse --short HEAD >superhead && + git -C submodule rev-parse --short HEAD >subhead && + git -C deepsubmodule rev-parse --short HEAD >deephead &&
These >superhead lines would not be necessary if we had our own function.
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 519
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+# In downstream, init "submodule2", but do not check it out while +# fetching. This lets us assert that unpopulated submodules can be +# fetched.
Firstly, "In upstream", I think? You want to fetch from it, so it has to be upstream.
Secondly, is this test needed? I thought the case in which the worktree has no submodules would be sufficient to test that unpopulated submodules can be fetched.
Quoted 1 line
+test_expect_success "'--recurse-submodules' should fetch submodule commits in changed submodules and the index" '
[snip]
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+ git checkout super && + ( + cd downstream && + git fetch --recurse-submodules >../actual.out 2>../actual.err + ) &&
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 586
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ test_must_be_empty actual.out && + + # Assert that the submodules in the super branch are fetched + git rev-parse --short HEAD >superhead && + git -C submodule rev-parse --short HEAD >subhead && + git -C deepsubmodule rev-parse --short HEAD >deephead && + verify_fetch_result actual.err && + # grep for the exact line to check that the submodule is read + # from the index, not from a commit + grep "^Fetching submodule submodule\$" actual.err &&
Instead of a grep, I think this should be done by precisely specifying what to fetch in the "git fetch" invocation, and then checking that the submodule has commits that it didn't have before.
In addition, I think the following cases also need to be tested:
- two fetched commits have submodules of the same name but different URL
- a fetched commit and a commit in the index have submodules of the same name but different URL
Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules
Quoted 5 lines · Jonathan Tan
Quoted 3 lines · Glen Choo
Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated (this is in addition to the current behavior of fetching populated submodules).
Quoted 8 lines
I think add a note that the current behavior is regardless of what is being fetched. So, maybe something like:
Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated. This is in addition to the current behavior of fetching populated submodules (which happens regardless of what was fetched in the superproject, or even if nothing was fetched in the superproject).
Makes sense, thanks. This description is true, though a bit misleading in the [--recurse-submodules=on-demand] case - if nothing was fetched, on-demand would still try to fetch submodules, though no submodules would be fetched. I'll tweak it a little.
Quoted 16 lines
Quoted 7 lines
As I mentioned in the cover letter, I'm not entirely happy with the name
repo_has_absorbed_submodules()- it's not a standardized term AFAIK and it's a little clunky."absorbed submodule" is just a stand-in for "submodule in .git/modules", so if we have a better term for "submodule in .git/modules", let's use that instead.
I think that this is OK if the doc comment is updated. I'll make the suggestion in the appropriate place below.
Quoted 5 lines
+/*
+ * Returns 1 if the repo has absorbed submodule gitdirs, and 0
+ * otherwise. Likesubmodule_name_to_gitdir(), this checks
+ * $GIT_DIR/modules, not $GIT_COMMON_DIR.
+ */
Quoted 8 lines
I think that if you replace the doc comment with something like:
Returns 1 if there is at least one submodule gitdir in $GIT_DIR/modules, and 0 otherwise. (End users can move submodule gitdirs into $GIT_DIR/modules by running "git submodule absorbgitdirs".) Like
submodule_name_to_gitdir()...then it would be fine.
Thanks! Sounds good.
Quoted 7 lines
submodule.c struct submodule_parallel_fetch { · line 1369
Quoted 5 lines
submodule.c +1
-0@@ -1338,6 +1366,7 @@ struct submodule_parallel_fetch {int result;struct string_list changed_submodule_names; +struct string_list seen_submodule_names;Could we unify the 2 lists instead of having 2 separate ones?
If I understand the suggestion correctly (Can we combine the two lists into a single changed_submodule_names list?) I don't think we can - at least not without other changes. The intent behind seen_submodule_names is to tell get_fetch_task_from_changed() which changed_submodule_names items can be ignored. If we only used one list, we wouldn't be able to tell whether we had already considered the submodule or not. If we stored this info elsewhere, e.g. an extra field in changed_submodule_data, then we could use a single list.
Quoted 76 lines
Quoted 56 lines
submodule.c +51
-0@@ -1529,12 +1569,77 @@ get_fetch_task(struct submodule_parallel_fetch *spf,return NULL ; } +static struct fetch_task * +get_fetch_task_from_changed (struct submodule_parallel_fetch *spf, +const char **default_argv,struct strbuf *err) +{ +for (; spf->changed_count < spf->changed_submodule_names.nr; + spf->changed_count++) { +struct string_list_item item = + spf->changed_submodule_names.items[spf->changed_count]; +struct changed_submodule_data *cs_data = item.util; +struct fetch_task *task; + +/* +* We might have already considered this submodule +* because we saw it in the index. +*/ +if (string_list_lookup (&spf->seen_submodule_names, item.string)) +continue ; + + task =fetch_task_create (spf->r, cs_data->path, + cs_data->super_oid); +if (!task) +continue ; + +switch (get_fetch_recurse_config (task->sub, spf)) { +default : +case RECURSE_SUBMODULES_DEFAULT : +case RECURSE_SUBMODULES_ON_DEMAND : + *default_argv ="on-demand" ; +break ; +case RECURSE_SUBMODULES_ON : + *default_argv ="yes" ; +break ; +case RECURSE_SUBMODULES_OFF : +continue ; + } + + task->repo =get_submodule_repo_for (spf->r, task->sub->path, + cs_data->super_oid); +if (!task->repo) { +fetch_task_release (task); +free (task); + +strbuf_addf (err,_ ("Could not access submodule '%s'\n" ), + cs_data->path); +continue ; + } +if (!is_tree_submodule_active (spf->r, cs_data->super_oid, + task->sub->path)) +continue ; + +if (!spf->quiet)submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, · line 1632
Quoted 10 lines
submodule.c +10
-0+strbuf_addf (err, +_ ("Fetching submodule %s%s at commit %s\n" ), + spf->prefix, task->sub->path, +find_unique_abbrev (cs_data->super_oid, +DEFAULT_ABBREV )); + spf->changed_count++; +return task; + } +return NULL ; +}This is very similar to
get_fetch_task_from_index(). Both:
1. loop over something
2. exit early if the submodule name is seen
3. create the fetch task
4. set the "recurse config"
5. get the submodule repo
6. if success, increment a counter
7. if failure, check for some conditions and maybe append to err
Quoted 1 line
Could a function be refactored that does 2-5?
Hm, it makes sense. I don't see a reason for 2-5 to be different for the different functions.
Quoted 7 lines
Review get_fetch_task(struct submodule_parallel_fetch *spf, · line 1634
Quoted 3 lines
(quoted diff) +2
-0+test_expect_success "'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules" ' + git -C downstream fetch --recurse-submodules &&First of all, thanks for updating the test - it is much easier to understand now.
Quoted 5 lines
About this line, we shouldn't use the code being tested to set up (we're testing "fetch --recurse-submodules", so we shouldn't use the same command to set up). Also, if we don't have confidence in the starting state, it may be a sign to write it out more explicitly instead of relying on a complicated command to do the right thing.
True. I think the easiest way to do this would be the "porcelain-downstream" you suggested in v1. But as you mentioned...
Quoted 3 lines
However, in this case, I don't think we need this. All we need is to see that the test contains a new superproject commit that points to a new submodule commit (and recursively). So we don't need this line.
this isn't necessary, so I don't know if this is worth the effort at the moment. I'll tinker with it.
As for the line itself, you're right, we don't need this. The state of the downstream was more important when we cared about the old branch head (it's needed for test_cmp), but we no longer do.
Quoted 20 lines
Quoted 4 lines
t/t5526-fetch-submodules.sh +4
-0+ # Create new superproject commit with updated submodules + add_upstream_commit && + ( + cd submodule &&t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 431
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ ( + cd subdir/deepsubmodule && + git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule" + ) && + git add submodule && + git commit -m "new submodule" &&I thought
add_upstream_commit()would do this, but apparently it just adds commits to the submodules (which works for the earlier tests that just tested that the submodules were fetched, but not for this one). I think that all this should go into its own function.
Quoted 5 lines
Also, I understand that "git fetch" is there to pick up the commit we made in
add_upstream_commit, but this indirection is unnecessary in a test, I think. Can we not useadd_upstream_commitand just create our own in subdir/deepsubmodule? (This might conflict with subsequent tests that use the old scheme, but I think that it should be fine.)
I copy-pasted this from existing tests, but I'm not happy with how noisy it is either. I'll tinker with this too.
Quoted 6 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 519
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+# In downstream, init "submodule2", but do not check it out while +# fetching. This lets us assert that unpopulated submodules can be +# fetched.Firstly, "In upstream", I think? You want to fetch from it, so it has to be upstream.
It is "in downstream" - we "git init" the upstream, but we need to "git submodule update --init" (which wraps "git submodule init") in the downstream. If we didn't init it in the downstream, downstream wouldn't have the clone and wouldn't fetch.
Quoted 3 lines
Secondly, is this test needed? I thought the case in which the worktree has no submodules would be sufficient to test that unpopulated submodules can be fetched.
I'd prefer to have this test because it tests the interaction between populated and unpopulated submodules. e.g. in a previous iteration, I only had the "no submodules" test, but accidentally reused the submodule_parallel_fetch.count variable for both populated and unpopulated submodules. The test suite didn't catch the bug - I only noticed the bug by a stroke of luck.
Quoted 15 lines
Quoted 1 line
+test_expect_success "'--recurse-submodules' should fetch submodule commits in changed submodules and the index" '
[snip]
Quoted 11 lines
t/t5526-fetch-submodules.sh +11
-0+ git checkout super && + ( + cd downstream && + git fetch --recurse-submodules >../actual.out 2>../actual.err + ) && + test_must_be_empty actual.out && + + # Assert that the submodules in the super branch are fetched + git rev-parse --short HEAD >superhead && + git -C submodule rev-parse --short HEAD >subhead && + git -C deepsubmodule rev-parse --short HEAD >deephead &&
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 586
Quoted 4 lines
t/t5526-fetch-submodules.sh +4
-0+ verify_fetch_result actual.err && + # grep for the exact line to check that the submodule is read + # from the index, not from a commit + grep "^Fetching submodule submodule\$" actual.err &&Instead of a grep, I think this should be done by precisely specifying what to fetch in the "git fetch" invocation, and then checking that the submodule has commits that it didn't have before.
verify_fetch_result() already tells us that the submodule has the new commit:
if [ -f subhead ]; then grep "Fetching submodule submodule" $ACTUAL_ERR && grep -E "\.\.$(cat subhead)\s+sub\s+-> origin/sub" $ACTUAL_ERR
but (by design) it does not tell us whether "git fetch" read the .gitmodules config from the index or a commit. The additional grep with "^$" tells us that we read from the index because it checks that the info message is not "Fetching submodule submodule at commit <id>". We want to have this check because we want "git fetch" to prefer the index in the event that the submodule is both changed and populated.
Quoted 5 lines
In addition, I think the following cases also need to be tested:
- two fetched commits have submodules of the same name but different URL
- a fetched commit and a commit in the index have submodules of the same name but different URL
It makes sense to test both cases. The former is a new edge case introduced by this commit, while the latter is a concern before this commit. I _believe_ that we already handle the latter gracefully, and that the same logic can be used to handle the former, but I don't think we have any tests proving either of these hypotheses.
Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules
Quoted 14 lines · Glen Choo
Quoted 12 lines · Jonathan Tan
Quoted 11 lines · Glen Choo
t/t5526-fetch-submodules.sh +11
-0+ # Create new superproject commit with updated submodules + add_upstream_commit && + ( + cd submodule && + ( + cd subdir/deepsubmodule && + git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule"
Quoted 8 lines
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 431
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ ) && + git add submodule && + git commit -m "new submodule" &&I thought
add_upstream_commit()would do this, but apparently it just adds commits to the submodules (which works for the earlier tests that just tested that the submodules were fetched, but not for this one). I think that all this should go into its own function.
I'm testing out a function that does exactly what these lines do, i.e. create a superproject commit containing a submodule change containing a deepsubmodule change. That works pretty well and it makes sense in the context of the tests.
Quoted 5 lines
Quoted 5 lines
Also, I understand that "git fetch" is there to pick up the commit we made in
add_upstream_commit, but this indirection is unnecessary in a test, I think. Can we not useadd_upstream_commitand just create our own in subdir/deepsubmodule? (This might conflict with subsequent tests that use the old scheme, but I think that it should be fine.)
We can avoid the "git fetch" if we first need to fix an inconsistency in how the submodules are set up. Right now, we have:
test_expect_success setup '
mkdir deepsubmodule &&
[...]
mkdir submodule &&
(
[...]
git submodule add "$pwd/deepsubmodule" subdir/deepsubmodule &&
git commit -a -m new &&
git branch -M sub
) &&
git submodule add "$pwd/submodule" submodule &&
[...]
(
cd downstream &&
git submodule update --init --recursive
)
'resulting in a directory structure like:
$pwd
|_submodule
|_subdir
|_deepsubmodule
|_deepsubmodule
and upstream/downstream dependencies like:
upstream downstream -------- ---------- $pwd/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)
$pwd/submodule/subdir/deepsubmodule
So we can't create the commit in submodule/subdir/deepsubmodule, because that's not where our SUT would fetch from.
Instead, we could fix this by having a more consistent upstream/downstream structure:
$pwd
|_submodule
|_subdir
|_deepsubmodule
upstream downstream -------- ---------- $pwd/submodule/subdir/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)
This seems more convenient to test, but before I commit to this, is there a downside to this that I'm not seeing?
Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules
Quoted 84 lines · Glen Choo
Quoted 14 lines · Glen Choo
Quoted 12 lines · Jonathan Tan
Quoted 11 lines · Glen Choo
t/t5526-fetch-submodules.sh +11
-0+ # Create new superproject commit with updated submodules + add_upstream_commit && + ( + cd submodule && + ( + cd subdir/deepsubmodule && + git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule"Quoted 8 lines
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 431
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ ) && + git add submodule && + git commit -m "new submodule" &&I thought
add_upstream_commit()would do this, but apparently it just adds commits to the submodules (which works for the earlier tests that just tested that the submodules were fetched, but not for this one). I think that all this should go into its own function.I'm testing out a function that does exactly what these lines do, i.e. create a superproject commit containing a submodule change containing a deepsubmodule change. That works pretty well and it makes sense in the context of the tests.
Quoted 5 lines
Quoted 5 lines
Also, I understand that "git fetch" is there to pick up the commit we made in
add_upstream_commit, but this indirection is unnecessary in a test, I think. Can we not useadd_upstream_commitand just create our own in subdir/deepsubmodule? (This might conflict with subsequent tests that use the old scheme, but I think that it should be fine.)We can avoid the "git fetch" if we first need to fix an inconsistency in how the submodules are set up. Right now, we have:
test_expect_success setup ' mkdir deepsubmodule && [...] mkdir submodule && ( [...] git submodule add "$pwd/deepsubmodule" subdir/deepsubmodule && git commit -a -m new && git branch -M sub ) && git submodule add "$pwd/submodule" submodule && [...] ( cd downstream && git submodule update --init --recursive ) 'resulting in a directory structure like:
$pwd
|_submodule
|_subdir
|_deepsubmodule
|_deepsubmoduleand upstream/downstream dependencies like:
upstream downstream -------- ---------- $pwd/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)$pwd/submodule/subdir/deepsubmodule
So we can't create the commit in submodule/subdir/deepsubmodule, because that's not where our SUT would fetch from.
Instead, we could fix this by having a more consistent upstream/downstream structure:
$pwd
|_submodule
|_subdir
|_deepsubmodule
Quoted 6 lines
upstream downstream -------- ---------- $pwd/submodule/subdir/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)This seems more convenient to test, but before I commit to this, is there a downside to this that I'm not seeing?
Won't this sort of arrangement create N copies of e.g. a zlib.git or some other common library that might be used by N submodules.
But I haven't read all the context, I'm assuming you're talking about how we store in-tree a/b and x/y/b submodules now, we store those in .git/ both as .git/modules/b.git or whatever? I can't remember ... :)
Whatever we do now I do think the caveat I've noted above is interesting when it comes to submodule design, e.g. if both git.git and some-random-thing.git both bring in the same sha1collisiondetection.git from the same github URL should those be the same in our underlying storage?
I think the answer to that would ideally be both "yes" and "no".
I.e. "yes" because it's surely handy for "git fetch", now you don't need to fetch the same stuff twice in the common case of just updating all our recursive submodules.
And also "no" because maybe some users would really consider them different. E.g. the you may want to "cd git/" and adjust the git.git one and create a branch there for some hotfix it needs, which would not be needed/wanted by some-random-thing.git.
Hrm...
Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules
Quoted 93 lines · Ævar Arnfjörð Bjarmason
Quoted 84 lines · Glen Choo
Quoted 14 lines · Glen Choo
Quoted 12 lines · Jonathan Tan
Quoted 11 lines · Glen Choo
t/t5526-fetch-submodules.sh +11
-0+ # Create new superproject commit with updated submodules + add_upstream_commit && + ( + cd submodule && + ( + cd subdir/deepsubmodule && + git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule"Quoted 8 lines
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 431
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ ) && + git add submodule && + git commit -m "new submodule" &&I thought
add_upstream_commit()would do this, but apparently it just adds commits to the submodules (which works for the earlier tests that just tested that the submodules were fetched, but not for this one). I think that all this should go into its own function.I'm testing out a function that does exactly what these lines do, i.e. create a superproject commit containing a submodule change containing a deepsubmodule change. That works pretty well and it makes sense in the context of the tests.
Quoted 5 lines
Quoted 5 lines
Also, I understand that "git fetch" is there to pick up the commit we made in
add_upstream_commit, but this indirection is unnecessary in a test, I think. Can we not useadd_upstream_commitand just create our own in subdir/deepsubmodule? (This might conflict with subsequent tests that use the old scheme, but I think that it should be fine.)We can avoid the "git fetch" if we first need to fix an inconsistency in how the submodules are set up. Right now, we have:
test_expect_success setup ' mkdir deepsubmodule && [...] mkdir submodule && ( [...] git submodule add "$pwd/deepsubmodule" subdir/deepsubmodule && git commit -a -m new && git branch -M sub ) && git submodule add "$pwd/submodule" submodule && [...] ( cd downstream && git submodule update --init --recursive ) 'resulting in a directory structure like:
$pwd
|_submodule
|_subdir
|_deepsubmodule
|_deepsubmoduleand upstream/downstream dependencies like:
upstream downstream -------- ---------- $pwd/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)$pwd/submodule/subdir/deepsubmodule
So we can't create the commit in submodule/subdir/deepsubmodule, because that's not where our SUT would fetch from.
Instead, we could fix this by having a more consistent upstream/downstream structure:
$pwd
|_submodule
|_subdir
|_deepsubmoduleQuoted 6 lines
upstream downstream -------- ---------- $pwd/submodule/subdir/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)This seems more convenient to test, but before I commit to this, is there a downside to this that I'm not seeing?
Quoted 6 lines
Won't this sort of arrangement create N copies of e.g. a zlib.git or some other common library that might be used by N submodules.
But I haven't read all the context, I'm assuming you're talking about how we store in-tree a/b and x/y/b submodules now, we store those in .git/ both as .git/modules/b.git or whatever? I can't remember ... :)
Ah the problem I'm describing is much simpler, it's just "how do we want our test setup (which has submodules) to look".
But we can also consider the question you are asking :)
Quoted 6 lines
Whatever we do now I do think the caveat I've noted above is interesting when it comes to submodule design, e.g. if both git.git and some-random-thing.git both bring in the same sha1collisiondetection.git from the same github URL should those be the same in our underlying storage?
Quoted 6 lines
I think the answer to that would ideally be both "yes" and "no".
I.e. "yes" because it's surely handy for "git fetch", now you don't need to fetch the same stuff twice in the common case of just updating all our recursive submodules.
Hm, and it would save space on disk.
Quoted 4 lines
And also "no" because maybe some users would really consider them different. E.g. the you may want to "cd git/" and adjust the git.git one and create a branch there for some hotfix it needs, which would not be needed/wanted by some-random-thing.git.
I don't think we could say "yes" for all users, because the subset of users you describe here will probably appreciate them being separate.
But I can imagine doing this manually, like a "git submodule dedupe", that lets users who really need it can opt into this risky setup where submodules are shared. Does anyone really need it though? I'm not sure yet.
Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules
In reply to Glen Choosubmodule.c get_fetch_task(struct submodule_parallel_fetch *spf, · line 1596
Quoted 2 lines · Glen Choo
submodule.c +2
-0+switch (get_fetch_recurse_config (task->sub, spf)) { +default :
Unfortunately get_fetch_recurse_config() returns "int", and the enum fields here defined in submodule.h aren't of a named type, so we can't get the advantage of a complier check for exhaustive enum member checking here...
submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, · line 1605
Quoted 9 lines
submodule.c +9
-0+case RECURSE_SUBMODULES_DEFAULT : +case RECURSE_SUBMODULES_ON_DEMAND : + *default_argv ="on-demand" ; +break ; +case RECURSE_SUBMODULES_ON : + *default_argv ="yes" ; +break ; +case RECURSE_SUBMODULES_OFF : +continue ;
...in any case there's a lot more of them, so just having this "default" case seems to make sense...
PATCH v2 8/9 submodule: read shallows when finding changed submodules
In reply to Glen ChooIn a repository with submodules, "git fetch --update-shallow" can fail. This happens because "git fetch" does not read shallows when rev walking the newly fetched commits to find changed submodules, thus the rev walk may try to read the parent of a shallow and fail. This can occur when --recurse-submodules is not passed, because the default behavior is to fetch changed submodules i.e. --recurse-submodules=on-demand.
Fix this by reading shallows before the rev walk, and test for it.
--- submodule.c | 4 ++++ t/t5526-fetch-submodules.sh | 10 ++++++++++ 2 files changed, 14 insertions(+)
submodule.c +4 -0
@@ -22,6 +22,7 @@#include "parse-options.h" #include "object-store.h" #include "commit-reach.h" +#include "shallow.h" static int config_update_recurse_submodules =RECURSE_SUBMODULES_OFF ;static int initialized_fetch_ref_tips;
@@ -901,6 +902,9 @@ static void collect_changed_submodules(struct repository *r, save_warning = warn_on_object_refname_ambiguity; warn_on_object_refname_ambiguity =0 ; +/* make sure shallows are read */ +is_repository_shallow (the_repository); +repo_init_revisions (r, &rev,NULL );setup_revisions (argv->nr, argv->v, &rev, &s_r_opt); warn_on_object_refname_ambiguity = save_warning;
t/t5526-fetch-submodules.sh +10 -0
@@ -1031,4 +1031,14 @@ test_expect_success 'recursive fetch after deinit a submodule' ' test_cmp expect actual ' +test_expect_success 'recursive fetch does not fail with --update-shallow' ' + git clone --no-local --depth=2 --recurse-submodules . shallow && + git init notshallow && + ( + cd notshallow && + git submodule add ../submodule sub && + git fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* --recurse-submodules + ) +' + test_done
Signature · 1 line
2.33.GIT
Re: [PATCH v2 8/9] submodule: read shallows when finding changed submodules
submodule.c static void collect_changed_submodules(struct repository *r, · line 906
Quoted 6 lines · Glen Choo
submodule.c +2
-0@@ -901,6 +902,9 @@ static void collect_changed_submodules(struct repository *r, save_warning = warn_on_object_refname_ambiguity; warn_on_object_refname_ambiguity =0 ; +/* make sure shallows are read */ +is_repository_shallow (the_repository);
This is presumably to initialize the data structures that some later code reads without first calling is_repository_shallow(). Do we know which part of the later code this is? If yes, it would be better to fix it there.
PATCH v2 9/9 submodule: fix latent check_has_commit() bug
In reply to Glen ChooWhen check_has_commit() is called on a missing submodule, initialization of the struct repository fails, but it attempts to clear the struct anyway (which is a fatal error). This bug is masked by its only caller, submodule_has_commits(), first calling add_submodule_odb() - the latter fails if the submodule does not exist, making submodule_has_commits() exit early and not invoke check_has_commit().
Fix this bug, and because calling add_submodule_odb() is no longer necessary as of 13a2f620b2 (submodule: pass repo to check_has_commit(), 2021-10-08), remove that call too.
This is the last caller of add_submodule_odb(), so remove that function. (Submodule ODBs are still added as alternates via add_submodule_odb_by_path().)
--- submodule.c | 35 ++--------------------------------- submodule.h | 9 ++++----- 2 files changed, 6 insertions(+), 38 deletions(-)
submodule.c +2 -33
@@ -168,26 +168,6 @@ void stage_updated_gitmodules(struct index_state *istate)static struct string_list added_submodule_odb_paths =STRING_LIST_INIT_NODUP ; -/* TODO: remove this function, use repo_submodule_init instead. */ -int add_submodule_odb (const char *path) -{ -struct strbuf objects_directory =STRBUF_INIT ; -int ret =0 ; - - ret =strbuf_git_path_submodule (&objects_directory, path,"objects/" ); -if (ret) - goto done; -if (!is_directory (objects_directory.buf)) { - ret =-1 ; - goto done; - } -string_list_insert (&added_submodule_odb_paths, -strbuf_detach (&objects_directory,NULL )); -done: -strbuf_release (&objects_directory); -return ret; -} -void add_submodule_odb_by_path (const char *path) {string_list_insert (&added_submodule_odb_paths,xstrdup (path));
@@ -966,7 +946,8 @@ static int check_has_commit(const struct object_id *oid, void *data)if (repo_submodule_init (&subrepo, cb->repo, cb->path, cb->super_oid)) { cb->result =0 ; - goto cleanup; +/* subrepo failed to init, so don't clean it up. */ +return 0 ; } type =oid_object_info (&subrepo, oid,NULL );
@@ -997,18 +978,6 @@ static int submodule_has_commits(struct repository *r, {struct has_commit_data has_commit = { r,1 , path, super_oid }; -/* -* Perform a cheap, but incorrect check for the existence of 'commits'. -* This is done by adding the submodule's object store to the in-core -* object store, and then querying for each commit's existence. If we -* do not have the commit object anywhere, there is no chance we have -* it in the object store of the correct submodule and have it -* reachable from a ref, so we can fail early without spawning rev-list -* which is expensive. -*/ -if (add_submodule_odb (path)) -return 0 ; -oid_array_for_each_unique (commits, check_has_commit, &has_commit);if (has_commit.result) {
submodule.h +4 -5
@@ -103,12 +103,11 @@ int submodule_uses_gitfile(const char *path);int bad_to_remove_submodule (const char *path,unsigned flags);/* -* Call add_submodule_odb() to add the submodule at the given path to a list. -* When register_all_submodule_odb_as_alternates() is called, the object stores -* of all submodules in that list will be added as alternates in -* the_repository. +* Call add_submodule_odb_by_path() to add the submodule at the given +* path to a list. When register_all_submodule_odb_as_alternates() is +* called, the object stores of all submodules in that list will be +* added as alternates in the_repository. */ -int add_submodule_odb (const char *path);void add_submodule_odb_by_path (const char *path);int register_all_submodule_odb_as_alternates (void );
Signature · 1 line
2.33.GIT
Re: [PATCH v2 9/9] submodule: fix latent check_has_commit() bug
Quoted 6 lines · Glen Choo
When
check_has_commit()is called on a missing submodule, initialization of the struct repository fails, but it attempts to clear the struct anyway (which is a fatal error). This bug is masked by its only caller,submodule_has_commits(), first callingadd_submodule_odb()- the latter fails if the submodule does not exist, makingsubmodule_has_commits()exit early and not invokecheck_has_commit().
This patch looks good, thanks.
v3 cover fetch --recurse-submodules: fetch unpopulated submodules
In reply to Glen ChooOriginal cover letter: https://lore.kernel.org/git/[email protected]
This series is based on gc/branch-recurse-submodules.
Thanks for the feedback, and sorry for the delay - I've made some substantial changes in response to the feedback, as well as unearthing some surprising bugs.
Patches 2 and 9 have some extra discussion in their --- descriptions. I'd appreciate feedback on those (especially the 'future work' described in patch 9).
= Patch organization
- Patches 1-3 are quality-of-life improvements to the test suite that make it easier to write the tests in patch 9.
- Patches 4-6 are preparation for "git fetch" to read .gitmodules from the superproject commit in patch 7.
- Patches 7-8 refactor out the logic of "finding which submodules to fetch" and "fetching the submodules", making it easier to tell "git fetch" to fetch unpopulated submodules.
- Patch 9 teaches "git fetch" to fetch changed, unpopulated submodules in addition to populated submodules.
- Patch 10 is an optional bugfix + cleanup of the "git fetch" code that removes the last caller of the deprecated "add_submodule_odb()".
= Changes
== Since v2
- Numerous small fixes to the code and commit message (thanks to all who helped spot these :))
- In patch 2, use test_cmp + sed to assert on test output, effectively reverting the "use grep" approach of v1-2 (see patch 2's description).
- New patch 3: introduce a test helper that creates the expected superproject commit (instead of copy-pasting the code over and over).
- I did not get rid of "git fetch" inside the test helper (as Jonathan suggested) though, because that requires a bigger change in the test setup, and I think the test helper makes the test straightforward enough.
- New patch 8: refactor some shared logic out into fetch_task_create(). This reduces code duplication between the get_fetch_task_from_* functions.
- In patch 9, add additional tests for 'submodules with the same name'.
- In patch 9, handle a bug where a submodule that is unpopulated by "git rm" still has "core.worktree" set and cannot be fetched (see patch 9's description).
- Remove the "git fetch --update-shallow" patch (I'll try to send it separately).
== Since v1
- Numerous style fixes suggested by Jonathan (thanks!)
- In patch 3, don't prematurely read submodules from the superproject commit (see:
<[email protected]>).
- In patch 7, stop using "git checkout" and "! grep" in tests.
- In patch 7, stop doing the "find changed submodules" rev walk unconditionally. Instead, continue to check for .gitmodules, but also check for submodules in $GIT_DIR/modules.
- I'm not entirely happy with the helper function name, see "---" for details.
- Move "git fetch --update-shallow" bugfix to patch 8.
- Because the "find changed submodules" rev walk is no longer unconditional, this fix is no longer needed for tests to pass.
- Rename fetch_populated_submodules() to fetch_submodules().
Glen Choo (10):
t5526: introduce test helper to assert on fetches
t5526: stop asserting on stderr literally
t5526: create superproject commits with test helper
submodule: make static functions read submodules from commits
submodule: inline submodule_commits() into caller
submodule: store new submodule commits oid_array in a struct
submodule: extract get_fetch_task()
submodule: move logic into fetch_task_create()
fetch: fetch unpopulated, changed submodules
submodule: fix latent check_has_commit() bug
Documentation/fetch-options.txt | 26 +- Documentation/git-fetch.txt | 10 +- builtin/fetch.c | 14 +- submodule.c | 395 ++++++++++++++--------- submodule.h | 21 +- t/t5526-fetch-submodules.sh | 533 ++++++++++++++++++++++++-------- 6 files changed, 687 insertions(+), 312 deletions(-)
Range-diff against v2:
1: a159cdaabb ! 1: b6d34b0f5c t5526: introduce test helper to assert on fetches
@@ Metadata
## Commit message ##
t5526: introduce test helper to assert on fetches
- A future commit will change the stderr of "git fetch
- --recurse-submodules" and add new tests to t/t5526-fetch-submodules.sh.
- This poses two challenges:
+ Tests in t/t5526-fetch-submodules.sh are unnecessarily noisy:
+
+ * The tests have extra logic in order to reproduce the expected stderr
+ literally, but not all of these details (e.g. the head of the
+ remote-tracking branch before the fetch) are relevant to the test.
- * The tests use test_cmp to assert on the stderr, which will fail on the
- future test because the stderr changes slightly, even though it still
- contains the information we expect.
* The expect.err file is constructed by the add_upstream_commit() helper as input into test_cmp, but most tests fetch a different combination of repos from expect.err. This results in noisy tests that modify
@@ Commit message
helper to t/t5526-fetch-submodules.sh that asserts on the output of "git fetch --recurse-submodules" and handles the ordering of expect.err.
- As a result, the tests no longer construct expect.err manually. test_cmp
- is still invoked by verify_fetch_result(), but that will be replaced in
- a later commit.
+ As a result, the tests no longer construct expect.err manually. Tests
+ still consider the old head of the remote-tracking branch ("$head1"),
+ but that will be fixed in a later commit.
2: 48894c6c43 ! 2: 0b85fa35c2 t5526: use grep to assert on fetches
@@ Metadata
Author: Glen Choo <[email protected]>
## Commit message ##
- t5526: use grep to assert on fetches
+ t5526: stop asserting on stderr literally
- In a previous commit, we replaced test_cmp invocations with
- verify_fetch_result(). Finish the process of removing test_cmp by using
- grep in verify_fetch_result() instead.
+ In the previous commit message, we noted that not all of the "git fetch"
+ stderr is relevant to the tests. Most of the test setup lines are
+ dedicated to these details of the stderr:
- This makes the tests less sensitive to changes because, instead of
- checking the whole stderr, we only grep for the lines of the form
+ 1. which repos (super/sub/deep) are involved in the fetch
+ 2. the head of the remote-tracking branch before the fetch (i.e. $head1)
+ 3. the head of the remote-tracking branch after the fetch (i.e. $head2)
- * "<old-head>..<new-head>\s+branch\s+-> origin/branch"
- * "Fetching submodule <submodule-path>" (if fetching a submodule)
+ 1. and 3. are relevant because they tell us that the expected commit is
+ fetched by the expected repo, but 2. is completely irrelevant.
- when we expect the repo to have fetched. If we expect the repo to not
- have fetched, grep to make sure the lines are absent. Also, simplify the
- assertions by using grep patterns that match only the relevant pieces of
- information, e.g. <old-head> is irrelevant because we only want to know
- if the fetch was performed, so we don't need to know where the branch
- was before the fetch.
+ Stop asserting on $head1 by replacing it with a dummy value in the
+ actual and expected output. Do this by introducing test
+ helpers (check_*()) that make it easier to construct the expected
+ output, and use sed to munge the actual output.
Signed-off-by: Glen Choo <[email protected]>
## t/t5526-fetch-submodules.sh ##
+@@ t/t5526-fetch-submodules.sh: export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB
+
+ pwd=$(pwd)
+
++check_sub() {
++ NEW_HEAD=$1 &&
++ cat <<-EOF >$pwd/expect.err.sub
++ Fetching submodule submodule
++ From $pwd/submodule
++ OLD_HEAD..$NEW_HEAD sub -> origin/sub
++ EOF
++}
++
++check_deep() {
++ NEW_HEAD=$1 &&
++ cat <<-EOF >$pwd/expect.err.deep
++ Fetching submodule submodule/subdir/deepsubmodule
++ From $pwd/deepsubmodule
++ OLD_HEAD..$NEW_HEAD deep -> origin/deep
++ EOF
++}
++
++check_super() {
++ NEW_HEAD=$1 &&
++ cat <<-EOF >$pwd/expect.err.super
++ From $pwd/.
++ OLD_HEAD..$NEW_HEAD super -> origin/super
++ EOF
++}
++
+ # For each submodule in the test setup, this creates a commit and writes
+ # a file that contains the expected err if that new commit were fetched.
+ # These output files get concatenated in the right order by
@@ t/t5526-fetch-submodules.sh: pwd=$(pwd)
add_upstream_commit() {
(
@@ t/t5526-fetch-submodules.sh: pwd=$(pwd)
- echo "Fetching submodule submodule" > ../expect.err.sub &&
- echo "From $pwd/submodule" >> ../expect.err.sub &&
- echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub
-+ git rev-parse --short HEAD >../subhead
++ new_head=$(git rev-parse --short HEAD) &&
++ check_sub $new_head
) &&
(
cd deepsubmodule &&
@@ t/t5526-fetch-submodules.sh: pwd=$(pwd)
- echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep
- echo "From $pwd/deepsubmodule" >> ../expect.err.deep &&
- echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep
-+ git rev-parse --short HEAD >../deephead
++ new_head=$(git rev-parse --short HEAD) &&
++ check_deep $new_head
)
}
- # Verifies that the expected repositories were fetched. This is done by
--# concatenating the files expect.err.[super|sub|deep] in the correct
--# order and comparing it to the actual stderr.
-+# checking that the branches of [super|sub|deep] were updated to
-+# [super|sub|deep]head if the corresponding file exists.
- #
--# If a repo should not be fetched in the test, its corresponding
--# expect.err file should be rm-ed.
-+# If the [super|sub|deep] head file does not exist, this verifies that
-+# the corresponding repo was not fetched. Thus, if a repo should not be
-+# fetched in the test, its corresponding head file should be
-+# rm-ed.
- verify_fetch_result() {
- ACTUAL_ERR=$1 &&
-- rm -f expect.err.combined &&
-- if [ -f expect.err.super ]; then
-- cat expect.err.super >>expect.err.combined
-+ # Each grep pattern is guaranteed to match the correct repo
-+ # because each repo uses a different name for their branch i.e.
-+ # "super", "sub" and "deep".
-+ if [ -f superhead ]; then
-+ grep -E "\.\.$(cat superhead)\s+super\s+-> origin/super" $ACTUAL_ERR
-+ else
-+ ! grep "super" $ACTUAL_ERR
- fi &&
-- if [ -f expect.err.sub ]; then
-- cat expect.err.sub >>expect.err.combined
-+ if [ -f subhead ]; then
-+ grep "Fetching submodule submodule" $ACTUAL_ERR &&
-+ grep -E "\.\.$(cat subhead)\s+sub\s+-> origin/sub" $ACTUAL_ERR
-+ else
-+ ! grep "Fetching submodule submodule" $ACTUAL_ERR
+@@ t/t5526-fetch-submodules.sh: verify_fetch_result() {
+ if [ -f expect.err.deep ]; then
+ cat expect.err.deep >>expect.err.combined
fi &&
-- if [ -f expect.err.deep ]; then
-- cat expect.err.deep >>expect.err.combined
-- fi &&
- test_cmp expect.err.combined $ACTUAL_ERR
-+ if [ -f deephead ]; then
-+ grep "Fetching submodule submodule/subdir/deepsubmodule" $ACTUAL_ERR &&
-+ grep -E "\.\.$(cat deephead)\s+deep\s+-> origin/deep" $ACTUAL_ERR
-+ else
-+ ! grep "Fetching submodule submodule/subdir/deepsubmodule" $ACTUAL_ERR
-+ fi
++ sed -E 's/[0-9a-f]+\.\./OLD_HEAD\.\./' $ACTUAL_ERR >actual.err.cmp &&
++ test_cmp expect.err.combined actual.err.cmp
}
test_expect_success setup '
@@ t/t5526-fetch-submodules.sh: test_expect_success "Recursion doesn't happen when
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >>expect.err.super &&
-- rm expect.err.deep &&
-+ git rev-parse --short HEAD >superhead &&
-+ rm deephead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
+ rm expect.err.deep &&
(
cd downstream &&
- git fetch >../actual.out 2>../actual.err
@@ t/t5526-fetch-submodules.sh: test_expect_success "Recursion stops when no new submodule commits are fetched"
test_expect_success "Recursion doesn't happen when new superproject commits don't change any submodules" '
@@ t/t5526-fetch-submodules.sh: test_expect_success "Recursion stops when no new su
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >> expect.err.super &&
-- rm expect.err.sub &&
-- rm expect.err.deep &&
-+ git rev-parse --short HEAD >superhead &&
-+ rm subhead &&
-+ rm deephead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
+ rm expect.err.sub &&
+ rm expect.err.deep &&
(
- cd downstream &&
- git fetch >../actual.out 2>../actual.err
@@ t/t5526-fetch-submodules.sh: test_expect_success "Recursion picks up config in submodule" '
)
) &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "Recursion picks up config in s
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >> expect.err.super &&
-+ git rev-parse --short HEAD >superhead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
(
cd downstream &&
git fetch >../actual.out 2>../actual.err &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "Recursion picks up all submodu
- echo "Fetching submodule submodule" > ../expect.err.sub &&
- echo "From $pwd/submodule" >> ../expect.err.sub &&
- echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub
-+ git rev-parse --short HEAD >../subhead
++ new_head=$(git rev-parse --short HEAD) &&
++ check_sub $new_head
) &&
- head1=$(git rev-parse --short HEAD) &&
git add submodule &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "Recursion picks up all submodu
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >> expect.err.super &&
-+ git rev-parse --short HEAD >superhead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
(
cd downstream &&
git fetch >../actual.out 2>../actual.err
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
- echo Fetching submodule submodule > ../expect.err.sub &&
- echo "From $pwd/submodule" >> ../expect.err.sub &&
- echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub
-+ git rev-parse --short HEAD >../subhead
++ new_head=$(git rev-parse --short HEAD) &&
++ check_sub $new_head
) &&
(
cd downstream &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >>expect.err.super &&
-+ git rev-parse --short HEAD >superhead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
(
cd downstream &&
git config fetch.recurseSubmodules false &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >> expect.err.super &&
-- rm expect.err.sub &&
-- rm expect.err.deep &&
-+ git rev-parse --short HEAD >superhead &&
-+ rm subhead &&
-+ rm deephead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
+ rm expect.err.sub &&
+ rm expect.err.deep &&
(
- cd downstream &&
- git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err
@@ t/t5526-fetch-submodules.sh: test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config
) &&
add_upstream_commit &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'fetch.recurseSubmodules=on-de
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >>expect.err.super &&
-- rm expect.err.deep &&
-+ git rev-parse --short HEAD >superhead &&
-+ rm deephead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
+ rm expect.err.deep &&
(
cd downstream &&
- git config fetch.recurseSubmodules on-demand &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override
) &&
add_upstream_commit &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'submodule.<sub>.fetchRecurseS
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >>expect.err.super &&
-- rm expect.err.deep &&
-+ git rev-parse --short HEAD >superhead &&
-+ rm deephead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
+ rm expect.err.deep &&
(
cd downstream &&
- git config submodule.submodule.fetchRecurseSubmodules on-demand &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "don't fetch submodule when newly recorded commits are alrea
cd submodule &&
git checkout -q HEAD^^
@@ t/t5526-fetch-submodules.sh: test_expect_success "don't fetch submodule when new
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." > expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >> expect.err.super &&
-- rm expect.err.sub &&
-+ git rev-parse --short HEAD >superhead &&
-+ rm subhead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
+ rm expect.err.sub &&
# This file does not exist, but rm -f for readability
-- rm -f expect.err.deep &&
-+ rm -f deephead &&
- (
- cd downstream &&
- git fetch >../actual.out 2>../actual.err
+ rm -f expect.err.deep &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git
git fetch --recurse-submodules
) &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'fetch.recurseSubmodules=on-de
- head2=$(git rev-parse --short HEAD) &&
- echo "From $pwd/." >expect.err.super &&
- echo " $head1..$head2 super -> origin/super" >>expect.err.super &&
-- rm expect.err.deep &&
-+ git rev-parse --short HEAD >superhead &&
-+ rm deephead &&
++ new_head=$(git rev-parse --short HEAD) &&
++ check_super $new_head &&
+ rm expect.err.deep &&
(
cd downstream &&
- rm .gitmodules &&
-: ---------- > 3: bb8ef6094a t5526: create superproject commits with test helper
3: 6cf5e76d62 ! 4: e83a1713c4 submodule: make static functions read submodules from commits
@@ Commit message
Submodules will be read from commits when we fetch unpopulated
submodules.
- The changed function signatures follow repo_submodule_init()'s argument
- order, i.e. "path" then "treeish_name". Where needed, reorder the
- arguments of functions that already take "path" and "treeish_name" to be
- consistent with this convention.
-
Signed-off-by: Glen Choo <[email protected]>
## submodule.c ##
@@ submodule.c: static int check_has_commit(const struct object_id *oid, void *data
struct oid_array *commits)
{
- struct has_commit_data has_commit = { r, 1, path };
-+ struct has_commit_data has_commit = { r, 1, path, super_oid };
++ struct has_commit_data has_commit = {
++ .repo = r,
++ .result = 1,
++ .path = path,
++ .super_oid = super_oid
++ };
/*
* Perform a cheap, but incorrect check for the existence of 'commits'.
4: 07fd4ff0a9 = 5: e27d402b9a submodule: inline submodule_commits() into caller
5: f049cb231b ! 6: 1c7c8218b8 submodule: store new submodule commits oid_array in a struct
@@ submodule.c: static void collect_changed_submodules(struct repository *r,
+static void free_submodules_data(struct string_list *submodules)
{
struct string_list_item *item;
-- for_each_string_list_item(item, submodules)
+ for_each_string_list_item(item, submodules)
- oid_array_clear((struct oid_array *) item->util);
-+ for_each_string_list_item(item, submodules) {
+ changed_submodule_data_clear(item->util);
-+ }
++
string_list_clear(submodules, 1);
} 6: 814073eecc ! 7: 80cf317722 submodule: extract get_fetch_task()
@@ Commit message
Signed-off-by: Glen Choo <[email protected]>
## submodule.c ##
+@@ submodule.c: struct fetch_task {
+ struct repository *repo;
+ const struct submodule *sub;
+ unsigned free_sub : 1; /* Do we need to free the submodule? */
++ const char *default_argv;
+
+ struct oid_array *commits; /* Ensure these commits are fetched */
+ };
@@ submodule.c: static struct repository *get_submodule_repo_for(struct repository *r,
return ret;
}
@@ submodule.c: static struct repository *get_submodule_repo_for(struct repository
-static int get_next_submodule(struct child_process *cp,
- struct strbuf *err, void *data, void **task_cb)
+static struct fetch_task *
-+get_fetch_task(struct submodule_parallel_fetch *spf,
-+ const char **default_argv, struct strbuf *err)
++get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)
{
- struct submodule_parallel_fetch *spf = data;
-
@@ submodule.c: static int get_next_submodule(struct child_process *cp,
task->sub->name))
continue;
- default_argv = "on-demand";
-+ *default_argv = "on-demand";
++ task->default_argv = "on-demand";
break;
case RECURSE_SUBMODULES_ON:
- default_argv = "yes";
-+ *default_argv = "yes";
++ task->default_argv = "yes";
break;
case RECURSE_SUBMODULES_OFF:
continue;
@@ submodule.c: static int get_next_submodule(struct child_process *cp,
+ void *data, void **task_cb)
+{
+ struct submodule_parallel_fetch *spf = data;
-+ const char *default_argv = NULL;
-+ struct fetch_task *task = get_fetch_task(spf, &default_argv, err);
++ struct fetch_task *task = get_fetch_task(spf, err);
+
+ if (task) {
+ struct strbuf submodule_prefix = STRBUF_INIT;
@@ submodule.c: static int get_next_submodule(struct child_process *cp,
+ cp->git_cmd = 1;
+ strvec_init(&cp->args);
+ strvec_pushv(&cp->args, spf->args.v);
-+ strvec_push(&cp->args, default_argv);
++ strvec_push(&cp->args, task->default_argv);
+ strvec_push(&cp->args, "--submodule-prefix");
+
+ strbuf_addf(&submodule_prefix, "%s%s/",
-: ---------- > 8: bf9cfa7054 submodule: move logic into fetch_task_create()
7: 10fd5bf921 ! 9: c7c2ff71b6 fetch: fetch unpopulated, changed submodules
@@ Commit message
commands like "git checkout --recurse-submodules" might fail.
Teach "git fetch" to fetch cloned, changed submodules regardless of
- whether they are populated (this is in addition to the current behavior
- of fetching populated submodules).
+ whether they are populated. This is in addition to the current behavior
+ of fetching populated submodules (which is always attempted regardless
+ of what was fetched in the superproject, or even if nothing was fetched
+ in the superproject).
- Since a submodule may be encountered multiple times (via the list of
- populated submodules or via the list of changed submodules), maintain a
- list of seen submodules to avoid fetching a submodule more than once.
+ A submodule may be encountered multiple times (via the list of
+ populated submodules or via the list of changed submodules). When this
+ happens, "git fetch" only reads the 'populated copy' and ignores the
+ 'changed copy'. Amend the verify_fetch_result() test helper so that we
+ can assert on which 'copy' is being read.
Signed-off-by: Glen Choo <[email protected]>
@@ submodule.c: void check_for_new_submodule_commits(struct object_id *oid)
}
+/*
-+ * Returns 1 if the repo has absorbed submodule gitdirs, and 0
-+ * otherwise. Like submodule_name_to_gitdir(), this checks
++ * Returns 1 if there is at least one submodule gitdir in
++ * $GIT_DIR/modules and 0 otherwise. This follows
++ * submodule_name_to_gitdir(), which looks for submodules in
+ * $GIT_DIR/modules, not $GIT_COMMON_DIR.
++ *
++ * A submodule can be moved to $GIT_DIR/modules manually by running "git
++ * submodule absorbgitdirs", or it may be initialized there by "git
++ * submodule update".
+ */
+static int repo_has_absorbed_submodules(struct repository *r)
+{
@@ submodule.c: struct submodule_parallel_fetch {
.submodules_with_errors = STRBUF_INIT, \
}
-@@ submodule.c: static struct repository *get_submodule_repo_for(struct repository *r,
+@@ submodule.c: struct fetch_task {
+ const struct submodule *sub;
+ unsigned free_sub : 1; /* Do we need to free the submodule? */
+ const char *default_argv;
++ struct strvec git_args;
+
+ struct oid_array *commits; /* Ensure these commits are fetched */
+ };
+@@ submodule.c: static void fetch_task_release(struct fetch_task *p)
+ if (p->repo)
+ repo_clear(p->repo);
+ FREE_AND_NULL(p->repo);
++
++ strvec_clear(&p->git_args);
+ }
+
+ static struct repository *get_submodule_repo_for(struct repository *r,
+@@ submodule.c: static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf
+ task->free_sub = 1;
+ }
+
++ if (string_list_lookup(&spf->seen_submodule_names, task->sub->name))
++ goto cleanup;
++
+ switch (get_fetch_recurse_config(task->sub, spf))
+ {
+ default:
+@@ submodule.c: static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf
}
static struct fetch_task *
--get_fetch_task(struct submodule_parallel_fetch *spf,
-- const char **default_argv, struct strbuf *err)
+-get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)
+get_fetch_task_from_index(struct submodule_parallel_fetch *spf,
-+ const char **default_argv, struct strbuf *err)
++ struct strbuf *err)
{
- for (; spf->count < spf->r->index->cache_nr; spf->count++) {
- const struct cache_entry *ce = spf->r->index->cache[spf->count];
@@ submodule.c: static struct repository *get_submodule_repo_for(struct repository
struct fetch_task *task;
if (!S_ISGITLINK(ce->ce_mode))
-@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
- if (!task)
- continue;
-
-+ /*
-+ * We might have already considered this submodule
-+ * because we saw it when iterating the changed
-+ * submodule names.
-+ */
-+ if (string_list_lookup(&spf->seen_submodule_names,
-+ task->sub->name))
-+ continue;
-+
- switch (get_fetch_recurse_config(task->sub, spf))
- {
- default:
-@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
+@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)
strbuf_addf(err, _("Fetching submodule %s%s\n"),
spf->prefix, ce->name);
@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
return task;
} else {
struct strbuf empty_submodule_path = STRBUF_INIT;
-@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
+@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)
return NULL;
}
+static struct fetch_task *
+get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,
-+ const char **default_argv, struct strbuf *err)
++ struct strbuf *err)
+{
+ for (; spf->changed_count < spf->changed_submodule_names.nr;
+ spf->changed_count++) {
@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
+ struct changed_submodule_data *cs_data = item.util;
+ struct fetch_task *task;
+
-+ /*
-+ * We might have already considered this submodule
-+ * because we saw it in the index.
-+ */
-+ if (string_list_lookup(&spf->seen_submodule_names, item.string))
++ if (!is_tree_submodule_active(spf->r, cs_data->super_oid,cs_data->path))
+ continue;
+
-+ task = fetch_task_create(spf->r, cs_data->path,
++ task = fetch_task_create(spf, cs_data->path,
+ cs_data->super_oid);
+ if (!task)
+ continue;
+
-+ switch (get_fetch_recurse_config(task->sub, spf)) {
-+ default:
-+ case RECURSE_SUBMODULES_DEFAULT:
-+ case RECURSE_SUBMODULES_ON_DEMAND:
-+ *default_argv = "on-demand";
-+ break;
-+ case RECURSE_SUBMODULES_ON:
-+ *default_argv = "yes";
-+ break;
-+ case RECURSE_SUBMODULES_OFF:
-+ continue;
-+ }
-+
-+ task->repo = get_submodule_repo_for(spf->r, task->sub->path,
-+ cs_data->super_oid);
+ if (!task->repo) {
++ strbuf_addf(err, _("Could not access submodule '%s' at commit %s\n"),
++ cs_data->path,
++ find_unique_abbrev(cs_data->super_oid, DEFAULT_ABBREV));
++
+ fetch_task_release(task);
+ free(task);
-+
-+ strbuf_addf(err, _("Could not access submodule '%s'\n"),
-+ cs_data->path);
+ continue;
+ }
-+ if (!is_tree_submodule_active(spf->r, cs_data->super_oid,
-+ task->sub->path))
-+ continue;
+
+ if (!spf->quiet)
+ strbuf_addf(err,
@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
+ spf->prefix, task->sub->path,
+ find_unique_abbrev(cs_data->super_oid,
+ DEFAULT_ABBREV));
++
+ spf->changed_count++;
++ /*
++ * NEEDSWORK: A submodule unpopulated by "git rm" will
++ * have core.worktree set, but the actual core.worktree
++ * directory won't exist, causing the child process to
++ * fail. Forcibly set --work-tree until we get smarter
++ * handling for core.worktree in unpopulated submodules.
++ */
++ strvec_push(&task->git_args, "--work-tree=.");
+ return task;
+ }
+ return NULL;
@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,
void *data, void **task_cb)
{
struct submodule_parallel_fetch *spf = data;
- const char *default_argv = NULL;
-- struct fetch_task *task = get_fetch_task(spf, &default_argv, err);
+- struct fetch_task *task = get_fetch_task(spf, err);
+ struct fetch_task *task =
-+ get_fetch_task_from_index(spf, &default_argv, err);
++ get_fetch_task_from_index(spf, err);
+ if (!task)
-+ task = get_fetch_task_from_changed(spf, &default_argv, err);
++ task = get_fetch_task_from_changed(spf, err);
if (task) {
struct strbuf submodule_prefix = STRBUF_INIT;
+@@ submodule.c: static int get_next_submodule(struct child_process *cp, struct strbuf *err,
+ prepare_submodule_repo_env_in_gitdir(&cp->env_array);
+ cp->git_cmd = 1;
+ strvec_init(&cp->args);
++ if (task->git_args.nr)
++ strvec_pushv(&cp->args, task->git_args.v);
+ strvec_pushv(&cp->args, spf->args.v);
+ strvec_push(&cp->args, task->default_argv);
+ strvec_push(&cp->args, "--submodule-prefix");
@@ submodule.c: static int get_next_submodule(struct child_process *cp, struct strbuf *err,
*task_cb = task;
@@ submodule.h: int should_update_submodules(void);
## t/t5526-fetch-submodules.sh ##
+@@ t/t5526-fetch-submodules.sh: pwd=$(pwd)
+
+ check_sub() {
+ NEW_HEAD=$1 &&
++ SUPER_HEAD=$2 &&
+ cat <<-EOF >$pwd/expect.err.sub
+- Fetching submodule submodule
++ Fetching submodule submodule${SUPER_HEAD:+ at commit $SUPER_HEAD}
+ From $pwd/submodule
+ OLD_HEAD..$NEW_HEAD sub -> origin/sub
+ EOF
+@@ t/t5526-fetch-submodules.sh: check_sub() {
+
+ check_deep() {
+ NEW_HEAD=$1 &&
++ SUB_HEAD=$2 &&
+ cat <<-EOF >$pwd/expect.err.deep
+- Fetching submodule submodule/subdir/deepsubmodule
++ Fetching submodule submodule/subdir/deepsubmodule${SUB_HEAD:+ at commit $SUB_HEAD}
+ From $pwd/deepsubmodule
+ OLD_HEAD..$NEW_HEAD deep -> origin/deep
+ EOF
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess
verify_fetch_result actual.err
'
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ (
+ cd downstream &&
+ git checkout --recurse-submodules -b no-submodules &&
-+ rm .gitmodules &&
++ git rm .gitmodules &&
+ git rm submodule &&
-+ git add .gitmodules &&
+ git commit -m "no submodules" &&
+ git checkout --recurse-submodules super
+ )
+'
+
+test_expect_success "'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules" '
-+ git -C downstream fetch --recurse-submodules &&
-+ # Create new superproject commit with updated submodules
-+ add_upstream_commit &&
-+ (
-+ cd submodule &&
-+ (
-+ cd subdir/deepsubmodule &&
-+ git fetch &&
-+ git checkout -q FETCH_HEAD
-+ ) &&
-+ git add subdir/deepsubmodule &&
-+ git commit -m "new deep submodule"
-+ ) &&
-+ git add submodule &&
-+ git commit -m "new submodule" &&
-+
++ add_submodule_commits &&
++ add_superproject_commits &&
+ # Fetch the new superproject commit
+ (
+ cd downstream &&
+ git switch --recurse-submodules no-submodules &&
+ git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err
+ ) &&
-+ test_must_be_empty actual.out &&
-+ git rev-parse --short HEAD >superhead &&
-+ git -C submodule rev-parse --short HEAD >subhead &&
-+ git -C deepsubmodule rev-parse --short HEAD >deephead &&
-+ verify_fetch_result actual.err &&
++ super_head=$(git rev-parse --short HEAD) &&
++ sub_head=$(git -C submodule rev-parse --short HEAD) &&
++ deep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&
++
++ # assert that these are fetched from commits, not the index
++ check_sub $sub_head $super_head &&
++ check_deep $deep_head $sub_head &&
+
-+ # Assert that the fetch happened at the non-HEAD commits
-+ grep "Fetching submodule submodule at commit $superhead" actual.err &&
-+ grep "Fetching submodule submodule/subdir/deepsubmodule at commit $subhead" actual.err
++ test_must_be_empty actual.out &&
++ verify_fetch_result actual.err
+'
+
+test_expect_success "'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules" '
-+ # Fetch any leftover commits from other tests.
-+ git -C downstream fetch --recurse-submodules &&
-+ # Create new superproject commit with updated submodules
-+ add_upstream_commit &&
-+ (
-+ cd submodule &&
-+ (
-+ cd subdir/deepsubmodule &&
-+ git fetch &&
-+ git checkout -q FETCH_HEAD
-+ ) &&
-+ git add subdir/deepsubmodule &&
-+ git commit -m "new deep submodule"
-+ ) &&
-+ git add submodule &&
-+ git commit -m "new submodule" &&
-+
++ add_submodule_commits &&
++ add_superproject_commits &&
+ # Fetch the new superproject commit
+ (
+ cd downstream &&
+ git switch --recurse-submodules no-submodules &&
+ git fetch --recurse-submodules >../actual.out 2>../actual.err
+ ) &&
-+ test_must_be_empty actual.out &&
-+ git rev-parse --short HEAD >superhead &&
-+ git -C submodule rev-parse --short HEAD >subhead &&
-+ git -C deepsubmodule rev-parse --short HEAD >deephead &&
-+ verify_fetch_result actual.err &&
++ super_head=$(git rev-parse --short HEAD) &&
++ sub_head=$(git -C submodule rev-parse --short HEAD) &&
++ deep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&
++
++ # assert that these are fetched from commits, not the index
++ check_sub $sub_head $super_head &&
++ check_deep $deep_head $sub_head &&
+
-+ # Assert that the fetch happened at the non-HEAD commits
-+ grep "Fetching submodule submodule at commit $superhead" actual.err &&
-+ grep "Fetching submodule submodule/subdir/deepsubmodule at commit $subhead" actual.err
++ test_must_be_empty actual.out &&
++ verify_fetch_result actual.err
+'
+
+test_expect_success "'--recurse-submodules' should ignore changed, inactive submodules" '
-+ # Fetch any leftover commits from other tests.
-+ git -C downstream fetch --recurse-submodules &&
-+ # Create new superproject commit with updated submodules
-+ add_upstream_commit &&
-+ (
-+ cd submodule &&
-+ (
-+ cd subdir/deepsubmodule &&
-+ git fetch &&
-+ git checkout -q FETCH_HEAD
-+ ) &&
-+ git add subdir/deepsubmodule &&
-+ git commit -m "new deep submodule"
-+ ) &&
-+ git add submodule &&
-+ git commit -m "new submodule" &&
++ add_submodule_commits &&
++ add_superproject_commits &&
+
+ # Fetch the new superproject commit
+ (
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ git -c submodule.submodule.active=false fetch --recurse-submodules >../actual.out 2>../actual.err
+ ) &&
+ test_must_be_empty actual.out &&
-+ git rev-parse --short HEAD >superhead &&
++ super_head=$(git rev-parse --short HEAD) &&
++ check_super $super_head &&
+ # Neither should be fetched because the submodule is inactive
-+ rm subhead &&
-+ rm deephead &&
++ rm expect.err.sub &&
++ rm expect.err.deep &&
+ verify_fetch_result actual.err
+'
+
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+'
+
+test_expect_success "'--recurse-submodules' should fetch submodule commits in changed submodules and the index" '
-+ # Fetch any leftover commits from other tests.
-+ git -C downstream fetch --recurse-submodules &&
+ # Create new commit in origin/super
-+ add_upstream_commit &&
-+ (
-+ cd submodule &&
-+ (
-+ cd subdir/deepsubmodule &&
-+ git fetch &&
-+ git checkout -q FETCH_HEAD
-+ ) &&
-+ git add subdir/deepsubmodule &&
-+ git commit -m "new deep submodule"
-+ ) &&
-+ git add submodule &&
-+ git commit -m "new submodule" &&
++ add_submodule_commits &&
++ add_superproject_commits &&
+
+ # Create new commit in origin/super-sub2-only
+ git checkout super-sub2-only &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ git fetch --recurse-submodules >../actual.out 2>../actual.err
+ ) &&
+ test_must_be_empty actual.out &&
-+
-+ # Assert that the submodules in the super branch are fetched
-+ git rev-parse --short HEAD >superhead &&
-+ git -C submodule rev-parse --short HEAD >subhead &&
-+ git -C deepsubmodule rev-parse --short HEAD >deephead &&
-+ verify_fetch_result actual.err &&
-+ # grep for the exact line to check that the submodule is read
-+ # from the index, not from a commit
-+ grep "^Fetching submodule submodule\$" actual.err &&
-+
-+ # Assert that super-sub2-only and submodule2 were fetched even
-+ # though another branch is checked out
++ deep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&
++ sub_head=$(git -C submodule rev-parse --short HEAD) &&
++ sub2_head=$(git -C submodule2 rev-parse --short HEAD) &&
++ super_head=$(git rev-parse --short HEAD) &&
+ super_sub2_only_head=$(git rev-parse --short super-sub2-only) &&
-+ grep -E "\.\.${super_sub2_only_head}\s+super-sub2-only\s+-> origin/super-sub2-only" actual.err &&
-+ grep "Fetching submodule submodule2 at commit $super_sub2_only_head" actual.err &&
-+ sub2head=$(git -C submodule2 rev-parse --short HEAD) &&
-+ grep -E "\.\.${sub2head}\s+sub2\s+-> origin/sub2" actual.err
++
++ # Use test_cmp manually because verify_fetch_result does not
++ # consider submodule2. All the repos should be fetched, but only
++ # submodule2 should be read from a commit
++ cat <<-EOF > expect.err.combined &&
++ From $pwd/.
++ OLD_HEAD..$super_head super -> origin/super
++ OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only
++ Fetching submodule submodule
++ From $pwd/submodule
++ OLD_HEAD..$sub_head sub -> origin/sub
++ Fetching submodule submodule/subdir/deepsubmodule
++ From $pwd/deepsubmodule
++ OLD_HEAD..$deep_head deep -> origin/deep
++ Fetching submodule submodule2 at commit $super_sub2_only_head
++ From $pwd/submodule2
++ OLD_HEAD..$sub2_head sub2 -> origin/sub2
++ EOF
++ sed -E "s/[0-9a-f]+\.\./OLD_HEAD\.\./" actual.err >actual.err.cmp &&
++ test_cmp expect.err.combined actual.err.cmp
+'
+
test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" '
- add_upstream_commit &&
+ add_submodule_commits &&
echo a >> file &&
+@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a submodule' '
+ test_cmp expect actual
+ '
+
++test_expect_success 'setup repo with upstreams that share a submodule name' '
++ mkdir same-name-1 &&
++ (
++ cd same-name-1 &&
++ git init &&
++ test_commit --no-tag a
++ ) &&
++ git clone same-name-1 same-name-2 &&
++ # same-name-1 and same-name-2 both add a submodule with the
++ # name "submodule"
++ (
++ cd same-name-1 &&
++ mkdir submodule &&
++ git -C submodule init &&
++ test_commit -C submodule --no-tag a1 &&
++ git submodule add "$pwd/same-name-1/submodule" &&
++ git add submodule &&
++ git commit -m "super-a1"
++ ) &&
++ (
++ cd same-name-2 &&
++ mkdir submodule &&
++ git -C submodule init &&
++ test_commit -C submodule --no-tag a2 &&
++ git submodule add "$pwd/same-name-2/submodule" &&
++ git add submodule &&
++ git commit -m "super-a2"
++ ) &&
++ git clone same-name-1 -o same-name-1 same-name-downstream &&
++ (
++ cd same-name-downstream &&
++ git remote add same-name-2 ../same-name-2 &&
++ git fetch --all &&
++ # init downstream with same-name-1
++ git submodule update --init
++ )
++'
++
++test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '
++ test_when_finished "git -C same-name-downstream checkout master" &&
++ (
++ cd same-name-1 &&
++ test_commit -C submodule --no-tag b1 &&
++ git add submodule &&
++ git commit -m "super-b1"
++ ) &&
++ (
++ cd same-name-2 &&
++ test_commit -C submodule --no-tag b2 &&
++ git add submodule &&
++ git commit -m "super-b2"
++ ) &&
++ (
++ cd same-name-downstream &&
++ # even though the .gitmodules is correct, we cannot
++ # fetch from same-name-2
++ git checkout same-name-2/master &&
++ git fetch --recurse-submodules same-name-1 &&
++ test_must_fail git fetch --recurse-submodules same-name-2
++ ) &&
++ super_head1=$(git -C same-name-1 rev-parse HEAD) &&
++ git -C same-name-downstream cat-file -e $super_head1 &&
++
++ super_head2=$(git -C same-name-2 rev-parse HEAD) &&
++ git -C same-name-downstream cat-file -e $super_head2 &&
++
++ sub_head1=$(git -C same-name-1/submodule rev-parse HEAD) &&
++ git -C same-name-downstream/submodule cat-file -e $sub_head1 &&
++
++ sub_head2=$(git -C same-name-2/submodule rev-parse HEAD) &&
++ test_must_fail git -C same-name-downstream/submodule cat-file -e $sub_head2
++'
++
++test_expect_success 'fetch --recurse-submodules updates name-conflicted, unpopulated submodule' '
++ (
++ cd same-name-1 &&
++ test_commit -C submodule --no-tag c1 &&
++ git add submodule &&
++ git commit -m "super-c1"
++ ) &&
++ (
++ cd same-name-2 &&
++ test_commit -C submodule --no-tag c2 &&
++ git add submodule &&
++ git commit -m "super-c2"
++ ) &&
++ (
++ cd same-name-downstream &&
++ git checkout master &&
++ git rm .gitmodules &&
++ git rm submodule &&
++ git commit -m "no submodules" &&
++ git fetch --recurse-submodules same-name-1
++ ) &&
++ head1=$(git -C same-name-1/submodule rev-parse HEAD) &&
++ head2=$(git -C same-name-2/submodule rev-parse HEAD) &&
++ (
++ cd same-name-downstream/.git/modules/submodule &&
++ # The submodule has core.worktree pointing to the "git
++ # rm"-ed directory, overwrite the invalid value.
++ git --work-tree=. cat-file -e $head1 &&
++ test_must_fail git --work-tree=. cat-file -e $head2
++ )
++'
++
+ test_done
8: 8aa68111b0 < -: ---------- submodule: read shallows when finding changed submodules
9: 05a8b93154 ! 10: e1ac74eee4 submodule: fix latent check_has_commit() bug @@ submodule.c: static int check_has_commit(const struct object_id *oid, void *data
type = oid_object_info(&subrepo, oid, NULL);
@@ submodule.c: static int submodule_has_commits(struct repository *r,
- {
- struct has_commit_data has_commit = { r, 1, path, super_oid };
+ .super_oid = super_oid
+ };
- /*
- * Perform a cheap, but incorrect check for the existence of 'commits'.
base-commit: 679e3693aba0c17af60c031f7eef68f2296b8dad
Signature · 1 line
2.33.GIT
PATCH v3 1/10 t5526: introduce test helper to assert on fetches
Tests in t/t5526-fetch-submodules.sh are unnecessarily noisy:
* The tests have extra logic in order to reproduce the expected stderr literally, but not all of these details (e.g. the head of the remote-tracking branch before the fetch) are relevant to the test.
* The expect.err file is constructed by the add_upstream_commit() helper as input into test_cmp, but most tests fetch a different combination of repos from expect.err. This results in noisy tests that modify parts of that expect.err to generate the expected output.
To address both of these issues, introduce a verify_fetch_result() helper to t/t5526-fetch-submodules.sh that asserts on the output of "git fetch --recurse-submodules" and handles the ordering of expect.err.
As a result, the tests no longer construct expect.err manually. Tests still consider the old head of the remote-tracking branch ("$head1"), but that will be fixed in a later commit.
--- t/t5526-fetch-submodules.sh | 136 +++++++++++++++++++++--------------- 1 file changed, 81 insertions(+), 55 deletions(-)
t/t5526-fetch-submodules.sh +81 -55
@@ -13,6 +13,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +# For each submodule in the test setup, this creates a commit and writes +# a file that contains the expected err if that new commit were fetched. +# These output files get concatenated in the right order by +# verify_fetch_result(). add_upstream_commit() { ( cd submodule &&
@@ -22,9 +26,9 @@ add_upstream_commit() { git add subfile && git commit -m new subfile && head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err && - echo "From $pwd/submodule" >> ../expect.err && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err + echo "Fetching submodule submodule" > ../expect.err.sub && + echo "From $pwd/submodule" >> ../expect.err.sub && + echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub ) && ( cd deepsubmodule &&
@@ -34,12 +38,33 @@ add_upstream_commit() { git add deepsubfile && git commit -m new deepsubfile && head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule/subdir/deepsubmodule" >> ../expect.err - echo "From $pwd/deepsubmodule" >> ../expect.err && - echo " $head1..$head2 deep -> origin/deep" >> ../expect.err + echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep + echo "From $pwd/deepsubmodule" >> ../expect.err.deep && + echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep ) } +# Verifies that the expected repositories were fetched. This is done by +# concatenating the files expect.err.[super|sub|deep] in the correct +# order and comparing it to the actual stderr. +# +# If a repo should not be fetched in the test, its corresponding +# expect.err file should be rm-ed. +verify_fetch_result() { + ACTUAL_ERR=$1 && + rm -f expect.err.combined && + if [ -f expect.err.super ]; then + cat expect.err.super >>expect.err.combined + fi && + if [ -f expect.err.sub ]; then + cat expect.err.sub >>expect.err.combined + fi && + if [ -f expect.err.deep ]; then + cat expect.err.deep >>expect.err.combined + fi && + test_cmp expect.err.combined $ACTUAL_ERR +} + test_expect_success setup ' mkdir deepsubmodule && (
@@ -77,7 +102,7 @@ test_expect_success "fetch --recurse-submodules recurses into submodules" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "submodule.recurse option triggers recursive fetch" '
@@ -87,7 +112,7 @@ test_expect_success "submodule.recurse option triggers recursive fetch" ' git -c submodule.recurse fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "fetch --recurse-submodules -j2 has the same output behaviour" '
@@ -97,7 +122,7 @@ test_expect_success "fetch --recurse-submodules -j2 has the same output behaviou GIT_TRACE="$TRASH_DIRECTORY/trace.out" git fetch --recurse-submodules -j2 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err && + verify_fetch_result actual.err && grep "2 tasks" trace.out '
@@ -127,7 +152,7 @@ test_expect_success "using fetchRecurseSubmodules=true in .gitmodules recurses i git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--no-recurse-submodules overrides .gitmodules config" '
@@ -158,7 +183,7 @@ test_expect_success "--recurse-submodules overrides fetchRecurseSubmodules setti git config --unset submodule.submodule.fetchRecurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--quiet propagates to submodules" '
@@ -186,7 +211,7 @@ test_expect_success "--dry-run propagates to submodules" ' git fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "Without --dry-run propagates to submodules" '
@@ -195,7 +220,7 @@ test_expect_success "Without --dry-run propagates to submodules" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "recurseSubmodules=true propagates into submodules" '
@@ -206,7 +231,7 @@ test_expect_success "recurseSubmodules=true propagates into submodules" ' git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--recurse-submodules overrides config in submodule" '
@@ -220,7 +245,7 @@ test_expect_success "--recurse-submodules overrides config in submodule" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--no-recurse-submodules overrides config setting" '
@@ -253,14 +278,14 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.sub && - echo " $head1..$head2 super -> origin/super" >>expect.err.sub && - head -3 expect.err >> expect.err.sub && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && - test_cmp expect.err.sub actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -271,14 +296,16 @@ test_expect_success "Recursion doesn't happen when new superproject commits don' git add file && git commit -m "new file" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.file && - echo " $head1..$head2 super -> origin/super" >> expect.err.file && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + rm expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err.file actual.err + verify_fetch_result actual.err ' test_expect_success "Recursion picks up config in submodule" '
@@ -295,9 +322,8 @@ test_expect_success "Recursion picks up config in submodule" ' git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.sub && - echo " $head1..$head2 super -> origin/super" >> expect.err.sub && - cat expect.err >> expect.err.sub && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && ( cd downstream && git fetch >../actual.out 2>../actual.err &&
@@ -306,7 +332,7 @@ test_expect_success "Recursion picks up config in submodule" ' git config --unset fetch.recurseSubmodules ) ) && - test_cmp expect.err.sub actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -331,15 +357,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >> expect.err.2 && - cat expect.err.sub >> expect.err.2 && - tail -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && - test_cmp expect.err.2 actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -375,11 +399,8 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - tail -3 expect.err > expect.err.deepsub && - echo "From $pwd/." > expect.err && - echo " $head1..$head2 super -> origin/super" >>expect.err && - cat expect.err.sub >> expect.err && - cat expect.err.deepsub >> expect.err && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -395,7 +416,7 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess ) ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" '
@@ -405,14 +426,16 @@ test_expect_success "'--recurse-submodules=on-demand' stops when no new submodul git add file && git commit -m "new file" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.file && - echo " $head1..$head2 super -> origin/super" >> expect.err.file && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + rm expect.err.deep && ( cd downstream && git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err.file actual.err + verify_fetch_result actual.err ' test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config" '
@@ -426,9 +449,9 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git config fetch.recurseSubmodules on-demand &&
@@ -440,7 +463,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config git config --unset fetch.recurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err + verify_fetch_result actual.err ' test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules" '
@@ -454,9 +477,9 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git config submodule.submodule.fetchRecurseSubmodules on-demand &&
@@ -468,7 +491,7 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override git config --unset submodule.submodule.fetchRecurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err + verify_fetch_result actual.err ' test_expect_success "don't fetch submodule when newly recorded commits are already present" '
@@ -480,14 +503,17 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea git add submodule && git commit -m "submodule rewound" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err && - echo " $head1..$head2 super -> origin/super" >> expect.err && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + # This file does not exist, but rm -f for readability + rm -f expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err && + verify_fetch_result actual.err && ( cd submodule && git checkout -q sub
@@ -505,9 +531,9 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git rm .gitmodules && git commit -m "new submodule without .gitmodules" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." >expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >>expect.err.2 && + echo "From $pwd/." >expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && rm .gitmodules &&
@@ -523,7 +549,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git reset --hard ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err && + verify_fetch_result actual.err && git checkout HEAD^ -- .gitmodules && git add .gitmodules && git commit -m "new submodule restored .gitmodules"
Signature · 1 line
2.33.GIT
Re: [PATCH v3 01/10] t5526: introduce test helper to assert on fetches
Quoted 1 line · Glen Choo
+verify_fetch_result() {
Style.
verify_fetch_result () {t/t5526-fetch-submodules.sh add_upstream_commit() { · line 56
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ ACTUAL_ERR=$1 && + rm -f expect.err.combined && + if [ -f expect.err.super ]; then
Style.
if test -f expect.err.super
then
PATCH v3 2/10 t5526: stop asserting on stderr literally
In reply to Glen ChooIn the previous commit message, we noted that not all of the "git fetch" stderr is relevant to the tests. Most of the test setup lines are dedicated to these details of the stderr:
1. which repos (super/sub/deep) are involved in the fetch
2. the head of the remote-tracking branch before the fetch (i.e. $head1)
3. the head of the remote-tracking branch after the fetch (i.e. $head2)
1. and 3. are relevant because they tell us that the expected commit is fetched by the expected repo, but 2. is completely irrelevant.
Stop asserting on $head1 by replacing it with a dummy value in the actual and expected output. Do this by introducing test helpers (check_*()) that make it easier to construct the expected output, and use sed to munge the actual output.
---
Per Ævar's suggestion [1], I reverted the test suite changes that replaced "test_cmp" with "grep", and opted to munge out irrelevant details with "sed". Opinion on "grep" vs "test_cmp" seems a bit split, and there are good arguments for either, but for these tests, I think test_cmp works better for a few reasons:
- The motivation for removing test_cmp in the old patch 1 [2], i.e. we _want_ to ignore changes in the test output, turned out to be absolutely incorrect. We actually care a lot about the changes in test output because it tells us where we are reading the submodule info from.
In the v1-2 test scheme, verify_fetch_result() would ignore those changes, so I added extra extra grep-s specifically to check where the submodule is read from. ([3] comments on one of these). I could have generalized verify_fetch_result() to handle those extra grep-s, but since the original motivation for using grep is gone, test_cmp seemed like a viable alternative for an intuitive test scheme.
- verify_fetch_result() is easier to reason about because we now assert on the output almost verbatim (besides munging), instead of mixing greps and negative greps to get a similar result. This should be helpful for someone updating the tests later.
- When a test can't use verify_fetch_result() (e.g. it involves other submodules, patch 9 adds some of these), I found it a lot easier to write the test using test_cmp instead of grep.
- test_cmp tests are sensitive to unmeaningful changes, but this behavior helps us catch unwanted regressions and (in these tests at least) it is relatively easy to change test_cmp tests to do the right thing.
[1] https://lore.kernel.org/git/[email protected]
[2] https://lore.kernel.org/git/[email protected]
[3] https://lore.kernel.org/git/[email protected]
t/t5526-fetch-submodules.sh | 117 +++++++++++++++++------------------- 1 file changed, 56 insertions(+), 61 deletions(-)
t/t5526-fetch-submodules.sh +56 -61
@@ -13,6 +13,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +check_sub() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.sub + Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$NEW_HEAD sub -> origin/sub + EOF +} + +check_deep() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.deep + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$NEW_HEAD deep -> origin/deep + EOF +} + +check_super() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.super + From $pwd/. + OLD_HEAD..$NEW_HEAD super -> origin/super + EOF +} + # For each submodule in the test setup, this creates a commit and writes # a file that contains the expected err if that new commit were fetched. # These output files get concatenated in the right order by
@@ -20,27 +46,21 @@ pwd=$(pwd) add_upstream_commit() { ( cd submodule && - head1=$(git rev-parse --short HEAD) && echo new >> subfile && test_tick && git add subfile && git commit -m new subfile && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + new_head=$(git rev-parse --short HEAD) && + check_sub $new_head ) && ( cd deepsubmodule && - head1=$(git rev-parse --short HEAD) && echo new >> deepsubfile && test_tick && git add deepsubfile && git commit -m new deepsubfile && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep - echo "From $pwd/deepsubmodule" >> ../expect.err.deep && - echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep + new_head=$(git rev-parse --short HEAD) && + check_deep $new_head ) }
@@ -62,7 +82,8 @@ verify_fetch_result() { if [ -f expect.err.deep ]; then cat expect.err.deep >>expect.err.combined fi && - test_cmp expect.err.combined $ACTUAL_ERR + sed -E 's/[0-9a-f]+\.\./OLD_HEAD\.\./' $ACTUAL_ERR >actual.err.cmp && + test_cmp expect.err.combined actual.err.cmp } test_expect_success setup '
@@ -274,12 +295,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in ' test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.deep && ( cd downstream &&
@@ -291,13 +310,11 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" test_expect_success "Recursion doesn't happen when new superproject commits don't change any submodules" ' add_upstream_commit && - head1=$(git rev-parse --short HEAD) && echo a > file && git add file && git commit -m "new file" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.sub && rm expect.err.deep && (
@@ -318,12 +335,10 @@ test_expect_success "Recursion picks up config in submodule" ' ) ) && add_upstream_commit && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && ( cd downstream && git fetch >../actual.out 2>../actual.err &&
@@ -345,20 +360,15 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + new_head=$(git rev-parse --short HEAD) && + check_sub $new_head ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -376,13 +386,10 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo Fetching submodule submodule > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + new_head=$(git rev-parse --short HEAD) && + check_sub $new_head ) && ( cd downstream &&
@@ -395,12 +402,10 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne ' test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -421,13 +426,11 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" ' add_upstream_commit && - head1=$(git rev-parse --short HEAD) && echo a >> file && git add file && git commit -m "new file" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.sub && rm expect.err.deep && (
@@ -445,12 +448,10 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config ) && add_upstream_commit && git config --global fetch.recurseSubmodules false && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.deep && ( cd downstream &&
@@ -473,12 +474,10 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override ) && add_upstream_commit && git config fetch.recurseSubmodules false && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.deep && ( cd downstream &&
@@ -499,12 +498,10 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea cd submodule && git checkout -q HEAD^^ ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "submodule rewound" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.sub && # This file does not exist, but rm -f for readability rm -f expect.err.deep &&
@@ -526,13 +523,11 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git fetch --recurse-submodules ) && add_upstream_commit && - head1=$(git rev-parse --short HEAD) && git add submodule && git rm .gitmodules && git commit -m "new submodule without .gitmodules" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." >expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.deep && ( cd downstream &&
Signature · 1 line
2.33.GIT
Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 18
Quoted 3 lines · Glen Choo
t/t5526-fetch-submodules.sh +3
-0+check_sub() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.sub
Hrm, I didn't know that would work, the usual style is:
cat >file <<...
Instead of:
cat <<.. >file
Maybe better to use that?
Quoted 12 lines
t/t5526-fetch-submodules.sh +12
-0+ Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$NEW_HEAD sub -> origin/sub + EOF +} + +check_deep() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.deep + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$NEW_HEAD deep -> origin/deep
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 40
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ EOF +} + +check_super() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.super + From $pwd/. + OLD_HEAD..$NEW_HEAD super -> origin/super + EOF +}
These look a lot better, but instead of always passing the result of "git rev-parse --short HEAD" can't we just always invoke that in these helpers?
Maybe there are cases where $NEW_HEAD is different, I've just skimmed this series.
t/t5526-fetch-submodules.sh verify_fetch_result() { · line 87
Quoted 8 lines
t/t5526-fetch-submodules.sh +2
-1@@ -62,7 +82,8 @@ verify_fetch_result() { if [ -f expect.err.deep ]; then cat expect.err.deep >>expect.err.combined fi && - test_cmp expect.err.combined $ACTUAL_ERR + sed -E 's/[0-9a-f]+\.\./OLD_HEAD\.\./' $ACTUAL_ERR >actual.err.cmp && + test_cmp expect.err.combined actual.err.cmp }
I think this is unportable per check-non-portable-shell.pl:
/\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)';
Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally
Quoted 8 lines · Ævar Arnfjörð Bjarmason
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 18
Quoted 3 lines · Glen Choo
t/t5526-fetch-submodules.sh +3
-0+check_sub() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.subHrm, I didn't know that would work, the usual style is:
Quoted 7 lines
cat >file <<...Instead of:
cat <<.. >fileMaybe better to use that?
Thanks, I somehow mixed things up when I wrote that.
Quoted 27 lines
Quoted 12 lines
t/t5526-fetch-submodules.sh +12
-0+ Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$NEW_HEAD sub -> origin/sub + EOF +} + +check_deep() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.deep + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$NEW_HEAD deep -> origin/deept/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 40
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ EOF +} + +check_super() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.super + From $pwd/. + OLD_HEAD..$NEW_HEAD super -> origin/super + EOF +}These look a lot better, but instead of always passing the result of "git rev-parse --short HEAD" can't we just always invoke that in these helpers?
Quoted 2 lines
Maybe there are cases where $NEW_HEAD is different, I've just skimmed this series.
I haven't found any other instances where $NEW_HEAD is different, so I suppose we could move it into the helpers. I don't think it benefits readability that much to do so, but if you think it's much better, I'll incorporate it when I reroll this.
Quoted 11 lines
t/t5526-fetch-submodules.sh verify_fetch_result() { · line 87
Quoted 8 lines
t/t5526-fetch-submodules.sh +2
-1@@ -62,7 +82,8 @@ verify_fetch_result() { if [ -f expect.err.deep ]; then cat expect.err.deep >>expect.err.combined fi && - test_cmp expect.err.combined $ACTUAL_ERR + sed -E 's/[0-9a-f]+\.\./OLD_HEAD\.\./' $ACTUAL_ERR >actual.err.cmp && + test_cmp expect.err.combined actual.err.cmp }I think this is unportable per check-non-portable-shell.pl:
Quoted 1 line
/\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)';
Ah thanks, my sed-fu is pretty poor, so I appreciate the tip :)
I used that because I wanted +, but I found what I needed from the sed manpage i.e. that + is equivalent to \{1,\}).
Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally
Quoted 10 lines · Glen Choo
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line · Glen Choo
+ sed -E 's/[0-9a-f]+\.\./OLD_HEAD\.\./' $ACTUAL_ERR >actual.err.cmp &&
I think this is unportable per check-non-portable-shell.pl:
/\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)';I used that because I wanted +, but I found what I needed from the sed manpage i.e. that + is equivalent to \{1,\}).
This isn't necessarily going to be portable either for older sed implementations. Most portable would be:
[0-9a-f][0-9a-f]*
(Whether or not we need to worry about those older sed impmenetations is a different question...)
Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally
In reply to Glen ChooQuoted 23 lines · Glen Choo
t/t5526-fetch-submodules.sh +15
-0@@ -13,6 +13,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +check_sub() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.sub + Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$NEW_HEAD sub -> origin/sub + EOF +} + +check_deep() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.deep + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$NEW_HEAD deep -> origin/deep
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 40
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ EOF +} + +check_super() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.super + From $pwd/. + OLD_HEAD..$NEW_HEAD super -> origin/super + EOF +}
These don't do any checking, but just write what's expected to a file. Could these be called something like write_sub_expected etc.?
Other than that, the patches up to this look fine (besides the comments left by others).
Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally
Quoted 29 lines · Jonathan Tan
Quoted 28 lines · Glen Choo
t/t5526-fetch-submodules.sh +20
-0@@ -13,6 +13,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +check_sub() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.sub + Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$NEW_HEAD sub -> origin/sub + EOF +} + +check_deep() { + NEW_HEAD=$1 && + cat <<-EOF >$pwd/expect.err.deep + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$NEW_HEAD deep -> origin/deep + EOF +} + +check_super() { + NEW_HEAD=$1 &&
Quoted 9 lines
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 40
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+ cat <<-EOF >$pwd/expect.err.super + From $pwd/. + OLD_HEAD..$NEW_HEAD super -> origin/super + EOF +}These don't do any checking, but just write what's expected to a file. Could these be called something like
write_sub_expectedetc.?
Thanks for the suggestion! I was struggling with names.
PATCH v3 3/10 t5526: create superproject commits with test helper
In reply to Glen ChooA few tests in t5526 use this pattern as part of their setup:
1. Create new commits in the upstream submodules (using add_upstream_commit()).
2. In the upstream superprojects, add the new submodule commits from the previous step.
A future commit will add more tests with this pattern, so reduce the verbosity of present and future tests by introducing a test helper that creates superproject commits. Since we now have two helpers that add upstream commits, rename add_upstream_commit() to add_submodule_commits().
--- t/t5526-fetch-submodules.sh | 94 +++++++++++++++++-------------------- 1 file changed, 44 insertions(+), 50 deletions(-)
t/t5526-fetch-submodules.sh +44 -50
@@ -43,7 +43,7 @@ check_super() { # a file that contains the expected err if that new commit were fetched. # These output files get concatenated in the right order by # verify_fetch_result(). -add_upstream_commit() { +add_submodule_commits() { ( cd submodule && echo new >> subfile &&
@@ -64,6 +64,30 @@ add_upstream_commit() { ) } +# For each superproject in the test setup, update its submodule, add the +# submodule and create a new commit with the submodule change. +# +# This requires add_submodule_commits() to be called first, otherwise +# the submodules will not have changed and cannot be "git add"-ed. +add_superproject_commits() { +( + cd submodule && + ( + cd subdir/deepsubmodule && + git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule" + ) && + git add submodule && + git commit -m "new submodule" && + super_head=$(git rev-parse --short HEAD) && + sub_head=$(git -C submodule rev-parse --short HEAD) && + check_super $super_head && + check_sub $sub_head +} + # Verifies that the expected repositories were fetched. This is done by # concatenating the files expect.err.[super|sub|deep] in the correct # order and comparing it to the actual stderr.
@@ -117,7 +141,7 @@ test_expect_success setup ' ' test_expect_success "fetch --recurse-submodules recurses into submodules" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git fetch --recurse-submodules >../actual.out 2>../actual.err
@@ -127,7 +151,7 @@ test_expect_success "fetch --recurse-submodules recurses into submodules" ' ' test_expect_success "submodule.recurse option triggers recursive fetch" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git -c submodule.recurse fetch >../actual.out 2>../actual.err
@@ -137,7 +161,7 @@ test_expect_success "submodule.recurse option triggers recursive fetch" ' ' test_expect_success "fetch --recurse-submodules -j2 has the same output behaviour" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && GIT_TRACE="$TRASH_DIRECTORY/trace.out" git fetch --recurse-submodules -j2 2>../actual.err
@@ -148,7 +172,7 @@ test_expect_success "fetch --recurse-submodules -j2 has the same output behaviou ' test_expect_success "fetch alone only fetches superproject" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -177,7 +201,7 @@ test_expect_success "using fetchRecurseSubmodules=true in .gitmodules recurses i ' test_expect_success "--no-recurse-submodules overrides .gitmodules config" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git fetch --no-recurse-submodules >../actual.out 2>../actual.err
@@ -226,7 +250,7 @@ test_expect_success "--quiet propagates to parallel submodules" ' ' test_expect_success "--dry-run propagates to submodules" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err
@@ -245,7 +269,7 @@ test_expect_success "Without --dry-run propagates to submodules" ' ' test_expect_success "recurseSubmodules=true propagates into submodules" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git config fetch.recurseSubmodules true &&
@@ -256,7 +280,7 @@ test_expect_success "recurseSubmodules=true propagates into submodules" ' ' test_expect_success "--recurse-submodules overrides config in submodule" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && (
@@ -270,7 +294,7 @@ test_expect_success "--recurse-submodules overrides config in submodule" ' ' test_expect_success "--no-recurse-submodules overrides config setting" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git config fetch.recurseSubmodules true &&
@@ -309,7 +333,7 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" ' test_expect_success "Recursion doesn't happen when new superproject commits don't change any submodules" ' - add_upstream_commit && + add_submodule_commits && echo a > file && git add file && git commit -m "new file" &&
@@ -334,7 +358,7 @@ test_expect_success "Recursion picks up config in submodule" ' git config fetch.recurseSubmodules true ) ) && - add_upstream_commit && + add_submodule_commits && git add submodule && git commit -m "new submodule" && new_head=$(git rev-parse --short HEAD) &&
@@ -352,23 +376,8 @@ test_expect_success "Recursion picks up config in submodule" ' ' test_expect_success "Recursion picks up all submodules when necessary" ' - add_upstream_commit && - ( - cd submodule && - ( - cd subdir/deepsubmodule && - git fetch && - git checkout -q FETCH_HEAD - ) && - git add subdir/deepsubmodule && - git commit -m "new deepsubmodule" && - new_head=$(git rev-parse --short HEAD) && - check_sub $new_head - ) && - git add submodule && - git commit -m "new submodule" && - new_head=$(git rev-parse --short HEAD) && - check_super $new_head && + add_submodule_commits && + add_superproject_commits && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -378,19 +387,7 @@ test_expect_success "Recursion picks up all submodules when necessary" ' ' test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)" ' - add_upstream_commit && - ( - cd submodule && - ( - cd subdir/deepsubmodule && - git fetch && - git checkout -q FETCH_HEAD - ) && - git add subdir/deepsubmodule && - git commit -m "new deepsubmodule" && - new_head=$(git rev-parse --short HEAD) && - check_sub $new_head - ) && + add_submodule_commits && ( cd downstream && git config fetch.recurseSubmodules true &&
@@ -402,10 +399,7 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne ' test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" ' - git add submodule && - git commit -m "new submodule" && - new_head=$(git rev-parse --short HEAD) && - check_super $new_head && + add_superproject_commits && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -425,7 +419,7 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess ' test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" ' - add_upstream_commit && + add_submodule_commits && echo a >> file && git add file && git commit -m "new file" &&
@@ -446,7 +440,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config cd downstream && git fetch --recurse-submodules ) && - add_upstream_commit && + add_submodule_commits && git config --global fetch.recurseSubmodules false && git add submodule && git commit -m "new submodule" &&
@@ -472,7 +466,7 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override cd downstream && git fetch --recurse-submodules ) && - add_upstream_commit && + add_submodule_commits && git config fetch.recurseSubmodules false && git add submodule && git commit -m "new submodule" &&
@@ -522,7 +516,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git cd downstream && git fetch --recurse-submodules ) && - add_upstream_commit && + add_submodule_commits && git add submodule && git rm .gitmodules && git commit -m "new submodule without .gitmodules" &&
Signature · 1 line
2.33.GIT
Re: [PATCH v3 03/10] t5526: create superproject commits with test helper
Quoted 6 lines · Glen Choo
t/t5526-fetch-submodules.sh +6
-0+# For each superproject in the test setup, update its submodule, add the +# submodule and create a new commit with the submodule change. +# +# This requires add_submodule_commits() to be called first, otherwise +# the submodules will not have changed and cannot be "git add"-ed. +add_superproject_commits() {
t/t5526-fetch-submodules.sh add_upstream_commit() { · line 82
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+( + cd submodule && + ( + cd subdir/deepsubmodule && + git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule" + ) &&
The indentation looks off. Also, no need for "-q".
Quoted 11 lines
t/t5526-fetch-submodules.sh +0
-7@@ -378,19 +387,7 @@ test_expect_success "Recursion picks up all submodules when necessary" ' ' test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)" ' - add_upstream_commit && - ( - cd submodule && - ( - cd subdir/deepsubmodule && - git fetch && - git checkout -q FETCH_HEAD
t/t5526-fetch-submodules.sh test_expect_success "Recursion picks up all submodules when necessary" ' · line 393
Quoted 10 lines
t/t5526-fetch-submodules.sh +1
-6- ) && - git add subdir/deepsubmodule && - git commit -m "new deepsubmodule" && - new_head=$(git rev-parse --short HEAD) && - check_sub $new_head - ) && + add_submodule_commits && ( cd downstream && git config fetch.recurseSubmodules true &&
Hmm...I'm surprised that this still passes even when code is deleted but the replacement is not added. What's happening here, I guess, is that we're checking that nothing has happened. The test probably should be rewritten but that's outside the scope of this patch set. So for now, just add the add_superproject_commits call.
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne · line 405
Quoted 12 lines
t/t5526-fetch-submodules.sh +1
-4@@ -402,10 +399,7 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne ' test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" ' - git add submodule && - git commit -m "new submodule" && - new_head=$(git rev-parse --short HEAD) && - check_super $new_head && + add_superproject_commits && ( cd downstream && git config fetch.recurseSubmodules false &&
add_superproject_commits without add_submodule_commits?
The rest looks good and overall this looks like a good idea to simplify the test.
Re: [PATCH v3 03/10] t5526: create superproject commits with test helper
Quoted 11 lines · Jonathan Tan
Quoted 10 lines · Glen Choo
t/t5526-fetch-submodules.sh +10
-0+# For each superproject in the test setup, update its submodule, add the +# submodule and create a new commit with the submodule change. +# +# This requires add_submodule_commits() to be called first, otherwise +# the submodules will not have changed and cannot be "git add"-ed. +add_superproject_commits() { +( + cd submodule && + ( + cd subdir/deepsubmodule &&
Quoted 8 lines
t/t5526-fetch-submodules.sh add_upstream_commit() { · line 82
Quoted 6 lines
t/t5526-fetch-submodules.sh +6
-0+ git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule" + ) &&The indentation looks off. Also, no need for "-q".
Ah thanks. I think the "-q" is there to suppress the detached HEAD warning, which is very large.
I'd prefer to keep it unless there are stronger reasons than "it's not needed for correctness".
Quoted 19 lines
Quoted 19 lines
t/t5526-fetch-submodules.sh +1
-13@@ -378,19 +387,7 @@ test_expect_success "Recursion picks up all submodules when necessary" ' ' test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)" ' - add_upstream_commit && - ( - cd submodule && - ( - cd subdir/deepsubmodule && - git fetch && - git checkout -q FETCH_HEAD - ) && - git add subdir/deepsubmodule && - git commit -m "new deepsubmodule" && - new_head=$(git rev-parse --short HEAD) && - check_sub $new_head - ) && + add_submodule_commits && (
Quoted 8 lines
Review test_expect_success "Recursion picks up all submodules when necessary" ' · line 393
Quoted 2 lines
(quoted diff) +0
-0cd downstream && git config fetch.recurseSubmodules true &&Hmm...I'm surprised that this still passes even when code is deleted but the replacement is not added. What's happening here, I guess, is that we're checking that nothing has happened. The test probably should be rewritten but that's outside the scope of this patch set. So for now, just add the
add_superproject_commitscall.
Yeah this test could use some fixing up; I spent a lot of time trying to understand this one. It could use comments at least.
The suggestion to add the add_superproject_commits call defeats the purpose of the test though - which is to assert that "on-demand" recursion only fetches submodule commits if a superproject commit says the submodule has changed, unlike "yes", which unconditionally fetches submodule commits.
So we need to consider these cases:
1. no new upstream commits
2. new upstream submodule commits, but not superproject (call add_submodule_commits() only)
3. new upstream submodule and superproject commits (call add_submodule_commits() and add_superproject_commits())
(1): "on-demand" and "yes" both fetch nothing
(2): "yes" fetches submodule commits but "on-demand" doesn't (3): "on-demand" and "yes" both fetch submodule and superproject commits
So this test can't call add_superproject_commits(), because we would no longer be testing scenario (2) - we'd be 'testing' (3) instead (which doesn't tell us how "on-demand" is different from "yes").
Quoted 14 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne · line 405
Quoted 12 lines
t/t5526-fetch-submodules.sh +1
-4@@ -402,10 +399,7 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne ' test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" ' - git add submodule && - git commit -m "new submodule" && - new_head=$(git rev-parse --short HEAD) && - check_super $new_head && + add_superproject_commits && ( cd downstream && git config fetch.recurseSubmodules false &&
add_superproject_commitswithoutadd_submodule_commits?
This is a silly holdover from before my rewrite.. These lines:
- git add submodule && - git commit -m "new submodule" && - new_head=$(git rev-parse --short HEAD) &&
don't make any sense either until you realize that these commits were set up in the _previous_ test. I should clean this up though, there's no reason for others to have to struggle with this the way I did.
The easiest approach would be to add the add_submodule_commits() call, with a comment explaining that it's technically unnecessary work (because the previous test already calls add_submodule_commits()) but it makes the test easier to read.
Re: [PATCH v3 03/10] t5526: create superproject commits with test helper
Quoted 22 lines · Glen Choo
Quoted 11 lines · Jonathan Tan
Quoted 10 lines · Glen Choo
t/t5526-fetch-submodules.sh +10
-0+# For each superproject in the test setup, update its submodule, add the +# submodule and create a new commit with the submodule change. +# +# This requires add_submodule_commits() to be called first, otherwise +# the submodules will not have changed and cannot be "git add"-ed. +add_superproject_commits() { +( + cd submodule && + ( + cd subdir/deepsubmodule &&Quoted 8 lines
t/t5526-fetch-submodules.sh add_upstream_commit() { · line 82
Quoted 6 lines
t/t5526-fetch-submodules.sh +6
-0+ git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule" + ) &&The indentation looks off. Also, no need for "-q".
Quoted 5 lines
Ah thanks. I think the "-q" is there to suppress the detached HEAD warning, which is very large.
I'd prefer to keep it unless there are stronger reasons than "it's not needed for correctness".
FWIW I was going to comment on the -q, but didn't because you're just moving this around.
I think even for large warnings it's fine to omit -q etc, since that's what --verbose (as in the test-lib.sh argument) is for.
But in this case it's probably better to leave it as-is.
Re: [PATCH v3 03/10] t5526: create superproject commits with test helper
Quoted 27 lines · Ævar Arnfjörð Bjarmason
Quoted 25 lines · Glen Choo
Quoted 11 lines · Jonathan Tan
Quoted 10 lines · Glen Choo
t/t5526-fetch-submodules.sh +10
-0+# For each superproject in the test setup, update its submodule, add the +# submodule and create a new commit with the submodule change. +# +# This requires add_submodule_commits() to be called first, otherwise +# the submodules will not have changed and cannot be "git add"-ed. +add_superproject_commits() { +( + cd submodule && + ( + cd subdir/deepsubmodule &&Quoted 8 lines
t/t5526-fetch-submodules.sh add_upstream_commit() { · line 82
Quoted 6 lines
t/t5526-fetch-submodules.sh +6
-0+ git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule" + ) &&The indentation looks off. Also, no need for "-q".
Ah thanks. I think the "-q" is there to suppress the detached HEAD warning, which is very large.
Quoted 8 lines
Quoted 2 lines
I'd prefer to keep it unless there are stronger reasons than "it's not needed for correctness".
FWIW I was going to comment on the -q, but didn't because you're just moving this around.
I think even for large warnings it's fine to omit -q etc, since that's what --verbose (as in the
test-lib.shargument) is for.
Ah interesting, I didn't consider that. Thanks!
Quoted 1 line
But in this case it's probably better to leave it as-is.
I'm also leaning towards this because I'm just moving things around, but I could be convinced otherwise.
PATCH v3 4/10 submodule: make static functions read submodules from commits
In reply to Glen ChooA future commit will teach "fetch --recurse-submodules" to fetch unpopulated submodules. To prepare for this, teach the necessary static functions how to read submodules from superproject commits using a "treeish_name" argument (instead of always reading from the index and filesystem) but do not actually change where submodules are read from. Submodules will be read from commits when we fetch unpopulated submodules.
--- submodule.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
submodule.c +19 -10
@@ -932,6 +932,7 @@ struct has_commit_data {struct repository *repo;int result;const char *path; +const struct object_id *super_oid; };static int check_has_commit (const struct object_id *oid,void *data)
@@ -940,7 +941,7 @@ static int check_has_commit(const struct object_id *oid, void *data)struct repository subrepo;enum object_type type; -if (repo_submodule_init (&subrepo, cb->repo, cb->path,null_oid ())) { +if (repo_submodule_init (&subrepo, cb->repo, cb->path, cb->super_oid)) { cb->result =0 ; goto cleanup; }
@@ -968,9 +969,15 @@ static int check_has_commit(const struct object_id *oid, void *data)static int submodule_has_commits (struct repository *r,const char *path, +const struct object_id *super_oid,struct oid_array *commits) { -struct has_commit_data has_commit = { r,1 , path }; +struct has_commit_data has_commit = { + .repo = r, + .result =1 , + .path = path, + .super_oid = super_oid + };/* * Perform a cheap, but incorrect check for the existence of 'commits'.
@@ -1017,7 +1024,7 @@ static int submodule_needs_pushing(struct repository *r,const char *path, struct oid_array *commits) { - if (!submodule_has_commits(r, path, commits)) + if (!submodule_has_commits(r, path, null_oid(), commits))/* * NOTE: We do consider it safe to return "no" here. The * correct answer would be "We do not know" instead of
@@ -1277,7 +1284,7 @@ static void calculate_changed_submodule_paths(struct repository *r,if (!path)continue ; -if (submodule_has_commits (r, path, commits)) { +if (submodule_has_commits (r, path,null_oid (), commits)) {oid_array_clear (commits); *name->string ='\0' ; }
@@ -1402,12 +1409,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path) }static struct fetch_task *fetch_task_create (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct fetch_task *task =xmalloc (sizeof (*task));memset (task,0 ,sizeof (*task)); - task->sub =submodule_from_path (r,null_oid (), path); + task->sub =submodule_from_path (r, treeish_name, path);if (!task->sub) {/* * No entry in .gitmodules? Technically not a submodule,
@@ -1439,11 +1447,12 @@ static void fetch_task_release(struct fetch_task *p) }static struct repository *get_submodule_repo_for (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct repository *ret =xmalloc (sizeof (*ret)); -if (repo_submodule_init (ret, r, path,null_oid ())) { +if (repo_submodule_init (ret, r, path, treeish_name)) {free (ret);return NULL ; }
@@ -1464,7 +1473,7 @@ static int get_next_submodule(struct child_process *cp,if (!S_ISGITLINK (ce->ce_mode))continue ; - task =fetch_task_create (spf->r, ce->name); + task =fetch_task_create (spf->r, ce->name,null_oid ());if (!task)continue ;
@@ -1487,7 +1496,7 @@ static int get_next_submodule(struct child_process *cp,continue ; } - task->repo =get_submodule_repo_for (spf->r, task->sub->path); + task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());if (task->repo) {struct strbuf submodule_prefix =STRBUF_INIT ;child_process_init (cp);
Signature · 1 line
2.33.GIT
PATCH v3 5/10 submodule: inline submodule_commits() into caller
In reply to Glen ChooWhen collecting the string_list of changed submodule names, the new submodules commits are stored in the string_list_item.util as an oid_array. A subsequent commit will replace the oid_array with a struct that has more information.
Prepare for this change by inlining submodule_commits() (which inserts into the string_list and initializes the string_list_item.util) into its only caller so that the code is easier to refactor later.
--- submodule.c | 22 ++++++---------------- 1 file changed, 6 insertions(+), 16 deletions(-)
submodule.c +6 -16
@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)return submodule_from_path (the_repository,null_oid (), ce->name); } -static struct oid_array *submodule_commits (struct string_list *submodules, -const char *name) -{ -struct string_list_item *item; - - item =string_list_insert (submodules, name); -if (item->util) -return (struct oid_array *) item->util; - -/* NEEDSWORK: should we have oid_array_init()? */ - item->util =xcalloc (1 ,sizeof (struct oid_array )); -return (struct oid_array *) item->util; -}struct collect_changed_submodules_cb_data {struct repository *repo;
@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,for (i =0 ; i < q->nr; i++) {struct diff_filepair *p = q->queue[i]; -struct oid_array *commits;const struct submodule *submodule;const char *name; +struct string_list_item *item;if (!S_ISGITLINK (p->two->mode))continue ;
@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,if (!name)continue ; - commits =submodule_commits (changed, name); -oid_array_append (commits, &p->two->oid); + item =string_list_insert (changed, name); +if (!item->util) +/* NEEDSWORK: should we have oid_array_init()? */ + item->util =xcalloc (1 ,sizeof (struct oid_array )); +oid_array_append (item->util, &p->two->oid); } }
Signature · 1 line
2.33.GIT
PATCH v3 6/10 submodule: store new submodule commits oid_array in a struct
In reply to Glen ChooThis commit prepares for a future commit that will teach git fetch --recurse-submodules how to fetch submodules that are present in <gitdir>/modules, but are not populated. To do this, we need to store more information about the changed submodule so that we can read the submodule configuration from the superproject commit instead of the filesystem.
Refactor the changed submodules string_list.util to hold a struct instead of an oid_array. This struct only holds the new_commits oid_array for now; more information will be added later.
--- submodule.c | 52 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 18 deletions(-)
submodule.c +34 -18
@@ -806,6 +806,20 @@ static const char *default_name_or_path(const char *path_or_name)return path_or_name; } +/* +* Holds relevant information for a changed submodule. Used as the .util +* member of the changed submodule string_list_item. +*/ +struct changed_submodule_data { +/* The submodule commits that have changed in the rev walk. */ +struct oid_array new_commits; +}; + +static void changed_submodule_data_clear (struct changed_submodule_data *cs_data) +{ +oid_array_clear (&cs_data->new_commits); +} +static void collect_changed_submodules_cb (struct diff_queue_struct *q,struct diff_options *options,void *data)
@@ -820,6 +834,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,const struct submodule *submodule;const char *name;struct string_list_item *item; +struct changed_submodule_data *cs_data;if (!S_ISGITLINK (p->two->mode))continue ;
@@ -848,9 +863,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q, item =string_list_insert (changed, name);if (!item->util) -/* NEEDSWORK: should we have oid_array_init()? */ - item->util =xcalloc (1 ,sizeof (struct oid_array )); -oid_array_append (item->util, &p->two->oid); + item->util =xcalloc (1 ,sizeof (struct changed_submodule_data )); + cs_data = item->util; +oid_array_append (&cs_data->new_commits, &p->two->oid); } }
@@ -897,11 +912,12 @@ static void collect_changed_submodules(struct repository *r,reset_revision_walk (); } -static void free_submodules_oids (struct string_list *submodules) +static void free_submodules_data (struct string_list *submodules) {struct string_list_item *item;for_each_string_list_item (item, submodules) -oid_array_clear ((struct oid_array *) item->util); +changed_submodule_data_clear (item->util); +string_list_clear (submodules,1 ); }
@@ -1074,7 +1090,7 @@ int find_unpushed_submodules(struct repository *r,collect_changed_submodules (r, &submodules, &argv);for_each_string_list_item (name, &submodules) { -struct oid_array *commits = name->util; +struct changed_submodule_data *cs_data = name->util;const struct submodule *submodule;const char *path =NULL ;
@@ -1087,11 +1103,11 @@ int find_unpushed_submodules(struct repository *r,if (!path)continue ; -if (submodule_needs_pushing (r, path, commits)) +if (submodule_needs_pushing (r, path, &cs_data->new_commits))string_list_insert (needs_pushing, path); } -free_submodules_oids (&submodules); +free_submodules_data (&submodules);strvec_clear (&argv);return needs_pushing->nr;
@@ -1261,7 +1277,7 @@ static void calculate_changed_submodule_paths(struct repository *r,collect_changed_submodules (r, changed_submodule_names, &argv);for_each_string_list_item (name, changed_submodule_names) { -struct oid_array *commits = name->util; +struct changed_submodule_data *cs_data = name->util;const struct submodule *submodule;const char *path =NULL ;
@@ -1274,8 +1290,8 @@ static void calculate_changed_submodule_paths(struct repository *r,if (!path)continue ; -if (submodule_has_commits (r, path,null_oid (), commits)) { -oid_array_clear (commits); +if (submodule_has_commits (r, path,null_oid (), &cs_data->new_commits)) { +changed_submodule_data_clear (cs_data); *name->string ='\0' ; } }
@@ -1312,7 +1328,7 @@ int submodule_touches_in_range(struct repository *r,strvec_clear (&args); -free_submodules_oids (&subs); +free_submodules_data (&subs);return ret; }
@@ -1596,7 +1612,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,struct fetch_task *task = task_cb;struct string_list_item *it; -struct oid_array *commits; +struct changed_submodule_data *cs_data;if (!task || !task->sub)BUG ("callback cookie bogus" );
@@ -1624,14 +1640,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,/* Could be an unchanged submodule, not contained in the list */ goto out; - commits = it->util; -oid_array_filter (commits, + cs_data = it->util; +oid_array_filter (&cs_data->new_commits, commit_missing_in_sub, task->repo);/* Are there commits we want, but do not exist? */ -if (commits->nr) { - task->commits = commits; +if (cs_data->new_commits.nr) { + task->commits = &cs_data->new_commits;ALLOC_GROW (spf->oid_fetch_tasks, spf->oid_fetch_tasks_nr +1 , spf->oid_fetch_tasks_alloc);
@@ -1689,7 +1705,7 @@ int fetch_populated_submodules(struct repository *r,strvec_clear (&spf.args); out: -free_submodules_oids (&spf.changed_submodule_names); +free_submodules_data (&spf.changed_submodule_names);return spf.result; }
Signature · 1 line
2.33.GIT
PATCH v3 7/10 submodule: extract get_fetch_task()
In reply to Glen Chooget_next_submodule() configures the parallel submodule fetch by performing two functions:
* iterate the index to find submodules
* configure the child processes to fetch the submodules found in the previous step
Extract the index iterating code into an iterator function, get_fetch_task(), so that get_next_submodule() is agnostic of how to find submodules. This prepares for a subsequent commit will teach the fetch machinery to also iterate through the list of changed submodules (in addition to the index).
--- submodule.c | 61 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 36 insertions(+), 25 deletions(-)
submodule.c +36 -25
@@ -1389,6 +1389,7 @@ struct fetch_task {struct repository *repo;const struct submodule *sub; unsigned free_sub : 1; /* Do we need to free the submodule? */ +const char *default_argv;struct oid_array *commits;/* Ensure these commits are fetched */ };
@@ -1466,14 +1467,11 @@ static struct repository *get_submodule_repo_for(struct repository *r,return ret; } -static int get_next_submodule (struct child_process *cp, -struct strbuf *err,void *data,void **task_cb) +static struct fetch_task * +get_fetch_task (struct submodule_parallel_fetch *spf,struct strbuf *err) { -struct submodule_parallel_fetch *spf = data; -for (; spf->count < spf->r->index->cache_nr; spf->count++) {const struct cache_entry *ce = spf->r->index->cache[spf->count]; -const char *default_argv;struct fetch_task *task;if (!S_ISGITLINK (ce->ce_mode))
@@ -1493,10 +1491,10 @@ static int get_next_submodule(struct child_process *cp, &spf->changed_submodule_names, task->sub->name)) continue; - default_argv ="on-demand" ; + task->default_argv ="on-demand" ;break ;case RECURSE_SUBMODULES_ON : - default_argv ="yes" ; + task->default_argv ="yes" ;break ;case RECURSE_SUBMODULES_OFF :continue ;
@@ -1504,29 +1502,12 @@ static int get_next_submodule(struct child_process *cp, task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());if (task->repo) { -struct strbuf submodule_prefix =STRBUF_INIT ; -child_process_init (cp); - cp->dir = task->repo->gitdir; -prepare_submodule_repo_env_in_gitdir (&cp->env_array); - cp->git_cmd =1 ;if (!spf->quiet)strbuf_addf (err,_ ("Fetching submodule %s%s\n" ), spf->prefix, ce->name); -strvec_init (&cp->args); -strvec_pushv (&cp->args, spf->args.v); -strvec_push (&cp->args, default_argv); -strvec_push (&cp->args,"--submodule-prefix" ); - -strbuf_addf (&submodule_prefix,"%s%s/" , - spf->prefix, - task->sub->path); -strvec_push (&cp->args, submodule_prefix.buf); spf->count++; - *task_cb = task; - -strbuf_release (&submodule_prefix); -return 1 ; +return task; }else {struct strbuf empty_submodule_path =STRBUF_INIT ;
@@ -1550,6 +1531,36 @@ static int get_next_submodule(struct child_process *cp,strbuf_release (&empty_submodule_path); } } +return NULL ; +} + +static int get_next_submodule (struct child_process *cp,struct strbuf *err, +void *data,void **task_cb) +{ +struct submodule_parallel_fetch *spf = data; +struct fetch_task *task =get_fetch_task (spf, err); + +if (task) { +struct strbuf submodule_prefix =STRBUF_INIT ; + +child_process_init (cp); + cp->dir = task->repo->gitdir; +prepare_submodule_repo_env_in_gitdir (&cp->env_array); + cp->git_cmd =1 ; +strvec_init (&cp->args); +strvec_pushv (&cp->args, spf->args.v); +strvec_push (&cp->args, task->default_argv); +strvec_push (&cp->args,"--submodule-prefix" ); + +strbuf_addf (&submodule_prefix,"%s%s/" , + spf->prefix, + task->sub->path); +strvec_push (&cp->args, submodule_prefix.buf); + *task_cb = task; + +strbuf_release (&submodule_prefix); +return 1 ; + }if (spf->oid_fetch_tasks_nr) { struct fetch_task *task =
Signature · 1 line
2.33.GIT
Re: [PATCH v3 07/10] submodule: extract get_fetch_task()
submodule.c struct fetch_task { · line 1395
Quoted 8 lines · Glen Choo
submodule.c +1
-0@@ -1389,6 +1389,7 @@ struct fetch_task {struct repository *repo;const struct submodule *sub; unsigned free_sub : 1; /* Do we need to free the submodule? */ +const char *default_argv;struct oid_array *commits;/* Ensure these commits are fetched */ };
I preferred the other way of passing default_argv in parallel, because it is only used for the interaction in between get_fetch_task() and get_next_submodule(), but I don't feel too strongly about this. In any case, up to and including this patch looks good.
PATCH v3 8/10 submodule: move logic into fetch_task_create()
In reply to Glen Chooget_fetch_task() gets a fetch task by iterating the index; a future commit will introduce a similar function, get_fetch_task_from_changed(), that gets a fetch task from the list of changed submodules. Both functions are similar in that they need to:
* create a fetch task
* initialize the submodule repo for the fetch task
* determine the default recursion mode
Move all of this logic into fetch_task_create() so that it is no longer split between fetch_task_create() and get_fetch_task(). This will make it easier to share code with get_fetch_task_from_changed().
---
I think this patch could be squashed into the previous one, let me know if this is a good idea.
submodule.c | 99 ++++++++++++++++++++++++++++------------------------- 1 file changed, 52 insertions(+), 47 deletions(-)
submodule.c +52 -47
@@ -1415,32 +1415,6 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)return (const struct submodule *) ret; } -static struct fetch_task *fetch_task_create (struct repository *r, -const char *path, -const struct object_id *treeish_name) -{ -struct fetch_task *task =xmalloc (sizeof (*task)); -memset (task,0 ,sizeof (*task)); - - task->sub =submodule_from_path (r, treeish_name, path); -if (!task->sub) { -/* -* No entry in .gitmodules? Technically not a submodule, -* but historically we supported repositories that happen to be -* in-place where a gitlink is. Keep supporting them. -*/ - task->sub =get_non_gitmodules_submodule (path); -if (!task->sub) { -free (task); -return NULL ; - } - - task->free_sub =1 ; - } - -return task; -} -static void fetch_task_release (struct fetch_task *p) {if (p->free_sub)
@@ -1467,6 +1441,57 @@ static struct repository *get_submodule_repo_for(struct repository *r,return ret; } +static struct fetch_task *fetch_task_create (struct submodule_parallel_fetch *spf, +const char *path, +const struct object_id *treeish_name) +{ +struct fetch_task *task =xmalloc (sizeof (*task)); +memset (task,0 ,sizeof (*task)); + + task->sub =submodule_from_path (spf->r, treeish_name, path); + +if (!task->sub) { +/* +* No entry in .gitmodules? Technically not a submodule, +* but historically we supported repositories that happen to be +* in-place where a gitlink is. Keep supporting them. +*/ + task->sub =get_non_gitmodules_submodule (path); +if (!task->sub) + goto cleanup; + + task->free_sub =1 ; + } + +switch (get_fetch_recurse_config (task->sub, spf)) + { +default : +case RECURSE_SUBMODULES_DEFAULT : +case RECURSE_SUBMODULES_ON_DEMAND : +if (!task->sub || + !string_list_lookup ( + &spf->changed_submodule_names, + task->sub->name)) + goto cleanup; + task->default_argv ="on-demand" ; +break ; +case RECURSE_SUBMODULES_ON : + task->default_argv ="yes" ; +break ; +case RECURSE_SUBMODULES_OFF : + goto cleanup; + } + + task->repo =get_submodule_repo_for (spf->r, path, treeish_name); + +return task; + + cleanup: +fetch_task_release (task); +free (task); +return NULL ; +} +static struct fetch_task *get_fetch_task (struct submodule_parallel_fetch *spf,struct strbuf *err) {
@@ -1477,30 +1502,10 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)if (!S_ISGITLINK (ce->ce_mode))continue ; - task =fetch_task_create (spf->r, ce->name,null_oid ()); + task =fetch_task_create (spf, ce->name,null_oid ());if (!task)continue ; -switch (get_fetch_recurse_config (task->sub, spf)) - { -default : -case RECURSE_SUBMODULES_DEFAULT : -case RECURSE_SUBMODULES_ON_DEMAND : -if (!task->sub || - !string_list_lookup ( - &spf->changed_submodule_names, - task->sub->name)) -continue ; - task->default_argv ="on-demand" ; -break ; -case RECURSE_SUBMODULES_ON : - task->default_argv ="yes" ; -break ; -case RECURSE_SUBMODULES_OFF : -continue ; - } - - task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());if (task->repo) {if (!spf->quiet) strbuf_addf(err, _("Fetching submodule %s%s\n"),
Signature · 1 line
2.33.GIT
Re: [PATCH v3 08/10] submodule: move logic into fetch_task_create()
Quoted 2 lines · Glen Choo
I think this patch could be squashed into the previous one, let me know if this is a good idea.
For what it's worth, as a reviewer, I appreciated this patch being its own. It made it easier to review.
It's unfortunate that the created diff has fetch_task_create() deleted and readded, but showing it with
--anchored=" memset(task, 0, sizeof(*task));"
does make it easier to see. (The space before memset is a tab.)
submodule.c static struct repository *get_submodule_repo_for(struct repository *r, · line 1493
Quoted 5 lines
+ cleanup:
submodule.c +4
-0+fetch_task_release (task); +free (task); +return NULL ; +}
No space between the left margin and "cleanup".
Otherwise, up to and including this patch looks good.
PATCH v3 9/10 fetch: fetch unpopulated, changed submodules
In reply to Glen Choo"git fetch --recurse-submodules" only considers populated submodules (i.e. submodules that can be found by iterating the index), which makes "git fetch" behave differently based on which commit is checked out. As a result, even if the user has initialized all submodules correctly, they may not fetch the necessary submodule commits, and commands like "git checkout --recurse-submodules" might fail.
Teach "git fetch" to fetch cloned, changed submodules regardless of whether they are populated. This is in addition to the current behavior of fetching populated submodules (which is always attempted regardless of what was fetched in the superproject, or even if nothing was fetched in the superproject).
A submodule may be encountered multiple times (via the list of populated submodules or via the list of changed submodules). When this happens, "git fetch" only reads the 'populated copy' and ignores the 'changed copy'. Amend the verify_fetch_result() test helper so that we can assert on which 'copy' is being read.
---
In the process of writing the new tests [1], I noticed some failures of the form:
# rm the submodule's working tree directory. $ git rm submodule [...]
# Do a fetch that requires running a child process from the submodule. $ git fetch --recurse-submodules same-name-1 [...]
# Fatal error tells us that we cannot chdir to the deleted working
tree.
fatal: cannot chdir to '../../../submodule': No such file or directory
This happens because submodules set/unset a value for core.worktree when they are checked out/"un-checked out" (see submodule_move_head() and connect_work_tree_and_git_dir()), but "git rm" doesn't know that core.worktree should be updated.
I've worked around this by passing "--work-tree=." to the child process [2], but this feels like a hack, especially because this bug should affect all child processes in a "git rm"-ed submodule (this probably includes the "git branch" processes in gc/branch-recurse-submodules, but I haven't confirmed it yet). Some more comprehensive solutions that could be future work are:
- Teach "git [add|rm]" to unset core.worktree (the reverse operation, "git restore", should already do the correct thing). This won't detect submodules removed with "rm -r" though.
- Teach submodule child processes to ignore stale core.worktree values.
- Do more things in-core instead of using child processes (avoiding the failing chdir() call).
I'm not sure what future work we should pursue, or even if the "--work-tree=." workaround is even good, so I'd appreciate feedback here.
[1] There is a similar, preexisting test that also removes the submodules. However, that test isn't affected because it invokes "git checkout" after doing "git rm".
[2] Since the submodule has a git dir but no working tree, I also tried working around the bug by passing "--bare". However, this doesn't work because work tree settings override "bare-ness" settings, as described by t/t1510-repo-setup.sh.
Documentation/fetch-options.txt | 26 ++-- Documentation/git-fetch.txt | 10 +- builtin/fetch.c | 14 +- submodule.c | 125 +++++++++++++-- submodule.h | 12 +- t/t5526-fetch-submodules.sh | 260 +++++++++++++++++++++++++++++++- 6 files changed, 404 insertions(+), 43 deletions(-)
Documentation/fetch-options.txt +17 -9
@@ -185,15 +185,23 @@ endif::git-pull[] ifndef::git-pull[] --recurse-submodules[=yes|on-demand|no]:: This option controls if and under what conditions new commits of - populated submodules should be fetched too. It can be used as a - boolean option to completely disable recursion when set to 'no' or to - unconditionally recurse into all populated submodules when set to - 'yes', which is the default when this option is used without any - value. Use 'on-demand' to only recurse into a populated submodule - when the superproject retrieves a commit that updates the submodule's - reference to a commit that isn't already in the local submodule - clone. By default, 'on-demand' is used, unless - `fetch.recurseSubmodules` is set (see linkgit:git-config[1]). + submodules should be fetched too. When recursing through submodules, + `git fetch` always attempts to fetch "changed" submodules, that is, a + submodule that has commits that are referenced by a newly fetched + superproject commit but are missing in the local submodule clone. A + changed submodule can be fetched as long as it is present locally e.g. + in `$GIT_DIR/modules/` (see linkgit:gitsubmodules[7]); if the upstream + adds a new submodule, that submodule cannot be fetched until it is + cloned e.g. by `git submodule update`. ++ +When set to 'on-demand', only changed submodules are fetched. When set +to 'yes', all populated submodules are fetched and submodules that are +both unpopulated and changed are fetched. When set to 'no', submodules +are never fetched. ++ +When unspecified, this uses the value of `fetch.recurseSubmodules` if it +is set (see linkgit:git-config[1]), defaulting to 'on-demand' if unset. +When this option is used without any value, it defaults to 'yes'. endif::git-pull[] -j::
Documentation/git-fetch.txt +4 -6
@@ -287,12 +287,10 @@ include::transfer-data-leaks.txt[] BUGS ---- -Using --recurse-submodules can only fetch new commits in already checked -out submodules right now. When e.g. upstream added a new submodule in the -just fetched commits of the superproject the submodule itself cannot be -fetched, making it impossible to check out that submodule later without -having to do a fetch again. This is expected to be fixed in a future Git -version. +Using --recurse-submodules can only fetch new commits in submodules that are +present locally e.g. in `$GIT_DIR/modules/`. If the upstream adds a new +submodule, that submodule cannot be fetched until it is cloned e.g. by `git +submodule update`. This is expected to be fixed in a future Git version. SEE ALSO --------
builtin/fetch.c +7 -7
@@ -2122,13 +2122,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) max_children = fetch_parallel_config;add_options_to_argv (&options); - result =fetch_populated_submodules (the_repository, - &options, - submodule_prefix, - recurse_submodules, - recurse_submodules_default, - verbosity <0 , - max_children); + result =fetch_submodules (the_repository, + &options, + submodule_prefix, + recurse_submodules, + recurse_submodules_default, + verbosity <0 , + max_children);strvec_clear (&options); }
submodule.c +112 -13
232 lines not on this page · raw message
submodule.h +6 -6
@@ -88,12 +88,12 @@ int should_update_submodules(void);*/ const struct submodule *submodule_from_ce (const struct cache_entry *ce);void check_for_new_submodule_commits (struct object_id *oid); -int fetch_populated_submodules (struct repository *r, -const struct strvec *options, -const char *prefix, -int command_line_option, -int default_option, -int quiet,int max_parallel_jobs); +int fetch_submodules (struct repository *r, +const struct strvec *options, +const char *prefix, +int command_line_option, +int default_option, +int quiet,int max_parallel_jobs);unsigned is_submodule_modified (const char *path,int ignore_untracked);int submodule_uses_gitfile (const char *path);
t/t5526-fetch-submodules.sh +258 -2
284 lines not on this page · raw message
Signature · 1 line
2.33.GIT
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
submodule.c struct changed_submodule_data { · line 831
Quoted 9 lines · Glen Choo
submodule.c +2
-0+char *path;/* The submodule commits that have changed in the rev walk. */ struct oid_array new_commits; };@@ -818,6 +828,7 @@ struct changed_submodule_data {static void changed_submodule_data_clear (struct changed_submodule_data *cs_data) {oid_array_clear (&cs_data->new_commits); +free (cs_data->path);
OK.
submodule.c static void collect_changed_submodules_cb(struct diff_queue_struct *q, · line 880
Quoted 9 lines
submodule.c +2
-0} static void collect_changed_submodules_cb(struct diff_queue_struct *q,@@ -865,6 +876,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,if (!item->util) item->util =xcalloc (1 ,sizeof (struct changed_submodule_data )); cs_data = item->util; + cs_data->super_oid = commit_oid; + cs_data->path =xstrdup (p->two->path);
Iffy. If item->util were populated already, wouldn't cs_data already have its .path member pointing at an allocated piece of memory? Can we safely free it before assigning a new value, or does somebody else still have a copy of .path and we cannot free it?
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
Quoted 16 lines · Junio C Hamano
submodule.c struct changed_submodule_data { · line 831
Quoted 9 lines · Glen Choo
submodule.c +2
-0+char *path;/* The submodule commits that have changed in the rev walk. */ struct oid_array new_commits; };@@ -818,6 +828,7 @@ struct changed_submodule_data {static void changed_submodule_data_clear (struct changed_submodule_data *cs_data) {oid_array_clear (&cs_data->new_commits); +free (cs_data->path);OK.
Quoted 2 lines
(quoted diff) +0
-0}
Quoted 12 lines
submodule.c static void collect_changed_submodules_cb(struct diff_queue_struct *q, · line 880
Quoted 7 lines
submodule.c +2
-0static void collect_changed_submodules_cb(struct diff_queue_struct *q,@@ -865,6 +876,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,if (!item->util) item->util =xcalloc (1 ,sizeof (struct changed_submodule_data )); cs_data = item->util; + cs_data->super_oid = commit_oid; + cs_data->path =xstrdup (p->two->path);Iffy. If
item->utilwere populated already, wouldn'tcs_dataalready have its .path member pointing at an allocated piece of memory? Can we safely free it before assigning a new value, or does somebody else still have a copy of .path and we cannot free it?
Great catch! This is a silly mistake, it looks like this because I copied the pattern that we used to _append_ new commit oids, but .super_oid and .path aren't appended, they're replaced.
But we don't even need to replace .super_oid and .path, we can use the first values we encounter and ignore subsequent ones.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
In reply to Glen Choosubmodule.c get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err) · line 1597
Quoted 13 lines · Glen Choo
submodule.c +13
-0+static struct fetch_task * +get_fetch_task_from_changed (struct submodule_parallel_fetch *spf, +struct strbuf *err) +{ +for (; spf->changed_count < spf->changed_submodule_names.nr; + spf->changed_count++) { +struct string_list_item item = + spf->changed_submodule_names.items[spf->changed_count]; +struct changed_submodule_data *cs_data = item.util; +struct fetch_task *task; + +if (!is_tree_submodule_active (spf->r, cs_data->super_oid,cs_data->path)) +continue ;
Where does this function come from? I seem to be getting compilation errors.
t/t5526-fetch-submodules.sh pwd=$(pwd) · line 16
Quoted 7 lines
t/t5526-fetch-submodules.sh +0
-0@@ -15,8 +15,9 @@ pwd=$(pwd) check_sub() {
Style.
check_sub () {t/t5526-fetch-submodules.sh pwd=$(pwd) · line 19
Quoted 3 lines
t/t5526-fetch-submodules.sh +1
-0NEW_HEAD=$1 && + SUPER_HEAD=$2 && cat <<-EOF >$pwd/expect.err.sub
Style.
cat <<-EOF >"$pwd/expect.err.sub"
You may swap the order of redirection (having <<here-doc at the end of the line might look more familiar to some people). Try to do as majority of surrounding code does.
Make sure you quote the redirection target filename if it involves variable interpolation (see Documentation/CodingGuidelines, look for "Redirection").
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 554
Quoted 1 line
t/t5526-fetch-submodules.sh +1
-0+ cat <<-EOF > expect.err.combined &&
Style.
cat <<-EOF >expect.err.combined &&
No SP between redirection operator and its target.
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 568
Quoted 1 line
t/t5526-fetch-submodules.sh +1
-0+ sed -E "s/[0-9a-f]+\.\./OLD_HEAD\.\./" actual.err >actual.err.cmp &&
No ERE in sed. "[0-9a-f][0-9a-f]*" instead of "[0-9a-f]+" should be sufficient, I think.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
Quoted 18 lines · Junio C Hamano
submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err) · line 1597
Quoted 13 lines · Glen Choo
submodule.c +13
-0+static struct fetch_task * +get_fetch_task_from_changed (struct submodule_parallel_fetch *spf, +struct strbuf *err) +{ +for (; spf->changed_count < spf->changed_submodule_names.nr; + spf->changed_count++) { +struct string_list_item item = + spf->changed_submodule_names.items[spf->changed_count]; +struct changed_submodule_data *cs_data = item.util; +struct fetch_task *task; + +if (!is_tree_submodule_active (spf->r, cs_data->super_oid,cs_data->path)) +continue ;Where does this function come from? I seem to be getting compilation errors.
Sorry, this was introduced in gc/branch-recurse-submodules, but I neglected to mention that I used that as the base in v1.
Quoted 32 lines
t/t5526-fetch-submodules.sh pwd=$(pwd) · line 16
Quoted 7 lines
t/t5526-fetch-submodules.sh +0
-0@@ -15,8 +15,9 @@ pwd=$(pwd) check_sub() {Style.
check_sub () {t/t5526-fetch-submodules.sh pwd=$(pwd) · line 19
Quoted 3 lines
t/t5526-fetch-submodules.sh +1
-0NEW_HEAD=$1 && + SUPER_HEAD=$2 && cat <<-EOF >$pwd/expect.err.subStyle.
cat <<-EOF >"$pwd/expect.err.sub"You may swap the order of redirection (having <<here-doc at the end of the line might look more familiar to some people). Try to do as majority of surrounding code does.
Make sure you quote the redirection target filename if it involves variable interpolation (see
Documentation/CodingGuidelines, look for "Redirection").t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 554
Quoted 1 line
t/t5526-fetch-submodules.sh +1
-0+ cat <<-EOF > expect.err.combined &&Style.
Quoted 8 lines
cat <<-EOF >expect.err.combined &&No SP between redirection operator and its target.
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 568
Quoted 1 line
t/t5526-fetch-submodules.sh +1
-0+ sed -E "s/[0-9a-f]+\.\./OLD_HEAD\.\./" actual.err >actual.err.cmp &&No ERE in sed. "[0-9a-f][0-9a-f]*" instead of "[0-9a-f]+" should be sufficient, I think.
Thanks :)
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
In reply to Glen ChooQuoted 20 lines · Glen Choo
In the process of writing the new tests [1], I noticed some failures of the form:
# rm the submodule's working tree directory. $ git rm submodule [...]# Do a fetch that requires running a child process from the submodule. $ git fetch --recurse-submodules same-name-1 [...]# Fatal error tells us that we cannot chdir to the deleted working
tree.
fatal: cannot chdir to '../../../submodule': No such file or directoryThis happens because submodules set/unset a value for core.worktree when they are checked out/"un-checked out" (see
submodule_move_head()andconnect_work_tree_and_git_dir()), but "git rm" doesn't know that core.worktree should be updated.
Quoted 5 lines
I've worked around this by passing "--work-tree=." to the child process [2], but this feels like a hack, especially because this bug should affect all child processes in a "git rm"-ed submodule (this probably includes the "git branch" processes in gc/branch-recurse-submodules, but I haven't confirmed it yet).
Ah...that's a tricky bug. Thanks for finding it.
Quoted 6 lines
Some more comprehensive solutions that
could be future work are:- Teach "git [add|rm]" to unset core.worktree (the reverse operation, "git restore", should already do the correct thing). This won't detect submodules removed with "rm -r" though.
This might work with the caveat you mentioned.
Quoted 1 line
- Teach submodule child processes to ignore stale core.worktree values.
This might work, coupled with Emily Shaffer's work on teaching submodules to know that they're submodules (so we know when a stale core.worktree can be safely ignored).
[1] https://lore.kernel.org/git/[email protected]/
Quoted 2 lines
- Do more things in-core instead of using child processes (avoiding the failing
chdir()call).
This might not work if the invocation needs to check the worktree (for example, as far as I know, we won't delete a branch if it's currently checked out in a worktree).
Quoted 3 lines
I'm not sure what future work we should pursue, or even if the "--work-tree=." workaround is even good, so I'd appreciate feedback here.
I can't think of better solutions than what you listed, unfortunately. I also can't think of a better workaround, but at least it's narrowly
scoped: we know that we're running on a submodule and that the operation is not affected by a worktree (for example, we're fetching, but we know we're not fetching with a refspec that updates a currently checked out branch). Let's see what others have to say.
submodule.c static void collect_changed_submodules_cb(struct diff_queue_struct *q, · line 880
Quoted 6 lines
submodule.c +2
-0@@ -865,6 +876,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,if (!item->util) item->util =xcalloc (1 ,sizeof (struct changed_submodule_data )); cs_data = item->util; + cs_data->super_oid = commit_oid; + cs_data->path =xstrdup (p->two->path);
Junio mentioned the possibility of cs_data->path already being non-NULL [1].
Quoted 11 lines
submodule.c +7
-0@@ -1253,14 +1266,33 @@ void check_for_new_submodule_commits(struct object_id *oid)oid_array_append (&ref_tips_after_fetch, oid); } +/* +* Returns 1 if there is at least one submodule gitdir in +* $GIT_DIR/modules and 0 otherwise. This follows +* submodule_name_to_gitdir(), which looks for submodules in +* $GIT_DIR/modules, not $GIT_COMMON_DIR. +* +* A submodule can be moved to $GIT_DIR/modules manually by running "git
submodule.c void check_for_new_submodule_commits(struct object_id *oid) · line 1285
Quoted 10 lines
submodule.c +10
-0+* submodule absorbgitdirs", or it may be initialized there by "git +* submodule update". +*/ +static int repo_has_absorbed_submodules (struct repository *r) +{ +struct strbuf buf =STRBUF_INIT ; + +strbuf_repo_git_path (&buf, r,"modules/" ); +return file_exists (buf.buf) && !is_empty_dir (buf.buf); +}
buf needs to be released?
submodule.c int submodule_touches_in_range(struct repository *r, · line 1372
Quoted 10 lines
submodule.c +2
-1@@ -1333,7 +1365,8 @@ int submodule_touches_in_range(struct repository *r, }struct submodule_parallel_fetch { -int count; +int index_count; +int changed_count;struct strvec args;struct repository *r;const char *prefix;
If we're sticking with these names, probably worth a comment. E.g. "index_count" is the number of submodules in <name of field that this is an index of> that we have processed, and likewise for "changed_count".
submodule.c struct submodule_parallel_fetch { · line 1382
Quoted 8 lines
submodule.c +1
-0@@ -1343,6 +1376,7 @@ struct submodule_parallel_fetch {int result;struct string_list changed_submodule_names; +struct string_list seen_submodule_names;/* Pending fetches by OIDs */ struct fetch_task **oid_fetch_tasks;
Also here - changed is the list that we generated from walking the fetched superproject commits, and seen is the list of submodules we've processed in <name of function>.
Quoted 1 line
(quoted diff) +0
-0@@ -1539,11 +1582,64 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)
[snip]
submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err) · line 1633
Quoted 12 lines
submodule.c +12
-0+/* +* NEEDSWORK: A submodule unpopulated by "git rm" will +* have core.worktree set, but the actual core.worktree +* directory won't exist, causing the child process to +* fail. Forcibly set --work-tree until we get smarter +* handling for core.worktree in unpopulated submodules. +*/ +strvec_push (&task->git_args,"--work-tree=." ); +return task; + } +return NULL ; +}
If we end up sticking to this workaround (which sounds reasonable to me), the comment here probably should contain a lot of what was written under the "---" in the commit message.
Quoted 1 line
+test_expect_success "'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules" '
[snip]
Quoted 17 lines
t/t5526-fetch-submodules.sh +17
-0+# In downstream, init "submodule2", but do not check it out while +# fetching. This lets us assert that unpopulated submodules can be +# fetched. +test_expect_success 'setup downstream branch with other submodule' ' + mkdir submodule2 && + ( + cd submodule2 && + git init && + echo sub2content >sub2file && + git add sub2file && + git commit -a -m new && + git branch -M sub2 + ) && + git checkout -b super-sub2-only && + git submodule add "$pwd/submodule2" submodule2 && + git commit -m "add sub2" && + git checkout super &&
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 523
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ ( + cd downstream && + git fetch --recurse-submodules origin && + git checkout super-sub2-only && + # Explicitly run "git submodule update" because sub2 is new + # and has not been cloned. + git submodule update --init && + git checkout --recurse-submodules super + ) +'
Hmm...what is the difference between this and the original case in which the index has no submodules? Both assert that unpopulated submodules (submodules that cannot be found by iterating the index, as described in your commit message) can be fetched.
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ # Use test_cmp manually because verify_fetch_result does not + # consider submodule2. All the repos should be fetched, but only + # submodule2 should be read from a commit + cat <<-EOF > expect.err.combined && + From $pwd/. + OLD_HEAD..$super_head super -> origin/super + OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only + Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$sub_head sub -> origin/sub
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 570
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$deep_head deep -> origin/deep + Fetching submodule submodule2 at commit $super_sub2_only_head + From $pwd/submodule2 + OLD_HEAD..$sub2_head sub2 -> origin/sub2 + EOF + sed -E "s/[0-9a-f]+\.\./OLD_HEAD\.\./" actual.err >actual.err.cmp && + test_cmp expect.err.combined actual.err.cmp +'
Could verify_fetch_result be modified to consider the new submodule instead?
Quoted 11 lines
t/t5526-fetch-submodules.sh +11
-0+test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' ' + test_when_finished "git -C same-name-downstream checkout master" && + ( + cd same-name-1 && + test_commit -C submodule --no-tag b1 && + git add submodule && + git commit -m "super-b1" + ) && + ( + cd same-name-2 && + test_commit -C submodule --no-tag b2 &&
t/t5526-fetch-submodules.sh test_expect_success 'recursive fetch after deinit a submodule' ' · line 1072
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ git add submodule && + git commit -m "super-b2" + ) && + ( + cd same-name-downstream && + # even though the .gitmodules is correct, we cannot + # fetch from same-name-2 + git checkout same-name-2/master && + git fetch --recurse-submodules same-name-1 && + test_must_fail git fetch --recurse-submodules same-name-2
What's the error message printed to the user here? (Just from reading the code, I would have expected this to succeed, with the submodule fetch being from same-name-1's submodule since we're fetching submodules by name, but apparently that is not the case.)
t/t5526-fetch-submodules.sh test_expect_success 'recursive fetch after deinit a submodule' ' · line 1116
Quoted 7 lines
t/t5526-fetch-submodules.sh +7
-0+ ( + cd same-name-downstream/.git/modules/submodule && + # The submodule has core.worktree pointing to the "git + # rm"-ed directory, overwrite the invalid value. + git --work-tree=. cat-file -e $head1 && + test_must_fail git --work-tree=. cat-file -e $head2 + )
Regarding the worktree workaround, also say "see comment in get_fetch_task() for more information" or something like that.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
Quoted 11 lines · Jonathan Tan
Quoted 3 lines · Glen Choo
I'm not sure what future work we should pursue, or even if the "--work-tree=." workaround is even good, so I'd appreciate feedback here.
I can't think of better solutions than what you listed, unfortunately. I also can't think of a better workaround, but at least it's narrowly
scoped: we know that we're running on a submodule and that the operation is not affected by a worktree (for example, we're fetching, but we know we're not fetching with a refspec that updates a currently checked out branch). Let's see what others have to say.
Thanks for sharing your thoughts :)
Quoted 15 lines
Quoted 15 lines
submodule.c +11
-0@@ -1253,14 +1266,33 @@ void check_for_new_submodule_commits(struct object_id *oid)oid_array_append (&ref_tips_after_fetch, oid); } +/* +* Returns 1 if there is at least one submodule gitdir in +* $GIT_DIR/modules and 0 otherwise. This follows +* submodule_name_to_gitdir(), which looks for submodules in +* $GIT_DIR/modules, not $GIT_COMMON_DIR. +* +* A submodule can be moved to $GIT_DIR/modules manually by running "git +* submodule absorbgitdirs", or it may be initialized there by "git +* submodule update". +*/ +static int repo_has_absorbed_submodules (struct repository *r)
Quoted 8 lines
submodule.c void check_for_new_submodule_commits(struct object_id *oid) · line 1285
Quoted 6 lines
submodule.c +6
-0+{ +struct strbuf buf =STRBUF_INIT ; + +strbuf_repo_git_path (&buf, r,"modules/" ); +return file_exists (buf.buf) && !is_empty_dir (buf.buf); +}buf needs to be released?
Ah, thanks.
Quoted 21 lines
submodule.c int submodule_touches_in_range(struct repository *r, · line 1372
Quoted 10 lines
submodule.c +2
-1@@ -1333,7 +1365,8 @@ int submodule_touches_in_range(struct repository *r, }struct submodule_parallel_fetch { -int count; +int index_count; +int changed_count;struct strvec args;struct repository *r;const char *prefix;If we're sticking with these names, probably worth a comment. E.g. "
index_count" is the number of submodules in <name of field that this is an index of> that we have processed, and likewise for "changed_count".Quoted 6 lines
submodule.c +1
-0@@ -1343,6 +1376,7 @@ struct submodule_parallel_fetch {int result;struct string_list changed_submodule_names; +struct string_list seen_submodule_names;
Quoted 6 lines
submodule.c struct submodule_parallel_fetch { · line 1382
Quoted 2 lines
submodule.c +0
-0/* Pending fetches by OIDs */ struct fetch_task **oid_fetch_tasks;Also here - changed is the list that we generated from walking the fetched superproject commits, and seen is the list of submodules we've processed in <name of function>.
Makes sense.
Quoted 4 lines
Quoted 1 line
(quoted diff) +0
-0@@ -1539,11 +1582,64 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)[snip]
Quoted 16 lines
submodule.c get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err) · line 1633
Quoted 12 lines
submodule.c +12
-0+/* +* NEEDSWORK: A submodule unpopulated by "git rm" will +* have core.worktree set, but the actual core.worktree +* directory won't exist, causing the child process to +* fail. Forcibly set --work-tree until we get smarter +* handling for core.worktree in unpopulated submodules. +*/ +strvec_push (&task->git_args,"--work-tree=." ); +return task; + } +return NULL ; +}If we end up sticking to this workaround (which sounds reasonable to me), the comment here probably should contain a lot of what was written under the "---" in the commit message.
I assume this includes documenting solutions (like your NEEDSWORK comment on submodule_name_to_gitdir()) and why core.worktree isn't usually a problem (because checkout et al do the right thing).
Quoted 28 lines
Quoted 1 line
+test_expect_success "'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules" '
[snip]
Quoted 24 lines
t/t5526-fetch-submodules.sh +24
-0+# In downstream, init "submodule2", but do not check it out while +# fetching. This lets us assert that unpopulated submodules can be +# fetched. +test_expect_success 'setup downstream branch with other submodule' ' + mkdir submodule2 && + ( + cd submodule2 && + git init && + echo sub2content >sub2file && + git add sub2file && + git commit -a -m new && + git branch -M sub2 + ) && + git checkout -b super-sub2-only && + git submodule add "$pwd/submodule2" submodule2 && + git commit -m "add sub2" && + git checkout super && + ( + cd downstream && + git fetch --recurse-submodules origin && + git checkout super-sub2-only && + # Explicitly run "git submodule update" because sub2 is new + # and has not been cloned. + git submodule update --init &&
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 434
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ git checkout --recurse-submodules super + ) +'Hmm...what is the difference between this and the original case in which the index has no submodules? Both assert that unpopulated submodules (submodules that cannot be found by iterating the index, as described in your commit message) can be fetched.
In the previous test, the index has no submodules (it's completely empty in fact, so we don't iterate the index at all), but in this test, it does. This lets us check that there aren't any buggy interactions when both changed and index submodules are present.
I think such mistakes are pretty easy to introduce on accident - I made one pre-v1 where I reused .count between both iterators (instead of having .index_count and .changed_count). It passed the previous test because we didn't care about the index, but it obviously wouldn't pass this one.
Quoted 15 lines
Quoted 15 lines
t/t5526-fetch-submodules.sh +15
-0+ # Use test_cmp manually because verify_fetch_result does not + # consider submodule2. All the repos should be fetched, but only + # submodule2 should be read from a commit + cat <<-EOF > expect.err.combined && + From $pwd/. + OLD_HEAD..$super_head super -> origin/super + OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only + Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$sub_head sub -> origin/sub + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$deep_head deep -> origin/deep + Fetching submodule submodule2 at commit $super_sub2_only_head + From $pwd/submodule2
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 570
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+ OLD_HEAD..$sub2_head sub2 -> origin/sub2 + EOF + sed -E "s/[0-9a-f]+\.\./OLD_HEAD\.\./" actual.err >actual.err.cmp && + test_cmp expect.err.combined actual.err.cmp +'Could
verify_fetch_resultbe modified to consider the new submodule instead?
Since submodule2 is on the end of the file, I could modify verify_fetch_result() to concatenate extra text on the end. But if it weren't in the middle, we'd need to insert arbitrary text in the middle of the file.
I can't think of a good way to do this without compromising test readability, so I'll just do concatenation for now.
Quoted 18 lines
Quoted 18 lines
t/t5526-fetch-submodules.sh +18
-0+test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' ' + test_when_finished "git -C same-name-downstream checkout master" && + ( + cd same-name-1 && + test_commit -C submodule --no-tag b1 && + git add submodule && + git commit -m "super-b1" + ) && + ( + cd same-name-2 && + test_commit -C submodule --no-tag b2 && + git add submodule && + git commit -m "super-b2" + ) && + ( + cd same-name-downstream && + # even though the .gitmodules is correct, we cannot + # fetch from same-name-2
Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success 'recursive fetch after deinit a submodule' ' · line 1072
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ git checkout same-name-2/master && + git fetch --recurse-submodules same-name-1 && + test_must_fail git fetch --recurse-submodules same-name-2What's the error message printed to the user here? (Just from reading the code, I would have expected this to succeed, with the submodule fetch being from same-name-1's submodule since we're fetching submodules by name, but apparently that is not the case.)
Yeah, I think this might trip up some readers. The message is:
From ../same-name-2
b7ebb59..944b5ac master -> same-name-2/master
Fetching submodule submodule
fatal: git upload-pack: not our ref 7ff6874077503acb9d0a52e280aaed9748276319
fatal: remote error: upload-pack: not our ref 7ff6874077503acb9d0a52e280aaed9748276319 Errors during submodule fetch:
submodule
Which, I believe, comes from how we fetch commits by oid:
static int get_next_submodule(struct child_process *cp, struct strbuf *err,
void *data, void **task_cb)
[...]
oid_array_for_each_unique(task->commits,
append_oid_to_argv, &cp->args);When the following is true:
- the submodule is found in the index
- we are fetching submodules unconditionally (--recurse-submodules=yes")
- no superproject commit "changes" the submodule
task->commits is empty, and we just fetch the from the submodule's remote by name. But as long as any superproject commit "changes" the submodule, we try to fetch by oid, which, as this test demonstrates, may fail.
Quoted 11 lines
t/t5526-fetch-submodules.sh test_expect_success 'recursive fetch after deinit a submodule' ' · line 1116
Quoted 8 lines
t/t5526-fetch-submodules.sh +7
-0+ ( + cd same-name-downstream/.git/modules/submodule && + # The submodule has core.worktree pointing to the "git + # rm"-ed directory, overwrite the invalid value. + git --work-tree=. cat-file -e $head1 && + test_must_fail git --work-tree=. cat-file -e $head2 + )Regarding the worktree workaround, also say "see comment in
get_fetch_task()for more information" or something like that.
Makes sense.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
Quoted 24 lines · Glen Choo
Quoted 15 lines
Quoted 15 lines
t/t5526-fetch-submodules.sh +15
-0+ # Use test_cmp manually because verify_fetch_result does not + # consider submodule2. All the repos should be fetched, but only + # submodule2 should be read from a commit + cat <<-EOF > expect.err.combined && + From $pwd/. + OLD_HEAD..$super_head super -> origin/super + OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only + Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$sub_head sub -> origin/sub + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$deep_head deep -> origin/deep + Fetching submodule submodule2 at commit $super_sub2_only_head + From $pwd/submodule2Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 570
Quoted 5 lines
t/t5526-fetch-submodules.sh +5
-0+ OLD_HEAD..$sub2_head sub2 -> origin/sub2 + EOF + sed -E "s/[0-9a-f]+\.\./OLD_HEAD\.\./" actual.err >actual.err.cmp && + test_cmp expect.err.combined actual.err.cmp +'Could
verify_fetch_resultbe modified to consider the new submodule instead?
Quoted 7 lines
Since submodule2 is on the end of the file, I could modify
verify_fetch_result()to concatenate extra text on the end. But if it weren't in the middle, we'd need to insert arbitrary text in the middle of the file.I can't think of a good way to do this without compromising test readability, so I'll just do concatenation for now.
Looking at it, I think you can do it by adding a section that verifies the "Fetching submodule submodule2" part if the file is present (so, no change in behavior in the rest of the tests since they don't write this file) and also modifying check_super to allow specification of the sub2 part (or making a new function for this).
Quoted 25 lines
Quoted 4 lines
What's the error message printed to the user here? (Just from reading the code, I would have expected this to succeed, with the submodule fetch being from same-name-1's submodule since we're fetching submodules by name, but apparently that is not the case.)
Yeah, I think this might trip up some readers. The message is:
From ../same-name-2
b7ebb59..944b5ac master -> same-name-2/masterFetching submodule submodule
fatal: git upload-pack: not our ref7ff6874077503acb9d0a52e280aaed9748276319
fatal: remote error: upload-pack: not our ref7ff6874077503acb9d0a52e280aaed9748276319Errors during submodule fetch:
submoduleWhich, I believe, comes from how we fetch commits by oid:
static int get_next_submodule(struct
child_process*cp, struct strbuf *err,
void *data, void **task_cb)
[...]oid_array_for_each_unique(task->commits, append_oid_to_argv, &cp->args);When the following is true:
Quoted 8 lines
- the submodule is found in the index
- we are fetching submodules unconditionally (--recurse-submodules=yes")
- no superproject commit "changes" the submodule
task->commitsis empty, and we just fetch the from the submodule's remote by name. But as long as any superproject commit "changes" the submodule, we try to fetch by oid, which, as this test demonstrates, may fail.
Ah, so we try to fetch an OID from a submodule given by a fetched commit, which is different from the submodule the client already has locally. This might be a sign that we need to store more information about the submodule so that we can print a clearer message. I haven't looked into this deeply, but this might be possible by putting more information in the util of changed_submodule_names, and when we have already seen that submodule, to add more information to the util instead of skipping it.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
Quoted 35 lines · Jonathan Tan
Quoted 25 lines · Glen Choo
Quoted 4 lines
What's the error message printed to the user here? (Just from reading the code, I would have expected this to succeed, with the submodule fetch being from same-name-1's submodule since we're fetching submodules by name, but apparently that is not the case.)
Yeah, I think this might trip up some readers. The message is:
From ../same-name-2
b7ebb59..944b5ac master -> same-name-2/masterFetching submodule submodule
fatal: git upload-pack: not our ref7ff6874077503acb9d0a52e280aaed9748276319
fatal: remote error: upload-pack: not our ref7ff6874077503acb9d0a52e280aaed9748276319Errors during submodule fetch:
submoduleWhich, I believe, comes from how we fetch commits by oid:
static int get_next_submodule(struct
child_process*cp, struct strbuf *err,
void *data, void **task_cb)
[...]oid_array_for_each_unique(task->commits, append_oid_to_argv, &cp->args);When the following is true:
Quoted 8 lines
- the submodule is found in the index
- we are fetching submodules unconditionally (--recurse-submodules=yes")
- no superproject commit "changes" the submodule
task->commitsis empty, and we just fetch the from the submodule's remote by name. But as long as any superproject commit "changes" the submodule, we try to fetch by oid, which, as this test demonstrates, may fail.
Quoted 8 lines
Ah, so we try to fetch an OID from a submodule given by a fetched commit, which is different from the submodule the client already has locally. This might be a sign that we need to store more information about the submodule so that we can print a clearer message. I haven't looked into this deeply, but this might be possible by putting more information in the util of
changed_submodule_names, and when we have already seen that submodule, to add more information to the util instead of skipping it.
Storing the submodule URL might achieve this purpose, but if the URL doesn't match, I think we'd want to skip the commit instead of trying to fetch a commit from an unrelated URL. I don't know if this is a good idea though yet - I haven't looked deeply into what Git uses the URL for and whether users might want to change the URL even though the submodule is the 'same' (e.g. pointing the URL to another remote instead of having two completely different repos with the same submodule name).
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
In reply to Glen ChooQuoted 38 lines · Glen Choo
Quoted 24 lines
Quoted 24 lines
t/t5526-fetch-submodules.sh +24
-0+# In downstream, init "submodule2", but do not check it out while +# fetching. This lets us assert that unpopulated submodules can be +# fetched. +test_expect_success 'setup downstream branch with other submodule' ' + mkdir submodule2 && + ( + cd submodule2 && + git init && + echo sub2content >sub2file && + git add sub2file && + git commit -a -m new && + git branch -M sub2 + ) && + git checkout -b super-sub2-only && + git submodule add "$pwd/submodule2" submodule2 && + git commit -m "add sub2" && + git checkout super && + ( + cd downstream && + git fetch --recurse-submodules origin && + git checkout super-sub2-only && + # Explicitly run "git submodule update" because sub2 is new + # and has not been cloned. + git submodule update --init &&Quoted 8 lines
t/t5526-fetch-submodules.sh test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess · line 434
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+ git checkout --recurse-submodules super + ) +'Hmm...what is the difference between this and the original case in which the index has no submodules? Both assert that unpopulated submodules (submodules that cannot be found by iterating the index, as described in your commit message) can be fetched.
In the previous test, the index has no submodules (it's completely empty in fact, so we don't iterate the index at all), but in this test, it does. This lets us check that there aren't any buggy interactions when both changed and index submodules are present.
Quoted 5 lines
I think such mistakes are pretty easy to introduce on accident - I made one pre-v1 where I reused .count between both iterators (instead of having .index_count and .changed_count). It passed the previous test because we didn't care about the index, but it obviously wouldn't pass this one.
In that case, describe this difference (one has no submodules in index, one has other submodules in index) and maybe position this so that both test cases (the no-submodule-in-index one and the other-submodule-in-index one) are next to each other.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
In reply to Glen ChooA few tests added by this patch have been failing on one specific job (linux-gcc ubuntu-latest) at GitHub CI.
https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968
https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520
Side note: you may need to be logged in to GitHub to view them. These two use different versions of CI to show the test traces; in the latter you may have to click on right-facing rectangle on the line with label "5520" to see the breakage.
I think there is some baked-in assumption in the failing test what the name of the initial branch by default is, which may be the reason why this particular job fails while others don't.
Can you take a look at it?
Thanks.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
Hi,
Quoted 11 lines · Junio C Hamano
A few tests added by this patch have been failing on one specific job (linux-gcc ubuntu-latest) at GitHub CI.
https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968
https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520Side note: you may need to be logged in to GitHub to view them. These two use different versions of CI to show the test traces; in the latter you may have to click on right-facing rectangle on the line with label "5520" to see the breakage.
Quoted 5 lines
I think there is some baked-in assumption in the failing test what the name of the initial branch by default is, which may be the reason why this particular job fails while others don't.
Can you take a look at it?
The log says this:
-- snip --
[...]
+ git commit -m super-b2
[main 00b85ba] super-b2
Author: A U Thor <[email protected]>
1 file changed, 1 insertion(+), 1 deletion(-)
+ cd same-name-downstream
+ git checkout same-name-2/master
error: pathspec 'same-name-2/master' did not match any file(s) known to git
error: last command exited with $?=1
+ git -C same-name-downstream checkout master
error: pathspec 'master' did not match any file(s) known to git
+ eval_ret=1
+ :
not ok 49 - fetch --recurse-submodules updates name-conflicted, populated submodule
#
# test_when_finished "git -C same-name-downstream checkout master" &&
# (
# cd same-name-1 &&
# test_commit -C submodule --no-tag b1 &&
# git add submodule &&
# git commit -m "super-b1"
# ) &&
# (
# cd same-name-2 &&
# test_commit -C submodule --no-tag b2 &&
# git add submodule &&
# git commit -m "super-b2"
# ) &&
# (
# cd same-name-downstream &&
# # even though the .gitmodules is correct, we
# cannot
# # fetch from same-name-2
# git checkout same-name-2/master &&
# git fetch --recurse-submodules same-name-1 &&
# test_must_fail git fetch --recurse-submodules
# same-name-2
# ) &&
# super_head1=$(git -C same-name-1 rev-parse HEAD) &&
# git -C same-name-downstream cat-file -e $super_head1 &&
#
# super_head2=$(git -C same-name-2 rev-parse HEAD) &&
# git -C same-name-downstream cat-file -e $super_head2 &&
#
# sub_head1=$(git -C same-name-1/submodule rev-parse HEAD)
# &&
# git -C same-name-downstream/submodule cat-file -e
# $sub_head1 &&
#
# sub_head2=$(git -C same-name-2/submodule rev-parse HEAD)
# &&
# test_must_fail git -C same-name-downstream/submodule
# cat-file -e $sub_head2
#
-- snap --
So yes, there is a lot of mastering going on.
I _think_ the remedy will be to use the -b <branch-name> option of git init in
https://github.com/git/git/blob/82dd0cbc7fcf2985a3dcfbd99caa9f80626b00df/t/t5526-fetch-submodules.sh#L1015
in
https://github.com/git/git/blob/82dd0cbc7fcf2985a3dcfbd99caa9f80626b00df/t/t5526-fetch-submodules.sh#L1024
and in
https://github.com/git/git/blob/82dd0cbc7fcf2985a3dcfbd99caa9f80626b00df/t/t5526-fetch-submodules.sh#L1033
e.g.
git -C submodule init -b main
At least that's how _I_ tried to address similar issues in the test suite in the past.
Ciao,
Dscho
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
Quoted 19 lines · Johannes Schindelin
Hi,
Quoted 15 lines · Junio C Hamano
A few tests added by this patch have been failing on one specific job (linux-gcc ubuntu-latest) at GitHub CI.
https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968
https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520Side note: you may need to be logged in to GitHub to view them. These two use different versions of CI to show the test traces; in the latter you may have to click on right-facing rectangle on the line with label "5520" to see the breakage.
I think there is some baked-in assumption in the failing test what the name of the initial branch by default is, which may be the reason why this particular job fails while others don't.
Quoted 6 lines
Quoted 1 line
Can you take a look at it?
The log says this:
...
At least that's how _I_ tried to address similar issues in the test suite in the past.
Yes, I had a squashable fix/workaround queued since last night.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
Quoted 21 lines · Junio C Hamano
Quoted 19 lines · Johannes Schindelin
Hi,
Quoted 15 lines · Junio C Hamano
A few tests added by this patch have been failing on one specific job (linux-gcc ubuntu-latest) at GitHub CI.
https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968
https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520Side note: you may need to be logged in to GitHub to view them. These two use different versions of CI to show the test traces; in the latter you may have to click on right-facing rectangle on the line with label "5520" to see the breakage.
I think there is some baked-in assumption in the failing test what the name of the initial branch by default is, which may be the reason why this particular job fails while others don't.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Can you take a look at it?
The log says this:
...
At least that's how _I_ tried to address similar issues in the test suite in the past.Yes, I had a squashable fix/workaround queued since last night.
Thanks, both! I especially appreciate the pointers because I couldn't remember how to set the default branch off the top of my head.
Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules
In reply to Junio C HamanoQuoted 11 lines · Junio C Hamano
A few tests added by this patch have been failing on one specific job (linux-gcc ubuntu-latest) at GitHub CI.
https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968
https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520Side note: you may need to be logged in to GitHub to view them. These two use different versions of CI to show the test traces; in the latter you may have to click on right-facing rectangle on the line with label "5520" to see the breakage.
Quoted 7 lines
I think there is some baked-in assumption in the failing test what the name of the initial branch by default is, which may be the reason why this particular job fails while others don't.
Can you take a look at it?
Thanks.
In case you haven't noticed, this is what I have near the tip of the topic to fix it.
t/t5526-fetch-submodules.sh +2 -0
@@ -3,6 +3,8 @@ test_description='Recursive "git fetch" for submodules' +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1 export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB
PATCH v3 10/10 submodule: fix latent check_has_commit() bug
In reply to Glen ChooWhen check_has_commit() is called on a missing submodule, initialization of the struct repository fails, but it attempts to clear the struct anyway (which is a fatal error). This bug is masked by its only caller, submodule_has_commits(), first calling add_submodule_odb(). The latter fails if the submodule does not exist, making submodule_has_commits() exit early and not invoke check_has_commit().
Fix this bug, and because calling add_submodule_odb() is no longer necessary as of 13a2f620b2 (submodule: pass repo to check_has_commit(), 2021-10-08), remove that call too.
This is the last caller of add_submodule_odb(), so remove that function. (Submodule ODBs are still added as alternates via add_submodule_odb_by_path().)
--- submodule.c | 35 ++--------------------------------- submodule.h | 9 ++++----- 2 files changed, 6 insertions(+), 38 deletions(-)
submodule.c +2 -33
@@ -167,26 +167,6 @@ void stage_updated_gitmodules(struct index_state *istate)static struct string_list added_submodule_odb_paths =STRING_LIST_INIT_NODUP ; -/* TODO: remove this function, use repo_submodule_init instead. */ -int add_submodule_odb (const char *path) -{ -struct strbuf objects_directory =STRBUF_INIT ; -int ret =0 ; - - ret =strbuf_git_path_submodule (&objects_directory, path,"objects/" ); -if (ret) - goto done; -if (!is_directory (objects_directory.buf)) { - ret =-1 ; - goto done; - } -string_list_insert (&added_submodule_odb_paths, -strbuf_detach (&objects_directory,NULL )); -done: -strbuf_release (&objects_directory); -return ret; -} -void add_submodule_odb_by_path (const char *path) {string_list_insert (&added_submodule_odb_paths,xstrdup (path));
@@ -962,7 +942,8 @@ static int check_has_commit(const struct object_id *oid, void *data)if (repo_submodule_init (&subrepo, cb->repo, cb->path, cb->super_oid)) { cb->result =0 ; - goto cleanup; +/* subrepo failed to init, so don't clean it up. */ +return 0 ; } type =oid_object_info (&subrepo, oid,NULL );
@@ -998,18 +979,6 @@ static int submodule_has_commits(struct repository *r, .super_oid = super_oid }; -/* -* Perform a cheap, but incorrect check for the existence of 'commits'. -* This is done by adding the submodule's object store to the in-core -* object store, and then querying for each commit's existence. If we -* do not have the commit object anywhere, there is no chance we have -* it in the object store of the correct submodule and have it -* reachable from a ref, so we can fail early without spawning rev-list -* which is expensive. -*/ -if (add_submodule_odb (path)) -return 0 ; -oid_array_for_each_unique (commits, check_has_commit, &has_commit);if (has_commit.result) {
submodule.h +4 -5
@@ -103,12 +103,11 @@ int submodule_uses_gitfile(const char *path);int bad_to_remove_submodule (const char *path,unsigned flags);/* -* Call add_submodule_odb() to add the submodule at the given path to a list. -* When register_all_submodule_odb_as_alternates() is called, the object stores -* of all submodules in that list will be added as alternates in -* the_repository. +* Call add_submodule_odb_by_path() to add the submodule at the given +* path to a list. When register_all_submodule_odb_as_alternates() is +* called, the object stores of all submodules in that list will be +* added as alternates in the_repository. */ -int add_submodule_odb (const char *path);void add_submodule_odb_by_path (const char *path);int register_all_submodule_odb_as_alternates (void );
Signature · 1 line
2.33.GIT
v4 cover fetch --recurse-submodules: fetch unpopulated submodules
In reply to Glen ChooOriginal cover letter: https://lore.kernel.org/git/[email protected]
This round of patches is now based on master - I prepared the previous rounds on top of gc/branch-recurse-submodules, but now that's merged onto master (and the actual branch for this series, gc/recursive-fetch-with-unused-submodules, is based off master anyway).
This round fixes up the comments from the previous round (thanks everyone!), all of which are fairly small.
= Patch organization
- Patches 1-3 are quality-of-life improvements to the test suite that make it easier to write the tests in patch 9.
- Patches 4-6 are preparation for "git fetch" to read .gitmodules from the superproject commit in patch 7.
- Patches 7-8 refactor out the logic of "finding which submodules to fetch" and "fetching the submodules", making it easier to tell "git fetch" to fetch unpopulated submodules.
- Patch 9 teaches "git fetch" to fetch changed, unpopulated submodules in addition to populated submodules.
- Patch 10 is an optional bugfix + cleanup of the "git fetch" code that removes the last caller of the deprecated "add_submodule_odb()".
= Changes
== Since v3
- Numerous style fixes + improved comments.
- Fix sed portability issues.
- Fix failing test due to default branch name assumptions.
- Patch 3: change a test so that it no longer depends on state from the previous test.
- Patch 9: fix memory leak when recording super_oid and path + add explanatory comment.
== Since v2
- Numerous small fixes to the code and commit message (thanks to all who helped spot these :))
- In patch 2, use test_cmp + sed to assert on test output, effectively reverting the "use grep" approach of v1-2 (see patch 2's description).
- New patch 3: introduce a test helper that creates the expected superproject commit (instead of copy-pasting the code over and over).
- I did not get rid of "git fetch" inside the test helper (as Jonathan suggested) though, because that requires a bigger change in the test setup, and I think the test helper makes the test straightforward enough.
- New patch 8: refactor some shared logic out into fetch_task_create(). This reduces code duplication between the get_fetch_task_from_* functions.
- In patch 9, add additional tests for 'submodules with the same name'.
- In patch 9, handle a bug where a submodule that is unpopulated by "git rm" still has "core.worktree" set and cannot be fetched (see patch 9's description).
- Remove the "git fetch --update-shallow" patch (I'll try to send it separately).
== Since v1
- Numerous style fixes suggested by Jonathan (thanks!)
- In patch 3, don't prematurely read submodules from the superproject commit (see:
<[email protected]>).
- In patch 7, stop using "git checkout" and "! grep" in tests.
- In patch 7, stop doing the "find changed submodules" rev walk unconditionally. Instead, continue to check for .gitmodules, but also check for submodules in $GIT_DIR/modules.
- I'm not entirely happy with the helper function name, see "---" for details.
- Move "git fetch --update-shallow" bugfix to patch 8.
- Because the "find changed submodules" rev walk is no longer unconditional, this fix is no longer needed for tests to pass.
- Rename fetch_populated_submodules() to fetch_submodules().
Glen Choo (10):
t5526: introduce test helper to assert on fetches
t5526: stop asserting on stderr literally
t5526: create superproject commits with test helper
submodule: make static functions read submodules from commits
submodule: inline submodule_commits() into caller
submodule: store new submodule commits oid_array in a struct
submodule: extract get_fetch_task()
submodule: move logic into fetch_task_create()
fetch: fetch unpopulated, changed submodules
submodule: fix latent check_has_commit() bug
Documentation/fetch-options.txt | 26 +- Documentation/git-fetch.txt | 10 +- builtin/fetch.c | 14 +- submodule.c | 442 +++++++++++++++++--------- submodule.h | 21 +- t/t5526-fetch-submodules.sh | 539 ++++++++++++++++++++++++-------- 6 files changed, 740 insertions(+), 312 deletions(-)
Range-diff against v3:
1: b6d34b0f5c ! 1: 57cd31afc2 t5526: introduce test helper to assert on fetches
@@ t/t5526-fetch-submodules.sh: add_upstream_commit() {
+verify_fetch_result() {
+ ACTUAL_ERR=$1 &&
+ rm -f expect.err.combined &&
-+ if [ -f expect.err.super ]; then
++ if test -f expect.err.super
++ then
+ cat expect.err.super >>expect.err.combined
+ fi &&
-+ if [ -f expect.err.sub ]; then
++ if test -f expect.err.sub
++ then
+ cat expect.err.sub >>expect.err.combined
+ fi &&
-+ if [ -f expect.err.deep ]; then
++ if test -f expect.err.deep
++ then
+ cat expect.err.deep >>expect.err.combined
+ fi &&
+ test_cmp expect.err.combined $ACTUAL_ERR
2: 0b85fa35c2 ! 2: b70c894cff t5526: stop asserting on stderr literally
@@ t/t5526-fetch-submodules.sh: export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB
pwd=$(pwd)
-+check_sub() {
++check_sub () {
+ NEW_HEAD=$1 &&
-+ cat <<-EOF >$pwd/expect.err.sub
++ cat >$pwd/expect.err.sub <<-EOF
+ Fetching submodule submodule
+ From $pwd/submodule
+ OLD_HEAD..$NEW_HEAD sub -> origin/sub
+ EOF
+}
+
-+check_deep() {
++check_deep () {
+ NEW_HEAD=$1 &&
-+ cat <<-EOF >$pwd/expect.err.deep
++ cat >$pwd/expect.err.deep <<-EOF
+ Fetching submodule submodule/subdir/deepsubmodule
+ From $pwd/deepsubmodule
+ OLD_HEAD..$NEW_HEAD deep -> origin/deep
+ EOF
+}
+
-+check_super() {
++check_super () {
+ NEW_HEAD=$1 &&
-+ cat <<-EOF >$pwd/expect.err.super
++ cat >$pwd/expect.err.super <<-EOF
+ From $pwd/.
+ OLD_HEAD..$NEW_HEAD super -> origin/super
+ EOF
@@ t/t5526-fetch-submodules.sh: pwd=$(pwd)
)
}
+@@ t/t5526-fetch-submodules.sh: add_upstream_commit() {
+ #
+ # If a repo should not be fetched in the test, its corresponding
+ # expect.err file should be rm-ed.
+-verify_fetch_result() {
++verify_fetch_result () {
+ ACTUAL_ERR=$1 &&
+ rm -f expect.err.combined &&
+ if test -f expect.err.super
@@ t/t5526-fetch-submodules.sh: verify_fetch_result() {
- if [ -f expect.err.deep ]; then
+ then
cat expect.err.deep >>expect.err.combined
fi &&
- test_cmp expect.err.combined $ACTUAL_ERR
-+ sed -E 's/[0-9a-f]+\.\./OLD_HEAD\.\./' $ACTUAL_ERR >actual.err.cmp &&
++ sed -e 's/[0-9a-f][0-9a-f]*\.\./OLD_HEAD\.\./' "$ACTUAL_ERR" >actual.err.cmp &&
+ test_cmp expect.err.combined actual.err.cmp
} 3: bb8ef6094a ! 3: 7e2a01164e t5526: create superproject commits with test helper
@@ Commit message
Signed-off-by: Glen Choo <[email protected]>
## t/t5526-fetch-submodules.sh ##
-@@ t/t5526-fetch-submodules.sh: check_super() {
+@@ t/t5526-fetch-submodules.sh: check_super () {
# a file that contains the expected err if that new commit were fetched.
# These output files get concatenated in the right order by
# verify_fetch_result().
-add_upstream_commit() {
-+add_submodule_commits() {
++add_submodule_commits () {
(
cd submodule &&
echo new >> subfile &&
@@ t/t5526-fetch-submodules.sh: add_upstream_commit() {
+#
+# This requires add_submodule_commits() to be called first, otherwise
+# the submodules will not have changed and cannot be "git add"-ed.
-+add_superproject_commits() {
-+(
-+ cd submodule &&
++add_superproject_commits () {
+ (
-+ cd subdir/deepsubmodule &&
-+ git fetch &&
-+ git checkout -q FETCH_HEAD
-+ ) &&
++ cd submodule &&
++ (
++ cd subdir/deepsubmodule &&
++ git fetch &&
++ git checkout -q FETCH_HEAD
++ ) &&
+ git add subdir/deepsubmodule &&
+ git commit -m "new deep submodule"
+ ) &&
4: e83a1713c4 = 4: 88112ee225 submodule: make static functions read submodules from commits
5: e27d402b9a = 5: 007cd97aba submodule: inline submodule_commits() into caller
6: 1c7c8218b8 = 6: f34ea88fe9 submodule: store new submodule commits oid_array in a struct
7: 80cf317722 ! 7: f66ab663c5 submodule: extract get_fetch_task()
@@ submodule.c: struct fetch_task {
struct repository *repo;
const struct submodule *sub;
unsigned free_sub : 1; /* Do we need to free the submodule? */
-+ const char *default_argv;
++ const char *default_argv; /* The default fetch mode. */
struct oid_array *commits; /* Ensure these commits are fetched */
};
8: bf9cfa7054 = 8: 4e3db1bc9d submodule: move logic into fetch_task_create()
9: c7c2ff71b6 ! 9: 9e7b1c1bbe fetch: fetch unpopulated, changed submodules
@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
## submodule.c ##
@@ submodule.c: static const char *default_name_or_path(const char *path_or_name)
- * member of the changed submodule string_list_item.
+
+ /*
+ * Holds relevant information for a changed submodule. Used as the .util
+- * member of the changed submodule string_list_item.
++ * member of the changed submodule name string_list_item.
++ *
++ * (super_oid, path) allows the submodule config to be read from _some_
++ * .gitmodules file. We store this information the first time we find a
++ * superproject commit that points to the submodule, but this is
++ * arbitrary - we can choose any (super_oid, path) that matches the
++ * submodule's name.
*/
struct changed_submodule_data {
+ /*
-+ * The first superproject commit in the rev walk that points to the
-+ * submodule.
++ * The first superproject commit in the rev walk that points to
++ * the submodule.
+ */
+ const struct object_id *super_oid;
+ /*
@@ submodule.c: struct changed_submodule_data {
static void collect_changed_submodules_cb(struct diff_queue_struct *q,
@@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct *q,
- if (!item->util)
+ continue;
+
+ item = string_list_insert(changed, name);
+- if (!item->util)
++ if (item->util)
++ cs_data = item->util;
++ else {
item->util = xcalloc(1, sizeof(struct changed_submodule_data));
- cs_data = item->util;
-+ cs_data->super_oid = commit_oid;
-+ cs_data->path = xstrdup(p->two->path);
+- cs_data = item->util;
++ cs_data = item->util;
++ cs_data->super_oid = commit_oid;
++ cs_data->path = xstrdup(p->two->path);
++ }
oid_array_append(&cs_data->new_commits, &p->two->oid);
}
}
@@ submodule.c: void check_for_new_submodule_commits(struct object_id *oid)
+ */
+static int repo_has_absorbed_submodules(struct repository *r)
+{
++ int ret;
+ struct strbuf buf = STRBUF_INIT;
+
+ strbuf_repo_git_path(&buf, r, "modules/");
-+ return file_exists(buf.buf) && !is_empty_dir(buf.buf);
++ ret = file_exists(buf.buf) && !is_empty_dir(buf.buf);
++ strbuf_release(&buf);
++ return ret;
+}
+
static void calculate_changed_submodule_paths(struct repository *r,
@@ submodule.c: int submodule_touches_in_range(struct repository *r,
struct submodule_parallel_fetch {
- int count;
++ /*
++ * The index of the last index entry processed by
++ * get_fetch_task_from_index().
++ */
+ int index_count;
++ /*
++ * The index of the last string_list entry processed by
++ * get_fetch_task_from_changed().
++ */
+ int changed_count;
struct strvec args;
struct repository *r;
const char *prefix;
@@ submodule.c: struct submodule_parallel_fetch {
+ int quiet;
int result;
++ /*
++ * Names of submodules that have new commits. Generated by
++ * walking the newly fetched superproject commits.
++ */
struct string_list changed_submodule_names;
++ /*
++ * Names of submodules that have already been processed. Lets us
++ * avoid fetching the same submodule more than once.
++ */
+ struct string_list seen_submodule_names;
/* Pending fetches by OIDs */
@@ submodule.c: struct submodule_parallel_fetch {
@@ submodule.c: struct fetch_task {
const struct submodule *sub;
unsigned free_sub : 1; /* Do we need to free the submodule? */
- const char *default_argv;
-+ struct strvec git_args;
+ const char *default_argv; /* The default fetch mode. */
++ struct strvec git_args; /* Args for the child git process. */
struct oid_array *commits; /* Ensure these commits are fetched */
};
@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf
+
+ spf->changed_count++;
+ /*
-+ * NEEDSWORK: A submodule unpopulated by "git rm" will
-+ * have core.worktree set, but the actual core.worktree
-+ * directory won't exist, causing the child process to
-+ * fail. Forcibly set --work-tree until we get smarter
-+ * handling for core.worktree in unpopulated submodules.
++ * NEEDSWORK: Submodules set/unset a value for
++ * core.worktree when they are populated/unpopulated by
++ * "git checkout" (and similar commands, see
++ * submodule_move_head() and
++ * connect_work_tree_and_git_dir()), but if the
++ * submodule is unpopulated in another way (e.g. "git
++ * rm", "rm -r"), core.worktree will still be set even
++ * though the directory doesn't exist, and the child
++ * process will crash while trying to chdir into the
++ * nonexistent directory.
++ *
++ * In this case, we know that the submodule has no
++ * working tree, so we can work around this by
++ * setting "--work-tree=." (--bare does not work because
++ * worktree settings take precedence over bare-ness).
++ * However, this is not necessarily true in other cases,
++ * so a generalized solution is still necessary.
++ *
++ * Possible solutions:
++ * - teach "git [add|rm]" to unset core.worktree and
++ * discourage users from removing submodules without
++ * using a Git command.
++ * - teach submodule child processes to ignore stale
++ * core.worktree values.
+ */
+ strvec_push(&task->git_args, "--work-tree=.");
+ return task;
@@ submodule.h: int should_update_submodules(void);
## t/t5526-fetch-submodules.sh ##
@@ t/t5526-fetch-submodules.sh: pwd=$(pwd)
- check_sub() {
+ check_sub () {
NEW_HEAD=$1 &&
+ SUPER_HEAD=$2 &&
- cat <<-EOF >$pwd/expect.err.sub
+ cat >$pwd/expect.err.sub <<-EOF
- Fetching submodule submodule
+ Fetching submodule submodule${SUPER_HEAD:+ at commit $SUPER_HEAD}
From $pwd/submodule
OLD_HEAD..$NEW_HEAD sub -> origin/sub
EOF
-@@ t/t5526-fetch-submodules.sh: check_sub() {
+@@ t/t5526-fetch-submodules.sh: check_sub () {
- check_deep() {
+ check_deep () {
NEW_HEAD=$1 &&
+ SUB_HEAD=$2 &&
- cat <<-EOF >$pwd/expect.err.deep
+ cat >$pwd/expect.err.deep <<-EOF
- Fetching submodule submodule/subdir/deepsubmodule
+ Fetching submodule submodule/subdir/deepsubmodule${SUB_HEAD:+ at commit $SUB_HEAD}
From $pwd/deepsubmodule
OLD_HEAD..$NEW_HEAD deep -> origin/deep
EOF
+@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne
+ '
+
+ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" '
++ add_submodule_commits &&
+ add_superproject_commits &&
+ (
+ cd downstream &&
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess
verify_fetch_result actual.err
'
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ # Use test_cmp manually because verify_fetch_result does not
+ # consider submodule2. All the repos should be fetched, but only
+ # submodule2 should be read from a commit
-+ cat <<-EOF > expect.err.combined &&
++ cat > expect.err.combined <<-EOF &&
+ From $pwd/.
+ OLD_HEAD..$super_head super -> origin/super
+ OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only
@@ t/t5526-fetch-submodules.sh: test_expect_success "'--recurse-submodules=on-deman
+ From $pwd/submodule2
+ OLD_HEAD..$sub2_head sub2 -> origin/sub2
+ EOF
-+ sed -E "s/[0-9a-f]+\.\./OLD_HEAD\.\./" actual.err >actual.err.cmp &&
++ sed -e "s/[0-9a-f][0-9a-f]*\.\./OLD_HEAD\.\./" actual.err >actual.err.cmp &&
+ test_cmp expect.err.combined actual.err.cmp
+'
+
@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a
+ mkdir same-name-1 &&
+ (
+ cd same-name-1 &&
-+ git init &&
++ git init -b main &&
+ test_commit --no-tag a
+ ) &&
+ git clone same-name-1 same-name-2 &&
@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a
+ (
+ cd same-name-1 &&
+ mkdir submodule &&
-+ git -C submodule init &&
++ git -C submodule init -b main &&
+ test_commit -C submodule --no-tag a1 &&
+ git submodule add "$pwd/same-name-1/submodule" &&
+ git add submodule &&
@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a
+ (
+ cd same-name-2 &&
+ mkdir submodule &&
-+ git -C submodule init &&
++ git -C submodule init -b main &&
+ test_commit -C submodule --no-tag a2 &&
+ git submodule add "$pwd/same-name-2/submodule" &&
+ git add submodule &&
@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a
+'
+
+test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '
-+ test_when_finished "git -C same-name-downstream checkout master" &&
++ test_when_finished "git -C same-name-downstream checkout main" &&
+ (
+ cd same-name-1 &&
+ test_commit -C submodule --no-tag b1 &&
@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a
+ cd same-name-downstream &&
+ # even though the .gitmodules is correct, we cannot
+ # fetch from same-name-2
-+ git checkout same-name-2/master &&
++ git checkout same-name-2/main &&
+ git fetch --recurse-submodules same-name-1 &&
+ test_must_fail git fetch --recurse-submodules same-name-2
+ ) &&
@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a
+ ) &&
+ (
+ cd same-name-downstream &&
-+ git checkout master &&
++ git checkout main &&
+ git rm .gitmodules &&
+ git rm submodule &&
+ git commit -m "no submodules" &&
@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a
+ (
+ cd same-name-downstream/.git/modules/submodule &&
+ # The submodule has core.worktree pointing to the "git
-+ # rm"-ed directory, overwrite the invalid value.
++ # rm"-ed directory, overwrite the invalid value. See
++ # comment in get_fetch_task_from_changed() for more
++ # information.
+ git --work-tree=. cat-file -e $head1 &&
+ test_must_fail git --work-tree=. cat-file -e $head2
+ )10: e1ac74eee4 = 10: 362ce3c7f8 submodule: fix latent check_has_commit() bug
base-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7
Signature · 1 line
2.33.GIT
PATCH v4 1/10 t5526: introduce test helper to assert on fetches
Tests in t/t5526-fetch-submodules.sh are unnecessarily noisy:
* The tests have extra logic in order to reproduce the expected stderr literally, but not all of these details (e.g. the head of the remote-tracking branch before the fetch) are relevant to the test.
* The expect.err file is constructed by the add_upstream_commit() helper as input into test_cmp, but most tests fetch a different combination of repos from expect.err. This results in noisy tests that modify parts of that expect.err to generate the expected output.
To address both of these issues, introduce a verify_fetch_result() helper to t/t5526-fetch-submodules.sh that asserts on the output of "git fetch --recurse-submodules" and handles the ordering of expect.err.
As a result, the tests no longer construct expect.err manually. Tests still consider the old head of the remote-tracking branch ("$head1"), but that will be fixed in a later commit.
--- t/t5526-fetch-submodules.sh | 139 ++++++++++++++++++++++-------------- 1 file changed, 84 insertions(+), 55 deletions(-)
t/t5526-fetch-submodules.sh +84 -55
@@ -10,6 +10,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +# For each submodule in the test setup, this creates a commit and writes +# a file that contains the expected err if that new commit were fetched. +# These output files get concatenated in the right order by +# verify_fetch_result(). add_upstream_commit() { ( cd submodule &&
@@ -19,9 +23,9 @@ add_upstream_commit() { git add subfile && git commit -m new subfile && head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err && - echo "From $pwd/submodule" >> ../expect.err && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err + echo "Fetching submodule submodule" > ../expect.err.sub && + echo "From $pwd/submodule" >> ../expect.err.sub && + echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub ) && ( cd deepsubmodule &&
@@ -31,12 +35,36 @@ add_upstream_commit() { git add deepsubfile && git commit -m new deepsubfile && head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule/subdir/deepsubmodule" >> ../expect.err - echo "From $pwd/deepsubmodule" >> ../expect.err && - echo " $head1..$head2 deep -> origin/deep" >> ../expect.err + echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep + echo "From $pwd/deepsubmodule" >> ../expect.err.deep && + echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep ) } +# Verifies that the expected repositories were fetched. This is done by +# concatenating the files expect.err.[super|sub|deep] in the correct +# order and comparing it to the actual stderr. +# +# If a repo should not be fetched in the test, its corresponding +# expect.err file should be rm-ed. +verify_fetch_result() { + ACTUAL_ERR=$1 && + rm -f expect.err.combined && + if test -f expect.err.super + then + cat expect.err.super >>expect.err.combined + fi && + if test -f expect.err.sub + then + cat expect.err.sub >>expect.err.combined + fi && + if test -f expect.err.deep + then + cat expect.err.deep >>expect.err.combined + fi && + test_cmp expect.err.combined $ACTUAL_ERR +} + test_expect_success setup ' mkdir deepsubmodule && (
@@ -74,7 +102,7 @@ test_expect_success "fetch --recurse-submodules recurses into submodules" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "submodule.recurse option triggers recursive fetch" '
@@ -84,7 +112,7 @@ test_expect_success "submodule.recurse option triggers recursive fetch" ' git -c submodule.recurse fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "fetch --recurse-submodules -j2 has the same output behaviour" '
@@ -94,7 +122,7 @@ test_expect_success "fetch --recurse-submodules -j2 has the same output behaviou GIT_TRACE="$TRASH_DIRECTORY/trace.out" git fetch --recurse-submodules -j2 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err && + verify_fetch_result actual.err && grep "2 tasks" trace.out '
@@ -124,7 +152,7 @@ test_expect_success "using fetchRecurseSubmodules=true in .gitmodules recurses i git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--no-recurse-submodules overrides .gitmodules config" '
@@ -155,7 +183,7 @@ test_expect_success "--recurse-submodules overrides fetchRecurseSubmodules setti git config --unset submodule.submodule.fetchRecurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--quiet propagates to submodules" '
@@ -183,7 +211,7 @@ test_expect_success "--dry-run propagates to submodules" ' git fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "Without --dry-run propagates to submodules" '
@@ -192,7 +220,7 @@ test_expect_success "Without --dry-run propagates to submodules" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "recurseSubmodules=true propagates into submodules" '
@@ -203,7 +231,7 @@ test_expect_success "recurseSubmodules=true propagates into submodules" ' git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--recurse-submodules overrides config in submodule" '
@@ -217,7 +245,7 @@ test_expect_success "--recurse-submodules overrides config in submodule" ' git fetch --recurse-submodules >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "--no-recurse-submodules overrides config setting" '
@@ -250,14 +278,14 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.sub && - echo " $head1..$head2 super -> origin/super" >>expect.err.sub && - head -3 expect.err >> expect.err.sub && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && - test_cmp expect.err.sub actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -268,14 +296,16 @@ test_expect_success "Recursion doesn't happen when new superproject commits don' git add file && git commit -m "new file" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.file && - echo " $head1..$head2 super -> origin/super" >> expect.err.file && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + rm expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err.file actual.err + verify_fetch_result actual.err ' test_expect_success "Recursion picks up config in submodule" '
@@ -292,9 +322,8 @@ test_expect_success "Recursion picks up config in submodule" ' git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.sub && - echo " $head1..$head2 super -> origin/super" >> expect.err.sub && - cat expect.err >> expect.err.sub && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && ( cd downstream && git fetch >../actual.out 2>../actual.err &&
@@ -303,7 +332,7 @@ test_expect_success "Recursion picks up config in submodule" ' git config --unset fetch.recurseSubmodules ) ) && - test_cmp expect.err.sub actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -328,15 +357,13 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >> expect.err.2 && - cat expect.err.sub >> expect.err.2 && - tail -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && - test_cmp expect.err.2 actual.err && + verify_fetch_result actual.err && test_must_be_empty actual.out '
@@ -372,11 +399,8 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - tail -3 expect.err > expect.err.deepsub && - echo "From $pwd/." > expect.err && - echo " $head1..$head2 super -> origin/super" >>expect.err && - cat expect.err.sub >> expect.err && - cat expect.err.deepsub >> expect.err && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -392,7 +416,7 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess ) ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err + verify_fetch_result actual.err ' test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" '
@@ -402,14 +426,16 @@ test_expect_success "'--recurse-submodules=on-demand' stops when no new submodul git add file && git commit -m "new file" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.file && - echo " $head1..$head2 super -> origin/super" >> expect.err.file && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + rm expect.err.deep && ( cd downstream && git fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err.file actual.err + verify_fetch_result actual.err ' test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config" '
@@ -423,9 +449,9 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git config fetch.recurseSubmodules on-demand &&
@@ -437,7 +463,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config git config --unset fetch.recurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err + verify_fetch_result actual.err ' test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules" '
@@ -451,9 +477,9 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override git add submodule && git commit -m "new submodule" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >> expect.err.2 && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && git config submodule.submodule.fetchRecurseSubmodules on-demand &&
@@ -465,7 +491,7 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override git config --unset submodule.submodule.fetchRecurseSubmodules ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err + verify_fetch_result actual.err ' test_expect_success "don't fetch submodule when newly recorded commits are already present" '
@@ -477,14 +503,17 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea git add submodule && git commit -m "submodule rewound" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err && - echo " $head1..$head2 super -> origin/super" >> expect.err && + echo "From $pwd/." > expect.err.super && + echo " $head1..$head2 super -> origin/super" >> expect.err.super && + rm expect.err.sub && + # This file does not exist, but rm -f for readability + rm -f expect.err.deep && ( cd downstream && git fetch >../actual.out 2>../actual.err ) && test_must_be_empty actual.out && - test_cmp expect.err actual.err && + verify_fetch_result actual.err && ( cd submodule && git checkout -q sub
@@ -502,9 +531,9 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git rm .gitmodules && git commit -m "new submodule without .gitmodules" && head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." >expect.err.2 && - echo " $head1..$head2 super -> origin/super" >>expect.err.2 && - head -3 expect.err >>expect.err.2 && + echo "From $pwd/." >expect.err.super && + echo " $head1..$head2 super -> origin/super" >>expect.err.super && + rm expect.err.deep && ( cd downstream && rm .gitmodules &&
@@ -520,7 +549,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git reset --hard ) && test_must_be_empty actual.out && - test_cmp expect.err.2 actual.err && + verify_fetch_result actual.err && git checkout HEAD^ -- .gitmodules && git add .gitmodules && git commit -m "new submodule restored .gitmodules"
Signature · 1 line
2.33.GIT
Re: [PATCH v4 01/10] t5526: introduce test helper to assert on fetches
t/t5526-fetch-submodules.sh add_upstream_commit() { · line 50
Quoted 7 lines · Glen Choo
t/t5526-fetch-submodules.sh +7
-0+# Verifies that the expected repositories were fetched. This is done by +# concatenating the files expect.err.[super|sub|deep] in the correct +# order and comparing it to the actual stderr. +# +# If a repo should not be fetched in the test, its corresponding +# expect.err file should be rm-ed. +verify_fetch_result() {
I think you updated 02/10 "check_sub () {" but this is leftover. If there aren't too many, I can fix them up locally. We'll see.
Re: [PATCH v4 01/10] t5526: introduce test helper to assert on fetches
Quoted 12 lines · Junio C Hamano
t/t5526-fetch-submodules.sh add_upstream_commit() { · line 50
Quoted 7 lines · Glen Choo
t/t5526-fetch-submodules.sh +7
-0+# Verifies that the expected repositories were fetched. This is done by +# concatenating the files expect.err.[super|sub|deep] in the correct +# order and comparing it to the actual stderr. +# +# If a repo should not be fetched in the test, its corresponding +# expect.err file should be rm-ed. +verify_fetch_result() {I think you updated 02/10 "
check_sub() {" but this is leftover. If there aren't too many, I can fix them up locally. We'll see.
Ugh, sorry about that.. I fixed this somewhere, but at some point I got my branches confused and must've missed this one.
Since you left comments on other patches (like [02/10] [email protected]), I'll just fix it on my end.
PATCH v4 2/10 t5526: stop asserting on stderr literally
In reply to Glen ChooIn the previous commit message, we noted that not all of the "git fetch" stderr is relevant to the tests. Most of the test setup lines are dedicated to these details of the stderr:
1. which repos (super/sub/deep) are involved in the fetch
2. the head of the remote-tracking branch before the fetch (i.e. $head1)
3. the head of the remote-tracking branch after the fetch (i.e. $head2)
1. and 3. are relevant because they tell us that the expected commit is fetched by the expected repo, but 2. is completely irrelevant.
Stop asserting on $head1 by replacing it with a dummy value in the actual and expected output. Do this by introducing test helpers (check_*()) that make it easier to construct the expected output, and use sed to munge the actual output.
--- t/t5526-fetch-submodules.sh | 119 +++++++++++++++++------------------- 1 file changed, 57 insertions(+), 62 deletions(-)
t/t5526-fetch-submodules.sh +57 -62
@@ -10,6 +10,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +check_sub () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.sub <<-EOF + Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$NEW_HEAD sub -> origin/sub + EOF +} + +check_deep () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.deep <<-EOF + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$NEW_HEAD deep -> origin/deep + EOF +} + +check_super () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.super <<-EOF + From $pwd/. + OLD_HEAD..$NEW_HEAD super -> origin/super + EOF +} + # For each submodule in the test setup, this creates a commit and writes # a file that contains the expected err if that new commit were fetched. # These output files get concatenated in the right order by
@@ -17,27 +43,21 @@ pwd=$(pwd) add_upstream_commit() { ( cd submodule && - head1=$(git rev-parse --short HEAD) && echo new >> subfile && test_tick && git add subfile && git commit -m new subfile && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + new_head=$(git rev-parse --short HEAD) && + check_sub $new_head ) && ( cd deepsubmodule && - head1=$(git rev-parse --short HEAD) && echo new >> deepsubfile && test_tick && git add deepsubfile && git commit -m new deepsubfile && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule/subdir/deepsubmodule" > ../expect.err.deep - echo "From $pwd/deepsubmodule" >> ../expect.err.deep && - echo " $head1..$head2 deep -> origin/deep" >> ../expect.err.deep + new_head=$(git rev-parse --short HEAD) && + check_deep $new_head ) }
@@ -47,7 +67,7 @@ add_upstream_commit() { # # If a repo should not be fetched in the test, its corresponding # expect.err file should be rm-ed. -verify_fetch_result() { +verify_fetch_result () { ACTUAL_ERR=$1 && rm -f expect.err.combined && if test -f expect.err.super
@@ -62,7 +82,8 @@ verify_fetch_result() { then cat expect.err.deep >>expect.err.combined fi && - test_cmp expect.err.combined $ACTUAL_ERR + sed -e 's/[0-9a-f][0-9a-f]*\.\./OLD_HEAD\.\./' "$ACTUAL_ERR" >actual.err.cmp && + test_cmp expect.err.combined actual.err.cmp } test_expect_success setup '
@@ -274,12 +295,10 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in ' test_expect_success "Recursion stops when no new submodule commits are fetched" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.deep && ( cd downstream &&
@@ -291,13 +310,11 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" test_expect_success "Recursion doesn't happen when new superproject commits don't change any submodules" ' add_upstream_commit && - head1=$(git rev-parse --short HEAD) && echo a > file && git add file && git commit -m "new file" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.sub && rm expect.err.deep && (
@@ -318,12 +335,10 @@ test_expect_success "Recursion picks up config in submodule" ' ) ) && add_upstream_commit && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && ( cd downstream && git fetch >../actual.out 2>../actual.err &&
@@ -345,20 +360,15 @@ test_expect_success "Recursion picks up all submodules when necessary" ' git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo "Fetching submodule submodule" > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + new_head=$(git rev-parse --short HEAD) && + check_sub $new_head ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -376,13 +386,10 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne git fetch && git checkout -q FETCH_HEAD ) && - head1=$(git rev-parse --short HEAD^) && git add subdir/deepsubmodule && git commit -m "new deepsubmodule" && - head2=$(git rev-parse --short HEAD) && - echo Fetching submodule submodule > ../expect.err.sub && - echo "From $pwd/submodule" >> ../expect.err.sub && - echo " $head1..$head2 sub -> origin/sub" >> ../expect.err.sub + new_head=$(git rev-parse --short HEAD) && + check_sub $new_head ) && ( cd downstream &&
@@ -395,12 +402,10 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne ' test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" ' - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -421,13 +426,11 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" ' add_upstream_commit && - head1=$(git rev-parse --short HEAD) && echo a >> file && git add file && git commit -m "new file" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.sub && rm expect.err.deep && (
@@ -445,12 +448,10 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config ) && add_upstream_commit && git config --global fetch.recurseSubmodules false && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.deep && ( cd downstream &&
@@ -473,12 +474,10 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override ) && add_upstream_commit && git config fetch.recurseSubmodules false && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "new submodule" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.deep && ( cd downstream &&
@@ -499,12 +498,10 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea cd submodule && git checkout -q HEAD^^ ) && - head1=$(git rev-parse --short HEAD) && git add submodule && git commit -m "submodule rewound" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." > expect.err.super && - echo " $head1..$head2 super -> origin/super" >> expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.sub && # This file does not exist, but rm -f for readability rm -f expect.err.deep &&
@@ -526,13 +523,11 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git git fetch --recurse-submodules ) && add_upstream_commit && - head1=$(git rev-parse --short HEAD) && git add submodule && git rm .gitmodules && git commit -m "new submodule without .gitmodules" && - head2=$(git rev-parse --short HEAD) && - echo "From $pwd/." >expect.err.super && - echo " $head1..$head2 super -> origin/super" >>expect.err.super && + new_head=$(git rev-parse --short HEAD) && + check_super $new_head && rm expect.err.deep && ( cd downstream &&
Signature · 1 line
2.33.GIT
Re: [PATCH v4 02/10] t5526: stop asserting on stderr literally
Quoted 26 lines · Glen Choo
In the previous commit message, we noted that not all of the "git fetch" stderr is relevant to the tests. Most of the test setup lines are dedicated to these details of the stderr:
1. which repos (super/sub/deep) are involved in the fetch
2. the head of the remote-tracking branch before the fetch (i.e. $head1)
3. the head of the remote-tracking branch after the fetch (i.e. $head2)1. and 3. are relevant because they tell us that the expected commit is fetched by the expected repo, but 2. is completely irrelevant.
Stop asserting on $head1 by replacing it with a dummy value in the actual and expected output. Do this by introducing test helpers (check_*()) that make it easier to construct the expected output, and use sed to munge the actual output.
Signed-off-by Glen Choo <[email protected]>--- t/t5526-fetch-submodules.sh | 119 +++++++++++++++++------------------- 1 file changed, 57 insertions(+), 62 deletions(-)t/t5526-fetch-submodules.sh +0
-0
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 15
Quoted 6 lines
t/t5526-fetch-submodules.sh +3
-0@@ -10,6 +10,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB pwd=$(pwd) +check_sub () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.sub <<-EOF
Style.
cat >"$pwd/expect.err.sub" <<-EOF
Here, $pwd most likely has an $IFS letter in it (because we deliberately use "trash directory.xxxx" as the place to run our tests in) but a redirection target does not go through $IFS word splitting, so such a quoting is not technically necessary, but some versions of bash are known to throw a warning if we don't, and an extra quoting does not hurt.
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 24
Quoted 3 lines
t/t5526-fetch-submodules.sh +3
-0+check_deep () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.deep <<-EOF
Likewise.
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 33
Quoted 9 lines
t/t5526-fetch-submodules.sh +9
-0+ Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$NEW_HEAD deep -> origin/deep + EOF +} + +check_super () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.super <<-EOF
Likewise.
Quoted 4 lines
t/t5526-fetch-submodules.sh +4
-0+ From $pwd/. + OLD_HEAD..$NEW_HEAD super -> origin/super + EOF +}
Re: [PATCH v4 02/10] t5526: stop asserting on stderr literally
In reply to Glen ChooQuoted 15 lines · Glen Choo
t/t5526-fetch-submodules.sh +15
-0+check_sub () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.sub <<-EOF + Fetching submodule submodule + From $pwd/submodule + OLD_HEAD..$NEW_HEAD sub -> origin/sub + EOF +} + +check_deep () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.deep <<-EOF + Fetching submodule submodule/subdir/deepsubmodule + From $pwd/deepsubmodule + OLD_HEAD..$NEW_HEAD deep -> origin/deep
t/t5526-fetch-submodules.sh export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB · line 37
Quoted 10 lines
t/t5526-fetch-submodules.sh +10
-0+ EOF +} + +check_super () { + NEW_HEAD=$1 && + cat >$pwd/expect.err.super <<-EOF + From $pwd/. + OLD_HEAD..$NEW_HEAD super -> origin/super + EOF +}
The check_ names still aren't changed (as I suggested in [1]) but perhaps it's fine to leave it. It doesn't seem to bother the other reviewers, and changing it would slightly disrupt the review in that there will be extra changes in the range-diff.
Re: [PATCH v4 02/10] t5526: stop asserting on stderr literally
Quoted 4 lines · Jonathan Tan
The
check_names still aren't changed (as I suggested in [1]) but perhaps it's fine to leave it. It doesn't seem to bother the other reviewers, and changing it would slightly disrupt the review in that there will be extra changes in the range-diff.
At least, please do not count my not mentioning it as such a vote. I didn't mention it because I saw you did.
Thanks.
Re: [PATCH v4 02/10] t5526: stop asserting on stderr literally
Quoted 9 lines · Junio C Hamano
Quoted 4 lines · Jonathan Tan
The
check_names still aren't changed (as I suggested in [1]) but perhaps it's fine to leave it. It doesn't seem to bother the other reviewers, and changing it would slightly disrupt the review in that there will be extra changes in the range-diff.At least, please do not count my not mentioning it as such a vote. I didn't mention it because I saw you did.
Oh! Sorry, I intended to do this - I missed this and another suggestion that you made re: test assertions [1]. Will incorporate this into the next round.
PATCH v4 3/10 t5526: create superproject commits with test helper
In reply to Glen ChooA few tests in t5526 use this pattern as part of their setup:
1. Create new commits in the upstream submodules (using add_upstream_commit()).
2. In the upstream superprojects, add the new submodule commits from the previous step.
A future commit will add more tests with this pattern, so reduce the verbosity of present and future tests by introducing a test helper that creates superproject commits. Since we now have two helpers that add upstream commits, rename add_upstream_commit() to add_submodule_commits().
--- t/t5526-fetch-submodules.sh | 94 +++++++++++++++++-------------------- 1 file changed, 44 insertions(+), 50 deletions(-)
t/t5526-fetch-submodules.sh +44 -50
@@ -40,7 +40,7 @@ check_super () { # a file that contains the expected err if that new commit were fetched. # These output files get concatenated in the right order by # verify_fetch_result(). -add_upstream_commit() { +add_submodule_commits () { ( cd submodule && echo new >> subfile &&
@@ -61,6 +61,30 @@ add_upstream_commit() { ) } +# For each superproject in the test setup, update its submodule, add the +# submodule and create a new commit with the submodule change. +# +# This requires add_submodule_commits() to be called first, otherwise +# the submodules will not have changed and cannot be "git add"-ed. +add_superproject_commits () { + ( + cd submodule && + ( + cd subdir/deepsubmodule && + git fetch && + git checkout -q FETCH_HEAD + ) && + git add subdir/deepsubmodule && + git commit -m "new deep submodule" + ) && + git add submodule && + git commit -m "new submodule" && + super_head=$(git rev-parse --short HEAD) && + sub_head=$(git -C submodule rev-parse --short HEAD) && + check_super $super_head && + check_sub $sub_head +} + # Verifies that the expected repositories were fetched. This is done by # concatenating the files expect.err.[super|sub|deep] in the correct # order and comparing it to the actual stderr.
@@ -117,7 +141,7 @@ test_expect_success setup ' ' test_expect_success "fetch --recurse-submodules recurses into submodules" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git fetch --recurse-submodules >../actual.out 2>../actual.err
@@ -127,7 +151,7 @@ test_expect_success "fetch --recurse-submodules recurses into submodules" ' ' test_expect_success "submodule.recurse option triggers recursive fetch" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git -c submodule.recurse fetch >../actual.out 2>../actual.err
@@ -137,7 +161,7 @@ test_expect_success "submodule.recurse option triggers recursive fetch" ' ' test_expect_success "fetch --recurse-submodules -j2 has the same output behaviour" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && GIT_TRACE="$TRASH_DIRECTORY/trace.out" git fetch --recurse-submodules -j2 2>../actual.err
@@ -148,7 +172,7 @@ test_expect_success "fetch --recurse-submodules -j2 has the same output behaviou ' test_expect_success "fetch alone only fetches superproject" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -177,7 +201,7 @@ test_expect_success "using fetchRecurseSubmodules=true in .gitmodules recurses i ' test_expect_success "--no-recurse-submodules overrides .gitmodules config" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git fetch --no-recurse-submodules >../actual.out 2>../actual.err
@@ -226,7 +250,7 @@ test_expect_success "--quiet propagates to parallel submodules" ' ' test_expect_success "--dry-run propagates to submodules" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err
@@ -245,7 +269,7 @@ test_expect_success "Without --dry-run propagates to submodules" ' ' test_expect_success "recurseSubmodules=true propagates into submodules" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git config fetch.recurseSubmodules true &&
@@ -256,7 +280,7 @@ test_expect_success "recurseSubmodules=true propagates into submodules" ' ' test_expect_success "--recurse-submodules overrides config in submodule" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && (
@@ -270,7 +294,7 @@ test_expect_success "--recurse-submodules overrides config in submodule" ' ' test_expect_success "--no-recurse-submodules overrides config setting" ' - add_upstream_commit && + add_submodule_commits && ( cd downstream && git config fetch.recurseSubmodules true &&
@@ -309,7 +333,7 @@ test_expect_success "Recursion stops when no new submodule commits are fetched" ' test_expect_success "Recursion doesn't happen when new superproject commits don't change any submodules" ' - add_upstream_commit && + add_submodule_commits && echo a > file && git add file && git commit -m "new file" &&
@@ -334,7 +358,7 @@ test_expect_success "Recursion picks up config in submodule" ' git config fetch.recurseSubmodules true ) ) && - add_upstream_commit && + add_submodule_commits && git add submodule && git commit -m "new submodule" && new_head=$(git rev-parse --short HEAD) &&
@@ -352,23 +376,8 @@ test_expect_success "Recursion picks up config in submodule" ' ' test_expect_success "Recursion picks up all submodules when necessary" ' - add_upstream_commit && - ( - cd submodule && - ( - cd subdir/deepsubmodule && - git fetch && - git checkout -q FETCH_HEAD - ) && - git add subdir/deepsubmodule && - git commit -m "new deepsubmodule" && - new_head=$(git rev-parse --short HEAD) && - check_sub $new_head - ) && - git add submodule && - git commit -m "new submodule" && - new_head=$(git rev-parse --short HEAD) && - check_super $new_head && + add_submodule_commits && + add_superproject_commits && ( cd downstream && git fetch >../actual.out 2>../actual.err
@@ -378,19 +387,7 @@ test_expect_success "Recursion picks up all submodules when necessary" ' ' test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)" ' - add_upstream_commit && - ( - cd submodule && - ( - cd subdir/deepsubmodule && - git fetch && - git checkout -q FETCH_HEAD - ) && - git add subdir/deepsubmodule && - git commit -m "new deepsubmodule" && - new_head=$(git rev-parse --short HEAD) && - check_sub $new_head - ) && + add_submodule_commits && ( cd downstream && git config fetch.recurseSubmodules true &&
@@ -402,10 +399,7 @@ test_expect_success "'--recurse-submodules=on-demand' doesn't recurse when no ne ' test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)" ' - git add submodule && - git commit -m "new submodule" && - new_head=$(git rev-parse --short HEAD) && - check_super $new_head && + add_superproject_commits && ( cd downstream && git config fetch.recurseSubmodules false &&
@@ -425,7 +419,7 @@ test_expect_success "'--recurse-submodules=on-demand' recurses as deep as necess ' test_expect_success "'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)" ' - add_upstream_commit && + add_submodule_commits && echo a >> file && git add file && git commit -m "new file" &&
@@ -446,7 +440,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' overrides global config cd downstream && git fetch --recurse-submodules ) && - add_upstream_commit && + add_submodule_commits && git config --global fetch.recurseSubmodules false && git add submodule && git commit -m "new submodule" &&
@@ -472,7 +466,7 @@ test_expect_success "'submodule.<sub>.fetchRecurseSubmodules=on-demand' override cd downstream && git fetch --recurse-submodules ) && - add_upstream_commit && + add_submodule_commits && git config fetch.recurseSubmodules false && git add submodule && git commit -m "new submodule" &&
@@ -522,7 +516,7 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git cd downstream && git fetch --recurse-submodules ) && - add_upstream_commit && + add_submodule_commits && git add submodule && git rm .gitmodules && git commit -m "new submodule without .gitmodules" &&
Signature · 1 line
2.33.GIT
Re: [PATCH v4 03/10] t5526: create superproject commits with test helper
Quoted 8 lines · Glen Choo
test_expect_success"'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)" 't/t5526-fetch-submodules.sh +0
-7- add_upstream_commit && - ( - cd submodule && - ( - cd subdir/deepsubmodule && - git fetch && - git checkout -q FETCH_HEAD
t/t5526-fetch-submodules.sh test_expect_success "Recursion picks up all submodules when necessary" ' · line 393
Quoted 10 lines
t/t5526-fetch-submodules.sh +1
-6- ) && - git add subdir/deepsubmodule && - git commit -m "new deepsubmodule" && - new_head=$(git rev-parse --short HEAD) && - check_sub $new_head - ) && + add_submodule_commits && ( cd downstream && git config fetch.recurseSubmodules true &&
The deletion of the block in which we updated submodule/subdir/deepsubmodule and submodule was a cause for concern, but now I think it's fine - add_upstream_commit already adds a commit in the submodule that would be fetched if we were to recurse into it. But since we are not recursing when no new commits are fetched in the superproject, we just have to detect that the commit added in add_upstream_commit wasn't fetched.
OK - patches up to and including this one look good.
PATCH v4 4/10 submodule: make static functions read submodules from commits
In reply to Glen ChooA future commit will teach "fetch --recurse-submodules" to fetch unpopulated submodules. To prepare for this, teach the necessary static functions how to read submodules from superproject commits using a "treeish_name" argument (instead of always reading from the index and filesystem) but do not actually change where submodules are read from. Submodules will be read from commits when we fetch unpopulated submodules.
--- submodule.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
submodule.c +19 -10
@@ -932,6 +932,7 @@ struct has_commit_data {struct repository *repo;int result;const char *path; +const struct object_id *super_oid; };static int check_has_commit (const struct object_id *oid,void *data)
@@ -940,7 +941,7 @@ static int check_has_commit(const struct object_id *oid, void *data)struct repository subrepo;enum object_type type; -if (repo_submodule_init (&subrepo, cb->repo, cb->path,null_oid ())) { +if (repo_submodule_init (&subrepo, cb->repo, cb->path, cb->super_oid)) { cb->result =0 ; goto cleanup; }
@@ -968,9 +969,15 @@ static int check_has_commit(const struct object_id *oid, void *data)static int submodule_has_commits (struct repository *r,const char *path, +const struct object_id *super_oid,struct oid_array *commits) { -struct has_commit_data has_commit = { r,1 , path }; +struct has_commit_data has_commit = { + .repo = r, + .result =1 , + .path = path, + .super_oid = super_oid + };/* * Perform a cheap, but incorrect check for the existence of 'commits'.
@@ -1017,7 +1024,7 @@ static int submodule_needs_pushing(struct repository *r,const char *path, struct oid_array *commits) { - if (!submodule_has_commits(r, path, commits)) + if (!submodule_has_commits(r, path, null_oid(), commits))/* * NOTE: We do consider it safe to return "no" here. The * correct answer would be "We do not know" instead of
@@ -1277,7 +1284,7 @@ static void calculate_changed_submodule_paths(struct repository *r,if (!path)continue ; -if (submodule_has_commits (r, path, commits)) { +if (submodule_has_commits (r, path,null_oid (), commits)) {oid_array_clear (commits); *name->string ='\0' ; }
@@ -1402,12 +1409,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path) }static struct fetch_task *fetch_task_create (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct fetch_task *task =xmalloc (sizeof (*task));memset (task,0 ,sizeof (*task)); - task->sub =submodule_from_path (r,null_oid (), path); + task->sub =submodule_from_path (r, treeish_name, path);if (!task->sub) {/* * No entry in .gitmodules? Technically not a submodule,
@@ -1439,11 +1447,12 @@ static void fetch_task_release(struct fetch_task *p) }static struct repository *get_submodule_repo_for (struct repository *r, -const char *path) +const char *path, +const struct object_id *treeish_name) {struct repository *ret =xmalloc (sizeof (*ret)); -if (repo_submodule_init (ret, r, path,null_oid ())) { +if (repo_submodule_init (ret, r, path, treeish_name)) {free (ret);return NULL ; }
@@ -1464,7 +1473,7 @@ static int get_next_submodule(struct child_process *cp,if (!S_ISGITLINK (ce->ce_mode))continue ; - task =fetch_task_create (spf->r, ce->name); + task =fetch_task_create (spf->r, ce->name,null_oid ());if (!task)continue ;
@@ -1487,7 +1496,7 @@ static int get_next_submodule(struct child_process *cp,continue ; } - task->repo =get_submodule_repo_for (spf->r, task->sub->path); + task->repo =get_submodule_repo_for (spf->r, task->sub->path,null_oid ());if (task->repo) {struct strbuf submodule_prefix =STRBUF_INIT ;child_process_init (cp);
Signature · 1 line
2.33.GIT
PATCH v4 5/10 submodule: inline submodule_commits() into caller
In reply to Glen ChooWhen collecting the string_list of changed submodule names, the new submodules commits are stored in the string_list_item.util as an oid_array. A subsequent commit will replace the oid_array with a struct that has more information.
Prepare for this change by inlining submodule_commits() (which inserts into the string_list and initializes the string_list_item.util) into its only caller so that the code is easier to refactor later.
--- submodule.c | 22 ++++++---------------- 1 file changed, 6 insertions(+), 16 deletions(-)
submodule.c +6 -16
@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)return submodule_from_path (the_repository,null_oid (), ce->name); } -static struct oid_array *submodule_commits (struct string_list *submodules, -const char *name) -{ -struct string_list_item *item; - - item =string_list_insert (submodules, name); -if (item->util) -return (struct oid_array *) item->util; - -/* NEEDSWORK: should we have oid_array_init()? */ - item->util =xcalloc (1 ,sizeof (struct oid_array )); -return (struct oid_array *) item->util; -}struct collect_changed_submodules_cb_data {struct repository *repo;
@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,for (i =0 ; i < q->nr; i++) {struct diff_filepair *p = q->queue[i]; -struct oid_array *commits;const struct submodule *submodule;const char *name; +struct string_list_item *item;if (!S_ISGITLINK (p->two->mode))continue ;
@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,if (!name)continue ; - commits =submodule_commits (changed, name); -oid_array_append (commits, &p->two->oid); + item =string_list_insert (changed, name); +if (!item->util) +/* NEEDSWORK: should we have oid_array_init()? */ + item->util =xcalloc (1 ,sizeof (struct oid_array )); +oid_array_append (item->util, &p->two->oid); } }
Signature · 1 line
2.33.GIT
PATCH v4 6/10 submodule: store new submodule commits oid_array in a struct
In reply to Glen ChooThis commit prepares for a future commit that will teach git fetch --recurse-submodules how to fetch submodules that are present in <gitdir>/modules, but are not populated. To do this, we need to store more information about the changed submodule so that we can read the submodule configuration from the superproject commit instead of the filesystem.
Refactor the changed submodules string_list.util to hold a struct instead of an oid_array. This struct only holds the new_commits oid_array for now; more information will be added later.
--- submodule.c | 52 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 18 deletions(-)
submodule.c +34 -18
@@ -806,6 +806,20 @@ static const char *default_name_or_path(const char *path_or_name)return path_or_name; } +/* +* Holds relevant information for a changed submodule. Used as the .util +* member of the changed submodule string_list_item. +*/ +struct changed_submodule_data { +/* The submodule commits that have changed in the rev walk. */ +struct oid_array new_commits; +}; + +static void changed_submodule_data_clear (struct changed_submodule_data *cs_data) +{ +oid_array_clear (&cs_data->new_commits); +} +static void collect_changed_submodules_cb (struct diff_queue_struct *q,struct diff_options *options,void *data)
@@ -820,6 +834,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,const struct submodule *submodule;const char *name;struct string_list_item *item; +struct changed_submodule_data *cs_data;if (!S_ISGITLINK (p->two->mode))continue ;
@@ -848,9 +863,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q, item =string_list_insert (changed, name);if (!item->util) -/* NEEDSWORK: should we have oid_array_init()? */ - item->util =xcalloc (1 ,sizeof (struct oid_array )); -oid_array_append (item->util, &p->two->oid); + item->util =xcalloc (1 ,sizeof (struct changed_submodule_data )); + cs_data = item->util; +oid_array_append (&cs_data->new_commits, &p->two->oid); } }
@@ -897,11 +912,12 @@ static void collect_changed_submodules(struct repository *r,reset_revision_walk (); } -static void free_submodules_oids (struct string_list *submodules) +static void free_submodules_data (struct string_list *submodules) {struct string_list_item *item;for_each_string_list_item (item, submodules) -oid_array_clear ((struct oid_array *) item->util); +changed_submodule_data_clear (item->util); +string_list_clear (submodules,1 ); }
@@ -1074,7 +1090,7 @@ int find_unpushed_submodules(struct repository *r,collect_changed_submodules (r, &submodules, &argv);for_each_string_list_item (name, &submodules) { -struct oid_array *commits = name->util; +struct changed_submodule_data *cs_data = name->util;const struct submodule *submodule;const char *path =NULL ;
@@ -1087,11 +1103,11 @@ int find_unpushed_submodules(struct repository *r,if (!path)continue ; -if (submodule_needs_pushing (r, path, commits)) +if (submodule_needs_pushing (r, path, &cs_data->new_commits))string_list_insert (needs_pushing, path); } -free_submodules_oids (&submodules); +free_submodules_data (&submodules);strvec_clear (&argv);return needs_pushing->nr;
@@ -1261,7 +1277,7 @@ static void calculate_changed_submodule_paths(struct repository *r,collect_changed_submodules (r, changed_submodule_names, &argv);for_each_string_list_item (name, changed_submodule_names) { -struct oid_array *commits = name->util; +struct changed_submodule_data *cs_data = name->util;const struct submodule *submodule;const char *path =NULL ;
@@ -1274,8 +1290,8 @@ static void calculate_changed_submodule_paths(struct repository *r,if (!path)continue ; -if (submodule_has_commits (r, path,null_oid (), commits)) { -oid_array_clear (commits); +if (submodule_has_commits (r, path,null_oid (), &cs_data->new_commits)) { +changed_submodule_data_clear (cs_data); *name->string ='\0' ; } }
@@ -1312,7 +1328,7 @@ int submodule_touches_in_range(struct repository *r,strvec_clear (&args); -free_submodules_oids (&subs); +free_submodules_data (&subs);return ret; }
@@ -1596,7 +1612,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,struct fetch_task *task = task_cb;struct string_list_item *it; -struct oid_array *commits; +struct changed_submodule_data *cs_data;if (!task || !task->sub)BUG ("callback cookie bogus" );
@@ -1624,14 +1640,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,/* Could be an unchanged submodule, not contained in the list */ goto out; - commits = it->util; -oid_array_filter (commits, + cs_data = it->util; +oid_array_filter (&cs_data->new_commits, commit_missing_in_sub, task->repo);/* Are there commits we want, but do not exist? */ -if (commits->nr) { - task->commits = commits; +if (cs_data->new_commits.nr) { + task->commits = &cs_data->new_commits;ALLOC_GROW (spf->oid_fetch_tasks, spf->oid_fetch_tasks_nr +1 , spf->oid_fetch_tasks_alloc);
@@ -1689,7 +1705,7 @@ int fetch_populated_submodules(struct repository *r,strvec_clear (&spf.args); out: -free_submodules_oids (&spf.changed_submodule_names); +free_submodules_data (&spf.changed_submodule_names);return spf.result; }
Signature · 1 line
2.33.GIT