Skip to contentSkip to conversation

fetch: introduce machine-parseable output

PATCH Superseded Patrick Steinhardt 7 participants 120 messages Apr 19, 2023 – May 15, 2023 Documentation

lore
Share & tools
lore
#0 lore raw

cover fetch: introduce machine-parseable output

Hi,

this is the second part of my quest to introduce a machine-parseable output for git-fetch(1) after the initial refactorings that have been merged via e9dffbc7f1 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06).

Parsing the output of fetches is mostly impossible. It prettifies reference names that are about to be updated, doesn't print the old and new object IDs the refs are being updated from and to, and prints all of that information in nice columns. In short, it is designed to be read by humans rather than machines.

This makes it hard to use in a script way though, e.g. to learn about which references actually have been updated or which have not been updated. This patch series intends to fix that by introducing a new machine-parseable interface:

```
$ git fetch --output-format=porcelain --no-progress
  fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
* 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
* 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
* 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
* 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
* 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
```

The series is structured as following:

- Patches 1 and 2 improve test coverage for output formats.

- Patch 3 fixes a bug with the current output format.

- Patch 4 to 6 perform some preliminary refactorings.

- Patch 7 introduces a new --output-format= option for git-fetch(1) that allows the user to configure the output more directly.

- Patch 8 introduces the new "porcelain" output format.

Patrick

Patrick Steinhardt (8):
fetch: split out tests for output format
fetch: add a test to exercise invalid output formats
fetch: fix missing from-reference when fetching HEAD:foo
fetch: introduce display_format enum
fetch: move display format parsing into main function
fetch: move option related variables into main function
fetch: introduce new --output-format option
fetch: introduce machine-parseable "porcelain" output format

 Documentation/config/fetch.txt  |   4 +-
 Documentation/fetch-options.txt |   5 +
 Documentation/git-fetch.txt     |  17 +-
 builtin/fetch.c                 | 406 +++++++++++++++++++-------------
 t/t5510-fetch.sh                |  53 -----
 t/t5574-fetch-output.sh         | 209 ++++++++++++++++
 6 files changed, 475 insertions(+), 219 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Signature · 1 line
2.40.0
#1 lore raw

PATCH 1/8 fetch: split out tests for output format

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
t/t5510-fetch.sh +0 -53
index dc44da9c79..4f289063ce 100755
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
t/t5574-fetch-output.sh new file +63 -0

63 lines not on this page · raw message

Signature · 1 line
2.40.0
#2 lore raw

PATCH 2/8 fetch: add a test to exercise invalid output formats

In reply to Patrick Steinhardt

Add a testcase that exercises the logic when an invalid output format is passed via the fetch.output configuration.

---
 t/t5574-fetch-output.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
t/t5574-fetch-output.sh +17 -0
index f91b654d38..0e45c27007 100755
@@ -7,6 +7,23 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_cmp expect actual &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
Signature · 1 line
2.40.0
#3 lore raw

PATCH 3/8 fetch: fix missing from-reference when fetching HEAD:foo

In reply to Patrick Steinhardt

When displaying reference updates, we print a line that looks similar to the following:

```
 * branch               master          -> master
```

The "branch" bit changes depending on what kind of reference we're updating, while both of the right-hand references are computed by stripping well-known prefixes like "refs/heads/" or "refs/tags".

The logic is kind of intertwined though and not easy to follow: we precompute both the kind (e.g. "branch") and the what, which is the abbreviated remote reference name, in store_updated_refs() and then pass it down the call chain to display_ref_update().

There is a set of different cases here:

- When the remote reference name is "HEAD" we assume no kind and will thus instead print "[new ref]". We keep what at the empty string.

- When the remote reference name has a well-known prefix then the kind would be "branch", "tag" or "remote-tracking branch". The what is the reference with the well-known prefix stripped and in fact matches the output that prettify_refname() would return.

- Otherwise, we'll again assume no kind and keep the what set to the fully qualified reference name.

Now there is a bug with the first case here, where the remote reference name is "HEAD". As noted, "what" will be set to the empty string. And that seems to be intentional because we also use this information to update the FETCH_HEAD, and in case we're updating HEAD we seemingly don't want to append that to our FETCH_HEAD value.

But as mentioned, we also use this value to display reference updates. And while the call to display_ref_update() correctly figures out that we meant "HEAD" when what is empty, the call to update_local_ref() doesn't. update_local_ref() will then call display_ref_update() with the empty string and cause the following broken output:

```
$ git fetch --dry-run origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands work as expected:

```
$ git fetch --dry-run origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

$ git fetch --dry-run origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

Fix this bug by instead unconditionally passing the full reference name to display_ref_update() which learns to call prettify_refname() on it. This does fix the above bug and is otherwise functionally the same as prettify_refname() would only ever strip the well-known prefixes just as intended. So at the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

---
 builtin/fetch.c         | 37 +++++++++++++++++++------------------
 t/t5574-fetch-output.sh | 19 +++++++++++++++++++
 2 files changed, 38 insertions(+), 18 deletions(-)
builtin/fetch.c +19 -18
index c310d89878..7c64f0c562 100644
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
t/t5574-fetch-output.sh +19 -0
index 0e45c27007..55f0f05b6a 100755
@@ -54,6 +54,25 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
Signature · 1 line
2.40.0
#4 lore raw

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

Quoted 70 lines · Patrick Steinhardt

When displaying reference updates, we print a line that looks similar to the following:

```
 * branch               master          -> master
```

The "branch" bit changes depending on what kind of reference we're updating, while both of the right-hand references are computed by stripping well-known prefixes like "refs/heads/" or "refs/tags".

The logic is kind of intertwined though and not easy to follow: we precompute both the kind (e.g. "branch") and the what, which is the abbreviated remote reference name, in store_updated_refs() and then pass it down the call chain to display_ref_update().

There is a set of different cases here:

- When the remote reference name is "HEAD" we assume no kind and will thus instead print "[new ref]". We keep what at the empty string.

- When the remote reference name has a well-known prefix then the kind would be "branch", "tag" or "remote-tracking branch". The what is the reference with the well-known prefix stripped and in fact matches the output that prettify_refname() would return.

- Otherwise, we'll again assume no kind and keep the what set to the fully qualified reference name.

Now there is a bug with the first case here, where the remote reference name is "HEAD". As noted, "what" will be set to the empty string. And that seems to be intentional because we also use this information to update the FETCH_HEAD, and in case we're updating HEAD we seemingly don't want to append that to our FETCH_HEAD value.

But as mentioned, we also use this value to display reference updates. And while the call to display_ref_update() correctly figures out that we meant "HEAD" when what is empty, the call to update_local_ref() doesn't. update_local_ref() will then call display_ref_update() with the empty string and cause the following broken output:

```
$ git fetch --dry-run origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands work as expected:

```
$ git fetch --dry-run origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

$ git fetch --dry-run origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

Fix this bug by instead unconditionally passing the full reference name to display_ref_update() which learns to call prettify_refname() on it. This does fix the above bug and is otherwise functionally the same as prettify_refname() would only ever strip the well-known prefixes just as intended. So at the same time, this also simplifies the code a bit.

Quoted 5 lines

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

The commit message here has a lot of context, but I found it a bit hard to parse through, especially relative to the actual fix in code.

One suggestion was to load the paragraphs a bit more with the actual problem being solved first, before beginning a lot of the context.

We also discussed the block of format changes and felt a bit mixed on whether to include it or not. It does match the coding style guidelines, but there is no actual functional change made to those lines in this series.

I think its a good improvement, and it does force some extra context into the diff which makes reading the resulting change easier.

#5 lore raw

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

Quoted 1 line · Jacob Keller

On 4/19/2023 5:31 AM, Patrick Steinhardt wrote:

[snip]

Quoted 6 lines

The commit message here has a lot of context, but I found it a bit hard to parse through, especially relative to the actual fix in code.

One suggestion was to load the paragraphs a bit more with the actual problem being solved first, before beginning a lot of the context.

Quoted 6 lines

We also discussed the block of format changes and felt a bit mixed on whether to include it or not. It does match the coding style guidelines, but there is no actual functional change made to those lines in this series.

I think its a good improvement, and it does force some extra context into the diff which makes reading the resulting change easier.

I initially really struggled to explain to myself why my change actually works, and that shows in the commit message. I've reworded it a bit and took some of the suggested changes that Glen posted to hopefully make this clearer.

Patrick

#6 lore raw

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

In reply to Patrick Steinhardt
Quoted 10 lines · Patrick Steinhardt
t/t5574-fetch-output.sh +9 -0
+test_expect_success 'fetch output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF

t/t5574-fetch-output.sh test_expect_success 'fetch compact output' ' · line 75

Quoted 10 lines
t/t5574-fetch-output.sh +10 -0
+	test_cmp expect actual &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_cmp expect actual
+'
+

The test mentions HEAD and --dry-run, but the bug seems to exist regardless of whether --dry-run is used. I understand the use of --dry-run for testing fetch output so that you can repeatably run git fetch and get the same results.

The tests here should probably also have a test that covers fetch without --dry-run though.

Thanks,
Jake

#7 lore raw

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

Quoted 29 lines · Jacob Keller
Quoted 10 lines · Patrick Steinhardt
t/t5574-fetch-output.sh +9 -0
+test_expect_success 'fetch output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF

t/t5574-fetch-output.sh test_expect_success 'fetch compact output' ' · line 75

Quoted 10 lines
t/t5574-fetch-output.sh +10 -0
+	test_cmp expect actual &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_cmp expect actual
+'
+

The test mentions HEAD and --dry-run, but the bug seems to exist regardless of whether --dry-run is used. I understand the use of --dry-run for testing fetch output so that you can repeatably run git fetch and get the same results.

Quoted 2 lines

The tests here should probably also have a test that covers fetch without --dry-run though.

Makes sense, will amend!

Patrick

#8 lore raw

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

In reply to Patrick Steinhardt

Rearranging the lines slightly,

Quoted 24 lines · Patrick Steinhardt

When displaying reference updates, we print a line that looks similar to the following:

```
 * branch               master          -> master
```

The "branch" bit changes depending on what kind of reference we're updating, while both of the right-hand references are computed by stripping well-known prefixes like "refs/heads/" or "refs/tags".

[...]
we also use this value to display reference updates.
And while the call to display_ref_update() correctly figures out that we meant "HEAD" when what is empty, the call to update_local_ref() doesn't. update_local_ref() will then call display_ref_update() with the empty string and cause the following broken output:

```
$ git fetch --dry-run origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```
Quoted 8 lines

[...]

Fix this bug by instead unconditionally passing the full reference name to display_ref_update() which learns to call prettify_refname() on it. This does fix the above bug and is otherwise functionally the same as prettify_refname() would only ever strip the well-known prefixes just as intended. So at the same time, this also simplifies the code a bit.

The bug fix is obviously good. I'm surprised we hadn't caught this sooner.

As a nitpicky comment, the commit message goes into a lot of detail, which makes it tricky to read on its own (though the level of detail makes it easy to match to the diff, making the diff quite easy to follow). I would have found this easier to read by summarizing the high-level mental model before diving into the background, e.g.

store_updated_refs() parses the remote ref name to create a 'note' to write to FETCH_HEAD. This note is usually the prettified ref name, so it is used to diplay ref updates (display_ref_update()). But if the remote ref is HEAD, the note is the empty string [insert bug description]. Instead, use the note only as a note and have display_ref_update() prettify the ref name itself...

Quoted 11 lines
builtin/fetch.c +2 -0
index c310d89878..7c64f0c562 100644
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 

builtin/fetch.c static void display_ref_update(struct display_state *display_state, char code, · line 931

Quoted 10 lines
builtin/fetch.c +2 -2
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');

As expected, we now prettify the name isntead of trusting the 'note' that came in the parameter...

Quoted 75 lines
builtin/fetch.c +9 -9
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }

...

Quoted 10 lines
builtin/fetch.c +1 -1
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {

builtin/fetch.c static int store_updated_refs(struct display_state *display_state, · line 1295

Quoted 8 lines
builtin/fetch.c +1 -1
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}

and we stop passing the 'note' as a parameter. Looks good.

Quoted 9 lines
builtin/fetch.c +1 -3
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))

builtin/fetch.c static int store_updated_refs(struct display_state *display_state, · line 1266

Quoted 10 lines
builtin/fetch.c +3 -2
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}

I really appreciate that this makes the patch easier to read. I don't really appreciate this sort of churn, but it _is_ following CodingGuidelines:

	- When there are multiple arms to a conditional and some of them
	  require braces, enclose even a single line block in braces for
	  consistency. E.g.:

		if (foo) {
			doit();
		} else {
			one();
			two();
			three();
		}

(I initialy thought it wasn't. Thanks to other Review Club participants for pointing this out).

t/t5574-fetch-output.sh test_expect_success 'fetch compact output' ' · line 57

Quoted 9 lines
t/t5574-fetch-output.sh +1 -0
index 0e45c27007..55f0f05b6a 100755
@@ -54,6 +54,25 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD and --dry-run' '

The commit message and diff didn't imply that this is a --dry-run only bug. I tested locally, and it seems to reproduce without --dry-run too, so I think we should drop "--dry-run" from this test name. In a later patch, you also add a test for porcelain output with --dry-run, but since this test seems designed for just this bug, I think we can drop the later test.

#9 lore raw

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

Quoted 48 lines · Glen Choo

Rearranging the lines slightly,

Quoted 24 lines · Patrick Steinhardt

When displaying reference updates, we print a line that looks similar to the following:

```
 * branch               master          -> master
```

The "branch" bit changes depending on what kind of reference we're updating, while both of the right-hand references are computed by stripping well-known prefixes like "refs/heads/" or "refs/tags".

[...]
we also use this value to display reference updates.
And while the call to display_ref_update() correctly figures out that we meant "HEAD" when what is empty, the call to update_local_ref() doesn't. update_local_ref() will then call display_ref_update() with the empty string and cause the following broken output:

```
$ git fetch --dry-run origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```
Quoted 8 lines

[...]

Fix this bug by instead unconditionally passing the full reference name to display_ref_update() which learns to call prettify_refname() on it. This does fix the above bug and is otherwise functionally the same as prettify_refname() would only ever strip the well-known prefixes just as intended. So at the same time, this also simplifies the code a bit.

The bug fix is obviously good. I'm surprised we hadn't caught this sooner.

As a nitpicky comment, the commit message goes into a lot of detail, which makes it tricky to read on its own (though the level of detail makes it easy to match to the diff, making the diff quite easy to follow). I would have found this easier to read by summarizing the high-level mental model before diving into the background, e.g.

Quoted 6 lines

store_updated_refs() parses the remote ref name to create a 'note' to write to FETCH_HEAD. This note is usually the prettified ref name, so it is used to diplay ref updates (display_ref_update()). But if the remote ref is HEAD, the note is the empty string [insert bug description]. Instead, use the note only as a note and have display_ref_update() prettify the ref name itself...

I like that and will use a variant of this.

[snip]

Quoted 8 lines
Quoted 8 lines
t/t5574-fetch-output.sh +0 -0
index 0e45c27007..55f0f05b6a 100755
@@ -54,6 +54,25 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
Quoted 8 lines

t/t5574-fetch-output.sh test_expect_success 'fetch compact output' ' · line 57

Quoted 1 line
t/t5574-fetch-output.sh +1 -0
+test_expect_success 'fetch output with HEAD and --dry-run' '

The commit message and diff didn't imply that this is a --dry-run only bug. I tested locally, and it seems to reproduce without --dry-run too, so I think we should drop "--dry-run" from this test name. In a later patch, you also add a test for porcelain output with --dry-run, but since this test seems designed for just this bug, I think we can drop the later test.

True, I'll amend the test.

Patrick

#10 lore raw

PATCH 4/8 fetch: introduce `display_format` enum

In reply to Patrick Steinhardt

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a enum display_format that is more readily extensible.

---
 builtin/fetch.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
builtin/fetch.c +64 -37
index 7c64f0c562..e03fcd1b2f 100644
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -784,7 +790,6 @@ static int refcol_width(const struct ref *ref, int compact_format)
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display foramt %d", display_state->format);
 	}
 }
 
@@ -904,30 +920,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
Signature · 1 line
2.40.0
#11 lore raw

PATCH 5/8 fetch: move display format parsing into main function

In reply to Patrick Steinhardt

Parsing the display format happens inside of display_state_init(). As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the fetch.output config key set. We're thus going to introduce a new --output-format switch for git-fetch(1) so that the output format can be configured more directly.

This means we'll have to hook parsing of the display format into the command line options parser. Having the code to determine the actual output format scattered across two different sites is hard to reason about though.

Refactor the code such that callers are expected to pass the display format that is to be used into display_state_init(). This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

---
 builtin/fetch.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
builtin/fetch.c +24 -17
index e03fcd1b2f..bcc156a9ce 100644
@@ -788,14 +788,13 @@ static int refcol_width(const struct ref *ref, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -812,15 +811,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT: {
@@ -1614,7 +1604,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1700,7 +1691,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2076,7 +2067,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2143,7 +2135,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2155,6 +2147,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2181,6 +2174,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}
+
 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
 		recurse_submodules = recurse_submodules_cli;
 
@@ -2309,7 +2315,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   display_format);
 	} else {
 		int max_children = max_jobs;
 
Signature · 1 line
2.40.0
#12 lore raw

PATCH 6/8 fetch: move option related variables into main function

In reply to Patrick Steinhardt

The options of git-fetch(1) which we pass to parse_options() are declared globally in builtin/fetch.c. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into cmd_fetch(). Move variables that were previously forced to be declared globally and which are only used by cmd_fetch() into function-local scope.

---
 builtin/fetch.c | 194 ++++++++++++++++++++++++------------------------
 1 file changed, 98 insertions(+), 96 deletions(-)
builtin/fetch.c +98 -96
index bcc156a9ce..81581b0033 100644
@@ -79,9 +79,8 @@ static int all, append, dry_run, force, keep, multiple, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
@@ -160,92 +153,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2145,13 +2052,108 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
-	int i;
 	const char *bundle_uri;
+	const char *submodule_prefix = "";
 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
Signature · 1 line
2.40.0
#13 lore raw

PATCH 7/8 fetch: introduce new `--output-format` option

In reply to Patrick Steinhardt

It is only possible to configure the output format that git-fetch(1) uses by setting it via a config key. While this interface may be fine as long as we only have the current "full" and "compact" output formats, where it is unlikely that the user will have to change them regularly. But we're about to introduce a new machine-parseable interface where the current mechanism feels a little bit indirect and rigid.

Introduce a new --output-format option that allows the user to change the desired output format more directly.

---
 Documentation/fetch-options.txt |  5 ++++
 builtin/fetch.c                 | 17 +++++++++++
 t/t5574-fetch-output.sh         | 50 +++++++++++++++++++++++++--------
 3 files changed, 60 insertions(+), 12 deletions(-)
Documentation/fetch-options.txt +5 -0
index 622bd84768..654f96f79d 100644
@@ -78,6 +78,11 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--output-format::
+	Control how ref update status is printed. Valid values are
+	`full` and `compact`. Default value is `full`. See section
+	OUTPUT in linkgit:git-fetch[1] for detail.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
builtin/fetch.c +17 -0
index 81581b0033..22ba75a958 100644
@@ -2050,6 +2050,21 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	return exit_code;
 }
 
+static int opt_parse_output_format(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value;
+	if (unset || !arg)
+		return 1;
+	else if (!strcmp(arg, "full"))
+		*format = DISPLAY_FORMAT_FULL;
+	else if (!strcmp(arg, "compact"))
+		*format = DISPLAY_FORMAT_COMPACT;
+	else
+		return error(_("unsupported output format '%s'"), arg);
+
+	return 0;
+}
+
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	const char *bundle_uri;
@@ -2101,6 +2116,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK(0, "output-format", &display_format, N_("format"), N_("output format"),
+			     opt_parse_output_format),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
t/t5574-fetch-output.sh +38 -12

65 lines not on this page · raw message

Signature · 1 line
2.40.0
#14 lore raw

Re: [PATCH 7/8] fetch: introduce new `--output-format` option

builtin/fetch.c int cmd_fetch(int argc, const char **argv, const char *prefix) · line 2123

Quoted 9 lines · Patrick Steinhardt
builtin/fetch.c +2 -0
@@ -2101,6 +2116,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK(0, "output-format", &display_format, N_("format"), N_("output format"),
+			     opt_parse_output_format),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),

This change is good enough for fetching from a single remote, but if we want to support "--all", we'd also need to propagate the CLI flag to the child "fetch" processes. (The config option wouldn't have this bug because the child processes would parse config and get the correct value.)

#15 lore raw

Re: [PATCH 7/8] fetch: introduce new `--output-format` option

Quoted 17 lines · Glen Choo

builtin/fetch.c int cmd_fetch(int argc, const char **argv, const char *prefix) · line 2123

Quoted 9 lines · Patrick Steinhardt
builtin/fetch.c +2 -0
@@ -2101,6 +2116,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK(0, "output-format", &display_format, N_("format"), N_("output format"),
+			     opt_parse_output_format),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),

This change is good enough for fetching from a single remote, but if we want to support "--all", we'd also need to propagate the CLI flag to the child "fetch" processes. (The config option wouldn't have this bug because the child processes would parse config and get the correct value.)

Oh, right, good catch. Will fix, thanks!

Patrick

#16 lore raw

PATCH 8/8 fetch: introduce machine-parseable "porcelain" output format

In reply to Patrick Steinhardt

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```

We assume two conditions which are generally true:

- The old and new object IDs have fixed known widths and cannot contain spaces.

- References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

---
 Documentation/config/fetch.txt  |  4 +-
 Documentation/fetch-options.txt |  4 +-
 Documentation/git-fetch.txt     | 17 ++++++-
 builtin/fetch.c                 | 50 +++++++++++++++-----
 t/t5574-fetch-output.sh         | 84 +++++++++++++++++++++++++++++++++
 5 files changed, 142 insertions(+), 17 deletions(-)
Documentation/config/fetch.txt +2 -2
index 568f0f75b3..70734226c0 100644
@@ -52,8 +52,8 @@ fetch.pruneTags::
 
 fetch.output::
 	Control how ref update status is printed. Valid values are
-	`full` and `compact`. Default value is `full`. See section
-	OUTPUT in linkgit:git-fetch[1] for detail.
+	`full`, `compact` and `porcelain`. Default value is `full`.
+	See section OUTPUT in linkgit:git-fetch[1] for detail.
 
 fetch.negotiationAlgorithm::
 	Control how information about the commits in the local repository
Documentation/fetch-options.txt +2 -2
index 654f96f79d..5ca8a67fe8 100644
@@ -80,8 +80,8 @@ linkgit:git-config[1].
 
 --output-format::
 	Control how ref update status is printed. Valid values are
-	`full` and `compact`. Default value is `full`. See section
-	OUTPUT in linkgit:git-fetch[1] for detail.
+	`full`, `compact` and `porcelain`. Default value is `full`.
+	See section OUTPUT in linkgit:git-fetch[1] for detail.
 
 ifndef::git-pull[]
 --[no-]write-fetch-head::
Documentation/git-fetch.txt +15 -2
index fba66f1460..efd22cd372 100644
@@ -197,13 +197,26 @@ The output of "git fetch" depends on the transport method used; this
 section describes the output when fetching over the Git protocol
 (either locally or via ssh) and Smart HTTP protocol.
 
-The status of the fetch is output in tabular form, with each line
-representing the status of a single ref. Each line is of the form:
+The output format can be chosen either via the `fetch.output` config
+(see linkgit:git-config[1]), or via the `--output-format` switch.
+Supported values include:
+
+For the `full` and `compact` output formats, the status of the fetch is
+output in tabular, with each line representing the status of a single
+ref. Each line is of the form:
 
 -------------------------------
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+The `porcelain` output format is intended to be machine-parseable. In
+contrast to the human-readable output formats it thus prints to standard
+output instead of standard error. Each line is of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
builtin/fetch.c +39 -11

150 lines not on this page · raw message

t/t5574-fetch-output.sh +84 -0

96 lines not on this page · raw message

Signature · 1 line
2.40.0
#17 lore raw

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 5 lines · Patrick Steinhardt

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```

This format doesn't show the remote name or url that was fetched. That seems okay when fetching with a single remote, but it seems necessary with "--all". Perhaps you were planning to add that in a later series? If so, I think it's okay to call the "porcelain" format experimental, and forbid porcelain + --all until then.

Quoted 7 lines

We assume two conditions which are generally true:

- The old and new object IDs have fixed known widths and cannot contain spaces.

- References cannot contain newlines.

This seems like a non-issue if we add a -z CLI option to indicate that entries should be NUL terminated instead of newline terminated, but that can be done as a followup.

Quoted 6 lines

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

Sending the 'main output' to stdout makes sense to me, but this (and possibly respecting -z) sounds like a different mode of operation, not just a matter of formats. It seems different enough that I'd prefer not to piggyback on "fetch.output" for this (even though this adds more surface to the interface...).

We could add --porcelain and say that "fetch.output" is ignored if --porcelain is also given. That also eliminates the need for --output-format, I think.

The .c changes look good to me.

Quoted 7 lines
t/t5574-fetch-output.sh +7 -0
+test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&

t/t5574-fetch-output.sh test_expect_success 'fetch output with HEAD and --dry-run' ' · line 184

Quoted 10 lines
t/t5574-fetch-output.sh +10 -0
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual
+'

As mentioned upthread, I think this test isn't needed because "porcelain" wouldn't run into the bug we are checking for anyway.

#18 lore raw

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 8 lines · Glen Choo
Quoted 5 lines · Patrick Steinhardt

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```
Quoted 5 lines

This format doesn't show the remote name or url that was fetched. That seems okay when fetching with a single remote, but it seems necessary with "--all". Perhaps you were planning to add that in a later series? If so, I think it's okay to call the "porcelain" format experimental, and forbid porcelain + --all until then.

The reason is mostly that I didn't find an output format that I really liked here. We'd basically have to repeat the remote URL for every single reference: just repeating it once per remote doesn't fly because with --parallel the output could be intermingled. But doing that feels wasteful to me, so I bailed. I guess I'm also biased here because it just wouldn't be useful to myself.

So with that in mind, I'd like to continue ignoring this issue for now and just not report the remote that the ref came from. But I'd also argue that we don't have to restrict porcelain mode to single-remote
fetches: it can still be useful to do multi-remote fetches even without the information where a certain reference update comes from. So any kind of restriction would feel artificial to me here.

Furthermore, I'd argue that it is not necessary to label the format as experimental only because of this limitation. With the refactorings done in this and the preceding patch series it is easy to add a new format in case there indeed is somebody that would have a usecase for this. The "porcelain" format should stay stable, and if we decide that we want to also report the remote for each reference in a follow-up we can easily add a "porcelain-v2" or "porcelain-with-remote" format.

As I said though: I'm clearly biased, so if you feel like my train of though is simply me being lazy then I'd carve in and adapt.

Quoted 10 lines
Quoted 6 lines

We assume two conditions which are generally true:

- The old and new object IDs have fixed known widths and cannot contain spaces.

- References cannot contain newlines.

This seems like a non-issue if we add a -z CLI option to indicate that entries should be NUL terminated instead of newline terminated, but that can be done as a followup.

Yeah, either via -z or a new porcelain output format. But both of these conditions should generally be true anyway, so I don't see that those should become a problem.

Quoted 13 lines
Quoted 6 lines

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

Sending the 'main output' to stdout makes sense to me, but this (and possibly respecting -z) sounds like a different mode of operation, not just a matter of formats. It seems different enough that I'd prefer not to piggyback on "fetch.output" for this (even though this adds more surface to the interface...).

Quoted 3 lines

We could add --porcelain and say that "fetch.output" is ignored if --porcelain is also given. That also eliminates the need for --output-format, I think.

I was thinking about this initially, as well. But ultimately I decided against this especially because of your second paragraph: we'd now need to think about precedence of options and mutual exclusion, and that to me feels like an interface that is less obvious than a single knob that works as you'd expect.

Quoted 14 lines

The .c changes look good to me.

Quoted 12 lines
t/t5574-fetch-output.sh +12 -0
+test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
Quoted 8 lines

t/t5574-fetch-output.sh test_expect_success 'fetch output with HEAD and --dry-run' ' · line 184

Quoted 5 lines
t/t5574-fetch-output.sh +5 -0
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual
+'

As mentioned upthread, I think this test isn't needed because "porcelain" wouldn't run into the bug we are checking for anyway.

The only reason that the other bug was able to survive for so long was that we didn't have test coverage there. So I think it makes sense to explicitly test this, too, also because it causes us to walk a different code path.

Last but not least: this test uncovered a segfault I had in a previous version. So I'd rather keep it :)

Patrick

#19 lore raw

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 15 lines · Patrick Steinhardt
Quoted 8 lines · Glen Choo
Quoted 5 lines · Patrick Steinhardt

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```
Quoted 5 lines

This format doesn't show the remote name or url that was fetched. That seems okay when fetching with a single remote, but it seems necessary with "--all". Perhaps you were planning to add that in a later series? If so, I think it's okay to call the "porcelain" format experimental, and forbid porcelain + --all until then.

Quoted 6 lines

The reason is mostly that I didn't find an output format that I really liked here. We'd basically have to repeat the remote URL for every single reference: just repeating it once per remote doesn't fly because with --parallel the output could be intermingled. But doing that feels wasteful to me, so I bailed. I guess I'm also biased here because it just wouldn't be useful to myself.

Yeah, I couldn't think of a good way to output the remote URL either.

Quoted 7 lines

So with that in mind, I'd like to continue ignoring this issue for now and just not report the remote that the ref came from. But I'd also argue that we don't have to restrict porcelain mode to single-remote
fetches: it can still be useful to do multi-remote fetches even without the information where a certain reference update comes from. So any kind of restriction would feel artificial to me here.

Quoted 7 lines

Furthermore, I'd argue that it is not necessary to label the format as experimental only because of this limitation. With the refactorings done in this and the preceding patch series it is easy to add a new format in case there indeed is somebody that would have a usecase for this. The "porcelain" format should stay stable, and if we decide that we want to also report the remote for each reference in a follow-up we can easily add a "porcelain-v2" or "porcelain-with-remote" format.

I agree that the current form is useful, but it seems quite likely that users will want the remote name/url at some point. In the majority of cases, the user can parse the remote name from the updated local ref and if they want, they look up the url by asking "git config", but that breaks down quite quickly if they have a custom refspec.

My suggestion to label it experimental was meant to give us the freedom to change the "first version" in backwards-incompatible ways that we otherwise couldn't. We could say "here's porcelain-v2, use this and ignore porcelain-v1", but it would be even nicer if the user didn't need to think about porcelain-v1. The idea is that we could call it experimental, see what users need in the wild, then make changes without having to create a new "porcelain-v2" mode.

(I realize in hindsight that calling it experimental sounds a bit insulting. I thought your proposal was quite sensible, actually. i apologize if it came across that way)

Perhaps part of the reason why you'd prefer not to make it experimental is that users who might want this feature are too afraid of backwards-incompatible changes. I think that's fair. A different way of achieving the same flexibility would be to make the "porcelain" format extensible (like git status --porcelain=v2). A future, multivalued "--porcelain-options=show-remote-url,show-remote-ref" would cover our bases.

I'm happy with either approach as long as we don't have to bikeshed about the "perfect" porcelain output :)

Quoted 17 lines
Quoted 13 lines
Quoted 6 lines

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

Sending the 'main output' to stdout makes sense to me, but this (and possibly respecting -z) sounds like a different mode of operation, not just a matter of formats. It seems different enough that I'd prefer not to piggyback on "fetch.output" for this (even though this adds more surface to the interface...).

Quoted 3 lines

We could add --porcelain and say that "fetch.output" is ignored if --porcelain is also given. That also eliminates the need for --output-format, I think.

Quoted 5 lines

I was thinking about this initially, as well. But ultimately I decided against this especially because of your second paragraph: we'd now need to think about precedence of options and mutual exclusion, and that to me feels like an interface that is less obvious than a single knob that works as you'd expect.

My full thoughts on this are in

https://lore.kernel.org/git/[email protected]

but the short version is that I'm not sure if I expect something as innocuous-looking as --output-format would imply other, machine-friendly things (like stdout instead of stderr), and using --porcelain might make option precedence clearer in some situtations (like if -z is given).

Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
t/t5574-fetch-output.sh +12 -0
+test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
Quoted 8 lines

t/t5574-fetch-output.sh test_expect_success 'fetch output with HEAD and --dry-run' ' · line 184

Quoted 5 lines
t/t5574-fetch-output.sh +5 -0
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual
+'

As mentioned upthread, I think this test isn't needed because "porcelain" wouldn't run into the bug we are checking for anyway.

Quoted 7 lines

The only reason that the other bug was able to survive for so long was that we didn't have test coverage there. So I think it makes sense to explicitly test this, too, also because it causes us to walk a different code path.

Last but not least: this test uncovered a segfault I had in a previous version. So I'd rather keep it :)

Ah, okay, both are fair. In that case, it probably makes sense to drop the "--dry-run".

#20 lore raw

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 3 lines · Glen Choo
Quoted 2 lines · Patrick Steinhardt
Quoted 1 line · Glen Choo

Patrick Steinhardt <[email protected]> writes:

[snip]

Quoted 29 lines
Quoted 7 lines

So with that in mind, I'd like to continue ignoring this issue for now and just not report the remote that the ref came from. But I'd also argue that we don't have to restrict porcelain mode to single-remote
fetches: it can still be useful to do multi-remote fetches even without the information where a certain reference update comes from. So any kind of restriction would feel artificial to me here.

Quoted 7 lines

Furthermore, I'd argue that it is not necessary to label the format as experimental only because of this limitation. With the refactorings done in this and the preceding patch series it is easy to add a new format in case there indeed is somebody that would have a usecase for this. The "porcelain" format should stay stable, and if we decide that we want to also report the remote for each reference in a follow-up we can easily add a "porcelain-v2" or "porcelain-with-remote" format.

I agree that the current form is useful, but it seems quite likely that users will want the remote name/url at some point. In the majority of cases, the user can parse the remote name from the updated local ref and if they want, they look up the url by asking "git config", but that breaks down quite quickly if they have a custom refspec.

My suggestion to label it experimental was meant to give us the freedom to change the "first version" in backwards-incompatible ways that we otherwise couldn't. We could say "here's porcelain-v2, use this and ignore porcelain-v1", but it would be even nicer if the user didn't need to think about porcelain-v1. The idea is that we could call it experimental, see what users need in the wild, then make changes without having to create a new "porcelain-v2" mode.

Quoted 3 lines

(I realize in hindsight that calling it experimental sounds a bit insulting. I thought your proposal was quite sensible, actually. i apologize if it came across that way)

No offense taken, so no need to apologize.

Quoted 3 lines

Perhaps part of the reason why you'd prefer not to make it experimental is that users who might want this feature are too afraid of backwards-incompatible changes. I think that's fair.

Exactly. I'm implementing this whole feature with a concrete usecase at GitLab in mind, and if we declare this as experimental then it will become a whole lot more risky for us to actually adopt it. Because if the output format may change at any point in time, then the code that builds on top of it is likely to be broken.

Quoted 5 lines

A different way of
achieving the same flexibility would be to make the "porcelain" format extensible (like git status --porcelain=v2). A future, multivalued "--porcelain-options=show-remote-url,show-remote-ref" would cover our bases.

Yeah, that would indeed be quite flexible. Theoretically, there is no reason why we couldn't have --output-format=porcelain,show-remote-url though.

Quoted 2 lines

I'm happy with either approach as long as we don't have to bikeshed about the "perfect" porcelain output :)

Agreed, and that's why I'm currently defending the "good enough" format. It should likely work for most usecases that exist out there. The target audience is going to be quite small here as this is not a user-directed feature. Furthermore, I assume that the overlap of machines that want to parse all reference updates done in a multi-remote fetch while also being able to exactly tell which remote brought in what updates is going to be tiny.

Quoted 23 lines
Quoted 17 lines
Quoted 13 lines
Quoted 6 lines

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

Sending the 'main output' to stdout makes sense to me, but this (and possibly respecting -z) sounds like a different mode of operation, not just a matter of formats. It seems different enough that I'd prefer not to piggyback on "fetch.output" for this (even though this adds more surface to the interface...).

Quoted 3 lines

We could add --porcelain and say that "fetch.output" is ignored if --porcelain is also given. That also eliminates the need for --output-format, I think.

Quoted 5 lines

I was thinking about this initially, as well. But ultimately I decided against this especially because of your second paragraph: we'd now need to think about precedence of options and mutual exclusion, and that to me feels like an interface that is less obvious than a single knob that works as you'd expect.

Quoted 8 lines

My full thoughts on this are in

https://lore.kernel.org/git/[email protected]

but the short version is that I'm not sure if I expect something as innocuous-looking as --output-format would imply other, machine-friendly things (like stdout instead of stderr), and using --porcelain might make option precedence clearer in some situtations (like if -z is given).

I'm not even sure that -z makes sense in this context. If we see cases where not using -z can cause the machine-readable interface to become unparseable then this is a bug in the output format, if you ask me. Mostly because the whole intent of it is to be machine-parseable. So if we output data that can e.g. contain newlines, then we must not use newlines as part of the output format or alternatively escape them. Why let the author of the script shoot themselves into the foot?

Anyway, I'm digressing. It's hard for me to decide what to do right now. The thread with Junio and Jacob points into the direction of keeping the --output-format= interface, while this thread points into the other direction. I'm naturally more inclined to keep --output-format=, mostly because I personally feel like it's the more obvious interface. But I also see your point, so it's not really a choice of right-or-wrong here, but rather of style.

Patrick

#21 lore raw

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 7 lines · Patrick Steinhardt
Quoted 2 lines

I'm happy with either approach as long as we don't have to bikeshed about the "perfect" porcelain output :)

Agreed, and that's why I'm currently defending the "good enough" format. It should likely work for most usecases that exist out there. The target audience is going to be quite small here as this is not a user-directed feature.

Okay, I agree that this is "good enough" for most cases and we can extend it if needed.

Quoted 9 lines
Quoted 8 lines

My full thoughts on this are in

https://lore.kernel.org/git/[email protected]

but the short version is that I'm not sure if I expect something as innocuous-looking as --output-format would imply other, machine-friendly things (like stdout instead of stderr), and using --porcelain might make option precedence clearer in some situtations (like if -z is given).

Quoted 7 lines

I'm not even sure that -z makes sense in this context. If we see cases where not using -z can cause the machine-readable interface to become unparseable then this is a bug in the output format, if you ask me. Mostly because the whole intent of it is to be machine-parseable. So if we output data that can e.g. contain newlines, then we must not use newlines as part of the output format or alternatively escape them. Why let the author of the script shoot themselves into the foot?

Agreed. I think the current output format is resilient enough, just that "-z" is something we choose to support for plumbing anyway.

Quoted 7 lines

Anyway, I'm digressing. It's hard for me to decide what to do right now. The thread with Junio and Jacob points into the direction of keeping the --output-format= interface, while this thread points into the other direction. I'm naturally more inclined to keep --output-format=, mostly because I personally feel like it's the more obvious interface. But I also see your point, so it's not really a choice of right-or-wrong here, but rather of style.

Yes. All sides seem to understand the tradeoffs, but we value them differently.

I'd personally prefer to err on the side of consistency, because even if the existing behavior is less-than-ideal since:

1) This makes it possible for us to fix it in a consistent, well-thought out way, so we don't have to decide the future for the whole project in a git-fetch series.
2) At least it still behaves how _some_ users have come to expect it.

As for which is more consistent, a grep for --porcelain over Documentation/* shows:

- git-push
- git-status
- git-worktree
- git-blame
- git-commit

Most of them already output to stdout. The lone exception is git-push, which does exactly this "use stdout for the main output, but keep using stderr for debugging output" behavior you added, so using --porcelain seems somewhat consistent.

On the other hand, I couldn't find any other option that switches from stderr to stdout (we sometimes say --stdout to say "use stdout instead of a file", but that's different), so if we added this behavior to "git fetch --[output-]format", this might be the first.

(Sidenote: most instances of --format are for templated-style format, but "git-replace" does accept an enum for it. I don't think this should weigh heavily in our decision though.)

For the other, non git-push commands, --porcelain is treated like an output format and respects "last one wins" regular option precedence, e.g. in the case of "git status --porcelain --short". I find this behavior somewhat confusing (I'd expect --porcelain to win, or at least be incompatible), but at least we can consistently change this in the future [*].

That said, I'm willing to accept that I'm biased toward my own ideas. If others don't find "--format=porcelain implies stdout" confusing, I can accept that decision.

[*] In all likelihood, we probably can't make the change to --porcelain for backwards-compatibility reasons, but we could introduce a synonym with the behavior we want ("--machine"?) and have OPT_PARSE_MACHINE handle both --porcelain and --machine.

#22 lore raw

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

In reply to Patrick Steinhardt
Quoted 15 lines · Patrick Steinhardt
Quoted 8 lines · Glen Choo
Quoted 5 lines · Patrick Steinhardt

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```
Quoted 5 lines

This format doesn't show the remote name or url that was fetched. That seems okay when fetching with a single remote, but it seems necessary with "--all". Perhaps you were planning to add that in a later series? If so, I think it's okay to call the "porcelain" format experimental, and forbid porcelain + --all until then.

Quoted 6 lines

The reason is mostly that I didn't find an output format that I really liked here. We'd basically have to repeat the remote URL for every single reference: just repeating it once per remote doesn't fly because with --parallel the output could be intermingled. But doing that feels wasteful to me, so I bailed. I guess I'm also biased here because it just wouldn't be useful to myself.

Couldn't each URL be assigned an index and then just show 0 in the output? I think it should be straightforward for a parser to figure out what 0 means but if not, that's something that could be figured out once the use case arrives (which very well could be never).

Signature · 1 line
Felipe Contreras
#23 lore raw
In reply to Patrick Steinhardt
Quoted 11 lines · Patrick Steinhardt

Parsing the output of fetches is mostly impossible. It prettifies reference names that are about to be updated, doesn't print the old and new object IDs the refs are being updated from and to, and prints all of that information in nice columns. In short, it is designed to be read by humans rather than machines.

This makes it hard to use in a script way though, e.g. to learn about which references actually have been updated or which have not been updated. This patch series intends to fix that by introducing a new machine-parseable interface:

Quoted 9 lines
```
$ git fetch --output-format=porcelain --no-progress
  fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
* 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
* 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
* 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
* 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
* 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
```

Makes sense, my only question is what other format could git fetch have? I think --format=porcelain is clear enough.

Signature · 1 line
Felipe Contreras
#24 lore raw
Quoted 22 lines · Felipe Contreras
Quoted 11 lines · Patrick Steinhardt

Parsing the output of fetches is mostly impossible. It prettifies reference names that are about to be updated, doesn't print the old and new object IDs the refs are being updated from and to, and prints all of that information in nice columns. In short, it is designed to be read by humans rather than machines.

This makes it hard to use in a script way though, e.g. to learn about which references actually have been updated or which have not been updated. This patch series intends to fix that by introducing a new machine-parseable interface:

Quoted 9 lines
```
$ git fetch --output-format=porcelain --no-progress
  fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
* 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
* 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
* 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
* 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
* 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
```
Quoted 2 lines

Makes sense, my only question is what other format could git fetch have? I think --format=porcelain is clear enough.

Yeah, I'd be perfectly happy to rename this to --format=porcelain. I'll wait for the Review Club that discusses this patch set tomorrow and will send a new version with that change afterwards if nobody disagrees.

Patrick

#25 lore raw
Quoted 23 lines · Patrick Steinhardt
Quoted 22 lines · Felipe Contreras
Quoted 20 lines · Patrick Steinhardt

Parsing the output of fetches is mostly impossible. It prettifies reference names that are about to be updated, doesn't print the old and new object IDs the refs are being updated from and to, and prints all of that information in nice columns. In short, it is designed to be read by humans rather than machines.

This makes it hard to use in a script way though, e.g. to learn about which references actually have been updated or which have not been updated. This patch series intends to fix that by introducing a new machine-parseable interface:

```
$ git fetch --output-format=porcelain --no-progress
  fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
* 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
* 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
* 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
* 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
* 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
```
Quoted 8 lines
Quoted 2 lines

Makes sense, my only question is what other format could git fetch have? I think --format=porcelain is clear enough.

Yeah, I'd be perfectly happy to rename this to --format=porcelain. I'll wait for the Review Club that discusses this patch set tomorrow and will send a new version with that change afterwards if nobody disagrees.

Patrick

We had some discussion during review club about this, where the idea of using "--porcelain" came up because many commands use that when switching into a machine readable format.

In addition, this format not only changes the output but also moves it from being on stderr to stdout, which is a hint that the intended usage of the command is now a little different.

I don't have strong opinion here but want to note that "--format" is often used by commands like log which changes how we structure the output of git objects.

Obviously using "--porcelain" is a bit weird when dealing with the pre-existing compact and full outputs, and perhaps --format wouldn't be confusing.

I didn't find any command which used --output-format today, and all the uses of --format I saw were for object formatting like git log.

I'm ok with --format, but wanted to note the potential confusion.

git status also uses --porcelain and has a -z option for NUL terminating instead of newline terminating. We thought -z might be useful to handle the potential of weird refnames that include characters. No one on the review could remember what rules were enforced on refnames to confirm if it was legal to have '\n' in a refname or not.

#26 lore raw
Quoted 6 lines · Jacob Keller
Quoted 5 lines

Yeah, I'd be perfectly happy to rename this to --format=porcelain. I'll wait for the Review Club that discusses this patch set tomorrow and will send a new version with that change afterwards if nobody disagrees.

Patrick

Quoted 7 lines

We had some discussion during review club about this, where the idea of using "--porcelain" came up because many commands use that when switching into a machine readable format.

In addition, this format not only changes the output but also moves it from being on stderr to stdout, which is a hint that the intended usage of the command is now a little different.

A little different from what? I do not think the answer would be "other program's --porcelain mode", as sending them to stdout would be one of the things that make the output easier for programs to parse, so it does sound like very much in the same spirit as "git status --porcelain" where its output format gets tweaked to be more machine friendly.

The output with "--porcelain" option enabled tend to be less human friendly and the distinction between Porcelain (for humans) and plumbing (for scripts) is reversed in the use of the word there---it started as "this is the option for those who write Porcelain commands to use", but still it is not a very good name for the option.

I am perfectly OK if the plan is to uniformly use --output-format (or something equally more descriptive) and migrate and deprecate the "--porcelain" option away from existing commands.

Thanks.

#27 lore raw
Quoted 16 lines · Junio C Hamano
Quoted 6 lines · Jacob Keller
Quoted 5 lines

Yeah, I'd be perfectly happy to rename this to --format=porcelain. I'll wait for the Review Club that discusses this patch set tomorrow and will send a new version with that change afterwards if nobody disagrees.

Patrick

Quoted 7 lines

We had some discussion during review club about this, where the idea of using "--porcelain" came up because many commands use that when switching into a machine readable format.

In addition, this format not only changes the output but also moves it from being on stderr to stdout, which is a hint that the intended usage of the command is now a little different.

Quoted 6 lines

A little different from what? I do not think the answer would be "other program's --porcelain mode", as sending them to stdout would be one of the things that make the output easier for programs to parse, so it does sound like very much in the same spirit as "git status --porcelain" where its output format gets tweaked to be more machine friendly.

A little different from using git fetch normally where all output is stderr and is generally "this is what I did" but which was obviously not intended to be parsed by scripts but instead by the human who ran the command.

Quoted 7 lines

The output with "--porcelain" option enabled tend to be less human friendly and the distinction between Porcelain (for humans) and plumbing (for scripts) is reversed in the use of the word there---it started as "this is the option for those who write Porcelain commands to use", but still it is not a very good name for the option.

Yea the option sort of means "to be used by those implementing porcelain" and its definitely a bit confusing.

Quoted 6 lines

I am perfectly OK if the plan is to uniformly use --output-format (or something equally more descriptive) and migrate and deprecate the "--porcelain" option away from existing commands.

Thanks.

That makes sense to me as well.

#28 lore raw
In reply to Junio C Hamano
Quoted 23 lines · Junio C Hamano
Quoted 6 lines · Jacob Keller
Quoted 5 lines

Yeah, I'd be perfectly happy to rename this to --format=porcelain. I'll wait for the Review Club that discusses this patch set tomorrow and will send a new version with that change afterwards if nobody disagrees.

Patrick

Quoted 7 lines

We had some discussion during review club about this, where the idea of using "--porcelain" came up because many commands use that when switching into a machine readable format.

In addition, this format not only changes the output but also moves it from being on stderr to stdout, which is a hint that the intended usage of the command is now a little different.

A little different from what? I do not think the answer would be "other program's --porcelain mode", as sending them to stdout would be one of the things that make the output easier for programs to parse, so it does sound like very much in the same spirit as "git status --porcelain" where its output format gets tweaked to be more machine friendly.

Quoted 6 lines

The output with "--porcelain" option enabled tend to be less human friendly and the distinction between Porcelain (for humans) and plumbing (for scripts) is reversed in the use of the word there---it started as "this is the option for those who write Porcelain commands to use", but still it is not a very good name for the option.

Ah, thanks for explaining where this reversed meaning actually comes from. I really wondered why we use "porcelain" in preexisting code to reflect machine-parseable interface, but that explanation does make sense.

Quoted 3 lines

I am perfectly OK if the plan is to uniformly use --output-format (or something equally more descriptive) and migrate and deprecate the "--porcelain" option away from existing commands.

I'm still not quite clear on where the consensus lies now. Personally, I think that both --format and --output-format work well and are a bit more descriptive than simply --porcelain, and wouldn't mind also migrating other binaries to use either of them.

Furthermore, I think that --[output-]format has the advantage that you don't need to handle priorities or mutual exclusion of different options that all apply to the reference format. To a user, it is not immediately obvious what git fetch --format=compact --porcelain would do, and which of both options ultimately get respected. But that's likely only true for future commands, because any migration would create the same kind of ambiguity for preexisting commands.

If we were to also migrate preexisting code to use --[output-]format then I'd argue that --output-format is likely the better name, mostly because it is less likely to be ambiguous compared to --format. The latter could e.g. easily confused with --object-format.

So I think I'll stick with --output-format for the time being.

Patrick

#29 lore raw
Quoted 9 lines · Patrick Steinhardt

On Wed, Apr 26, 2023 at 01:23:12PM -0700, Junio C Hamano wrote:
Furthermore, I think that --[output-]format has the advantage that you don't need to handle priorities or mutual exclusion of different options that all apply to the reference format. To a user, it is not immediately obvious what git fetch --format=compact --porcelain would do, and which of both options ultimately get respected. But that's likely only true for future commands, because any migration would create the same kind of ambiguity for preexisting commands.

Quoted 7 lines

If we were to also migrate preexisting code to use --[output-]format then I'd argue that --output-format is likely the better name, mostly because it is less likely to be ambiguous compared to --format. The latter could e.g. easily confused with --object-format.

So I think I'll stick with --output-format for the time being.

I agree. I think using --output-format and migrating existing commands that have --porcelain to use --output-format is good. I'm not sure whether to keep using --output-format=porcelain or whether to use a different term there.

I definitely think avoiding confusion with --format is good, and I don't think its too much of a burden to have the longer --output-format as the option name.

Thanks,
Jake

Quoted 1 line

Patrick

#30 lore raw
In reply to Junio C Hamano
Quoted 22 lines · Junio C Hamano
Quoted 7 lines

We had some discussion during review club about this, where the idea of using "--porcelain" came up because many commands use that when switching into a machine readable format.

In addition, this format not only changes the output but also moves it from being on stderr to stdout, which is a hint that the intended usage of the command is now a little different.

A little different from what? I do not think the answer would be "other program's --porcelain mode", as sending them to stdout would be one of the things that make the output easier for programs to parse, so it does sound like very much in the same spirit as "git status --porcelain" where its output format gets tweaked to be more machine friendly.

The output with "--porcelain" option enabled tend to be less human friendly and the distinction between Porcelain (for humans) and plumbing (for scripts) is reversed in the use of the word there---it started as "this is the option for those who write Porcelain commands to use", but still it is not a very good name for the option.

Quoted 3 lines

I am perfectly OK if the plan is to uniformly use --output-format (or something equally more descriptive) and migrate and deprecate the "--porcelain" option away from existing commands.

I agree that --porcelain is a confusing name that would be nice to deprecate, but I don't think --output-format captures all of the intent of "operate in a machine-friendly mode instead of a human-friendly one". Unfortunately, if we had picked --plumbing from the beginning, I doubt we would be having this discussion today :/

E.g. machines (Unix ones at least?) like to have output on stdout and to be able to request NUL-terminated output. It's unfortunate that if we don't piggyback onto --output-format, we run into option precedence problems (like Patrick mentioned), but I'd find it more confusing that --output-format=[porcelain|full|compact] don't behave the same way.

I don't think this puts us in a better spot with regards to option precedence either. Consider:

  git fetch --output-format=full -z <...>

The only way to respect both options is to have -z affect the human-readable output, which isn't the end of the world, but it seems unnecessary.

Perhaps something like --machine instead?

#31 lore raw
In reply to Jacob Keller
Quoted 6 lines · Jacob Keller
Quoted 5 lines

Yeah, I'd be perfectly happy to rename this to --format=porcelain. I'll wait for the Review Club that discusses this patch set tomorrow and will send a new version with that change afterwards if nobody disagrees.

Patrick

Quoted 7 lines

We had some discussion during review club about this, where the idea of using "--porcelain" came up because many commands use that when switching into a machine readable format.

In addition, this format not only changes the output but also moves it from being on stderr to stdout, which is a hint that the intended usage of the command is now a little different.

A little different from what? I do not think the answer would be "other program's --porcelain mode", as sending them to stdout would be one of the things that make the output easier for programs to parse, so it does sound like very much in the same spirit as "git status --porcelain" where its output format gets tweaked to be more machine friendly.

The output with "--porcelain" option enabled tend to be less human friendly and the distinction between Porcelain (for humans) and plumbing (for scripts) is reversed in the use of the word there---it started as "this is the option for those who write Porcelain commands to use", but still it is not a very good name for the option.

I am perfectly OK if the plan is to uniformly use --output-format (or something equally more descriptive) and migrate and deprecate the "--porcelain" option away from existing commands.

Thanks.

#32 lore raw
In reply to Patrick Steinhardt

Hi Patrick!

Thanks for the pleasant read! I thought this was a great topic for Review Club. It's too bad that we missed you, but we post all relevant feedback here anyway.

Nevertheless, if you'd like to see the meeting notes, you can find them at:

https://docs.google.com/document/d/14L8BAumGTpsXpjDY8VzZ4rRtpAjuGrFSRqn3stCuS_w/edit

Quoted 11 lines · Patrick Steinhardt

Parsing the output of fetches is mostly impossible. It prettifies reference names that are about to be updated, doesn't print the old and new object IDs the refs are being updated from and to, and prints all of that information in nice columns. In short, it is designed to be read by humans rather than machines.

This makes it hard to use in a script way though, e.g. to learn about which references actually have been updated or which have not been updated. This patch series intends to fix that by introducing a new machine-parseable interface:

Quoted 9 lines
```
$ git fetch --output-format=porcelain --no-progress
  fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
* 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
* 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
* 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
* 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
* 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
```

Having machine-parseable output seems like an obviously good goal to me. The finer points of the interface and output format are worth discussing. I'll do so in the patches themselves.

Overall, I found the series very well-structured and easy to follow along. Thanks.

#33 lore raw
Quoted 10 lines · Glen Choo

Hi Patrick!

Thanks for the pleasant read! I thought this was a great topic for Review Club. It's too bad that we missed you, but we post all relevant feedback here anyway.

Nevertheless, if you'd like to see the meeting notes, you can find them at:

https://docs.google.com/document/d/14L8BAumGTpsXpjDY8VzZ4rRtpAjuGrFSRqn3stCuS_w/edit

It seems like Jacob Keller and I have mostly the same comments (since we did both go to the same Review Club, after all :)), so feel free to respond to only Jacob's messages and ignore mine (where the messages overlap) - I'll catch up on those conversations.

#34 lore raw
In reply to Patrick Steinhardt
Quoted 11 lines · Patrick Steinhardt
```
$ git fetch --output-format=porcelain --no-progress
  fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
* 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
* 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
* 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
* 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
* 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
```

One thing that the standard output (maybe with --progress?) shows to stderr is which remote is being fetched and what URL was fetched.

This seems like useful data to include in a machine readable format. It wasn't clear if such output would still be made and whether it would go to stderr or stdout, nor whether that existing output was machine readable.

Obviously you can somewhat infer remotes based on the refs/remotes, but that doesn't include other refs like tags.

When fetching from multiple reports (--all or explicitly asking), I wonder if it makes sense to include some lines to differentiate where each block of updates for a given remote starts and ends?

Thanks,
Jake

#35 lore raw

v2 cover fetch: introduce machine-parseable output

In reply to Patrick Steinhardt

Hi,

this is the second version of my patch series that introduces a machine-parseable output.

Changes compared to v1:

- Patch 3/8: I've reworded the commit message to hopefully be more straight-forward. I've also amended tests so that we don't only test with --dry-run, but also without.

- Patch 6/8: I've also moved the all and multiple variables from their global scope into cmd__fetch.

- Patch 7/8: Fixed a bug where --output-format= wasn't honored for child processes when doing multi-remote fetches. Furthermore, I've unified parsing of the actual format so that we don't have to repeat it thrice.

- Patch 8/8: Added a note to the commit message that tries to argue why I didn't add remote information to the interface. I'm still open to change this if you disagree with my reasoning here.

Thanks a bunch for all the feedback received so far, really appreciate it!

Patrick

Patrick Steinhardt (8):
fetch: split out tests for output format
fetch: add a test to exercise invalid output formats
fetch: fix missing from-reference when fetching HEAD:foo
fetch: introduce display_format enum
fetch: move display format parsing into main function
fetch: move option related variables into main function
fetch: introduce new --output-format option
fetch: introduce machine-parseable "porcelain" output format

 Documentation/config/fetch.txt  |   4 +-
 Documentation/fetch-options.txt |   5 +
 Documentation/git-fetch.txt     |  17 +-
 builtin/fetch.c                 | 427 ++++++++++++++++++++------------
 t/t5510-fetch.sh                |  53 ----
 t/t5574-fetch-output.sh         | 237 ++++++++++++++++++
 6 files changed, 521 insertions(+), 222 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh

Range-diff against v1:

1:  0d0d50d14c = 1:  0d0d50d14c fetch: split out tests for output format
2:  29d2c58914 = 2:  29d2c58914 fetch: add a test to exercise invalid output formats
3:  596e12f03a ! 3:  d1fb6eeae7 fetch: fix missing from-reference when fetching HEAD:foo

@@ Metadata
## Commit message ##
fetch: fix missing from-reference when fetching HEAD:foo

- When displaying reference updates, we print a line that looks similar to
- the following:
+ store_updated_refs() parses the remote reference for two purposes:
+
+ - It gets used as a note when writing FETCH_HEAD.
+
+ - It is passed through to display_ref_update() to display
+ updated references in the following format:
+
+ ```
+ * branch master -> master
+ ```
+
+ In most cases, the parsed remote reference is the prettified reference
+ name and can thus be used for both cases. But if the remote reference is
+ HEAD, the parsed remote reference becomes empty. This is intended when
+ we write the FETCH_HEAD, where we skip writing the note in that case.
+ But it is not intended when displaying the updated references and would
+ cause us to miss the left-hand side of the displayed reference update:

```
- * branch master -> master
- ```
-
- The "branch" bit changes depending on what kind of reference we're
- updating, while both of the right-hand references are computed by
- stripping well-known prefixes like "refs/heads/" or "refs/tags".
-
- The logic is kind of intertwined though and not easy to follow: we
- precompute both the kind (e.g. "branch") and the what, which is the
- abbreviated remote reference name, in store_updated_refs() and then
- pass it down the call chain to display_ref_update().
-
- There is a set of different cases here:
-
- - When the remote reference name is "HEAD" we assume no kind and
- will thus instead print "[new ref]". We keep what at the empty
- string.
-
- - When the remote reference name has a well-known prefix then the
- kind would be "branch", "tag" or "remote-tracking branch". The
- what is the reference with the well-known prefix stripped and in
- fact matches the output that prettify_refname() would return.
-
- - Otherwise, we'll again assume no kind and keep the what set to the
- fully qualified reference name.
-
- Now there is a bug with the first case here, where the remote reference
- name is "HEAD". As noted, "what" will be set to the empty string. And
- that seems to be intentional because we also use this information to
- update the FETCH_HEAD, and in case we're updating HEAD we seemingly
- don't want to append that to our FETCH_HEAD value.
-
- But as mentioned, we also use this value to display reference updates.
- And while the call to display_ref_update() correctly figures out that
- we meant "HEAD" when what is empty, the call to update_local_ref()
- doesn't. update_local_ref() will then call display_ref_update() with
- the empty string and cause the following broken output:
-
- ```
- $ git fetch --dry-run origin HEAD:foo
+ $ git fetch origin HEAD:foo
From https://github.com/git/git
* [new ref] -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow,
- which is further stressed by the point that the following commands work
- as expected:
+ which is further stressed by the point that the following commands show
+ the left-hand side as expected:

```
- $ git fetch --dry-run origin HEAD
+ $ git fetch origin HEAD
From https://github.com/git/git
* branch HEAD -> FETCH_HEAD

- $ git fetch --dry-run origin master
+ $ git fetch origin master
From https://github.com/git/git
* branch master -> FETCH_HEAD
* branch master -> origin/master
```

- Fix this bug by instead unconditionally passing the full reference name
- to display_ref_update() which learns to call prettify_refname() on
- it. This does fix the above bug and is otherwise functionally the same
- as prettify_refname() would only ever strip the well-known prefixes
- just as intended. So at the same time, this also simplifies the code a
- bit.
+ The logic of how we compute the remote reference name that we ultimately
+ pass to display_ref_update() is not easy to follow. There are three
+ different cases here:
+
+ - When the remote reference name is "HEAD" we set the remote
+ reference name to the empty string. This is the case that causes
+ the bug to occur, where we would indeed want to print "HEAD"
+ instead of the empty string. This is what prettify_refname()
+ would return.
+
+ - When the remote reference name has a well-known prefix then we
+ strip this prefix. This matches what prettify_refname() does.
+
+ - Otherwise, we keep the fully qualified reference name. This also
+ matches what prettify_refname() does.
+
+ As the return value of prettify_refname() would do the correct thing
+ for us in all three cases, we can fix the bug by passing through the
+ full remote reference name to display_ref_update(), which learns to
+ call prettify_refname(). At the same time, this also simplifies the
+ code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is
@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
+ EOF
+ test_cmp expect actual &&
+
++ git -C head fetch origin HEAD >actual 2>&1 &&
++ test_cmp expect actual &&
++
+ git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
+ cat >expect <<-EOF &&
+ From $(test-tool path-utils real_path .)/.
+ * [new ref] HEAD -> foo
+ EOF
++ test_cmp expect actual &&
++
++ git -C head fetch origin HEAD:foo >actual 2>&1 &&
+ test_cmp expect actual
+'
+

4:  8571363be1 = 4:  b545bf8bb9 fetch: introduce `display_format` enum
5:  d98c3ee0ce = 5:  4990d35998 fetch: move display format parsing into main function
6:  640a8840e1 ! 6:  cfe84129ab fetch: move option related variables into main function
    @@ Commit message
         Signed-off-by: Patrick Steinhardt <[email protected]>
     
      ## builtin/fetch.c ##
    -@@ builtin/fetch.c: static int all, append, dry_run, force, keep, multiple, update_head_ok;
    +@@ builtin/fetch.c: static int fetch_prune_tags_config = -1; /* unspecified */
    + static int prune_tags = -1; /* unspecified */
    + #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
    + 
    +-static int all, append, dry_run, force, keep, multiple, update_head_ok;
    ++static int append, dry_run, force, keep, update_head_ok;
      static int write_fetch_head = 1;
      static int verbosity, deepen_relative, set_upstream, refetch;
      static int progress = -1;
    @@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const cha
      	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
      	struct string_list list = STRING_LIST_INIT_DUP;
      	struct remote *remote = NULL;
    ++	int all = 0, multiple = 0;
      	int result = 0;
      	int prune_tags_ok = 1;
     +	int enable_auto_gc = 1;
7:  3b2cad066a ! 7:  0335e5eeb4 fetch: introduce new `--output-format` option
    @@ Commit message
         current mechanism feels a little bit indirect and rigid.
     
         Introduce a new `--output-format` option that allows the user to change
    -    the desired output format more directly.
    +    the desired format more directly.
     
         Signed-off-by: Patrick Steinhardt <[email protected]>
     
    @@ Documentation/fetch-options.txt: linkgit:git-config[1].
      	Write the list of remote refs fetched in the `FETCH_HEAD`
     
      ## builtin/fetch.c ##
    +@@ builtin/fetch.c: enum display_format {
    + 	DISPLAY_FORMAT_UNKNOWN = 0,
    + 	DISPLAY_FORMAT_FULL,
    + 	DISPLAY_FORMAT_COMPACT,
    ++	DISPLAY_FORMAT_MAX,
    ++};
    ++
    ++static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
    ++	NULL,
    ++	"full",
    ++	"compact",
    + };
    + 
    + struct display_state {
    +@@ builtin/fetch.c: static int fetch_finished(int result, struct strbuf *out,
    + 	return 0;
    + }
    + 
    +-static int fetch_multiple(struct string_list *list, int max_children)
    ++static int fetch_multiple(struct string_list *list, int max_children,
    ++			  enum display_format format)
    + {
    + 	int i, result = 0;
    + 	struct strvec argv = STRVEC_INIT;
    +@@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_children)
    + 		     "--no-write-commit-graph", NULL);
    + 	add_options_to_argv(&argv);
    + 
    ++	if (format != DISPLAY_FORMAT_UNKNOWN)
    ++		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
    ++
    + 	if (max_children != 1 && list->nr != 1) {
    + 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
    + 		const struct run_process_parallel_opts opts = {
     @@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv,
      	return exit_code;
      }
      
    ++static enum display_format parse_display_format(const char *format)
    ++{
    ++	for (int i = 0; i < ARRAY_SIZE(display_formats); i++)
    ++		if (display_formats[i] && !strcmp(display_formats[i], format))
    ++			return i;
    ++	return DISPLAY_FORMAT_UNKNOWN;
    ++}
    ++
     +static int opt_parse_output_format(const struct option *opt, const char *arg, int unset)
     +{
    -+	enum display_format *format = opt->value;
    ++	enum display_format *format = opt->value, parsed;
    ++
     +	if (unset || !arg)
     +		return 1;
    -+	else if (!strcmp(arg, "full"))
    -+		*format = DISPLAY_FORMAT_FULL;
    -+	else if (!strcmp(arg, "compact"))
    -+		*format = DISPLAY_FORMAT_COMPACT;
    -+	else
    ++
    ++	parsed = parse_display_format(arg);
    ++	if (parsed == DISPLAY_FORMAT_UNKNOWN)
     +		return error(_("unsupported output format '%s'"), arg);
    ++	*format = parsed;
     +
     +	return 0;
     +}
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
      			 N_("write fetched references to the FETCH_HEAD file")),
      		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 		const char *format = "full";
    + 
    + 		git_config_get_string_tmp("fetch.output", &format);
    +-		if (!strcasecmp(format, "full"))
    +-			display_format = DISPLAY_FORMAT_FULL;
    +-		else if (!strcasecmp(format, "compact"))
    +-			display_format = DISPLAY_FORMAT_COMPACT;
    +-		else
    ++
    ++		display_format = parse_display_format(format);
    ++		if (display_format == DISPLAY_FORMAT_UNKNOWN)
    + 			die(_("invalid value for '%s': '%s'"),
    + 			    "fetch.output", format);
    + 	}
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 			max_children = fetch_parallel_config;
    + 
    + 		/* TODO should this also die if we have a previous partial-clone? */
    +-		result = fetch_multiple(&list, max_children);
    ++		result = fetch_multiple(&list, max_children, display_format);
    + 	}
    + 
    + 
     
      ## t/t5574-fetch-output.sh ##
     @@ t/t5574-fetch-output.sh: test_expect_success 'fetch with invalid output format configuration' '
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch aligned output' '
      	cat >expect <<-\EOF &&
      	main       -> origin/*
      	extraaa    -> *
    +@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
    + 	test_cmp expect actual
    + '
    + 
    ++test_expect_success 'fetch compact output with multiple remotes' '
    ++	test_when_finished "rm -rf compact-cfg compact-cli" &&
    ++
    ++	git clone . compact-cli &&
    ++	git -C compact-cli remote add second-remote "$PWD" &&
    ++	git clone . compact-cfg &&
    ++	git -C compact-cfg remote add second-remote "$PWD" &&
    ++	test_commit multi-commit &&
    ++
    ++	git -C compact-cfg -c fetch.output=compact fetch --all >actual-cfg 2>&1 &&
    ++	git -C compact-cli fetch --output-format=compact --all >actual-cli 2>&1 &&
    ++	test_cmp actual-cfg actual-cli &&
    ++
    ++	grep -e "->" actual-cfg | cut -c 22- >actual &&
    ++	cat >expect <<-\EOF &&
    ++	main         -> origin/*
    ++	multi-commit -> *
    ++	main       -> second-remote/*
    ++	EOF
    ++	test_cmp expect actual
    ++'
    ++
    + test_expect_success 'fetch output with HEAD and --dry-run' '
    + 	test_when_finished "rm -rf head" &&
    + 	git clone . head &&

8: 301138da03 ! 8: d7c1bc1a80 fetch: introduce machine-parseable "porcelain" output format
@@ Commit message
so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

+ A notable ommission here is that the output format does not include the
+ remote from which a reference was fetched, which might be important
+ information especially in the context of multi-remote fetches. But as
+ such a format would require us to print the remote for every single
+ reference update due to parallelizable fetches it feels wasteful for the
+ most likely usecase, which is when fetching from a single remote. If
+ usecases come up for this in the future though it is easy enough to add
+ a new "porcelain-v2" format that adds this information.
+

## Documentation/config/fetch.txt ##
@@ builtin/fetch.c: enum display_format {
DISPLAY_FORMAT_FULL,
DISPLAY_FORMAT_COMPACT,
+ DISPLAY_FORMAT_PORCELAIN,
+ DISPLAY_FORMAT_MAX,
+ };
+
+@@ builtin/fetch.c: static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
+ NULL,
+ "full",
+ "compact",
++ "porcelain",
};

struct display_state {
@@ builtin/fetch.c: static int prune_refs(struct display_state *display_state,
summary_width);
warn_dangling_symref(stderr, dangling_msg, ref->name);
}
-@@ builtin/fetch.c: static int opt_parse_output_format(const struct option *opt, const char *arg, in
- *format = DISPLAY_FORMAT_FULL;
- else if (!strcmp(arg, "compact"))
- *format = DISPLAY_FORMAT_COMPACT;
-+ else if (!strcmp(arg, "porcelain"))
-+ *format = DISPLAY_FORMAT_PORCELAIN;
- else
- return error(_("unsupported output format '%s'"), arg);
-
-@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
- display_format = DISPLAY_FORMAT_FULL;
- else if (!strcasecmp(format, "compact"))
- display_format = DISPLAY_FORMAT_COMPACT;
-+ else if (!strcasecmp(format, "porcelain"))
-+ display_format = DISPLAY_FORMAT_PORCELAIN;
- else
- die(_("invalid value for '%s': '%s'"),
- "fetch.output", format);

## t/t5574-fetch-output.sh ##
-@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
+@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output with multiple remotes' '
test_cmp expect actual
'

Signature · 1 line
2.40.1
#36 lore raw

PATCH v2 1/8 fetch: split out tests for output format

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
t/t5510-fetch.sh +0 -53
index dc44da9c79..4f289063ce 100755
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
t/t5574-fetch-output.sh new file +63 -0

63 lines not on this page · raw message

Signature · 1 line
2.40.1
#37 lore raw

Re: [PATCH v2 1/8] fetch: split out tests for output format

Quoted 29 lines · Patrick Steinhardt

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
t/t5574-fetch-output.sh new file +5 -0
new file mode 100755
index 0000000000..f91b654d38
@@ -0,0 +1,63 @@
+#!/bin/sh
+
+test_description='git fetch output format'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main

t/t5574-fetch-output.sh line 15

Quoted 10 lines
t/t5574-fetch-output.sh new file +10 -0
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+test_expect_success 'fetch aligned output' '
+	git clone . full-output &&
+	test_commit looooooooooooong-tag &&
+	(
+		cd full-output &&
+		git -c fetch.output=full fetch origin >actual 2>&1 &&

Why is that 2>&1 redirection used here?
If the output the test case is interested in goes to the command's standard output, then it's unnecessary. However, if it goes to standard error, then why is standard output redirected as well?

I understand that this patch just moves existing test cases around as-is, so this is not something you introduced, but I point it out here, because later patches of this series add several new test cases following this anti-pattern.

Since this series creates a new test script, perhaps this might be the right time to clean this up.

Quoted 28 lines
t/t5574-fetch-output.sh new file +28 -0
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main                 -> origin/main
+	looooooooooooong-tag -> looooooooooooong-tag
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'fetch compact output' '
+	git clone . compact &&
+	test_commit extraaa &&
+	(
+		cd compact &&
+		git -c fetch.output=compact fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main       -> origin/*
+	extraaa    -> *
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success '--no-show-forced-updates' '
+	mkdir forced-updates &&
+	(
+		cd forced-updates &&

t/t5574-fetch-output.sh line 53

Quoted 10 lines
t/t5574-fetch-output.sh new file +10 -0
+		git init &&
+		test_commit 1 &&
+		test_commit 2
+	) &&
+	git clone forced-updates forced-update-clone &&
+	git clone forced-updates no-forced-update-clone &&
+	git -C forced-updates reset --hard HEAD~1 &&
+	(
+		cd forced-update-clone &&
+		git fetch --show-forced-updates origin 2>output &&

Oh, look, there are some good examples to follow here as well :)

Quoted 13 lines
t/t5574-fetch-output.sh new file +10 -0
+		test_i18ngrep "(forced update)" output
+	) &&
+	(
+		cd no-forced-update-clone &&
+		git fetch --no-show-forced-updates origin 2>output &&
+		test_i18ngrep ! "(forced update)" output
+	)
+'
+
+test_done
Signature · 1 line
2.40.1
#38 lore raw

Re: [PATCH v2 1/8] fetch: split out tests for output format

Quoted 46 lines · SZEDER Gábor
Quoted 29 lines · Patrick Steinhardt

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
t/t5574-fetch-output.sh new file +5 -0
new file mode 100755
index 0000000000..f91b654d38
@@ -0,0 +1,63 @@
+#!/bin/sh
+
+test_description='git fetch output format'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main

t/t5574-fetch-output.sh line 15

Quoted 10 lines
t/t5574-fetch-output.sh new file +10 -0
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+test_expect_success 'fetch aligned output' '
+	git clone . full-output &&
+	test_commit looooooooooooong-tag &&
+	(
+		cd full-output &&
+		git -c fetch.output=full fetch origin >actual 2>&1 &&

Why is that 2>&1 redirection used here?
If the output the test case is interested in goes to the command's standard output, then it's unnecessary. However, if it goes to standard error, then why is standard output redirected as well?

Quoted 7 lines

I understand that this patch just moves existing test cases around as-is, so this is not something you introduced, but I point it out here, because later patches of this series add several new test cases following this anti-pattern.

Since this series creates a new test script, perhaps this might be the right time to clean this up.

I feel like this patch series is big enough on its own already, so for now I'd like to refrain from touching up the old tests. But you've got a good point here, and I'll make sure that any new tests don't followw the same pattern.

That being said, I think we should keep on testing both stdout and stderr. The tests here are explicitly about the output format, so it does make sense to verify that both of them match what we expect. We should treat them as separate things though.

Patrick

#39 lore raw

PATCH v2 2/8 fetch: add a test to exercise invalid output formats

In reply to Patrick Steinhardt

Add a testcase that exercises the logic when an invalid output format is passed via the fetch.output configuration.

---
 t/t5574-fetch-output.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
t/t5574-fetch-output.sh +17 -0
index f91b654d38..0e45c27007 100755
@@ -7,6 +7,23 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_cmp expect actual &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
Signature · 1 line
2.40.1
#40 lore raw

PATCH v2 3/8 fetch: fix missing from-reference when fetching HEAD:foo

In reply to Patrick Steinhardt

store_updated_refs() parses the remote reference for two purposes:

- It gets used as a note when writing FETCH_HEAD.

- It is passed through to display_ref_update() to display updated references in the following format:

```
* branch master -> master
```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

```
$ git fetch origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands show the left-hand side as expected:

```
$ git fetch origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

$ git fetch origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

The logic of how we compute the remote reference name that we ultimately pass to display_ref_update() is not easy to follow. There are three different cases here:

- When the remote reference name is "HEAD" we set the remote reference name to the empty string. This is the case that causes the bug to occur, where we would indeed want to print "HEAD" instead of the empty string. This is what prettify_refname() would return.

- When the remote reference name has a well-known prefix then we strip this prefix. This matches what prettify_refname() does.

- Otherwise, we keep the fully qualified reference name. This also matches what prettify_refname() does.

As the return value of prettify_refname() would do the correct thing for us in all three cases, we can fix the bug by passing through the full remote reference name to display_ref_update(), which learns to call prettify_refname(). At the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

---
 builtin/fetch.c         | 37 +++++++++++++++++++------------------
 t/t5574-fetch-output.sh | 25 +++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 18 deletions(-)
builtin/fetch.c +19 -18
index c310d89878..7c64f0c562 100644
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
t/t5574-fetch-output.sh +25 -0
index 0e45c27007..b9dcdade63 100755
@@ -54,6 +54,31 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch origin HEAD >actual 2>&1 &&
+	test_cmp expect actual &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch origin HEAD:foo >actual 2>&1 &&
+	test_cmp expect actual
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
Signature · 1 line
2.40.1
#41 lore raw

Re: [PATCH v2 3/8] fetch: fix missing from-reference when fetching HEAD:foo

For v2, I've only read through this patch for now. I'm still formulating my thoughts on the output format and the CLI option name.

Quoted 18 lines · Patrick Steinhardt

store_updated_refs() parses the remote reference for two purposes:

- It gets used as a note when writing FETCH_HEAD.

- It is passed through to display_ref_update() to display updated references in the following format:

```
* branch master -> master
```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

Quoted 7 lines

[...]

As the return value of prettify_refname() would do the correct thing for us in all three cases, we can fix the bug by passing through the full remote reference name to display_ref_update(), which learns to call prettify_refname(). At the same time, this also simplifies the code a bit.

This rewritten section explains the bug and the fix very nicely :) Thanks!

Quoted 11 lines
t/t5574-fetch-output.sh +3 -0
index 0e45c27007..b9dcdade63 100755
@@ -54,6 +54,31 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&

t/t5574-fetch-output.sh test_expect_success 'fetch compact output' ' · line 69

Quoted 10 lines
t/t5574-fetch-output.sh +10 -0
+
+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch origin HEAD >actual 2>&1 &&
+	test_cmp expect actual &&

Maybe rename the test to just 'fetch output with HEAD'? The 'and --dry-run' part seems completely obsolete.

#42 lore raw

Re: [PATCH v2 3/8] fetch: fix missing from-reference when fetching HEAD:foo

In reply to Patrick Steinhardt

t/t5574-fetch-output.sh test_expect_success 'fetch compact output' ' · line 58

Quoted 9 lines · Patrick Steinhardt

index 0e45c27007..b9dcdade63 100755

t/t5574-fetch-output.sh +2 -0
@@ -54,6 +54,31 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&

The test name could drop the --dry-run mention since this now tests both variants.

Everything else looks good on this patch. The new commit message is much better to me now!

#43 lore raw

PATCH v2 4/8 fetch: introduce `display_format` enum

In reply to Patrick Steinhardt

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a enum display_format that is more readily extensible.

---
 builtin/fetch.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
builtin/fetch.c +64 -37
index 7c64f0c562..e03fcd1b2f 100644
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -784,7 +790,6 @@ static int refcol_width(const struct ref *ref, int compact_format)
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display foramt %d", display_state->format);
 	}
 }
 
@@ -904,30 +920,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
Signature · 1 line
2.40.1
#44 lore raw

PATCH v2 5/8 fetch: move display format parsing into main function

In reply to Patrick Steinhardt

Parsing the display format happens inside of display_state_init(). As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the fetch.output config key set. We're thus going to introduce a new --output-format switch for git-fetch(1) so that the output format can be configured more directly.

This means we'll have to hook parsing of the display format into the command line options parser. Having the code to determine the actual output format scattered across two different sites is hard to reason about though.

Refactor the code such that callers are expected to pass the display format that is to be used into display_state_init(). This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

---
 builtin/fetch.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
builtin/fetch.c +24 -17
index e03fcd1b2f..bcc156a9ce 100644
@@ -788,14 +788,13 @@ static int refcol_width(const struct ref *ref, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -812,15 +811,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT: {
@@ -1614,7 +1604,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1700,7 +1691,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2076,7 +2067,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2143,7 +2135,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2155,6 +2147,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2181,6 +2174,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}
+
 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
 		recurse_submodules = recurse_submodules_cli;
 
@@ -2309,7 +2315,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   display_format);
 	} else {
 		int max_children = max_jobs;
 
Signature · 1 line
2.40.1
#45 lore raw

PATCH v2 6/8 fetch: move option related variables into main function

In reply to Patrick Steinhardt

The options of git-fetch(1) which we pass to parse_options() are declared globally in builtin/fetch.c. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into cmd_fetch(). Move variables that were previously forced to be declared globally and which are only used by cmd_fetch() into function-local scope.

---
 builtin/fetch.c | 197 ++++++++++++++++++++++++------------------------
 1 file changed, 100 insertions(+), 97 deletions(-)
builtin/fetch.c +100 -97
index bcc156a9ce..97a510649c 100644
@@ -75,13 +75,12 @@ static int fetch_prune_tags_config = -1; /* unspecified */
 static int prune_tags = -1; /* unspecified */
 #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok;
+static int append, dry_run, force, keep, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
@@ -160,92 +153,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2145,13 +2052,109 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
-	int i;
 	const char *bundle_uri;
+	const char *submodule_prefix = "";
 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
+	int all = 0, multiple = 0;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
Signature · 1 line
2.40.1
#46 lore raw

Re: [PATCH v2 6/8] fetch: move option related variables into main function

Quoted 8 lines · Patrick Steinhardt

The options of git-fetch(1) which we pass to parse_options() are declared globally in builtin/fetch.c. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into cmd_fetch(). Move variables that were previously forced to be declared globally and which are only used by cmd_fetch() into function-local scope.

Very well done.

#47 lore raw

PATCH v2 7/8 fetch: introduce new `--output-format` option

In reply to Patrick Steinhardt

It is only possible to configure the output format that git-fetch(1) uses by setting it via a config key. While this interface may be fine as long as we only have the current "full" and "compact" output formats, where it is unlikely that the user will have to change them regularly. But we're about to introduce a new machine-parseable interface where the current mechanism feels a little bit indirect and rigid.

Introduce a new --output-format option that allows the user to change the desired format more directly.

---
 Documentation/fetch-options.txt |  5 +++
 builtin/fetch.c                 | 48 ++++++++++++++++++----
 t/t5574-fetch-output.sh         | 72 +++++++++++++++++++++++++++------
 3 files changed, 106 insertions(+), 19 deletions(-)
Documentation/fetch-options.txt +5 -0
index 622bd84768..654f96f79d 100644
@@ -78,6 +78,11 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--output-format::
+	Control how ref update status is printed. Valid values are
+	`full` and `compact`. Default value is `full`. See section
+	OUTPUT in linkgit:git-fetch[1] for detail.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
builtin/fetch.c +41 -7

90 lines not on this page · raw message

t/t5574-fetch-output.sh +60 -12

93 lines not on this page · raw message

Signature · 1 line
2.40.1
#48 lore raw

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

Quoted 6 lines · Patrick Steinhardt
builtin/fetch.c +0 -0
index 97a510649c..30099b2ac3 100644
@@ -52,6 +52,13 @@ enum display_format {
 	DISPLAY_FORMAT_UNKNOWN = 0,

builtin/fetch.c enum display_format { · line 62

Quoted 10 lines
builtin/fetch.c +7 -0
 	DISPLAY_FORMAT_FULL,
 	DISPLAY_FORMAT_COMPACT,
+	DISPLAY_FORMAT_MAX,
+};
+
+static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
+	NULL,
+	"full",
+	"compact",
 };

Hmph, the _MAX thing that is only needed to size the array and never used elsewhere (i.e. parse_display_format() uses ARRAY_SIZE() of the thing, instead of the constant, and that is just fine) is an eyesore.

I wonder if

	static const char *const display_format[] = {
		[DISPLAY_FORMAT_UNKNOWN] = NULL,
		[DISPLAY_FORMAT_FULL] = "full",
		[DISPLAY_FORMAT_COMPACT] = "compact",
	};

would be easier to maintain?

I'll omit my usual "name your array in singular" lecture, as I think you've heard it already.

Thanks.

#49 lore raw

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

Quoted 9 lines · Junio C Hamano

I wonder if

	static const char *const display_format[] = {
		[DISPLAY_FORMAT_UNKNOWN] = NULL,
		[DISPLAY_FORMAT_FULL] = "full",
		[DISPLAY_FORMAT_COMPACT] = "compact",
	};

would be easier to maintain?

It's easier to read, so I'd think so.

#50 lore raw

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

Quoted 10 lines · Glen Choo
Quoted 8 lines · Junio C Hamano

I wonder if

	static const char *const display_format[] = {
		[DISPLAY_FORMAT_UNKNOWN] = NULL,
		[DISPLAY_FORMAT_FULL] = "full",
		[DISPLAY_FORMAT_COMPACT] = "compact",
	};
Quoted 3 lines
Quoted 1 line

would be easier to maintain?

It's easier to read, so I'd think so.

Yeah, I'll adopt this approach in v3.

Patrick

#51 lore raw

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

In reply to Patrick Steinhardt

builtin/fetch.c static int fetch_multiple(struct string_list *list, int max_children) · line 1907

Quoted 7 lines · Patrick Steinhardt
builtin/fetch.c +3 -0
@@ -1894,6 +1902,9 @@ static int fetch_multiple(struct string_list *list, int max_children)
 		     "--no-write-commit-graph", NULL);
 	add_options_to_argv(&argv);
 
+	if (format != DISPLAY_FORMAT_UNKNOWN)
+		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
+

I think these lines belong inside add_options_to_argv(), since that's also used to prepare argv for fetch_submodules(), so we'd also get support for --recurse-submodules. (I wish I had spotted that in v1, sorry. Thankfully they use the same helper function, so we only have to do this once.)

----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 422e29a914..7aa385aed5 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1796,8 +1796,11 @@ static int add_remote_or_group(const char *name, struct string_list *list)

    return 1;

}

  -static void add_options_to_argv(struct strvec *argv)
  +static void add_options_to_argv(struct strvec *argv,
  +				enum display_format format)
  {
  /* Maybe this shouldn't be first, idk */
  +	if (format != DISPLAY_FORMAT_UNKNOWN)
  +		strvec_pushf(argv, "--output-format=%s", display_formats[format]);
    if (dry_run)
      strvec_push(argv, "--dry-run");
    if (prune != -1)
  @@ -1908,10 +1911,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
    strvec_pushl(&argv, "-c", "fetch.bundleURI=",
          "fetch", "--append", "--no-auto-gc",
          "--no-write-commit-graph", NULL);
  -	add_options_to_argv(&argv);
  -
  -	if (format != DISPLAY_FORMAT_UNKNOWN)
  -		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
  +	add_options_to_argv(&argv, format);
    if (max_children != 1 && list->nr != 1) {
      struct parallel_fetch_state state = { argv.v, list, 0, 0 };

@@ -2403,7 +2403,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
if (max_children < 0)
max_children = fetch_parallel_config;

  -		add_options_to_argv(&options);
  +		add_options_to_argv(&options, display_format);
      result = fetch_submodules(the_repository,
              &options,
              submodule_prefix,

----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----

I tested the result of that locally with --recurse-submodules, and it works.

#52 lore raw

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

Quoted 57 lines · Glen Choo

builtin/fetch.c static int fetch_multiple(struct string_list *list, int max_children) · line 1907

Quoted 7 lines · Patrick Steinhardt
builtin/fetch.c +3 -0
@@ -1894,6 +1902,9 @@ static int fetch_multiple(struct string_list *list, int max_children)
 		     "--no-write-commit-graph", NULL);
 	add_options_to_argv(&argv);
 
+	if (format != DISPLAY_FORMAT_UNKNOWN)
+		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
+

I think these lines belong inside add_options_to_argv(), since that's also used to prepare argv for fetch_submodules(), so we'd also get support for --recurse-submodules. (I wish I had spotted that in v1, sorry. Thankfully they use the same helper function, so we only have to do this once.)

----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 422e29a914..7aa385aed5 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1796,8 +1796,11 @@ static int add_remote_or_group(const char *name, struct string_list *list)

    return 1;

}

  -static void add_options_to_argv(struct strvec *argv)
  +static void add_options_to_argv(struct strvec *argv,
  +				enum display_format format)
  {
  /* Maybe this shouldn't be first, idk */
  +	if (format != DISPLAY_FORMAT_UNKNOWN)
  +		strvec_pushf(argv, "--output-format=%s", display_formats[format]);
    if (dry_run)
      strvec_push(argv, "--dry-run");
    if (prune != -1)
  @@ -1908,10 +1911,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
    strvec_pushl(&argv, "-c", "fetch.bundleURI=",
          "fetch", "--append", "--no-auto-gc",
          "--no-write-commit-graph", NULL);
  -	add_options_to_argv(&argv);
  -
  -	if (format != DISPLAY_FORMAT_UNKNOWN)
  -		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
  +	add_options_to_argv(&argv, format);
    if (max_children != 1 && list->nr != 1) {
      struct parallel_fetch_state state = { argv.v, list, 0, 0 };

@@ -2403,7 +2403,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
if (max_children < 0)
max_children = fetch_parallel_config;

  -		add_options_to_argv(&options);
  +		add_options_to_argv(&options, display_format);
      result = fetch_submodules(the_repository,
              &options,
              submodule_prefix,
Quoted 4 lines

----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----

I tested the result of that locally with --recurse-submodules, and it works.

Unfortunately it doesn't quite work alright: while the porcelain format does indeed get inherited to the child process correctly, the parallel process API will cause us to group output per submodule-fetch. This has the consequence that stdout will be redirected into stderr, and that then breaks the assumption that all machine-parseable output goes to stdout.

My initial reflex is to just outright reject porcelain mode when submodule fetches are enabled. But that would require the caller to always explicitly pass --recurse-submodules=off, which isn't exactly great usability-wise.

The alternative would be to ungroup the output so that we can continue to print to the correct output streams. That works alright, and I've got a working version that does exactly that. But now we have the issue that the porcelain output is misleading: you cannot tell whether a specific reference update happens in the parent repository or in the submodule as that information is not part of the output.

I consider the second option to be much worse than the first option because it can cause scripts do to the wrong thing. So I'll send v3 with the first option, even though it's kind of an awful workaround. I'd be happy to hear any alternative proposals though.

Patrick

#53 lore raw

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

Quoted 82 lines · Patrick Steinhardt
Quoted 57 lines · Glen Choo

builtin/fetch.c static int fetch_multiple(struct string_list *list, int max_children) · line 1907

Quoted 7 lines · Patrick Steinhardt
builtin/fetch.c +3 -0
@@ -1894,6 +1902,9 @@ static int fetch_multiple(struct string_list *list, int max_children)
 		     "--no-write-commit-graph", NULL);
 	add_options_to_argv(&argv);
 
+	if (format != DISPLAY_FORMAT_UNKNOWN)
+		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
+

I think these lines belong inside add_options_to_argv(), since that's also used to prepare argv for fetch_submodules(), so we'd also get support for --recurse-submodules. (I wish I had spotted that in v1, sorry. Thankfully they use the same helper function, so we only have to do this once.)

----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 422e29a914..7aa385aed5 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1796,8 +1796,11 @@ static int add_remote_or_group(const char *name, struct string_list *list)

    return 1;

}

  -static void add_options_to_argv(struct strvec *argv)
  +static void add_options_to_argv(struct strvec *argv,
  +				enum display_format format)
  {
  /* Maybe this shouldn't be first, idk */
  +	if (format != DISPLAY_FORMAT_UNKNOWN)
  +		strvec_pushf(argv, "--output-format=%s", display_formats[format]);
    if (dry_run)
      strvec_push(argv, "--dry-run");
    if (prune != -1)
  @@ -1908,10 +1911,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
    strvec_pushl(&argv, "-c", "fetch.bundleURI=",
          "fetch", "--append", "--no-auto-gc",
          "--no-write-commit-graph", NULL);
  -	add_options_to_argv(&argv);
  -
  -	if (format != DISPLAY_FORMAT_UNKNOWN)
  -		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
  +	add_options_to_argv(&argv, format);
    if (max_children != 1 && list->nr != 1) {
      struct parallel_fetch_state state = { argv.v, list, 0, 0 };

@@ -2403,7 +2403,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
if (max_children < 0)
max_children = fetch_parallel_config;

  -		add_options_to_argv(&options);
  +		add_options_to_argv(&options, display_format);
      result = fetch_submodules(the_repository,
              &options,
              submodule_prefix,
Quoted 4 lines

----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----

I tested the result of that locally with --recurse-submodules, and it works.

Unfortunately it doesn't quite work alright: while the porcelain format does indeed get inherited to the child process correctly, the parallel process API will cause us to group output per submodule-fetch. This has the consequence that stdout will be redirected into stderr, and that then breaks the assumption that all machine-parseable output goes to stdout.

My initial reflex is to just outright reject porcelain mode when submodule fetches are enabled. But that would require the caller to always explicitly pass --recurse-submodules=off, which isn't exactly great usability-wise.

The alternative would be to ungroup the output so that we can continue to print to the correct output streams. That works alright, and I've got a working version that does exactly that. But now we have the issue that the porcelain output is misleading: you cannot tell whether a specific reference update happens in the parent repository or in the submodule as that information is not part of the output.

Quoted 6 lines

I consider the second option to be much worse than the first option because it can cause scripts do to the wrong thing. So I'll send v3 with the first option, even though it's kind of an awful workaround. I'd be happy to hear any alternative proposals though.

Patrick

I've gone with a slightly different variant of the first option that is inspired by --negotiate-only: instead of refusing to run, we disable submodule-fetches unless explicitly specified on the command line. In that case we return an error.

Patrick

#54 lore raw

PATCH v2 8/8 fetch: introduce machine-parseable "porcelain" output format

In reply to Patrick Steinhardt

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```

We assume two conditions which are generally true:

- The old and new object IDs have fixed known widths and cannot contain spaces.

- References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote. If usecases come up for this in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

---
 Documentation/config/fetch.txt  |  4 +-
 Documentation/fetch-options.txt |  4 +-
 Documentation/git-fetch.txt     | 17 ++++++-
 builtin/fetch.c                 | 47 +++++++++++++-----
 t/t5574-fetch-output.sh         | 84 +++++++++++++++++++++++++++++++++
 5 files changed, 139 insertions(+), 17 deletions(-)
Documentation/config/fetch.txt +2 -2
index 568f0f75b3..70734226c0 100644
@@ -52,8 +52,8 @@ fetch.pruneTags::
 
 fetch.output::
 	Control how ref update status is printed. Valid values are
-	`full` and `compact`. Default value is `full`. See section
-	OUTPUT in linkgit:git-fetch[1] for detail.
+	`full`, `compact` and `porcelain`. Default value is `full`.
+	See section OUTPUT in linkgit:git-fetch[1] for detail.
 
 fetch.negotiationAlgorithm::
 	Control how information about the commits in the local repository
Documentation/fetch-options.txt +2 -2
index 654f96f79d..5ca8a67fe8 100644
@@ -80,8 +80,8 @@ linkgit:git-config[1].
 
 --output-format::
 	Control how ref update status is printed. Valid values are
-	`full` and `compact`. Default value is `full`. See section
-	OUTPUT in linkgit:git-fetch[1] for detail.
+	`full`, `compact` and `porcelain`. Default value is `full`.
+	See section OUTPUT in linkgit:git-fetch[1] for detail.
 
 ifndef::git-pull[]
 --[no-]write-fetch-head::
Documentation/git-fetch.txt +15 -2
index fba66f1460..efd22cd372 100644
@@ -197,13 +197,26 @@ The output of "git fetch" depends on the transport method used; this
 section describes the output when fetching over the Git protocol
 (either locally or via ssh) and Smart HTTP protocol.
 
-The status of the fetch is output in tabular form, with each line
-representing the status of a single ref. Each line is of the form:
+The output format can be chosen either via the `fetch.output` config
+(see linkgit:git-config[1]), or via the `--output-format` switch.
+Supported values include:
+
+For the `full` and `compact` output formats, the status of the fetch is
+output in tabular, with each line representing the status of a single
+ref. Each line is of the form:
 
 -------------------------------
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+The `porcelain` output format is intended to be machine-parseable. In
+contrast to the human-readable output formats it thus prints to standard
+output instead of standard error. Each line is of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
builtin/fetch.c +36 -11

141 lines not on this page · raw message

t/t5574-fetch-output.sh +84 -0

96 lines not on this page · raw message

Signature · 1 line
2.40.1
#55 lore raw

Re: [PATCH v2 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 8 lines · Patrick Steinhardt

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote. If usecases come up for this in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

Thanks for clarifying this. I think generally its ok to leave out since you can often infer what remote a ref came from (either because of refs/remotes, or because of how you configured your refspecs for fetching).

We do have the option to extend this in the future if needed, so I think that's ok. In that case if we wanted an even more verbose output I'd probably argue for something more structured like adding it as part of a JSON output.

Thanks,
Jake

#56 lore raw

Re: [PATCH v2 8/8] fetch: introduce machine-parseable "porcelain" output format

In reply to Patrick Steinhardt
Quoted 8 lines · Patrick Steinhardt

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote. If usecases come up for this in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

We discussed this elsewhere in the thread, but if we are just adding information (and not omitting existing information or shuffling it around), I would prefer for us to make the format extensible using flags than to add a whole new format enum. We can't imagine what other information users might want (maybe the remote ref name?), and it would be nice to avoid bumping the 'porcelain version' unnecessarily.

I agree that this format is good enough as a starting point, though.

It's not new to v2, but I've mentioned my reservations on --output-format on:

https://lore.kernel.org/git/[email protected]/

I will be out of office all of next week. If, in the meantime, everyone else decides that --output-format=porcelain is good enough, I'm happy to accept the result.

Quoted 12 lines
t/t5574-fetch-output.sh +8 -0
@@ -127,6 +193,24 @@ test_expect_success 'fetch output with HEAD and --dry-run' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD

t/t5574-fetch-output.sh test_expect_success 'fetch output with HEAD and --dry-run' ' · line 213

Quoted 10 lines
t/t5574-fetch-output.sh +10 -0
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual
+'
+

Now that the earlier test also tests without --dry-run, shouldn't this one too?

#57 lore raw

v3 cover fetch: introduce machine-parseable output

In reply to Patrick Steinhardt

Hi,

this is the third version of my patch series to introduce a machine-parseable output for git-fetch(1).

Changes compared to v2:

- A new commit that fixes --no-recurse-submodules not being honored for multi-remote fetches was added. This fixes issues that would otherwise be hit later in the patch series.

- Tests were adjusted to explicitly verify stdout and stderr as standalone units instead of redirecting 2>&1.

- Instead of introducing --output-format and a new "porcelain" format I've changed this to instead introduce --porcelain. What finally convinced me to go this way is Glen's list of preexisting commands that have --porcelain. So I decided to not be the only outlier here.

- The DISPLAY_FORMAT_MAX macro was dropped in favor of Junio's proposal that uses designated initializers for the array.

- Similar to --negotiate-only, the porcelain output is now incompatible with --recurse-submodules=[yes|on-demand]. This is done to avoid ambiguity when references in both the superproject and any of its submodules is updated.

Thanks again for all the feedback!

Patrick

Patrick Steinhardt (8):
fetch: fix --no-recurse-submodules with multi-remote fetches
fetch: split out tests for output format
fetch: add a test to exercise invalid output formats
fetch: fix missing from-reference when fetching HEAD:foo
fetch: introduce display_format enum
fetch: move display format parsing into main function
fetch: move option related variables into main function
fetch: introduce machine-parseable "porcelain" output format

 Documentation/fetch-options.txt |   6 +
 Documentation/git-fetch.txt     |   9 +
 builtin/fetch.c                 | 437 +++++++++++++++++++-------------
 t/t5510-fetch.sh                |  53 ----
 t/t5526-fetch-submodules.sh     |  31 +++
 t/t5574-fetch-output.sh         | 231 +++++++++++++++++
 6 files changed, 543 insertions(+), 224 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh

Range-diff against v2:

-:  ---------- > 1:  4b2b0cfe15 fetch: fix `--no-recurse-submodules` with multi-remote fetches
1:  0d0d50d14c = 2:  6ebc7450ba fetch: split out tests for output format
2:  29d2c58914 ! 3:  78479922ac fetch: add a test to exercise invalid output formats
    @@ t/t5574-fetch-output.sh: export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
     +	test_when_finished "rm -rf clone" &&
     +	git clone . clone &&
     +
    -+	test_must_fail git -C clone -c fetch.output= fetch origin >actual 2>&1 &&
    ++	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
     +	EOF
    -+	test_cmp expect actual &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
     +
    -+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual 2>&1 &&
    ++	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
     +	EOF
    -+	test_cmp expect actual
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err
     +'
     +
      test_expect_success 'fetch aligned output' '
3:  d1fb6eeae7 ! 4:  46e1266ab0 fetch: fix missing from-reference when fetching HEAD:foo
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
      	test_cmp expect actual
      '
      
    -+test_expect_success 'fetch output with HEAD and --dry-run' '
    ++test_expect_success 'fetch output with HEAD' '
     +	test_when_finished "rm -rf head" &&
     +	git clone . head &&
     +
    -+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
    ++	git -C head fetch --dry-run origin HEAD >actual.out 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	From $(test-tool path-utils real_path .)/.
     +	 * branch            HEAD       -> FETCH_HEAD
     +	EOF
    -+	test_cmp expect actual &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
     +
    -+	git -C head fetch origin HEAD >actual 2>&1 &&
    -+	test_cmp expect actual &&
    ++	git -C head fetch origin HEAD >actual.out 2>actual.err &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
     +
    -+	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
    ++	git -C head fetch --dry-run origin HEAD:foo >actual.out 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	From $(test-tool path-utils real_path .)/.
     +	 * [new ref]         HEAD       -> foo
     +	EOF
    -+	test_cmp expect actual &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
     +
    -+	git -C head fetch origin HEAD:foo >actual 2>&1 &&
    -+	test_cmp expect actual
    ++	git -C head fetch origin HEAD:foo >actual.out 2>actual.err &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err
     +'
     +
      test_expect_success '--no-show-forced-updates' '
4:  b545bf8bb9 ! 5:  acc0f7f520 fetch: introduce `display_format` enum
    @@ builtin/fetch.c: static void display_state_init(struct display_state *display_st
     +		break;
     +	}
     +	default:
    -+		BUG("unexpected display foramt %d", display_state->format);
    ++		BUG("unexpected display format %d", display_state->format);
      	}
      }
5:  4990d35998 = 6:  cd23440128 fetch: move display format parsing into main function
6:  cfe84129ab = 7:  edbc31013f fetch: move option related variables into main function
7:  0335e5eeb4 < -:  ---------- fetch: introduce new `--output-format` option
8:  d7c1bc1a80 ! 8:  e132d9494e fetch: introduce machine-parseable "porcelain" output format

@@ Commit message
information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the
- most likely usecase, which is when fetching from a single remote. If
- usecases come up for this in the future though it is easy enough to add
- a new "porcelain-v2" format that adds this information.
+ most likely usecase, which is when fetching from a single remote.
+
+ In a similar spirit, a second restriction is that this cannot be used
+ with --recurse-submodules. This is because any reference updates would
+ be ambiguous without also printing the repository in which the update
+ happens.
+
+ Considering that both multi-remote and submodule fetches are rather
+ niche and likely not going to be useful for the majority of usecases
+ these omissions feel acceptable. If usecases for either of these come up
+ in the future though it is easy enough to add a new "porcelain-v2"
+ format that adds this information.

- ## Documentation/config/fetch.txt ##
-@@ Documentation/config/fetch.txt: fetch.pruneTags::
-
- fetch.output::
- Control how ref update status is printed. Valid values are
-- full and compact. Default value is full. See section
-- OUTPUT in linkgit:git-fetch[1] for detail.
-+ full, compact and porcelain. Default value is full.
-+ See section OUTPUT in linkgit:git-fetch[1] for detail.
-
- fetch.negotiationAlgorithm::
- Control how information about the commits in the local repository
-
## Documentation/fetch-options.txt ##
@@ Documentation/fetch-options.txt: linkgit:git-config[1].
+ --dry-run::
+ Show what would be done, without making any changes.

- --output-format::
- Control how ref update status is printed. Valid values are
-- full and compact. Default value is full. See section
-- OUTPUT in linkgit:git-fetch[1] for detail.
-+ full, compact and porcelain. Default value is full.
-+ See section OUTPUT in linkgit:git-fetch[1] for detail.
-
++--porcelain::
++ Print the output to standard output in an easy-to-parse format for
++ scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
+++
++This is incompatible with --recurse-submodules=[yes|on-demand]. ++
ifndef::git-pull[]
--[no-]write-fetch-head::
+ Write the list of remote refs fetched in the FETCH_HEAD

## Documentation/git-fetch.txt ##
-@@ Documentation/git-fetch.txt: The output of "git fetch" depends on the transport method used; this
- section describes the output when fetching over the Git protocol
- (either locally or via ssh) and Smart HTTP protocol.
-
--The status of the fetch is output in tabular form, with each line
--representing the status of a single ref. Each line is of the form:
-+The output format can be chosen either via the fetch.output config
-+(see linkgit:git-config[1]), or via the --output-format switch.
-+Supported values include:
-+
-+For the full and compact output formats, the status of the fetch is
-+output in tabular, with each line representing the status of a single
-+ref. Each line is of the form:
-
- -------------------------------
+@@ Documentation/git-fetch.txt: representing the status of a single ref. Each line is of the form:
<flag> <summary> <from> -> <to> [<reason>]
-------------------------------

-+The porcelain output format is intended to be machine-parseable. In -+contrast to the human-readable output formats it thus prints to standard -+output instead of standard error. Each line is of the form:
++When using --porcelain, the output format is intended to be ++machine-parseable. In contrast to the human-readable output formats it ++thus prints to standard output instead of standard error. Each line is ++of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
@@ builtin/fetch.c: enum display_format {
DISPLAY_FORMAT_FULL,
DISPLAY_FORMAT_COMPACT,
+ DISPLAY_FORMAT_PORCELAIN,
- DISPLAY_FORMAT_MAX,
- };
-
-@@ builtin/fetch.c: static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
- NULL,
- "full",
- "compact",
-+ "porcelain",
};

struct display_state {
@@ builtin/fetch.c: static void display_state_init(struct display_state *display_st
+ /* We don't need to precompute anything here. */
+ break;
default:
-- BUG("unexpected display foramt %d", display_state->format);
-+ BUG("unexpected display format %d", display_state->format);
+ BUG("unexpected display format %d", display_state->format);
}
- }
-
@@ builtin/fetch.c: static void print_compact(struct display_state *display_state,
static void display_ref_update(struct display_state *display_state, char code,
const char *summary, const char *error,
@@ builtin/fetch.c: static int prune_refs(struct display_state *display_state,
summary_width);
warn_dangling_symref(stderr, dangling_msg, ref->name);
}
+@@ builtin/fetch.c: static int add_remote_or_group(const char *name, struct string_list *list)
+ return 1;
+ }
+
+-static void add_options_to_argv(struct strvec *argv)
++static void add_options_to_argv(struct strvec *argv,
++ enum display_format format)
+ {
+ if (dry_run)
+ strvec_push(argv, "--dry-run");
+@@ builtin/fetch.c: static void add_options_to_argv(struct strvec *argv)
+ strvec_push(argv, "--ipv6");
+ if (!write_fetch_head)
+ strvec_push(argv, "--no-write-fetch-head");
++ if (format == DISPLAY_FORMAT_PORCELAIN)
++ strvec_pushf(argv, "--porcelain");
+ }
+
+ /* Fetch multiple remotes in parallel */
+@@ builtin/fetch.c: struct parallel_fetch_state {
+ const char **argv;
+ struct string_list *remotes;
+ int next, result;
++ enum display_format format;
+ };
+
+ static int fetch_next_remote(struct child_process *cp,
+@@ builtin/fetch.c: static int fetch_next_remote(struct child_process *cp,
+ strvec_push(&cp->args, remote);
+ cp->git_cmd = 1;
+
+- if (verbosity >= 0)
++ if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
+ printf(_("Fetching %s\n"), remote);
+
+ return 1;
+@@ builtin/fetch.c: static int fetch_finished(int result, struct strbuf *out,
+ return 0;
+ }
+
+-static int fetch_multiple(struct string_list *list, int max_children) ++static int fetch_multiple(struct string_list *list, int max_children,
++ enum display_format format)
+ {
+ int i, result = 0;
+ struct strvec argv = STRVEC_INIT;
+@@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_children) +
+ strvec_pushl(&argv, "fetch", "--append", "--no-auto-gc",
+ "--no-write-commit-graph", NULL);
+- add_options_to_argv(&argv);
++ add_options_to_argv(&argv, format);
+
+ if (max_children != 1 && list->nr != 1) {
+- struct parallel_fetch_state state = { argv.v, list, 0, 0 };
++ struct parallel_fetch_state state = { argv.v, list, 0, 0, format };
+ const struct run_process_parallel_opts opts = {
+ .tr2_category = "fetch",
+ .tr2_label = "parallel/fetch",
+@@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_children) +
+ strvec_pushv(&cmd.args, argv.v);
+ strvec_push(&cmd.args, name);
+- if (verbosity >= 0)
++ if (verbosity >= 0 && format != DISPLAY_FORMAT_PORCELAIN)
+ printf(_("Fetching %s\n"), name);
+ cmd.git_cmd = 1;
+ if (run_command(&cmd)) {
+@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv,
+ return exit_code;
+ }
+
++static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset) ++{
++ enum display_format *format = opt->value;
++ *format = DISPLAY_FORMAT_PORCELAIN;
++ return 0;
++}
++
+ int cmd_fetch(int argc, const char **argv, const char *prefix)
+ {
+ const char *bundle_uri;
+@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
+ PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+ OPT_BOOL(0, "dry-run", &dry_run,
+ N_("dry run")),
++ OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
++ PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
+ OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+ N_("write fetched references to the FETCH_HEAD file")),
+ OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
+ fetch_config_from_gitmodules(sfjc, rs);
+ }
+
++ if (display_format == DISPLAY_FORMAT_PORCELAIN) {
++ switch (recurse_submodules_cli) {
++ case RECURSE_SUBMODULES_OFF:
++ case RECURSE_SUBMODULES_DEFAULT:
++ /*
++ * Reference updates in submodules would be ambiguous
++ * in porcelain mode, so we reject this combination.
++ */
++ recurse_submodules = RECURSE_SUBMODULES_OFF;
++ break;
++
++ default:
++ die(_("options '%s' and '%s' cannot be used together"),
++ "--porcelain", "--recurse-submodules");
++ }
++ }
++
+ if (negotiate_only && !negotiation_tip.nr)
+ die(_("--negotiate-only needs one or more --negotiation-tip=*"));
+
+@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
+ max_children = fetch_parallel_config;
+
+ /* TODO should this also die if we have a previous partial-clone? */
+- result = fetch_multiple(&list, max_children);
++ result = fetch_multiple(&list, max_children, display_format);
+ }
+
+
+@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
+ if (max_children < 0)
+ max_children = fetch_parallel_config;
+
+- add_options_to_argv(&options);
++ add_options_to_argv(&options, display_format);
+ result = fetch_submodules(the_repository,
+ &options,
+ submodule_prefix,

## t/t5574-fetch-output.sh ##
-@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output with multiple remotes' ' +@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
test_cmp expect actual
'

+test_expect_success 'fetch porcelain output' '
-+ test_when_finished "rm -rf porcelain-cfg porcelain-cli" &&
++ test_when_finished "rm -rf porcelain" &&
+
+ # Set up a bunch of references that we can use to demonstrate different
+ # kinds of flag symbols in the output format.
@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output with multiple
+ # namespaces so that we can test that rejected and force-updated
+ # references are reported properly.
+ refspecs="refs/heads/*:refs/unforced/* +refs/heads/*:refs/forced/*" &&
-+ git clone . porcelain-cli &&
-+ git clone . porcelain-cfg &&
-+ git -C porcelain-cfg fetch origin $refspecs &&
-+ git -C porcelain-cli fetch origin $refspecs &&
++ git clone . porcelain &&
++ git -C porcelain fetch origin $refspecs &&
+
+ # Now that we have set up the client repositories we can change our
+ # local references.
@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output with multiple
+ # Execute a dry-run fetch first. We do this to assert that the dry-run
+ # and non-dry-run fetches produces the same output. Execution of the
+ # fetch is expected to fail as we have a rejected reference update.
-+ test_must_fail git -C porcelain-cfg -c fetch.output=porcelain fetch --dry-run --prune origin $refspecs >actual-dry-run-cfg &&
-+ test_must_fail git -C porcelain-cli fetch --output-format=porcelain --dry-run --prune origin $refspecs >actual-dry-run-cli &&
-+ test_cmp actual-dry-run-cfg actual-dry-run-cli &&
-+ test_cmp expect actual-dry-run-cfg &&
++ test_must_fail git -C porcelain fetch \
++ --porcelain --dry-run --prune origin $refspecs >actual &&
++ test_cmp expect actual &&
+
+ # And now we perform a non-dry-run fetch.
-+ test_must_fail git -C porcelain-cfg -c fetch.output=porcelain fetch --prune origin $refspecs >actual-cfg &&
-+ test_must_fail git -C porcelain-cli fetch --output-format=porcelain --prune origin $refspecs >actual-cli &&
-+ test_cmp actual-cfg actual-cli &&
-+ test_cmp expect actual-cfg &&
-+
-+ # Ensure that the dry-run and non-dry-run output matches.
-+ test_cmp actual-dry-run-cfg actual-cfg
++ test_must_fail git -C porcelain fetch \
++ --porcelain --prune origin $refspecs >actual 2>stderr &&
++ test_cmp expect actual &&
++ test_must_be_empty stderr
+'
+
- test_expect_success 'fetch output with HEAD and --dry-run' ' ++test_expect_success 'fetch porcelain with multiple remotes' '
++ test_when_finished "rm -rf porcelain" &&
++
++ git clone . porcelain &&
++ git -C porcelain remote add second-remote "$PWD" &&
++ git -C porcelain fetch second-remote &&
++
++ test_commit --no-tag multi-commit &&
++ old_commit=$(git rev-parse HEAD~) &&
++ new_commit=$(git rev-parse HEAD) &&
++
++ cat >expect <<-EOF &&
++ $old_commit $new_commit refs/remotes/origin/force-updated
++ $old_commit $new_commit refs/remotes/second-remote/force-updated
++ EOF
++
++ git -C porcelain fetch --porcelain --all >actual 2>stderr &&
++ test_cmp expect actual &&
++ test_must_be_empty stderr
++'
++
++test_expect_success 'fetch porcelain refuses to work with submodules' '
++ test_when_finished "rm -rf porcelain" &&
++
++ cat >expect <<-EOF &&
++ fatal: options ${SQ}--porcelain${SQ} and ${SQ}--recurse-submodules${SQ} cannot be used together
++ EOF
++
++ git init porcelain &&
++ test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=yes 2>stderr &&
++ test_cmp expect stderr &&
++
++ test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=on-demand 2>stderr &&
++ test_cmp expect stderr
++'
++
+ test_expect_success 'fetch output with HEAD' '
test_when_finished "rm -rf head" &&
git clone . head &&
-@@ t/t5574-fetch-output.sh: test_expect_success 'fetch output with HEAD and --dry-run' '
- test_cmp expect actual
+@@ t/t5574-fetch-output.sh: test_expect_success 'fetch output with HEAD' '
+ test_cmp expect actual.err
'

-+test_expect_success 'fetch porcelain output with HEAD and --dry-run' ' ++test_expect_success 'fetch porcelain output with HEAD' '
+ test_when_finished "rm -rf head" &&
+ git clone . head &&
+ COMMIT_ID=$(git rev-parse HEAD) &&
+
-+ git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
++ git -C head fetch --porcelain --dry-run origin HEAD >actual &&
+ cat >expect <<-EOF &&
+ * $ZERO_OID $COMMIT_ID FETCH_HEAD
+ EOF
+ test_cmp expect actual &&
+
-+ git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
++ git -C head fetch --porcelain origin HEAD >actual &&
++ test_cmp expect actual &&
++
++ git -C head fetch --porcelain --dry-run origin HEAD:foo >actual &&
+ cat >expect <<-EOF &&
+ * $ZERO_OID $COMMIT_ID refs/heads/foo
+ EOF
++ test_cmp expect actual &&
++
++ git -C head fetch --porcelain origin HEAD:foo >actual &&
+ test_cmp expect actual
+'
+

Signature · 1 line
2.40.1
#58 lore raw

PATCH v3 1/8 fetch: fix `--no-recurse-submodules` with multi-remote fetches

When running git fetch --no-recurse-submodules, the exectation is that we don't fetch any submodules. And while this works for fetches of a single remote, it doesn't when fetching multiple remotes at once. The result is that we do recurse into submodules even though the user has explicitly asked us not to.

This is because while we pass on --recurse-submodules={yes,on-demand} if specified by the user, we don't pass on --no-recurse-submodules to the subprocess spawned to perform the submodule fetch.

Fix this by also forwarding this flag as expected.

---
 builtin/fetch.c             |  2 ++
 t/t5526-fetch-submodules.sh | 31 +++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
builtin/fetch.c +2 -0
index c310d89878..08d7fc7233 100644
@@ -1876,6 +1876,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--keep");
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		strvec_push(argv, "--recurse-submodules");
+	else if (recurse_submodules == RECURSE_SUBMODULES_OFF)
+		strvec_push(argv, "--no-recurse-submodules");
 	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
 		strvec_push(argv, "--recurse-submodules=on-demand");
 	if (tags == TAGS_SET)
t/t5526-fetch-submodules.sh +31 -0
index dcdbe26a08..162e5bac2f 100755
@@ -1180,4 +1180,35 @@ test_expect_success 'fetch --all with --recurse-submodules with multiple' '
 	test_line_count = 2 fetch-subs
 '
 
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "git -C downstream remote remove second" &&
+
+	# We need to add a second remote, otherwise --all falls back to the
+	# normal fetch-one case.
+	git -C downstream remote add second .. &&
+	git -C downstream fetch --all &&
+
+	add_submodule_commits &&
+	add_superproject_commits &&
+	old_commit=$(git rev-parse --short HEAD~) &&
+	new_commit=$(git rev-parse --short HEAD) &&
+
+	git -C downstream fetch --all --no-recurse-submodules >actual.out 2>actual.err &&
+
+	cat >expect.out <<-EOF &&
+	Fetching origin
+	Fetching second
+	EOF
+
+	cat >expect.err <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	   $old_commit..$new_commit  super      -> origin/super
+	From ..
+	   $old_commit..$new_commit  super      -> second/super
+	EOF
+
+	test_cmp expect.out actual.out &&
+	test_cmp expect.err actual.err
+'
+
 test_done
Signature · 1 line
2.40.1
#59 lore raw

Re: [PATCH v3 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

Quoted 23 lines · Patrick Steinhardt

When running git fetch --no-recurse-submodules, the exectation is that we don't fetch any submodules. And while this works for fetches of a single remote, it doesn't when fetching multiple remotes at once. The result is that we do recurse into submodules even though the user has explicitly asked us not to.

This is because while we pass on --recurse-submodules={yes,on-demand} if specified by the user, we don't pass on --no-recurse-submodules to the subprocess spawned to perform the submodule fetch.

Fix this by also forwarding this flag as expected.

---
 builtin/fetch.c             |  2 ++
 t/t5526-fetch-submodules.sh | 31 +++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
builtin/fetch.c +0 -0
index c310d89878..08d7fc7233 100644

builtin/fetch.c static void add_options_to_argv(struct strvec *argv) · line 1883

Quoted 8 lines
builtin/fetch.c +2 -0
@@ -1876,6 +1876,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--keep");
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		strvec_push(argv, "--recurse-submodules");
+	else if (recurse_submodules == RECURSE_SUBMODULES_OFF)
+		strvec_push(argv, "--no-recurse-submodules");
 	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
 		strvec_push(argv, "--recurse-submodules=on-demand");
 	if (tags == TAGS_SET)

Makes sense.

I wondered for a bit whether this should have been checking recurse_submodules_cli (the actual CLI flag) back in 386c076a86 (fetch --negotiate-only: do not update submodules, 2022-01-18). I think this current state is correct, though. After we have told the superproject what submodule recursion mode to use, we want to continue using that mode when recursing through submodules regardless of whether that mode originally came from the CLI or config.

Quoted 20 lines
builtin/fetch.c +20 -0
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "git -C downstream remote remove second" &&
+
+	# We need to add a second remote, otherwise --all falls back to the
+	# normal fetch-one case.
+	git -C downstream remote add second .. &&
+	git -C downstream fetch --all &&
+
+	add_submodule_commits &&
+	add_superproject_commits &&
+	old_commit=$(git rev-parse --short HEAD~) &&
+	new_commit=$(git rev-parse --short HEAD) &&
+
+	git -C downstream fetch --all --no-recurse-submodules >actual.out 2>actual.err &&
+
+	cat >expect.out <<-EOF &&
+	Fetching origin
+	Fetching second
+	EOF
+

builtin/fetch.c

Quoted 10 lines
builtin/fetch.c +10 -0
+	cat >expect.err <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	   $old_commit..$new_commit  super      -> origin/super
+	From ..
+	   $old_commit..$new_commit  super      -> second/super
+	EOF
+
+	test_cmp expect.out actual.out &&
+	test_cmp expect.err actual.err
+'

The test looks okay, though is there a reason you didn't copy the style of the previous test? It is nearly exactly what you want, I think, like (untested)

  test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
    test_when_finished "rm -fr src_clone" &&
    git clone --recurse-submodules src src_clone &&
    (
      cd src_clone &&
      git remote add secondary ../src &&
      git config submodule.recurse true &&
      git config fetch.parallel 0 &&
      git fetch --all 2>../fetch-log
    ) &&
    grep "Fetching submodule" fetch-log >fetch-subs &&
    test_must_be_empty fetch-subs
  '

which has the handy benefit of not needing the test-tools invocation.

#60 lore raw

Re: [PATCH v3 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

Quoted 1 line · Glen Choo

Patrick Steinhardt <[email protected]> writes:

[snip]

Quoted 49 lines
Quoted 20 lines
t/t5526-fetch-submodules.sh +20 -0
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "git -C downstream remote remove second" &&
+
+	# We need to add a second remote, otherwise --all falls back to the
+	# normal fetch-one case.
+	git -C downstream remote add second .. &&
+	git -C downstream fetch --all &&
+
+	add_submodule_commits &&
+	add_superproject_commits &&
+	old_commit=$(git rev-parse --short HEAD~) &&
+	new_commit=$(git rev-parse --short HEAD) &&
+
+	git -C downstream fetch --all --no-recurse-submodules >actual.out 2>actual.err &&
+
+	cat >expect.out <<-EOF &&
+	Fetching origin
+	Fetching second
+	EOF
+

t/t5526-fetch-submodules.sh test_expect_success 'fetch --all with --recurse-submodules with multiple' ' · line 1212

Quoted 10 lines
t/t5526-fetch-submodules.sh +10 -0
+	cat >expect.err <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	   $old_commit..$new_commit  super      -> origin/super
+	From ..
+	   $old_commit..$new_commit  super      -> second/super
+	EOF
+
+	test_cmp expect.out actual.out &&
+	test_cmp expect.err actual.err
+'

The test looks okay, though is there a reason you didn't copy the style of the previous test? It is nearly exactly what you want, I think, like (untested)

  test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
    test_when_finished "rm -fr src_clone" &&
    git clone --recurse-submodules src src_clone &&
    (
      cd src_clone &&
      git remote add secondary ../src &&
      git config submodule.recurse true &&
      git config fetch.parallel 0 &&
      git fetch --all 2>../fetch-log
    ) &&
    grep "Fetching submodule" fetch-log >fetch-subs &&
    test_must_be_empty fetch-subs
  '
Quoted 1 line

which has the handy benefit of not needing the test-tools invocation.

That is indeed much simpler, thanks.

Patrick

#61 lore raw

PATCH v3 2/8 fetch: split out tests for output format

In reply to Patrick Steinhardt

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
t/t5510-fetch.sh +0 -53
index dc44da9c79..4f289063ce 100755
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
t/t5574-fetch-output.sh new file +63 -0

63 lines not on this page · raw message

Signature · 1 line
2.40.1
#62 lore raw

PATCH v3 3/8 fetch: add a test to exercise invalid output formats

In reply to Patrick Steinhardt

Add a testcase that exercises the logic when an invalid output format is passed via the fetch.output configuration.

---
 t/t5574-fetch-output.sh | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
t/t5574-fetch-output.sh +19 -0
index f91b654d38..a09750d225 100755
@@ -7,6 +7,25 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
Signature · 1 line
2.40.1
#63 lore raw

PATCH v3 4/8 fetch: fix missing from-reference when fetching HEAD:foo

In reply to Patrick Steinhardt

store_updated_refs() parses the remote reference for two purposes:

- It gets used as a note when writing FETCH_HEAD.

- It is passed through to display_ref_update() to display updated references in the following format:

```
* branch master -> master
```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

```
$ git fetch origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands show the left-hand side as expected:

```
$ git fetch origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

$ git fetch origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

The logic of how we compute the remote reference name that we ultimately pass to display_ref_update() is not easy to follow. There are three different cases here:

- When the remote reference name is "HEAD" we set the remote reference name to the empty string. This is the case that causes the bug to occur, where we would indeed want to print "HEAD" instead of the empty string. This is what prettify_refname() would return.

- When the remote reference name has a well-known prefix then we strip this prefix. This matches what prettify_refname() does.

- Otherwise, we keep the fully qualified reference name. This also matches what prettify_refname() does.

As the return value of prettify_refname() would do the correct thing for us in all three cases, we can fix the bug by passing through the full remote reference name to display_ref_update(), which learns to call prettify_refname(). At the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

---
 builtin/fetch.c         | 37 +++++++++++++++++++------------------
 t/t5574-fetch-output.sh | 29 +++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 18 deletions(-)
builtin/fetch.c +19 -18
index 08d7fc7233..6aecf549e8 100644
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
t/t5574-fetch-output.sh +29 -0
index a09750d225..6e0f7e0046 100755
@@ -56,6 +56,35 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD:foo >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
Signature · 1 line
2.40.1
#64 lore raw

PATCH v3 5/8 fetch: introduce `display_format` enum

In reply to Patrick Steinhardt

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a enum display_format that is more readily extensible.

---
 builtin/fetch.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
builtin/fetch.c +64 -37
index 6aecf549e8..9e7e45344d 100644
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -784,7 +790,6 @@ static int refcol_width(const struct ref *ref, int compact_format)
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display format %d", display_state->format);
 	}
 }
 
@@ -904,30 +920,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
Signature · 1 line
2.40.1
#65 lore raw

PATCH v3 6/8 fetch: move display format parsing into main function

In reply to Patrick Steinhardt

Parsing the display format happens inside of display_state_init(). As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the fetch.output config key set. We're thus going to introduce a new --output-format switch for git-fetch(1) so that the output format can be configured more directly.

This means we'll have to hook parsing of the display format into the command line options parser. Having the code to determine the actual output format scattered across two different sites is hard to reason about though.

Refactor the code such that callers are expected to pass the display format that is to be used into display_state_init(). This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

---
 builtin/fetch.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
builtin/fetch.c +24 -17
index 9e7e45344d..e15d43dc1e 100644
@@ -788,14 +788,13 @@ static int refcol_width(const struct ref *ref, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -812,15 +811,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT: {
@@ -1614,7 +1604,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1700,7 +1691,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2078,7 +2069,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2145,7 +2137,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2157,6 +2149,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2183,6 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}
+
 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
 		recurse_submodules = recurse_submodules_cli;
 
@@ -2311,7 +2317,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   display_format);
 	} else {
 		int max_children = max_jobs;
 
Signature · 1 line
2.40.1
#66 lore raw

PATCH v3 7/8 fetch: move option related variables into main function

In reply to Patrick Steinhardt

The options of git-fetch(1) which we pass to parse_options() are declared globally in builtin/fetch.c. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into cmd_fetch(). Move variables that were previously forced to be declared globally and which are only used by cmd_fetch() into function-local scope.

---
 builtin/fetch.c | 197 ++++++++++++++++++++++++------------------------
 1 file changed, 100 insertions(+), 97 deletions(-)
builtin/fetch.c +100 -97
index e15d43dc1e..820ec7285c 100644
@@ -75,13 +75,12 @@ static int fetch_prune_tags_config = -1; /* unspecified */
 static int prune_tags = -1; /* unspecified */
 #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok;
+static int append, dry_run, force, keep, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
@@ -160,92 +153,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2147,13 +2054,109 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
-	int i;
 	const char *bundle_uri;
+	const char *submodule_prefix = "";
 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
+	int all = 0, multiple = 0;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
Signature · 1 line
2.40.1
#67 lore raw

PATCH v3 8/8 fetch: introduce machine-parseable "porcelain" output format

In reply to Patrick Steinhardt

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```

We assume two conditions which are generally true:

- The old and new object IDs have fixed known widths and cannot contain spaces.

- References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote.

In a similar spirit, a second restriction is that this cannot be used with --recurse-submodules. This is because any reference updates would be ambiguous without also printing the repository in which the update happens.

Considering that both multi-remote and submodule fetches are rather niche and likely not going to be useful for the majority of usecases these omissions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

---
 Documentation/fetch-options.txt |   6 ++
 Documentation/git-fetch.txt     |   9 +++
 builtin/fetch.c                 |  91 +++++++++++++++++++-----
 t/t5574-fetch-output.sh         | 120 ++++++++++++++++++++++++++++++++
 4 files changed, 208 insertions(+), 18 deletions(-)
Documentation/fetch-options.txt +6 -0
index 622bd84768..02a5ae21ef 100644
@@ -78,6 +78,12 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--porcelain::
+	Print the output to standard output in an easy-to-parse format for
+	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
++
+This is incompatible with `--recurse-submodules=[yes|on-demand]`.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
Documentation/git-fetch.txt +9 -0
index fba66f1460..f123139c58 100644
@@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+When using `--porcelain`, the output format is intended to be
+machine-parseable. In contrast to the human-readable output formats it
+thus prints to standard output instead of standard error. Each line is
+of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
builtin/fetch.c +73 -18

252 lines not on this page · raw message

t/t5574-fetch-output.sh +120 -0

132 lines not on this page · raw message

Signature · 1 line
2.40.1
#68 lore raw

Re: [PATCH v3 8/8] fetch: introduce machine-parseable "porcelain" output format

This version looks great! I only have minor comments.

Quoted 12 lines · Patrick Steinhardt

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote.

In a similar spirit, a second restriction is that this cannot be used with --recurse-submodules. This is because any reference updates would be ambiguous without also printing the repository in which the update happens.

Quoted 5 lines

Considering that both multi-remote and submodule fetches are rather niche and likely not going to be useful for the majority of usecases these omissions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

As a point of clarification, I think these options aren't niche in themselves, but they are more user-facing, so using them _in conjunction with_ --porcelain is probably pretty niche, so I think this is okay for now.

Quoted 24 lines
Documentation/fetch-options.txt +6 -0
@@ -78,6 +78,12 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--porcelain::
+	Print the output to standard output in an easy-to-parse format for
+	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
++
+This is incompatible with `--recurse-submodules=[yes|on-demand]`.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
Documentation/git-fetch.txt +1 -0
index fba66f1460..f123139c58 100644
@@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+When using `--porcelain`, the output format is intended to be

Documentation/git-fetch.txt representing the status of a single ref. Each line is of the form: · line 217

Quoted 10 lines
Documentation/git-fetch.txt +8 -0
+machine-parseable. In contrast to the human-readable output formats it
+thus prints to standard output instead of standard error. Each line is
+of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.

I hadn't commented on the docs before, but they are very clear. Thanks!

Quoted 9 lines
Documentation/git-fetch.txt +1 -0
@@ -1830,6 +1857,7 @@ struct parallel_fetch_state {
 	const char **argv;
 	struct string_list *remotes;
 	int next, result;
+	enum display_format format;
 };
 
 static int fetch_next_remote(struct child_process *cp,

Documentation/git-fetch.txt static int fetch_next_remote(struct child_process *cp, · line 1883

Quoted 8 lines
Documentation/git-fetch.txt +1 -1
@@ -1849,7 +1877,7 @@ static int fetch_next_remote(struct child_process *cp,
 	strvec_push(&cp->args, remote);
 	cp->git_cmd = 1;
 
-	if (verbosity >= 0)
+	if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
 		printf(_("Fetching %s\n"), remote);
 
 	return 1;

Here and elsewhere, I wonder if it's clearer to name the variable "porcelain" and separate it from "enum display_format". Then we can check "porcelain" directly instead of using "format == DISPLAY_FORMAT_PORCELAIN" as a proxy...

Quoted 11 lines
Documentation/git-fetch.txt +7 -0
+static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value;
+	*format = DISPLAY_FORMAT_PORCELAIN;
+	return 0;
+}
+
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	const char *bundle_uri;

Documentation/git-fetch.txt int cmd_fetch(int argc, const char **argv, const char *prefix) · line 2147

Quoted 8 lines
Documentation/git-fetch.txt +2 -0
@@ -2104,6 +2140,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
+			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),

e.g. since we are reusing the "display_format" variable, we need to make sure we parse "--porcelain" after we read "fetch.output". I double-checked to make sure we were doing the right thing, though it would be nice to not have to worry about those sorts of things. This shouldn't hold up the series though.

Quoted 5 lines
Documentation/git-fetch.txt +5 -0
+test_expect_success 'fetch porcelain with multiple remotes' '
+	test_when_finished "rm -rf porcelain" &&
+
+	git clone . porcelain &&
+	git -C porcelain remote add second-remote "$PWD" &&

Documentation/git-fetch.txt

Quoted 10 lines
Documentation/git-fetch.txt +10 -0
+	git -C porcelain fetch second-remote &&
+
+	test_commit --no-tag multi-commit &&
+	old_commit=$(git rev-parse HEAD~) &&
+	new_commit=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	  $old_commit $new_commit refs/remotes/origin/force-updated
+	  $old_commit $new_commit refs/remotes/second-remote/force-updated
+	EOF

The only thing in this test that relies on the previous test is that HEAD is pointing to "force-updated", and it's hard to tell where HEAD is since the previous test is so long. Could we create a new branch instead?

#69 lore raw

Re: [PATCH v3 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 22 lines · Glen Choo

This version looks great! I only have minor comments.

Quoted 12 lines · Patrick Steinhardt

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote.

In a similar spirit, a second restriction is that this cannot be used with --recurse-submodules. This is because any reference updates would be ambiguous without also printing the repository in which the update happens.

Quoted 5 lines

Considering that both multi-remote and submodule fetches are rather niche and likely not going to be useful for the majority of usecases these omissions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

Quoted 4 lines

As a point of clarification, I think these options aren't niche in themselves, but they are more user-facing, so using them _in conjunction with_ --porcelain is probably pretty niche, so I think this is okay for now.

Yeah, that's what I indeed intended to say. Will clarify.

[snip]

Quoted 16 lines
Quoted 16 lines
builtin/fetch.c +2 -1
@@ -1830,6 +1857,7 @@ struct parallel_fetch_state {
 	const char **argv;
 	struct string_list *remotes;
 	int next, result;
+	enum display_format format;
 };
 
 static int fetch_next_remote(struct child_process *cp,
@@ -1849,7 +1877,7 @@ static int fetch_next_remote(struct child_process *cp,
 	strvec_push(&cp->args, remote);
 	cp->git_cmd = 1;
 
-	if (verbosity >= 0)
+	if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
 		printf(_("Fetching %s\n"), remote);
 
Quoted 6 lines

Review static int fetch_next_remote(struct child_process *cp, · line 1883

Quoted 1 line
(quoted diff) +0 -0
 	return 1;

Here and elsewhere, I wonder if it's clearer to name the variable "porcelain" and separate it from "enum display_format". Then we can check "porcelain" directly instead of using "format == DISPLAY_FORMAT_PORCELAIN" as a proxy...

For now I'd like to keep this as-is: it's easier to keep track of this when there is only a single variable that keeps track of the output format. But if we were to add additional porcelain formats in the future I agree that it would be nice to refactor the code as you propose.

I'd rather keep the simpler version for now though where we only have a single state to worry about.

Quoted 17 lines
Quoted 17 lines
builtin/fetch.c +9 -0
+static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value;
+	*format = DISPLAY_FORMAT_PORCELAIN;
+	return 0;
+}
+
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	const char *bundle_uri;
@@ -2104,6 +2140,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
+			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
Quoted 8 lines

builtin/fetch.c int cmd_fetch(int argc, const char **argv, const char *prefix) · line 2147

Quoted 2 lines
builtin/fetch.c +0 -0
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),

e.g. since we are reusing the "display_format" variable, we need to make sure we parse "--porcelain" after we read "fetch.output". I double-checked to make sure we were doing the right thing, though it would be nice to not have to worry about those sorts of things. This shouldn't hold up the series though.

In fact it is the other way round: we parse --porcelain first and then only read "fetch.output" in the case where the display_format variable is still set to DISPLAY_FORMAT_UNKNOWN.

So in the end there is no fragile order dependence here -- it would work just the same regardless of whether we first parse command line options or the configuration.

Quoted 12 lines
Quoted 12 lines
t/t5574-fetch-output.sh +12 -0
+test_expect_success 'fetch porcelain with multiple remotes' '
+	test_when_finished "rm -rf porcelain" &&
+
+	git clone . porcelain &&
+	git -C porcelain remote add second-remote "$PWD" &&
+	git -C porcelain fetch second-remote &&
+
+	test_commit --no-tag multi-commit &&
+	old_commit=$(git rev-parse HEAD~) &&
+	new_commit=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
Quoted 8 lines

t/t5574-fetch-output.sh test_expect_success 'fetch compact output' ' · line 133

Quoted 3 lines
t/t5574-fetch-output.sh +3 -0
+	  $old_commit $new_commit refs/remotes/origin/force-updated
+	  $old_commit $new_commit refs/remotes/second-remote/force-updated
+	EOF

The only thing in this test that relies on the previous test is that HEAD is pointing to "force-updated", and it's hard to tell where HEAD is since the previous test is so long. Could we create a new branch instead?

Makes sense, done.

Patrick

#70 lore raw

Re: [PATCH v3 8/8] fetch: introduce machine-parseable "porcelain" output format

In reply to Patrick Steinhardt

Documentation/fetch-options.txt linkgit:git-config[1]. · line 86

Quoted 6 lines · Patrick Steinhardt

+--porcelain::

Documentation/fetch-options.txt +5 -0
+	Print the output to standard output in an easy-to-parse format for
+	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
++
+This is incompatible with `--recurse-submodules=[yes|on-demand]`.
+

We should probably specify that this will override any setting of "fetch.output" (and perhaps add a test for it). Most readers will probably guess that this is the case, but the extra clarity wouldn't hurt.

#71 lore raw
In reply to Patrick Steinhardt
Quoted 2 lines · Patrick Steinhardt

this is the third version of my patch series to introduce a machine-parseable output for git-fetch(1).

Which base did you pick to build these 8 patches? It seems that 6/8 is not happy when the series is attempted to be applied on top of many recent tips of 'master' that I tried (including 667fcf4e15 that the previous iteration was queued on top of in my tree).

Thanks.

#72 lore raw
Quoted 9 lines · Junio C Hamano
Quoted 2 lines · Patrick Steinhardt

this is the third version of my patch series to introduce a machine-parseable output for git-fetch(1).

Which base did you pick to build these 8 patches? It seems that 6/8 is not happy when the series is attempted to be applied on top of many recent tips of 'master' that I tried (including 667fcf4e15 that the previous iteration was queued on top of in my tree).

OK, I found e9dffbc7 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06). I can manage from here.

Thanks.

#73 lore raw
Quoted 12 lines · Junio C Hamano
Quoted 9 lines · Junio C Hamano
Quoted 2 lines · Patrick Steinhardt

this is the third version of my patch series to introduce a machine-parseable output for git-fetch(1).

Which base did you pick to build these 8 patches? It seems that 6/8 is not happy when the series is attempted to be applied on top of many recent tips of 'master' that I tried (including 667fcf4e15 that the previous iteration was queued on top of in my tree).

Quoted 4 lines

OK, I found e9dffbc7 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06). I can manage from here.

Thanks.

Yeah, that's where I based it off from. Sorry for not explicitly mentioning this fact.

Patrick

#74 lore raw
In reply to Patrick Steinhardt
Quoted 4 lines · Patrick Steinhardt

Hi,

this is the third version of my patch series to introduce a machine-parseable output for git-fetch(1).

This version looks great! I only had minor comments this time, mainly on docs and tests.

Thanks for your patience and receptiveness, and for the fruitful discussions :)

#75 lore raw
Quoted 4 lines · Glen Choo
Quoted 2 lines · Patrick Steinhardt

Hi,

Quoted 8 lines
Quoted 2 lines

this is the third version of my patch series to introduce a machine-parseable output for git-fetch(1).

This version looks great! I only had minor comments this time, mainly on docs and tests.

Thanks for your patience and receptiveness, and for the fruitful discussions :)

Thank _you_ for the extensive feedback and constructive criticism.

Patrick

#76 lore raw

v4 cover fetch: introduce machine-parseable output

In reply to Patrick Steinhardt

Hi,

this is the fourth version of my patch series to introduce a machine-parseable output format for git-fetch(1). It applies on top of e9dffbc7f1 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06).

There's only a small set of change compared to v3:

- Patch 1/8: The test added by this change has been significantly simplified.

- Patch 8/8: Reworded the commit message to clarify that parallel fetches and --recurse-submodules aren't niche on their own, but are likely niche combined with --porcelain given that they are user-facing while --porcelain is script-facing.

- Patch 8/8: Clarified that --porcelain takes precedence over the fetch.output config option in the user-facing documentation. I've also added a test to verify that this is the case.

- Patch 8/8: Amended one test to not depend on the state that its preceding test cretaes.

Thanks to Glen for the feedback.

Patrick

Patrick Steinhardt (8):
fetch: fix --no-recurse-submodules with multi-remote fetches
fetch: split out tests for output format
fetch: add a test to exercise invalid output formats
fetch: fix missing from-reference when fetching HEAD:foo
fetch: introduce display_format enum
fetch: move display format parsing into main function
fetch: move option related variables into main function
fetch: introduce machine-parseable "porcelain" output format

 Documentation/fetch-options.txt |   7 +
 Documentation/git-fetch.txt     |   9 +
 builtin/fetch.c                 | 437 +++++++++++++++++++-------------
 t/t5510-fetch.sh                |  53 ----
 t/t5526-fetch-submodules.sh     |  19 ++
 t/t5574-fetch-output.sh         | 251 ++++++++++++++++++
 6 files changed, 552 insertions(+), 224 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh

Range-diff against v3:
1: 4b2b0cfe15 ! 1: d82b42ed34 fetch: fix --no-recurse-submodules with multi-remote fetches

    @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetch --all with --recurse-sub
      '
      
     +test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
    -+	test_when_finished "git -C downstream remote remove second" &&
    ++	test_when_finished "rm -rf src_clone" &&
     +
    -+	# We need to add a second remote, otherwise --all falls back to the
    -+	# normal fetch-one case.
    -+	git -C downstream remote add second .. &&
    -+	git -C downstream fetch --all &&
    ++	git clone --recurse-submodules src src_clone &&
    ++	(
    ++		cd src_clone &&
    ++		git remote add secondary ../src &&
    ++		git config submodule.recurse true &&
    ++		git config fetch.parallel 0 &&
    ++		git fetch --all --no-recurse-submodules 2>../actual
    ++	) &&
     +
    -+	add_submodule_commits &&
    -+	add_superproject_commits &&
    -+	old_commit=$(git rev-parse --short HEAD~) &&
    -+	new_commit=$(git rev-parse --short HEAD) &&
    -+
    -+	git -C downstream fetch --all --no-recurse-submodules >actual.out 2>actual.err &&
    -+
    -+	cat >expect.out <<-EOF &&
    -+	Fetching origin
    -+	Fetching second
    ++	cat >expect <<-EOF &&
    ++	From ../src
    ++	 * [new branch]      master     -> secondary/master
     +	EOF
    -+
    -+	cat >expect.err <<-EOF &&
    -+	From $(test-tool path-utils real_path .)/.
    -+	   $old_commit..$new_commit  super      -> origin/super
    -+	From ..
    -+	   $old_commit..$new_commit  super      -> second/super
    -+	EOF
    -+
    -+	test_cmp expect.out actual.out &&
    -+	test_cmp expect.err actual.err
    ++	test_cmp expect actual
     +'
     +
      test_done
2:  6ebc7450ba = 2:  33112dc51a fetch: split out tests for output format
3:  78479922ac = 3:  006ea93afb fetch: add a test to exercise invalid output formats
4:  46e1266ab0 = 4:  e599ea6d33 fetch: fix missing from-reference when fetching HEAD:foo
5:  acc0f7f520 = 5:  80ac00b0c4 fetch: introduce `display_format` enum
6:  cd23440128 = 6:  826b8b7bc0 fetch: move display format parsing into main function
7:  edbc31013f = 7:  20f2e061d6 fetch: move option related variables into main function
8:  e132d9494e ! 8:  24ae381950 fetch: introduce machine-parseable "porcelain" output format
    @@ Commit message
         be ambiguous without also printing the repository in which the update
         happens.
     
    -    Considering that both multi-remote and submodule fetches are rather
    -    niche and likely not going to be useful for the majority of usecases
    -    these omissions feel acceptable. If usecases for either of these come up
    -    in the future though it is easy enough to add a new "porcelain-v2"
    -    format that adds this information.
    +    Considering that both multi-remote and submodule fetches are user-facing
    +    features, using them in conjunction with `--porcelain` that is intended
    +    for scripting purposes is likely not going to be useful in the majority
    +    of cases. With that in mind these restrictions feel acceptable. If
    +    usecases for either of these come up in the future though it is easy
    +    enough to add a new "porcelain-v2" format that adds this information.
     
         Signed-off-by: Patrick Steinhardt <[email protected]>
     
    @@ Documentation/fetch-options.txt: linkgit:git-config[1].
     +	Print the output to standard output in an easy-to-parse format for
     +	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
     ++
    -+This is incompatible with `--recurse-submodules=[yes|on-demand]`.
    ++This is incompatible with `--recurse-submodules=[yes|on-demand]` and takes
    ++precedence over the `fetch.output` config option.
     +
      ifndef::git-pull[]
      --[no-]write-fetch-head::
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +test_expect_success 'fetch porcelain with multiple remotes' '
     +	test_when_finished "rm -rf porcelain" &&
     +
    ++	git switch --create multiple-remotes &&
     +	git clone . porcelain &&
     +	git -C porcelain remote add second-remote "$PWD" &&
     +	git -C porcelain fetch second-remote &&
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +	new_commit=$(git rev-parse HEAD) &&
     +
     +	cat >expect <<-EOF &&
    -+	  $old_commit $new_commit refs/remotes/origin/force-updated
    -+	  $old_commit $new_commit refs/remotes/second-remote/force-updated
    ++	  $old_commit $new_commit refs/remotes/origin/multiple-remotes
    ++	  $old_commit $new_commit refs/remotes/second-remote/multiple-remotes
     +	EOF
     +
     +	git -C porcelain fetch --porcelain --all >actual 2>stderr &&
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=on-demand 2>stderr &&
     +	test_cmp expect stderr
     +'
    ++
    ++test_expect_success 'fetch porcelain overrides fetch.output config' '
    ++	test_when_finished "rm -rf porcelain" &&
    ++
    ++	git switch --create config-override &&
    ++	git clone . porcelain &&
    ++	test_commit new-commit &&
    ++	old_commit=$(git rev-parse HEAD~) &&
    ++	new_commit=$(git rev-parse HEAD) &&
    ++
    ++	cat >expect <<-EOF &&
    ++	  $old_commit $new_commit refs/remotes/origin/config-override
    ++	* $ZERO_OID $new_commit refs/tags/new-commit
    ++	EOF
    ++
    ++	git -C porcelain -c fetch.output=compact fetch --porcelain >stdout 2>stderr &&
    ++	test_must_be_empty stderr &&
    ++	test_cmp expect stdout
    ++'
     +
      test_expect_success 'fetch output with HEAD' '
      	test_when_finished "rm -rf head" &&
Signature · 1 line
2.40.1
#77 lore raw

PATCH v4 1/8 fetch: fix `--no-recurse-submodules` with multi-remote fetches

When running git fetch --no-recurse-submodules, the exectation is that we don't fetch any submodules. And while this works for fetches of a single remote, it doesn't when fetching multiple remotes at once. The result is that we do recurse into submodules even though the user has explicitly asked us not to.

This is because while we pass on --recurse-submodules={yes,on-demand} if specified by the user, we don't pass on --no-recurse-submodules to the subprocess spawned to perform the submodule fetch.

Fix this by also forwarding this flag as expected.

---
 builtin/fetch.c             |  2 ++
 t/t5526-fetch-submodules.sh | 19 +++++++++++++++++++
 2 files changed, 21 insertions(+)
builtin/fetch.c +2 -0
index c310d89878..08d7fc7233 100644
@@ -1876,6 +1876,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--keep");
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		strvec_push(argv, "--recurse-submodules");
+	else if (recurse_submodules == RECURSE_SUBMODULES_OFF)
+		strvec_push(argv, "--no-recurse-submodules");
 	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
 		strvec_push(argv, "--recurse-submodules=on-demand");
 	if (tags == TAGS_SET)
t/t5526-fetch-submodules.sh +19 -0
index dcdbe26a08..ba69cd583f 100755
@@ -1180,4 +1180,23 @@ test_expect_success 'fetch --all with --recurse-submodules with multiple' '
 	test_line_count = 2 fetch-subs
 '
 
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "rm -rf src_clone" &&
+
+	git clone --recurse-submodules src src_clone &&
+	(
+		cd src_clone &&
+		git remote add secondary ../src &&
+		git config submodule.recurse true &&
+		git config fetch.parallel 0 &&
+		git fetch --all --no-recurse-submodules 2>../actual
+	) &&
+
+	cat >expect <<-EOF &&
+	From ../src
+	 * [new branch]      master     -> secondary/master
+	EOF
+	test_cmp expect actual
+'
+
 test_done
Signature · 1 line
2.40.1
#78 lore raw

Re: [PATCH v4 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

Quoted 11 lines · Patrick Steinhardt

When running git fetch --no-recurse-submodules, the exectation is that we don't fetch any submodules. And while this works for fetches of a single remote, it doesn't when fetching multiple remotes at once. The result is that we do recurse into submodules even though the user has explicitly asked us not to.

This is because while we pass on --recurse-submodules={yes,on-demand} if specified by the user, we don't pass on --no-recurse-submodules to the subprocess spawned to perform the submodule fetch.

Fix this by also forwarding this flag as expected.

Makes sense.

t/t5526-fetch-submodules.sh test_expect_success 'fetch --all with --recurse-submodules with multiple' ' · line 1190

Quoted 8 lines
t/t5526-fetch-submodules.sh +8 -0
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "rm -rf src_clone" &&
+
+	git clone --recurse-submodules src src_clone &&
+	(
+		cd src_clone &&
+		git remote add secondary ../src &&
+		git config submodule.recurse true &&

The above two is essential to this test; we are interested in making sure that --no-recurse-submodules is propagated down even when the "--all" option is used, and we want another remote for that. We set the default to recurse, so that passing "--no-recurse-submodules" would defeat it, but just refraining to pass "--recurse-submodules" would cause us to recurse.

t/t5526-fetch-submodules.sh test_expect_success 'fetch --all with --recurse-submodules with multiple' ' · line 1191

Quoted 1 line
t/t5526-fetch-submodules.sh +1 -0
+		git config fetch.parallel 0 &&

Is this necessary for the purpose of the test, though? It should not hurt, but we do not require the end-users to set it in real life for the parallel fetching to work, either, right?

Just being curious.

t/t5526-fetch-submodules.sh test_expect_success 'fetch --all with --recurse-submodules with multiple' ' · line 1200

Quoted 9 lines
t/t5526-fetch-submodules.sh +9 -0
+		git fetch --all --no-recurse-submodules 2>../actual
+	) &&
+
+	cat >expect <<-EOF &&
+	From ../src
+	 * [new branch]      master     -> secondary/master
+	EOF
+	test_cmp expect actual
+'

In the context of a series that attempts to introduce a new stable output format for machine consumption, which implies the traditional output can change to match human users' preference, this test feels a bit brittle, but let's wait until the end of the series to judge that.

Looking good. Thanks.

Quoted 1 line

test_done

#79 lore raw

Re: [PATCH v4 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

Quoted 5 lines · Junio C Hamano

t/t5526-fetch-submodules.sh test_expect_success 'fetch --all with --recurse-submodules with multiple' ' · line 1191

Quoted 1 line
t/t5526-fetch-submodules.sh +1 -0
+		git config fetch.parallel 0 &&

Is this necessary for the purpose of the test, though? It should not hurt, but we do not require the end-users to set it in real life for the parallel fetching to work, either, right?

IIUC it would make the test output deterministic if we fetched from both remotes. That doesn't happen here though, so I guess it's not doing anything right now.

Quoted 7 lines
Quoted 7 lines
t/t5526-fetch-submodules.sh +7 -0
+		git fetch --all --no-recurse-submodules 2>../actual
+	) &&
+
+	cat >expect <<-EOF &&
+	From ../src
+	 * [new branch]      master     -> secondary/master
+	EOF
Quoted 8 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch --all with --recurse-submodules with multiple' ' · line 1200

Quoted 2 lines
t/t5526-fetch-submodules.sh +2 -0
+	test_cmp expect actual
+'

In the context of a series that attempts to introduce a new stable output format for machine consumption, which implies the traditional output can change to match human users' preference, this test feels a bit brittle, but let's wait until the end of the series to judge that.

I also find it a bit brittle to assert on the whole output when this test is about checking that we do not fetch the superproject.

Is there a reason you didn't go with the "grep for submodule lines" approach in the previous tests? If it's about catching regressions, IMO your PATCH 2/8 does a good enough job of doing that.

Wondering out loud, I wonder if it makes sense for us to make a bigger distinction between "tests whose purpose is to guard against unexpected changes in output" (i.e. snapshot tests) vs "tests that happen to use output as a way to assert behavior" (i.e. 'regular' behavioral tests). Many GUI app codebases have such a distinction and have different best practices around them.

#80 lore raw

Re: [PATCH v4 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

Quoted 11 lines · Glen Choo
Quoted 5 lines · Junio C Hamano

t/t5526-fetch-submodules.sh test_expect_success 'fetch --all with --recurse-submodules with multiple' ' · line 1191

Quoted 1 line
t/t5526-fetch-submodules.sh +1 -0
+		git config fetch.parallel 0 &&

Is this necessary for the purpose of the test, though? It should not hurt, but we do not require the end-users to set it in real life for the parallel fetching to work, either, right?

IIUC it would make the test output deterministic if we fetched from both remotes. That doesn't happen here though, so I guess it's not doing anything right now.

Right, will drop.

Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
t/t5526-fetch-submodules.sh +7 -0
+		git fetch --all --no-recurse-submodules 2>../actual
+	) &&
+
+	cat >expect <<-EOF &&
+	From ../src
+	 * [new branch]      master     -> secondary/master
+	EOF
Quoted 8 lines

t/t5526-fetch-submodules.sh test_expect_success 'fetch --all with --recurse-submodules with multiple' ' · line 1200

Quoted 2 lines
t/t5526-fetch-submodules.sh +2 -0
+	test_cmp expect actual
+'

In the context of a series that attempts to introduce a new stable output format for machine consumption, which implies the traditional output can change to match human users' preference, this test feels a bit brittle, but let's wait until the end of the series to judge that.

I also find it a bit brittle to assert on the whole output when this test is about checking that we do not fetch the superproject.

Is there a reason you didn't go with the "grep for submodule lines" approach in the previous tests? If it's about catching regressions, IMO your PATCH 2/8 does a good enough job of doing that.

Quoted 6 lines

Wondering out loud, I wonder if it makes sense for us to make a bigger distinction between "tests whose purpose is to guard against unexpected changes in output" (i.e. snapshot tests) vs "tests that happen to use output as a way to assert behavior" (i.e. 'regular' behavioral tests). Many GUI app codebases have such a distinction and have different best practices around them.

Fair enough, will switch to ! grep "Fetching submodule".

Patrick

#81 lore raw

PATCH v4 2/8 fetch: split out tests for output format

In reply to Patrick Steinhardt

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
t/t5510-fetch.sh +0 -53
index dc44da9c79..4f289063ce 100755
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
t/t5574-fetch-output.sh new file +63 -0

63 lines not on this page · raw message

Signature · 1 line
2.40.1
#82 lore raw

PATCH v4 3/8 fetch: add a test to exercise invalid output formats

In reply to Patrick Steinhardt

Add a testcase that exercises the logic when an invalid output format is passed via the fetch.output configuration.

---
 t/t5574-fetch-output.sh | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
t/t5574-fetch-output.sh +19 -0
index f91b654d38..a09750d225 100755
@@ -7,6 +7,25 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
Signature · 1 line
2.40.1
#83 lore raw

Re: [PATCH v4 3/8] fetch: add a test to exercise invalid output formats

Quoted 7 lines · Patrick Steinhardt

Add a testcase that exercises the logic when an invalid output format is passed via the fetch.output configuration.

---
 t/t5574-fetch-output.sh | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

It makes perfect sense to make sure that invalid input gets rejected and the command exits with non-zero status, and it is probably a good thing that the end-user sees a message that explains why the particular input is rejected (even though it adds one more thing that needs to be updated when the message gets reworded).

But do we need to insist on no output to the standard output stream when the command errors out?

Other than that, looking good.

THanks.

Quoted 30 lines
t/t5574-fetch-output.sh +19 -0
index f91b654d38..a09750d225 100755
@@ -7,6 +7,25 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
#84 lore raw

Re: [PATCH v4 3/8] fetch: add a test to exercise invalid output formats

Quoted 16 lines · Junio C Hamano
Quoted 7 lines · Patrick Steinhardt

Add a testcase that exercises the logic when an invalid output format is passed via the fetch.output configuration.

---
 t/t5574-fetch-output.sh | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

It makes perfect sense to make sure that invalid input gets rejected and the command exits with non-zero status, and it is probably a good thing that the end-user sees a message that explains why the particular input is rejected (even though it adds one more thing that needs to be updated when the message gets reworded).

Quoted 4 lines

But do we need to insist on no output to the standard output stream when the command errors out?

Other than that, looking good.

Not really, no. Will change.

Patrick

#85 lore raw

PATCH v4 4/8 fetch: fix missing from-reference when fetching HEAD:foo

In reply to Patrick Steinhardt

store_updated_refs() parses the remote reference for two purposes:

- It gets used as a note when writing FETCH_HEAD.

- It is passed through to display_ref_update() to display updated references in the following format:

```
* branch master -> master
```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

```
$ git fetch origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands show the left-hand side as expected:

```
$ git fetch origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

$ git fetch origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

The logic of how we compute the remote reference name that we ultimately pass to display_ref_update() is not easy to follow. There are three different cases here:

- When the remote reference name is "HEAD" we set the remote reference name to the empty string. This is the case that causes the bug to occur, where we would indeed want to print "HEAD" instead of the empty string. This is what prettify_refname() would return.

- When the remote reference name has a well-known prefix then we strip this prefix. This matches what prettify_refname() does.

- Otherwise, we keep the fully qualified reference name. This also matches what prettify_refname() does.

As the return value of prettify_refname() would do the correct thing for us in all three cases, we can fix the bug by passing through the full remote reference name to display_ref_update(), which learns to call prettify_refname(). At the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

---
 builtin/fetch.c         | 37 +++++++++++++++++++------------------
 t/t5574-fetch-output.sh | 29 +++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 18 deletions(-)
builtin/fetch.c +19 -18
index 08d7fc7233..6aecf549e8 100644
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
t/t5574-fetch-output.sh +29 -0
index a09750d225..6e0f7e0046 100755
@@ -56,6 +56,35 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD:foo >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
Signature · 1 line
2.40.1
#86 lore raw

Re: [PATCH v4 4/8] fetch: fix missing from-reference when fetching HEAD:foo

Quoted 8 lines · Patrick Steinhardt

But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

```
$ git fetch origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```
Quoted 8 lines

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands show the left-hand side as expected:

```

$ git fetch origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

I do not mind being explicit and showing HEAD in this case for the sake of consistency.

But speaking for the past developers, it was deliberate to omit what is common from the output to make it more terse, IIRC, and I think it is unfair to call it a "BUG".

Back when we wrote git-fetch-script, the output was a lot more verbose, and through efforts like 165f3902 (git-fetch: more terse fetch output, 2007-11-03) and numerous others over time, we got to the current output.

Quoted 3 lines

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

Just to help readers, "kind" is the category like branch, tag, etc. and "what" is the concrete name like 'master' and 'foo'.

Quoted 31 lines
builtin/fetch.c +5 -3
index 08d7fc7233..6aecf549e8 100644
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;

builtin/fetch.c static int update_local_ref(struct ref *ref, · line 951

Quoted 5 lines
builtin/fetch.c +1 -1
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);

Makes sense. The variable "remote" (now removed) holds what to write to FETCH_HEAD to be used to formulate a merge message by the caller, but this function is purely to report the ref updates and has no need to have access to that information.

Quoted 9 lines
builtin/fetch.c +1 -3
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))

builtin/fetch.c static int store_updated_refs(struct display_state *display_state, · line 1266

Quoted 10 lines
builtin/fetch.c +3 -2
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}

This is a bit noisier than necessary. It took me a while until I realized that this hunk is a no-op.

builtin/fetch.c static int store_updated_refs(struct display_state *display_state, · line 1283

Quoted 8 lines
builtin/fetch.c +1 -1
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);

Good.

builtin/fetch.c static int store_updated_refs(struct display_state *display_state, · line 1293

Quoted 7 lines
builtin/fetch.c +1 -1
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);

Good, too. The original cleared "what" and then to compensate for that had a logic to turn it back to "HEAD", but that is all gone by passing rm->name down.

I think we could pass "rm" and leave it to display_ref_update() what string to use, if we wanted to further refine the output later.

Then somebody in the future may even want to see "HEAD" to be shown as an empty string and that can all be done in display_ref_update(). It would fix the inconsistency that "git fetch origin HEAD" reports "HEAD -> FETCH_HEAD" by hiding "HEAD" just like the case where fetching "HEAD:foo" does, going in the other direction, I would think.

Thanks.

#87 lore raw

Re: [PATCH v4 4/8] fetch: fix missing from-reference when fetching HEAD:foo

Quoted 22 lines · Junio C Hamano
Quoted 8 lines · Patrick Steinhardt

But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

```
$ git fetch origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```
Quoted 8 lines

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands show the left-hand side as expected:

```

$ git fetch origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

I do not mind being explicit and showing HEAD in this case for the sake of consistency.

Quoted 8 lines

But speaking for the past developers, it was deliberate to omit what is common from the output to make it more terse, IIRC, and I think it is unfair to call it a "BUG".

Back when we wrote git-fetch-script, the output was a lot more verbose, and through efforts like 165f3902 (git-fetch: more terse fetch output, 2007-11-03) and numerous others over time, we got to the current output.

That's fair. It's still not quite clear whether this behaviour is in fact intentional though. Quoting 165f3902 (git-fetch: more terse fetch output, 2007-11-03), the weird corner case is not documented:

    This makes the fetch output much more terse and prettier on a 80 column
    display, based on a consensus reached on the mailing list.  Here's an
    example output:

    Receiving objects: 100% (5439/5439), 1.60 MiB | 636 KiB/s, done.
    Resolving deltas: 100% (4604/4604), done.
    From git://git.kernel.org/pub/scm/git/git
     ! [rejected]        html -> origin/html  (non fast forward)
       136e631..f45e867  maint -> origin/maint  (fast forward)
       9850e2e..44dd7e0  man -> origin/man  (fast forward)
       3e4bb08..e3d6d56  master -> origin/master  (fast forward)
       fa3665c..536f64a  next -> origin/next  (fast forward)
     + 4f6d9d6...768326f pu -> origin/pu  (forced update)
     * [new branch]      todo -> origin/todo

I've reformulated the commit message to talk about an inconsistency instead of a bug.

Patrick

#88 lore raw

PATCH v4 5/8 fetch: introduce `display_format` enum

In reply to Patrick Steinhardt

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a enum display_format that is more readily extensible.

---
 builtin/fetch.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
builtin/fetch.c +64 -37
index 6aecf549e8..9e7e45344d 100644
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -784,7 +790,6 @@ static int refcol_width(const struct ref *ref, int compact_format)
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display format %d", display_state->format);
 	}
 }
 
@@ -904,30 +920,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
Signature · 1 line
2.40.1
#89 lore raw

Re: [PATCH v4 5/8] fetch: introduce `display_format` enum

Quoted 8 lines · Patrick Steinhardt

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a enum display_format that is more readily extensible.

Makes sense.

builtin/fetch.c enum {

Quoted 12 lines
builtin/fetch.c +6 -1
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
 
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;

OK. Preparatory conversion without adding anything new.

builtin/fetch.c static void display_state_init(struct display_state *display_state, struct ref * · line 822

Quoted 12 lines
builtin/fetch.c +2 -2
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);

Naturally.

Quoted 35 lines
builtin/fetch.c +17 -15
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.

builtin/fetch.c static void display_state_init(struct display_state *display_state, struct ref * · line 853

Quoted 10 lines
builtin/fetch.c +9 -0
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display format %d", display_state->format);
 	}

Due to reindentation, the patch is noisier than what it does (which should be "nothing, other than allowing another value in the .format member").

It makes me wonder if it would make it easier to read to move the bulk of this code to a helper function. If we are to give a name to what is being done in the above hunk, what would it be? It computes display->refcol_width in which all records would fit, but presumably if we are to add more things to be shown per ref and align them in a simlar way, we would compute widths for these other things there as well. Perhaps compute_display_alignment() or somesuch?

#90 lore raw

Re: [PATCH v4 5/8] fetch: introduce `display_format` enum

Quoted 1 line · Junio C Hamano

Patrick Steinhardt <[email protected]> writes:

[snip]

Quoted 50 lines
Quoted 35 lines
builtin/fetch.c +17 -15
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.

builtin/fetch.c static void display_state_init(struct display_state *display_state, struct ref * · line 853

Quoted 10 lines
builtin/fetch.c +9 -0
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display format %d", display_state->format);
 	}

Due to reindentation, the patch is noisier than what it does (which should be "nothing, other than allowing another value in the .format member").

Quoted 7 lines

It makes me wonder if it would make it easier to read to move the bulk of this code to a helper function. If we are to give a name to what is being done in the above hunk, what would it be? It computes display->refcol_width in which all records would fit, but presumably if we are to add more things to be shown per ref and align them in a simlar way, we would compute widths for these other things there as well. Perhaps compute_display_alignment() or somesuch?

The code already has such a function and calls it refcol_width(), but it only computes the width for a single reference. The most natural thing to do here would thus be to merge the loop over the references into that function. This would also allow us to skip some weirdness, like the fact that we skip some references inside refcol_width() while we skip other references in the ref_map loop.

This refactoring is also quite noisy, but it makes the code simpler overall and will make the patch introducing the enum less so.

Patrick

#91 lore raw

PATCH v4 6/8 fetch: move display format parsing into main function

In reply to Patrick Steinhardt

Parsing the display format happens inside of display_state_init(). As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the fetch.output config key set. We're thus going to introduce a new --output-format switch for git-fetch(1) so that the output format can be configured more directly.

This means we'll have to hook parsing of the display format into the command line options parser. Having the code to determine the actual output format scattered across two different sites is hard to reason about though.

Refactor the code such that callers are expected to pass the display format that is to be used into display_state_init(). This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

---
 builtin/fetch.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
builtin/fetch.c +24 -17
index 9e7e45344d..e15d43dc1e 100644
@@ -788,14 +788,13 @@ static int refcol_width(const struct ref *ref, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -812,15 +811,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT: {
@@ -1614,7 +1604,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1700,7 +1691,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2078,7 +2069,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2145,7 +2137,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2157,6 +2149,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2183,6 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}
+
 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
 		recurse_submodules = recurse_submodules_cli;
 
@@ -2311,7 +2317,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   display_format);
 	} else {
 		int max_children = max_jobs;
 
Signature · 1 line
2.40.1
#92 lore raw

Re: [PATCH v4 6/8] fetch: move display format parsing into main function

Quoted 6 lines · Patrick Steinhardt

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the fetch.output config key set. We're thus going to introduce a new --output-format switch for git-fetch(1) so that the output format can be configured more directly.

Good. I was wondering about that code in the context of the previous patch, especially the error message had a hard-coded assumption that it comes from a configuration variable.

And the changes to display_state_init() to lift the responsibility of finding and validating .format out of the function, and the changes to intermediate functions to pass the .format through the callchain, are all expected and there was nothing questionable.

Quoted 14 lines
builtin/fetch.c +3 -0
@@ -2157,6 +2149,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2183,6 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";

builtin/fetch.c int cmd_fetch(int argc, const char **argv, const char *prefix) · line 2190

Quoted 10 lines
builtin/fetch.c +10 -0
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}

OK, but isn't the usual way to do this to have configuration parser before parse_options() and then let parse_options() override whatever display_format set by it?

That way, we do not have to worry about DISPLAY_FORMAT_UNKNOWN at all. Just initialize the variable to whatever default format at the beginning of this function, read "fetch.output" to override it if the configuration exists, and then let parse_options() to handle "--output-format" or "--porcelain" or whatever to further override it.

Thanks.

#93 lore raw

Re: [PATCH v4 6/8] fetch: move display format parsing into main function

In reply to Patrick Steinhardt
Quoted 3 lines · Patrick Steinhardt

with the fetch.output config key set. We're thus going to introduce a new --output-format switch for git-fetch(1) so that the output format can be configured more directly.

This is stale as of v3, since isn't named --output-format any more. Let me see if there are other instances of this.

(I should have caught this earlier; I only saw it because Junio quoted it in a reply of his.)

#94 lore raw

Re: [PATCH v4 6/8] fetch: move display format parsing into main function

Quoted 11 lines · Glen Choo
Quoted 3 lines · Patrick Steinhardt

with the fetch.output config key set. We're thus going to introduce a new --output-format switch for git-fetch(1) so that the output format can be configured more directly.

This is stale as of v3, since isn't named --output-format any more. Let me see if there are other instances of this.

(I should have caught this earlier; I only saw it because Junio quoted it in a reply of his.)

Good catch, fixed now.

Patrick

#95 lore raw

PATCH v4 7/8 fetch: move option related variables into main function

In reply to Patrick Steinhardt

The options of git-fetch(1) which we pass to parse_options() are declared globally in builtin/fetch.c. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into cmd_fetch(). Move variables that were previously forced to be declared globally and which are only used by cmd_fetch() into function-local scope.

---
 builtin/fetch.c | 197 ++++++++++++++++++++++++------------------------
 1 file changed, 100 insertions(+), 97 deletions(-)
builtin/fetch.c +100 -97
index e15d43dc1e..820ec7285c 100644
@@ -75,13 +75,12 @@ static int fetch_prune_tags_config = -1; /* unspecified */
 static int prune_tags = -1; /* unspecified */
 #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok;
+static int append, dry_run, force, keep, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
@@ -160,92 +153,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2147,13 +2054,109 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
-	int i;
 	const char *bundle_uri;
+	const char *submodule_prefix = "";
 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
+	int all = 0, multiple = 0;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
Signature · 1 line
2.40.1
#96 lore raw

PATCH v4 8/8 fetch: introduce machine-parseable "porcelain" output format

In reply to Patrick Steinhardt

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```

We assume two conditions which are generally true:

- The old and new object IDs have fixed known widths and cannot contain spaces.

- References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote.

In a similar spirit, a second restriction is that this cannot be used with --recurse-submodules. This is because any reference updates would be ambiguous without also printing the repository in which the update happens.

Considering that both multi-remote and submodule fetches are user-facing features, using them in conjunction with --porcelain that is intended for scripting purposes is likely not going to be useful in the majority of cases. With that in mind these restrictions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

---
 Documentation/fetch-options.txt |   7 ++
 Documentation/git-fetch.txt     |   9 ++
 builtin/fetch.c                 |  91 +++++++++++++++++----
 t/t5574-fetch-output.sh         | 140 ++++++++++++++++++++++++++++++++
 4 files changed, 229 insertions(+), 18 deletions(-)
Documentation/fetch-options.txt +7 -0
index 622bd84768..41fc7ca3c6 100644
@@ -78,6 +78,13 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--porcelain::
+	Print the output to standard output in an easy-to-parse format for
+	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
++
+This is incompatible with `--recurse-submodules=[yes|on-demand]` and takes
+precedence over the `fetch.output` config option.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
Documentation/git-fetch.txt +9 -0
index fba66f1460..f123139c58 100644
@@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+When using `--porcelain`, the output format is intended to be
+machine-parseable. In contrast to the human-readable output formats it
+thus prints to standard output instead of standard error. Each line is
+of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
builtin/fetch.c +73 -18

252 lines not on this page · raw message

t/t5574-fetch-output.sh +140 -0

152 lines not on this page · raw message

Signature · 1 line
2.40.1
#97 lore raw

Re: [PATCH v4 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 6 lines · Patrick Steinhardt

Considering that both multi-remote and submodule fetches are user-facing features, using them in conjunction with --porcelain that is intended for scripting purposes is likely not going to be useful in the majority of cases. With that in mind these restrictions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

Two steps ago, the proposed log message still mentioned "--output-format", which may want to be proofread again and revised.

Quoted 11 lines
builtin/fetch.c +2 -1
@@ -1786,7 +1810,8 @@ static int add_remote_or_group(const char *name, struct string_list *list)
 	return 1;
 }
 
-static void add_options_to_argv(struct strvec *argv)
+static void add_options_to_argv(struct strvec *argv,
+				enum display_format format)
 {
 	if (dry_run)
 		strvec_push(argv, "--dry-run");

builtin/fetch.c static void add_options_to_argv(struct strvec *argv) · line 1852

Quoted 6 lines
builtin/fetch.c +2 -0
@@ -1822,6 +1847,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--ipv6");
 	if (!write_fetch_head)
 		strvec_push(argv, "--no-write-fetch-head");
+	if (format == DISPLAY_FORMAT_PORCELAIN)
+		strvec_pushf(argv, "--porcelain");
 }

Hmph.

[PATCH 9/8] may want to also introduce and pass down the "--output-format=full/compact" option, but that is clearly outside of the scope of this step.

builtin/fetch.c static int fetch_one(struct remote *remote, int argc, const char **argv, · line 2089

Quoted 6 lines
builtin/fetch.c +6 -0
+static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value;
+	*format = DISPLAY_FORMAT_PORCELAIN;
+	return 0;
+}

Lack of "if (unset) ..." worries me. Shouldn't the code allow

	git fetch --porcelain --no-porcelain

to defeat an earlier one and revert back to the default?

Other than that the changes in this step were mostly expected and I didn't spot anything glaringly wrong, but I didn't spend as much time as I did for the other steps.

Thanks.

#98 lore raw

Re: [PATCH v4 8/8] fetch: introduce machine-parseable "porcelain" output format

Quoted 11 lines · Junio C Hamano
Quoted 6 lines · Patrick Steinhardt

Considering that both multi-remote and submodule fetches are user-facing features, using them in conjunction with --porcelain that is intended for scripting purposes is likely not going to be useful in the majority of cases. With that in mind these restrictions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

Two steps ago, the proposed log message still mentioned "--output-format", which may want to be proofread again and revised.

It's a relict from previous iterations. I've fixed the preceding patch to talk about --porcelain instead.

Quoted 20 lines
Quoted 11 lines
builtin/fetch.c +2 -1
@@ -1786,7 +1810,8 @@ static int add_remote_or_group(const char *name, struct string_list *list)
 	return 1;
 }
 
-static void add_options_to_argv(struct strvec *argv)
+static void add_options_to_argv(struct strvec *argv,
+				enum display_format format)
 {
 	if (dry_run)
 		strvec_push(argv, "--dry-run");

builtin/fetch.c static void add_options_to_argv(struct strvec *argv) · line 1852

Quoted 6 lines
builtin/fetch.c +2 -0
@@ -1822,6 +1847,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--ipv6");
 	if (!write_fetch_head)
 		strvec_push(argv, "--no-write-fetch-head");
+	if (format == DISPLAY_FORMAT_PORCELAIN)
+		strvec_pushf(argv, "--porcelain");
 }

Hmph.

Quoted 3 lines

[PATCH 9/8] may want to also introduce and pass down the "--output-format=full/compact" option, but that is clearly outside of the scope of this step.

We don't have --output-format now though, but only the --porcelain switch. The only way to configure "full"/"compact" output formats is via the "fetch.output" config variable right now.

Quoted 9 lines

builtin/fetch.c static int fetch_one(struct remote *remote, int argc, const char **argv, · line 2089

Quoted 6 lines
builtin/fetch.c +6 -0
+static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value;
+	*format = DISPLAY_FORMAT_PORCELAIN;
+	return 0;
+}

Lack of "if (unset) ..." worries me. Shouldn't the code allow

Quoted 3 lines
	git fetch --porcelain --no-porcelain

to defeat an earlier one and revert back to the default?

We pass PARSE_OPT_NOARG|PARSE_OPT_NONEG, so this isn't really much of a concern. But yeah, it wouldn't allow --no-porcelain, and supporting it is trivial enough to do. I'll change this to a OPT_BOOL().

Patrick

#99 lore raw

v5 cover fetch: introduce machine-parseable output

In reply to Patrick Steinhardt

Hi,

this is the fourth version of my patch series to introduce a machine-parseable output format for git-fetch(1). It applies on top of e9dffbc7f1 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06).

Changes compared to v4:

- Patch 1/9: Simplified the test as proposed by Junio and Glen.

- Patch 3/9: Added a test to verify that git fetch -c fetch.output without a value set fails as expected. Also dropped the tests that checked whether stdout was empty.

- Patch 4/9: Reformulated the commit message to treat the missing left-hand side of displayed references as an inconsistency instead of a bug. I've also added a testcase to verify that direct OID fetches continue to work as expected.

- Patch 5/9: New patch that makes calculation of the table width for displayed reference updates self-contained in refcol_width(). This is a preparatory refactoring that should make patch 6/9 easier to review.

- Patch 7/9: Refactored the code to parse the "fetch.output" config variable inside of git_fetch_config() before we parse command line options. Also fixed that the commit message was still referring to --output-format=porcelain instead of the new --porcelain switch.

- Patch 9/9: The --porcelain option is now a simple OPT_BOOL() that can be negated. Added a test that --no-porcelain works as expected.

Thanks for your feedback, Junio and Glen!

Patrick

Patrick Steinhardt (9):
fetch: fix --no-recurse-submodules with multi-remote fetches
fetch: split out tests for output format
fetch: add a test to exercise invalid output formats
fetch: print left-hand side when fetching HEAD:foo
fetch: refactor calculation of the display table width
fetch: introduce display_format enum
fetch: lift up parsing of "fetch.output" config variable
fetch: move option related variables into main function
fetch: introduce machine-parseable "porcelain" output format

 Documentation/fetch-options.txt |   7 +
 Documentation/git-fetch.txt     |   9 +
 builtin/fetch.c                 | 490 +++++++++++++++++++-------------
 t/t5510-fetch.sh                |  53 ----
 t/t5526-fetch-submodules.sh     |  13 +
 t/t5574-fetch-output.sh         | 293 +++++++++++++++++++
 6 files changed, 611 insertions(+), 254 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh

Range-diff against v4:
1: d82b42ed34 ! 1: 02ee4fab7e fetch: fix --no-recurse-submodules with multi-remote fetches

    @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetch --all with --recurse-sub
     +		cd src_clone &&
     +		git remote add secondary ../src &&
     +		git config submodule.recurse true &&
    -+		git config fetch.parallel 0 &&
    -+		git fetch --all --no-recurse-submodules 2>../actual
    ++		git fetch --all --no-recurse-submodules 2>../fetch-log
     +	) &&
    -+
    -+	cat >expect <<-EOF &&
    -+	From ../src
    -+	 * [new branch]      master     -> secondary/master
    -+	EOF
    -+	test_cmp expect actual
    ++	! grep "Fetching submodule" fetch-log
     +'
     +
      test_done
 2:  33112dc51a =  2:  e459d8a1ec fetch: split out tests for output format
 3:  006ea93afb !  3:  d503c425fe fetch: add a test to exercise invalid output formats
    @@ t/t5574-fetch-output.sh: export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
     +	test_when_finished "rm -rf clone" &&
     +	git clone . clone &&
     +
    -+	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
    ++	test_must_fail git -C clone -c fetch.output fetch origin 2>actual.err &&
    ++	cat >expect <<-EOF &&
    ++	error: missing value for ${SQ}fetch.output${SQ}
    ++	fatal: unable to parse ${SQ}fetch.output${SQ} from command-line config
    ++	EOF
    ++	test_cmp expect actual.err &&
    ++
    ++	test_must_fail git -C clone -c fetch.output= fetch origin 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
     +	EOF
    -+	test_must_be_empty actual.out &&
     +	test_cmp expect actual.err &&
     +
    -+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
    ++	test_must_fail git -C clone -c fetch.output=garbage fetch origin 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
     +	EOF
    -+	test_must_be_empty actual.out &&
     +	test_cmp expect actual.err
     +'
     +

4: e599ea6d33 ! 4: 2cc7318697 fetch: fix missing from-reference when fetching HEAD:foo
@@ Metadata
Author: Patrick Steinhardt <[email protected]>

## Commit message ##
- fetch: fix missing from-reference when fetching HEAD:foo
+ fetch: print left-hand side when fetching HEAD:foo

store_updated_refs() parses the remote reference for two purposes:

@@ Commit message
name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case.
- But it is not intended when displaying the updated references and would
- cause us to miss the left-hand side of the displayed reference update:
+ But when displaying the updated references this leads to inconsistent
+ output where the left-hand side of reference updates is missing in some
+ cases:

```
- $ git fetch origin HEAD:foo
- From https://github.com/git/git
- * [new ref] -> foo
- ```
-
- The HEAD string is clearly missing from the left-hand side of the arrow,
- which is further stressed by the point that the following commands show
- the left-hand side as expected:
-
- ```
- $ git fetch origin HEAD
+ $ git fetch origin HEAD HEAD:explicit-head :implicit-head main
From https://github.com/git/git
* branch HEAD -> FETCH_HEAD
-
- $ git fetch origin master
- From https://github.com/git/git
- * branch master -> FETCH_HEAD
- * branch master -> origin/master
+ * [new ref] -> explicit-head
+ * [new ref] -> implicit-head
+ * branch main -> FETCH_HEAD
```

+ This behaviour has existed ever since the table-based output has been
+ introduced for git-fetch(1) via 165f390250 (git-fetch: more terse fetch
+ output, 2007-11-03) and was never explicitly documented either in the
+ commit message or in any of our tests. So while it may not be a bug per
+ se, it feels like a weird inconsistency and not like it was a concious
+ design decision.
+
The logic of how we compute the remote reference name that we ultimately pass to display_ref_update() is not easy to follow. There are three different cases here:

- When the remote reference name is "HEAD" we set the remote reference name to the empty string. This is the case that causes
- the bug to occur, where we would indeed want to print "HEAD"
- instead of the empty string. This is what prettify_refname()
- would return.
+ the left-hand side to go missing, where we would indeed want to
+ print "HEAD" instead of the empty string. This is what
+ prettify_refname() would return.

- When the remote reference name has a well-known prefix then we strip this prefix. This matches what prettify_refname() does.
@@ Commit message
matches what prettify_refname() does.

As the return value of prettify_refname() would do the correct thing
- for us in all three cases, we can fix the bug by passing through the
- full remote reference name to display_ref_update(), which learns to
- call prettify_refname(). At the same time, this also simplifies the
- code a bit.
+ for us in all three cases, we can thus fix the inconsistency by passing
+ through the full remote reference name to display_ref_update(), which
+ learns to call prettify_refname(). At the same time, this also
+ simplifies the code a bit.

Note that this patch also changes formatting of the block that computes
- the "kind" and "what" variables. This is done on purpose so that it is
- part of the diff, hopefully making the change easier to comprehend.
+ the "kind" (which is the category like "branch" or "tag") and "what"
+ (which is the prettified reference name like "master" or "v1.0")
+ variables. This is done on purpose so that it is part of the diff,
+ hopefully making the change easier to comprehend.

@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
+ test_must_be_empty actual.out &&
+ test_cmp expect actual.err
+'
++
++test_expect_success 'fetch output with object ID' '
++ test_when_finished "rm -rf object-id" &&
++ git clone . object-id &&
++ commit=$(git rev-parse HEAD) &&
++
++ git -C object-id fetch --dry-run origin $commit:object-id >actual.out 2>actual.err &&
++ cat >expect <<-EOF &&
++ From $(test-tool path-utils real_path .)/.
++ * [new ref] $commit -> object-id
++ EOF
++ test_must_be_empty actual.out &&
++ test_cmp expect actual.err &&
++
++ git -C object-id fetch origin $commit:object-id >actual.out 2>actual.err &&
++ test_must_be_empty actual.out &&
++ test_cmp expect actual.err
++'
+
test_expect_success '--no-show-forced-updates' '
mkdir forced-updates &&

 -:  ---------- >  5:  bb1a591c2f fetch: refactor calculation of the display table width
 5:  80ac00b0c4 !  6:  3cac552f5f fetch: introduce `display_format` enum
    @@ builtin/fetch.c: enum {
      
      	char *url;
      	int url_len, shown_url;
    -@@ builtin/fetch.c: static int refcol_width(const struct ref *ref, int compact_format)
    - static void display_state_init(struct display_state *display_state, struct ref *ref_map,
    - 			       const char *raw_url)
    - {
    --	struct ref *rm;
    - 	const char *format = "full";
    - 	int i;
    - 
     @@ builtin/fetch.c: static void display_state_init(struct display_state *display_state, struct ref *
      
      	git_config_get_string_tmp("fetch.output", &format);
    @@ builtin/fetch.c: static void display_state_init(struct display_state *display_st
      		die(_("invalid value for '%s': '%s'"),
      		    "fetch.output", format);
      
    --	display_state->refcol_width = 10;
    --	for (rm = ref_map; rm; rm = rm->next) {
    --		int width;
    +-	display_state->refcol_width = refcol_width(ref_map, display_state->compact_format);
     +	switch (display_state->format) {
     +	case DISPLAY_FORMAT_FULL:
    -+	case DISPLAY_FORMAT_COMPACT: {
    -+		struct ref *rm;
    - 
    --		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
    --		    !rm->peer_ref ||
    --		    !strcmp(rm->name, "HEAD"))
    --			continue;
    -+		display_state->refcol_width = 10;
    -+		for (rm = ref_map; rm; rm = rm->next) {
    -+			int width;
    - 
    --		width = refcol_width(rm, display_state->compact_format);
    -+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
    -+			    !rm->peer_ref ||
    -+			    !strcmp(rm->name, "HEAD"))
    -+				continue;
    - 
    --		/*
    --		 * Not precise calculation for compact mode because '*' can
    --		 * appear on the left hand side of '->' and shrink the column
    --		 * back.
    --		 */
    --		if (display_state->refcol_width < width)
    --			display_state->refcol_width = width;
    -+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
    -+
    -+			/*
    -+			 * Not precise calculation for compact mode because '*' can
    -+			 * appear on the left hand side of '->' and shrink the column
    -+			 * back.
    -+			 */
    -+			if (display_state->refcol_width < width)
    -+				display_state->refcol_width = width;
    -+		}
    -+
    ++	case DISPLAY_FORMAT_COMPACT:
    ++		display_state->refcol_width = refcol_width(ref_map,
    ++							   display_state->format == DISPLAY_FORMAT_COMPACT);
     +		break;
    -+	}
     +	default:
     +		BUG("unexpected display format %d", display_state->format);
    - 	}
    ++	}
      }
      
    + static void display_state_release(struct display_state *display_state)
     @@ builtin/fetch.c: static void display_ref_update(struct display_state *display_state, char code,
      			       const char *remote, const char *local,
      			       int summary_width)

6: 826b8b7bc0 ! 7: 0c3dbcd09f fetch: move display format parsing into main function
@@ Metadata
Author: Patrick Steinhardt <[email protected]>

## Commit message ##
- fetch: move display format parsing into main function
+ fetch: lift up parsing of "fetch.output" config variable

Parsing the display format happens inside of display_state_init(). As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

- We're about to introduce a output format though that is intended to be
- parseable by machines, for example inside of a script. In that case it
- becomes a bit awkward of an interface if you have to call git-fetch(1)
- with the fetch.output config key set. We're thus going to introduce a
- new --output-format switch for git-fetch(1) so that the output format
- can be configured more directly.
-
- This means we'll have to hook parsing of the display format into the
- command line options parser. Having the code to determine the actual
- output format scattered across two different sites is hard to reason
- about though.
+ We're about to introduce a new "porcelain" output format though that is
+ intended to be parseable by machines, for example inside of a script.
+ This format can be enabled by passing the --porcelain switch to
+ git-fetch(1). As a consequence, we'll have to add a second callsite that
+ influences the output format, which will become awkward to handle.

Refactor the code such that callers are expected to pass the display format that is to be used into display_state_init(). This allows us to
@@ Commit message

## builtin/fetch.c ##
-@@ builtin/fetch.c: static int refcol_width(const struct ref *ref, int compact_format) +@@ builtin/fetch.c: static int fetch_write_commit_graph = -1;
+ static int stdin_refspecs = 0;
+ static int negotiate_only;
+
++struct fetch_config {
++ enum display_format display_format;
++};
++
+ static int git_fetch_config(const char *k, const char *v, void *cb)
+ {
++ struct fetch_config *fetch_config = cb;
++
+ if (!strcmp(k, "fetch.prune")) {
+ fetch_prune_config = git_config_bool(k, v);
+ return 0;
+@@ builtin/fetch.c: static int git_fetch_config(const char *k, const char *v, void *cb)
+ return 0;
+ }
+
++ if (!strcmp(k, "fetch.output")) {
++ if (!v)
++ return config_error_nonbool(k);
++ else if (!strcasecmp(v, "full"))
++ fetch_config->display_format = DISPLAY_FORMAT_FULL;
++ else if (!strcasecmp(v, "compact"))
++ fetch_config->display_format = DISPLAY_FORMAT_COMPACT;
++ else
++ die(_("invalid value for '%s': '%s'"),
++ "fetch.output", v);
++ }
++
+ return git_default_config(k, v, cb);
+ }
+
+@@ builtin/fetch.c: static int refcol_width(const struct ref *ref_map, int compact_format)
}

static void display_state_init(struct display_state *display_state, struct ref *ref_map,
@@ builtin/fetch.c: static void display_state_init(struct display_state *display_st
-
switch (display_state->format) {
case DISPLAY_FORMAT_FULL:
- case DISPLAY_FORMAT_COMPACT: {
+ case DISPLAY_FORMAT_COMPACT:
@@ builtin/fetch.c: static int backfill_tags(struct display_state *display_state,
}

@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const cha
sigchain_pop(SIGPIPE);
refspec_clear(&rs);
transport_disconnect(gtransport);
-@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix) +@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv, +
+ int cmd_fetch(int argc, const char **argv, const char *prefix)
{
++ struct fetch_config config = {
++ .display_format = DISPLAY_FORMAT_FULL,
++ };
int i;
const char *bundle_uri;
-+ enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
struct string_list list = STRING_LIST_INIT_DUP;
- struct remote *remote = NULL;
- int result = 0;
@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
- argc = parse_options(argc, argv, prefix,
- builtin_fetch_options, builtin_fetch_usage, 0);
-
-+ if (display_format == DISPLAY_FORMAT_UNKNOWN) {
-+ const char *format = "full";
-+
-+ git_config_get_string_tmp("fetch.output", &format);
-+ if (!strcasecmp(format, "full"))
-+ display_format = DISPLAY_FORMAT_FULL;
-+ else if (!strcasecmp(format, "compact"))
-+ display_format = DISPLAY_FORMAT_COMPACT;
-+ else
-+ die(_("invalid value for '%s': '%s'"),
-+ "fetch.output", format);
-+ }
-+
- if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
- recurse_submodules = recurse_submodules_cli;
+ free(anon);
+ }

+- git_config(git_fetch_config, NULL);
++ git_config(git_fetch_config, &config);
+ if (the_repository->gitdir) {
+ prepare_repo_settings(the_repository);
+ the_repository->settings.command_requires_full_index = 0;
@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
} else if (remote) {
if (filter_options.choice || has_promisor_remote())
fetch_one_setup_partial(remote);
- result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+ result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
-+ display_format);
++ config.display_format);
} else {
int max_children = max_jobs;

7: 20f2e061d6 ! 8: 8e33a08c35 fetch: move option related variables into main function
@@ builtin/fetch.c: static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
-static int stdin_refspecs = 0;
-static int negotiate_only;

- static int git_fetch_config(const char *k, const char *v, void *cb)
- {
+ struct fetch_config {
+ enum display_format display_format;
@@ builtin/fetch.c: static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
return 0;
}
@@ builtin/fetch.c: static int parse_refmap_arg(const struct option *opt, const cha
static void unlock_pack(unsigned int flags)
{
if (gtransport)
-@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv, -
- int cmd_fetch(int argc, const char **argv, const char *prefix)
- {
+@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
+ struct fetch_config config = {
+ .display_format = DISPLAY_FORMAT_FULL,
+ };
- int i;
- const char *bundle_uri;
+ const char *submodule_prefix = "";
- enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
+ const char *bundle_uri;
struct string_list list = STRING_LIST_INIT_DUP;
struct remote *remote = NULL;
+ int all = 0, multiple = 0;
8: 24ae381950 ! 9: d49152c220 fetch: introduce machine-parseable "porcelain" output format

    @@ builtin/fetch.c: enum display_format {
      
      struct display_state {
     @@ builtin/fetch.c: static void display_state_init(struct display_state *display_state, struct ref *
    - 
    + 		display_state->refcol_width = refcol_width(ref_map,
    + 							   display_state->format == DISPLAY_FORMAT_COMPACT);
      		break;
    - 	}
     +	case DISPLAY_FORMAT_PORCELAIN:
     +		/* We don't need to precompute anything here. */
     +		break;
    @@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_chi
      				printf(_("Fetching %s\n"), name);
      			cmd.git_cmd = 1;
      			if (run_command(&cmd)) {
    -@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv,
    - 	return exit_code;
    - }
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 	int fetch_write_commit_graph = -1;
    + 	int stdin_refspecs = 0;
    + 	int negotiate_only = 0;
    ++	int porcelain = 0;
    + 	int i;
      
    -+static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
    -+{
    -+	enum display_format *format = opt->value;
    -+	*format = DISPLAY_FORMAT_PORCELAIN;
    -+	return 0;
    -+}
    -+
    - int cmd_fetch(int argc, const char **argv, const char *prefix)
    - {
    - 	const char *bundle_uri;
    + 	struct option builtin_fetch_options[] = {
     @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
      		OPT_BOOL(0, "dry-run", &dry_run,
      			 N_("dry run")),
    -+		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
    -+			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
    ++		OPT_BOOL(0, "porcelain", &porcelain, N_("machine-readable output")),
      		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
      			 N_("write fetched references to the FETCH_HEAD file")),
      		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      		fetch_config_from_gitmodules(sfjc, rs);
      	}
      
    -+	if (display_format == DISPLAY_FORMAT_PORCELAIN) {
    ++
    ++	if (porcelain) {
     +		switch (recurse_submodules_cli) {
     +		case RECURSE_SUBMODULES_OFF:
     +		case RECURSE_SUBMODULES_DEFAULT:
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
     +			die(_("options '%s' and '%s' cannot be used together"),
     +			    "--porcelain", "--recurse-submodules");
     +		}
    ++
    ++		config.display_format = DISPLAY_FORMAT_PORCELAIN;
     +	}
     +
      	if (negotiate_only && !negotiation_tip.nr)
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      
      		/* TODO should this also die if we have a previous partial-clone? */
     -		result = fetch_multiple(&list, max_children);
    -+		result = fetch_multiple(&list, max_children, display_format);
    ++		result = fetch_multiple(&list, max_children, config.display_format);
      	}
      
    - 
    +-
    + 	/*
    + 	 * This is only needed after fetch_one(), which does not fetch
    + 	 * submodules by itself.
     @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      		if (max_children < 0)
      			max_children = fetch_parallel_config;
      
     -		add_options_to_argv(&options);
    -+		add_options_to_argv(&options, display_format);
    ++		add_options_to_argv(&options, config.display_format);
      		result = fetch_submodules(the_repository,
      					  &options,
      					  submodule_prefix,
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +	test_must_be_empty stderr &&
     +	test_cmp expect stdout
     +'
    ++
    ++test_expect_success 'fetch --no-porcelain overrides previous --porcelain' '
    ++	test_when_finished "rm -rf no-porcelain" &&
    ++
    ++	git switch --create no-porcelain &&
    ++	git clone . no-porcelain &&
    ++	test_commit --no-tag no-porcelain &&
    ++	old_commit=$(git rev-parse --short HEAD~) &&
    ++	new_commit=$(git rev-parse --short HEAD) &&
    ++
    ++	cat >expect <<-EOF &&
    ++	From $(test-tool path-utils real_path .)/.
    ++	   $old_commit..$new_commit  no-porcelain -> origin/no-porcelain
    ++	EOF
    ++
    ++	git -C no-porcelain fetch --porcelain --no-porcelain >stdout 2>stderr &&
    ++	test_cmp expect stderr &&
    ++	test_must_be_empty stdout
    ++'
     +
      test_expect_success 'fetch output with HEAD' '
      	test_when_finished "rm -rf head" &&
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch output with HEAD' '
     +	test_cmp expect actual
     +'
     +
    - test_expect_success '--no-show-forced-updates' '
    - 	mkdir forced-updates &&
    - 	(
    + test_expect_success 'fetch output with object ID' '
    + 	test_when_finished "rm -rf object-id" &&
    + 	git clone . object-id &&
Signature · 1 line
2.40.1
#100 lore raw

PATCH v5 1/9 fetch: fix `--no-recurse-submodules` with multi-remote fetches

When running git fetch --no-recurse-submodules, the exectation is that we don't fetch any submodules. And while this works for fetches of a single remote, it doesn't when fetching multiple remotes at once. The result is that we do recurse into submodules even though the user has explicitly asked us not to.

This is because while we pass on --recurse-submodules={yes,on-demand} if specified by the user, we don't pass on --no-recurse-submodules to the subprocess spawned to perform the submodule fetch.

Fix this by also forwarding this flag as expected.

---
 builtin/fetch.c             |  2 ++
 t/t5526-fetch-submodules.sh | 13 +++++++++++++
 2 files changed, 15 insertions(+)
builtin/fetch.c +2 -0
index c310d89878..08d7fc7233 100644
@@ -1876,6 +1876,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--keep");
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		strvec_push(argv, "--recurse-submodules");
+	else if (recurse_submodules == RECURSE_SUBMODULES_OFF)
+		strvec_push(argv, "--no-recurse-submodules");
 	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
 		strvec_push(argv, "--recurse-submodules=on-demand");
 	if (tags == TAGS_SET)
t/t5526-fetch-submodules.sh +13 -0
index dcdbe26a08..26e933f93a 100755
@@ -1180,4 +1180,17 @@ test_expect_success 'fetch --all with --recurse-submodules with multiple' '
 	test_line_count = 2 fetch-subs
 '
 
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "rm -rf src_clone" &&
+
+	git clone --recurse-submodules src src_clone &&
+	(
+		cd src_clone &&
+		git remote add secondary ../src &&
+		git config submodule.recurse true &&
+		git fetch --all --no-recurse-submodules 2>../fetch-log
+	) &&
+	! grep "Fetching submodule" fetch-log
+'
+
 test_done
Signature · 1 line
2.40.1
#101 lore raw

PATCH v5 2/9 fetch: split out tests for output format

In reply to Patrick Steinhardt

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
t/t5510-fetch.sh +0 -53
index dc44da9c79..4f289063ce 100755
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
t/t5574-fetch-output.sh new file +63 -0

63 lines not on this page · raw message

Signature · 1 line
2.40.1
#102 lore raw

PATCH v5 3/9 fetch: add a test to exercise invalid output formats

In reply to Patrick Steinhardt

Add a testcase that exercises the logic when an invalid output format is passed via the fetch.output configuration.

---
 t/t5574-fetch-output.sh | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
t/t5574-fetch-output.sh +24 -0
index f91b654d38..8a344e6790 100755
@@ -7,6 +7,30 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output fetch origin 2>actual.err &&
+	cat >expect <<-EOF &&
+	error: missing value for ${SQ}fetch.output${SQ}
+	fatal: unable to parse ${SQ}fetch.output${SQ} from command-line config
+	EOF
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_cmp expect actual.err
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
Signature · 1 line
2.40.1
#103 lore raw

PATCH v5 4/9 fetch: print left-hand side when fetching HEAD:foo

In reply to Patrick Steinhardt

store_updated_refs() parses the remote reference for two purposes:

- It gets used as a note when writing FETCH_HEAD.

- It is passed through to display_ref_update() to display updated references in the following format:

```
* branch master -> master
```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But when displaying the updated references this leads to inconsistent output where the left-hand side of reference updates is missing in some cases:

```
$ git fetch origin HEAD HEAD:explicit-head :implicit-head main
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD
 * [new ref]                          -> explicit-head
 * [new ref]                          -> implicit-head
 * branch                  main       -> FETCH_HEAD
```

This behaviour has existed ever since the table-based output has been introduced for git-fetch(1) via 165f390250 (git-fetch: more terse fetch output, 2007-11-03) and was never explicitly documented either in the commit message or in any of our tests. So while it may not be a bug per se, it feels like a weird inconsistency and not like it was a concious design decision.

The logic of how we compute the remote reference name that we ultimately pass to display_ref_update() is not easy to follow. There are three different cases here:

- When the remote reference name is "HEAD" we set the remote reference name to the empty string. This is the case that causes the left-hand side to go missing, where we would indeed want to print "HEAD" instead of the empty string. This is what prettify_refname() would return.

- When the remote reference name has a well-known prefix then we strip this prefix. This matches what prettify_refname() does.

- Otherwise, we keep the fully qualified reference name. This also matches what prettify_refname() does.

As the return value of prettify_refname() would do the correct thing for us in all three cases, we can thus fix the inconsistency by passing through the full remote reference name to display_ref_update(), which learns to call prettify_refname(). At the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" (which is the category like "branch" or "tag") and "what" (which is the prettified reference name like "master" or "v1.0") variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

---
 builtin/fetch.c         | 37 ++++++++++++++++----------------
 t/t5574-fetch-output.sh | 47 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 18 deletions(-)
builtin/fetch.c +19 -18
index 08d7fc7233..6aecf549e8 100644
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
t/t5574-fetch-output.sh +47 -0

53 lines not on this page · raw message

Signature · 1 line
2.40.1
#104 lore raw

Re: [PATCH v5 4/9] fetch: print left-hand side when fetching HEAD:foo

Quoted 6 lines · Patrick Steinhardt

This behaviour has existed ever since the table-based output has been introduced for git-fetch(1) via 165f390250 (git-fetch: more terse fetch output, 2007-11-03) and was never explicitly documented either in the commit message or in any of our tests. So while it may not be a bug per se, it feels like a weird inconsistency and not like it was a concious design decision.

The change seems well-justified. I agree that it does feel strange and unintended. I was really surprised to learn that it was not a bug.

#105 lore raw

Re: [PATCH v5 4/9] fetch: print left-hand side when fetching HEAD:foo

In reply to Patrick Steinhardt

builtin/fetch.c static int update_local_ref(struct ref *ref, · line 953

Quoted 8 lines · Patrick Steinhardt
builtin/fetch.c +1 -1
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}

Here (and in other hunks) we now dereference remote_ref unconditionally. But in existing parts of the code, we guard against remote_ref being NULL. E.g., later on:

          if (!current || !updated) {
                  const char *msg;
                  const char *what;
                  int r;
                  /*
                   * Nicely describe the new ref we're fetching.
                   * Base this on the remote's ref name, as it's
                   * more likely to follow a standard layout.
                   */
                  const char *name = remote_ref ? remote_ref->name : "";
		  [...]

I'm not sure if the old code was being overly defensive, or if the new code is ripe for a segfault. But it's probably worth looking into (it was noticed by Coverity).

Looking at the caller, it is always store_update_refs() which passes its own "rm", a pointer iterating over ref_map. And it should always be non-NULL, since that's the loop condition.

So I think your code is fine, but you might want to double-check my logic. (And it may be worth cleaning up the existing redundant check to prevent confusion).

-Peff

#106 lore raw

Re: [PATCH v5 4/9] fetch: print left-hand side when fetching HEAD:foo

Quoted 31 lines · Jeff King

builtin/fetch.c static int update_local_ref(struct ref *ref, · line 953

Quoted 8 lines · Patrick Steinhardt
builtin/fetch.c +1 -1
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}

Here (and in other hunks) we now dereference remote_ref unconditionally. But in existing parts of the code, we guard against remote_ref being NULL. E.g., later on:

          if (!current || !updated) {
                  const char *msg;
                  const char *what;
                  int r;
                  /*
                   * Nicely describe the new ref we're fetching.
                   * Base this on the remote's ref name, as it's
                   * more likely to follow a standard layout.
                   */
                  const char *name = remote_ref ? remote_ref->name : "";
		  [...]

I'm not sure if the old code was being overly defensive, or if the new code is ripe for a segfault. But it's probably worth looking into (it was noticed by Coverity).

Quoted 7 lines

Looking at the caller, it is always store_update_refs() which passes its own "rm", a pointer iterating over ref_map. And it should always be non-NULL, since that's the loop condition.

So I think your code is fine, but you might want to double-check my logic. (And it may be worth cleaning up the existing redundant check to prevent confusion).

I really think that the code is overly defensive. As you mention, there is a single caller of update_local_ref(), only, and that caller already dereferences the remote reference unconditionally anyway. So if there was any way for rm to become NULL we'd already have a segfault earlier than in update_local_ref().

I'll send a follow-up patch series after the dust has settled that removes the check.

Thanks!

Patrick

#107 lore raw

PATCH v5 5/9 fetch: refactor calculation of the display table width

In reply to Patrick Steinhardt

When displaying reference updates, we try to print the references in a neat table. As the table's width is determined its contents we thus need to precalculate the overall width before we can start printing updated references.

The calculation is driven by display_state_init(), which invokes refcol_width() for every reference that is to be printed. This split is somewhat confusing. For one, we filter references that shall be attributed to the overall width in both places. And second, we needlessly recalculate the maximum line length based on the terminal columns and display format for every reference.

Refactor the code so that the complete width calculations are neatly contained in refcol_width().

---
 builtin/fetch.c | 81 ++++++++++++++++++++++---------------------------
 1 file changed, 37 insertions(+), 44 deletions(-)
builtin/fetch.c +37 -44
index 6aecf549e8..007eb3693d 100644
@@ -751,40 +751,51 @@ static int s_update_ref(const char *action,
 	return ret;
 }
 
-static int refcol_width(const struct ref *ref, int compact_format)
+static int refcol_width(const struct ref *ref_map, int compact_format)
 {
-	int max, rlen, llen, len;
+	const struct ref *ref;
+	int max, width = 10;
 
-	/* uptodate lines are only shown on high verbosity level */
-	if (verbosity <= 0 && oideq(&ref->peer_ref->old_oid, &ref->old_oid))
-		return 0;
-
-	max    = term_columns();
-	rlen   = utf8_strwidth(prettify_refname(ref->name));
-
-	llen   = utf8_strwidth(prettify_refname(ref->peer_ref->name));
-
-	/*
-	 * rough estimation to see if the output line is too long and
-	 * should not be counted (we can't do precise calculation
-	 * anyway because we don't know if the error explanation part
-	 * will be printed in update_local_ref)
-	 */
-	if (compact_format) {
-		llen = 0;
+	max = term_columns();
+	if (compact_format)
 		max = max * 2 / 3;
-	}
-	len = 21 /* flag and summary */ + rlen + 4 /* -> */ + llen;
-	if (len >= max)
-		return 0;
 
-	return rlen;
+	for (ref = ref_map; ref; ref = ref->next) {
+		int rlen, llen = 0, len;
+
+		if (ref->status == REF_STATUS_REJECT_SHALLOW ||
+		    !ref->peer_ref ||
+		    !strcmp(ref->name, "HEAD"))
+			continue;
+
+		/* uptodate lines are only shown on high verbosity level */
+		if (verbosity <= 0 && oideq(&ref->peer_ref->old_oid, &ref->old_oid))
+			continue;
+
+		rlen = utf8_strwidth(prettify_refname(ref->name));
+		if (!compact_format)
+			llen = utf8_strwidth(prettify_refname(ref->peer_ref->name));
+
+		/*
+		 * rough estimation to see if the output line is too long and
+		 * should not be counted (we can't do precise calculation
+		 * anyway because we don't know if the error explanation part
+		 * will be printed in update_local_ref)
+		 */
+		len = 21 /* flag and summary */ + rlen + 4 /* -> */ + llen;
+		if (len >= max)
+			continue;
+
+		if (width < rlen)
+			width = rlen;
+	}
+
+	return width;
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -816,25 +827,7 @@ static void display_state_init(struct display_state *display_state, struct ref *
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
-
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
-
-		width = refcol_width(rm, display_state->compact_format);
-
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
-	}
+	display_state->refcol_width = refcol_width(ref_map, display_state->compact_format);
 }
 
 static void display_state_release(struct display_state *display_state)
Signature · 1 line
2.40.1
#108 lore raw

Re: [PATCH v5 5/9] fetch: refactor calculation of the display table width

Quoted 12 lines · Patrick Steinhardt

When displaying reference updates, we try to print the references in a neat table. As the table's width is determined its contents we thus need to precalculate the overall width before we can start printing updated references.

The calculation is driven by display_state_init(), which invokes refcol_width() for every reference that is to be printed. This split is somewhat confusing. For one, we filter references that shall be attributed to the overall width in both places. And second, we needlessly recalculate the maximum line length based on the terminal columns and display format for every reference.

Quoted 2 lines

Refactor the code so that the complete width calculations are neatly contained in refcol_width().

Through no fault of yours, I have to admit that I found this refactor quite hard to read. I ended up redoing the refactor and ended up with a result very similar to yours. That was probably overkill since we have pretty extensive tests in this area, but I'm quite happy with the change since it's far more readable.

Quoted 5 lines
builtin/fetch.c +0 -5
-	/*
-	 * rough estimation to see if the output line is too long and
-	 * should not be counted (we can't do precise calculation
-	 * anyway because we don't know if the error explanation part
-	 * will be printed in update_local_ref)

builtin/fetch.c static int s_update_ref(const char *action, · line 779

Quoted 10 lines
builtin/fetch.c +2 -7
-	 */
-	if (compact_format) {
-		llen = 0;
+	max = term_columns();
+	if (compact_format)
 		max = max * 2 / 3;
-	}
-	len = 21 /* flag and summary */ + rlen + 4 /* -> */ + llen;
-	if (len >= max)
-		return 0;

The one thing that changed for the better (vs keeping the lines in the same order as before) is that this comment that used to be anchored on "if (compact_format) {"...

builtin/fetch.c static int s_update_ref(const char *action, · line 787

Quoted 9 lines
builtin/fetch.c +9 -0
+		/*
+		 * rough estimation to see if the output line is too long and
+		 * should not be counted (we can't do precise calculation
+		 * anyway because we don't know if the error explanation part
+		 * will be printed in update_local_ref)
+		 */
+		len = 21 /* flag and summary */ + rlen + 4 /* -> */ + llen;
+		if (len >= max)
+			continue;

is now more accurately anchored to the check that throws away the line.

Looks good.

#109 lore raw

PATCH v5 6/9 fetch: introduce `display_format` enum

In reply to Patrick Steinhardt

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a enum display_format that is more readily extensible.

---
 builtin/fetch.c | 69 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 47 insertions(+), 22 deletions(-)
builtin/fetch.c +47 -22
index 007eb3693d..4c58ee503a 100644
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -820,14 +826,22 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = refcol_width(ref_map, display_state->compact_format);
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT:
+		display_state->refcol_width = refcol_width(ref_map,
+							   display_state->format == DISPLAY_FORMAT_COMPACT);
+		break;
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	}
 }
 
 static void display_state_release(struct display_state *display_state)
@@ -897,30 +911,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
Signature · 1 line
2.40.1
#110 lore raw

PATCH v5 7/9 fetch: lift up parsing of "fetch.output" config variable

In reply to Patrick Steinhardt

Parsing the display format happens inside of display_state_init(). As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a new "porcelain" output format though that is intended to be parseable by machines, for example inside of a script. This format can be enabled by passing the --porcelain switch to git-fetch(1). As a consequence, we'll have to add a second callsite that influences the output format, which will become awkward to handle.

Refactor the code such that callers are expected to pass the display format that is to be used into display_state_init(). This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

---
 builtin/fetch.c | 50 +++++++++++++++++++++++++++++++------------------
 1 file changed, 32 insertions(+), 18 deletions(-)
builtin/fetch.c +32 -18
index 4c58ee503a..439d3c667b 100644
@@ -104,8 +104,14 @@ static int fetch_write_commit_graph = -1;
 static int stdin_refspecs = 0;
 static int negotiate_only;
 
+struct fetch_config {
+	enum display_format display_format;
+};
+
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
+	struct fetch_config *fetch_config = cb;
+
 	if (!strcmp(k, "fetch.prune")) {
 		fetch_prune_config = git_config_bool(k, v);
 		return 0;
@@ -144,6 +150,18 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(k, "fetch.output")) {
+		if (!v)
+			return config_error_nonbool(k);
+		else if (!strcasecmp(v, "full"))
+			fetch_config->display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(v, "compact"))
+			fetch_config->display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", v);
+	}
+
 	return git_default_config(k, v, cb);
 }
 
@@ -800,14 +818,13 @@ static int refcol_width(const struct ref *ref_map, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -824,15 +841,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT:
@@ -1605,7 +1613,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1691,7 +1700,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2069,7 +2078,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2136,7 +2146,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2146,6 +2156,9 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
+	struct fetch_config config = {
+		.display_format = DISPLAY_FORMAT_FULL,
+	};
 	int i;
 	const char *bundle_uri;
 	struct string_list list = STRING_LIST_INIT_DUP;
@@ -2165,7 +2178,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		free(anon);
 	}
 
-	git_config(git_fetch_config, NULL);
+	git_config(git_fetch_config, &config);
 	if (the_repository->gitdir) {
 		prepare_repo_settings(the_repository);
 		the_repository->settings.command_requires_full_index = 0;
@@ -2302,7 +2315,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   config.display_format);
 	} else {
 		int max_children = max_jobs;
 
Signature · 1 line
2.40.1
#111 lore raw

PATCH v5 8/9 fetch: move option related variables into main function

In reply to Patrick Steinhardt

The options of git-fetch(1) which we pass to parse_options() are declared globally in builtin/fetch.c. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into cmd_fetch(). Move variables that were previously forced to be declared globally and which are only used by cmd_fetch() into function-local scope.

---
 builtin/fetch.c | 197 ++++++++++++++++++++++++------------------------
 1 file changed, 100 insertions(+), 97 deletions(-)
builtin/fetch.c +100 -97
index 439d3c667b..7a3d620c4c 100644
@@ -75,13 +75,12 @@ static int fetch_prune_tags_config = -1; /* unspecified */
 static int prune_tags = -1; /* unspecified */
 #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok;
+static int append, dry_run, force, keep, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 struct fetch_config {
 	enum display_format display_format;
@@ -178,92 +171,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2159,12 +2066,108 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	struct fetch_config config = {
 		.display_format = DISPLAY_FORMAT_FULL,
 	};
-	int i;
+	const char *submodule_prefix = "";
 	const char *bundle_uri;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
+	int all = 0, multiple = 0;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
Signature · 1 line
2.40.1
#112 lore raw

PATCH v5 9/9 fetch: introduce machine-parseable "porcelain" output format

In reply to Patrick Steinhardt

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

```
<flag> <old-object-id> <new-object-id> <local-reference>\n
```

We assume two conditions which are generally true:

- The old and new object IDs have fixed known widths and cannot contain spaces.

- References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote.

In a similar spirit, a second restriction is that this cannot be used with --recurse-submodules. This is because any reference updates would be ambiguous without also printing the repository in which the update happens.

Considering that both multi-remote and submodule fetches are user-facing features, using them in conjunction with --porcelain that is intended for scripting purposes is likely not going to be useful in the majority of cases. With that in mind these restrictions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

---
 Documentation/fetch-options.txt |   7 ++
 Documentation/git-fetch.txt     |   9 ++
 builtin/fetch.c                 |  88 ++++++++++++++----
 t/t5574-fetch-output.sh         | 159 ++++++++++++++++++++++++++++++++
 4 files changed, 244 insertions(+), 19 deletions(-)
Documentation/fetch-options.txt +7 -0
index 622bd84768..41fc7ca3c6 100644
@@ -78,6 +78,13 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--porcelain::
+	Print the output to standard output in an easy-to-parse format for
+	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
++
+This is incompatible with `--recurse-submodules=[yes|on-demand]` and takes
+precedence over the `fetch.output` config option.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
Documentation/git-fetch.txt +9 -0
index fba66f1460..f123139c58 100644
@@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+When using `--porcelain`, the output format is intended to be
+machine-parseable. In contrast to the human-readable output formats it
+thus prints to standard output instead of standard error. Each line is
+of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
builtin/fetch.c +69 -19

251 lines not on this page · raw message

t/t5574-fetch-output.sh +159 -0

171 lines not on this page · raw message

Signature · 1 line
2.40.1
#113 lore raw

Re: [PATCH v5 9/9] fetch: introduce machine-parseable "porcelain" output format

Quoted 8 lines · Patrick Steinhardt
t/t5574-fetch-output.sh +8 -0
+test_expect_success 'fetch --no-porcelain overrides previous --porcelain' '
+	test_when_finished "rm -rf no-porcelain" &&
+
+	git switch --create no-porcelain &&
+	git clone . no-porcelain &&
+	test_commit --no-tag no-porcelain &&
+	old_commit=$(git rev-parse --short HEAD~) &&
+	new_commit=$(git rev-parse --short HEAD) &&

t/t5574-fetch-output.sh test_expect_success 'fetch compact output' ' · line 197

Quoted 10 lines
t/t5574-fetch-output.sh +10 -0
+
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	   $old_commit..$new_commit  no-porcelain -> origin/no-porcelain
+	EOF
+
+	git -C no-porcelain fetch --porcelain --no-porcelain >stdout 2>stderr &&
+	test_cmp expect stderr &&
+	test_must_be_empty stdout
+'

(This shouldn't block the series) As a matter of taste, it feels like overkill to construct the whole output to check if --no-porcelain wins. Maybe in a future cleanup, we would go through the tests in this file and see where it is really important that we have the exact output and where it is not.

#114 lore raw
In reply to Patrick Steinhardt
Quoted 3 lines · Patrick Steinhardt

- Patch 3/9: Added a test to verify that git fetch -c fetch.output without a value set fails as expected. Also dropped the tests that checked whether stdout was empty.

Thank you for the attention to this small detail of valueless configuration variable definition.

Quoted 4 lines

- Patch 4/9: Reformulated the commit message to treat the missing left-hand side of displayed references as an inconsistency instead of a bug. I've also added a testcase to verify that direct OID fetches continue to work as expected.

Again, the direct OID fetch is a good thing to test here. I noticed that the test added here insists that the standard output stream is empty when the command errors out, which is not consistent with [3/9] above.

Quoted 4 lines

- Patch 5/9: New patch that makes calculation of the table width for displayed reference updates self-contained in refcol_width(). This is a preparatory refactoring that should make patch 6/9 easier to review.

It is an excellent idea to avoid calls to refcol_width() for each ref that gets shown and make the helper responsible for computing the required maxwidth. The result indeed has become easier to follow as you mentioned in your response to my review on the previous round.

Quoted 5 lines

- Patch 7/9: Refactored the code to parse the "fetch.output" config variable inside of git_fetch_config() before we parse command line options. Also fixed that the commit message was still referring to --output-format=porcelain instead of the new --porcelain switch.

As a standalone step, it leaves an impression that the step makes the way we handle the output-format configuration variable inconsistent with the way we handle the other configuration variables, but I think it is a good place to stop for the purpose of this topic. It lays a good foundation for future clean-up after the dust settles from this topic---we may want to move global variables assigned in the git_fetch_config() into the fetch_config structure.

Quoted 3 lines

- Patch 9/9: The --porcelain option is now a simple OPT_BOOL() that can be negated. Added a test that --no-porcelain works as expected.

OK, this time the familiar "prepare a variable to its default, let config callback to overwrite it by reading configuration variables, and then let the command line option override it" is used and the result easy to follow. I do not think .display_format is never assigned DISPLAY_FORMAT_UNKNOWN with this change, so [6/9] could lose the value from the enum, I think. The defensive switch statement that has BUG() to notice an erroneous caller that pass values other than DISPLAY_FORMAT_{FULL,COMPACT} is still a good idea.

Looking good.

Thanks.

#115 lore raw
Quoted 1 line · Junio C Hamano

Patrick Steinhardt <[email protected]> writes:

[snip]

Quoted 9 lines
Quoted 4 lines

- Patch 4/9: Reformulated the commit message to treat the missing left-hand side of displayed references as an inconsistency instead of a bug. I've also added a testcase to verify that direct OID fetches continue to work as expected.

Again, the direct OID fetch is a good thing to test here. I noticed that the test added here insists that the standard output stream is empty when the command errors out, which is not consistent with [3/9] above.

I think you misread the test: yes, we do test stdout and stderr separately and in many cases assert that stdout is in fact empty. But none of the added tests are about failing commands. So given that:

- The added tests explicitly are about verifying the output format.

- The distinction between stdout and stderr matters.

- The distinction matters even more with the addition of --porcelain.

I think that explicitly verifing both output streams is the correct thing to do.

Quoted 4 lines
Quoted 3 lines

- Patch 9/9: The --porcelain option is now a simple OPT_BOOL() that can be negated. Added a test that --no-porcelain works as expected.

Quoted 9 lines

OK, this time the familiar "prepare a variable to its default, let config callback to overwrite it by reading configuration variables, and then let the command line option override it" is used and the result easy to follow. I do not think .display_format is never assigned DISPLAY_FORMAT_UNKNOWN with this change, so [6/9] could lose the value from the enum, I think. The defensive switch statement that has BUG() to notice an erroneous caller that pass values other than DISPLAY_FORMAT_{FULL,COMPACT} is still a good idea.

Ah, right, DISPLAY_FORMAT_UNKNOWN isn't really needed anymore. I think it's still good to have valid values of the enum start with 1 so that it becomes easier to detect cases where we accidentally use a default initialized variable. But that can be achieved without giving the default value an explicit name.

I'll refrain from sending a new version just to remove this constant as it doesn't really feel worth it, though. But please, let me know in case you disagree and I'll send an updated version.

Thanks
Patrick

#116 lore raw
Quoted 8 lines · Patrick Steinhardt
Quoted 1 line · Junio C Hamano

Patrick Steinhardt <[email protected]> writes:

[snip]

Quoted 5 lines
Quoted 4 lines

- Patch 4/9: Reformulated the commit message to treat the missing left-hand side of displayed references as an inconsistency instead of a bug. I've also added a testcase to verify that direct OID fetches continue to work as expected.

Quoted 8 lines
Quoted 4 lines

Again, the direct OID fetch is a good thing to test here. I noticed that the test added here insists that the standard output stream is empty when the command errors out, which is not consistent with [3/9] above.

I think you misread the test: yes, we do test stdout and stderr separately and in many cases assert that stdout is in fact empty. But none of the added tests are about failing commands.

Ah, you are absolutely right. Sorry for the noise---I do not have an objection to ensure that program output in the successful case is predictable. My main concern is to avoid giving unneeded assurance in the failing case.

Quoted 11 lines
Quoted 5 lines

assigned DISPLAY_FORMAT_UNKNOWN with this change, so [6/9] could lose the value from the enum, I think. The defensive switch statement that has BUG() to notice an erroneous caller that pass values other than DISPLAY_FORMAT_{FULL,COMPACT} is still a good idea.

Ah, right, DISPLAY_FORMAT_UNKNOWN isn't really needed anymore. I think it's still good to have valid values of the enum start with 1 so that it becomes easier to detect cases where we accidentally use a default initialized variable. But that can be achieved without giving the default value an explicit name.

Unless we explicitly take advantage of UNKNOWN being 0 and perform clever defaulting, the result is far safer if you get rid of it, and instead give 0 to a real choice that is used as the default, for two
reasons: (1) naturally 0 initialization work fine, (2) we have one less enum constant people (or editor's auto-completer) assign to a variable by mistake instead of a real one they intended to. I agree with the last sentence of your paragraph above.

#117 lore raw
In reply to Patrick Steinhardt
Quoted 12 lines · Patrick Steinhardt
Quoted 9 lines · Junio C Hamano

OK, this time the familiar "prepare a variable to its default, let config callback to overwrite it by reading configuration variables, and then let the command line option override it" is used and the result easy to follow. I do not think .display_format is never assigned DISPLAY_FORMAT_UNKNOWN with this change, so [6/9] could lose the value from the enum, I think. The defensive switch statement that has BUG() to notice an erroneous caller that pass values other than DISPLAY_FORMAT_{FULL,COMPACT} is still a good idea.

Quoted 5 lines

Ah, right, DISPLAY_FORMAT_UNKNOWN isn't really needed anymore. I think it's still good to have valid values of the enum start with 1 so that it becomes easier to detect cases where we accidentally use a default initialized variable. But that can be achieved without giving the default value an explicit name.

I think it's standard to have an element like that. It could be UNKNOWN, NULL, INVALID, or even DEFAULT.

For example, you could have code like this:

if (format == DISPLAY_FORMAT_DEFAULT)

        format = DISPLAY_FORMAT_FULL;

This would distinguish cases in which the user did not specify a display format and we choose a default, from those where the user explicitely chose the format that happens to be the default.

Signature · 1 line
Felipe Contreras
Reviewed-by on the v5 series
#118 lore raw
In reply to Patrick Steinhardt
Quoted 24 lines · Patrick Steinhardt

Changes compared to v4:

- Patch 1/9: Simplified the test as proposed by Junio and Glen.

- Patch 3/9: Added a test to verify that git fetch -c fetch.output without a value set fails as expected. Also dropped the tests that checked whether stdout was empty.

- Patch 4/9: Reformulated the commit message to treat the missing left-hand side of displayed references as an inconsistency instead of a bug. I've also added a testcase to verify that direct OID fetches continue to work as expected.

- Patch 5/9: New patch that makes calculation of the table width for displayed reference updates self-contained in refcol_width(). This is a preparatory refactoring that should make patch 6/9 easier to review.

- Patch 7/9: Refactored the code to parse the "fetch.output" config variable inside of git_fetch_config() before we parse command line options. Also fixed that the commit message was still referring to --output-format=porcelain instead of the new --porcelain switch.

Quoted 3 lines

- Patch 9/9: The --porcelain option is now a simple OPT_BOOL() that can be negated. Added a test that --no-porcelain works as expected.

I didn't spot any blocking issues in this version, and the various improvements (especially 6-7/9) are really welcome. I also read through Junio's comments, but I didn't spot anything that I thought should block the series, so I'm happy to leave

#119 lore raw
Quoted 30 lines · Glen Choo
Quoted 24 lines · Patrick Steinhardt

Changes compared to v4:

- Patch 1/9: Simplified the test as proposed by Junio and Glen.

- Patch 3/9: Added a test to verify that git fetch -c fetch.output without a value set fails as expected. Also dropped the tests that checked whether stdout was empty.

- Patch 4/9: Reformulated the commit message to treat the missing left-hand side of displayed references as an inconsistency instead of a bug. I've also added a testcase to verify that direct OID fetches continue to work as expected.

- Patch 5/9: New patch that makes calculation of the table width for displayed reference updates self-contained in refcol_width(). This is a preparatory refactoring that should make patch 6/9 easier to review.

- Patch 7/9: Refactored the code to parse the "fetch.output" config variable inside of git_fetch_config() before we parse command line options. Also fixed that the commit message was still referring to --output-format=porcelain instead of the new --porcelain switch.

Quoted 3 lines

- Patch 9/9: The --porcelain option is now a simple OPT_BOOL() that can be negated. Added a test that --no-porcelain works as expected.

Quoted 6 lines

I didn't spot any blocking issues in this version, and the various improvements (especially 6-7/9) are really welcome. I also read through Junio's comments, but I didn't spot anything that I thought should block the series, so I'm happy to leave

Thanks a lot for your reviews!

Patrick