Skip to contentSkip to conversation

fetch --prune: exit with error if pruning fails

PATCH Superseded Thomas Gummerer 3 participants 11 messages Jan 27, 2022 – Jan 31, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fetch --prune: exit with error if pruning fails

When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.

The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.

---
 builtin/fetch.c  | 12 ++++++++----
 t/t5510-fetch.sh | 10 ++++++++++
 2 files changed, 18 insertions(+), 4 deletions(-)
builtin/fetch.c +8 -4
index 5f06b21f8e..54545efedd 100644
@@ -1609,11 +1609,15 @@ static int do_fetch(struct transport *transport,
 		 * don't care whether --tags was specified.
 		 */
 		if (rs->nr) {
-			prune_refs(rs, ref_map, transport->url);
+			retcode = prune_refs(rs, ref_map, transport->url);
 		} else {
-			prune_refs(&transport->remote->fetch,
-				   ref_map,
-				   transport->url);
+			retcode = prune_refs(&transport->remote->fetch,
+					     ref_map,
+					     transport->url);
+		}
+		if (retcode) {
+			free_refs(ref_map);
+			goto cleanup;
 		}
 	}
 	if (fetch_and_consume_refs(transport, ref_map, worktrees)) {
t/t5510-fetch.sh +10 -0
index 20f7110ec1..df824cc3d0 100755
@@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
 	git rev-parse sometag
 '
 
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	>.git/packed-refs.new &&
+
+	test_must_fail git fetch --prune origin
+'
+
 test_expect_success 'fetch --atomic works with a single branch' '
 	test_when_finished "rm -rf \"$D\"/atomic" &&
 
Signature · 1 line
2.31.1
#1 lore raw
Quoted 8 lines · Thomas Gummerer

When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.

The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.

It is a good idea to signal, with a non-zero status, that the user needs to inspect the situation and possibly take a corrective action. I however do not know if it is sufficient to just give the same exit status as returned by prune_refs(), which may or may not happen to crash with other possible non-zero exit status to make it indistinguishable from other kinds of errors.

builtin/fetch.c static int do_fetch(struct transport *transport, · line 1617

Quoted 11 lines
builtin/fetch.c +5 -4
 		if (rs->nr) {
-			prune_refs(rs, ref_map, transport->url);
+			retcode = prune_refs(rs, ref_map, transport->url);
 		} else {
-			prune_refs(&transport->remote->fetch,
-				   ref_map,
-				   transport->url);
+			retcode = prune_refs(&transport->remote->fetch,
+					     ref_map,
+					     transport->url);
+		}

It seems that it is possible for do_fetch() to return a negative value, even without this patch. The return value from prune_refs() comes from refs.c::delete_refs(), which can come from error_errno() that is -1, so this patch adds more of the same problem to existing ones, though.

And the value propagates up to cmd_fetch() via fetch_one(). This will be relayed to exit() without changing via this callchain:

    git.c::cmd_main()
      git.c::run_argv()
        git.c::handle_builtin()
          git.c::run_builtin()

This is not a new problem, which does not want to be corrected as part of this patch, but let's leave a mental note as #leftoverbits for now.

builtin/fetch.c static int do_fetch(struct transport *transport, · line 1621

Quoted 4 lines
builtin/fetch.c +3 -0
+		if (retcode) {
+			free_refs(ref_map);
+			goto cleanup;
 		}

This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.

Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.

Quoted 5 lines
t/t5510-fetch.sh +0 -0
index 20f7110ec1..df824cc3d0 100755

t/t5510-fetch.sh test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' · line 173

Quoted 10 lines
t/t5510-fetch.sh +7 -0
@@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
 	git rev-parse sometag
 '
 
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	>.git/packed-refs.new &&
+	test_must_fail git fetch --prune origin

Is it because the lockfile ".new" on packed-refs prevents deletion of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.

Another tangent #leftoverbits is that many tests in this script are so old-style that chdir around without isolating themselves in subshells, while some do. We may want to clean them up when the file is quiescent.

Thanks.

#2 lore raw

Hi Junio,

Quoted 12 lines · Junio C Hamano

builtin/fetch.c static int do_fetch(struct transport *transport, · line 1621

Quoted 4 lines · Thomas Gummerer
builtin/fetch.c +3 -0
+		if (retcode) {
+			free_refs(ref_map);
+			goto cleanup;
 		}

This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.

Quoted 6 lines

Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.

I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.

Maybe Thomas has an even stronger argument in favor of erroring out. In any case, I don't think that --prune should be a "best effort, otherwise just shrug" option. If we wanted that, we could introduce --prune-best-effort or some such...

Ciao,
Dscho

#3 lore raw
Quoted 23 lines · Johannes Schindelin

Hi Junio,

Quoted 12 lines · Junio C Hamano

builtin/fetch.c static int do_fetch(struct transport *transport, · line 1621

Quoted 4 lines · Thomas Gummerer
builtin/fetch.c +3 -0
+		if (retcode) {
+			free_refs(ref_map);
+			goto cleanup;
 		}

This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.

Quoted 6 lines

Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.

Quoted 5 lines

I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.

I was thinking along similar lines here. I was going back and forth between letting the fetch continue, and then exiting with a non-zero exit code, and just erroring out directly. I ended up with the latter because it felt like the right thing to do for the user. The command failed, so I'd think it's more confusing that it does more work after that (even if we already did part of that work).

But I don't care too much one way or another, as long as we end up with a non-zero exit code when the fetch fails.

Quoted 4 lines

Maybe Thomas has an even stronger argument in favor of erroring out. In any case, I don't think that --prune should be a "best effort, otherwise just shrug" option. If we wanted that, we could introduce --prune-best-effort or some such...

I don't have a strong argument on whether we should error out immediately after failing to prune the refs, or just erroring out after doing the rest of the work, other than exiting early feels right to me.

From the comments upthread it seems to me that the argument is more whether we should continue after failing to prune, and exit with a non-zero exit code, or if we should just fail early.

Though I'm happy to introduce a '--prune-best-effort' option or something like that if we still want to preserve that option for users. I don't really see people wanting to use that though, and it should be very rare that '--prune' fails, but the rest of the fetch succeeds.

#4 lore raw

Hi Thomas & Junio,

Quoted 29 lines · Thomas Gummerer
Quoted 21 lines · Johannes Schindelin
Quoted 12 lines · Junio C Hamano

builtin/fetch.c static int do_fetch(struct transport *transport, · line 1621

Quoted 4 lines · Thomas Gummerer
builtin/fetch.c +3 -0
+		if (retcode) {
+			free_refs(ref_map);
+			goto cleanup;
 		}

This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.

Quoted 6 lines

Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.

Quoted 5 lines

I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.

Quoted 3 lines

I was thinking along similar lines here. I was going back and forth between letting the fetch continue, and then exiting with a non-zero exit code, and just erroring out directly.

Oh, I think I misunderstood Junio. As long as the failed prune will cause a non-zero exit code, I am fine with continuing to try to fetch.

Ciao,
Dscho

#5 lore raw
In reply to Johannes Schindelin
Quoted 10 lines · Johannes Schindelin

I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.

Maybe Thomas has an even stronger argument in favor of erroring out. In any case, I don't think that --prune should be a "best effort, otherwise just shrug" option. If we wanted that, we could introduce --prune-best-effort or some such...

I am not opposed to reporting an error by exiting with non-zero exit code. I never said it should be best effort, and doing the "fetch" part after a failed prune does not make it best effort.

What I am questioning is if it makes sense to stop the fetching part. When we fetch to update multiple refs, we do not stop at the first ref-update failure, but try to do as much as possible and then report an error, no? It is the same thing.

#6 lore raw
In reply to Junio C Hamano
Quoted 11 lines · Junio C Hamano
Quoted 8 lines · Thomas Gummerer

When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.

The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.

Quoted 6 lines

It is a good idea to signal, with a non-zero status, that the user needs to inspect the situation and possibly take a corrective action. I however do not know if it is sufficient to just give the same exit status as returned by prune_refs(), which may or may not happen to crash with other possible non-zero exit status to make it indistinguishable from other kinds of errors.

Not sure I understand this correctly. Are you saying that we should take the return value from prune refs, and always munge that to the same exit code for 'git fetch --prune' if it fails? E.g. so prune_refs() could return 1 or 2 or 3 or whatever, but we would always want the exit code for 'fetch' to be 1?

Happy to implement that.

Quoted 17 lines
Quoted 5 lines
t/t5510-fetch.sh +0 -0
index 20f7110ec1..df824cc3d0 100755

t/t5510-fetch.sh test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' · line 173

Quoted 10 lines
t/t5510-fetch.sh +7 -0
@@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
 	git rev-parse sometag
 '
 
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	>.git/packed-refs.new &&
+	test_must_fail git fetch --prune origin

Is it because the lockfile ".new" on packed-refs prevents deletion

Quoted 8 lines

of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.

Yes, that's correct. New refs will be created as loose refs, so they don't care about packed-refs. However deletions can potentially be happening in packed-refs, and that's why it fails when 'packed-refs.new' exists.

I don't love the test either, but I also can't think of a better way to do this.

Quoted 6 lines

Another tangent #leftoverbits is that many tests in this script are so old-style that chdir around without isolating themselves in subshells, while some do. We may want to clean them up when the file is quiescent.

Thanks.

#7 lore raw

Hi Thomas,

Quoted 22 lines · Thomas Gummerer
Quoted 11 lines · Junio C Hamano

t/t5510-fetch.sh test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' · line 174

Quoted 7 lines · Thomas Gummerer
t/t5510-fetch.sh +7 -0
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	>.git/packed-refs.new &&
+	test_must_fail git fetch --prune origin

Is it because the lockfile ".new" on packed-refs prevents deletion

Quoted 8 lines

of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.

Quoted 7 lines

Yes, that's correct. New refs will be created as loose refs, so they don't care about packed-refs. However deletions can potentially be happening in packed-refs, and that's why it fails when 'packed-refs.new' exists.

I don't love the test either, but I also can't think of a better way to do this.

Maybe add a code comment about it? Something like:

	[...]
	: this will prevent --prune from locking packed-refs &&
	>.git/packed-refs.new &&
	[...]

Ciao,
Dscho

#8 lore raw
Quoted 34 lines · Johannes Schindelin

Hi Thomas,

Quoted 22 lines · Thomas Gummerer
Quoted 11 lines · Junio C Hamano

t/t5510-fetch.sh test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' · line 174

Quoted 7 lines · Thomas Gummerer
t/t5510-fetch.sh +7 -0
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	>.git/packed-refs.new &&
+	test_must_fail git fetch --prune origin

Is it because the lockfile ".new" on packed-refs prevents deletion

Quoted 8 lines

of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.

Quoted 7 lines

Yes, that's correct. New refs will be created as loose refs, so they don't care about packed-refs. However deletions can potentially be happening in packed-refs, and that's why it fails when 'packed-refs.new' exists.

I don't love the test either, but I also can't think of a better way to do this.

Quoted 6 lines

Maybe add a code comment about it? Something like:

	[...]
	: this will prevent --prune from locking packed-refs &&
	>.git/packed-refs.new &&
	[...]

Yeah, I think a comment here is a good idea, thanks!

#9 lore raw

PATCH v2 1/1 fetch --prune: exit with error if pruning fails

In reply to Thomas Gummerer

When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.

The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.

---

changes in v2:
- retcode will now always be 1 if we get an error from prune_refs, no matter what the original error code was.
- we will now continue the fetch, and only exit with a non-zero exit code after having finished the rest of the fetch operation
- added a comment in the test explaining why creating a packed-refs.new file will make the fetch --prune fail.

 builtin/fetch.c  | 10 ++++++----
 t/t5510-fetch.sh | 11 +++++++++++
 2 files changed, 17 insertions(+), 4 deletions(-)
builtin/fetch.c +6 -4
index 5f06b21f8e..648f0694f2 100644
@@ -1609,12 +1609,14 @@ static int do_fetch(struct transport *transport,
 		 * don't care whether --tags was specified.
 		 */
 		if (rs->nr) {
-			prune_refs(rs, ref_map, transport->url);
+			retcode = prune_refs(rs, ref_map, transport->url);
 		} else {
-			prune_refs(&transport->remote->fetch,
-				   ref_map,
-				   transport->url);
+			retcode = prune_refs(&transport->remote->fetch,
+					     ref_map,
+					     transport->url);
 		}
+		if (retcode != 0)
+			retcode = 1;
 	}
 	if (fetch_and_consume_refs(transport, ref_map, worktrees)) {
 		free_refs(ref_map);
t/t5510-fetch.sh +11 -0
index 20f7110ec1..ef0da0a63b 100755
@@ -164,6 +164,17 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
 	git rev-parse sometag
 '
 
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	: this will prevent --prune from locking packed-refs for deleting refs, but adding loose refs still succeeds  &&
+	>.git/packed-refs.new &&
+
+	test_must_fail git fetch --prune origin
+'
+
 test_expect_success 'fetch --atomic works with a single branch' '
 	test_when_finished "rm -rf \"$D\"/atomic" &&
 
Signature · 1 line
2.31.1
#10 lore raw
Quoted 11 lines · Thomas Gummerer
builtin/fetch.c +1 -1
index 5f06b21f8e..648f0694f2 100644
@@ -1609,12 +1609,14 @@ static int do_fetch(struct transport *transport,
 		 * don't care whether --tags was specified.
 		 */
 		if (rs->nr) {
-			prune_refs(rs, ref_map, transport->url);
+			retcode = prune_refs(rs, ref_map, transport->url);
 		} else {

builtin/fetch.c static int do_fetch(struct transport *transport, · line 1620

Quoted 10 lines
builtin/fetch.c +5 -3
-			prune_refs(&transport->remote->fetch,
-				   ref_map,
-				   transport->url);
+			retcode = prune_refs(&transport->remote->fetch,
+					     ref_map,
+					     transport->url);
 		}
+		if (retcode != 0)
+			retcode = 1;
 	}

Looks trivially correct, even though a few style things look a bit irritating to my eyes ;-).

Thanks, will queue.