fetch --prune: exit with error if pruning fails
PATCH 1/1 fetch --prune: exit with error if pruning fails
When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.
The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.
--- builtin/fetch.c | 12 ++++++++---- t/t5510-fetch.sh | 10 ++++++++++ 2 files changed, 18 insertions(+), 4 deletions(-)
builtin/fetch.c +8 -4
@@ -1609,11 +1609,15 @@ static int do_fetch(struct transport *transport,* don't care whether --tags was specified. */ if (rs->nr) { -prune_refs (rs, ref_map, transport->url); + retcode =prune_refs (rs, ref_map, transport->url); }else { -prune_refs (&transport->remote->fetch, - ref_map, - transport->url); + retcode =prune_refs (&transport->remote->fetch, + ref_map, + transport->url); + } +if (retcode) { +free_refs (ref_map); + goto cleanup; } }if (fetch_and_consume_refs (transport, ref_map, worktrees)) {
t/t5510-fetch.sh +10 -0
@@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' git rev-parse sometag ' +test_expect_success REFFILES 'fetch --prune fails to delete branches' ' + cd "$D" && + git clone . prune-fail && + cd prune-fail && + git update-ref refs/remotes/origin/extrabranch main && + >.git/packed-refs.new && + + test_must_fail git fetch --prune origin +' + test_expect_success 'fetch --atomic works with a single branch' ' test_when_finished "rm -rf \"$D\"/atomic" &&
Signature · 1 line
2.31.1
Quoted 8 lines · Thomas Gummerer
When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.
The --prune option was introduced in
f360d844de("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.
It is a good idea to signal, with a non-zero status, that the user needs to inspect the situation and possibly take a corrective action. I however do not know if it is sufficient to just give the same exit status as returned by prune_refs(), which may or may not happen to crash with other possible non-zero exit status to make it indistinguishable from other kinds of errors.
builtin/fetch.c static int do_fetch(struct transport *transport, · line 1617
Quoted 11 lines
builtin/fetch.c +5
-4if (rs->nr) { -prune_refs (rs, ref_map, transport->url); + retcode =prune_refs (rs, ref_map, transport->url); }else { -prune_refs (&transport->remote->fetch, - ref_map, - transport->url); + retcode =prune_refs (&transport->remote->fetch, + ref_map, + transport->url); + }
It seems that it is possible for do_fetch() to return a negative value, even without this patch. The return value from prune_refs() comes from refs.c::delete_refs(), which can come from error_errno() that is -1, so this patch adds more of the same problem to existing ones, though.
And the value propagates up to cmd_fetch() via fetch_one(). This will be relayed to exit() without changing via this callchain:
git.c::cmd_main()
git.c::run_argv()
git.c::handle_builtin()
git.c::run_builtin()This is not a new problem, which does not want to be corrected as part of this patch, but let's leave a mental note as #leftoverbits for now.
builtin/fetch.c static int do_fetch(struct transport *transport, · line 1621
Quoted 4 lines
builtin/fetch.c +3
-0+if (retcode) { +free_refs (ref_map); + goto cleanup; }
This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.
Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.
Quoted 5 lines
t/t5510-fetch.sh +0
-0
t/t5510-fetch.sh test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' · line 173
Quoted 10 lines
t/t5510-fetch.sh +7
-0@@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' git rev-parse sometag ' +test_expect_success REFFILES 'fetch --prune fails to delete branches' ' + cd "$D" && + git clone . prune-fail && + cd prune-fail && + git update-ref refs/remotes/origin/extrabranch main && + >.git/packed-refs.new && + test_must_fail git fetch --prune origin
Is it because the lockfile ".new" on packed-refs prevents deletion of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.
Another tangent #leftoverbits is that many tests in this script are so old-style that chdir around without isolating themselves in subshells, while some do. We may want to clean them up when the file is quiescent.
Thanks.
Hi Junio,
Quoted 12 lines · Junio C Hamano
builtin/fetch.c static int do_fetch(struct transport *transport, · line 1621
Quoted 4 lines · Thomas Gummerer
builtin/fetch.c +3
-0+if (retcode) { +free_refs (ref_map); + goto cleanup; }This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.
Quoted 6 lines
Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.
I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.
Maybe Thomas has an even stronger argument in favor of erroring out. In any case, I don't think that --prune should be a "best effort, otherwise just shrug" option. If we wanted that, we could introduce --prune-best-effort or some such...
Ciao,
Dscho
Quoted 23 lines · Johannes Schindelin
Hi Junio,
Quoted 12 lines · Junio C Hamano
builtin/fetch.c static int do_fetch(struct transport *transport, · line 1621
Quoted 4 lines · Thomas Gummerer
builtin/fetch.c +3
-0+if (retcode) { +free_refs (ref_map); + goto cleanup; }This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.
Quoted 6 lines
Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.
Quoted 5 lines
I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.
I was thinking along similar lines here. I was going back and forth between letting the fetch continue, and then exiting with a non-zero exit code, and just erroring out directly. I ended up with the latter because it felt like the right thing to do for the user. The command failed, so I'd think it's more confusing that it does more work after that (even if we already did part of that work).
But I don't care too much one way or another, as long as we end up with a non-zero exit code when the fetch fails.
Quoted 4 lines
Maybe Thomas has an even stronger argument in favor of erroring out. In any case, I don't think that
--pruneshould be a "best effort, otherwise just shrug" option. If we wanted that, we could introduce--prune-best-effortor some such...
I don't have a strong argument on whether we should error out immediately after failing to prune the refs, or just erroring out after doing the rest of the work, other than exiting early feels right to me.
From the comments upthread it seems to me that the argument is more whether we should continue after failing to prune, and exit with a non-zero exit code, or if we should just fail early.
Though I'm happy to introduce a '--prune-best-effort' option or something like that if we still want to preserve that option for users. I don't really see people wanting to use that though, and it should be very rare that '--prune' fails, but the rest of the fetch succeeds.
Quoted 11 lines · Junio C Hamano
Quoted 8 lines · Thomas Gummerer
When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.
The --prune option was introduced in
f360d844de("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.
Quoted 6 lines
It is a good idea to signal, with a non-zero status, that the user needs to inspect the situation and possibly take a corrective action. I however do not know if it is sufficient to just give the same exit status as returned by
prune_refs(), which may or may not happen to crash with other possible non-zero exit status to make it indistinguishable from other kinds of errors.
Not sure I understand this correctly. Are you saying that we should take the return value from prune refs, and always munge that to the same exit code for 'git fetch --prune' if it fails? E.g. so prune_refs() could return 1 or 2 or 3 or whatever, but we would always want the exit code for 'fetch' to be 1?
Happy to implement that.
Quoted 17 lines
Quoted 5 lines
t/t5510-fetch.sh +0
-0t/t5510-fetch.sh test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' · line 173
Quoted 10 lines
t/t5510-fetch.sh +7
-0@@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' git rev-parse sometag ' +test_expect_success REFFILES 'fetch --prune fails to delete branches' ' + cd "$D" && + git clone . prune-fail && + cd prune-fail && + git update-ref refs/remotes/origin/extrabranch main && + >.git/packed-refs.new && + test_must_fail git fetch --prune originIs it because the lockfile ".new" on packed-refs prevents deletion
Quoted 8 lines
of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.
Yes, that's correct. New refs will be created as loose refs, so they don't care about packed-refs. However deletions can potentially be happening in packed-refs, and that's why it fails when 'packed-refs.new' exists.
I don't love the test either, but I also can't think of a better way to do this.
Quoted 6 lines
Another tangent #leftoverbits is that many tests in this script are so old-style that chdir around without isolating themselves in subshells, while some do. We may want to clean them up when the file is quiescent.
Thanks.
Hi Thomas,
Quoted 22 lines · Thomas Gummerer
Quoted 11 lines · Junio C Hamano
t/t5510-fetch.sh test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' · line 174
Quoted 7 lines · Thomas Gummerer
t/t5510-fetch.sh +7
-0+test_expect_success REFFILES 'fetch --prune fails to delete branches' ' + cd "$D" && + git clone . prune-fail && + cd prune-fail && + git update-ref refs/remotes/origin/extrabranch main && + >.git/packed-refs.new && + test_must_fail git fetch --prune originIs it because the lockfile ".new" on packed-refs prevents deletion
Quoted 8 lines
of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.
Quoted 7 lines
Yes, that's correct. New refs will be created as loose refs, so they don't care about packed-refs. However deletions can potentially be happening in packed-refs, and that's why it fails when 'packed-refs.new' exists.
I don't love the test either, but I also can't think of a better way to do this.
Maybe add a code comment about it? Something like:
[...] : this will prevent --prune from locking packed-refs && >.git/packed-refs.new && [...]
Ciao,
Dscho
Hi Thomas & Junio,
Quoted 29 lines · Thomas Gummerer
Quoted 21 lines · Johannes Schindelin
Quoted 12 lines · Junio C Hamano
builtin/fetch.c static int do_fetch(struct transport *transport, · line 1621
Quoted 4 lines · Thomas Gummerer
builtin/fetch.c +3
-0+if (retcode) { +free_refs (ref_map); + goto cleanup; }This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.
Quoted 6 lines
Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.
Quoted 5 lines
I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.
Quoted 3 lines
I was thinking along similar lines here. I was going back and forth between letting the fetch continue, and then exiting with a non-zero exit code, and just erroring out directly.
Oh, I think I misunderstood Junio. As long as the failed prune will cause a non-zero exit code, I am fine with continuing to try to fetch.
Ciao,
Dscho
Quoted 10 lines · Johannes Schindelin
I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.
Maybe Thomas has an even stronger argument in favor of erroring out. In any case, I don't think that
--pruneshould be a "best effort, otherwise just shrug" option. If we wanted that, we could introduce--prune-best-effortor some such...
I am not opposed to reporting an error by exiting with non-zero exit code. I never said it should be best effort, and doing the "fetch" part after a failed prune does not make it best effort.
What I am questioning is if it makes sense to stop the fetching part. When we fetch to update multiple refs, we do not stop at the first ref-update failure, but try to do as much as possible and then report an error, no? It is the same thing.
Quoted 34 lines · Johannes Schindelin
Hi Thomas,
Quoted 22 lines · Thomas Gummerer
Quoted 11 lines · Junio C Hamano
t/t5510-fetch.sh test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' · line 174
Quoted 7 lines · Thomas Gummerer
t/t5510-fetch.sh +7
-0+test_expect_success REFFILES 'fetch --prune fails to delete branches' ' + cd "$D" && + git clone . prune-fail && + cd prune-fail && + git update-ref refs/remotes/origin/extrabranch main && + >.git/packed-refs.new && + test_must_fail git fetch --prune originIs it because the lockfile ".new" on packed-refs prevents deletion
Quoted 8 lines
of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.
Quoted 7 lines
Yes, that's correct. New refs will be created as loose refs, so they don't care about packed-refs. However deletions can potentially be happening in packed-refs, and that's why it fails when 'packed-refs.new' exists.
I don't love the test either, but I also can't think of a better way to do this.
Quoted 6 lines
Maybe add a code comment about it? Something like:
[...] : this will prevent --prune from locking packed-refs && >.git/packed-refs.new && [...]
Yeah, I think a comment here is a good idea, thanks!
PATCH v2 1/1 fetch --prune: exit with error if pruning fails
In reply to Thomas GummererWhen pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.
The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.
---
changes in v2:
- retcode will now always be 1 if we get an error from prune_refs, no matter what the original error code was.
- we will now continue the fetch, and only exit with a non-zero exit code after having finished the rest of the fetch operation
- added a comment in the test explaining why creating a packed-refs.new file will make the fetch --prune fail.
builtin/fetch.c | 10 ++++++---- t/t5510-fetch.sh | 11 +++++++++++ 2 files changed, 17 insertions(+), 4 deletions(-)
builtin/fetch.c +6 -4
@@ -1609,12 +1609,14 @@ static int do_fetch(struct transport *transport,* don't care whether --tags was specified. */ if (rs->nr) { -prune_refs (rs, ref_map, transport->url); + retcode =prune_refs (rs, ref_map, transport->url); }else { -prune_refs (&transport->remote->fetch, - ref_map, - transport->url); + retcode =prune_refs (&transport->remote->fetch, + ref_map, + transport->url); } +if (retcode !=0 ) + retcode =1 ; }if (fetch_and_consume_refs (transport, ref_map, worktrees)) {free_refs (ref_map);
t/t5510-fetch.sh +11 -0
@@ -164,6 +164,17 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' git rev-parse sometag ' +test_expect_success REFFILES 'fetch --prune fails to delete branches' ' + cd "$D" && + git clone . prune-fail && + cd prune-fail && + git update-ref refs/remotes/origin/extrabranch main && + : this will prevent --prune from locking packed-refs for deleting refs, but adding loose refs still succeeds && + >.git/packed-refs.new && + + test_must_fail git fetch --prune origin +' + test_expect_success 'fetch --atomic works with a single branch' ' test_when_finished "rm -rf \"$D\"/atomic" &&
Signature · 1 line
2.31.1
Quoted 11 lines · Thomas Gummerer
builtin/fetch.c +1
-1@@ -1609,12 +1609,14 @@ static int do_fetch(struct transport *transport,* don't care whether --tags was specified. */ if (rs->nr) { -prune_refs (rs, ref_map, transport->url); + retcode =prune_refs (rs, ref_map, transport->url); }else {
builtin/fetch.c static int do_fetch(struct transport *transport, · line 1620
Quoted 10 lines
builtin/fetch.c +5
-3-prune_refs (&transport->remote->fetch, - ref_map, - transport->url); + retcode =prune_refs (&transport->remote->fetch, + ref_map, + transport->url); } +if (retcode !=0 ) + retcode =1 ; }
Looks trivially correct, even though a few style things look a bit irritating to my eyes ;-).
Thanks, will queue.