remote-curl: pass on atomic capability to remote side
PATCH 1/1 remote-curl: pass on atomic capability to remote side
When pushing more than one reference with the --atomic option, the server is supposed to perform a single atomic transaction to update the references, leaving them either all to succeed or all to fail. This works fine when pushing locally or over SSH, but when pushing over HTTP, we fail to pass the atomic capability to the remote side. In fact, we have not reported this capability to any remote helpers during the life of the feature.
Now normally, things happen to work nevertheless, since we actually check for most types of failures, such as non-fast-forward updates, on the client side, and just abort the entire attempt. However, if the server side reports a problem, such as the inability to lock a ref, the transaction isn't atomic, because we haven't passed the appropriate capability over and the remote side has no way of knowing that we wanted atomic behavior.
Fix this by passing the option from the transport code through to remote helpers, and from the HTTP remote helper down to send-pack. With this change, we can detect if the server side rejects the push and report back appropriately. Note the difference in the messages: the remote side reports "atomic transaction failed", while our own checking rejects pushes with the message "atomic push failed".
Document the atomic option in the remote helper documentation, so other implementers can implement it if they like.
---
I discovered this at work when implementing support for atomic pushes. Our equivalent of the pre-receive hook is not all-or-nothing, and passing the atomic capability through to our backend worked for SSH, but not HTTP. I discovered with GIT_TRACE_PACKET=1 that we didn't pass the atomic capability through during HTTP, so I thought I'd send a patch.
As I mentioned in the commit message, to my knowledge, this functionality has been broken since the atomic capability was introduced circa 2.4.0.
Documentation/gitremote-helpers.txt | 5 ++++ remote-curl.c | 13 +++++++++- t/t5541-http-push-smart.sh | 40 +++++++++++++++++++++++++++-- transport-helper.c | 4 +++ transport.h | 3 +++ 5 files changed, 62 insertions(+), 3 deletions(-)
Documentation/gitremote-helpers.txt +5 -0
@@ -509,6 +509,11 @@ set by Git if the remote helper has the 'option' capability. Indicate that only the objects wanted need to be fetched, not their dependents. +'option atomic' {'true'|'false'}:: + When pushing, request the remote server to update refs in a single atomic + transaction. If successful, all refs will be updated, or none will. If the + remote side does not support this capability, the push will fail. + SEE ALSO -------- linkgit:git-remote[1]
remote-curl.c +12 -1
@@ -40,7 +40,8 @@ struct options { push_cert : 2, deepen_relative : 1, from_promisor : 1, - no_dependents : 1; + no_dependents : 1, + atomic : 1; };static struct options options;static struct string_list cas_options =STRING_LIST_INIT_DUP ;
@@ -148,6 +149,14 @@ static int set_option(const char *name, const char *value)else return -1 ;return 0 ; + }else if (!strcmp (name,"atomic" )) { +if (!strcmp (value,"true" )) + options.atomic =1 ; +else if (!strcmp (value,"false" )) + options.atomic =0 ; +else +return -1 ; +return 0 ; }else if (!strcmp (name,"push-option" )) {if (*value !='"' )string_list_append (&options.push_options, value);
@@ -1196,6 +1205,8 @@ static int push_git(struct discovery *heads, int nr_spec, char **specs)argv_array_push (&args,"--signed=yes" ); else if (options.push_cert == SEND_PACK_PUSH_CERT_IF_ASKED) argv_array_push(&args, "--signed=if-asked"); + if (options.atomic) +argv_array_push (&args,"--atomic" );if (options.verbosity ==0 )argv_array_push (&args,"--quiet" );else if (options.verbosity >1 )
t/t5541-http-push-smart.sh +38 -2
@@ -184,11 +184,12 @@ test_expect_success 'push --atomic also prevents branch creation, reports collat test_config -C "$d" http.receivepack true && up="$HTTPD_URL"/smart/atomic-branches.git && - # Tell "$up" about two branches for now + # Tell "$up" about three branches for now test_commit atomic1 && test_commit atomic2 && git branch collateral && - git push "$up" master collateral && + git branch other && + git push "$up" master collateral other && # collateral is a valid push, but should be failed by atomic push git checkout collateral &&
@@ -226,6 +227,41 @@ test_expect_success 'push --atomic also prevents branch creation, reports collat grep "^ ! .*rejected.* collateral -> collateral .*atomic push failed" output ' +test_expect_success 'push --atomic fails on server-side errors' ' + # Use previously set up repository + d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git && + test_config -C "$d" http.receivepack true && + up="$HTTPD_URL"/smart/atomic-branches.git && + + # break ref updates for other on the remote site + mkdir "$d/refs/heads/other.lock" && + + # add the new commit to other + git branch -f other collateral && + + # --atomic should cause entire push to be rejected + test_must_fail git push --atomic "$up" atomic other 2>output && + + # the new branch should not have been created upstream + test_must_fail git -C "$d" show-ref --verify refs/heads/atomic && + + # upstream should still reflect atomic2, the last thing we pushed + # successfully + git rev-parse atomic2 >expected && + # ...to other. + git -C "$d" rev-parse refs/heads/other >actual && + test_cmp expected actual && + + # the new branch should not have been created upstream + test_must_fail git -C "$d" show-ref --verify refs/heads/atomic && + + # the failed refs should be indicated to the user + grep "^ ! .*rejected.* other -> other .*atomic transaction failed" output && + + # the collateral failure refs should be indicated to the user + grep "^ ! .*rejected.* atomic -> atomic .*atomic transaction failed" output +' + test_expect_success 'push --all can push to empty repo' ' d=$HTTPD_DOCUMENT_ROOT_PATH/empty-all.git && git init --bare "$d" &&
transport-helper.c +4 -0
@@ -854,6 +854,10 @@ static void set_common_push_options(struct transport *transport,die (_ ("helper %s does not support --signed=if-asked" ), name); } +if (flags &TRANSPORT_PUSH_ATOMIC ) +if (set_helper_option (transport,TRANS_OPT_ATOMIC ,"true" ) !=0 ) +die (_ ("helper %s does not support --atomic" ), name); +if (flags &TRANSPORT_PUSH_OPTIONS ) {struct string_list_item *item;for_each_string_list_item (item, transport->push_options)
transport.h +3 -0
@@ -208,6 +208,9 @@ void transport_check_allowed(const char *type);/* Filter objects for partial clone and fetch */ #define TRANS_OPT_LIST_OBJECTS_FILTER "filter" +/* Request atomic (all-or-nothing) updates when pushing */ +#define TRANS_OPT_ATOMIC "atomic" +/** * Returns 0 if the option was used, non-zero otherwise. Prints a * message to stderr if the option is not used.
Quoted 6 lines · brian m. carlson
Fix this by passing the option from the transport code through to remote helpers, and from the HTTP remote helper down to send-pack. With this change, we can detect if the server side rejects the push and report back appropriately. Note the difference in the messages: the remote side reports "atomic transaction failed", while our own checking rejects pushes with the message "atomic push failed".
Good find. The patch looks good to me, except for one minor style nit in the documentation (see below).
Quoted 2 lines
Document the atomic option in the remote helper documentation, so other implementers can implement it if they like.
I wondered what would happen for existing helpers that do not implement the option, but the behavior here:
transport-helper.c static void set_common_push_options(struct transport *transport, · line 860
Quoted 4 lines
transport-helper.c +4
-0+if (flags &TRANSPORT_PUSH_ATOMIC ) +if (set_helper_option (transport,TRANS_OPT_ATOMIC ,"true" ) !=0 ) +die (_ ("helper %s does not support --atomic" ), name); +
looks like the right thing.
Quoted 3 lines
As I mentioned in the commit message, to my knowledge, this functionality has been broken since the atomic capability was introduced circa 2.4.0.
Yeah, I tried this with v2.4.0 and it had the same problem (plus it's very clear from your patch that it's not a regression, but just that nobody bothered to implement it in the first place).
Documentation/gitremote-helpers.txt set by Git if the remote helper has the 'option' capability. · line 516
Quoted 13 lines
Documentation/gitremote-helpers.txt +5
-0@@ -509,6 +509,11 @@ set by Git if the remote helper has the 'option' capability. Indicate that only the objects wanted need to be fetched, not their dependents. +'option atomic' {'true'|'false'}:: + When pushing, request the remote server to update refs in a single atomic + transaction. If successful, all refs will be updated, or none will. If the + remote side does not support this capability, the push will fail. +
This is implemented with a single space, but the rest of the option bodies are indented with a tab. Asciidoc seems to format it identically either way, though.
-Peff
Quoted 9 lines · Jeff King
Quoted 8 lines · brian m. carlson
Documentation/gitremote-helpers.txt +0
-0@@ -509,6 +509,11 @@ set by Git if the remote helper has the 'option' capability. Indicate that only the objects wanted need to be fetched, not their dependents.
Quoted 9 lines
Documentation/gitremote-helpers.txt set by Git if the remote helper has the 'option' capability. · line 516
Quoted 5 lines
Documentation/gitremote-helpers.txt +5
-0+'option atomic' {'true'|'false'}:: + When pushing, request the remote server to update refs in a single atomic + transaction. If successful, all refs will be updated, or none will. If the + remote side does not support this capability, the push will fail. +This is implemented with a single space, but the rest of the option bodies are indented with a tab. Asciidoc seems to format it identically either way, though.
Yeah, my default editor configuration for AsciiDoc is two spaces. I noticed that Junio's already picked it up for next, but I'll send a v2 with this fixed in case he wants to merge the fixed version to master instead.
If it's more convenient, I can send a follow-up patch fixing the whitespace.
Signature · 2 lines
brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204
Quoted 6 lines · brian m. carlson
Yeah, my default editor configuration for AsciiDoc is two spaces. I noticed that Junio's already picked it up for next, but I'll send a v2 with this fixed in case he wants to merge the fixed version to master instead.
If it's more convenient, I can send a follow-up patch fixing the whitespace.
Either is fine. At this point in the cycle where we are very close to -rc0, I do not mind too much about having a few reverts of 'next' to give a clean slate to an obvious and trivial fix.
Thanks.
PATCH v2 1/1 remote-curl: pass on atomic capability to remote side
In reply to brian m. carlsonWhen pushing more than one reference with the --atomic option, the server is supposed to perform a single atomic transaction to update the references, leaving them either all to succeed or all to fail. This works fine when pushing locally or over SSH, but when pushing over HTTP, we fail to pass the atomic capability to the remote side. In fact, we have not reported this capability to any remote helpers during the life of the feature.
Now normally, things happen to work nevertheless, since we actually check for most types of failures, such as non-fast-forward updates, on the client side, and just abort the entire attempt. However, if the server side reports a problem, such as the inability to lock a ref, the transaction isn't atomic, because we haven't passed the appropriate capability over and the remote side has no way of knowing that we wanted atomic behavior.
Fix this by passing the option from the transport code through to remote helpers, and from the HTTP remote helper down to send-pack. With this change, we can detect if the server side rejects the push and report back appropriately. Note the difference in the messages: the remote side reports "atomic transaction failed", while our own checking rejects pushes with the message "atomic push failed".
Document the atomic option in the remote helper documentation, so other implementers can implement it if they like.
---
Changes from v1:
* Use tabs instead of spaces in docs.
Documentation/gitremote-helpers.txt | 5 ++++ remote-curl.c | 13 +++++++++- t/t5541-http-push-smart.sh | 40 +++++++++++++++++++++++++++-- transport-helper.c | 4 +++ transport.h | 3 +++ 5 files changed, 62 insertions(+), 3 deletions(-)
Documentation/gitremote-helpers.txt +5 -0
@@ -509,6 +509,11 @@ set by Git if the remote helper has the 'option' capability. Indicate that only the objects wanted need to be fetched, not their dependents. +'option atomic' {'true'|'false'}:: + When pushing, request the remote server to update refs in a single atomic + transaction. If successful, all refs will be updated, or none will. If the + remote side does not support this capability, the push will fail. + SEE ALSO -------- linkgit:git-remote[1]
remote-curl.c +12 -1
@@ -40,7 +40,8 @@ struct options { push_cert : 2, deepen_relative : 1, from_promisor : 1, - no_dependents : 1; + no_dependents : 1, + atomic : 1; };static struct options options;static struct string_list cas_options =STRING_LIST_INIT_DUP ;
@@ -148,6 +149,14 @@ static int set_option(const char *name, const char *value)else return -1 ;return 0 ; + }else if (!strcmp (name,"atomic" )) { +if (!strcmp (value,"true" )) + options.atomic =1 ; +else if (!strcmp (value,"false" )) + options.atomic =0 ; +else +return -1 ; +return 0 ; }else if (!strcmp (name,"push-option" )) {if (*value !='"' )string_list_append (&options.push_options, value);
@@ -1196,6 +1205,8 @@ static int push_git(struct discovery *heads, int nr_spec, char **specs)argv_array_push (&args,"--signed=yes" ); else if (options.push_cert == SEND_PACK_PUSH_CERT_IF_ASKED) argv_array_push(&args, "--signed=if-asked"); + if (options.atomic) +argv_array_push (&args,"--atomic" );if (options.verbosity ==0 )argv_array_push (&args,"--quiet" );else if (options.verbosity >1 )
t/t5541-http-push-smart.sh +38 -2
@@ -184,11 +184,12 @@ test_expect_success 'push --atomic also prevents branch creation, reports collat test_config -C "$d" http.receivepack true && up="$HTTPD_URL"/smart/atomic-branches.git && - # Tell "$up" about two branches for now + # Tell "$up" about three branches for now test_commit atomic1 && test_commit atomic2 && git branch collateral && - git push "$up" master collateral && + git branch other && + git push "$up" master collateral other && # collateral is a valid push, but should be failed by atomic push git checkout collateral &&
@@ -226,6 +227,41 @@ test_expect_success 'push --atomic also prevents branch creation, reports collat grep "^ ! .*rejected.* collateral -> collateral .*atomic push failed" output ' +test_expect_success 'push --atomic fails on server-side errors' ' + # Use previously set up repository + d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git && + test_config -C "$d" http.receivepack true && + up="$HTTPD_URL"/smart/atomic-branches.git && + + # break ref updates for other on the remote site + mkdir "$d/refs/heads/other.lock" && + + # add the new commit to other + git branch -f other collateral && + + # --atomic should cause entire push to be rejected + test_must_fail git push --atomic "$up" atomic other 2>output && + + # the new branch should not have been created upstream + test_must_fail git -C "$d" show-ref --verify refs/heads/atomic && + + # upstream should still reflect atomic2, the last thing we pushed + # successfully + git rev-parse atomic2 >expected && + # ...to other. + git -C "$d" rev-parse refs/heads/other >actual && + test_cmp expected actual && + + # the new branch should not have been created upstream + test_must_fail git -C "$d" show-ref --verify refs/heads/atomic && + + # the failed refs should be indicated to the user + grep "^ ! .*rejected.* other -> other .*atomic transaction failed" output && + + # the collateral failure refs should be indicated to the user + grep "^ ! .*rejected.* atomic -> atomic .*atomic transaction failed" output +' + test_expect_success 'push --all can push to empty repo' ' d=$HTTPD_DOCUMENT_ROOT_PATH/empty-all.git && git init --bare "$d" &&
transport-helper.c +4 -0
@@ -854,6 +854,10 @@ static void set_common_push_options(struct transport *transport,die (_ ("helper %s does not support --signed=if-asked" ), name); } +if (flags &TRANSPORT_PUSH_ATOMIC ) +if (set_helper_option (transport,TRANS_OPT_ATOMIC ,"true" ) !=0 ) +die (_ ("helper %s does not support --atomic" ), name); +if (flags &TRANSPORT_PUSH_OPTIONS ) {struct string_list_item *item;for_each_string_list_item (item, transport->push_options)
transport.h +3 -0
@@ -208,6 +208,9 @@ void transport_check_allowed(const char *type);/* Filter objects for partial clone and fetch */ #define TRANS_OPT_LIST_OBJECTS_FILTER "filter" +/* Request atomic (all-or-nothing) updates when pushing */ +#define TRANS_OPT_ATOMIC "atomic" +/** * Returns 0 if the option was used, non-zero otherwise. Prints a * message to stderr if the option is not used.