Skip ls-refs if possible for HTTP
cover Skip ls-refs if possible for HTTP
This was discovered by one of my colleagues when using a partial clone. I thought I had resolved the problem with the commits mentioned in patch 1 (e70a3030e7 and ancestors), but apparently that is not the case (that only worked for native protocols). So here is a fix for HTTP.
I'm not sure of the value of the test in patch 2, but that test does fail if I don't update fetch_refs_from_bundle() to first call get_refs_from_bundle() if it hasn't already been called.
Jonathan Tan (2):
transport-helper: skip ls-refs if unnecessary
transport: teach all vtables to allow fetch first
t/t5607-clone-bundle.sh | 11 +++++++++++ t/t5702-protocol-v2.sh | 13 +++++++++++++ transport-helper.c | 38 ++++++++++++++++++++++++++++++++------ transport-internal.h | 6 ------ transport.c | 18 ++++++------------ 5 files changed, 62 insertions(+), 24 deletions(-)
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
PATCH 1/2 transport-helper: skip ls-refs if unnecessary
Commit e70a3030e7 ("fetch: do not list refs if fetching only hashes", 2018-10-07) and its ancestors taught Git, as an optimization, to skip the ls-refs step when it is not necessary during a protocol v2 fetch (for example, when lazy fetching a missing object in a partial clone, or when running "git fetch --no-tags <remote> <SHA-1>"). But that was only done for natively supported protocols; in particular, HTTP was not supported.
Teach Git to skip ls-refs when using remote helpers that support connect or stateless-connect. To do this, fetch() is made an acceptable entry point. Because fetch() can now be the first function in the vtable called, "get_helper(transport);" has to be added to the beginning of that function to set the transport up (if not yet set up) before process_connect() is invoked.
When fetch() is called, the transport could be taken over (this happens if "connect" or "stateless-connect" is successfully run without any "fallback" response), or not. If the transport is taken over, execution continues like execution for natively supported protocols (fetch_refs_via_pack() is executed, which will fetch refs using ls-refs if needed). If not, the remote helper interface will invoke get_refs_list() if it hasn't been invoked yet, preserving existing behavior.
--- t/t5702-protocol-v2.sh | 13 +++++++++++++ transport-helper.c | 39 +++++++++++++++++++++++++++++++++------ 2 files changed, 46 insertions(+), 6 deletions(-)
t/t5702-protocol-v2.sh +13 -0
@@ -631,6 +631,19 @@ test_expect_success 'fetch with http:// using protocol v2' ' grep "git< version 2" log ' +test_expect_success 'fetch with http:// by hash without tag following with protocol v2 does not list refs' ' + test_when_finished "rm -f log" && + + test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" two_a && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" rev-parse two_a >two_a_hash && + + GIT_TRACE_PACKET="$(pwd)/log" git -C http_child -c protocol.version=2 \ + fetch --no-tags origin $(cat two_a_hash) && + + grep "fetch< version 2" log && + ! grep "fetch> command=ls-refs" log +' + test_expect_success 'fetch from namespaced repo respects namespaces' ' test_when_finished "rm -f log" &&
transport-helper.c +33 -6
@@ -33,6 +33,16 @@ struct helper_data { check_connectivity : 1, no_disconnect_req : 1, no_private_update : 1; + +/* +* As an optimization, the transport code may invoke fetch before +* get_refs_list. If this happens, and if the transport helper doesn't +* support connect or stateless_connect, we need to invoke +* get_refs_list ourselves if we haven't already done so. Keep track of +* whether we have invoked get_refs_list. +*/ + unsigned get_refs_list_called : 1; +char *export_marks;char *import_marks;/* These go from remote name (as in "list") to private name */
@@ -652,17 +662,25 @@ static int connect_helper(struct transport *transport, const char *name,return 0 ; } +static struct ref *get_refs_list_using_list (struct transport *transport, +int for_push); +static int fetch (struct transport *transport,int nr_heads,struct ref **to_fetch) {struct helper_data *data = transport->data;int i, count; +get_helper (transport); +if (process_connect (transport,0 )) {do_take_over (transport);return transport->vtable->fetch (transport, nr_heads, to_fetch); } +if (!data->get_refs_list_called) +get_refs_list_using_list (transport,0 ); + count =0 ;for (i =0 ; i < nr_heads; i++)if (!(to_fetch[i]->status &REF_STATUS_UPTODATE ))
@@ -1058,6 +1076,19 @@ static int has_attribute(const char *attrs, const char *attr)static struct ref *get_refs_list (struct transport *transport,int for_push,const struct argv_array *ref_prefixes) +{ +get_helper (transport); + +if (process_connect (transport, for_push)) { +do_take_over (transport); +return transport->vtable->get_refs_list (transport, for_push, ref_prefixes); + } + +return get_refs_list_using_list (transport, for_push); +} + +static struct ref *get_refs_list_using_list (struct transport *transport, +int for_push) {struct helper_data *data = transport->data;struct child_process *helper;
@@ -1066,13 +1097,9 @@ static struct ref *get_refs_list(struct transport *transport, int for_push,struct ref *posn;struct strbuf buf =STRBUF_INIT ; + data->get_refs_list_called =1 ; helper =get_helper (transport); -if (process_connect (transport, for_push)) { -do_take_over (transport); -return transport->vtable->get_refs_list (transport, for_push, ref_prefixes); - } -if (data->push && for_push)write_str_in_full (helper->in,"list for-push\n" );else
@@ -1119,7 +1146,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push, }static struct transport_vtable vtable = { -0 , +1 , set_helper_option, get_refs_list, fetch,
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
PATCH 2/2 transport: teach all vtables to allow fetch first
In reply to Jonathan TanThe only transport that does not allow fetch() to be called before get_refs_list() is the bundle transport. Clean up the code by teaching the bundle transport the ability to do this, and removing support for transports that don't support this order of invocation.
--- t/t5607-clone-bundle.sh | 11 +++++++++++ transport-helper.c | 1 - transport-internal.h | 6 ------ transport.c | 18 ++++++------------ 4 files changed, 17 insertions(+), 19 deletions(-)
t/t5607-clone-bundle.sh +11 -0
@@ -83,4 +83,15 @@ test_expect_success 'failed bundle creation does not leave cruft' ' test_path_is_missing fail.bundle.lock ' +test_expect_success 'fetch SHA-1 from bundle' ' + test_create_repo foo && + test_commit -C foo x && + git -C foo bundle create tip.bundle -1 master && + git -C foo rev-parse HEAD >hash && + + # Exercise to ensure that fetching a SHA-1 from a bundle works with no + # errors + git fetch --no-tags foo/tip.bundle "$(cat hash)" +' + test_done
transport-helper.c +0 -1
@@ -1146,7 +1146,6 @@ static struct ref *get_refs_list_using_list(struct transport *transport, }static struct transport_vtable vtable = { -1 , set_helper_option, get_refs_list, fetch,
transport-internal.h +0 -6
@@ -6,12 +6,6 @@ struct transport;struct argv_array ;struct transport_vtable { -/** -* This transport supports the fetch() function being called -* without get_refs_list() first being called. -*/ -unsigned fetch_without_list :1 ; -/** * Returns 0 if successful, positive if the option is not * recognized or is inapplicable, and negative if the option
transport.c +6 -12
@@ -122,6 +122,7 @@ static void set_upstreams(struct transport *transport, struct ref *refs,struct bundle_transport_data {int fd;struct bundle_header header; +unsigned get_refs_from_bundle_called :1 ; }; static struct ref *get_refs_from_bundle(struct transport *transport,
@@ -135,6 +136,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport,if (for_push)return NULL ; + data->get_refs_from_bundle_called =1 ; +if (data->fd >0 )close (data->fd); data->fd =read_bundle_header (transport->url, &data->header);
@@ -154,6 +157,9 @@ static int fetch_refs_from_bundle(struct transport *transport, int nr_heads, struct ref **to_fetch) { struct bundle_transport_data *data = transport->data; + +if (!data->get_refs_from_bundle_called) +get_refs_from_bundle (transport,0 ,NULL );return unbundle (the_repository, &data->header, data->fd, transport->progress ?BUNDLE_VERBOSE :0 ); }
@@ -742,7 +748,6 @@ static int disconnect_git(struct transport *transport) }static struct transport_vtable taken_over_vtable = { -1 ,NULL , get_refs_via_connect, fetch_refs_via_pack,
@@ -892,7 +897,6 @@ void transport_check_allowed(const char *type) }static struct transport_vtable bundle_vtable = { -0 ,NULL , get_refs_from_bundle, fetch_refs_from_bundle,
@@ -902,7 +906,6 @@ static struct transport_vtable bundle_vtable = { };static struct transport_vtable builtin_smart_vtable = { -1 ,NULL , get_refs_via_connect, fetch_refs_via_pack,
@@ -1285,15 +1288,6 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)struct ref **heads =NULL ;struct ref *rm; -if (!transport->vtable->fetch_without_list) -/* -* Some transports (e.g. the built-in bundle transport and the -* transport helper interface) do not work when fetching is -* done immediately after transport creation. List the remote -* refs anyway (if not already listed) as a workaround. -*/ -transport_get_remote_refs (transport,NULL ); -for (rm = refs; rm; rm = rm->next) { nr_refs++; if (rm->peer_ref &&
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
Quoted 8 lines · Jonathan Tan
This was discovered by one of my colleagues when using a partial clone. I thought I had resolved the problem with the commits mentioned in patch 1 (
e70a3030e7and ancestors), but apparently that is not the case (that only worked for native protocols). So here is a fix for HTTP.I'm not sure of the value of the test in patch 2, but that test does fail if I don't update
fetch_refs_from_bundle()to first callget_refs_from_bundle()if it hasn't already been called.
This probably is totally off-tangent, but do any of these "let's advertise fewer" changes at the protocol level have to take into account the use of --prune option on the client side?
Quoted 10 lines
Jonathan Tan (2):
transport-helper: skip ls-refs if unnecessary
transport: teach all vtables to allow fetch firstt/t5607-clone-bundle.sh | 11 +++++++++++ t/t5702-protocol-v2.sh | 13 +++++++++++++ transport-helper.c | 38 ++++++++++++++++++++++++++++++++------ transport-internal.h | 6 ------ transport.c | 18 ++++++------------ 5 files changed, 62 insertions(+), 24 deletions(-)
Quoted 3 lines
This probably is totally off-tangent, but do any of these "let's advertise fewer" changes at the protocol level have to take into account the use of --prune option on the client side?
I don't think so. According to what I understand from the documentation, the prune option prunes based on the RHS of the refspec, and it doesn't affect anything non-matching. When we advertise fewer, the only refs that are missing are those that are non-matching anyway.
Some experimentation seems to show that that is the case also:
$ git init one [snip] $ git -C one commit --allow-empty -m x [master (root-commit) 67056ac] x $ git -C one branch maste $ git -C one branch other $ git clone "$(pwd)/one" two Cloning into 'two'... done. $ git -C one branch -d maste Deleted branch maste (was 67056ac). $ git -C two checkout --detach HEAD HEAD is now at 67056ac x $ GIT_TRACE_PACKET=1 git -c protocol.version=2 -C two fetch --prune origin refs/heads/m*:refs/remotes/origin/m* [snipped lots of stuff; basically, only refs/heads/master is sent] 10:15:46.308264 pkt-line.c:80 packet: fetch> ref-prefix refs/heads/m 10:15:46.308276 pkt-line.c:80 packet: fetch> ref-prefix refs/tags/ 10:15:46.308523 pkt-line.c:80 packet: fetch< 67056ac6d07814334716df760054ac5bec05b66a refs/heads/master From /usr/local/google/home/jonathantanmy/tmp/g/one - [deleted] (none) -> origin/maste $ git -C two for-each-ref 67056ac6d07814334716df760054ac5bec05b66a commit refs/heads/master 67056ac6d07814334716df760054ac5bec05b66a commit refs/remotes/origin/HEAD 67056ac6d07814334716df760054ac5bec05b66a commit refs/remotes/origin/master 67056ac6d07814334716df760054ac5bec05b66a commit refs/remotes/origin/other
(Notice that refs/remotes/origin/other is untouched.)