ref-in-want
cover ref-in-want
This series adds the ref-in-want feature which was originally proposed by Jonathan Tan
(https://public-inbox.org/git/[email protected]/).
Back when ref-in-want was first discussed it was decided that we should first solve the issue of moving to a new wire format and find a way to limit the ref-advertisement before moving forward with ref-in-want. Now that protocol version 2 is a reality, and that refs can be filtered on the server side, we can revisit ref-in-want.
This version of ref-in-want is a bit more restrictive than what Jonathan originally proposed (only full ref names are allowed instead of globs and OIDs), but it is meant to accomplish the same goal (solve the issues of refs changing during negotiation).
Brandon Williams (8):
test-pkt-line: add unpack-sideband subcommand
upload-pack: implement ref-in-want
upload-pack: test negotiation with changing repository
fetch: refactor the population of peer ref OIDs
fetch: refactor fetch_refs into two functions
fetch: refactor to make function args narrower
fetch-pack: put shallow info in output parameter
fetch-pack: implement ref-in-want
Documentation/config.txt | 4 + Documentation/technical/protocol-v2.txt | 28 ++- builtin/clone.c | 4 +- builtin/fetch.c | 126 +++++++----- fetch-object.c | 2 +- fetch-pack.c | 52 +++-- remote.c | 1 + remote.h | 1 + t/helper/test-pkt-line.c | 37 ++++ t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 + t/lib-httpd/one-time-sed.sh | 16 ++ t/t5703-upload-pack-ref-in-want.sh | 245 ++++++++++++++++++++++++ transport-helper.c | 6 +- transport-internal.h | 9 +- transport.c | 34 +++- transport.h | 3 +- upload-pack.c | 64 +++++++ 18 files changed, 564 insertions(+), 77 deletions(-) create mode 100644 t/lib-httpd/one-time-sed.sh create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Signature · 1 line
2.17.1.1185.g55be947832-goog
PATCH 1/8 test-pkt-line: add unpack-sideband subcommand
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
--- t/helper/test-pkt-line.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
t/helper/test-pkt-line.c +37 -0
@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h" static void pack_line (const char *line)
@@ -48,6 +49,40 @@ static void unpack(void) } } +static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE ); + +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) { +case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ; +if (band ==1 ) + fd =1 ; +else + fd =2 ; + +write_or_die (fd, reader.line+1 , reader.pktlen-1 ); + +if (band ==3 ) +die ("sind-band error" ); +break ; +case PACKET_READ_FLUSH : +return ; +case PACKET_READ_DELIM : +break ; + } + } +} +int cmd_main (int argc,const char **argv) {if (argc <2 )
@@ -57,6 +92,8 @@ int cmd_main(int argc, const char **argv)pack (argc -2 , argv +2 ); else if (!strcmp(argv[1], "unpack")) unpack(); + else if (!strcmp(argv[1], "unpack-sideband")) + unpack_sideband(); else die("invalid argument '%s'", argv[1]);
Signature · 1 line
2.17.1.1185.g55be947832-goog
PATCH 2/8 upload-pack: implement ref-in-want
In reply to Brandon WilliamsCurrently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement.
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
--- Documentation/config.txt | 4 + Documentation/technical/protocol-v2.txt | 28 ++++- t/t5703-upload-pack-ref-in-want.sh | 153 ++++++++++++++++++++++++ upload-pack.c | 64 ++++++++++ 4 files changed, 248 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Documentation/config.txt +4 -0
@@ -3479,6 +3479,10 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories). +uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. + url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves a
Documentation/technical/protocol-v2.txt +27 -1
@@ -299,12 +299,21 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values. +If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server than the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. output = *section - section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt) acknowledgments = PKT-LINE("acknowledgments" LF)
@@ -319,6 +328,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-Line(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff)
@@ -379,6 +392,19 @@ header. * This section is only included if a packfile section is also included in the response. + wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs" + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * Ther server MUST NOT send any refs which were not requested + using 'want-ref' lines. + packfile section * This section is only included if the client has sent 'want' lines in its request and either requested that no more
t/t5703-upload-pack-ref-in-want.sh new file +153 -0
153 lines not on this page · raw message
upload-pack.c +64 -0
136 lines not on this page · raw message
Signature · 1 line
2.17.1.1185.g55be947832-goog
Re: [PATCH 2/8] upload-pack: implement ref-in-want
Quoted 65 lines · Brandon Williams
Currently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement.
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
Signed-off-by Brandon Williams <[email protected]>--- Documentation/config.txt | 4 + Documentation/technical/protocol-v2.txt | 28 ++++- t/t5703-upload-pack-ref-in-want.sh | 153 ++++++++++++++++++++++++ upload-pack.c | 64 ++++++++++ 4 files changed, 248 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.shDocumentation/config.txt +4
-0@@ -3479,6 +3479,10 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories). +uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. + url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves aDocumentation/technical/protocol-v2.txt +10
-1@@ -299,12 +299,21 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values. +If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server than the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. output = *section - section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt) acknowledgments = PKT-LINE("acknowledgments" LF)
Documentation/technical/protocol-v2.txt header. · line 332
Quoted 5 lines
Documentation/technical/protocol-v2.txt +2
-0@@ -319,6 +328,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-Line(wanted-ref LF)
s/PKT-Line/PKT-LINE/
ATB,
Ramsay Jones
Re: [PATCH 2/8] upload-pack: implement ref-in-want
In reply to Brandon WilliamsDocumentation/config.txt Note that this configuration variable is ignored if it is seen in the · line 3485
Quoted 4 lines · Brandon Williams
Documentation/config.txt +4
-0+uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. +
I think it makes sense to elaborate a bit on what this is for. Having read this series through, and to make sure I understood this, maybe something like this:
This feature is intended for the benefit of load-balanced servers which may not have the same view of what SHA-1s their refs point to, but are guaranteed to never advertise a reference that another server serving the request doesn't know about.
I.e. from what I can tell this gives no benefits for someone using a monolithic git server, except insofar as there would be a slight decrease in network traffic if the average length of refs is less than the length of a SHA-1.
That's fair enough, just something we should prominently say.
It does have the "disadvantage", if you can call it that, that it's introducing a race condition between when we read the ref advertisement and are promised XYZ refs, but may actually get ABC, but I can't think of a reason anyone would care about this in practice.
The reason I'm saying "another server [...] doesn't know about" above is that 2/8 has this:
if (read_ref(arg, &oid))
die("unknown ref %s", arg);Doesn't that mean that if server A in your pool advertises master, next & pu, and you then go and fetch from server B advertising master & next, but not "pu" that the clone will die?
Presumably at Google you either have something to ensure a consistent view, e.g. only advertise refs by name older than N seconds, or globally update ref name but not their contents, and don't allow deleting refs (or give them the same treatment).
But that, and again, I may have misunderstood this whole thing, significantly reduces the utility of this feature for anyone "in the wild" since nothing shipped with "git" gives you that feature.
The naïve way to do slave mirroring with stock git is to have a post-receive hook that pushes to your mirrors in a for-loop, or has them fetch from the master in a loop, and then round-robin LB those servers. Due to the "die on nonexisting" semantics in this extension that'll result in failed clones.
So I think we should either be really vocal about that caveat, or perhaps think of how we could make that configurable, e.g. what happens if the server says "sorry, don't know about that one", and carries on with the rest it does know about?
Is there a way for client & server to gracefully recover from that? E.g. send "master" & "next" now, and when I pull again in a few seconds I get the new "pu"?
Also, as a digression isn't that a problem shared with protocol v2 in general? I.e. without this extension isn't it going to make another connection to the naïve LB'd mirroring setup described above and find that SHA-1s as well as refs don't match?
BREAK.
Also is if this E-Mail wasn't long enough, on a completely different topic, in an earlier discussion in
https://public-inbox.org/git/[email protected]/ I noted that it would be neat-o to have optional wildmatch/pcre etc. matching for the use case you're not caring about here (and I don't expect you to, you're solving a different problem).
But let's say I want to add that after this, and being unfamiliar with the protocol v2 conventions. Would that be a whole new ref-in-want-wildmatch-prefix capability with a new want-ref-wildmatch-prefix verb, or is there some less verbose way we can anticipate that use-case and internally version / advertise sub-capabilities?
I don't know if that makes any sense, and would be fine with just a ref-in-want-wildmatch-prefix if that's the way to do it. I just think it's inevitable that we'll have such a thing eventually, so it's worth thinking about how such a future extension fits in.
Re: [PATCH 2/8] upload-pack: implement ref-in-want
Quoted 17 lines · Ævar Arnfjörð Bjarmason
Documentation/config.txt Note that this configuration variable is ignored if it is seen in the · line 3485
Quoted 4 lines · Brandon Williams
Documentation/config.txt +4
-0+uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. +I think it makes sense to elaborate a bit on what this is for. Having read this series through, and to make sure I understood this, maybe something like this:
This feature is intended for the benefit of load-balanced servers which may not have the same view of what SHA-1s their refs point to, but are guaranteed to never advertise a reference that another server serving the request doesn't know about.
Quoted 4 lines
I.e. from what I can tell this gives no benefits for someone using a monolithic git server, except insofar as there would be a slight decrease in network traffic if the average length of refs is less than the length of a SHA-1.
Yeah I agree that the motivation should probably be spelled out more, thanks for the suggestion.
Quoted 33 lines
That's fair enough, just something we should prominently say.
It does have the "disadvantage", if you can call it that, that it's introducing a race condition between when we read the ref advertisement and are promised XYZ refs, but may actually get ABC, but I can't think of a reason anyone would care about this in practice.
The reason I'm saying "another server [...] doesn't know about" above is that 2/8 has this:
if (read_ref(arg, &oid)) die("unknown ref %s", arg);Doesn't that mean that if server A in your pool advertises master, next & pu, and you then go and fetch from server B advertising master & next, but not "pu" that the clone will die?
Presumably at Google you either have something to ensure a consistent view, e.g. only advertise refs by name older than N seconds, or globally update ref name but not their contents, and don't allow deleting refs (or give them the same treatment).
But that, and again, I may have misunderstood this whole thing, significantly reduces the utility of this feature for anyone "in the wild" since nothing shipped with "git" gives you that feature.
The naïve way to do slave mirroring with stock git is to have a post-receive hook that pushes to your mirrors in a for-loop, or has them fetch from the master in a loop, and then round-robin LB those servers. Due to the "die on nonexisting" semantics in this extension that'll result in failed clones.
Quoted 4 lines
So I think we should either be really vocal about that caveat, or perhaps think of how we could make that configurable, e.g. what happens if the server says "sorry, don't know about that one", and carries on with the rest it does know about?
Jonathan actually pointed this out to me earlier and I think the best way to deal with this is to just ignore the refs that the server doesn't know about instead of dying here. I mean its no worse than what we already have and we shouldn't hit this case too often. And that way the fetch can still proceed.
Quoted 4 lines
Is there a way for client & server to gracefully recover from that? E.g. send "master" & "next" now, and when I pull again in a few seconds I get the new "pu"?
I think in this case the client would just need to wait for some amount of replication delay and attempt fetching at a later point.
Quoted 5 lines
Also, as a digression isn't that a problem shared with protocol v2 in general? I.e. without this extension isn't it going to make another connection to the naïve LB'd mirroring setup described above and find that SHA-1s as well as refs don't match?
This is actually an issue with fetch using either v2 or v0. Unless I'm misunderstanding what you're asking here.
Quoted 17 lines
BREAK.
Also is if this E-Mail wasn't long enough, on a completely different topic, in an earlier discussion in
https://public-inbox.org/git/[email protected]/ I noted that it would be neat-o to have optional wildmatch/pcre etc. matching for the use case you're not caring about here (and I don't expect you to, you're solving a different problem).But let's say I want to add that after this, and being unfamiliar with the protocol v2 conventions. Would that be a whole new ref-in-want-wildmatch-prefix capability with a new want-ref-wildmatch-prefix verb, or is there some less verbose way we can anticipate that use-case and internally version / advertise sub-capabilities?
Quoted 4 lines
I don't know if that makes any sense, and would be fine with just a ref-in-want-wildmatch-prefix if that's the way to do it. I just think it's inevitable that we'll have such a thing eventually, so it's worth thinking about how such a future extension fits in.
Yes back when introducing the server-side ref filtering in ls-refs we originally talked about included wildmatch or other forms of pattern matching. We opted to not over complicate things and favored prefix matching because it didn't bake in some subset of globbing or regex and it was easier to compute on the server side.
Anyway back to your question. Yes if at some point in the future we wanted to add in wildmatch/pcre to the protocol for ls-refs or for ref-in-want then it could be added as a feature or capability. I don't think it would require adding a whole new verb (it probably would for the ls-refs case since the verb used there is "ref-prefix") but the capability could mean that the "want-ref" verb now understands wildmatch patterns in addition to fully qualified refs.
Signature · 1 line
Brandon Williams
Re: [PATCH 2/8] upload-pack: implement ref-in-want
Quoted 79 lines · Brandon Williams
Quoted 17 lines · Ævar Arnfjörð Bjarmason
Documentation/config.txt Note that this configuration variable is ignored if it is seen in the · line 3485
Quoted 4 lines · Brandon Williams
Documentation/config.txt +4
-0+uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. +I think it makes sense to elaborate a bit on what this is for. Having read this series through, and to make sure I understood this, maybe something like this:
This feature is intended for the benefit of load-balanced servers which may not have the same view of what SHA-1s their refs point to, but are guaranteed to never advertise a reference that another server serving the request doesn't know about.
Quoted 4 lines
I.e. from what I can tell this gives no benefits for someone using a monolithic git server, except insofar as there would be a slight decrease in network traffic if the average length of refs is less than the length of a SHA-1.
Yeah I agree that the motivation should probably be spelled out more, thanks for the suggestion.
Quoted 33 lines
That's fair enough, just something we should prominently say.
It does have the "disadvantage", if you can call it that, that it's introducing a race condition between when we read the ref advertisement and are promised XYZ refs, but may actually get ABC, but I can't think of a reason anyone would care about this in practice.
The reason I'm saying "another server [...] doesn't know about" above is that 2/8 has this:
if (read_ref(arg, &oid)) die("unknown ref %s", arg);Doesn't that mean that if server A in your pool advertises master, next & pu, and you then go and fetch from server B advertising master & next, but not "pu" that the clone will die?
Presumably at Google you either have something to ensure a consistent view, e.g. only advertise refs by name older than N seconds, or globally update ref name but not their contents, and don't allow deleting refs (or give them the same treatment).
But that, and again, I may have misunderstood this whole thing, significantly reduces the utility of this feature for anyone "in the wild" since nothing shipped with "git" gives you that feature.
The naïve way to do slave mirroring with stock git is to have a post-receive hook that pushes to your mirrors in a for-loop, or has them fetch from the master in a loop, and then round-robin LB those servers. Due to the "die on nonexisting" semantics in this extension that'll result in failed clones.
Quoted 4 lines
So I think we should either be really vocal about that caveat, or perhaps think of how we could make that configurable, e.g. what happens if the server says "sorry, don't know about that one", and carries on with the rest it does know about?
Jonathan actually pointed this out to me earlier and I think the best way to deal with this is to just ignore the refs that the server doesn't know about instead of dying here. I mean its no worse than what we already have and we shouldn't hit this case too often. And that way the fetch can still proceed.
Quoted 4 lines
Is there a way for client & server to gracefully recover from that? E.g. send "master" & "next" now, and when I pull again in a few seconds I get the new "pu"?
I think in this case the client would just need to wait for some amount of replication delay and attempt fetching at a later point.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Also, as a digression isn't that a problem shared with protocol v2 in general? I.e. without this extension isn't it going to make another connection to the naïve LB'd mirroring setup described above and find that SHA-1s as well as refs don't match?
This is actually an issue with fetch using either v2 or v0. Unless I'm misunderstanding what you're asking here.
Isn't the whole dialog in v1 guaranteed to be with one server from intial ref advertisement to the client saying have/want, or is that just with ssh?
In any case the reason the above is an issue here is because you're getting the advertisement from a different server than you're negotiating the pack with, right?
Quoted 28 lines
Quoted 17 lines
BREAK.
Also is if this E-Mail wasn't long enough, on a completely different topic, in an earlier discussion in
https://public-inbox.org/git/[email protected]/ I noted that it would be neat-o to have optional wildmatch/pcre etc. matching for the use case you're not caring about here (and I don't expect you to, you're solving a different problem).But let's say I want to add that after this, and being unfamiliar with the protocol v2 conventions. Would that be a whole new ref-in-want-wildmatch-prefix capability with a new want-ref-wildmatch-prefix verb, or is there some less verbose way we can anticipate that use-case and internally version / advertise sub-capabilities?
Quoted 4 lines
I don't know if that makes any sense, and would be fine with just a ref-in-want-wildmatch-prefix if that's the way to do it. I just think it's inevitable that we'll have such a thing eventually, so it's worth thinking about how such a future extension fits in.
Yes back when introducing the server-side ref filtering in ls-refs we originally talked about included wildmatch or other forms of pattern matching. We opted to not over complicate things and favored prefix matching because it didn't bake in some subset of globbing or regex and it was easier to compute on the server side.
Quoted 7 lines
Anyway back to your question. Yes if at some point in the future we wanted to add in wildmatch/pcre to the protocol for ls-refs or for ref-in-want then it could be added as a feature or capability. I don't think it would require adding a whole new verb (it probably would for the ls-refs case since the verb used there is "ref-prefix") but the capability could mean that the "want-ref" verb now understands wildmatch patterns in addition to fully qualified refs.
Probably still makes sense to have it be a different verb since some things in wildmatch / regex are metachars but may be valid in ref names.
Thanks!
Re: [PATCH 2/8] upload-pack: implement ref-in-want
Quoted 87 lines · Ævar Arnfjörð Bjarmason
Quoted 84 lines · Brandon Williams
Quoted 17 lines · Ævar Arnfjörð Bjarmason
Documentation/config.txt Note that this configuration variable is ignored if it is seen in the · line 3485
Quoted 4 lines · Brandon Williams
Documentation/config.txt +4
-0+uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. +I think it makes sense to elaborate a bit on what this is for. Having read this series through, and to make sure I understood this, maybe something like this:
This feature is intended for the benefit of load-balanced servers which may not have the same view of what SHA-1s their refs point to, but are guaranteed to never advertise a reference that another server serving the request doesn't know about.
Quoted 4 lines
I.e. from what I can tell this gives no benefits for someone using a monolithic git server, except insofar as there would be a slight decrease in network traffic if the average length of refs is less than the length of a SHA-1.
Yeah I agree that the motivation should probably be spelled out more, thanks for the suggestion.
Quoted 33 lines
That's fair enough, just something we should prominently say.
It does have the "disadvantage", if you can call it that, that it's introducing a race condition between when we read the ref advertisement and are promised XYZ refs, but may actually get ABC, but I can't think of a reason anyone would care about this in practice.
The reason I'm saying "another server [...] doesn't know about" above is that 2/8 has this:
if (read_ref(arg, &oid)) die("unknown ref %s", arg);Doesn't that mean that if server A in your pool advertises master, next & pu, and you then go and fetch from server B advertising master & next, but not "pu" that the clone will die?
Presumably at Google you either have something to ensure a consistent view, e.g. only advertise refs by name older than N seconds, or globally update ref name but not their contents, and don't allow deleting refs (or give them the same treatment).
But that, and again, I may have misunderstood this whole thing, significantly reduces the utility of this feature for anyone "in the wild" since nothing shipped with "git" gives you that feature.
The naïve way to do slave mirroring with stock git is to have a post-receive hook that pushes to your mirrors in a for-loop, or has them fetch from the master in a loop, and then round-robin LB those servers. Due to the "die on nonexisting" semantics in this extension that'll result in failed clones.
Quoted 4 lines
So I think we should either be really vocal about that caveat, or perhaps think of how we could make that configurable, e.g. what happens if the server says "sorry, don't know about that one", and carries on with the rest it does know about?
Jonathan actually pointed this out to me earlier and I think the best way to deal with this is to just ignore the refs that the server doesn't know about instead of dying here. I mean its no worse than what we already have and we shouldn't hit this case too often. And that way the fetch can still proceed.
Quoted 4 lines
Is there a way for client & server to gracefully recover from that? E.g. send "master" & "next" now, and when I pull again in a few seconds I get the new "pu"?
I think in this case the client would just need to wait for some amount of replication delay and attempt fetching at a later point.
Quoted 5 lines
Also, as a digression isn't that a problem shared with protocol v2 in general? I.e. without this extension isn't it going to make another connection to the naïve LB'd mirroring setup described above and find that SHA-1s as well as refs don't match?
Quoted 6 lines
Quoted 2 lines
This is actually an issue with fetch using either v2 or v0. Unless I'm misunderstanding what you're asking here.
Isn't the whole dialog in v1 guaranteed to be with one server from intial ref advertisement to the client saying have/want, or is that just with ssh?
That's only guaranteed with statefull connections (git:// and ssh://),
http:// has this issue because its stateless.
Quoted 4 lines
In any case the reason the above is an issue here is because you're getting the advertisement from a different server than you're negotiating the pack with, right?
Yes correct, or even a different server on each negotiation round-trip.
Quoted 37 lines
Quoted 29 lines
Quoted 18 lines
BREAK.
Also is if this E-Mail wasn't long enough, on a completely different topic, in an earlier discussion in
https://public-inbox.org/git/[email protected]/ I noted that it would be neat-o to have optional wildmatch/pcre etc. matching for the use case you're not caring about here (and I don't expect you to, you're solving a different problem).But let's say I want to add that after this, and being unfamiliar with the protocol v2 conventions. Would that be a whole new ref-in-want-wildmatch-prefix capability with a new want-ref-wildmatch-prefix verb, or is there some less verbose way we can anticipate that use-case and internally version / advertise sub-capabilities?
Quoted 4 lines
I don't know if that makes any sense, and would be fine with just a ref-in-want-wildmatch-prefix if that's the way to do it. I just think it's inevitable that we'll have such a thing eventually, so it's worth thinking about how such a future extension fits in.
Yes back when introducing the server-side ref filtering in ls-refs we originally talked about included wildmatch or other forms of pattern matching. We opted to not over complicate things and favored prefix matching because it didn't bake in some subset of globbing or regex and it was easier to compute on the server side.
Quoted 7 lines
Anyway back to your question. Yes if at some point in the future we wanted to add in wildmatch/pcre to the protocol for ls-refs or for ref-in-want then it could be added as a feature or capability. I don't think it would require adding a whole new verb (it probably would for the ls-refs case since the verb used there is "ref-prefix") but the capability could mean that the "want-ref" verb now understands wildmatch patterns in addition to fully qualified refs.
Quoted 2 lines
Probably still makes sense to have it be a different verb since some things in wildmatch / regex are metachars but may be valid in ref names.
Yeah we can leave that up to the designer of such a feature ;)
Quoted 2 lines
Thanks!
Signature · 1 line
Brandon Williams
PATCH 3/8 upload-pack: test negotiation with changing repository
In reply to Brandon WilliamsAdd tests to check the behavior of fetching from a repository which changes between rounds of negotiation (for example, when different servers in a load-balancing agreement participate in the same stateless RPC negotiation). This forms a baseline of comparison to the ref-in-want functionality (which will be introduced to the client in subsequent commits), and ensures that subsequent commits do not change existing behavior.
As part of this effort, a mechanism to substitute strings in a single HTTP response is added.
--- t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 +++ t/lib-httpd/one-time-sed.sh | 16 ++++++ t/t5703-upload-pack-ref-in-want.sh | 92 ++++++++++++++++++++++++++++++ 4 files changed, 117 insertions(+) create mode 100644 t/lib-httpd/one-time-sed.sh
t/lib-httpd.sh +1 -0
@@ -132,6 +132,7 @@ prepare_httpd() { cp "$TEST_PATH"/passwd "$HTTPD_ROOT_PATH" install_script broken-smart-http.sh install_script error.sh + install_script one-time-sed.sh ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +8 -0
@@ -111,9 +111,14 @@ Alias /auth/dumb/ www/auth/dumb/ SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} SetEnv GIT_HTTP_EXPORT_ALL </LocationMatch> +<LocationMatch /one_time_sed/> + SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} + SetEnv GIT_HTTP_EXPORT_ALL +</LocationMatch> ScriptAliasMatch /smart_*[^/]*/(.*) ${GIT_EXEC_PATH}/git-http-backend/$1 ScriptAlias /broken_smart/ broken-smart-http.sh/ ScriptAlias /error/ error.sh/ +ScriptAliasMatch /one_time_sed/(.*) one-time-sed.sh/$1 <Directory ${GIT_EXEC_PATH}> Options FollowSymlinks </Directory>
@@ -123,6 +128,9 @@ ScriptAlias /error/ error.sh/ <Files error.sh> Options ExecCGI </Files> +<Files one-time-sed.sh> + Options ExecCGI +</Files> <Files ${GIT_EXEC_PATH}/git-http-backend> Options ExecCGI </Files>
t/lib-httpd/one-time-sed.sh new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/sh + +if [ -e one-time-sed ]; then + "$GIT_EXEC_PATH/git-http-backend" >out + + sed "$(cat one-time-sed)" <out >out_modified + + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fi
t/t5703-upload-pack-ref-in-want.sh +92 -0
96 lines not on this page · raw message
Signature · 1 line
2.17.1.1185.g55be947832-goog
PATCH 4/8 fetch: refactor the population of peer ref OIDs
In reply to Brandon WilliamsPopulate peer ref OIDs in get_ref_map instead of do_fetch. Besides tightening scopes of variables in the code, this also prepares for get_ref_map being able to be called multiple times within do_fetch.
--- builtin/fetch.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
builtin/fetch.c +18 -18
@@ -351,6 +351,7 @@ static struct ref *get_ref_map(struct transport *transport,/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs; +struct string_list existing_refs =STRING_LIST_INIT_DUP ;const struct ref *remote_refs;if (rs->nr)
@@ -458,7 +459,23 @@ static struct ref *get_ref_map(struct transport *transport, tail = &rm->next; } -return ref_remove_duplicates (ref_map); + ref_map =ref_remove_duplicates (ref_map); + +for_each_ref (add_existing, &existing_refs); +for (rm = ref_map; rm; rm = rm->next) { +if (rm->peer_ref) { +struct string_list_item *peer_item = +string_list_lookup (&existing_refs, + rm->peer_ref->name); +if (peer_item) { +struct object_id *old_oid = peer_item->util; +oidcpy (&rm->peer_ref->old_oid, old_oid); + } + } + } +string_list_clear (&existing_refs,1 ); + +return ref_map; }#define STORE_REF_ERROR_OTHER 1
@@ -1110,14 +1127,10 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)static int do_fetch (struct transport *transport,struct refspec *rs) { -struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct ref *ref_map; -struct ref *rm;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; -for_each_ref (add_existing, &existing_refs); -if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 ) tags =TAGS_SET ;
@@ -1136,18 +1149,6 @@ static int do_fetch(struct transport *transport,if (!update_head_ok)check_not_current_branch (ref_map); -for (rm = ref_map; rm; rm = rm->next) { -if (rm->peer_ref) { -struct string_list_item *peer_item = -string_list_lookup (&existing_refs, - rm->peer_ref->name); -if (peer_item) { -struct object_id *old_oid = peer_item->util; -oidcpy (&rm->peer_ref->old_oid, old_oid); - } - } - } -if (tags ==TAGS_DEFAULT && autotags)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,"1" );if (prune) {
@@ -1183,7 +1184,6 @@ static int do_fetch(struct transport *transport, } cleanup: -string_list_clear (&existing_refs,1 );return retcode; }
Signature · 1 line
2.17.1.1185.g55be947832-goog
PATCH 5/8 fetch: refactor fetch_refs into two functions
In reply to Brandon WilliamsRefactor the fetch_refs function into a function that does the fetching of refs and another function that stores them.
--- builtin/fetch.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
builtin/fetch.c +13 -6
@@ -967,10 +967,16 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)int ret =quickfetch (ref_map);if (ret) ret =transport_fetch_refs (transport, ref_map); -if (!ret) - ret |=store_updated_refs (transport->url, - transport->remote->name, - ref_map); +if (ret) +transport_unlock_pack (transport); +return ret; +} + +static int consume_refs (struct transport *transport,struct ref *ref_map) +{ +int ret =store_updated_refs (transport->url, + transport->remote->name, + ref_map);transport_unlock_pack (transport);return ret; }
@@ -1116,7 +1122,8 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -fetch_refs (transport, ref_map); +if (!fetch_refs (transport, ref_map)) +consume_refs (transport, ref_map);if (gsecondary) {transport_disconnect (gsecondary);
@@ -1165,7 +1172,7 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map)) { +if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
Signature · 1 line
2.17.1.1185.g55be947832-goog
PATCH 6/8 fetch: refactor to make function args narrower
In reply to Brandon WilliamsRefactor find_non_local_tags and get_ref_map to only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
--- builtin/fetch.c | 52 ++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
builtin/fetch.c +26 -26
@@ -254,9 +254,9 @@ static int will_fetch(struct ref **head, const unsigned char *sha1)return 0 ; } -static void find_non_local_tags (struct transport *transport, -struct ref **head, -struct ref ***tail) +static void find_non_local_tags (const struct ref *refs, +struct ref **head, +struct ref ***tail) {struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct string_list remote_refs =STRING_LIST_INIT_NODUP ;
@@ -264,7 +264,7 @@ static void find_non_local_tags(struct transport *transport,struct string_list_item *item =NULL ;for_each_ref (add_existing, &existing_refs); -for (ref =transport_get_remote_refs (transport,NULL ); ref; ref = ref->next) { +for (ref = refs; ref; ref = ref->next) {if (!starts_with (ref->name,"refs/tags/" ))continue ;
@@ -338,7 +338,8 @@ static void find_non_local_tags(struct transport *transport,string_list_clear (&remote_refs,0 ); } -static struct ref *get_ref_map (struct transport *transport, +static struct ref *get_ref_map (struct remote *remote, +const struct ref *remote_refs,struct refspec *rs,int tags,int *autotags) {
@@ -346,27 +347,11 @@ static struct ref *get_ref_map(struct transport *transport,struct ref *rm;struct ref *ref_map =NULL ;struct ref **tail = &ref_map; -struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs;struct string_list existing_refs =STRING_LIST_INIT_DUP ; -const struct ref *remote_refs; - -if (rs->nr) -refspec_ref_prefixes (rs, &ref_prefixes); -else if (transport->remote && transport->remote->fetch.nr) -refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); - -if (ref_prefixes.argc && - (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { -argv_array_push (&ref_prefixes,"refs/tags/" ); - } - - remote_refs =transport_get_remote_refs (transport, &ref_prefixes); - -argv_array_clear (&ref_prefixes);if (rs->nr) {struct refspec *fetch_refspec;
@@ -403,7 +388,7 @@ static struct ref *get_ref_map(struct transport *transport,if (refmap.nr) fetch_refspec = &refmap;else - fetch_refspec = &transport->remote->fetch; + fetch_refspec = &remote->fetch;for (i =0 ; i < fetch_refspec->nr; i++)get_fetch_map (ref_map, &fetch_refspec->items[i], &oref_tail,1 );
@@ -411,7 +396,6 @@ static struct ref *get_ref_map(struct transport *transport,die ("--refmap option is only meaningful with command-line refspec(s)." ); }else {/* Use the defaults */ -struct remote *remote = transport->remote;struct branch *branch =branch_get (NULL );int has_merge =branch_has_merge_config (branch); if (remote &&
@@ -450,7 +434,7 @@ static struct ref *get_ref_map(struct transport *transport,/* also fetch all tags */ get_fetch_map (remote_refs, tag_refspec, &tail,0 ); else if (tags == TAGS_DEFAULT && *autotags) - find_non_local_tags(transport, &ref_map, &tail); + find_non_local_tags(remote_refs, &ref_map, &tail);/* Now append any refs to be updated opportunistically: */ *tail = orefs;
@@ -1137,6 +1121,8 @@ static int do_fetch(struct transport *transport,struct ref *ref_map;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; +const struct ref *remote_refs; +struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 )
@@ -1152,7 +1138,21 @@ static int do_fetch(struct transport *transport, goto cleanup; } - ref_map =get_ref_map (transport, rs, tags, &autotags); +if (rs->nr) +refspec_ref_prefixes (rs, &ref_prefixes); +else if (transport->remote && transport->remote->fetch.nr) +refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); + +if (ref_prefixes.argc && + (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { +argv_array_push (&ref_prefixes,"refs/tags/" ); + } + + remote_refs =transport_get_remote_refs (transport, &ref_prefixes); +argv_array_clear (&ref_prefixes); + + ref_map =get_ref_map (transport->remote, remote_refs, rs, + tags, &autotags);if (!update_head_ok)check_not_current_branch (ref_map);
@@ -1184,7 +1184,7 @@ static int do_fetch(struct transport *transport,if (tags ==TAGS_DEFAULT && autotags) {struct ref **tail = &ref_map; ref_map =NULL ; -find_non_local_tags (transport, &ref_map, &tail); +find_non_local_tags (remote_refs, &ref_map, &tail);if (ref_map)backfill_tags (transport, ref_map);free_refs (ref_map);
Signature · 1 line
2.17.1.1185.g55be947832-goog
PATCH 7/8 fetch-pack: put shallow info in output parameter
In reply to Brandon WilliamsExpand the transport fetch method signature, by adding an output parameter, to allow transports to return information about the refs they have fetched. Then communicate shallow status information through this mechanism instead of by modifying the input list of refs.
This does require clients to sometimes generate the ref map twice: once from the list of refs provided by the remote (as is currently done) and potentially once from the new list of refs that the fetch mechanism provides.
--- builtin/clone.c | 4 ++-- builtin/fetch.c | 23 +++++++++++++++++++---- fetch-object.c | 2 +- fetch-pack.c | 17 +++++++++-------- transport-helper.c | 6 ++++-- transport-internal.h | 9 ++++++++- transport.c | 34 ++++++++++++++++++++++++++++------ transport.h | 3 ++- 8 files changed, 73 insertions(+), 25 deletions(-)
builtin/clone.c +2 -2
@@ -1155,7 +1155,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (!is_local && !complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); remote_head =find_ref_by_name (refs,"HEAD" ); remote_head_points_at =
@@ -1197,7 +1197,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)if (is_local)clone_local (path, git_dir);else if (refs && complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); update_remote_refs(refs, mapped_refs, remote_head_points_at, branch_top.buf, reflog_msg.buf, transport,
builtin/fetch.c +19 -4
@@ -946,11 +946,13 @@ static int quickfetch(struct ref *ref_map)return check_connected (iterate_ref_map, &rm, &opt); } -static int fetch_refs (struct transport *transport,struct ref *ref_map) +static int fetch_refs (struct transport *transport,struct ref *ref_map, +struct ref **updated_remote_refs) {int ret =quickfetch (ref_map);if (ret) - ret =transport_fetch_refs (transport, ref_map); + ret =transport_fetch_refs (transport, ref_map, + updated_remote_refs);if (ret)transport_unlock_pack (transport);return ret;
@@ -1106,7 +1108,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -if (!fetch_refs (transport, ref_map)) +if (!fetch_refs (transport, ref_map,NULL ))consume_refs (transport, ref_map);if (gsecondary) {
@@ -1122,6 +1124,7 @@ static int do_fetch(struct transport *transport,int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ;const struct ref *remote_refs; +struct ref *new_remote_refs =NULL ;struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {
@@ -1172,7 +1175,19 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) { + +if (fetch_refs (transport, ref_map, &new_remote_refs)) { +free_refs (ref_map); + retcode =1 ; + goto cleanup; + } +if (new_remote_refs) { +free_refs (ref_map); + ref_map =get_ref_map (transport->remote, new_remote_refs, rs, + tags, &autotags); +free_refs (new_remote_refs); + } +if (consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
fetch-object.c +1 -1
@@ -19,7 +19,7 @@ static void fetch_refs(const char *remote_name, struct ref *ref)transport_set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" );transport_set_option (transport,TRANS_OPT_NO_DEPENDENTS ,"1" ); -transport_fetch_refs (transport, ref); +transport_fetch_refs (transport, ref,NULL ); fetch_if_missing = original_fetch_if_missing; }
fetch-pack.c +9 -8
@@ -1470,12 +1470,13 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr) }static void update_shallow (struct fetch_pack_args *args, -struct ref **sought,int nr_sought, +struct ref *refs,struct shallow_info *si) {struct oid_array ref =OID_ARRAY_INIT ;int *status; -int i; +int i =0 ; +struct ref *r;if (args->deepen && alternate_shallow_file) {if (*alternate_shallow_file =='\0' ) {/* --unshallow */
@@ -1517,8 +1518,8 @@ static void update_shallow(struct fetch_pack_args *args,remove_nonexistent_theirs_shallow (si);if (!si->nr_ours && !si->nr_theirs)return ; -for (i =0 ; i < nr_sought; i++) -oid_array_append (&ref, &sought[i]->old_oid); +for (r = refs; r; r = r->next) +oid_array_append (&ref, &r->old_oid); si->ref = &ref;if (args->update_shallow) {
@@ -1552,12 +1553,12 @@ static void update_shallow(struct fetch_pack_args *args,* remote is also shallow, check what ref is safe to update * without updating .git/shallow */ - status =xcalloc (nr_sought,sizeof (*status)); + status =xcalloc (ref.nr,sizeof (*status));assign_shallow_commits_to_refs (si,NULL , status);if (si->nr_ours || si->nr_theirs) { -for (i =0 ; i < nr_sought; i++) +for (r = refs; r; r = r->next, i++)if (status[i]) - sought[i]->status =REF_STATUS_REJECT_SHALLOW ; + r->status =REF_STATUS_REJECT_SHALLOW ; }free (status);oid_array_clear (&ref);
@@ -1591,7 +1592,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args, ref_cpy =do_fetch_pack (args, fd, ref, sought, nr_sought, &si, pack_lockfile);reprepare_packed_git (the_repository); -update_shallow (args, sought, nr_sought, &si); +update_shallow (args, ref_cpy, &si);clear_shallow_info (&si);return ref_cpy; }
transport-helper.c +4 -2
@@ -651,14 +651,16 @@ static int connect_helper(struct transport *transport, const char *name, }static int fetch (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct helper_data *data = transport->data;int i, count;if (process_connect (transport,0 )) {do_take_over (transport); -return transport->vtable->fetch (transport, nr_heads, to_fetch); +return transport->vtable->fetch (transport, nr_heads, to_fetch, + fetched_refs); } count =0 ;
transport-internal.h +8 -1
@@ -36,11 +36,18 @@ struct transport_vtable {* Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. * +* The transport *may* provide, in fetched_refs, the list of refs that +* it fetched. If the transport knows anything about the fetched refs +* that the caller does not know (for example, shallow status), it +* should provide that list of refs and include that information in the +* list. +* * If the transport did not get hashes for refs in * get_refs_list(), it should set the old_sha1 fields in the * provided refs now. **/ -int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs); +int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs, +struct ref **fetched_refs);/** * Push the objects and refs. Send the necessary objects, and
transport.c +28 -6
@@ -151,7 +151,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport, }static int fetch_refs_from_bundle (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct bundle_transport_data *data = transport->data; return unbundle(&data->header, data->fd,
@@ -287,7 +288,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus }static int fetch_refs_via_pack (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -354,8 +356,12 @@ static int fetch_refs_via_pack(struct transport *transport,if (report_unmatched_refs (to_fetch, nr_heads)) ret =-1 ; +if (fetched_refs) + *fetched_refs = refs; +else +free_refs (refs); +free_refs (refs_tmp); -free_refs (refs);free (dest);return ret; }
@@ -1215,19 +1221,31 @@ const struct ref *transport_get_remote_refs(struct transport *transport,return transport->remote_refs; } -int transport_fetch_refs (struct transport *transport,struct ref *refs) +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs) {int rc;int nr_heads =0 , nr_alloc =0 , nr_refs =0 ;struct ref **heads =NULL ; +struct ref *nop_head =NULL , **nop_tail = &nop_head;struct ref *rm;for (rm = refs; rm; rm = rm->next) { nr_refs++;if (rm->peer_ref && !is_null_oid (&rm->old_oid) && - !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) + !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don not +* actually need to fetch them. +*/ +if (fetched_refs) { +struct ref *nop_ref =copy_ref (rm); + *nop_tail = nop_ref; + nop_tail = &nop_ref->next; + }continue ; + }ALLOC_GROW (heads, nr_heads +1 , nr_alloc); heads[nr_heads++] = rm; }
@@ -1245,7 +1263,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) heads[nr_heads++] = rm; } - rc = transport->vtable->fetch (transport, nr_heads, heads); + rc = transport->vtable->fetch (transport, nr_heads, heads, fetched_refs); +if (fetched_refs && nop_head) { + *nop_tail = *fetched_refs; + *fetched_refs = nop_head; + }free (heads);return rc;
transport.h +2 -1
@@ -218,7 +218,8 @@ int transport_push(struct transport *connection,const struct ref *transport_get_remote_refs (struct transport *transport,const struct argv_array *ref_prefixes); -int transport_fetch_refs (struct transport *transport,struct ref *refs); +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs);void transport_unlock_pack (struct transport *transport);int transport_disconnect (struct transport *transport);char *transport_anonymize_url (const char *url);
Signature · 1 line
2.17.1.1185.g55be947832-goog
PATCH 8/8 fetch-pack: implement ref-in-want
In reply to Brandon WilliamsImplement ref-in-want on the client side so that when a server supports the "ref-in-want" feature, a client will send "want-ref" lines for each reference the client wants to fetch.
--- fetch-pack.c | 35 +++++++++++++++++++++++++++--- remote.c | 1 + remote.h | 1 + t/t5703-upload-pack-ref-in-want.sh | 4 ++-- 4 files changed, 36 insertions(+), 5 deletions(-)
fetch-pack.c +32 -3
@@ -1102,9 +1102,10 @@ static void add_shallow_requests(struct strbuf *req_buf,static void add_wants (const struct ref *wants,struct strbuf *req_buf) { +int use_ref_in_want =server_supports_feature ("fetch" ,"ref-in-want" ,0 ); +for ( ; wants ; wants = wants->next) {const struct object_id *remote = &wants->old_oid; -const char *remote_hex;struct object *o;/*
@@ -1122,8 +1123,10 @@ static void add_wants(const struct ref *wants, struct strbuf *req_buf)continue ; } - remote_hex =oid_to_hex (remote); -packet_buf_write (req_buf,"want %s\n" , remote_hex); +if (!use_ref_in_want || wants->exact_sha1) +packet_buf_write (req_buf,"want %s\n" ,oid_to_hex (remote)); +else +packet_buf_write (req_buf,"want-ref %s\n" , wants->name); } }
@@ -1334,6 +1337,29 @@ static void receive_shallow_info(struct fetch_pack_args *args, args->deepen =1 ; } +static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line); + +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + } + } + +if (reader->status !=PACKET_READ_DELIM ) +die ("error processing wanted refs: %d" , reader->status); +} +enum fetch_state {FETCH_CHECK_LOCAL =0 ,FETCH_SEND_REQUEST ,
@@ -1408,6 +1434,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,if (process_section_header (&reader,"shallow-info" ,1 ))receive_shallow_info (args, &reader); +if (process_section_header (&reader,"wanted-refs" ,1 )) +receive_wanted_refs (&reader, ref); +/* get the pack */ process_section_header (&reader,"packfile" ,0 );if (get_pack (args, fd, pack_lockfile))
remote.c +1 -0
@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_sha1 =1 ; }else { ref_map =get_remote_ref (remote_refs, name); }
remote.h +1 -0
@@ -73,6 +73,7 @@ struct ref { force:1, forced_update:1, expect_old_sha1:1, + exact_sha1:1, deletion:1;enum {
t/t5703-upload-pack-ref-in-want.sh +2 -2
@@ -204,7 +204,7 @@ test_expect_success 'server is initially ahead - no ref in want' ' grep "ERR upload-pack: not our ref" err ' -test_expect_failure 'server is initially ahead - ref in want' ' +test_expect_success 'server is initially ahead - ref in want' ' git -C "$REPO" config uploadpack.allowRefInWant true && rm -rf local && cp -r "$LOCAL_PRISTINE" local &&
@@ -228,7 +228,7 @@ test_expect_success 'server is initially behind - no ref in want' ' test_cmp expected actual ' -test_expect_failure 'server is initially behind - ref in want' ' +test_expect_success 'server is initially behind - ref in want' ' git -C "$REPO" config uploadpack.allowRefInWant true && rm -rf local && cp -r "$LOCAL_PRISTINE" local &&
Signature · 1 line
2.17.1.1185.g55be947832-goog
v2 cover ref-in-want
In reply to Brandon WilliamsChanges in v2:
* issuing a want-ref line to a ref which doesn't exist is just ignored.
* fixed some typos
Brandon Williams (8):
test-pkt-line: add unpack-sideband subcommand
upload-pack: implement ref-in-want
upload-pack: test negotiation with changing repository
fetch: refactor the population of peer ref OIDs
fetch: refactor fetch_refs into two functions
fetch: refactor to make function args narrower
fetch-pack: put shallow info in output parameter
fetch-pack: implement ref-in-want
Documentation/config.txt | 7 + Documentation/technical/protocol-v2.txt | 29 ++- builtin/clone.c | 4 +- builtin/fetch.c | 126 +++++++----- fetch-object.c | 2 +- fetch-pack.c | 52 +++-- remote.c | 1 + remote.h | 1 + t/helper/test-pkt-line.c | 37 ++++ t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 + t/lib-httpd/one-time-sed.sh | 16 ++ t/t5703-upload-pack-ref-in-want.sh | 245 ++++++++++++++++++++++++ transport-helper.c | 6 +- transport-internal.h | 9 +- transport.c | 34 +++- transport.h | 3 +- upload-pack.c | 64 +++++++ 18 files changed, 568 insertions(+), 77 deletions(-) create mode 100644 t/lib-httpd/one-time-sed.sh create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
PATCH v2 1/8 test-pkt-line: add unpack-sideband subcommand
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
--- t/helper/test-pkt-line.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
t/helper/test-pkt-line.c +37 -0
@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h" static void pack_line (const char *line)
@@ -48,6 +49,40 @@ static void unpack(void) } } +static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE ); + +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) { +case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ; +if (band ==1 ) + fd =1 ; +else + fd =2 ; + +write_or_die (fd, reader.line+1 , reader.pktlen-1 ); + +if (band ==3 ) +die ("sind-band error" ); +break ; +case PACKET_READ_FLUSH : +return ; +case PACKET_READ_DELIM : +break ; + } + } +} +int cmd_main (int argc,const char **argv) {if (argc <2 )
@@ -57,6 +92,8 @@ int cmd_main(int argc, const char **argv)pack (argc -2 , argv +2 ); else if (!strcmp(argv[1], "unpack")) unpack(); + else if (!strcmp(argv[1], "unpack-sideband")) + unpack_sideband(); else die("invalid argument '%s'", argv[1]);
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
Re: [PATCH v2 1/8] test-pkt-line: add unpack-sideband subcommand
Quoted 34 lines · Brandon Williams
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
Signed-off-by Brandon Williams <[email protected]>--- t/helper/test-pkt-line.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)t/helper/test-pkt-line.c +13
-0@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h" static void pack_line (const char *line)@@ -48,6 +49,40 @@ static void unpack(void) } } +static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE ); + +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) {
t/helper/test-pkt-line.c static void unpack(void) · line 73
Quoted 10 lines
t/helper/test-pkt-line.c +10
-0+case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ; +if (band ==1 ) + fd =1 ; +else + fd =2 ; + +write_or_die (fd, reader.line+1 , reader.pktlen-1 );
white space around + and - ?
t/helper/test-pkt-line.c static void unpack(void) · line 76
Quoted 3 lines
t/helper/test-pkt-line.c +3
-0+ +if (band ==3 ) +die ("sind-band error" );
s/sind/side/ ?
What values for band are possible?
e.g. band==4 would also just write to fd=1;
but I suspect we don't want that, yet.
So maybe
band = reader.line[0] & 0xff;
if (band < 1 || band > 2)
die("unexpected side band %d", band)
fd = band;instead?
Re: [PATCH v2 1/8] test-pkt-line: add unpack-sideband subcommand
Quoted 39 lines · Stefan Beller
Quoted 38 lines · Brandon Williams
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
Signed-off-by Brandon Williams <[email protected]>--- t/helper/test-pkt-line.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)t/helper/test-pkt-line.c +17
-0@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h" static void pack_line (const char *line)@@ -48,6 +49,40 @@ static void unpack(void) } } +static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE ); + +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) { +case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ;
Quoted 8 lines
t/helper/test-pkt-line.c static void unpack(void) · line 73
Quoted 6 lines
t/helper/test-pkt-line.c +6
-0+if (band ==1 ) + fd =1 ; +else + fd =2 ; + +write_or_die (fd, reader.line+1 , reader.pktlen-1 );white space around + and - ?
Will fix.
Quoted 6 lines
t/helper/test-pkt-line.c static void unpack(void) · line 76
Quoted 4 lines
t/helper/test-pkt-line.c +3
-0+ +if (band ==3 ) +die ("sind-band error" );s/sind/side/ ?
Thanks for catching this.
Quoted 5 lines
What values for band are possible?
e.g. band==4 would also just write to fd=1;
but I suspect we don't want that, yet.
Quoted 8 lines
So maybe
band = reader.line[0] & 0xff; if (band < 1 || band > 2) die("unexpected side band %d", band) fd = band;instead?
Yeah that's must cleaner logic.
Signature · 1 line
Brandon Williams
PATCH v2 2/8 upload-pack: implement ref-in-want
In reply to Brandon WilliamsCurrently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement.
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
--- Documentation/config.txt | 7 ++ Documentation/technical/protocol-v2.txt | 29 ++++- t/t5703-upload-pack-ref-in-want.sh | 153 ++++++++++++++++++++++++ upload-pack.c | 64 ++++++++++ 4 files changed, 252 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Documentation/config.txt +7 -0
@@ -3479,6 +3479,13 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories). +uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. This feature + is intended for the benefit of load-balanced servers which may + not have the same view of what OIDs their refs point to due to + replication delay. + url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves a
Documentation/technical/protocol-v2.txt +28 -1
@@ -299,12 +299,22 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values. +If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. A server should ignore any "want-ref <ref>" lines where + <ref> doesn't exist on the server. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. output = *section - section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt) acknowledgments = PKT-LINE("acknowledgments" LF)
@@ -319,6 +329,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff)
@@ -379,6 +393,19 @@ header. * This section is only included if a packfile section is also included in the response. + wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs" + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server MUST NOT send any refs which were not requested + using 'want-ref' lines. + packfile section * This section is only included if the client has sent 'want' lines in its request and either requested that no more
t/t5703-upload-pack-ref-in-want.sh new file +153 -0
153 lines not on this page · raw message
upload-pack.c +64 -0
136 lines not on this page · raw message
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
Re: [PATCH v2 2/8] upload-pack: implement ref-in-want
Hi Brandon,
Quoted 4 lines · Brandon Williams
Currently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during
I stopped reading when stumbling upon 'vulnerability to failure' a I found it hard to read. A quick search turns out this is insider slang of civil engineers. :)
Quoted 2 lines
negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement.
... and the repository is not replicated evenly to all servers, yet.
Quoted 6 lines
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
This paragraph makes it sound as if it can be combined technically, i.e.
client:
want 01234...
want-ref master.. usual back and forth + pack..
server:
wanted-ref: master 2345..
What happens if the client "wants" a sha1 that is advertised, but happens to be the same as a wanted-ref?
Quoted 14 lines
Signed-off-by Brandon Williams <[email protected]>--- Documentation/config.txt | 7 ++ Documentation/technical/protocol-v2.txt | 29 ++++- t/t5703-upload-pack-ref-in-want.sh | 153 ++++++++++++++++++++++++ upload-pack.c | 64 ++++++++++ 4 files changed, 252 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.shDocumentation/config.txt +0
-0
Documentation/config.txt Note that this configuration variable is ignored if it is seen in the · line 3487
Quoted 9 lines
Documentation/config.txt +6
-0@@ -3479,6 +3479,13 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories). +uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. This feature + is intended for the benefit of load-balanced servers which may + not have the same view of what OIDs their refs point to due to + replication delay.
Instead of saying who benefits, can we also say what the feature is about? Didn't someone mention on the first round of this series, that technically ref-in-want also provides smaller net work load as refs usually are shorter than oids (specifically as oids will grow in the hash transisition plan later)? Is that worth mentioning?
When using this feature is a ref advertisement still needed?
Quoted 11 lines
Documentation/config.txt +1
-0+ url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves aDocumentation/technical/protocol-v2.txt +0
-0@@ -299,12 +299,22 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values.
Documentation/technical/protocol-v2.txt included in the client's request: · line 310
Quoted 10 lines
Documentation/technical/protocol-v2.txt +9
-0+If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. A server should ignore any "want-ref <ref>" lines where + <ref> doesn't exist on the server.
Are patterns allowed?, e.g. I might want refs/tags/* at all times.
Quoted 12 lines
Documentation/technical/protocol-v2.txt +4
-0@@ -319,6 +329,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff)
Documentation/technical/protocol-v2.txt header. · line 399
Quoted 7 lines
Documentation/technical/protocol-v2.txt +4
-0@@ -379,6 +393,19 @@ header. * This section is only included if a packfile section is also included in the response. + wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response.
Is it possible to fetch non-fast-forwarded refs this way? Or specifcially refs that were reset to an older point in history such that no pack file is needed to transfer; would we transfer an empty pack and then the wanted-refs section for that use case?
Documentation/technical/protocol-v2.txt header. · line 406
Quoted 7 lines
Documentation/technical/protocol-v2.txt +7
-0+ +# c(o/foo) d(o/bar) +# \ / +# b e(baz) f(master) +# \__ | __/ +# \ | / +# a
time is up in this diagram, most diagrams I looked at in tests are sideways. Should be fine either way.
Documentation/technical/protocol-v2.txt header.
Quoted 12 lines
Documentation/technical/protocol-v2.txt +12
-0+test_expect_success 'invalid want-ref line' ' + test-pkt-line pack >in <<-EOF && + command=fetch + 0001 + no-progress + want-ref refs/heads/non-existent + done + 0000 + EOF + + test_must_fail git serve --stateless-rpc 2>out <in && + grep "unknown ref" out
The docs disagree with the test?
A server should ignore any "want-ref <ref>" lines where
<ref> doesn't exist on the server.
Documentation/technical/protocol-v2.txt
Quoted 2 lines
Documentation/technical/protocol-v2.txt +2
-0+ +test_expect_success 'mix want and want-ref' '
cool!
Re: [PATCH v2 2/8] upload-pack: implement ref-in-want
Quoted 6 lines · Stefan Beller
Hi Brandon,
Quoted 2 lines · Brandon Williams
negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement.
... and the repository is not replicated evenly to all servers, yet.
I'll update the commit msg to also include this.
Quoted 15 lines
Quoted 7 lines
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
This paragraph makes it sound as if it can be combined technically, i.e.
client:
want 01234... want-ref master
Quoted 8 lines
.. usual back and forth + pack..
server:
wanted-ref: master 2345..
What happens if the client "wants" a sha1 that is advertised, but happens to be the same as a wanted-ref?
This would be fine, same as sending a want line with the same sha1 lots of times. Though there would still be a wanted-ref section from the server for the wanted-ref.
Quoted 18 lines
Quoted 18 lines
Signed-off-by Brandon Williams <[email protected]>--- Documentation/config.txt | 7 ++ Documentation/technical/protocol-v2.txt | 29 ++++- t/t5703-upload-pack-ref-in-want.sh | 153 ++++++++++++++++++++++++ upload-pack.c | 64 ++++++++++ 4 files changed, 252 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.shDocumentation/config.txt +0
-0@@ -3479,6 +3479,13 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories).
Quoted 12 lines
Documentation/config.txt Note that this configuration variable is ignored if it is seen in the · line 3487
Quoted 6 lines
Documentation/config.txt +6
-0+uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. This feature + is intended for the benefit of load-balanced servers which may + not have the same view of what OIDs their refs point to due to + replication delay.Instead of saying who benefits, can we also say what the feature is about? Didn't someone mention on the first round of this series, that technically ref-in-want also provides smaller net work load as refs usually are shorter than oids (specifically as oids will grow in the hash transisition plan later)? Is that worth mentioning?
Well I basically just took this from what a previous reviewer thought it should say. I think what you have listed here isn't really a big benefit of using ref-in-want, its the issue with load-balanced servers that this is trying to solve.
Quoted 2 lines
When using this feature is a ref advertisement still needed?
Maybe in the future no, but as of right now the code is structured to still request a ref advertisement.
Quoted 13 lines
Quoted 13 lines
Documentation/config.txt +1
-0+ url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves aDocumentation/technical/protocol-v2.txt +0
-0@@ -299,12 +299,22 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values.
Quoted 11 lines
Documentation/technical/protocol-v2.txt included in the client's request: · line 310
Quoted 9 lines
Documentation/technical/protocol-v2.txt +9
-0+If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. A server should ignore any "want-ref <ref>" lines where + <ref> doesn't exist on the server.Are patterns allowed?, e.g. I might want refs/tags/* at all times.
Nope, "Where <ref> is the full name of a ref". We can maybe allow this at a later point in time.
Quoted 16 lines
Quoted 16 lines
Documentation/technical/protocol-v2.txt +4
-0@@ -319,6 +329,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff)@@ -379,6 +393,19 @@ header. * This section is only included if a packfile section is also included in the response.
Quoted 9 lines
Documentation/technical/protocol-v2.txt header. · line 399
Quoted 4 lines
Documentation/technical/protocol-v2.txt +4
-0+ wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response.Is it possible to fetch non-fast-forwarded refs this way? Or specifcially refs that were reset to an older point in history such that no pack file is needed to transfer; would we transfer an empty pack and then the wanted-refs section for that use case?
Yeah there are cases where an empty packfile would be sent like you've described.
Quoted 13 lines
Documentation/technical/protocol-v2.txt header. · line 406
Quoted 9 lines
Documentation/technical/protocol-v2.txt +7
-0+ +# c(o/foo) d(o/bar) +# \ / +# b e(baz) f(master) +# \__ | __/ +# \ | / +# atime is up in this diagram, most diagrams I looked at in tests are sideways. Should be fine either way.
Quoted 16 lines
Documentation/technical/protocol-v2.txt header.
Quoted 12 lines
Documentation/technical/protocol-v2.txt +12
-0+test_expect_success 'invalid want-ref line' ' + test-pkt-line pack >in <<-EOF && + command=fetch + 0001 + no-progress + want-ref refs/heads/non-existent + done + 0000 + EOF + + test_must_fail git serve --stateless-rpc 2>out <in && + grep "unknown ref" outThe docs disagree with the test?
A server should ignore any "want-ref <ref>" lines where
<ref> doesn't exist on the server.
I forgot to remove this when i updated the docs. I'll remove this test as it fails now :(
Quoted 6 lines
Documentation/technical/protocol-v2.txt
Quoted 4 lines
Documentation/technical/protocol-v2.txt +2
-0+ +test_expect_success 'mix want and want-ref' 'cool!
Signature · 1 line
Brandon Williams
Re: [PATCH v2 2/8] upload-pack: implement ref-in-want
In reply to Brandon WilliamsQuoted 5 lines · Brandon Williams
Currently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement.
In other words, your "git fetch refs/heads/master" initially contacts a mirror A and learns that its refs/heads/master is at commit X, starts negotiation, and end up with getting served by a mirror B that is slightly beind mirror A, which wants to give you a packfile that brings you up to say commit X~3. You want to be able to say "I want to update my refs/remotes/origin/master with whatever is (close to) the latest at your refs/heads/master", not "I just learned refs/heads/master is at X from one of you, I demand getting updated to that exact commit."
For that, you'd say, after getting ref advertisement, "I want the tip of refs/heads/master that one of you have, whoever ends up serving me eventually".
Documentation/technical/protocol-v2.txt included in the client's request: · line 310
Quoted 5 lines
Documentation/technical/protocol-v2.txt +5
-0+ want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. A server should ignore any "want-ref <ref>" lines where + <ref> doesn't exist on the server.
But when you said "I want some version of refs/heads/bo", is it sensible for me to say "Oh, among 7 mirrors, you unluckily ended up with me who is most behind and do not even have 'bo' branch yet, so I won't be giving you that branch (or history leading to the commit my 6 friends may have at refs/heads/bo)"?
What is the end-user visible effect of "ignoring"? Does your "git fetch bo" that were unluckily served by me who do not yet have the branch error out? If so, preferrably the conversation should faile before packfile generation and transfer.
I am guessing that you do not want to fail the negotiation if your "git fetch bo" happen to contact me (who lack 'bo') first, as long as the conversation continues and switches later to one of my friends who can fulfill the request, and that is why you are forbidding me (who got initial contact and saw "want-ref bo") from failing the whole thing. But it is unclear who is responsible for erroring out the whole "git fetch bo" *if* unlucky you ended up getting served by the most stale mirror that does not even have 'bo'.
The story would be different if your request were
git fetch refs/heads/*:refs/remotes/origin/*
in which case, you are not even saying "I want this and that ref"; you are saying "all refs in refs/heads/* whoever ends up serving me happens to have". You may initially contact one of my friends and learn that there are 'master' and 'bo' branches (and probably others), and after conversation end up talking with me who is stale and lack 'bo'. In such a case, I agree that it is not sensible for me to fail the request as a whole and instead serve you whatever branches I happen to have. I may lack 'bo' branch due to mirroring lag, but I may also have 'extra' branch that others no longer have due to mirroring lag of deletion of that branch!
But then I think your "git fetch refs/heads/*:refs/remotes/origin/*" should not fail not just because I do not have 'bo', but you also should grab other old branches I have, which you didn't hear about when you made the initial contact with my friend in the mirror pool.
So, given that, would it make sense for 'want-ref <ref>' request to name "a particular ref" as the above document says? I have a feeling that it should allow a pattern to be matched at the server side (and it is not an error if the pattern did not match anything), in addition to asking for a particular ref (in which case, lack of that ref should be a hard failure, at least for the mirror that ends up serving the packfile and the final "here are the refs your request ended up fetching, with their values").
Documentation/technical/protocol-v2.txt included in the client's request: · line 318
Quoted 8 lines
The response of
fetchis broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header.output = *section
Documentation/technical/protocol-v2.txt +1
-1- section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt)
OK. In my initial reading, I somehow failed to read this piece before going on to the next hunk ...
Documentation/technical/protocol-v2.txt header. · line 335
Quoted 8 lines
Documentation/technical/protocol-v2.txt +4
-0@@ -319,6 +329,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname +
... and wondered how the reader knows where wanted-ref list ends; we will see a flush (or is it delim? either is accepted?) where the list ends, which is good.
Re: [PATCH v2 2/8] upload-pack: implement ref-in-want
Quoted 11 lines · Junio C Hamano
The story would be different if your request were
git fetch refs/heads/*:refs/remotes/origin/*
in which case, you are not even saying "I want this and that ref"; you are saying "all refs in refs/heads/* whoever ends up serving me happens to have". You may initially contact one of my friends and learn that there are 'master' and 'bo' branches (and probably others), and after conversation end up talking with me who is stale and lack 'bo'. In such a case, I agree that it is not sensible for me to fail the request as a whole and instead serve you whatever
s/whole and instead/whole. I should instead/;
Quoted 3 lines
branches I happen to have. I may lack 'bo' branch due to mirroring lag, but I may also have 'extra' branch that others no longer have due to mirroring lag of deletion of that branch!
Re: [PATCH v2 2/8] upload-pack: implement ref-in-want
In reply to Junio C HamanoQuoted 20 lines · Junio C Hamano
The story would be different if your request were
git fetch refs/heads/*:refs/remotes/origin/*
in which case, you are not even saying "I want this and that ref"; you are saying "all refs in refs/heads/* whoever ends up serving me happens to have". You may initially contact one of my friends and learn that there are 'master' and 'bo' branches (and probably others), and after conversation end up talking with me who is stale and lack 'bo'. In such a case, I agree that it is not sensible for me to fail the request as a whole and instead serve you whatever branches I happen to have. I may lack 'bo' branch due to mirroring lag, but I may also have 'extra' branch that others no longer have due to mirroring lag of deletion of that branch!
But then I think your "git fetch refs/heads/*:refs/remotes/origin/*" should not fail not just because I do not have 'bo', but you also should grab other old branches I have, which you didn't hear about when you made the initial contact with my friend in the mirror pool.
Quoted 8 lines
So, given that, would it make sense for 'want-ref <ref>' request to name "a particular ref" as the above document says? I have a feeling that it should allow a pattern to be matched at the server side (and it is not an error if the pattern did not match anything), in addition to asking for a particular ref (in which case, lack of that ref should be a hard failure, at least for the mirror that ends up serving the packfile and the final "here are the refs your request ended up fetching, with their values").
After some more in-office discussion about this I think I should revert back to making it a hard failure when a client requests a ref which doesn't exist on the server. This makes things more consistent with what happens today if I request a non-existent ref (Although that error is purely on the client). Its no worse than we have today and even with this solution not solving the issues of new/deleted refs (which are rare operations wrt updates) we still can get the benefit of not failing due to a ref updating. This is also very valuable for the servers which have to to ACL checks on individual ref names.
I also think that we should keep this first implementation of ref-in-want simple and *not* include patterns, even if that's what we may want someday down the road. Adding a new capability in the future for support of such patterns would be relatively simple and easy. The reason why I don't think we should add pattern support just yet is due to a request for "want-ref refs/tags/*" or a like request resulting in a larger than expected packfile every time "fetch --tags" is run. The issue being that in a fetch request "refs/tags/*" is too broad of a request and could be requesting 100s of tags when all we really wanted was to get the one or two new tags which are present on the remote (because we already have all the other tags present locally).
So I think the best way is to limit these patterns to the ls-refs request where we can then discover the few tags we're missing and then request just those tags explicitly, keeping the resulting packfile smaller.
Thoughts?
Signature · 1 line
Brandon Williams
Re: [PATCH v2 2/8] upload-pack: implement ref-in-want
Quoted 4 lines · Brandon Williams
I also think that we should keep this first implementation of ref-in-want simple and *not* include patterns, even if that's what we may want someday down the road. Adding a new capability in the future for support of such patterns would be relatively simple and easy.
I am all for many-small-steps over a single-giant-step approach.
Quoted 8 lines
The
reason why I don't think we should add pattern support just yet is due to a request for "want-ref refs/tags/*" or a like request resulting in a larger than expected packfile every time "fetch --tags" is run. The issue being that in a fetch request "refs/tags/*" is too broad of a request and could be requesting 100s of tags when all we really wanted was to get the one or two new tags which are present on the remote (because we already have all the other tags present locally).
I do not quite get this. Why does it have to result in a large packfile? Doesn't the requester of refs/tags/* still show what it has via "have" exchange?
Re: [PATCH v2 2/8] upload-pack: implement ref-in-want
Quoted 18 lines · Junio C Hamano
Quoted 4 lines · Brandon Williams
I also think that we should keep this first implementation of ref-in-want simple and *not* include patterns, even if that's what we may want someday down the road. Adding a new capability in the future for support of such patterns would be relatively simple and easy.
I am all for many-small-steps over a single-giant-step approach.
Quoted 8 lines
The
reason why I don't think we should add pattern support just yet is due to a request for "want-ref refs/tags/*" or a like request resulting in a larger than expected packfile every time "fetch --tags" is run. The issue being that in a fetch request "refs/tags/*" is too broad of a request and could be requesting 100s of tags when all we really wanted was to get the one or two new tags which are present on the remote (because we already have all the other tags present locally).
Quoted 3 lines
I do not quite get this. Why does it have to result in a large packfile? Doesn't the requester of refs/tags/* still show what it has via "have" exchange?
Sorry Jonathan Tan said it much clearer here:
https://public-inbox.org/git/[email protected]/
Signature · 1 line
Brandon Williams
Re: [PATCH v2 2/8] upload-pack: implement ref-in-want
Quoted 19 lines · Brandon Williams
Quoted 18 lines · Junio C Hamano
Quoted 4 lines · Brandon Williams
I also think that we should keep this first implementation of ref-in-want simple and *not* include patterns, even if that's what we may want someday down the road. Adding a new capability in the future for support of such patterns would be relatively simple and easy.
I am all for many-small-steps over a single-giant-step approach.
Quoted 8 lines
The
reason why I don't think we should add pattern support just yet is due to a request for "want-ref refs/tags/*" or a like request resulting in a larger than expected packfile every time "fetch --tags" is run. The issue being that in a fetch request "refs/tags/*" is too broad of a request and could be requesting 100s of tags when all we really wanted was to get the one or two new tags which are present on the remote (because we already have all the other tags present locally).
Quoted 6 lines
Quoted 3 lines
I do not quite get this. Why does it have to result in a large packfile? Doesn't the requester of refs/tags/* still show what it has via "have" exchange?
Sorry Jonathan Tan said it much clearer here:
https://public-inbox.org/git/[email protected]/
Hmph, it feels like that the real culprit in that illustration is that the negotiation stops after seeing A, and assumes that the chain leading to B and C need to be re-sent, without the parties in the negotiation even discussing that lines of the history. When you say "want-ref refs/tags/*", intending to update remote-tracking refs on your side with the result, shouldn't the protocol make sure that the current state of the remote-tracking refs (i.e. B and C in the example) is taken into account? It does not quite sound like a problem inherent to "want-ref refs/tags/*" or a like request; the real cause of the problem seems to lie elsewhere, at least to me.
Thanks.
PATCH v2 3/8 upload-pack: test negotiation with changing repository
In reply to Brandon WilliamsAdd tests to check the behavior of fetching from a repository which changes between rounds of negotiation (for example, when different servers in a load-balancing agreement participate in the same stateless RPC negotiation). This forms a baseline of comparison to the ref-in-want functionality (which will be introduced to the client in subsequent commits), and ensures that subsequent commits do not change existing behavior.
As part of this effort, a mechanism to substitute strings in a single HTTP response is added.
--- t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 +++ t/lib-httpd/one-time-sed.sh | 16 ++++++ t/t5703-upload-pack-ref-in-want.sh | 92 ++++++++++++++++++++++++++++++ 4 files changed, 117 insertions(+) create mode 100644 t/lib-httpd/one-time-sed.sh
t/lib-httpd.sh +1 -0
@@ -132,6 +132,7 @@ prepare_httpd() { cp "$TEST_PATH"/passwd "$HTTPD_ROOT_PATH" install_script broken-smart-http.sh install_script error.sh + install_script one-time-sed.sh ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +8 -0
@@ -111,9 +111,14 @@ Alias /auth/dumb/ www/auth/dumb/ SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} SetEnv GIT_HTTP_EXPORT_ALL </LocationMatch> +<LocationMatch /one_time_sed/> + SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} + SetEnv GIT_HTTP_EXPORT_ALL +</LocationMatch> ScriptAliasMatch /smart_*[^/]*/(.*) ${GIT_EXEC_PATH}/git-http-backend/$1 ScriptAlias /broken_smart/ broken-smart-http.sh/ ScriptAlias /error/ error.sh/ +ScriptAliasMatch /one_time_sed/(.*) one-time-sed.sh/$1 <Directory ${GIT_EXEC_PATH}> Options FollowSymlinks </Directory>
@@ -123,6 +128,9 @@ ScriptAlias /error/ error.sh/ <Files error.sh> Options ExecCGI </Files> +<Files one-time-sed.sh> + Options ExecCGI +</Files> <Files ${GIT_EXEC_PATH}/git-http-backend> Options ExecCGI </Files>
t/lib-httpd/one-time-sed.sh new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/sh + +if [ -e one-time-sed ]; then + "$GIT_EXEC_PATH/git-http-backend" >out + + sed "$(cat one-time-sed)" <out >out_modified + + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fi
t/t5703-upload-pack-ref-in-want.sh +92 -0
96 lines not on this page · raw message
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
Re: [PATCH v2 3/8] upload-pack: test negotiation with changing repository
Quoted 11 lines · Brandon Williams
Add tests to check the behavior of fetching from a repository which changes between rounds of negotiation (for example, when different servers in a load-balancing agreement participate in the same stateless RPC negotiation). This forms a baseline of comparison to the ref-in-want functionality (which will be introduced to the client in subsequent commits), and ensures that subsequent commits do not change existing behavior.
As part of this effort, a mechanism to substitute strings in a single HTTP response is added.
This patch looks good to me.
Thanks,
Stefan
PATCH v2 4/8 fetch: refactor the population of peer ref OIDs
In reply to Brandon WilliamsPopulate peer ref OIDs in get_ref_map instead of do_fetch. Besides tightening scopes of variables in the code, this also prepares for get_ref_map being able to be called multiple times within do_fetch.
--- builtin/fetch.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
builtin/fetch.c +18 -18
@@ -351,6 +351,7 @@ static struct ref *get_ref_map(struct transport *transport,/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs; +struct string_list existing_refs =STRING_LIST_INIT_DUP ;const struct ref *remote_refs;if (rs->nr)
@@ -458,7 +459,23 @@ static struct ref *get_ref_map(struct transport *transport, tail = &rm->next; } -return ref_remove_duplicates (ref_map); + ref_map =ref_remove_duplicates (ref_map); + +for_each_ref (add_existing, &existing_refs); +for (rm = ref_map; rm; rm = rm->next) { +if (rm->peer_ref) { +struct string_list_item *peer_item = +string_list_lookup (&existing_refs, + rm->peer_ref->name); +if (peer_item) { +struct object_id *old_oid = peer_item->util; +oidcpy (&rm->peer_ref->old_oid, old_oid); + } + } + } +string_list_clear (&existing_refs,1 ); + +return ref_map; }#define STORE_REF_ERROR_OTHER 1
@@ -1110,14 +1127,10 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)static int do_fetch (struct transport *transport,struct refspec *rs) { -struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct ref *ref_map; -struct ref *rm;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; -for_each_ref (add_existing, &existing_refs); -if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 ) tags =TAGS_SET ;
@@ -1136,18 +1149,6 @@ static int do_fetch(struct transport *transport,if (!update_head_ok)check_not_current_branch (ref_map); -for (rm = ref_map; rm; rm = rm->next) { -if (rm->peer_ref) { -struct string_list_item *peer_item = -string_list_lookup (&existing_refs, - rm->peer_ref->name); -if (peer_item) { -struct object_id *old_oid = peer_item->util; -oidcpy (&rm->peer_ref->old_oid, old_oid); - } - } - } -if (tags ==TAGS_DEFAULT && autotags)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,"1" );if (prune) {
@@ -1183,7 +1184,6 @@ static int do_fetch(struct transport *transport, } cleanup: -string_list_clear (&existing_refs,1 );return retcode; }
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
PATCH v2 5/8 fetch: refactor fetch_refs into two functions
In reply to Brandon WilliamsRefactor the fetch_refs function into a function that does the fetching of refs and another function that stores them.
--- builtin/fetch.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
builtin/fetch.c +13 -6
@@ -967,10 +967,16 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)int ret =quickfetch (ref_map);if (ret) ret =transport_fetch_refs (transport, ref_map); -if (!ret) - ret |=store_updated_refs (transport->url, - transport->remote->name, - ref_map); +if (ret) +transport_unlock_pack (transport); +return ret; +} + +static int consume_refs (struct transport *transport,struct ref *ref_map) +{ +int ret =store_updated_refs (transport->url, + transport->remote->name, + ref_map);transport_unlock_pack (transport);return ret; }
@@ -1116,7 +1122,8 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -fetch_refs (transport, ref_map); +if (!fetch_refs (transport, ref_map)) +consume_refs (transport, ref_map);if (gsecondary) {transport_disconnect (gsecondary);
@@ -1165,7 +1172,7 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map)) { +if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
PATCH v2 6/8 fetch: refactor to make function args narrower
In reply to Brandon WilliamsRefactor find_non_local_tags and get_ref_map to only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
--- builtin/fetch.c | 52 ++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
builtin/fetch.c +26 -26
@@ -254,9 +254,9 @@ static int will_fetch(struct ref **head, const unsigned char *sha1)return 0 ; } -static void find_non_local_tags (struct transport *transport, -struct ref **head, -struct ref ***tail) +static void find_non_local_tags (const struct ref *refs, +struct ref **head, +struct ref ***tail) {struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct string_list remote_refs =STRING_LIST_INIT_NODUP ;
@@ -264,7 +264,7 @@ static void find_non_local_tags(struct transport *transport,struct string_list_item *item =NULL ;for_each_ref (add_existing, &existing_refs); -for (ref =transport_get_remote_refs (transport,NULL ); ref; ref = ref->next) { +for (ref = refs; ref; ref = ref->next) {if (!starts_with (ref->name,"refs/tags/" ))continue ;
@@ -338,7 +338,8 @@ static void find_non_local_tags(struct transport *transport,string_list_clear (&remote_refs,0 ); } -static struct ref *get_ref_map (struct transport *transport, +static struct ref *get_ref_map (struct remote *remote, +const struct ref *remote_refs,struct refspec *rs,int tags,int *autotags) {
@@ -346,27 +347,11 @@ static struct ref *get_ref_map(struct transport *transport,struct ref *rm;struct ref *ref_map =NULL ;struct ref **tail = &ref_map; -struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs;struct string_list existing_refs =STRING_LIST_INIT_DUP ; -const struct ref *remote_refs; - -if (rs->nr) -refspec_ref_prefixes (rs, &ref_prefixes); -else if (transport->remote && transport->remote->fetch.nr) -refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); - -if (ref_prefixes.argc && - (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { -argv_array_push (&ref_prefixes,"refs/tags/" ); - } - - remote_refs =transport_get_remote_refs (transport, &ref_prefixes); - -argv_array_clear (&ref_prefixes);if (rs->nr) {struct refspec *fetch_refspec;
@@ -403,7 +388,7 @@ static struct ref *get_ref_map(struct transport *transport,if (refmap.nr) fetch_refspec = &refmap;else - fetch_refspec = &transport->remote->fetch; + fetch_refspec = &remote->fetch;for (i =0 ; i < fetch_refspec->nr; i++)get_fetch_map (ref_map, &fetch_refspec->items[i], &oref_tail,1 );
@@ -411,7 +396,6 @@ static struct ref *get_ref_map(struct transport *transport,die ("--refmap option is only meaningful with command-line refspec(s)." ); }else {/* Use the defaults */ -struct remote *remote = transport->remote;struct branch *branch =branch_get (NULL );int has_merge =branch_has_merge_config (branch); if (remote &&
@@ -450,7 +434,7 @@ static struct ref *get_ref_map(struct transport *transport,/* also fetch all tags */ get_fetch_map (remote_refs, tag_refspec, &tail,0 ); else if (tags == TAGS_DEFAULT && *autotags) - find_non_local_tags(transport, &ref_map, &tail); + find_non_local_tags(remote_refs, &ref_map, &tail);/* Now append any refs to be updated opportunistically: */ *tail = orefs;
@@ -1137,6 +1121,8 @@ static int do_fetch(struct transport *transport,struct ref *ref_map;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; +const struct ref *remote_refs; +struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 )
@@ -1152,7 +1138,21 @@ static int do_fetch(struct transport *transport, goto cleanup; } - ref_map =get_ref_map (transport, rs, tags, &autotags); +if (rs->nr) +refspec_ref_prefixes (rs, &ref_prefixes); +else if (transport->remote && transport->remote->fetch.nr) +refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); + +if (ref_prefixes.argc && + (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { +argv_array_push (&ref_prefixes,"refs/tags/" ); + } + + remote_refs =transport_get_remote_refs (transport, &ref_prefixes); +argv_array_clear (&ref_prefixes); + + ref_map =get_ref_map (transport->remote, remote_refs, rs, + tags, &autotags);if (!update_head_ok)check_not_current_branch (ref_map);
@@ -1184,7 +1184,7 @@ static int do_fetch(struct transport *transport,if (tags ==TAGS_DEFAULT && autotags) {struct ref **tail = &ref_map; ref_map =NULL ; -find_non_local_tags (transport, &ref_map, &tail); +find_non_local_tags (remote_refs, &ref_map, &tail);if (ref_map)backfill_tags (transport, ref_map);free_refs (ref_map);
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
Re: [PATCH v2 6/8] fetch: refactor to make function args narrower
Quoted 7 lines · Brandon Williams
Refactor
find_non_local_tagsandget_ref_mapto only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
This patch and the two prior refactoring patches are
PATCH v2 7/8 fetch-pack: put shallow info in output parameter
In reply to Brandon WilliamsExpand the transport fetch method signature, by adding an output parameter, to allow transports to return information about the refs they have fetched. Then communicate shallow status information through this mechanism instead of by modifying the input list of refs.
This does require clients to sometimes generate the ref map twice: once from the list of refs provided by the remote (as is currently done) and potentially once from the new list of refs that the fetch mechanism provides.
--- builtin/clone.c | 4 ++-- builtin/fetch.c | 23 +++++++++++++++++++---- fetch-object.c | 2 +- fetch-pack.c | 17 +++++++++-------- transport-helper.c | 6 ++++-- transport-internal.h | 9 ++++++++- transport.c | 34 ++++++++++++++++++++++++++++------ transport.h | 3 ++- 8 files changed, 73 insertions(+), 25 deletions(-)
builtin/clone.c +2 -2
@@ -1155,7 +1155,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (!is_local && !complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); remote_head =find_ref_by_name (refs,"HEAD" ); remote_head_points_at =
@@ -1197,7 +1197,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)if (is_local)clone_local (path, git_dir);else if (refs && complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); update_remote_refs(refs, mapped_refs, remote_head_points_at, branch_top.buf, reflog_msg.buf, transport,
builtin/fetch.c +19 -4
@@ -946,11 +946,13 @@ static int quickfetch(struct ref *ref_map)return check_connected (iterate_ref_map, &rm, &opt); } -static int fetch_refs (struct transport *transport,struct ref *ref_map) +static int fetch_refs (struct transport *transport,struct ref *ref_map, +struct ref **updated_remote_refs) {int ret =quickfetch (ref_map);if (ret) - ret =transport_fetch_refs (transport, ref_map); + ret =transport_fetch_refs (transport, ref_map, + updated_remote_refs);if (ret)transport_unlock_pack (transport);return ret;
@@ -1106,7 +1108,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -if (!fetch_refs (transport, ref_map)) +if (!fetch_refs (transport, ref_map,NULL ))consume_refs (transport, ref_map);if (gsecondary) {
@@ -1122,6 +1124,7 @@ static int do_fetch(struct transport *transport,int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ;const struct ref *remote_refs; +struct ref *new_remote_refs =NULL ;struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {
@@ -1172,7 +1175,19 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) { + +if (fetch_refs (transport, ref_map, &new_remote_refs)) { +free_refs (ref_map); + retcode =1 ; + goto cleanup; + } +if (new_remote_refs) { +free_refs (ref_map); + ref_map =get_ref_map (transport->remote, new_remote_refs, rs, + tags, &autotags); +free_refs (new_remote_refs); + } +if (consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
fetch-object.c +1 -1
@@ -19,7 +19,7 @@ static void fetch_refs(const char *remote_name, struct ref *ref)transport_set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" );transport_set_option (transport,TRANS_OPT_NO_DEPENDENTS ,"1" ); -transport_fetch_refs (transport, ref); +transport_fetch_refs (transport, ref,NULL ); fetch_if_missing = original_fetch_if_missing; }
fetch-pack.c +9 -8
@@ -1470,12 +1470,13 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr) }static void update_shallow (struct fetch_pack_args *args, -struct ref **sought,int nr_sought, +struct ref *refs,struct shallow_info *si) {struct oid_array ref =OID_ARRAY_INIT ;int *status; -int i; +int i =0 ; +struct ref *r;if (args->deepen && alternate_shallow_file) {if (*alternate_shallow_file =='\0' ) {/* --unshallow */
@@ -1517,8 +1518,8 @@ static void update_shallow(struct fetch_pack_args *args,remove_nonexistent_theirs_shallow (si);if (!si->nr_ours && !si->nr_theirs)return ; -for (i =0 ; i < nr_sought; i++) -oid_array_append (&ref, &sought[i]->old_oid); +for (r = refs; r; r = r->next) +oid_array_append (&ref, &r->old_oid); si->ref = &ref;if (args->update_shallow) {
@@ -1552,12 +1553,12 @@ static void update_shallow(struct fetch_pack_args *args,* remote is also shallow, check what ref is safe to update * without updating .git/shallow */ - status =xcalloc (nr_sought,sizeof (*status)); + status =xcalloc (ref.nr,sizeof (*status));assign_shallow_commits_to_refs (si,NULL , status);if (si->nr_ours || si->nr_theirs) { -for (i =0 ; i < nr_sought; i++) +for (r = refs; r; r = r->next, i++)if (status[i]) - sought[i]->status =REF_STATUS_REJECT_SHALLOW ; + r->status =REF_STATUS_REJECT_SHALLOW ; }free (status);oid_array_clear (&ref);
@@ -1591,7 +1592,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args, ref_cpy =do_fetch_pack (args, fd, ref, sought, nr_sought, &si, pack_lockfile);reprepare_packed_git (the_repository); -update_shallow (args, sought, nr_sought, &si); +update_shallow (args, ref_cpy, &si);clear_shallow_info (&si);return ref_cpy; }
transport-helper.c +4 -2
@@ -651,14 +651,16 @@ static int connect_helper(struct transport *transport, const char *name, }static int fetch (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct helper_data *data = transport->data;int i, count;if (process_connect (transport,0 )) {do_take_over (transport); -return transport->vtable->fetch (transport, nr_heads, to_fetch); +return transport->vtable->fetch (transport, nr_heads, to_fetch, + fetched_refs); } count =0 ;
transport-internal.h +8 -1
@@ -36,11 +36,18 @@ struct transport_vtable {* Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. * +* The transport *may* provide, in fetched_refs, the list of refs that +* it fetched. If the transport knows anything about the fetched refs +* that the caller does not know (for example, shallow status), it +* should provide that list of refs and include that information in the +* list. +* * If the transport did not get hashes for refs in * get_refs_list(), it should set the old_sha1 fields in the * provided refs now. **/ -int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs); +int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs, +struct ref **fetched_refs);/** * Push the objects and refs. Send the necessary objects, and
transport.c +28 -6
@@ -151,7 +151,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport, }static int fetch_refs_from_bundle (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct bundle_transport_data *data = transport->data; return unbundle(&data->header, data->fd,
@@ -287,7 +288,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus }static int fetch_refs_via_pack (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -354,8 +356,12 @@ static int fetch_refs_via_pack(struct transport *transport,if (report_unmatched_refs (to_fetch, nr_heads)) ret =-1 ; +if (fetched_refs) + *fetched_refs = refs; +else +free_refs (refs); +free_refs (refs_tmp); -free_refs (refs);free (dest);return ret; }
@@ -1215,19 +1221,31 @@ const struct ref *transport_get_remote_refs(struct transport *transport,return transport->remote_refs; } -int transport_fetch_refs (struct transport *transport,struct ref *refs) +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs) {int rc;int nr_heads =0 , nr_alloc =0 , nr_refs =0 ;struct ref **heads =NULL ; +struct ref *nop_head =NULL , **nop_tail = &nop_head;struct ref *rm;for (rm = refs; rm; rm = rm->next) { nr_refs++;if (rm->peer_ref && !is_null_oid (&rm->old_oid) && - !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) + !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don not +* actually need to fetch them. +*/ +if (fetched_refs) { +struct ref *nop_ref =copy_ref (rm); + *nop_tail = nop_ref; + nop_tail = &nop_ref->next; + }continue ; + }ALLOC_GROW (heads, nr_heads +1 , nr_alloc); heads[nr_heads++] = rm; }
@@ -1245,7 +1263,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) heads[nr_heads++] = rm; } - rc = transport->vtable->fetch (transport, nr_heads, heads); + rc = transport->vtable->fetch (transport, nr_heads, heads, fetched_refs); +if (fetched_refs && nop_head) { + *nop_tail = *fetched_refs; + *fetched_refs = nop_head; + }free (heads);return rc;
transport.h +2 -1
@@ -218,7 +218,8 @@ int transport_push(struct transport *connection,const struct ref *transport_get_remote_refs (struct transport *transport,const struct argv_array *ref_prefixes); -int transport_fetch_refs (struct transport *transport,struct ref *refs); +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs);void transport_unlock_pack (struct transport *transport);int transport_disconnect (struct transport *transport);char *transport_anonymize_url (const char *url);
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
Re: [PATCH v2 7/8] fetch-pack: put shallow info in output parameter
transport.c const struct ref *transport_get_remote_refs(struct transport *transport, · line 1239
Quoted 3 lines
transport.c +3
-0+ !oidcmp(&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don not
do not or don't; there is no middle way. ;)
Re: [PATCH v2 7/8] fetch-pack: put shallow info in output parameter
In reply to Brandon Williamsbuiltin/fetch.c static int do_fetch(struct transport *transport, · line 1127
Quoted 5 lines
builtin/fetch.c +1
-0@@ -1122,6 +1124,7 @@ static int do_fetch(struct transport *transport,int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ;const struct ref *remote_refs; +struct ref *new_remote_refs =NULL ;
Above, you use the name "updated_remote_refs" - it's probably better to standardize on one. I think "updated" is better.
(The transport calling it "fetched_refs" is fine, because that's what they are from the perspective of the transport. From the perspective of fetch-pack, it is indeed a new or updated set of remote refs.)
Quoted 7 lines
builtin/fetch.c +6
-1-if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) { + +if (fetch_refs (transport, ref_map, &new_remote_refs)) { +free_refs (ref_map); + retcode =1 ; + goto cleanup; + }
builtin/fetch.c static int do_fetch(struct transport *transport, · line 1193
Quoted 10 lines
builtin/fetch.c +7
-0+if (new_remote_refs) { +free_refs (ref_map); + ref_map =get_ref_map (transport->remote, new_remote_refs, rs, + tags, &autotags); +free_refs (new_remote_refs); + } +if (consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
Here, if we got updated remote refs, we need to regenerate ref_map, since it is the source of truth.
Maybe add a comment in the "if (new_remote_refs)" block explaining this
- something like: Regenerate ref_map using the updated remote refs, because the transport would place shallow (and other) information there.
fetch-pack.c static void update_shallow(struct fetch_pack_args *args, · line 1561
Quoted 5 lines
fetch-pack.c +2
-2-for (i =0 ; i < nr_sought; i++) +for (r = refs; r; r = r->next, i++)if (status[i]) - sought[i]->status =REF_STATUS_REJECT_SHALLOW ; + r->status =REF_STATUS_REJECT_SHALLOW ;
You use i here without initializing it to 0. t5703 also fails with this patch - probably related to this, but I didn't check.
If you initialize i here, I don't think you need to initialize it to 0 at the top of this function.
Re: [PATCH v2 7/8] fetch-pack: put shallow info in output parameter
Quoted 8 lines · Jonathan Tan
builtin/fetch.c static int do_fetch(struct transport *transport, · line 1127
Quoted 5 lines
builtin/fetch.c +1
-0@@ -1122,6 +1124,7 @@ static int do_fetch(struct transport *transport,int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ;const struct ref *remote_refs; +struct ref *new_remote_refs =NULL ;Above, you use the name "
updated_remote_refs" - it's probably better to standardize on one. I think "updated" is better.
Good catch I'll update the variable name.
Quoted 26 lines
(The transport calling it "
fetched_refs" is fine, because that's what they are from the perspective of the transport. From the perspective of fetch-pack, it is indeed a new or updated set of remote refs.)Quoted 7 lines
builtin/fetch.c +6
-1-if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) { + +if (fetch_refs (transport, ref_map, &new_remote_refs)) { +free_refs (ref_map); + retcode =1 ; + goto cleanup; + }builtin/fetch.c static int do_fetch(struct transport *transport, · line 1193
Quoted 10 lines
builtin/fetch.c +7
-0+if (new_remote_refs) { +free_refs (ref_map); + ref_map =get_ref_map (transport->remote, new_remote_refs, rs, + tags, &autotags); +free_refs (new_remote_refs); + } +if (consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;Here, if we got updated remote refs, we need to regenerate
ref_map, since it is the source of truth.
Quoted 4 lines
Maybe add a comment in the "if (
new_remote_refs)" block explaining this
- something like: Regenerateref_mapusing the updated remote refs, because the transport would place shallow (and other) information there.
That's probably a good idea to give future readers more context into why this is happening.
Quoted 9 lines
fetch-pack.c static void update_shallow(struct fetch_pack_args *args, · line 1561
Quoted 6 lines
fetch-pack.c +2
-2-for (i =0 ; i < nr_sought; i++) +for (r = refs; r; r = r->next, i++)if (status[i]) - sought[i]->status =REF_STATUS_REJECT_SHALLOW ; + r->status =REF_STATUS_REJECT_SHALLOW ;You use i here without initializing it to 0. t5703 also fails with this patch - probably related to this, but I didn't check.
Oh yeah that's definitely a bug, thanks for catching that.
Quoted 3 lines
If you initialize i here, I don't think you need to initialize it to 0 at the top of this function.
Signature · 1 line
Brandon Williams
PATCH v2 8/8 fetch-pack: implement ref-in-want
In reply to Brandon WilliamsImplement ref-in-want on the client side so that when a server supports the "ref-in-want" feature, a client will send "want-ref" lines for each reference the client wants to fetch.
--- fetch-pack.c | 35 +++++++++++++++++++++++++++--- remote.c | 1 + remote.h | 1 + t/t5703-upload-pack-ref-in-want.sh | 4 ++-- 4 files changed, 36 insertions(+), 5 deletions(-)
fetch-pack.c +32 -3
@@ -1102,9 +1102,10 @@ static void add_shallow_requests(struct strbuf *req_buf,static void add_wants (const struct ref *wants,struct strbuf *req_buf) { +int use_ref_in_want =server_supports_feature ("fetch" ,"ref-in-want" ,0 ); +for ( ; wants ; wants = wants->next) {const struct object_id *remote = &wants->old_oid; -const char *remote_hex;struct object *o;/*
@@ -1122,8 +1123,10 @@ static void add_wants(const struct ref *wants, struct strbuf *req_buf)continue ; } - remote_hex =oid_to_hex (remote); -packet_buf_write (req_buf,"want %s\n" , remote_hex); +if (!use_ref_in_want || wants->exact_sha1) +packet_buf_write (req_buf,"want %s\n" ,oid_to_hex (remote)); +else +packet_buf_write (req_buf,"want-ref %s\n" , wants->name); } }
@@ -1334,6 +1337,29 @@ static void receive_shallow_info(struct fetch_pack_args *args, args->deepen =1 ; } +static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line); + +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + } + } + +if (reader->status !=PACKET_READ_DELIM ) +die ("error processing wanted refs: %d" , reader->status); +} +enum fetch_state {FETCH_CHECK_LOCAL =0 ,FETCH_SEND_REQUEST ,
@@ -1408,6 +1434,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,if (process_section_header (&reader,"shallow-info" ,1 ))receive_shallow_info (args, &reader); +if (process_section_header (&reader,"wanted-refs" ,1 )) +receive_wanted_refs (&reader, ref); +/* get the pack */ process_section_header (&reader,"packfile" ,0 );if (get_pack (args, fd, pack_lockfile))
remote.c +1 -0
@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_sha1 =1 ; }else { ref_map =get_remote_ref (remote_refs, name); }
remote.h +1 -0
@@ -73,6 +73,7 @@ struct ref { force:1, forced_update:1, expect_old_sha1:1, + exact_sha1:1, deletion:1;enum {
t/t5703-upload-pack-ref-in-want.sh +2 -2
@@ -204,7 +204,7 @@ test_expect_success 'server is initially ahead - no ref in want' ' grep "ERR upload-pack: not our ref" err ' -test_expect_failure 'server is initially ahead - ref in want' ' +test_expect_success 'server is initially ahead - ref in want' ' git -C "$REPO" config uploadpack.allowRefInWant true && rm -rf local && cp -r "$LOCAL_PRISTINE" local &&
@@ -228,7 +228,7 @@ test_expect_success 'server is initially behind - no ref in want' ' test_cmp expected actual ' -test_expect_failure 'server is initially behind - ref in want' ' +test_expect_success 'server is initially behind - ref in want' ' git -C "$REPO" config uploadpack.allowRefInWant true && rm -rf local && cp -r "$LOCAL_PRISTINE" local &&
Signature · 1 line
2.18.0.rc1.242.g61856ae69a-goog
Re: [PATCH v2 8/8] fetch-pack: implement ref-in-want
Quoted 2 lines · Brandon Williams
fetch-pack.c +2
-0+static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{
...
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1356
Quoted 7 lines
fetch-pack.c +7
-0+ +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + }
The server is documented as MUST NOT send additional refs, which is fine here, as we'd have no way of storing them anyway. Do we want to issue a warning, though?
if (!r) /* never break'd */
warning ("server send unexpected line '%s'", reader.line);Quoted 17 lines
remote.c +1
-0@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_sha1 =1 ; }else { ref_map =get_remote_ref (remote_refs, name); }remote.h +0
-0
remote.h struct ref { · line 76
Quoted 4 lines
remote.h +1
-0@@ -73,6 +73,7 @@ struct ref { force:1, forced_update:1, expect_old_sha1:1, + exact_sha1:1,
Can we rename that to exact_oid ?
(bonus points for also converting expect_old_sha1)
Thanks,
Stefan
Re: [PATCH v2 8/8] fetch-pack: implement ref-in-want
Quoted 13 lines · Stefan Beller
Quoted 2 lines · Brandon Williams
fetch-pack.c +2
-0+static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{...
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1356
Quoted 7 lines
fetch-pack.c +7
-0+ +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + }
Quoted 6 lines
The server is documented as MUST NOT send additional refs, which is fine here, as we'd have no way of storing them anyway. Do we want to issue a warning, though?
if (!r) /* never break'd */ warning ("server send unexpected line '%s'", reader.line);
Depends, does this warning help out the end user or do you think it would confuse users to see this and still have their fetch succeed?
Quoted 18 lines
Quoted 18 lines
remote.c +1
-0@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_sha1 =1 ; }else { ref_map =get_remote_ref (remote_refs, name); }remote.h +0
-1--- a/remote.h
Quoted 8 lines
remote.h struct ref { · line 76
Quoted 6 lines
+++
b/remote.hremote.h +1
-0@@ -73,6 +73,7 @@ struct ref { force:1, forced_update:1, expect_old_sha1:1, + exact_sha1:1,Can we rename that to
exact_oid?
I'll fix this.
Signature · 1 line
Brandon Williams
Re: [PATCH v2 8/8] fetch-pack: implement ref-in-want
Hi,
Quoted 14 lines · Brandon Williams
Quoted 13 lines · Stefan Beller
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1356
Quoted 6 lines · Brandon Williams
fetch-pack.c +6
-0+for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + }The server is documented as MUST NOT send additional refs, which is fine here, as we'd have no way of storing them anyway. Do we want to issue a warning, though?
Quoted 5 lines
Quoted 2 lines
if (!r) /* never break'd */ warning ("server send unexpected line '%s'", reader.line);Depends, does this warning help out the end user or do you think it would confuse users to see this and still have their fetch succeed?
I think we'd want to error out instead of warning. That keeps the spec simple and that way, server implementors will notice early if they are doing something that clients aren't going to understand anyway, which would benefit users.
Thanks,
Jonathan
Quoted 3 lines · Brandon Williams
Changes in v2:
* issuing a want-ref line to a ref which doesn't exist is just ignored.
* fixed some typos
Do I lock some (logical) prerequisite changes? On 2.18-rc2 they apply cleanly but the series fails its own test.
t5703-upload-pack-ref-in-want.sh
expecting success:
test-pkt-line pack >in <<-EOF &&
command=fetch
0001
no-progress
want-ref refs/heads/non-existent
done
0000
EOF
test_must_fail git serve --stateless-rpc 2>out <in &&
grep "unknown ref" outtest_must_fail: command succeeded: git serve --stateless-rpc not ok 3 - invalid want-ref line
Quoted 8 lines · Junio C Hamano
Quoted 3 lines · Brandon Williams
Changes in v2:
* issuing a want-ref line to a ref which doesn't exist is just ignored.
* fixed some typosDo I lock some (logical) prerequisite changes? On 2.18-rc2 they apply cleanly but the series fails its own test.
No this is an error I made in this version of the series which another reviewer pointed out, I have a local v3 which addresses this (by removing the test since it isn't necessary anymore). Sorry for the mistake :)
Quoted 17 lines
t5703-upload-pack-ref-in-want.sh
expecting success:test-pkt-line pack >in <<-EOF && command=fetch 0001 no-progress want-ref refs/heads/non-existent done 0000 EOF test_must_fail git serve --stateless-rpc 2>out <in && grep "unknown ref" out
test_must_fail: command succeeded: git serve --stateless-rpc not ok 3 - invalid want-ref line
Signature · 1 line
Brandon Williams
v3 cover ref-in-want
In reply to Brandon WilliamsChanges in v3:
* Discussion seemed to settle on keeping the simplified version of ref-in-want where the "want-ref" line only accepts full ref names. If we want to we can add patterns at a later time.
* Reverted back to v1's behavior where requesting a ref that doesn't exists is a hard error on the server. I went back and forth many times on what the right thing to do here is and decided that a hard error works much cleaner for the time being.
* Some typos.
Brandon Williams (8):
test-pkt-line: add unpack-sideband subcommand
upload-pack: implement ref-in-want
upload-pack: test negotiation with changing repository
fetch: refactor the population of peer ref OIDs
fetch: refactor fetch_refs into two functions
fetch: refactor to make function args narrower
fetch-pack: put shallow info in output parameter
fetch-pack: implement ref-in-want
Documentation/config.txt | 7 + Documentation/technical/protocol-v2.txt | 28 ++- builtin/clone.c | 4 +- builtin/fetch.c | 131 ++++++++----- fetch-object.c | 2 +- fetch-pack.c | 52 +++-- remote.c | 1 + remote.h | 1 + t/helper/test-pkt-line.c | 33 ++++ t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 + t/lib-httpd/one-time-sed.sh | 16 ++ t/t5703-upload-pack-ref-in-want.sh | 245 ++++++++++++++++++++++++ transport-helper.c | 6 +- transport-internal.h | 9 +- transport.c | 34 +++- transport.h | 3 +- upload-pack.c | 64 +++++++ 18 files changed, 568 insertions(+), 77 deletions(-) create mode 100644 t/lib-httpd/one-time-sed.sh create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
PATCH v3 1/8 test-pkt-line: add unpack-sideband subcommand
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
--- t/helper/test-pkt-line.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
t/helper/test-pkt-line.c +33 -0
@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h" static void pack_line (const char *line)
@@ -48,6 +49,36 @@ static void unpack(void) } } +static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE ); + +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) { +case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ; +if (band <1 || band >2 ) +die ("unexpected side band %d" , band); + fd = band; + +write_or_die (fd, reader.line +1 , reader.pktlen -1 ); +break ; +case PACKET_READ_FLUSH : +return ; +case PACKET_READ_DELIM : +break ; + } + } +} +int cmd_main (int argc,const char **argv) {if (argc <2 )
@@ -57,6 +88,8 @@ int cmd_main(int argc, const char **argv)pack (argc -2 , argv +2 ); else if (!strcmp(argv[1], "unpack")) unpack(); + else if (!strcmp(argv[1], "unpack-sideband")) + unpack_sideband(); else die("invalid argument '%s'", argv[1]);
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
Re: [PATCH v3 1/8] test-pkt-line: add unpack-sideband subcommand
Hi,
Quoted 7 lines · Brandon Williams
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
Signed-off-by Brandon Williams <[email protected]>--- t/helper/test-pkt-line.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
Neat. It appears that this writes sideband channel 1 (packfile data) to stdout, sideband channel 2 (progress) to stderr, and sideband channel 3 (errors) cause the helper to fail. It would have been nice if a comment or the commit message said that, but it's no reason to reroll --- the code is clear enough.
t/helper/test-pkt-line.c line 2
Quoted 5 lines
t/helper/test-pkt-line.c +1
-0@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h"
I think this is for write_or_die. Makes sense (well, in the same way as any of the other functions that ended up in cache.h instead of a more thought-through place do).
The old #includes were problematic, since the caller cannot count on git-compat-util.h to be the first include of pkt-line.h. See Documentation/CodingGuidelines "The first #include" for more on this subject.
[...]
Quoted 6 lines
t/helper/test-pkt-line.c +6
-0+static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE );
t/helper/test-pkt-line.c static void unpack(void) · line 67
Quoted 10 lines
t/helper/test-pkt-line.c +10
-0+ +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) { +case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ;
reader.line[0] is a char. This promotes it to an 'int' and then ANDs against 0xff, which would ensure it is a positive value. In other words, this does the same thing as
band = (int) (unsigned char) reader.line[0];
but more concisely.
More importantly, it matches what recv_sideband does. Good.
Thanks.
PATCH v3 2/8 upload-pack: implement ref-in-want
In reply to Brandon WilliamsCurrently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement and there exists replication delay.
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
--- Documentation/config.txt | 7 ++ Documentation/technical/protocol-v2.txt | 28 ++++- t/t5703-upload-pack-ref-in-want.sh | 153 ++++++++++++++++++++++++ upload-pack.c | 64 ++++++++++ 4 files changed, 251 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Documentation/config.txt +7 -0
@@ -3479,6 +3479,13 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories). +uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. This feature + is intended for the benefit of load-balanced servers which may + not have the same view of what OIDs their refs point to due to + replication delay. + url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves a
Documentation/technical/protocol-v2.txt +27 -1
@@ -299,12 +299,21 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values. +If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. output = *section - section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt) acknowledgments = PKT-LINE("acknowledgments" LF)
@@ -319,6 +328,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff)
@@ -379,6 +392,19 @@ header. * This section is only included if a packfile section is also included in the response. + wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs" + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server MUST NOT send any refs which were not requested + using 'want-ref' lines. + packfile section * This section is only included if the client has sent 'want' lines in its request and either requested that no more
t/t5703-upload-pack-ref-in-want.sh new file +153 -0
153 lines not on this page · raw message
upload-pack.c +64 -0
136 lines not on this page · raw message
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
Re: [PATCH v3 2/8] upload-pack: implement ref-in-want
Documentation/technical/protocol-v2.txt header. · line 400
Quoted 6 lines
Documentation/technical/protocol-v2.txt +6
-0+ wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs"
Add a period at the end to be consistent with the others.
Documentation/technical/protocol-v2.txt header. · line 406
Quoted 5 lines
Documentation/technical/protocol-v2.txt +5
-0+ * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server MUST NOT send any refs which were not requested + using 'want-ref' lines.
We might want tag following refs to be included here in the future, but at that time, I think we can amend this to say that if include-tag-ref is sent by the user, the server may send additional refs, otherwise the server must not do so. So this is fine.
Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh new file +10
-0+test_expect_success 'mix want and want-ref' ' + cat >expected_refs <<-EOF && + $(git rev-parse f) refs/heads/master + EOF + git rev-parse e f | sort >expected_commits && + + test-pkt-line pack >in <<-EOF && + command=fetch + 0001 + no-progress
t/t5703-upload-pack-ref-in-want.sh line 131
Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh new file +10
-0+ want-ref refs/heads/master + want $(git rev-parse e) + have $(git rev-parse a) + done + 0000 + EOF + + git serve --stateless-rpc >out <in && + check_output +'
Overall the tests look good, although I might be a bit biased since they are based on what I wrote a while ago [1]. I was wondering about the behavior when the client mixes "want" and "want-ref" (as will happen if they fetch both a ref by name and an exact SHA-1), and this test indeed shows the expected behavior.
Quoted 9 lines
t/t5703-upload-pack-ref-in-want.sh new file +9
-0+test_expect_success 'want-ref with ref we already have commit for' ' + cat >expected_refs <<-EOF && + $(git rev-parse c) refs/heads/o/foo + EOF + >expected_commits && + + test-pkt-line pack >in <<-EOF && + command=fetch + 0001
t/t5703-upload-pack-ref-in-want.sh line 151
Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh new file +10
-0+ no-progress + want-ref refs/heads/o/foo + have $(git rev-parse c) + done + 0000 + EOF + + git serve --stateless-rpc >out <in && + check_output +'
Likewise for this test - the ref is still reported, but the packfile does not contain the requested object.
upload-pack.c void upload_pack(struct upload_pack_options *options) · line 1120
Quoted 3 lines
struct
upload_pack_data{upload-pack.c +1
-0struct object_array wants; +struct string_list wanted_refs;
Document here that this is a map from ref names to owned struct object_id instances.
Quoted 9 lines
upload-pack.c +9
-0+static int parse_want_ref (const char *line,struct string_list *wanted_refs) +{ +const char *arg; +if (skip_prefix (line,"want-ref " , &arg)) { +struct object_id oid; +struct string_list_item *item; +struct object *o; + +if (read_ref (arg, &oid))
upload-pack.c static int parse_want(const char *line) · line 1213
Quoted 10 lines
upload-pack.c +10
-0+die ("unknown ref %s" , arg); + + item =string_list_append (wanted_refs, arg); + item->util =oiddup (&oid); + + o =parse_object_or_die (&oid, arg); +if (!(o->flags &WANTED )) { + o->flags |=WANTED ; +add_object_array (o,NULL , &want_obj); + }
Makes sense - besides adding it to wanted_refs, this adds the object to want_obj, just like how the other code paths for "want" adds it.
Quoted 7 lines
upload-pack.c +7
-0+static void send_wanted_ref_info (struct upload_pack_data *data) +{ +const struct string_list_item *item; + +if (!data->wanted_refs.nr) +return ; +
upload-pack.c static int process_haves_and_send_acks(struct upload_pack_data *data) · line 1406
Quoted 10 lines
upload-pack.c +10
-0+packet_write_fmt (1 ,"wanted-refs\n" ); + +for_each_string_list_item (item, &data->wanted_refs) { +packet_write_fmt (1 ,"%s %s\n" , +oid_to_hex (item->util), + item->string); + } + +packet_delim (1 ); +}
The documentation states that the "wanted-refs" section is only sent if there is at least one "want-ref" from the client, and each "want-ref" causes one entry to be added to data->wanted_refs, so this is correct.
Thanks - besides adding the period in the documentation, this patch looks good to me.
Re: [PATCH v3 2/8] upload-pack: implement ref-in-want
In reply to Brandon Williamsupload-pack.c static int parse_want(const char *line) · line 1204
Quoted 10 lines
upload-pack.c +10
-0+static int parse_want_ref (const char *line,struct string_list *wanted_refs) +{ +const char *arg; +if (skip_prefix (line,"want-ref " , &arg)) { +struct object_id oid; +struct string_list_item *item; +struct object *o; + +if (read_ref (arg, &oid)) +die ("unknown ref %s" , arg);
One more thing - if you're planning to "die" here, also write out an error to the user, just like in parse_want().
Re: [PATCH v3 2/8] upload-pack: implement ref-in-want
Quoted 5 lines · Jonathan Tan
Quoted 5 lines
upload-pack.c +5
-0+static int parse_want_ref (const char *line,struct string_list *wanted_refs) +{ +const char *arg; +if (skip_prefix (line,"want-ref " , &arg)) { +struct object_id oid;
Quoted 8 lines
upload-pack.c static int parse_want(const char *line) · line 1204
Quoted 5 lines
upload-pack.c +5
-0+struct string_list_item *item; +struct object *o; + +if (read_ref (arg, &oid)) +die ("unknown ref %s" , arg);One more thing - if you're planning to "die" here, also write out an error to the user, just like in
parse_want().
Oh good idea, I'll add an ERR pkt here
Signature · 1 line
Brandon Williams
PATCH v3 3/8 upload-pack: test negotiation with changing repository
In reply to Brandon WilliamsAdd tests to check the behavior of fetching from a repository which changes between rounds of negotiation (for example, when different servers in a load-balancing agreement participate in the same stateless RPC negotiation). This forms a baseline of comparison to the ref-in-want functionality (which will be introduced to the client in subsequent commits), and ensures that subsequent commits do not change existing behavior.
As part of this effort, a mechanism to substitute strings in a single HTTP response is added.
--- t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 +++ t/lib-httpd/one-time-sed.sh | 16 ++++++ t/t5703-upload-pack-ref-in-want.sh | 92 ++++++++++++++++++++++++++++++ 4 files changed, 117 insertions(+) create mode 100644 t/lib-httpd/one-time-sed.sh
t/lib-httpd.sh +1 -0
@@ -132,6 +132,7 @@ prepare_httpd() { cp "$TEST_PATH"/passwd "$HTTPD_ROOT_PATH" install_script broken-smart-http.sh install_script error.sh + install_script one-time-sed.sh ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +8 -0
@@ -111,9 +111,14 @@ Alias /auth/dumb/ www/auth/dumb/ SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} SetEnv GIT_HTTP_EXPORT_ALL </LocationMatch> +<LocationMatch /one_time_sed/> + SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} + SetEnv GIT_HTTP_EXPORT_ALL +</LocationMatch> ScriptAliasMatch /smart_*[^/]*/(.*) ${GIT_EXEC_PATH}/git-http-backend/$1 ScriptAlias /broken_smart/ broken-smart-http.sh/ ScriptAlias /error/ error.sh/ +ScriptAliasMatch /one_time_sed/(.*) one-time-sed.sh/$1 <Directory ${GIT_EXEC_PATH}> Options FollowSymlinks </Directory>
@@ -123,6 +128,9 @@ ScriptAlias /error/ error.sh/ <Files error.sh> Options ExecCGI </Files> +<Files one-time-sed.sh> + Options ExecCGI +</Files> <Files ${GIT_EXEC_PATH}/git-http-backend> Options ExecCGI </Files>
t/lib-httpd/one-time-sed.sh new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/sh + +if [ -e one-time-sed ]; then + "$GIT_EXEC_PATH/git-http-backend" >out + + sed "$(cat one-time-sed)" <out >out_modified + + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fi
t/t5703-upload-pack-ref-in-want.sh +92 -0
96 lines not on this page · raw message
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
PATCH v3 4/8 fetch: refactor the population of peer ref OIDs
In reply to Brandon WilliamsPopulate peer ref OIDs in get_ref_map instead of do_fetch. Besides tightening scopes of variables in the code, this also prepares for get_ref_map being able to be called multiple times within do_fetch.
--- builtin/fetch.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
builtin/fetch.c +18 -18
@@ -351,6 +351,7 @@ static struct ref *get_ref_map(struct transport *transport,/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs; +struct string_list existing_refs =STRING_LIST_INIT_DUP ;const struct ref *remote_refs;if (rs->nr)
@@ -458,7 +459,23 @@ static struct ref *get_ref_map(struct transport *transport, tail = &rm->next; } -return ref_remove_duplicates (ref_map); + ref_map =ref_remove_duplicates (ref_map); + +for_each_ref (add_existing, &existing_refs); +for (rm = ref_map; rm; rm = rm->next) { +if (rm->peer_ref) { +struct string_list_item *peer_item = +string_list_lookup (&existing_refs, + rm->peer_ref->name); +if (peer_item) { +struct object_id *old_oid = peer_item->util; +oidcpy (&rm->peer_ref->old_oid, old_oid); + } + } + } +string_list_clear (&existing_refs,1 ); + +return ref_map; }#define STORE_REF_ERROR_OTHER 1
@@ -1110,14 +1127,10 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)static int do_fetch (struct transport *transport,struct refspec *rs) { -struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct ref *ref_map; -struct ref *rm;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; -for_each_ref (add_existing, &existing_refs); -if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 ) tags =TAGS_SET ;
@@ -1136,18 +1149,6 @@ static int do_fetch(struct transport *transport,if (!update_head_ok)check_not_current_branch (ref_map); -for (rm = ref_map; rm; rm = rm->next) { -if (rm->peer_ref) { -struct string_list_item *peer_item = -string_list_lookup (&existing_refs, - rm->peer_ref->name); -if (peer_item) { -struct object_id *old_oid = peer_item->util; -oidcpy (&rm->peer_ref->old_oid, old_oid); - } - } - } -if (tags ==TAGS_DEFAULT && autotags)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,"1" );if (prune) {
@@ -1183,7 +1184,6 @@ static int do_fetch(struct transport *transport, } cleanup: -string_list_clear (&existing_refs,1 );return retcode; }
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
Re: [PATCH v3 4/8] fetch: refactor the population of peer ref OIDs
Quoted 3 lines
Populate peer ref OIDs in
get_ref_mapinstead ofdo_fetch. Besides tightening scopes of variables in the code, this also prepares forget_ref_mapbeing able to be called multiple times withindo_fetch.
get_ref_map() is only called in one place in builtin/fetch.c, and that place is in do_fetch(), so moving functionality from do_fetch() to get_ref_map() is perfectly fine, and also allows tightening of the scope of the existing_refs variable.
PATCH v3 5/8 fetch: refactor fetch_refs into two functions
In reply to Brandon WilliamsRefactor the fetch_refs function into a function that does the fetching of refs and another function that stores them.
--- builtin/fetch.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
builtin/fetch.c +13 -6
@@ -967,10 +967,16 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)int ret =quickfetch (ref_map);if (ret) ret =transport_fetch_refs (transport, ref_map); -if (!ret) - ret |=store_updated_refs (transport->url, - transport->remote->name, - ref_map); +if (ret) +transport_unlock_pack (transport); +return ret; +} + +static int consume_refs (struct transport *transport,struct ref *ref_map) +{ +int ret =store_updated_refs (transport->url, + transport->remote->name, + ref_map);transport_unlock_pack (transport);return ret; }
@@ -1116,7 +1122,8 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -fetch_refs (transport, ref_map); +if (!fetch_refs (transport, ref_map)) +consume_refs (transport, ref_map);if (gsecondary) {transport_disconnect (gsecondary);
@@ -1165,7 +1172,7 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map)) { +if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
Re: [PATCH v3 5/8] fetch: refactor fetch_refs into two functions
Hi,
Quoted 9 lines · Brandon Williams
[Subject: fetch: refactor
fetch_refsinto two functions]Refactor the
fetch_refsfunction into a function that does the fetching of refs and another function that stores them.Signed-off-by Brandon Williams <[email protected]>--- builtin/fetch.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
It's hard to understand the context for this patch based on this description alone. E.g. is fetch_refs too long to follow? Or are we about to expand it? Or are we going to use the factored-out subpart for something else?
[...]
Quoted 4 lines
builtin/fetch.c +0
-0@@ -967,10 +967,16 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)int ret =quickfetch (ref_map);
builtin/fetch.c static int fetch_refs(struct transport *transport, struct ref *ref_map) · line 973
Quoted 10 lines
builtin/fetch.c +4
-4if (ret) ret =transport_fetch_refs (transport, ref_map); -if (!ret) - ret |=store_updated_refs (transport->url, - transport->remote->name, - ref_map); +if (ret) +transport_unlock_pack (transport); +return ret; +}
Paraphrasing the old code:
try quickfetch
if that fails, we have to fetch for real
both of the above "lock" a pack using a .keep file to avoid races
if the fetch succeeded, now update the refs
finally, "unlock" the pack by rm-ing the .keep file
Paraphrasing the new code:
try quickfetch
if that fails, we have to fetch for real
both of the above "lock" a pack using a .keep file to avoid races
if the fetch failed, "unlock" the pack by rm-ing the .keep file.
Do I understand correctly that this is preparation for changing the 'update refs' step?
As a minor nit, I think this would be easier to read if we treat the unlock_pack as a destructor. Something like this:
int ret = quickfetch(ref_map); if (ret) ret = transport_fetch_refs(transport, ref_map); if (!ret) /* * Keep the new pack's ".keep" file around to allow * the caller time to update refs to reference the new * objects. */ return 0; transport_unlock_pack(transport); return ret;
builtin/fetch.c static int fetch_refs(struct transport *transport, struct ref *ref_map) · line 975
Quoted 2 lines
builtin/fetch.c +2
-0+ +static int consume_refs (struct transport *transport,struct ref *ref_map)
The name consume_refs doesn't make it clear to me what this function is going to do. Maybe a function comment can help.
I.e. either the name or a comment would tell me that this is going to update local refs based on the ref values that the remote end told us.
The rest of the patch looks good.
Thanks and hope that helps,
Jonathan
Re: [PATCH v3 5/8] fetch: refactor fetch_refs into two functions
In reply to Brandon WilliamsQuoted 23 lines · Brandon Williams
builtin/fetch.c +10
-4@@ -967,10 +967,16 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)int ret =quickfetch (ref_map);if (ret) ret =transport_fetch_refs (transport, ref_map); -if (!ret) - ret |=store_updated_refs (transport->url, - transport->remote->name, - ref_map); +if (ret) +transport_unlock_pack (transport); +return ret; +} + +static int consume_refs (struct transport *transport,struct ref *ref_map) +{ +int ret =store_updated_refs (transport->url, + transport->remote->name, + ref_map);transport_unlock_pack (transport);return ret; }
[...]
builtin/fetch.c static void backfill_tags(struct transport *transport, struct ref *ref_map) · line 1126
Quoted 4 lines
builtin/fetch.c +2
-1-fetch_refs (transport, ref_map); +if (!fetch_refs (transport, ref_map)) +consume_refs (transport, ref_map);
Ah, I missed something in my previous reply.
If transport_fetch_refs succeeds and store_updated_refs fails, then in the old code, transport_unlock_pack would clean up by removing the no longer needed .keep file. In the new code, that's consume_refs's responsibility, which I find much nicer. It's probably worth mentioning that in the commit message as well.
Thanks again,
Jonathan
PATCH v3 6/8 fetch: refactor to make function args narrower
In reply to Brandon WilliamsRefactor find_non_local_tags and get_ref_map to only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
--- builtin/fetch.c | 52 ++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
builtin/fetch.c +26 -26
@@ -254,9 +254,9 @@ static int will_fetch(struct ref **head, const unsigned char *sha1)return 0 ; } -static void find_non_local_tags (struct transport *transport, -struct ref **head, -struct ref ***tail) +static void find_non_local_tags (const struct ref *refs, +struct ref **head, +struct ref ***tail) {struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct string_list remote_refs =STRING_LIST_INIT_NODUP ;
@@ -264,7 +264,7 @@ static void find_non_local_tags(struct transport *transport,struct string_list_item *item =NULL ;for_each_ref (add_existing, &existing_refs); -for (ref =transport_get_remote_refs (transport,NULL ); ref; ref = ref->next) { +for (ref = refs; ref; ref = ref->next) {if (!starts_with (ref->name,"refs/tags/" ))continue ;
@@ -338,7 +338,8 @@ static void find_non_local_tags(struct transport *transport,string_list_clear (&remote_refs,0 ); } -static struct ref *get_ref_map (struct transport *transport, +static struct ref *get_ref_map (struct remote *remote, +const struct ref *remote_refs,struct refspec *rs,int tags,int *autotags) {
@@ -346,27 +347,11 @@ static struct ref *get_ref_map(struct transport *transport,struct ref *rm;struct ref *ref_map =NULL ;struct ref **tail = &ref_map; -struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs;struct string_list existing_refs =STRING_LIST_INIT_DUP ; -const struct ref *remote_refs; - -if (rs->nr) -refspec_ref_prefixes (rs, &ref_prefixes); -else if (transport->remote && transport->remote->fetch.nr) -refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); - -if (ref_prefixes.argc && - (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { -argv_array_push (&ref_prefixes,"refs/tags/" ); - } - - remote_refs =transport_get_remote_refs (transport, &ref_prefixes); - -argv_array_clear (&ref_prefixes);if (rs->nr) {struct refspec *fetch_refspec;
@@ -403,7 +388,7 @@ static struct ref *get_ref_map(struct transport *transport,if (refmap.nr) fetch_refspec = &refmap;else - fetch_refspec = &transport->remote->fetch; + fetch_refspec = &remote->fetch;for (i =0 ; i < fetch_refspec->nr; i++)get_fetch_map (ref_map, &fetch_refspec->items[i], &oref_tail,1 );
@@ -411,7 +396,6 @@ static struct ref *get_ref_map(struct transport *transport,die ("--refmap option is only meaningful with command-line refspec(s)." ); }else {/* Use the defaults */ -struct remote *remote = transport->remote;struct branch *branch =branch_get (NULL );int has_merge =branch_has_merge_config (branch); if (remote &&
@@ -450,7 +434,7 @@ static struct ref *get_ref_map(struct transport *transport,/* also fetch all tags */ get_fetch_map (remote_refs, tag_refspec, &tail,0 ); else if (tags == TAGS_DEFAULT && *autotags) - find_non_local_tags(transport, &ref_map, &tail); + find_non_local_tags(remote_refs, &ref_map, &tail);/* Now append any refs to be updated opportunistically: */ *tail = orefs;
@@ -1137,6 +1121,8 @@ static int do_fetch(struct transport *transport,struct ref *ref_map;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; +const struct ref *remote_refs; +struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 )
@@ -1152,7 +1138,21 @@ static int do_fetch(struct transport *transport, goto cleanup; } - ref_map =get_ref_map (transport, rs, tags, &autotags); +if (rs->nr) +refspec_ref_prefixes (rs, &ref_prefixes); +else if (transport->remote && transport->remote->fetch.nr) +refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); + +if (ref_prefixes.argc && + (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { +argv_array_push (&ref_prefixes,"refs/tags/" ); + } + + remote_refs =transport_get_remote_refs (transport, &ref_prefixes); +argv_array_clear (&ref_prefixes); + + ref_map =get_ref_map (transport->remote, remote_refs, rs, + tags, &autotags);if (!update_head_ok)check_not_current_branch (ref_map);
@@ -1184,7 +1184,7 @@ static int do_fetch(struct transport *transport,if (tags ==TAGS_DEFAULT && autotags) {struct ref **tail = &ref_map; ref_map =NULL ; -find_non_local_tags (transport, &ref_map, &tail); +find_non_local_tags (remote_refs, &ref_map, &tail);if (ref_map)backfill_tags (transport, ref_map);free_refs (ref_map);
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
PATCH v3 7/8 fetch-pack: put shallow info in output parameter
In reply to Brandon WilliamsExpand the transport fetch method signature, by adding an output parameter, to allow transports to return information about the refs they have fetched. Then communicate shallow status information through this mechanism instead of by modifying the input list of refs.
This does require clients to sometimes generate the ref map twice: once from the list of refs provided by the remote (as is currently done) and potentially once from the new list of refs that the fetch mechanism provides.
--- builtin/clone.c | 4 ++-- builtin/fetch.c | 28 ++++++++++++++++++++++++---- fetch-object.c | 2 +- fetch-pack.c | 17 +++++++++-------- transport-helper.c | 6 ++++-- transport-internal.h | 9 ++++++++- transport.c | 34 ++++++++++++++++++++++++++++------ transport.h | 3 ++- 8 files changed, 78 insertions(+), 25 deletions(-)
builtin/clone.c +2 -2
@@ -1155,7 +1155,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (!is_local && !complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); remote_head =find_ref_by_name (refs,"HEAD" ); remote_head_points_at =
@@ -1197,7 +1197,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)if (is_local)clone_local (path, git_dir);else if (refs && complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); update_remote_refs(refs, mapped_refs, remote_head_points_at, branch_top.buf, reflog_msg.buf, transport,
builtin/fetch.c +24 -4
@@ -946,11 +946,13 @@ static int quickfetch(struct ref *ref_map)return check_connected (iterate_ref_map, &rm, &opt); } -static int fetch_refs (struct transport *transport,struct ref *ref_map) +static int fetch_refs (struct transport *transport,struct ref *ref_map, +struct ref **updated_remote_refs) {int ret =quickfetch (ref_map);if (ret) - ret =transport_fetch_refs (transport, ref_map); + ret =transport_fetch_refs (transport, ref_map, + updated_remote_refs);if (ret)transport_unlock_pack (transport);return ret;
@@ -1106,7 +1108,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -if (!fetch_refs (transport, ref_map)) +if (!fetch_refs (transport, ref_map,NULL ))consume_refs (transport, ref_map);if (gsecondary) {
@@ -1122,6 +1124,7 @@ static int do_fetch(struct transport *transport,int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ;const struct ref *remote_refs; +struct ref *updated_remote_refs =NULL ;struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {
@@ -1172,7 +1175,24 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) { + +if (fetch_refs (transport, ref_map, &updated_remote_refs)) { +free_refs (ref_map); + retcode =1 ; + goto cleanup; + } +if (updated_remote_refs) { +/* +* Regenerate ref_map using the updated remote refs. This is +* to account for additional information which may be provided +* by the transport (e.g. shallow info). +*/ +free_refs (ref_map); + ref_map =get_ref_map (transport->remote, updated_remote_refs, rs, + tags, &autotags); +free_refs (updated_remote_refs); + } +if (consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
fetch-object.c +1 -1
@@ -19,7 +19,7 @@ static void fetch_refs(const char *remote_name, struct ref *ref)transport_set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" );transport_set_option (transport,TRANS_OPT_NO_DEPENDENTS ,"1" ); -transport_fetch_refs (transport, ref); +transport_fetch_refs (transport, ref,NULL ); fetch_if_missing = original_fetch_if_missing; }
fetch-pack.c +9 -8
@@ -1470,12 +1470,13 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr) }static void update_shallow (struct fetch_pack_args *args, -struct ref **sought,int nr_sought, +struct ref *refs,struct shallow_info *si) {struct oid_array ref =OID_ARRAY_INIT ;int *status; -int i; +int i =0 ; +struct ref *r;if (args->deepen && alternate_shallow_file) {if (*alternate_shallow_file =='\0' ) {/* --unshallow */
@@ -1517,8 +1518,8 @@ static void update_shallow(struct fetch_pack_args *args,remove_nonexistent_theirs_shallow (si);if (!si->nr_ours && !si->nr_theirs)return ; -for (i =0 ; i < nr_sought; i++) -oid_array_append (&ref, &sought[i]->old_oid); +for (r = refs; r; r = r->next) +oid_array_append (&ref, &r->old_oid); si->ref = &ref;if (args->update_shallow) {
@@ -1552,12 +1553,12 @@ static void update_shallow(struct fetch_pack_args *args,* remote is also shallow, check what ref is safe to update * without updating .git/shallow */ - status =xcalloc (nr_sought,sizeof (*status)); + status =xcalloc (ref.nr,sizeof (*status));assign_shallow_commits_to_refs (si,NULL , status);if (si->nr_ours || si->nr_theirs) { -for (i =0 ; i < nr_sought; i++) +for (r = refs, i =0 ; r; r = r->next, i++)if (status[i]) - sought[i]->status =REF_STATUS_REJECT_SHALLOW ; + r->status =REF_STATUS_REJECT_SHALLOW ; }free (status);oid_array_clear (&ref);
@@ -1591,7 +1592,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args, ref_cpy =do_fetch_pack (args, fd, ref, sought, nr_sought, &si, pack_lockfile);reprepare_packed_git (the_repository); -update_shallow (args, sought, nr_sought, &si); +update_shallow (args, ref_cpy, &si);clear_shallow_info (&si);return ref_cpy; }
transport-helper.c +4 -2
@@ -651,14 +651,16 @@ static int connect_helper(struct transport *transport, const char *name, }static int fetch (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct helper_data *data = transport->data;int i, count;if (process_connect (transport,0 )) {do_take_over (transport); -return transport->vtable->fetch (transport, nr_heads, to_fetch); +return transport->vtable->fetch (transport, nr_heads, to_fetch, + fetched_refs); } count =0 ;
transport-internal.h +8 -1
@@ -36,11 +36,18 @@ struct transport_vtable {* Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. * +* The transport *may* provide, in fetched_refs, the list of refs that +* it fetched. If the transport knows anything about the fetched refs +* that the caller does not know (for example, shallow status), it +* should provide that list of refs and include that information in the +* list. +* * If the transport did not get hashes for refs in * get_refs_list(), it should set the old_sha1 fields in the * provided refs now. **/ -int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs); +int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs, +struct ref **fetched_refs);/** * Push the objects and refs. Send the necessary objects, and
transport.c +28 -6
@@ -151,7 +151,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport, }static int fetch_refs_from_bundle (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct bundle_transport_data *data = transport->data; return unbundle(&data->header, data->fd,
@@ -287,7 +288,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus }static int fetch_refs_via_pack (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -354,8 +356,12 @@ static int fetch_refs_via_pack(struct transport *transport,if (report_unmatched_refs (to_fetch, nr_heads)) ret =-1 ; +if (fetched_refs) + *fetched_refs = refs; +else +free_refs (refs); +free_refs (refs_tmp); -free_refs (refs);free (dest);return ret; }
@@ -1215,19 +1221,31 @@ const struct ref *transport_get_remote_refs(struct transport *transport,return transport->remote_refs; } -int transport_fetch_refs (struct transport *transport,struct ref *refs) +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs) {int rc;int nr_heads =0 , nr_alloc =0 , nr_refs =0 ;struct ref **heads =NULL ; +struct ref *nop_head =NULL , **nop_tail = &nop_head;struct ref *rm;for (rm = refs; rm; rm = rm->next) { nr_refs++;if (rm->peer_ref && !is_null_oid (&rm->old_oid) && - !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) + !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don't +* actually need to fetch them. +*/ +if (fetched_refs) { +struct ref *nop_ref =copy_ref (rm); + *nop_tail = nop_ref; + nop_tail = &nop_ref->next; + }continue ; + }ALLOC_GROW (heads, nr_heads +1 , nr_alloc); heads[nr_heads++] = rm; }
@@ -1245,7 +1263,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) heads[nr_heads++] = rm; } - rc = transport->vtable->fetch (transport, nr_heads, heads); + rc = transport->vtable->fetch (transport, nr_heads, heads, fetched_refs); +if (fetched_refs && nop_head) { + *nop_tail = *fetched_refs; + *fetched_refs = nop_head; + }free (heads);return rc;
transport.h +2 -1
@@ -218,7 +218,8 @@ int transport_push(struct transport *connection,const struct ref *transport_get_remote_refs (struct transport *transport,const struct argv_array *ref_prefixes); -int transport_fetch_refs (struct transport *transport,struct ref *refs); +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs);void transport_unlock_pack (struct transport *transport);int transport_disconnect (struct transport *transport);char *transport_anonymize_url (const char *url);
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
Re: [PATCH v3 7/8] fetch-pack: put shallow info in output parameter
fetch-pack.c static int remove_duplicates_in_refs(struct ref **ref, int nr) · line 1473
Quoted 3 lines
static void update_shallow(struct
fetch_pack_args*args,fetch-pack.c +1
-1- struct ref **sought, int nr_sought, + struct ref *refs,
update_shallow() now takes in a linked list of refs instead of an array. I see that the translation of this function is straightforward - occasionally, we need to iterate through the linked list and count up from 0 at the same time, but that is not a problem.
fetch-pack.c static int remove_duplicates_in_refs(struct ref **ref, int nr) · line 1478
Quoted 6 lines
fetch-pack.c +1
-1struct shallow_info *si) {struct oid_array ref =OID_ARRAY_INIT ;int *status; -int i; +int i =0 ;
Remove the " = 0" - I've verified that it does not need to be there, and it might inhibit useful "unintialized variable" warnings if others were to change the code later.
Optional: I would also remove this declaration and declare "int i;" in each of the blocks that need it.
Quoted 11 lines
static int fetch_refs_via_pack(struct transport *transport,
transport.c +2
-1- int nr_heads, struct ref **to_fetch) + int nr_heads, struct ref **to_fetch, + struct ref **fetched_refs) { int ret = 0;struct git_transport_data *data = transport->data;@@ -354,8 +356,12 @@ static int fetch_refs_via_pack(struct transport *transport,if (report_unmatched_refs (to_fetch, nr_heads)) ret =-1 ;
transport.c static int fetch_refs_via_pack(struct transport *transport, · line 367
Quoted 10 lines
transport.c +5
-1+if (fetched_refs) + *fetched_refs = refs; +else +free_refs (refs); +free_refs (refs_tmp); -free_refs (refs);free (dest);return ret; }
Instead of just freeing the linked list, we return it if requested by the client. This makes sense.
Quoted 33 lines
transport.c +14
-2-int transport_fetch_refs (struct transport *transport,struct ref *refs) +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs) {int rc;int nr_heads =0 , nr_alloc =0 , nr_refs =0 ;struct ref **heads =NULL ; +struct ref *nop_head =NULL , **nop_tail = &nop_head;struct ref *rm;for (rm = refs; rm; rm = rm->next) { nr_refs++;if (rm->peer_ref && !is_null_oid (&rm->old_oid) && - !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) + !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don't +* actually need to fetch them. +*/ +if (fetched_refs) { +struct ref *nop_ref =copy_ref (rm); + *nop_tail = nop_ref; + nop_tail = &nop_ref->next; + }continue ; + }ALLOC_GROW (heads, nr_heads +1 , nr_alloc); heads[nr_heads++] = rm; }@@ -1245,7 +1263,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) heads[nr_heads++] = rm; }
transport.c int transport_fetch_refs(struct transport *transport, struct ref *refs) · line 1273
Quoted 10 lines
transport.c +5
-1- rc = transport->vtable->fetch (transport, nr_heads, heads); + rc = transport->vtable->fetch (transport, nr_heads, heads, fetched_refs); +if (fetched_refs && nop_head) { + *nop_tail = *fetched_refs; + *fetched_refs = nop_head; + }free (heads);return rc;
And sometimes, even if we are merely simulating the fetching of refs, we still need to report those refs in fetched_refs. This is correct.
I also see that t5703 now passes.
Besides enabling the writing of subsequent patches, I see that this also makes the API clearer in that the input refs to transport_fetch_refs() are not overloaded to output shallow information. Other than the " = 0" change above, this patch looks good to me.
Re: [PATCH v3 7/8] fetch-pack: put shallow info in output parameter
Quoted 95 lines · Jonathan Tan
fetch-pack.c static int remove_duplicates_in_refs(struct ref **ref, int nr) · line 1473
Quoted 3 lines
static void update_shallow(struct
fetch_pack_args*args,fetch-pack.c +1
-1- struct ref **sought, int nr_sought, + struct ref *refs,
update_shallow()now takes in a linked list of refs instead of an array. I see that the translation of this function is straightforward - occasionally, we need to iterate through the linked list and count up from 0 at the same time, but that is not a problem.fetch-pack.c static int remove_duplicates_in_refs(struct ref **ref, int nr) · line 1478
Quoted 6 lines
fetch-pack.c +1
-1struct shallow_info *si) {struct oid_array ref =OID_ARRAY_INIT ;int *status; -int i; +int i =0 ;Remove the " = 0" - I've verified that it does not need to be there, and it might inhibit useful "unintialized variable" warnings if others were to change the code later.
Optional: I would also remove this declaration and declare "int i;" in each of the blocks that need it.
Quoted 11 lines
static int fetch_refs_via_pack(struct transport *transport,
transport.c +2
-1- int nr_heads, struct ref **to_fetch) + int nr_heads, struct ref **to_fetch, + struct ref **fetched_refs) { int ret = 0;struct git_transport_data *data = transport->data;@@ -354,8 +356,12 @@ static int fetch_refs_via_pack(struct transport *transport,if (report_unmatched_refs (to_fetch, nr_heads)) ret =-1 ;transport.c static int fetch_refs_via_pack(struct transport *transport, · line 367
Quoted 10 lines
transport.c +5
-1+if (fetched_refs) + *fetched_refs = refs; +else +free_refs (refs); +free_refs (refs_tmp); -free_refs (refs);free (dest);return ret; }Instead of just freeing the linked list, we return it if requested by the client. This makes sense.
Quoted 33 lines
transport.c +14
-2-int transport_fetch_refs (struct transport *transport,struct ref *refs) +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs) {int rc;int nr_heads =0 , nr_alloc =0 , nr_refs =0 ;struct ref **heads =NULL ; +struct ref *nop_head =NULL , **nop_tail = &nop_head;struct ref *rm;for (rm = refs; rm; rm = rm->next) { nr_refs++;if (rm->peer_ref && !is_null_oid (&rm->old_oid) && - !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) + !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don't +* actually need to fetch them. +*/ +if (fetched_refs) { +struct ref *nop_ref =copy_ref (rm); + *nop_tail = nop_ref; + nop_tail = &nop_ref->next; + }continue ; + }ALLOC_GROW (heads, nr_heads +1 , nr_alloc); heads[nr_heads++] = rm; }@@ -1245,7 +1263,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) heads[nr_heads++] = rm; }transport.c int transport_fetch_refs(struct transport *transport, struct ref *refs) · line 1273
Quoted 10 lines
transport.c +5
-1- rc = transport->vtable->fetch (transport, nr_heads, heads); + rc = transport->vtable->fetch (transport, nr_heads, heads, fetched_refs); +if (fetched_refs && nop_head) { + *nop_tail = *fetched_refs; + *fetched_refs = nop_head; + }free (heads);return rc;And sometimes, even if we are merely simulating the fetching of refs, we still need to report those refs in
fetched_refs. This is correct.
Quoted 6 lines
I also see that t5703 now passes.
Besides enabling the writing of subsequent patches, I see that this also makes the API clearer in that the input refs to
transport_fetch_refs()are not overloaded to output shallow information. Other than the " = 0" change above, this patch looks good to me.
Perfect, I'll just drop the " = 0" part (making the diff slightly smaller)
Signature · 1 line
Brandon Williams
PATCH v3 8/8 fetch-pack: implement ref-in-want
In reply to Brandon WilliamsImplement ref-in-want on the client side so that when a server supports the "ref-in-want" feature, a client will send "want-ref" lines for each reference the client wants to fetch.
--- fetch-pack.c | 35 +++++++++++++++++++++++++++--- remote.c | 1 + remote.h | 1 + t/t5703-upload-pack-ref-in-want.sh | 4 ++-- 4 files changed, 36 insertions(+), 5 deletions(-)
fetch-pack.c +32 -3
@@ -1102,9 +1102,10 @@ static void add_shallow_requests(struct strbuf *req_buf,static void add_wants (const struct ref *wants,struct strbuf *req_buf) { +int use_ref_in_want =server_supports_feature ("fetch" ,"ref-in-want" ,0 ); +for ( ; wants ; wants = wants->next) {const struct object_id *remote = &wants->old_oid; -const char *remote_hex;struct object *o;/*
@@ -1122,8 +1123,10 @@ static void add_wants(const struct ref *wants, struct strbuf *req_buf)continue ; } - remote_hex =oid_to_hex (remote); -packet_buf_write (req_buf,"want %s\n" , remote_hex); +if (!use_ref_in_want || wants->exact_oid) +packet_buf_write (req_buf,"want %s\n" ,oid_to_hex (remote)); +else +packet_buf_write (req_buf,"want-ref %s\n" , wants->name); } }
@@ -1334,6 +1337,29 @@ static void receive_shallow_info(struct fetch_pack_args *args, args->deepen =1 ; } +static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line); + +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + } + } + +if (reader->status !=PACKET_READ_DELIM ) +die ("error processing wanted refs: %d" , reader->status); +} +enum fetch_state {FETCH_CHECK_LOCAL =0 ,FETCH_SEND_REQUEST ,
@@ -1408,6 +1434,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,if (process_section_header (&reader,"shallow-info" ,1 ))receive_shallow_info (args, &reader); +if (process_section_header (&reader,"wanted-refs" ,1 )) +receive_wanted_refs (&reader, ref); +/* get the pack */ process_section_header (&reader,"packfile" ,0 );if (get_pack (args, fd, pack_lockfile))
remote.c +1 -0
@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_oid =1 ; }else { ref_map =get_remote_ref (remote_refs, name); }
remote.h +1 -0
@@ -73,6 +73,7 @@ struct ref { force:1, forced_update:1, expect_old_sha1:1, + exact_oid:1, deletion:1;enum {
t/t5703-upload-pack-ref-in-want.sh +2 -2
@@ -204,7 +204,7 @@ test_expect_success 'server is initially ahead - no ref in want' ' grep "ERR upload-pack: not our ref" err ' -test_expect_failure 'server is initially ahead - ref in want' ' +test_expect_success 'server is initially ahead - ref in want' ' git -C "$REPO" config uploadpack.allowRefInWant true && rm -rf local && cp -r "$LOCAL_PRISTINE" local &&
@@ -228,7 +228,7 @@ test_expect_success 'server is initially behind - no ref in want' ' test_cmp expected actual ' -test_expect_failure 'server is initially behind - ref in want' ' +test_expect_success 'server is initially behind - ref in want' ' git -C "$REPO" config uploadpack.allowRefInWant true && rm -rf local && cp -r "$LOCAL_PRISTINE" local &&
Signature · 1 line
2.18.0.rc1.244.gcf134e6275-goog
Re: [PATCH v3 8/8] fetch-pack: implement ref-in-want
Hi,
Quoted 11 lines · Brandon Williams
Implement ref-in-want on the client side so that when a server supports the "ref-in-want" feature, a client will send "want-ref" lines for each reference the client wants to fetch.
Signed-off-by Brandon Williams <[email protected]>--- fetch-pack.c | 35 +++++++++++++++++++++++++++--- remote.c | 1 + remote.h | 1 + t/t5703-upload-pack-ref-in-want.sh | 4 ++-- 4 files changed, 36 insertions(+), 5 deletions(-)
This commit message doesn't tell me what ref-in-want is or is for. Could it include
A. a pointer to Documentation/technical/protocol-v2.txt, or B. an example illustrating the effect e.g. using GIT_TRACE_PACKET
or both?
[...]
fetch-pack.c static void add_shallow_requests(struct strbuf *req_buf, · line 1107
Quoted 9 lines
fetch-pack.c +2
-0@@ -1102,9 +1102,10 @@ static void add_shallow_requests(struct strbuf *req_buf,static void add_wants (const struct ref *wants,struct strbuf *req_buf) { +int use_ref_in_want =server_supports_feature ("fetch" ,"ref-in-want" ,0 ); +for ( ; wants ; wants = wants->next) {
Not about this patch: it's kind of confusing that the iterator is called 'wants' even though it points into the middle of the list. I would even be tempted to do
const struct ref *want;
for (want = wants; want; want = want->next) {It wouldn't make sense to do in this patch, though.
[...]
fetch-pack.c static void add_wants(const struct ref *wants, struct strbuf *req_buf) · line 1129
Quoted 10 lines
fetch-pack.c +4
-2@@ -1122,8 +1123,10 @@ static void add_wants(const struct ref *wants, struct strbuf *req_buf)continue ; } - remote_hex =oid_to_hex (remote); -packet_buf_write (req_buf,"want %s\n" , remote_hex); +if (!use_ref_in_want || wants->exact_oid) +packet_buf_write (req_buf,"want %s\n" ,oid_to_hex (remote)); +else +packet_buf_write (req_buf,"want-ref %s\n" , wants->name);
Very neat.
[...]
Quoted 10 lines
fetch-pack.c +6
-0@@ -1334,6 +1337,29 @@ static void receive_shallow_info(struct fetch_pack_args *args, args->deepen =1 ; } +static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end;
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1355
Quoted 10 lines
fetch-pack.c +10
-0+struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line); + +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + }
Stefan mentioned that the spec says
* The server MUST NOT send any refs which were not requested using 'want-ref' lines.
Can client enforce that? If not, can the spec say SHOULD NOT for the server and add a MUST describing appropriate client behavior?
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1359
Quoted 4 lines
fetch-pack.c +4
-0+ } + } + +if (reader->status !=PACKET_READ_DELIM )
The spec says
* This section is only included if the client has requested a ref using a 'want-ref' line and if a packfile section is also included in the response.
What should happen if the client already has all the relevant objects (or in other words if there is no packfile to send in the packfile section)? Is the idea that the client should already have known that based on the ref advertisement? What if ref values change to put us in that state between the ls-refs and fetch steps?
[...]
remote.c int get_fetch_map(const struct ref *remote_refs, · line 1738
Quoted 7 lines
remote.c +1
-0@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_oid =1 ;
Sensible. The alternative would be that we check whether the refname is oid-shaped at want-ref generation time, which would be unnecessarily complicated.
[...]
remote.h struct ref {
Quoted 3 lines
remote.h +0
-0@@ -73,6 +73,7 @@ struct ref {
Not about this patch: why is this in remote.h instead of ref.h?
remote.h struct ref { · line 77
Quoted 5 lines
remote.h +1
-0force:1, forced_update:1, expect_old_sha1:1, + exact_oid:1, deletion:1;
Looks good, and we have room for plenty more bits there.
[...]
remote.h struct ref { · line 78
Quoted 1 line
remote.h +1
-0+++ b/t/t5703-upload-pack-ref-in-want.sh
Nice.
Thanks for a pleasant read.
Sincerely,
Jonathan
Re: [PATCH v3 8/8] fetch-pack: implement ref-in-want
Quoted 16 lines · Jonathan Nieder
Hi,
Quoted 11 lines · Brandon Williams
Implement ref-in-want on the client side so that when a server supports the "ref-in-want" feature, a client will send "want-ref" lines for each reference the client wants to fetch.
Signed-off-by Brandon Williams <[email protected]>--- fetch-pack.c | 35 +++++++++++++++++++++++++++--- remote.c | 1 + remote.h | 1 + t/t5703-upload-pack-ref-in-want.sh | 4 ++-- 4 files changed, 36 insertions(+), 5 deletions(-)
Quoted 7 lines
This commit message doesn't tell me what ref-in-want is or is for. Could it include
A. a pointer to
Documentation/technical/protocol-v2.txt, or B. an example illustrating the effect e.g. usingGIT_TRACE_PACKETor both?
Yeah I can imporve the message here.
Quoted 9 lines
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1355
Quoted 6 lines
fetch-pack.c +6
-0+ +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + }Stefan mentioned that the spec says
Quoted 5 lines
* The server MUST NOT send any refs which were not requested using 'want-ref' lines.
Can client enforce that? If not, can the spec say SHOULD NOT for the server and add a MUST describing appropriate client behavior?
Yeah I can update the docs in an earlier patch.
Quoted 12 lines
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1359
Quoted 5 lines
fetch-pack.c +4
-0+ } + } + +if (reader->status !=PACKET_READ_DELIM )The spec says
* This section is only included if the client has requested a ref using a 'want-ref' line and if a packfile section is also included in the response.
Quoted 5 lines
What should happen if the client already has all the relevant objects (or in other words if there is no packfile to send in the packfile section)? Is the idea that the client should already have known that based on the ref advertisement? What if ref values change to put us in that state between the ls-refs and fetch steps?
I believe the current functionality is that if all wants are already satisfied by all haves then an empty packfile is sent, so that would fall under that case.
Signature · 1 line
Brandon Williams
v4 cover ref-in-want
In reply to Brandon WilliamsChanges in v4 are fairly minor. There are a few documentation changes, commit message updates, as well as a few small style tweaks based on reviewer comments.
Brandon Williams (8):
test-pkt-line: add unpack-sideband subcommand
upload-pack: implement ref-in-want
upload-pack: test negotiation with changing repository
fetch: refactor the population of peer ref OIDs
fetch: refactor fetch_refs into two functions
fetch: refactor to make function args narrower
fetch-pack: put shallow info in output parameter
fetch-pack: implement ref-in-want
Documentation/config.txt | 7 + Documentation/technical/protocol-v2.txt | 28 ++- builtin/clone.c | 4 +- builtin/fetch.c | 135 ++++++++----- fetch-object.c | 2 +- fetch-pack.c | 50 ++++- remote.c | 1 + remote.h | 1 + t/helper/test-pkt-line.c | 33 ++++ t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 + t/lib-httpd/one-time-sed.sh | 16 ++ t/t5703-upload-pack-ref-in-want.sh | 245 ++++++++++++++++++++++++ transport-helper.c | 6 +- transport-internal.h | 9 +- transport.c | 34 +++- transport.h | 3 +- upload-pack.c | 66 +++++++ 18 files changed, 574 insertions(+), 75 deletions(-) create mode 100644 t/lib-httpd/one-time-sed.sh create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v4 1/8 test-pkt-line: add unpack-sideband subcommand
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
--- t/helper/test-pkt-line.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
t/helper/test-pkt-line.c +33 -0
@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h" static void pack_line (const char *line)
@@ -48,6 +49,36 @@ static void unpack(void) } } +static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE ); + +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) { +case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ; +if (band <1 || band >2 ) +die ("unexpected side band %d" , band); + fd = band; + +write_or_die (fd, reader.line +1 , reader.pktlen -1 ); +break ; +case PACKET_READ_FLUSH : +return ; +case PACKET_READ_DELIM : +break ; + } + } +} +int cmd_main (int argc,const char **argv) {if (argc <2 )
@@ -57,6 +88,8 @@ int cmd_main(int argc, const char **argv)pack (argc -2 , argv +2 ); else if (!strcmp(argv[1], "unpack")) unpack(); + else if (!strcmp(argv[1], "unpack-sideband")) + unpack_sideband(); else die("invalid argument '%s'", argv[1]);
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v4 2/8 upload-pack: implement ref-in-want
In reply to Brandon WilliamsCurrently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement and there exists replication delay.
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
--- Documentation/config.txt | 7 ++ Documentation/technical/protocol-v2.txt | 28 ++++- t/t5703-upload-pack-ref-in-want.sh | 153 ++++++++++++++++++++++++ upload-pack.c | 66 ++++++++++ 4 files changed, 253 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Documentation/config.txt +7 -0
@@ -3479,6 +3479,13 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories). +uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. This feature + is intended for the benefit of load-balanced servers which may + not have the same view of what OIDs their refs point to due to + replication delay. + url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves a
Documentation/technical/protocol-v2.txt +27 -1
@@ -299,12 +299,21 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values. +If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. output = *section - section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt) acknowledgments = PKT-LINE("acknowledgments" LF)
@@ -319,6 +328,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff)
@@ -379,6 +392,19 @@ header. * This section is only included if a packfile section is also included in the response. + wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs" + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server SHOULD NOT send any refs which were not requested + using 'want-ref' lines. + packfile section * This section is only included if the client has sent 'want' lines in its request and either requested that no more
t/t5703-upload-pack-ref-in-want.sh new file +153 -0
153 lines not on this page · raw message
upload-pack.c +66 -0
138 lines not on this page · raw message
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v4 3/8 upload-pack: test negotiation with changing repository
In reply to Brandon WilliamsAdd tests to check the behavior of fetching from a repository which changes between rounds of negotiation (for example, when different servers in a load-balancing agreement participate in the same stateless RPC negotiation). This forms a baseline of comparison to the ref-in-want functionality (which will be introduced to the client in subsequent commits), and ensures that subsequent commits do not change existing behavior.
As part of this effort, a mechanism to substitute strings in a single HTTP response is added.
--- t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 +++ t/lib-httpd/one-time-sed.sh | 16 ++++++ t/t5703-upload-pack-ref-in-want.sh | 92 ++++++++++++++++++++++++++++++ 4 files changed, 117 insertions(+) create mode 100644 t/lib-httpd/one-time-sed.sh
t/lib-httpd.sh +1 -0
@@ -132,6 +132,7 @@ prepare_httpd() { cp "$TEST_PATH"/passwd "$HTTPD_ROOT_PATH" install_script broken-smart-http.sh install_script error.sh + install_script one-time-sed.sh ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +8 -0
@@ -111,9 +111,14 @@ Alias /auth/dumb/ www/auth/dumb/ SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} SetEnv GIT_HTTP_EXPORT_ALL </LocationMatch> +<LocationMatch /one_time_sed/> + SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} + SetEnv GIT_HTTP_EXPORT_ALL +</LocationMatch> ScriptAliasMatch /smart_*[^/]*/(.*) ${GIT_EXEC_PATH}/git-http-backend/$1 ScriptAlias /broken_smart/ broken-smart-http.sh/ ScriptAlias /error/ error.sh/ +ScriptAliasMatch /one_time_sed/(.*) one-time-sed.sh/$1 <Directory ${GIT_EXEC_PATH}> Options FollowSymlinks </Directory>
@@ -123,6 +128,9 @@ ScriptAlias /error/ error.sh/ <Files error.sh> Options ExecCGI </Files> +<Files one-time-sed.sh> + Options ExecCGI +</Files> <Files ${GIT_EXEC_PATH}/git-http-backend> Options ExecCGI </Files>
t/lib-httpd/one-time-sed.sh new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/sh + +if [ -e one-time-sed ]; then + "$GIT_EXEC_PATH/git-http-backend" >out + + sed "$(cat one-time-sed)" <out >out_modified + + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fi
t/t5703-upload-pack-ref-in-want.sh +92 -0
96 lines not on this page · raw message
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
Re: [PATCH v4 3/8] upload-pack: test negotiation with changing repository
Quoted 9 lines
t/lib-httpd/one-time-sed.sh new file +6
-0@@ -0,0 +1,16 @@ +#!/bin/sh + +if [ -e one-time-sed ]; then + "$GIT_EXEC_PATH/git-http-backend" >out + + sed "$(cat one-time-sed)" <out >out_modified
t/lib-httpd/one-time-sed.sh line 16
Quoted 10 lines
t/lib-httpd/one-time-sed.sh new file +10
-0+ + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fi
Add an explanatory comment somewhere (maybe, at the beginning), something like:
If "one-time-sed" exists in $HTTPD_ROOT_PATH, run sed on the HTTP response, using the contents of "one-time-sed" as the sed command to be run. If the response was modified as a result, delete "one-time-sed" so that subsequent HTTP responses are no longer modified.
This can be used to simulate the effects of the repository changing in between HTTP request-response pairs.
Quoted 1 line
+test_expect_failure 'server is initially ahead - ref in want' '
[snip]
Quoted 1 line
+test_expect_failure 'server is initially behind - ref in want' '
[snip]
These are test_expect_failure, I assume because the fetch part has not been implemented yet. Can this be moved to the end of the patch set, once the fetch part has been implemented?
There's also the case of when the server initially has a ref but later does not - can this be tested here too? The server part is already covered by the upload-pack test in which we craft a request with a non-existent ref, but it would be good to test the client part too.
PATCH v4 4/8 fetch: refactor the population of peer ref OIDs
In reply to Brandon WilliamsPopulate peer ref OIDs in get_ref_map instead of do_fetch. Besides tightening scopes of variables in the code, this also prepares for get_ref_map being able to be called multiple times within do_fetch.
--- builtin/fetch.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
builtin/fetch.c +18 -18
@@ -351,6 +351,7 @@ static struct ref *get_ref_map(struct transport *transport,/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs; +struct string_list existing_refs =STRING_LIST_INIT_DUP ;const struct ref *remote_refs;if (rs->nr)
@@ -458,7 +459,23 @@ static struct ref *get_ref_map(struct transport *transport, tail = &rm->next; } -return ref_remove_duplicates (ref_map); + ref_map =ref_remove_duplicates (ref_map); + +for_each_ref (add_existing, &existing_refs); +for (rm = ref_map; rm; rm = rm->next) { +if (rm->peer_ref) { +struct string_list_item *peer_item = +string_list_lookup (&existing_refs, + rm->peer_ref->name); +if (peer_item) { +struct object_id *old_oid = peer_item->util; +oidcpy (&rm->peer_ref->old_oid, old_oid); + } + } + } +string_list_clear (&existing_refs,1 ); + +return ref_map; }#define STORE_REF_ERROR_OTHER 1
@@ -1110,14 +1127,10 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)static int do_fetch (struct transport *transport,struct refspec *rs) { -struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct ref *ref_map; -struct ref *rm;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; -for_each_ref (add_existing, &existing_refs); -if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 ) tags =TAGS_SET ;
@@ -1136,18 +1149,6 @@ static int do_fetch(struct transport *transport,if (!update_head_ok)check_not_current_branch (ref_map); -for (rm = ref_map; rm; rm = rm->next) { -if (rm->peer_ref) { -struct string_list_item *peer_item = -string_list_lookup (&existing_refs, - rm->peer_ref->name); -if (peer_item) { -struct object_id *old_oid = peer_item->util; -oidcpy (&rm->peer_ref->old_oid, old_oid); - } - } - } -if (tags ==TAGS_DEFAULT && autotags)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,"1" );if (prune) {
@@ -1183,7 +1184,6 @@ static int do_fetch(struct transport *transport, } cleanup: -string_list_clear (&existing_refs,1 );return retcode; }
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v4 5/8 fetch: refactor fetch_refs into two functions
In reply to Brandon WilliamsRefactor the fetch_refs function into a function that does the fetching of refs and another function that stores them. This is in preparation for allowing additional processing of the fetched refs before updating the local ref store.
--- builtin/fetch.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-)
builtin/fetch.c +18 -5
@@ -968,9 +968,21 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)if (ret) ret =transport_fetch_refs (transport, ref_map);if (!ret) - ret |=store_updated_refs (transport->url, - transport->remote->name, - ref_map); +/* +* Keep the new pack's ".keep" file around to allow the caller +* time to update refs to reference the new objects. +*/ +return 0 ; +transport_unlock_pack (transport); +return ret; +} + +/* Update local refs based on the ref values fetched from a remote */ +static int consume_refs (struct transport *transport,struct ref *ref_map) +{ +int ret =store_updated_refs (transport->url, + transport->remote->name, + ref_map);transport_unlock_pack (transport);return ret; }
@@ -1116,7 +1128,8 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -fetch_refs (transport, ref_map); +if (!fetch_refs (transport, ref_map)) +consume_refs (transport, ref_map);if (gsecondary) {transport_disconnect (gsecondary);
@@ -1165,7 +1178,7 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map)) { +if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v4 6/8 fetch: refactor to make function args narrower
In reply to Brandon WilliamsRefactor find_non_local_tags and get_ref_map to only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
--- builtin/fetch.c | 52 ++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
builtin/fetch.c +26 -26
@@ -254,9 +254,9 @@ static int will_fetch(struct ref **head, const unsigned char *sha1)return 0 ; } -static void find_non_local_tags (struct transport *transport, -struct ref **head, -struct ref ***tail) +static void find_non_local_tags (const struct ref *refs, +struct ref **head, +struct ref ***tail) {struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct string_list remote_refs =STRING_LIST_INIT_NODUP ;
@@ -264,7 +264,7 @@ static void find_non_local_tags(struct transport *transport,struct string_list_item *item =NULL ;for_each_ref (add_existing, &existing_refs); -for (ref =transport_get_remote_refs (transport,NULL ); ref; ref = ref->next) { +for (ref = refs; ref; ref = ref->next) {if (!starts_with (ref->name,"refs/tags/" ))continue ;
@@ -338,7 +338,8 @@ static void find_non_local_tags(struct transport *transport,string_list_clear (&remote_refs,0 ); } -static struct ref *get_ref_map (struct transport *transport, +static struct ref *get_ref_map (struct remote *remote, +const struct ref *remote_refs,struct refspec *rs,int tags,int *autotags) {
@@ -346,27 +347,11 @@ static struct ref *get_ref_map(struct transport *transport,struct ref *rm;struct ref *ref_map =NULL ;struct ref **tail = &ref_map; -struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs;struct string_list existing_refs =STRING_LIST_INIT_DUP ; -const struct ref *remote_refs; - -if (rs->nr) -refspec_ref_prefixes (rs, &ref_prefixes); -else if (transport->remote && transport->remote->fetch.nr) -refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); - -if (ref_prefixes.argc && - (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { -argv_array_push (&ref_prefixes,"refs/tags/" ); - } - - remote_refs =transport_get_remote_refs (transport, &ref_prefixes); - -argv_array_clear (&ref_prefixes);if (rs->nr) {struct refspec *fetch_refspec;
@@ -403,7 +388,7 @@ static struct ref *get_ref_map(struct transport *transport,if (refmap.nr) fetch_refspec = &refmap;else - fetch_refspec = &transport->remote->fetch; + fetch_refspec = &remote->fetch;for (i =0 ; i < fetch_refspec->nr; i++)get_fetch_map (ref_map, &fetch_refspec->items[i], &oref_tail,1 );
@@ -411,7 +396,6 @@ static struct ref *get_ref_map(struct transport *transport,die ("--refmap option is only meaningful with command-line refspec(s)." ); }else {/* Use the defaults */ -struct remote *remote = transport->remote;struct branch *branch =branch_get (NULL );int has_merge =branch_has_merge_config (branch); if (remote &&
@@ -450,7 +434,7 @@ static struct ref *get_ref_map(struct transport *transport,/* also fetch all tags */ get_fetch_map (remote_refs, tag_refspec, &tail,0 ); else if (tags == TAGS_DEFAULT && *autotags) - find_non_local_tags(transport, &ref_map, &tail); + find_non_local_tags(remote_refs, &ref_map, &tail);/* Now append any refs to be updated opportunistically: */ *tail = orefs;
@@ -1143,6 +1127,8 @@ static int do_fetch(struct transport *transport,struct ref *ref_map;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; +const struct ref *remote_refs; +struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 )
@@ -1158,7 +1144,21 @@ static int do_fetch(struct transport *transport, goto cleanup; } - ref_map =get_ref_map (transport, rs, tags, &autotags); +if (rs->nr) +refspec_ref_prefixes (rs, &ref_prefixes); +else if (transport->remote && transport->remote->fetch.nr) +refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); + +if (ref_prefixes.argc && + (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { +argv_array_push (&ref_prefixes,"refs/tags/" ); + } + + remote_refs =transport_get_remote_refs (transport, &ref_prefixes); +argv_array_clear (&ref_prefixes); + + ref_map =get_ref_map (transport->remote, remote_refs, rs, + tags, &autotags);if (!update_head_ok)check_not_current_branch (ref_map);
@@ -1190,7 +1190,7 @@ static int do_fetch(struct transport *transport,if (tags ==TAGS_DEFAULT && autotags) {struct ref **tail = &ref_map; ref_map =NULL ; -find_non_local_tags (transport, &ref_map, &tail); +find_non_local_tags (remote_refs, &ref_map, &tail);if (ref_map)backfill_tags (transport, ref_map);free_refs (ref_map);
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
Re: [PATCH v4 6/8] fetch: refactor to make function args narrower
Quoted 5 lines
Refactor
find_non_local_tagsandget_ref_mapto only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
I see that due to the narrowing of get_ref_map() to take the refs (and the remote) instead of the whole transport, the computation of the refs (including computation of the ref prefixes) is also moved from get_ref_map() to its caller, do_fetch(). As in a previous patch, get_ref_map() is only used once, so this movement is safe.
PATCH v4 7/8 fetch-pack: put shallow info in output parameter
In reply to Brandon WilliamsExpand the transport fetch method signature, by adding an output parameter, to allow transports to return information about the refs they have fetched. Then communicate shallow status information through this mechanism instead of by modifying the input list of refs.
This does require clients to sometimes generate the ref map twice: once from the list of refs provided by the remote (as is currently done) and potentially once from the new list of refs that the fetch mechanism provides.
--- builtin/clone.c | 4 ++-- builtin/fetch.c | 28 ++++++++++++++++++++++++---- fetch-object.c | 2 +- fetch-pack.c | 15 ++++++++------- transport-helper.c | 6 ++++-- transport-internal.h | 9 ++++++++- transport.c | 34 ++++++++++++++++++++++++++++------ transport.h | 3 ++- 8 files changed, 77 insertions(+), 24 deletions(-)
builtin/clone.c +2 -2
@@ -1155,7 +1155,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (!is_local && !complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); remote_head =find_ref_by_name (refs,"HEAD" ); remote_head_points_at =
@@ -1197,7 +1197,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)if (is_local)clone_local (path, git_dir);else if (refs && complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); update_remote_refs(refs, mapped_refs, remote_head_points_at, branch_top.buf, reflog_msg.buf, transport,
builtin/fetch.c +24 -4
@@ -946,11 +946,13 @@ static int quickfetch(struct ref *ref_map)return check_connected (iterate_ref_map, &rm, &opt); } -static int fetch_refs (struct transport *transport,struct ref *ref_map) +static int fetch_refs (struct transport *transport,struct ref *ref_map, +struct ref **updated_remote_refs) {int ret =quickfetch (ref_map);if (ret) - ret =transport_fetch_refs (transport, ref_map); + ret =transport_fetch_refs (transport, ref_map, + updated_remote_refs);if (!ret)/* * Keep the new pack's ".keep" file around to allow the caller
@@ -1112,7 +1114,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -if (!fetch_refs (transport, ref_map)) +if (!fetch_refs (transport, ref_map,NULL ))consume_refs (transport, ref_map);if (gsecondary) {
@@ -1128,6 +1130,7 @@ static int do_fetch(struct transport *transport,int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ;const struct ref *remote_refs; +struct ref *updated_remote_refs =NULL ;struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {
@@ -1178,7 +1181,24 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) { + +if (fetch_refs (transport, ref_map, &updated_remote_refs)) { +free_refs (ref_map); + retcode =1 ; + goto cleanup; + } +if (updated_remote_refs) { +/* +* Regenerate ref_map using the updated remote refs. This is +* to account for additional information which may be provided +* by the transport (e.g. shallow info). +*/ +free_refs (ref_map); + ref_map =get_ref_map (transport->remote, updated_remote_refs, rs, + tags, &autotags); +free_refs (updated_remote_refs); + } +if (consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
fetch-object.c +1 -1
@@ -19,7 +19,7 @@ static void fetch_refs(const char *remote_name, struct ref *ref)transport_set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" );transport_set_option (transport,TRANS_OPT_NO_DEPENDENTS ,"1" ); -transport_fetch_refs (transport, ref); +transport_fetch_refs (transport, ref,NULL ); fetch_if_missing = original_fetch_if_missing; }
fetch-pack.c +8 -7
@@ -1470,12 +1470,13 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr) }static void update_shallow (struct fetch_pack_args *args, -struct ref **sought,int nr_sought, +struct ref *refs,struct shallow_info *si) {struct oid_array ref =OID_ARRAY_INIT ;int *status;int i; +struct ref *r;if (args->deepen && alternate_shallow_file) {if (*alternate_shallow_file =='\0' ) {/* --unshallow */
@@ -1517,8 +1518,8 @@ static void update_shallow(struct fetch_pack_args *args,remove_nonexistent_theirs_shallow (si);if (!si->nr_ours && !si->nr_theirs)return ; -for (i =0 ; i < nr_sought; i++) -oid_array_append (&ref, &sought[i]->old_oid); +for (r = refs; r; r = r->next) +oid_array_append (&ref, &r->old_oid); si->ref = &ref;if (args->update_shallow) {
@@ -1552,12 +1553,12 @@ static void update_shallow(struct fetch_pack_args *args,* remote is also shallow, check what ref is safe to update * without updating .git/shallow */ - status =xcalloc (nr_sought,sizeof (*status)); + status =xcalloc (ref.nr,sizeof (*status));assign_shallow_commits_to_refs (si,NULL , status);if (si->nr_ours || si->nr_theirs) { -for (i =0 ; i < nr_sought; i++) +for (r = refs, i =0 ; r; r = r->next, i++)if (status[i]) - sought[i]->status =REF_STATUS_REJECT_SHALLOW ; + r->status =REF_STATUS_REJECT_SHALLOW ; }free (status);oid_array_clear (&ref);
@@ -1591,7 +1592,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args, ref_cpy =do_fetch_pack (args, fd, ref, sought, nr_sought, &si, pack_lockfile);reprepare_packed_git (the_repository); -update_shallow (args, sought, nr_sought, &si); +update_shallow (args, ref_cpy, &si);clear_shallow_info (&si);return ref_cpy; }
transport-helper.c +4 -2
@@ -651,14 +651,16 @@ static int connect_helper(struct transport *transport, const char *name, }static int fetch (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct helper_data *data = transport->data;int i, count;if (process_connect (transport,0 )) {do_take_over (transport); -return transport->vtable->fetch (transport, nr_heads, to_fetch); +return transport->vtable->fetch (transport, nr_heads, to_fetch, + fetched_refs); } count =0 ;
transport-internal.h +8 -1
@@ -36,11 +36,18 @@ struct transport_vtable {* Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. * +* The transport *may* provide, in fetched_refs, the list of refs that +* it fetched. If the transport knows anything about the fetched refs +* that the caller does not know (for example, shallow status), it +* should provide that list of refs and include that information in the +* list. +* * If the transport did not get hashes for refs in * get_refs_list(), it should set the old_sha1 fields in the * provided refs now. **/ -int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs); +int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs, +struct ref **fetched_refs);/** * Push the objects and refs. Send the necessary objects, and
transport.c +28 -6
@@ -151,7 +151,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport, }static int fetch_refs_from_bundle (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct bundle_transport_data *data = transport->data; return unbundle(&data->header, data->fd,
@@ -287,7 +288,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus }static int fetch_refs_via_pack (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -354,8 +356,12 @@ static int fetch_refs_via_pack(struct transport *transport,if (report_unmatched_refs (to_fetch, nr_heads)) ret =-1 ; +if (fetched_refs) + *fetched_refs = refs; +else +free_refs (refs); +free_refs (refs_tmp); -free_refs (refs);free (dest);return ret; }
@@ -1215,19 +1221,31 @@ const struct ref *transport_get_remote_refs(struct transport *transport,return transport->remote_refs; } -int transport_fetch_refs (struct transport *transport,struct ref *refs) +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs) {int rc;int nr_heads =0 , nr_alloc =0 , nr_refs =0 ;struct ref **heads =NULL ; +struct ref *nop_head =NULL , **nop_tail = &nop_head;struct ref *rm;for (rm = refs; rm; rm = rm->next) { nr_refs++;if (rm->peer_ref && !is_null_oid (&rm->old_oid) && - !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) + !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don't +* actually need to fetch them. +*/ +if (fetched_refs) { +struct ref *nop_ref =copy_ref (rm); + *nop_tail = nop_ref; + nop_tail = &nop_ref->next; + }continue ; + }ALLOC_GROW (heads, nr_heads +1 , nr_alloc); heads[nr_heads++] = rm; }
@@ -1245,7 +1263,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) heads[nr_heads++] = rm; } - rc = transport->vtable->fetch (transport, nr_heads, heads); + rc = transport->vtable->fetch (transport, nr_heads, heads, fetched_refs); +if (fetched_refs && nop_head) { + *nop_tail = *fetched_refs; + *fetched_refs = nop_head; + }free (heads);return rc;
transport.h +2 -1
@@ -218,7 +218,8 @@ int transport_push(struct transport *connection,const struct ref *transport_get_remote_refs (struct transport *transport,const struct argv_array *ref_prefixes); -int transport_fetch_refs (struct transport *transport,struct ref *refs); +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs);void transport_unlock_pack (struct transport *transport);int transport_disconnect (struct transport *transport);char *transport_anonymize_url (const char *url);
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v4 8/8 fetch-pack: implement ref-in-want
In reply to Brandon WilliamsImplement ref-in-want on the client side so that when a server supports the "ref-in-want" feature, a client will send "want-ref" lines for each reference the client wants to fetch. This feature allows clients to tolerate inconsistencies that exist when a remote repository's refs change during the course of negotiation.
This allows a client to request to request a particular ref without specifying the OID of the ref. This means that instead of hitting an error when a ref no longer points at the OID it did at the beginning of negotiation, negotiation can continue and the value of that ref will be sent at the termination of negotiation, just before a packfile is sent.
More information on the ref-in-want feature can be found in Documentation/technical/protocol-v2.txt.
--- fetch-pack.c | 35 +++++++++++++++++++++++++++--- remote.c | 1 + remote.h | 1 + t/t5703-upload-pack-ref-in-want.sh | 4 ++-- 4 files changed, 36 insertions(+), 5 deletions(-)
fetch-pack.c +32 -3
@@ -1102,9 +1102,10 @@ static void add_shallow_requests(struct strbuf *req_buf,static void add_wants (const struct ref *wants,struct strbuf *req_buf) { +int use_ref_in_want =server_supports_feature ("fetch" ,"ref-in-want" ,0 ); +for ( ; wants ; wants = wants->next) {const struct object_id *remote = &wants->old_oid; -const char *remote_hex;struct object *o;/*
@@ -1122,8 +1123,10 @@ static void add_wants(const struct ref *wants, struct strbuf *req_buf)continue ; } - remote_hex =oid_to_hex (remote); -packet_buf_write (req_buf,"want %s\n" , remote_hex); +if (!use_ref_in_want || wants->exact_oid) +packet_buf_write (req_buf,"want %s\n" ,oid_to_hex (remote)); +else +packet_buf_write (req_buf,"want-ref %s\n" , wants->name); } }
@@ -1334,6 +1337,29 @@ static void receive_shallow_info(struct fetch_pack_args *args, args->deepen =1 ; } +static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line); + +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + } + } + +if (reader->status !=PACKET_READ_DELIM ) +die ("error processing wanted refs: %d" , reader->status); +} +enum fetch_state {FETCH_CHECK_LOCAL =0 ,FETCH_SEND_REQUEST ,
@@ -1408,6 +1434,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,if (process_section_header (&reader,"shallow-info" ,1 ))receive_shallow_info (args, &reader); +if (process_section_header (&reader,"wanted-refs" ,1 )) +receive_wanted_refs (&reader, ref); +/* get the pack */ process_section_header (&reader,"packfile" ,0 );if (get_pack (args, fd, pack_lockfile))
remote.c +1 -0
@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_oid =1 ; }else { ref_map =get_remote_ref (remote_refs, name); }
remote.h +1 -0
@@ -73,6 +73,7 @@ struct ref { force:1, forced_update:1, expect_old_sha1:1, + exact_oid:1, deletion:1;enum {
t/t5703-upload-pack-ref-in-want.sh +2 -2
@@ -204,7 +204,7 @@ test_expect_success 'server is initially ahead - no ref in want' ' grep "ERR upload-pack: not our ref" err ' -test_expect_failure 'server is initially ahead - ref in want' ' +test_expect_success 'server is initially ahead - ref in want' ' git -C "$REPO" config uploadpack.allowRefInWant true && rm -rf local && cp -r "$LOCAL_PRISTINE" local &&
@@ -228,7 +228,7 @@ test_expect_success 'server is initially behind - no ref in want' ' test_cmp expected actual ' -test_expect_failure 'server is initially behind - ref in want' ' +test_expect_success 'server is initially behind - ref in want' ' git -C "$REPO" config uploadpack.allowRefInWant true && rm -rf local && cp -r "$LOCAL_PRISTINE" local &&
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
Quoted 3 lines
Changes in v4 are fairly minor. There are a few documentation changes, commit message updates, as well as a few small style tweaks based on reviewer comments.
Patches 4 and 7, which I have commented on previously, look good.
As for patch 2, it still has a missing period in the documentation that I remarked upon in [1], but I'm not too worried about that. Having said that, Jonathan Nieder suggested [2]:
Quoted 7 lines
Stefan mentioned that the spec says
* The server MUST NOT send any refs which were not requested using 'want-ref' lines.
Can client enforce that? If not, can the spec say SHOULD NOT for the server and add a MUST describing appropriate client behavior?
I noticed that you did use "SHOULD NOT" instead of "MUST NOT" - in this case, you should probably also follow the second part about appropriate client behavior - it's probably best to document and implement that we ignore all unwanted refs. But considering this situation, though, I think it's better to just put "MUST NOT" and have the client enforce this.
One more thing - I think that the fetch part needs to be tested more. In particular, test cases similar to that of the upload-pack tests (multiple ref names, ref name + exact SHA-1), and in addition, handling of wildcards (for example, a wildcard that expands to nothing and a wildcard that expands to 2 refs).
v5 cover ref-in-want
In reply to Brandon WilliamsChanges in v5:
* Added a comment explaining the one-time-sed.sh script
* Added a number of tests per reviewer feedback
* Fixed a typo in documentation
Brandon Williams (8):
test-pkt-line: add unpack-sideband subcommand
upload-pack: implement ref-in-want
upload-pack: test negotiation with changing repository
fetch: refactor the population of peer ref OIDs
fetch: refactor fetch_refs into two functions
fetch: refactor to make function args narrower
fetch-pack: put shallow info in output parameter
fetch-pack: implement ref-in-want
Documentation/config.txt | 7 + Documentation/technical/protocol-v2.txt | 29 +- builtin/clone.c | 4 +- builtin/fetch.c | 135 +++++---- fetch-object.c | 2 +- fetch-pack.c | 50 +++- remote.c | 1 + remote.h | 1 + t/helper/test-pkt-line.c | 33 +++ t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 + t/lib-httpd/one-time-sed.sh | 22 ++ t/t5703-upload-pack-ref-in-want.sh | 351 ++++++++++++++++++++++++ transport-helper.c | 6 +- transport-internal.h | 9 +- transport.c | 34 ++- transport.h | 3 +- upload-pack.c | 66 +++++ 18 files changed, 687 insertions(+), 75 deletions(-) create mode 100644 t/lib-httpd/one-time-sed.sh create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v5 1/8 test-pkt-line: add unpack-sideband subcommand
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
--- t/helper/test-pkt-line.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
t/helper/test-pkt-line.c +33 -0
@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h" static void pack_line (const char *line)
@@ -48,6 +49,36 @@ static void unpack(void) } } +static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE ); + +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) { +case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ; +if (band <1 || band >2 ) +die ("unexpected side band %d" , band); + fd = band; + +write_or_die (fd, reader.line +1 , reader.pktlen -1 ); +break ; +case PACKET_READ_FLUSH : +return ; +case PACKET_READ_DELIM : +break ; + } + } +} +int cmd_main (int argc,const char **argv) {if (argc <2 )
@@ -57,6 +88,8 @@ int cmd_main(int argc, const char **argv)pack (argc -2 , argv +2 ); else if (!strcmp(argv[1], "unpack")) unpack(); + else if (!strcmp(argv[1], "unpack-sideband")) + unpack_sideband(); else die("invalid argument '%s'", argv[1]);
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v5 2/8 upload-pack: implement ref-in-want
In reply to Brandon WilliamsCurrently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement and there exists replication delay.
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
--- Documentation/config.txt | 7 ++ Documentation/technical/protocol-v2.txt | 29 ++++- t/t5703-upload-pack-ref-in-want.sh | 153 ++++++++++++++++++++++++ upload-pack.c | 66 ++++++++++ 4 files changed, 254 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Documentation/config.txt +7 -0
@@ -3479,6 +3479,13 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories). +uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. This feature + is intended for the benefit of load-balanced servers which may + not have the same view of what OIDs their refs point to due to + replication delay. + url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves a
Documentation/technical/protocol-v2.txt +28 -1
@@ -299,12 +299,21 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values. +If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. output = *section - section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt) acknowledgments = PKT-LINE("acknowledgments" LF)
@@ -319,6 +328,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff)
@@ -379,6 +392,20 @@ header. * This section is only included if a packfile section is also included in the response. + wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs". + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server SHOULD NOT send any refs which were not requested + using 'want-ref' lines and a client MUST ignore refs which + weren't requested. + packfile section * This section is only included if the client has sent 'want' lines in its request and either requested that no more
t/t5703-upload-pack-ref-in-want.sh new file +153 -0
153 lines not on this page · raw message
upload-pack.c +66 -0
138 lines not on this page · raw message
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
Re: [PATCH v5 2/8] upload-pack: implement ref-in-want
Documentation/technical/protocol-v2.txt header. · line 407
Quoted 13 lines · Brandon Williams
Documentation/technical/protocol-v2.txt +13
-0+ wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs". + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server SHOULD NOT send any refs which were not requested + using 'want-ref' lines and a client MUST ignore refs which + weren't requested.
Just being curious, but the above feels the other way around. Why are we being more lenient to writers of broken server than writers of broken clients? The number of installations they need to take back and replace is certainly lower for the former, which means that, if exchanges of unsoliclited refs are unwanted, clients should notice and barf (or warn) if the server misbehaves, and the server should be forbidden from sending unsolicited refs, no?
t/t5703-upload-pack-ref-in-want.sh line 7
Quoted 13 lines
t/t5703-upload-pack-ref-in-want.sh new file +7
-0@@ -0,0 +1,153 @@ +#!/bin/sh + +test_description='upload-pack ref-in-want' + +. ./test-lib.sh + +get_actual_refs() {
Style. "get_actual_refs () {"
t/t5703-upload-pack-ref-in-want.sh line 8
Quoted 1 line
t/t5703-upload-pack-ref-in-want.sh new file +1
-0+ sed -n '/wanted-refs/,/0001/p' <out | sed '1d;$d' | test-pkt-line unpack >actual_refs
Unnecessary piping of sed output into another sed invocation?
sed -n -e '/wanted-refs/,/0001/{
/wanted-refs/d
/0001/d
p
}'or something like that?
Re: [PATCH v5 2/8] upload-pack: implement ref-in-want
Quoted 16 lines · Junio C Hamano
Documentation/technical/protocol-v2.txt header. · line 407
Quoted 13 lines · Brandon Williams
Documentation/technical/protocol-v2.txt +13
-0+ wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs". + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server SHOULD NOT send any refs which were not requested + using 'want-ref' lines and a client MUST ignore refs which + weren't requested.
Quoted 7 lines
Just being curious, but the above feels the other way around. Why are we being more lenient to writers of broken server than writers of broken clients? The number of installations they need to take back and replace is certainly lower for the former, which means that, if exchanges of unsoliclited refs are unwanted, clients should notice and barf (or warn) if the server misbehaves, and the server should be forbidden from sending unsolicited refs, no?
Ok so should I change the server part to "MUST NOT" and the client part to "SHOULD"? And I can add code to die when we see refs that weren't requested, but i feel like if we add an ability to request a pattern in the future this will completely change, which is why I currently have a client just ignoring anything else.
Quoted 22 lines
Quoted 8 lines
t/t5703-upload-pack-ref-in-want.sh new file +0
-0t/t5703-upload-pack-ref-in-want.sh line 7
Quoted 7 lines
t/t5703-upload-pack-ref-in-want.sh new file +7
-0@@ -0,0 +1,153 @@ +#!/bin/sh + +test_description='upload-pack ref-in-want' + +. ./test-lib.sh + +get_actual_refs() {Style. "
get_actual_refs() {"t/t5703-upload-pack-ref-in-want.sh line 8
Quoted 1 line
t/t5703-upload-pack-ref-in-want.sh new file +1
-0+ sed -n '/wanted-refs/,/0001/p' <out | sed '1d;$d' | test-pkt-line unpack >actual_refsUnnecessary piping of sed output into another sed invocation?
Quoted 7 lines
sed -n -e '/wanted-refs/,/0001/{ /wanted-refs/d /0001/d p }'or something like that?
Yeah thanks for the help with sed :)
Signature · 1 line
Brandon Williams
Re: [PATCH v5 2/8] upload-pack: implement ref-in-want
Quoted 12 lines · Brandon Williams
Quoted 11 lines
Documentation/technical/protocol-v2.txt header. · line 407
Quoted 3 lines
Documentation/technical/protocol-v2.txt +3
-0+ * The server SHOULD NOT send any refs which were not requested + using 'want-ref' lines and a client MUST ignore refs which + weren't requested.Just being curious, but the above feels the other way around. Why are we being more lenient to writers of broken server than writers of broken clients? The number of installations they need to take back and replace is certainly lower for the former, which means that, if exchanges of unsoliclited refs are unwanted, clients should notice and barf (or warn) if the server misbehaves, and the server should be forbidden from sending unsolicited refs, no?
Quoted 5 lines
Ok so should I change the server part to "MUST NOT" and the client part to "SHOULD"? And I can add code to die when we see refs that weren't requested, but i feel like if we add an ability to request a pattern in the future this will completely change, which is why I currently have a client just ignoring anything else.
I did not have enough information to give an answer to "should I do X?"; that is why I asked these questions prefixed with "Just being curious". I do not quite get a good feeling that I now know enough to answer, still, but let me try.
If we anticipate backward incompatible changes between this early WIP stage and the final completed protocol, it would be GOOD to make sure that an early WIP clients/servers fail when seeing the other side gives them something they do not understand, no?
So...
Re: [PATCH v5 2/8] upload-pack: implement ref-in-want
Quoted 25 lines · Junio C Hamano
Quoted 12 lines · Brandon Williams
Quoted 11 lines
Documentation/technical/protocol-v2.txt header. · line 407
Quoted 3 lines
Documentation/technical/protocol-v2.txt +3
-0+ * The server SHOULD NOT send any refs which were not requested + using 'want-ref' lines and a client MUST ignore refs which + weren't requested.Just being curious, but the above feels the other way around. Why are we being more lenient to writers of broken server than writers of broken clients? The number of installations they need to take back and replace is certainly lower for the former, which means that, if exchanges of unsoliclited refs are unwanted, clients should notice and barf (or warn) if the server misbehaves, and the server should be forbidden from sending unsolicited refs, no?
Quoted 5 lines
Ok so should I change the server part to "MUST NOT" and the client part to "SHOULD"? And I can add code to die when we see refs that weren't requested, but i feel like if we add an ability to request a pattern in the future this will completely change, which is why I currently have a client just ignoring anything else.
I did not have enough information to give an answer to "should I do X?"; that is why I asked these questions prefixed with "Just being curious". I do not quite get a good feeling that I now know enough to answer, still, but let me try.
Quoted 6 lines
If we anticipate backward incompatible changes between this early WIP stage and the final completed protocol, it would be GOOD to make sure that an early WIP clients/servers fail when seeing the other side gives them something they do not understand, no?
So...
Yeah after thinking more about this I agree, we should have the client fail out and require that the server MUST not send additional refs. This can of course be loosened through a capability if we want to do something else in the future. Thanks for sanity checking me :)
Signature · 1 line
Brandon Williams
Re: [PATCH v5 2/8] upload-pack: implement ref-in-want
Quoted 1 line
Yeah after thinking more about this
I wonder if we have some mental model that we want to teach to the users? What is the fetch command (using the ref-in-want capability) supposed to do?
* update to the latest state observed by the latest remote talked to?
* update to some approximate state that is converged from multiple remotes?
* update to a state that the first remote had, that we talked to
Having such a model would make it easier for me to follow this discussion.
Quoted 2 lines
I agree, we should have the client
fail out and require that the server MUST not send additional refs.
This is reasoned for by the way we evolve the client, not some state the user expects to see short or longterm?
Quoted 2 lines
This can of course be loosened through a capability if we want to do something else in the future. Thanks for sanity checking me :)
ok, that is a sensible way to go forward.
Re: [PATCH v5 2/8] upload-pack: implement ref-in-want
In reply to Junio C HamanoQuoted 16 lines
Documentation/technical/protocol-v2.txt header. · line 407
Quoted 13 lines · Brandon Williams
Documentation/technical/protocol-v2.txt +13
-0+ wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs". + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server SHOULD NOT send any refs which were not requested + using 'want-ref' lines and a client MUST ignore refs which + weren't requested.
Quoted 7 lines
Just being curious, but the above feels the other way around. Why are we being more lenient to writers of broken server than writers of broken clients? The number of installations they need to take back and replace is certainly lower for the former, which means that, if exchanges of unsoliclited refs are unwanted, clients should notice and barf (or warn) if the server misbehaves, and the server should be forbidden from sending unsolicited refs, no?
I agree - if you were thinking of later relaxing this constraint to allow the server to supply tag ref information during tag following, writing "MUST NOT" here is still fine. (We can later change this to "if 'include-tag-ref' is sent, the server MAY send refs which were not requested, otherwise, the server MUST NOT".)
Both Jonathan Nieder and I also suggested client enforcement [1] - I see that in patch 8, in receive_wanted_refs(), unrecognized wanted-ref lines are silently ignored. Maybe at least a warning would be good.
PATCH v5 3/8 upload-pack: test negotiation with changing repository
In reply to Brandon WilliamsAdd tests to check the behavior of fetching from a repository which changes between rounds of negotiation (for example, when different servers in a load-balancing agreement participate in the same stateless RPC negotiation). This forms a baseline of comparison to the ref-in-want functionality (which will be introduced to the client in subsequent commits), and ensures that subsequent commits do not change existing behavior.
As part of this effort, a mechanism to substitute strings in a single HTTP response is added.
--- t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 ++++ t/lib-httpd/one-time-sed.sh | 22 ++++++++++ t/t5703-upload-pack-ref-in-want.sh | 68 ++++++++++++++++++++++++++++++ 4 files changed, 99 insertions(+) create mode 100644 t/lib-httpd/one-time-sed.sh
t/lib-httpd.sh +1 -0
@@ -132,6 +132,7 @@ prepare_httpd() { cp "$TEST_PATH"/passwd "$HTTPD_ROOT_PATH" install_script broken-smart-http.sh install_script error.sh + install_script one-time-sed.sh ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +8 -0
@@ -111,9 +111,14 @@ Alias /auth/dumb/ www/auth/dumb/ SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} SetEnv GIT_HTTP_EXPORT_ALL </LocationMatch> +<LocationMatch /one_time_sed/> + SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} + SetEnv GIT_HTTP_EXPORT_ALL +</LocationMatch> ScriptAliasMatch /smart_*[^/]*/(.*) ${GIT_EXEC_PATH}/git-http-backend/$1 ScriptAlias /broken_smart/ broken-smart-http.sh/ ScriptAlias /error/ error.sh/ +ScriptAliasMatch /one_time_sed/(.*) one-time-sed.sh/$1 <Directory ${GIT_EXEC_PATH}> Options FollowSymlinks </Directory>
@@ -123,6 +128,9 @@ ScriptAlias /error/ error.sh/ <Files error.sh> Options ExecCGI </Files> +<Files one-time-sed.sh> + Options ExecCGI +</Files> <Files ${GIT_EXEC_PATH}/git-http-backend> Options ExecCGI </Files>
t/lib-httpd/one-time-sed.sh new file +22 -0
@@ -0,0 +1,22 @@ +#!/bin/sh + +# If "one-time-sed" exists in $HTTPD_ROOT_PATH, run sed on the HTTP response, +# using the contents of "one-time-sed" as the sed command to be run. If the +# response was modified as a result, delete "one-time-sed" so that subsequent +# HTTP responses are no longer modified. +# +# This can be used to simulate the effects of the repository changing in +# between HTTP request-response pairs. +if [ -e one-time-sed ]; then "$GIT_EXEC_PATH/git-http-backend" >out + + sed "$(cat one-time-sed)" <out >out_modified + + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fi
t/t5703-upload-pack-ref-in-want.sh +68 -0
72 lines not on this page · raw message
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
Re: [PATCH v5 3/8] upload-pack: test negotiation with changing repository
t/lib-httpd/one-time-sed.sh line 6
Quoted 12 lines · Brandon Williams
t/lib-httpd/one-time-sed.sh new file +6
-0@@ -0,0 +1,22 @@ +#!/bin/sh + +# If "one-time-sed" exists in $HTTPD_ROOT_PATH, run sed on the HTTP response, +# using the contents of "one-time-sed" as the sed command to be run. If the +# response was modified as a result, delete "one-time-sed" so that subsequent +# HTTP responses are no longer modified.
;-) clever.
t/lib-httpd/one-time-sed.sh line 10
Quoted 4 lines
t/lib-httpd/one-time-sed.sh new file +4
-0+# +# This can be used to simulate the effects of the repository changing in +# between HTTP request-response pairs. +if [ -e one-time-sed ]; then "$GIT_EXEC_PATH/git-http-backend" >out
Style (cf. Documentation/CodingGuidelines).
t/lib-httpd/one-time-sed.sh line 22
Quoted 12 lines
t/lib-httpd/one-time-sed.sh new file +12
-0+ + sed "$(cat one-time-sed)" <out >out_modified + + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fi
OK. I was worried if the removal-after-use was about _this_ script (in which case it is a bad hygiene), but this is not a one-time script, but merely the mechanism to use the one-time sed script.
Perhaps rename this to t/lib-httpd/apply-one-time-sed.sh or something to avoid confusion?
Quoted 31 lines
t/t5703-upload-pack-ref-in-want.sh +23
-0@@ -150,4 +150,72 @@ test_expect_success 'want-ref with ref we already have commit for' ' check_output ' +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +REPO="$HTTPD_DOCUMENT_ROOT_PATH/repo" +LOCAL_PRISTINE="$(pwd)/local_pristine" + +test_expect_success 'setup repos for change-while-negotiating test' ' + ( + git init "$REPO" && + cd "$REPO" && + >.git/git-daemon-export-ok && + test_commit m1 && + git tag -d m1 && + + # Local repo with many commits (so that negotiation will take + # more than 1 request/response pair) + git clone "http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo" "$LOCAL_PRISTINE" && + cd "$LOCAL_PRISTINE" && + git checkout -b side && + for i in $(seq 1 33); do test_commit s$i; done && + + # Add novel commits to upstream + git checkout master &&
t/t5703-upload-pack-ref-in-want.sh test_expect_success 'want-ref with ref we already have commit for' ' · line 185
Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10
-0+ cd "$REPO" && + test_commit m2 && + test_commit m3 && + git tag -d m2 m3 + ) && + git -C "$LOCAL_PRISTINE" remote set-url origin "http://127.0.0.1:$LIB_HTTPD_PORT/one_time_sed/repo" && + git -C "$LOCAL_PRISTINE" config protocol.version 2 +' + +inconsistency() {
Style. "inconsistency () {"
Quoted 36 lines
t/t5703-upload-pack-ref-in-want.sh +35
-0+ # Simulate that the server initially reports $2 as the ref + # corresponding to $1, and after that, $1 as the ref corresponding to + # $1. This corresponds to the real-life situation where the server's + # repository appears to change during negotiation, for example, when + # different servers in a load-balancing arrangement serve (stateless) + # RPCs during a single negotiation. + printf "s/%s/%s/" \ + $(git -C "$REPO" rev-parse $1 | tr -d "\n") \ + $(git -C "$REPO" rev-parse $2 | tr -d "\n") \ + >"$HTTPD_ROOT_PATH/one-time-sed" +} + +test_expect_success 'server is initially ahead - no ref in want' ' + git -C "$REPO" config uploadpack.allowRefInWant false && + rm -rf local && + cp -r "$LOCAL_PRISTINE" local && + inconsistency master 1234567890123456789012345678901234567890 && + test_must_fail git -C local fetch 2>err && + grep "ERR upload-pack: not our ref" err +' + +test_expect_success 'server is initially behind - no ref in want' ' + git -C "$REPO" config uploadpack.allowRefInWant false && + rm -rf local && + cp -r "$LOCAL_PRISTINE" local && + inconsistency master "master^" && + git -C local fetch && + + git -C "$REPO" rev-parse --verify "master^" >expected && + git -C local rev-parse --verify refs/remotes/origin/master >actual && + test_cmp expected actual +' + +stop_httpd + test_done
Re: [PATCH v5 3/8] upload-pack: test negotiation with changing repository
Quoted 41 lines · Junio C Hamano
t/lib-httpd/one-time-sed.sh line 6
Quoted 12 lines · Brandon Williams
t/lib-httpd/one-time-sed.sh new file +6
-0@@ -0,0 +1,22 @@ +#!/bin/sh + +# If "one-time-sed" exists in $HTTPD_ROOT_PATH, run sed on the HTTP response, +# using the contents of "one-time-sed" as the sed command to be run. If the +# response was modified as a result, delete "one-time-sed" so that subsequent +# HTTP responses are no longer modified.;-) clever.
t/lib-httpd/one-time-sed.sh line 10
Quoted 4 lines
t/lib-httpd/one-time-sed.sh new file +4
-0+# +# This can be used to simulate the effects of the repository changing in +# between HTTP request-response pairs. +if [ -e one-time-sed ]; then "$GIT_EXEC_PATH/git-http-backend" >outStyle (cf.
Documentation/CodingGuidelines).t/lib-httpd/one-time-sed.sh line 22
Quoted 12 lines
t/lib-httpd/one-time-sed.sh new file +12
-0+ + sed "$(cat one-time-sed)" <out >out_modified + + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fiOK. I was worried if the removal-after-use was about _this_ script (in which case it is a bad hygiene), but this is not a one-time script, but merely the mechanism to use the one-time sed script.
Quoted 2 lines
Perhaps rename this to
t/lib-httpd/apply-one-time-sed.shor something to avoid confusion?
Sure, I'll go ahead and rename it to that.
Quoted 81 lines
Quoted 32 lines
t/t5703-upload-pack-ref-in-want.sh +23
-0@@ -150,4 +150,72 @@ test_expect_success 'want-ref with ref we already have commit for' ' check_output ' +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +REPO="$HTTPD_DOCUMENT_ROOT_PATH/repo" +LOCAL_PRISTINE="$(pwd)/local_pristine" + +test_expect_success 'setup repos for change-while-negotiating test' ' + ( + git init "$REPO" && + cd "$REPO" && + >.git/git-daemon-export-ok && + test_commit m1 && + git tag -d m1 && + + # Local repo with many commits (so that negotiation will take + # more than 1 request/response pair) + git clone "http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo" "$LOCAL_PRISTINE" && + cd "$LOCAL_PRISTINE" && + git checkout -b side && + for i in $(seq 1 33); do test_commit s$i; done && + + # Add novel commits to upstream + git checkout master &&t/t5703-upload-pack-ref-in-want.sh test_expect_success 'want-ref with ref we already have commit for' ' · line 185
Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10
-0+ cd "$REPO" && + test_commit m2 && + test_commit m3 && + git tag -d m2 m3 + ) && + git -C "$LOCAL_PRISTINE" remote set-url origin "http://127.0.0.1:$LIB_HTTPD_PORT/one_time_sed/repo" && + git -C "$LOCAL_PRISTINE" config protocol.version 2 +' + +inconsistency() {Style. "inconsistency () {"
Quoted 36 lines
t/t5703-upload-pack-ref-in-want.sh +35
-0+ # Simulate that the server initially reports $2 as the ref + # corresponding to $1, and after that, $1 as the ref corresponding to + # $1. This corresponds to the real-life situation where the server's + # repository appears to change during negotiation, for example, when + # different servers in a load-balancing arrangement serve (stateless) + # RPCs during a single negotiation. + printf "s/%s/%s/" \ + $(git -C "$REPO" rev-parse $1 | tr -d "\n") \ + $(git -C "$REPO" rev-parse $2 | tr -d "\n") \ + >"$HTTPD_ROOT_PATH/one-time-sed" +} + +test_expect_success 'server is initially ahead - no ref in want' ' + git -C "$REPO" config uploadpack.allowRefInWant false && + rm -rf local && + cp -r "$LOCAL_PRISTINE" local && + inconsistency master 1234567890123456789012345678901234567890 && + test_must_fail git -C local fetch 2>err && + grep "ERR upload-pack: not our ref" err +' + +test_expect_success 'server is initially behind - no ref in want' ' + git -C "$REPO" config uploadpack.allowRefInWant false && + rm -rf local && + cp -r "$LOCAL_PRISTINE" local && + inconsistency master "master^" && + git -C local fetch && + + git -C "$REPO" rev-parse --verify "master^" >expected && + git -C local rev-parse --verify refs/remotes/origin/master >actual && + test_cmp expected actual +' + +stop_httpd + test_done
Signature · 1 line
Brandon Williams
Re: [PATCH v5 3/8] upload-pack: test negotiation with changing repository
In reply to Brandon Williamst/lib-httpd/one-time-sed.sh line 7
Quoted 5 lines
t/lib-httpd/one-time-sed.sh new file +5
-0+# If "one-time-sed" exists in $HTTPD_ROOT_PATH, run sed on the HTTP response, +# using the contents of "one-time-sed" as the sed command to be run. If the +# response was modified as a result, delete "one-time-sed" so that subsequent +# HTTP responses are no longer modified. +#
Whitespace error on this line (the line immediately after "are no longer modified.").
PATCH v5 4/8 fetch: refactor the population of peer ref OIDs
In reply to Brandon WilliamsPopulate peer ref OIDs in get_ref_map instead of do_fetch. Besides tightening scopes of variables in the code, this also prepares for get_ref_map being able to be called multiple times within do_fetch.
--- builtin/fetch.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
builtin/fetch.c +18 -18
@@ -351,6 +351,7 @@ static struct ref *get_ref_map(struct transport *transport,/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs; +struct string_list existing_refs =STRING_LIST_INIT_DUP ;const struct ref *remote_refs;if (rs->nr)
@@ -458,7 +459,23 @@ static struct ref *get_ref_map(struct transport *transport, tail = &rm->next; } -return ref_remove_duplicates (ref_map); + ref_map =ref_remove_duplicates (ref_map); + +for_each_ref (add_existing, &existing_refs); +for (rm = ref_map; rm; rm = rm->next) { +if (rm->peer_ref) { +struct string_list_item *peer_item = +string_list_lookup (&existing_refs, + rm->peer_ref->name); +if (peer_item) { +struct object_id *old_oid = peer_item->util; +oidcpy (&rm->peer_ref->old_oid, old_oid); + } + } + } +string_list_clear (&existing_refs,1 ); + +return ref_map; }#define STORE_REF_ERROR_OTHER 1
@@ -1110,14 +1127,10 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)static int do_fetch (struct transport *transport,struct refspec *rs) { -struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct ref *ref_map; -struct ref *rm;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; -for_each_ref (add_existing, &existing_refs); -if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 ) tags =TAGS_SET ;
@@ -1136,18 +1149,6 @@ static int do_fetch(struct transport *transport,if (!update_head_ok)check_not_current_branch (ref_map); -for (rm = ref_map; rm; rm = rm->next) { -if (rm->peer_ref) { -struct string_list_item *peer_item = -string_list_lookup (&existing_refs, - rm->peer_ref->name); -if (peer_item) { -struct object_id *old_oid = peer_item->util; -oidcpy (&rm->peer_ref->old_oid, old_oid); - } - } - } -if (tags ==TAGS_DEFAULT && autotags)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,"1" );if (prune) {
@@ -1183,7 +1184,6 @@ static int do_fetch(struct transport *transport, } cleanup: -string_list_clear (&existing_refs,1 );return retcode; }
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v5 5/8 fetch: refactor fetch_refs into two functions
In reply to Brandon WilliamsRefactor the fetch_refs function into a function that does the fetching of refs and another function that stores them. This is in preparation for allowing additional processing of the fetched refs before updating the local ref store.
--- builtin/fetch.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-)
builtin/fetch.c +18 -5
@@ -968,9 +968,21 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)if (ret) ret =transport_fetch_refs (transport, ref_map);if (!ret) - ret |=store_updated_refs (transport->url, - transport->remote->name, - ref_map); +/* +* Keep the new pack's ".keep" file around to allow the caller +* time to update refs to reference the new objects. +*/ +return 0 ; +transport_unlock_pack (transport); +return ret; +} + +/* Update local refs based on the ref values fetched from a remote */ +static int consume_refs (struct transport *transport,struct ref *ref_map) +{ +int ret =store_updated_refs (transport->url, + transport->remote->name, + ref_map);transport_unlock_pack (transport);return ret; }
@@ -1116,7 +1128,8 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -fetch_refs (transport, ref_map); +if (!fetch_refs (transport, ref_map)) +consume_refs (transport, ref_map);if (gsecondary) {transport_disconnect (gsecondary);
@@ -1165,7 +1178,7 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map)) { +if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
PATCH v5 6/8 fetch: refactor to make function args narrower
In reply to Brandon WilliamsRefactor find_non_local_tags and get_ref_map to only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
--- builtin/fetch.c | 52 ++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
builtin/fetch.c +26 -26
@@ -254,9 +254,9 @@ static int will_fetch(struct ref **head, const unsigned char *sha1)return 0 ; } -static void find_non_local_tags (struct transport *transport, -struct ref **head, -struct ref ***tail) +static void find_non_local_tags (const struct ref *refs, +struct ref **head, +struct ref ***tail) {struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct string_list remote_refs =STRING_LIST_INIT_NODUP ;
@@ -264,7 +264,7 @@ static void find_non_local_tags(struct transport *transport,struct string_list_item *item =NULL ;for_each_ref (add_existing, &existing_refs); -for (ref =transport_get_remote_refs (transport,NULL ); ref; ref = ref->next) { +for (ref = refs; ref; ref = ref->next) {if (!starts_with (ref->name,"refs/tags/" ))continue ;
@@ -338,7 +338,8 @@ static void find_non_local_tags(struct transport *transport,string_list_clear (&remote_refs,0 ); } -static struct ref *get_ref_map (struct transport *transport, +static struct ref *get_ref_map (struct remote *remote, +const struct ref *remote_refs,struct refspec *rs,int tags,int *autotags) {
@@ -346,27 +347,11 @@ static struct ref *get_ref_map(struct transport *transport,struct ref *rm;struct ref *ref_map =NULL ;struct ref **tail = &ref_map; -struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs;struct string_list existing_refs =STRING_LIST_INIT_DUP ; -const struct ref *remote_refs; - -if (rs->nr) -refspec_ref_prefixes (rs, &ref_prefixes); -else if (transport->remote && transport->remote->fetch.nr) -refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); - -if (ref_prefixes.argc && - (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { -argv_array_push (&ref_prefixes,"refs/tags/" ); - } - - remote_refs =transport_get_remote_refs (transport, &ref_prefixes); - -argv_array_clear (&ref_prefixes);if (rs->nr) {struct refspec *fetch_refspec;
@@ -403,7 +388,7 @@ static struct ref *get_ref_map(struct transport *transport,if (refmap.nr) fetch_refspec = &refmap;else - fetch_refspec = &transport->remote->fetch; + fetch_refspec = &remote->fetch;for (i =0 ; i < fetch_refspec->nr; i++)get_fetch_map (ref_map, &fetch_refspec->items[i], &oref_tail,1 );
@@ -411,7 +396,6 @@ static struct ref *get_ref_map(struct transport *transport,die ("--refmap option is only meaningful with command-line refspec(s)." ); }else {/* Use the defaults */ -struct remote *remote = transport->remote;struct branch *branch =branch_get (NULL );int has_merge =branch_has_merge_config (branch); if (remote &&
@@ -450,7 +434,7 @@ static struct ref *get_ref_map(struct transport *transport,/* also fetch all tags */ get_fetch_map (remote_refs, tag_refspec, &tail,0 ); else if (tags == TAGS_DEFAULT && *autotags) - find_non_local_tags(transport, &ref_map, &tail); + find_non_local_tags(remote_refs, &ref_map, &tail);/* Now append any refs to be updated opportunistically: */ *tail = orefs;
@@ -1143,6 +1127,8 @@ static int do_fetch(struct transport *transport,struct ref *ref_map;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; +const struct ref *remote_refs; +struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 )
@@ -1158,7 +1144,21 @@ static int do_fetch(struct transport *transport, goto cleanup; } - ref_map =get_ref_map (transport, rs, tags, &autotags); +if (rs->nr) +refspec_ref_prefixes (rs, &ref_prefixes); +else if (transport->remote && transport->remote->fetch.nr) +refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); + +if (ref_prefixes.argc && + (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { +argv_array_push (&ref_prefixes,"refs/tags/" ); + } + + remote_refs =transport_get_remote_refs (transport, &ref_prefixes); +argv_array_clear (&ref_prefixes); + + ref_map =get_ref_map (transport->remote, remote_refs, rs, + tags, &autotags);if (!update_head_ok)check_not_current_branch (ref_map);
@@ -1190,7 +1190,7 @@ static int do_fetch(struct transport *transport,if (tags ==TAGS_DEFAULT && autotags) {struct ref **tail = &ref_map; ref_map =NULL ; -find_non_local_tags (transport, &ref_map, &tail); +find_non_local_tags (remote_refs, &ref_map, &tail);if (ref_map)backfill_tags (transport, ref_map);free_refs (ref_map);
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
Re: [PATCH v5 6/8] fetch: refactor to make function args narrower
Quoted 5 lines · Brandon Williams
Refactor
find_non_local_tagsandget_ref_mapto only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
Makes sense. One can argue that the original has less risk of set of refs used for calls to this function vs calls for others go out of sync, but the objective of this series is to allow them to be different ;-)
I am not sure if I understand/agree with the split of parameters done to get_ref_map() at this step, but hopefully its benefit would become obvious once I read later steps.
PATCH v5 7/8 fetch-pack: put shallow info in output parameter
In reply to Brandon WilliamsExpand the transport fetch method signature, by adding an output parameter, to allow transports to return information about the refs they have fetched. Then communicate shallow status information through this mechanism instead of by modifying the input list of refs.
This does require clients to sometimes generate the ref map twice: once from the list of refs provided by the remote (as is currently done) and potentially once from the new list of refs that the fetch mechanism provides.
--- builtin/clone.c | 4 ++-- builtin/fetch.c | 28 ++++++++++++++++++++++++---- fetch-object.c | 2 +- fetch-pack.c | 15 ++++++++------- transport-helper.c | 6 ++++-- transport-internal.h | 9 ++++++++- transport.c | 34 ++++++++++++++++++++++++++++------ transport.h | 3 ++- 8 files changed, 77 insertions(+), 24 deletions(-)
builtin/clone.c +2 -2
@@ -1155,7 +1155,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (!is_local && !complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); remote_head =find_ref_by_name (refs,"HEAD" ); remote_head_points_at =
@@ -1197,7 +1197,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)if (is_local)clone_local (path, git_dir);else if (refs && complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); update_remote_refs(refs, mapped_refs, remote_head_points_at, branch_top.buf, reflog_msg.buf, transport,
builtin/fetch.c +24 -4
@@ -946,11 +946,13 @@ static int quickfetch(struct ref *ref_map)return check_connected (iterate_ref_map, &rm, &opt); } -static int fetch_refs (struct transport *transport,struct ref *ref_map) +static int fetch_refs (struct transport *transport,struct ref *ref_map, +struct ref **updated_remote_refs) {int ret =quickfetch (ref_map);if (ret) - ret =transport_fetch_refs (transport, ref_map); + ret =transport_fetch_refs (transport, ref_map, + updated_remote_refs);if (!ret)/* * Keep the new pack's ".keep" file around to allow the caller
@@ -1112,7 +1114,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -if (!fetch_refs (transport, ref_map)) +if (!fetch_refs (transport, ref_map,NULL ))consume_refs (transport, ref_map);if (gsecondary) {
@@ -1128,6 +1130,7 @@ static int do_fetch(struct transport *transport,int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ;const struct ref *remote_refs; +struct ref *updated_remote_refs =NULL ;struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {
@@ -1178,7 +1181,24 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) { + +if (fetch_refs (transport, ref_map, &updated_remote_refs)) { +free_refs (ref_map); + retcode =1 ; + goto cleanup; + } +if (updated_remote_refs) { +/* +* Regenerate ref_map using the updated remote refs. This is +* to account for additional information which may be provided +* by the transport (e.g. shallow info). +*/ +free_refs (ref_map); + ref_map =get_ref_map (transport->remote, updated_remote_refs, rs, + tags, &autotags); +free_refs (updated_remote_refs); + } +if (consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
fetch-object.c +1 -1
@@ -19,7 +19,7 @@ static void fetch_refs(const char *remote_name, struct ref *ref)transport_set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" );transport_set_option (transport,TRANS_OPT_NO_DEPENDENTS ,"1" ); -transport_fetch_refs (transport, ref); +transport_fetch_refs (transport, ref,NULL ); fetch_if_missing = original_fetch_if_missing; }
fetch-pack.c +8 -7
@@ -1470,12 +1470,13 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr) }static void update_shallow (struct fetch_pack_args *args, -struct ref **sought,int nr_sought, +struct ref *refs,struct shallow_info *si) {struct oid_array ref =OID_ARRAY_INIT ;int *status;int i; +struct ref *r;if (args->deepen && alternate_shallow_file) {if (*alternate_shallow_file =='\0' ) {/* --unshallow */
@@ -1517,8 +1518,8 @@ static void update_shallow(struct fetch_pack_args *args,remove_nonexistent_theirs_shallow (si);if (!si->nr_ours && !si->nr_theirs)return ; -for (i =0 ; i < nr_sought; i++) -oid_array_append (&ref, &sought[i]->old_oid); +for (r = refs; r; r = r->next) +oid_array_append (&ref, &r->old_oid); si->ref = &ref;if (args->update_shallow) {
@@ -1552,12 +1553,12 @@ static void update_shallow(struct fetch_pack_args *args,* remote is also shallow, check what ref is safe to update * without updating .git/shallow */ - status =xcalloc (nr_sought,sizeof (*status)); + status =xcalloc (ref.nr,sizeof (*status));assign_shallow_commits_to_refs (si,NULL , status);if (si->nr_ours || si->nr_theirs) { -for (i =0 ; i < nr_sought; i++) +for (r = refs, i =0 ; r; r = r->next, i++)if (status[i]) - sought[i]->status =REF_STATUS_REJECT_SHALLOW ; + r->status =REF_STATUS_REJECT_SHALLOW ; }free (status);oid_array_clear (&ref);
@@ -1591,7 +1592,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args, ref_cpy =do_fetch_pack (args, fd, ref, sought, nr_sought, &si, pack_lockfile);reprepare_packed_git (the_repository); -update_shallow (args, sought, nr_sought, &si); +update_shallow (args, ref_cpy, &si);clear_shallow_info (&si);return ref_cpy; }
transport-helper.c +4 -2
@@ -651,14 +651,16 @@ static int connect_helper(struct transport *transport, const char *name, }static int fetch (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct helper_data *data = transport->data;int i, count;if (process_connect (transport,0 )) {do_take_over (transport); -return transport->vtable->fetch (transport, nr_heads, to_fetch); +return transport->vtable->fetch (transport, nr_heads, to_fetch, + fetched_refs); } count =0 ;
transport-internal.h +8 -1
@@ -36,11 +36,18 @@ struct transport_vtable {* Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. * +* The transport *may* provide, in fetched_refs, the list of refs that +* it fetched. If the transport knows anything about the fetched refs +* that the caller does not know (for example, shallow status), it +* should provide that list of refs and include that information in the +* list. +* * If the transport did not get hashes for refs in * get_refs_list(), it should set the old_sha1 fields in the * provided refs now. **/ -int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs); +int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs, +struct ref **fetched_refs);/** * Push the objects and refs. Send the necessary objects, and
transport.c +28 -6
@@ -151,7 +151,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport, }static int fetch_refs_from_bundle (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct bundle_transport_data *data = transport->data; return unbundle(&data->header, data->fd,
@@ -287,7 +288,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus }static int fetch_refs_via_pack (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -354,8 +356,12 @@ static int fetch_refs_via_pack(struct transport *transport,if (report_unmatched_refs (to_fetch, nr_heads)) ret =-1 ; +if (fetched_refs) + *fetched_refs = refs; +else +free_refs (refs); +free_refs (refs_tmp); -free_refs (refs);free (dest);return ret; }
@@ -1215,19 +1221,31 @@ const struct ref *transport_get_remote_refs(struct transport *transport,return transport->remote_refs; } -int transport_fetch_refs (struct transport *transport,struct ref *refs) +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs) {int rc;int nr_heads =0 , nr_alloc =0 , nr_refs =0 ;struct ref **heads =NULL ; +struct ref *nop_head =NULL , **nop_tail = &nop_head;struct ref *rm;for (rm = refs; rm; rm = rm->next) { nr_refs++;if (rm->peer_ref && !is_null_oid (&rm->old_oid) && - !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) + !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don't +* actually need to fetch them. +*/ +if (fetched_refs) { +struct ref *nop_ref =copy_ref (rm); + *nop_tail = nop_ref; + nop_tail = &nop_ref->next; + }continue ; + }ALLOC_GROW (heads, nr_heads +1 , nr_alloc); heads[nr_heads++] = rm; }
@@ -1245,7 +1263,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) heads[nr_heads++] = rm; } - rc = transport->vtable->fetch (transport, nr_heads, heads); + rc = transport->vtable->fetch (transport, nr_heads, heads, fetched_refs); +if (fetched_refs && nop_head) { + *nop_tail = *fetched_refs; + *fetched_refs = nop_head; + }free (heads);return rc;
transport.h +2 -1
@@ -218,7 +218,8 @@ int transport_push(struct transport *connection,const struct ref *transport_get_remote_refs (struct transport *transport,const struct argv_array *ref_prefixes); -int transport_fetch_refs (struct transport *transport,struct ref *refs); +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs);void transport_unlock_pack (struct transport *transport);int transport_disconnect (struct transport *transport);char *transport_anonymize_url (const char *url);
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
Re: [PATCH v5 7/8] fetch-pack: put shallow info in output parameter
Quoted 4 lines · Brandon Williams
Expand the transport fetch method signature, by adding an output parameter, to allow transports to return information about the refs they have fetched. Then communicate shallow status information through this mechanism instead of by modifying the input list of refs.
Makes sense. Would this mechanism also allow us to be more explicit about the "tag following"?
Re: [PATCH v5 7/8] fetch-pack: put shallow info in output parameter
Quoted 10 lines · Junio C Hamano
Quoted 4 lines · Brandon Williams
Expand the transport fetch method signature, by adding an output parameter, to allow transports to return information about the refs they have fetched. Then communicate shallow status information through this mechanism instead of by modifying the input list of refs.
Makes sense. Would this mechanism also allow us to be more explicit about the "tag following"?
Yes most likely. We could change it so that when a packfile is sent the result of tag following could be sent along too (the actual tag refs themselves that is) instead of having the client rely on the ref advertisement for tag following.
Signature · 1 line
Brandon Williams
PATCH v5 8/8 fetch-pack: implement ref-in-want
In reply to Brandon WilliamsImplement ref-in-want on the client side so that when a server supports the "ref-in-want" feature, a client will send "want-ref" lines for each reference the client wants to fetch. This feature allows clients to tolerate inconsistencies that exist when a remote repository's refs change during the course of negotiation.
This allows a client to request to request a particular ref without specifying the OID of the ref. This means that instead of hitting an error when a ref no longer points at the OID it did at the beginning of negotiation, negotiation can continue and the value of that ref will be sent at the termination of negotiation, just before a packfile is sent.
More information on the ref-in-want feature can be found in Documentation/technical/protocol-v2.txt.
--- fetch-pack.c | 35 +++++++- remote.c | 1 + remote.h | 1 + t/t5703-upload-pack-ref-in-want.sh | 130 +++++++++++++++++++++++++++++ 4 files changed, 164 insertions(+), 3 deletions(-)
fetch-pack.c +32 -3
@@ -1102,9 +1102,10 @@ static void add_shallow_requests(struct strbuf *req_buf,static void add_wants (const struct ref *wants,struct strbuf *req_buf) { +int use_ref_in_want =server_supports_feature ("fetch" ,"ref-in-want" ,0 ); +for ( ; wants ; wants = wants->next) {const struct object_id *remote = &wants->old_oid; -const char *remote_hex;struct object *o;/*
@@ -1122,8 +1123,10 @@ static void add_wants(const struct ref *wants, struct strbuf *req_buf)continue ; } - remote_hex =oid_to_hex (remote); -packet_buf_write (req_buf,"want %s\n" , remote_hex); +if (!use_ref_in_want || wants->exact_oid) +packet_buf_write (req_buf,"want %s\n" ,oid_to_hex (remote)); +else +packet_buf_write (req_buf,"want-ref %s\n" , wants->name); } }
@@ -1334,6 +1337,29 @@ static void receive_shallow_info(struct fetch_pack_args *args, args->deepen =1 ; } +static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line); + +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + } + } + +if (reader->status !=PACKET_READ_DELIM ) +die ("error processing wanted refs: %d" , reader->status); +} +enum fetch_state {FETCH_CHECK_LOCAL =0 ,FETCH_SEND_REQUEST ,
@@ -1408,6 +1434,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,if (process_section_header (&reader,"shallow-info" ,1 ))receive_shallow_info (args, &reader); +if (process_section_header (&reader,"wanted-refs" ,1 )) +receive_wanted_refs (&reader, ref); +/* get the pack */ process_section_header (&reader,"packfile" ,0 );if (get_pack (args, fd, pack_lockfile))
remote.c +1 -0
@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_oid =1 ; }else { ref_map =get_remote_ref (remote_refs, name); }
remote.h +1 -0
@@ -73,6 +73,7 @@ struct ref { force:1, forced_update:1, expect_old_sha1:1, + exact_oid:1, deletion:1;enum {
t/t5703-upload-pack-ref-in-want.sh +130 -0
142 lines not on this page · raw message
Signature · 1 line
2.18.0.rc2.346.g013aa6912e-goog
Re: [PATCH v5 8/8] fetch-pack: implement ref-in-want
Quoted 1 line
+test_expect_success 'setup repos for change-while-negotiating test' '
The tests that follow are basic ref-in-want tests, not tests on a repo that changes during negotiation - this would be just "setup repos for fetch tests".
t/t5703-upload-pack-ref-in-want.sh test_expect_success 'server is initially behind - no ref in want' ' · line 312
Quoted 9 lines
t/t5703-upload-pack-ref-in-want.sh +9
-0+test_expect_success 'fetching with exact OID' ' + rm -rf local && + cp -r "$LOCAL_PRISTINE" local && + git -C local fetch origin $(git -C "$REPO" rev-parse d):refs/heads/actual && + + git -C "$REPO" rev-parse "d" >expected && + git -C local rev-parse refs/heads/actual >actual && + test_cmp expected actual +'
Also verify that "want-ref refs/tags/d" is being sent over the wire, and not any "want ...". (If not we can't distinguish these from the usual non-want-ref behavior.) Same comment for the other tests.
Other than that (and my other comments), this patch series looks good.
Re: [PATCH v5 8/8] fetch-pack: implement ref-in-want
Quoted 5 lines · Jonathan Tan
Quoted 1 line
+test_expect_success 'setup repos for change-while-negotiating test' '
The tests that follow are basic ref-in-want tests, not tests on a repo that changes during negotiation - this would be just "setup repos for fetch tests".
That looks like a copy-paste error.
Quoted 14 lines
t/t5703-upload-pack-ref-in-want.sh test_expect_success 'server is initially behind - no ref in want' ' · line 312
Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +9
-0+test_expect_success 'fetching with exact OID' ' + rm -rf local && + cp -r "$LOCAL_PRISTINE" local && + git -C local fetch origin $(git -C "$REPO" rev-parse d):refs/heads/actual && + + git -C "$REPO" rev-parse "d" >expected && + git -C local rev-parse refs/heads/actual >actual && + test_cmp expected actual +'Also verify that "want-ref refs/tags/d" is being sent over the wire, and not any "want ...". (If not we can't distinguish these from the usual non-want-ref behavior.) Same comment for the other tests.
I think your mistaken on how what this test is looking for. no want-ref line is going to be sent because we're requesting an exact OID here, not a ref. But I can add checks for want-ref in the tests that should be sending want-ref.
Quoted 2 lines
Other than that (and my other comments), this patch series looks good.
Signature · 1 line
Brandon Williams
v6 cover ref-in-want
In reply to Brandon WilliamsChanges in v6:
* Stricter checks in tests
* Change the name of a test to better reflect what is being tested
* Various style issues fixed in shell scripts
* Renamed one-time-sed.sh to apply-one-time-sed.sh
* Client now errors out when an unexpected ref is sent from the server during the wanted-ref section. Also changed the docs around the server's responsibility with the refs that are sent during this section.
Brandon Williams (8):
test-pkt-line: add unpack-sideband subcommand
upload-pack: implement ref-in-want
upload-pack: test negotiation with changing repository
fetch: refactor the population of peer ref OIDs
fetch: refactor fetch_refs into two functions
fetch: refactor to make function args narrower
fetch-pack: put shallow info in output parameter
fetch-pack: implement ref-in-want
Documentation/config.txt | 7 + Documentation/technical/protocol-v2.txt | 28 +- builtin/clone.c | 4 +- builtin/fetch.c | 135 +++++---- fetch-object.c | 2 +- fetch-pack.c | 53 +++- remote.c | 1 + remote.h | 1 + t/helper/test-pkt-line.c | 33 +++ t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 + t/lib-httpd/apply-one-time-sed.sh | 22 ++ t/t5703-upload-pack-ref-in-want.sh | 377 ++++++++++++++++++++++++ transport-helper.c | 6 +- transport-internal.h | 9 +- transport.c | 34 ++- transport.h | 3 +- upload-pack.c | 66 +++++ 18 files changed, 715 insertions(+), 75 deletions(-) create mode 100644 t/lib-httpd/apply-one-time-sed.sh create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
PATCH v6 1/8 test-pkt-line: add unpack-sideband subcommand
Add an 'unpack-sideband' subcommand to the test-pkt-line helper to enable unpacking packet line data sent multiplexed using a sideband.
--- t/helper/test-pkt-line.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
t/helper/test-pkt-line.c +33 -0
@@ -1,3 +1,4 @@ +#include "cache.h" #include "pkt-line.h" static void pack_line (const char *line)
@@ -48,6 +49,36 @@ static void unpack(void) } } +static void unpack_sideband (void ) +{ +struct packet_reader reader; +packet_reader_init (&reader,0 ,NULL ,0 , +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_CHOMP_NEWLINE ); + +while (packet_reader_read (&reader) !=PACKET_READ_EOF ) { +int band; +int fd; + +switch (reader.status) { +case PACKET_READ_EOF : +break ; +case PACKET_READ_NORMAL : + band = reader.line[0 ] &0xff ; +if (band <1 || band >2 ) +die ("unexpected side band %d" , band); + fd = band; + +write_or_die (fd, reader.line +1 , reader.pktlen -1 ); +break ; +case PACKET_READ_FLUSH : +return ; +case PACKET_READ_DELIM : +break ; + } + } +} +int cmd_main (int argc,const char **argv) {if (argc <2 )
@@ -57,6 +88,8 @@ int cmd_main(int argc, const char **argv)pack (argc -2 , argv +2 ); else if (!strcmp(argv[1], "unpack")) unpack(); + else if (!strcmp(argv[1], "unpack-sideband")) + unpack_sideband(); else die("invalid argument '%s'", argv[1]);
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
PATCH v6 2/8 upload-pack: implement ref-in-want
In reply to Brandon WilliamsCurrently, while performing packfile negotiation, clients are only allowed to specify their desired objects using object ids. This causes a vulnerability to failure when an object turns non-existent during negotiation, which may happen if, for example, the desired repository is provided by multiple Git servers in a load-balancing arrangement and there exists replication delay.
In order to eliminate this vulnerability, implement the ref-in-want feature for the 'fetch' command in protocol version 2. This feature enables the 'fetch' command to support requests in the form of ref names through a new "want-ref <ref>" parameter. At the conclusion of negotiation, the server will send a list of all of the wanted references (as provided by "want-ref" lines) in addition to the generated packfile.
--- Documentation/config.txt | 7 ++ Documentation/technical/protocol-v2.txt | 28 ++++- t/t5703-upload-pack-ref-in-want.sh | 160 ++++++++++++++++++++++++ upload-pack.c | 66 ++++++++++ 4 files changed, 260 insertions(+), 1 deletion(-) create mode 100755 t/t5703-upload-pack-ref-in-want.sh
Documentation/config.txt +7 -0
@@ -3479,6 +3479,13 @@ Note that this configuration variable is ignored if it is seen in the repository-level config (this is a safety measure against fetching from untrusted repositories). +uploadpack.allowRefInWant:: + If this option is set, `upload-pack` will support the `ref-in-want` + feature of the protocol version 2 `fetch` command. This feature + is intended for the benefit of load-balanced servers which may + not have the same view of what OIDs their refs point to due to + replication delay. + url.<base>.insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with <base>. In cases where some site serves a
Documentation/technical/protocol-v2.txt +27 -1
@@ -299,12 +299,21 @@ included in the client's request: for use with partial clone and partial fetch operations. See `rev-list` for possible "filter-spec" values. +If the 'ref-in-want' feature is advertised, the following argument can +be included in the client's request as well as the potential addition of +the 'wanted-refs' section in the server's response as explained below. + + want-ref <ref> + Indicates to the server that the client wants to retrieve a + particular ref, where <ref> is the full name of a ref on the + server. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. output = *section - section = (acknowledgments | shallow-info | packfile) + section = (acknowledgments | shallow-info | wanted-refs | packfile) (flush-pkt | delim-pkt) acknowledgments = PKT-LINE("acknowledgments" LF)
@@ -319,6 +328,10 @@ header. shallow = "shallow" SP obj-id unshallow = "unshallow" SP obj-id + wanted-refs = PKT-LINE("wanted-refs" LF) + *PKT-LINE(wanted-ref LF) + wanted-ref = obj-id SP refname + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff)
@@ -379,6 +392,19 @@ header. * This section is only included if a packfile section is also included in the response. + wanted-refs section + * This section is only included if the client has requested a + ref using a 'want-ref' line and if a packfile section is also + included in the response. + + * Always begins with the section header "wanted-refs". + + * The server will send a ref listing ("<oid> <refname>") for + each reference requested using 'want-ref' lines. + + * The server MUST NOT send any refs which were not requested + using 'want-ref' lines. + packfile section * This section is only included if the client has sent 'want' lines in its request and either requested that no more
t/t5703-upload-pack-ref-in-want.sh new file +160 -0
160 lines not on this page · raw message
upload-pack.c +66 -0
138 lines not on this page · raw message
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
PATCH v6 3/8 upload-pack: test negotiation with changing repository
In reply to Brandon WilliamsAdd tests to check the behavior of fetching from a repository which changes between rounds of negotiation (for example, when different servers in a load-balancing agreement participate in the same stateless RPC negotiation). This forms a baseline of comparison to the ref-in-want functionality (which will be introduced to the client in subsequent commits), and ensures that subsequent commits do not change existing behavior.
As part of this effort, a mechanism to substitute strings in a single HTTP response is added.
--- t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 8 ++++ t/lib-httpd/apply-one-time-sed.sh | 22 ++++++++++ t/t5703-upload-pack-ref-in-want.sh | 68 ++++++++++++++++++++++++++++++ 4 files changed, 99 insertions(+) create mode 100644 t/lib-httpd/apply-one-time-sed.sh
t/lib-httpd.sh +1 -0
@@ -132,6 +132,7 @@ prepare_httpd() { cp "$TEST_PATH"/passwd "$HTTPD_ROOT_PATH" install_script broken-smart-http.sh install_script error.sh + install_script apply-one-time-sed.sh ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +8 -0
@@ -111,9 +111,14 @@ Alias /auth/dumb/ www/auth/dumb/ SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} SetEnv GIT_HTTP_EXPORT_ALL </LocationMatch> +<LocationMatch /one_time_sed/> + SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} + SetEnv GIT_HTTP_EXPORT_ALL +</LocationMatch> ScriptAliasMatch /smart_*[^/]*/(.*) ${GIT_EXEC_PATH}/git-http-backend/$1 ScriptAlias /broken_smart/ broken-smart-http.sh/ ScriptAlias /error/ error.sh/ +ScriptAliasMatch /one_time_sed/(.*) apply-one-time-sed.sh/$1 <Directory ${GIT_EXEC_PATH}> Options FollowSymlinks </Directory>
@@ -123,6 +128,9 @@ ScriptAlias /error/ error.sh/ <Files error.sh> Options ExecCGI </Files> +<Files apply-one-time-sed.sh> + Options ExecCGI +</Files> <Files ${GIT_EXEC_PATH}/git-http-backend> Options ExecCGI </Files>
t/lib-httpd/apply-one-time-sed.sh new file +22 -0
@@ -0,0 +1,22 @@ +#!/bin/sh + +# If "one-time-sed" exists in $HTTPD_ROOT_PATH, run sed on the HTTP response, +# using the contents of "one-time-sed" as the sed command to be run. If the +# response was modified as a result, delete "one-time-sed" so that subsequent +# HTTP responses are no longer modified. +# +# This can be used to simulate the effects of the repository changing in +# between HTTP request-response pairs. +if [ -e one-time-sed ]; then + "$GIT_EXEC_PATH/git-http-backend" >out + sed "$(cat one-time-sed)" <out >out_modified + + if diff out out_modified >/dev/null; then + cat out + else + cat out_modified + rm one-time-sed + fi +else + "$GIT_EXEC_PATH/git-http-backend" +fi
t/t5703-upload-pack-ref-in-want.sh +68 -0
72 lines not on this page · raw message
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
PATCH v6 4/8 fetch: refactor the population of peer ref OIDs
In reply to Brandon WilliamsPopulate peer ref OIDs in get_ref_map instead of do_fetch. Besides tightening scopes of variables in the code, this also prepares for get_ref_map being able to be called multiple times within do_fetch.
--- builtin/fetch.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
builtin/fetch.c +18 -18
@@ -351,6 +351,7 @@ static struct ref *get_ref_map(struct transport *transport,/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs; +struct string_list existing_refs =STRING_LIST_INIT_DUP ;const struct ref *remote_refs;if (rs->nr)
@@ -458,7 +459,23 @@ static struct ref *get_ref_map(struct transport *transport, tail = &rm->next; } -return ref_remove_duplicates (ref_map); + ref_map =ref_remove_duplicates (ref_map); + +for_each_ref (add_existing, &existing_refs); +for (rm = ref_map; rm; rm = rm->next) { +if (rm->peer_ref) { +struct string_list_item *peer_item = +string_list_lookup (&existing_refs, + rm->peer_ref->name); +if (peer_item) { +struct object_id *old_oid = peer_item->util; +oidcpy (&rm->peer_ref->old_oid, old_oid); + } + } + } +string_list_clear (&existing_refs,1 ); + +return ref_map; }#define STORE_REF_ERROR_OTHER 1
@@ -1110,14 +1127,10 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)static int do_fetch (struct transport *transport,struct refspec *rs) { -struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct ref *ref_map; -struct ref *rm;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; -for_each_ref (add_existing, &existing_refs); -if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 ) tags =TAGS_SET ;
@@ -1136,18 +1149,6 @@ static int do_fetch(struct transport *transport,if (!update_head_ok)check_not_current_branch (ref_map); -for (rm = ref_map; rm; rm = rm->next) { -if (rm->peer_ref) { -struct string_list_item *peer_item = -string_list_lookup (&existing_refs, - rm->peer_ref->name); -if (peer_item) { -struct object_id *old_oid = peer_item->util; -oidcpy (&rm->peer_ref->old_oid, old_oid); - } - } - } -if (tags ==TAGS_DEFAULT && autotags)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,"1" );if (prune) {
@@ -1183,7 +1184,6 @@ static int do_fetch(struct transport *transport, } cleanup: -string_list_clear (&existing_refs,1 );return retcode; }
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
PATCH v6 5/8 fetch: refactor fetch_refs into two functions
In reply to Brandon WilliamsRefactor the fetch_refs function into a function that does the fetching of refs and another function that stores them. This is in preparation for allowing additional processing of the fetched refs before updating the local ref store.
--- builtin/fetch.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-)
builtin/fetch.c +18 -5
@@ -968,9 +968,21 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map)if (ret) ret =transport_fetch_refs (transport, ref_map);if (!ret) - ret |=store_updated_refs (transport->url, - transport->remote->name, - ref_map); +/* +* Keep the new pack's ".keep" file around to allow the caller +* time to update refs to reference the new objects. +*/ +return 0 ; +transport_unlock_pack (transport); +return ret; +} + +/* Update local refs based on the ref values fetched from a remote */ +static int consume_refs (struct transport *transport,struct ref *ref_map) +{ +int ret =store_updated_refs (transport->url, + transport->remote->name, + ref_map);transport_unlock_pack (transport);return ret; }
@@ -1116,7 +1128,8 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -fetch_refs (transport, ref_map); +if (!fetch_refs (transport, ref_map)) +consume_refs (transport, ref_map);if (gsecondary) {transport_disconnect (gsecondary);
@@ -1165,7 +1178,7 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map)) { +if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
PATCH v6 6/8 fetch: refactor to make function args narrower
In reply to Brandon WilliamsRefactor find_non_local_tags and get_ref_map to only take the information they need instead of the entire transport struct. Besides improving code clarity, this also improves their flexibility, allowing for a different set of refs to be used instead of relying on the ones stored in the transport struct.
--- builtin/fetch.c | 52 ++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
builtin/fetch.c +26 -26
@@ -254,9 +254,9 @@ static int will_fetch(struct ref **head, const unsigned char *sha1)return 0 ; } -static void find_non_local_tags (struct transport *transport, -struct ref **head, -struct ref ***tail) +static void find_non_local_tags (const struct ref *refs, +struct ref **head, +struct ref ***tail) {struct string_list existing_refs =STRING_LIST_INIT_DUP ;struct string_list remote_refs =STRING_LIST_INIT_NODUP ;
@@ -264,7 +264,7 @@ static void find_non_local_tags(struct transport *transport,struct string_list_item *item =NULL ;for_each_ref (add_existing, &existing_refs); -for (ref =transport_get_remote_refs (transport,NULL ); ref; ref = ref->next) { +for (ref = refs; ref; ref = ref->next) {if (!starts_with (ref->name,"refs/tags/" ))continue ;
@@ -338,7 +338,8 @@ static void find_non_local_tags(struct transport *transport,string_list_clear (&remote_refs,0 ); } -static struct ref *get_ref_map (struct transport *transport, +static struct ref *get_ref_map (struct remote *remote, +const struct ref *remote_refs,struct refspec *rs,int tags,int *autotags) {
@@ -346,27 +347,11 @@ static struct ref *get_ref_map(struct transport *transport,struct ref *rm;struct ref *ref_map =NULL ;struct ref **tail = &ref_map; -struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;/* opportunistically-updated references: */ struct ref *orefs =NULL , **oref_tail = &orefs;struct string_list existing_refs =STRING_LIST_INIT_DUP ; -const struct ref *remote_refs; - -if (rs->nr) -refspec_ref_prefixes (rs, &ref_prefixes); -else if (transport->remote && transport->remote->fetch.nr) -refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); - -if (ref_prefixes.argc && - (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { -argv_array_push (&ref_prefixes,"refs/tags/" ); - } - - remote_refs =transport_get_remote_refs (transport, &ref_prefixes); - -argv_array_clear (&ref_prefixes);if (rs->nr) {struct refspec *fetch_refspec;
@@ -403,7 +388,7 @@ static struct ref *get_ref_map(struct transport *transport,if (refmap.nr) fetch_refspec = &refmap;else - fetch_refspec = &transport->remote->fetch; + fetch_refspec = &remote->fetch;for (i =0 ; i < fetch_refspec->nr; i++)get_fetch_map (ref_map, &fetch_refspec->items[i], &oref_tail,1 );
@@ -411,7 +396,6 @@ static struct ref *get_ref_map(struct transport *transport,die ("--refmap option is only meaningful with command-line refspec(s)." ); }else {/* Use the defaults */ -struct remote *remote = transport->remote;struct branch *branch =branch_get (NULL );int has_merge =branch_has_merge_config (branch); if (remote &&
@@ -450,7 +434,7 @@ static struct ref *get_ref_map(struct transport *transport,/* also fetch all tags */ get_fetch_map (remote_refs, tag_refspec, &tail,0 ); else if (tags == TAGS_DEFAULT && *autotags) - find_non_local_tags(transport, &ref_map, &tail); + find_non_local_tags(remote_refs, &ref_map, &tail);/* Now append any refs to be updated opportunistically: */ *tail = orefs;
@@ -1143,6 +1127,8 @@ static int do_fetch(struct transport *transport,struct ref *ref_map;int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ; +const struct ref *remote_refs; +struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {if (transport->remote->fetch_tags ==2 )
@@ -1158,7 +1144,21 @@ static int do_fetch(struct transport *transport, goto cleanup; } - ref_map =get_ref_map (transport, rs, tags, &autotags); +if (rs->nr) +refspec_ref_prefixes (rs, &ref_prefixes); +else if (transport->remote && transport->remote->fetch.nr) +refspec_ref_prefixes (&transport->remote->fetch, &ref_prefixes); + +if (ref_prefixes.argc && + (tags ==TAGS_SET || (tags ==TAGS_DEFAULT && !rs->nr))) { +argv_array_push (&ref_prefixes,"refs/tags/" ); + } + + remote_refs =transport_get_remote_refs (transport, &ref_prefixes); +argv_array_clear (&ref_prefixes); + + ref_map =get_ref_map (transport->remote, remote_refs, rs, + tags, &autotags);if (!update_head_ok)check_not_current_branch (ref_map);
@@ -1190,7 +1190,7 @@ static int do_fetch(struct transport *transport,if (tags ==TAGS_DEFAULT && autotags) {struct ref **tail = &ref_map; ref_map =NULL ; -find_non_local_tags (transport, &ref_map, &tail); +find_non_local_tags (remote_refs, &ref_map, &tail);if (ref_map)backfill_tags (transport, ref_map);free_refs (ref_map);
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
PATCH v6 7/8 fetch-pack: put shallow info in output parameter
In reply to Brandon WilliamsExpand the transport fetch method signature, by adding an output parameter, to allow transports to return information about the refs they have fetched. Then communicate shallow status information through this mechanism instead of by modifying the input list of refs.
This does require clients to sometimes generate the ref map twice: once from the list of refs provided by the remote (as is currently done) and potentially once from the new list of refs that the fetch mechanism provides.
--- builtin/clone.c | 4 ++-- builtin/fetch.c | 28 ++++++++++++++++++++++++---- fetch-object.c | 2 +- fetch-pack.c | 15 ++++++++------- transport-helper.c | 6 ++++-- transport-internal.h | 9 ++++++++- transport.c | 34 ++++++++++++++++++++++++++++------ transport.h | 3 ++- 8 files changed, 77 insertions(+), 24 deletions(-)
builtin/clone.c +2 -2
@@ -1155,7 +1155,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (!is_local && !complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); remote_head =find_ref_by_name (refs,"HEAD" ); remote_head_points_at =
@@ -1197,7 +1197,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)if (is_local)clone_local (path, git_dir);else if (refs && complete_refs_before_fetch) -transport_fetch_refs (transport, mapped_refs); +transport_fetch_refs (transport, mapped_refs,NULL ); update_remote_refs(refs, mapped_refs, remote_head_points_at, branch_top.buf, reflog_msg.buf, transport,
builtin/fetch.c +24 -4
@@ -946,11 +946,13 @@ static int quickfetch(struct ref *ref_map)return check_connected (iterate_ref_map, &rm, &opt); } -static int fetch_refs (struct transport *transport,struct ref *ref_map) +static int fetch_refs (struct transport *transport,struct ref *ref_map, +struct ref **updated_remote_refs) {int ret =quickfetch (ref_map);if (ret) - ret =transport_fetch_refs (transport, ref_map); + ret =transport_fetch_refs (transport, ref_map, + updated_remote_refs);if (!ret)/* * Keep the new pack's ".keep" file around to allow the caller
@@ -1112,7 +1114,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)transport_set_option (transport,TRANS_OPT_FOLLOWTAGS ,NULL );transport_set_option (transport,TRANS_OPT_DEPTH ,"0" );transport_set_option (transport,TRANS_OPT_DEEPEN_RELATIVE ,NULL ); -if (!fetch_refs (transport, ref_map)) +if (!fetch_refs (transport, ref_map,NULL ))consume_refs (transport, ref_map);if (gsecondary) {
@@ -1128,6 +1130,7 @@ static int do_fetch(struct transport *transport,int autotags = (transport->remote->fetch_tags ==1 );int retcode =0 ;const struct ref *remote_refs; +struct ref *updated_remote_refs =NULL ;struct argv_array ref_prefixes =ARGV_ARRAY_INIT ;if (tags ==TAGS_DEFAULT ) {
@@ -1178,7 +1181,24 @@ static int do_fetch(struct transport *transport, transport->url); } } -if (fetch_refs (transport, ref_map) ||consume_refs (transport, ref_map)) { + +if (fetch_refs (transport, ref_map, &updated_remote_refs)) { +free_refs (ref_map); + retcode =1 ; + goto cleanup; + } +if (updated_remote_refs) { +/* +* Regenerate ref_map using the updated remote refs. This is +* to account for additional information which may be provided +* by the transport (e.g. shallow info). +*/ +free_refs (ref_map); + ref_map =get_ref_map (transport->remote, updated_remote_refs, rs, + tags, &autotags); +free_refs (updated_remote_refs); + } +if (consume_refs (transport, ref_map)) {free_refs (ref_map); retcode =1 ; goto cleanup;
fetch-object.c +1 -1
@@ -19,7 +19,7 @@ static void fetch_refs(const char *remote_name, struct ref *ref)transport_set_option (transport,TRANS_OPT_FROM_PROMISOR ,"1" );transport_set_option (transport,TRANS_OPT_NO_DEPENDENTS ,"1" ); -transport_fetch_refs (transport, ref); +transport_fetch_refs (transport, ref,NULL ); fetch_if_missing = original_fetch_if_missing; }
fetch-pack.c +8 -7
@@ -1470,12 +1470,13 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr) }static void update_shallow (struct fetch_pack_args *args, -struct ref **sought,int nr_sought, +struct ref *refs,struct shallow_info *si) {struct oid_array ref =OID_ARRAY_INIT ;int *status;int i; +struct ref *r;if (args->deepen && alternate_shallow_file) {if (*alternate_shallow_file =='\0' ) {/* --unshallow */
@@ -1517,8 +1518,8 @@ static void update_shallow(struct fetch_pack_args *args,remove_nonexistent_theirs_shallow (si);if (!si->nr_ours && !si->nr_theirs)return ; -for (i =0 ; i < nr_sought; i++) -oid_array_append (&ref, &sought[i]->old_oid); +for (r = refs; r; r = r->next) +oid_array_append (&ref, &r->old_oid); si->ref = &ref;if (args->update_shallow) {
@@ -1552,12 +1553,12 @@ static void update_shallow(struct fetch_pack_args *args,* remote is also shallow, check what ref is safe to update * without updating .git/shallow */ - status =xcalloc (nr_sought,sizeof (*status)); + status =xcalloc (ref.nr,sizeof (*status));assign_shallow_commits_to_refs (si,NULL , status);if (si->nr_ours || si->nr_theirs) { -for (i =0 ; i < nr_sought; i++) +for (r = refs, i =0 ; r; r = r->next, i++)if (status[i]) - sought[i]->status =REF_STATUS_REJECT_SHALLOW ; + r->status =REF_STATUS_REJECT_SHALLOW ; }free (status);oid_array_clear (&ref);
@@ -1591,7 +1592,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args, ref_cpy =do_fetch_pack (args, fd, ref, sought, nr_sought, &si, pack_lockfile);reprepare_packed_git (the_repository); -update_shallow (args, sought, nr_sought, &si); +update_shallow (args, ref_cpy, &si);clear_shallow_info (&si);return ref_cpy; }
transport-helper.c +4 -2
@@ -651,14 +651,16 @@ static int connect_helper(struct transport *transport, const char *name, }static int fetch (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct helper_data *data = transport->data;int i, count;if (process_connect (transport,0 )) {do_take_over (transport); -return transport->vtable->fetch (transport, nr_heads, to_fetch); +return transport->vtable->fetch (transport, nr_heads, to_fetch, + fetched_refs); } count =0 ;
transport-internal.h +8 -1
@@ -36,11 +36,18 @@ struct transport_vtable {* Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. * +* The transport *may* provide, in fetched_refs, the list of refs that +* it fetched. If the transport knows anything about the fetched refs +* that the caller does not know (for example, shallow status), it +* should provide that list of refs and include that information in the +* list. +* * If the transport did not get hashes for refs in * get_refs_list(), it should set the old_sha1 fields in the * provided refs now. **/ -int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs); +int (*fetch)(struct transport *transport,int refs_nr,struct ref **refs, +struct ref **fetched_refs);/** * Push the objects and refs. Send the necessary objects, and
transport.c +28 -6
@@ -151,7 +151,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport, }static int fetch_refs_from_bundle (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {struct bundle_transport_data *data = transport->data; return unbundle(&data->header, data->fd,
@@ -287,7 +288,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus }static int fetch_refs_via_pack (struct transport *transport, -int nr_heads,struct ref **to_fetch) +int nr_heads,struct ref **to_fetch, +struct ref **fetched_refs) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -354,8 +356,12 @@ static int fetch_refs_via_pack(struct transport *transport,if (report_unmatched_refs (to_fetch, nr_heads)) ret =-1 ; +if (fetched_refs) + *fetched_refs = refs; +else +free_refs (refs); +free_refs (refs_tmp); -free_refs (refs);free (dest);return ret; }
@@ -1215,19 +1221,31 @@ const struct ref *transport_get_remote_refs(struct transport *transport,return transport->remote_refs; } -int transport_fetch_refs (struct transport *transport,struct ref *refs) +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs) {int rc;int nr_heads =0 , nr_alloc =0 , nr_refs =0 ;struct ref **heads =NULL ; +struct ref *nop_head =NULL , **nop_tail = &nop_head;struct ref *rm;for (rm = refs; rm; rm = rm->next) { nr_refs++;if (rm->peer_ref && !is_null_oid (&rm->old_oid) && - !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) + !oidcmp (&rm->peer_ref->old_oid, &rm->old_oid)) { +/* +* These need to be reported as fetched, but we don't +* actually need to fetch them. +*/ +if (fetched_refs) { +struct ref *nop_ref =copy_ref (rm); + *nop_tail = nop_ref; + nop_tail = &nop_ref->next; + }continue ; + }ALLOC_GROW (heads, nr_heads +1 , nr_alloc); heads[nr_heads++] = rm; }
@@ -1245,7 +1263,11 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) heads[nr_heads++] = rm; } - rc = transport->vtable->fetch (transport, nr_heads, heads); + rc = transport->vtable->fetch (transport, nr_heads, heads, fetched_refs); +if (fetched_refs && nop_head) { + *nop_tail = *fetched_refs; + *fetched_refs = nop_head; + }free (heads);return rc;
transport.h +2 -1
@@ -218,7 +218,8 @@ int transport_push(struct transport *connection,const struct ref *transport_get_remote_refs (struct transport *transport,const struct argv_array *ref_prefixes); -int transport_fetch_refs (struct transport *transport,struct ref *refs); +int transport_fetch_refs (struct transport *transport,struct ref *refs, +struct ref **fetched_refs);void transport_unlock_pack (struct transport *transport);int transport_disconnect (struct transport *transport);char *transport_anonymize_url (const char *url);
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
PATCH v6 8/8 fetch-pack: implement ref-in-want
In reply to Brandon WilliamsImplement ref-in-want on the client side so that when a server supports the "ref-in-want" feature, a client will send "want-ref" lines for each reference the client wants to fetch. This feature allows clients to tolerate inconsistencies that exist when a remote repository's refs change during the course of negotiation.
This allows a client to request to request a particular ref without specifying the OID of the ref. This means that instead of hitting an error when a ref no longer points at the OID it did at the beginning of negotiation, negotiation can continue and the value of that ref will be sent at the termination of negotiation, just before a packfile is sent.
More information on the ref-in-want feature can be found in Documentation/technical/protocol-v2.txt.
--- fetch-pack.c | 38 +++++++- remote.c | 1 + remote.h | 1 + t/t5703-upload-pack-ref-in-want.sh | 149 +++++++++++++++++++++++++++++ 4 files changed, 186 insertions(+), 3 deletions(-)
fetch-pack.c +35 -3
@@ -1102,9 +1102,10 @@ static void add_shallow_requests(struct strbuf *req_buf,static void add_wants (const struct ref *wants,struct strbuf *req_buf) { +int use_ref_in_want =server_supports_feature ("fetch" ,"ref-in-want" ,0 ); +for ( ; wants ; wants = wants->next) {const struct object_id *remote = &wants->old_oid; -const char *remote_hex;struct object *o;/*
@@ -1122,8 +1123,10 @@ static void add_wants(const struct ref *wants, struct strbuf *req_buf)continue ; } - remote_hex =oid_to_hex (remote); -packet_buf_write (req_buf,"want %s\n" , remote_hex); +if (!use_ref_in_want || wants->exact_oid) +packet_buf_write (req_buf,"want %s\n" ,oid_to_hex (remote)); +else +packet_buf_write (req_buf,"want-ref %s\n" , wants->name); } }
@@ -1334,6 +1337,32 @@ static void receive_shallow_info(struct fetch_pack_args *args, args->deepen =1 ; } +static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line); + +for (r = refs; r; r = r->next) { +if (!strcmp (end, r->name)) { +oidcpy (&r->old_oid, &oid); +break ; + } + } + +if (!r) +die ("unexpected wanted-ref: '%s'" , reader->line); + } + +if (reader->status !=PACKET_READ_DELIM ) +die ("error processing wanted refs: %d" , reader->status); +} +enum fetch_state {FETCH_CHECK_LOCAL =0 ,FETCH_SEND_REQUEST ,
@@ -1408,6 +1437,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,if (process_section_header (&reader,"shallow-info" ,1 ))receive_shallow_info (args, &reader); +if (process_section_header (&reader,"wanted-refs" ,1 )) +receive_wanted_refs (&reader, ref); +/* get the pack */ process_section_header (&reader,"packfile" ,0 );if (get_pack (args, fd, pack_lockfile))
remote.c +1 -0
@@ -1735,6 +1735,7 @@ int get_fetch_map(const struct ref *remote_refs,if (refspec->exact_sha1) { ref_map =alloc_ref (name);get_oid_hex (name, &ref_map->old_oid); + ref_map->exact_oid =1 ; }else { ref_map =get_remote_ref (remote_refs, name); }
remote.h +1 -0
@@ -73,6 +73,7 @@ struct ref { force:1, forced_update:1, expect_old_sha1:1, + exact_oid:1, deletion:1;enum {
t/t5703-upload-pack-ref-in-want.sh +149 -0
161 lines not on this page · raw message
Signature · 1 line
2.18.0.399.gad0ab374a1-goog
Re: [PATCH v6 8/8] fetch-pack: implement ref-in-want
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1349
Quoted 10 lines · Brandon Williams
fetch-pack.c +10
-0+static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line);
Could you do a follow and wrap all these strings in _() since this one is already in 'next'?
Signature · 1 line
Duy
Re: [PATCH v6 8/8] fetch-pack: implement ref-in-want
Quoted 14 lines · Duy Nguyen
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1349
Quoted 10 lines · Brandon Williams
fetch-pack.c +10
-0+static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line);Could you do a follow and wrap all these strings in
_()since this one is already in 'next'?
What criteria is used to determine if something should be translated? To me, this looks like a wire-protocol error which would benefit from not being translated because it would be easier to grep for if it occurs. That and if a user sees this sort of error I don't think that they could really do anything about it anyway.
Of course it appears as if all other 'die' calls in fetch-pack have been marked for translation so I guess my though process doesn't hold :)
Signature · 1 line
Brandon Williams
Re: [PATCH v6 8/8] fetch-pack: implement ref-in-want
Quoted 17 lines · Brandon Williams
Quoted 14 lines · Duy Nguyen
fetch-pack.c static void receive_shallow_info(struct fetch_pack_args *args, · line 1349
Quoted 10 lines · Brandon Williams
fetch-pack.c +10
-0+static void receive_wanted_refs (struct packet_reader *reader,struct ref *refs) +{ +process_section_header (reader,"wanted-refs" ,0 ); +while (packet_reader_read (reader) ==PACKET_READ_NORMAL ) { +struct object_id oid; +const char *end; +struct ref *r =NULL ; + +if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) +die ("expected wanted-ref, got '%s'" , reader->line);Could you do a follow and wrap all these strings in
_()since this one is already in 'next'?
Quoted 5 lines
What criteria is used to determine if something should be translated? To me, this looks like a wire-protocol error which would benefit from not being translated because it would be easier to grep for if it occurs. That and if a user sees this sort of error I don't think that they could really do anything about it anyway.
Devs are users too and for me, even if I can read English just fine, I prefer fully translated interface, not a mix of non-English and English. Users can still google around and find out about wanted-ref (at least linux users 10 years ago did). If they show up here asking for support, we can ask them to translate back if needed (or look into .po files). We have the same problem anyway if their bug reports contain other non-English strings.
Besides drawing the line "benefit from (not) being translated" varies from one developer to another. I think it's just easier and more consistent to stick to "if it's not machine-readable (or really meant for devs, like BUG()), translate it" and leave it to translators to decide.
Signature · 1 line
Duy
Re: [PATCH v6 8/8] fetch-pack: implement ref-in-want
Hi,
Quoted 3 lines · Duy Nguyen
Quoted 1 line · Brandon Williams
What criteria is used to determine if something should be translated?
[...]
Quoted 5 lines
Besides drawing the line "benefit from (not) being translated" varies from one developer to another. I think it's just easier and more consistent to stick to "if it's not machine-readable (or really meant for devs, like
BUG()), translate it" and leave it to translators to decide.
Yup, this is my understanding too: everything user-facing (as opposed to machine-facing) that isn't a "this can't happen" local assertion error (BUG) should be translated.
Protocol and format errors aren't BUG (assertion error) because they reflect something that shouldn't be possible to happen due to someone *else* behaving badly. BUG is for errors that shouldn't be possible due to the version of Git that produces the message behaving badly.
po/README says
- Don't mark everything for translation, only strings which will be read by humans (the porcelain interface) should be translated.
The output from Git's plumbing utilities will primarily be read by programs and would break scripts under non-C locales if it was translated. Plumbing strings should not be translated, since they're part of Git's API.
I kind of wish we had a rule that was less fuzzy, since then we could enforce it. For example, JGit has a rule that *all* strings should be translatable unless they have a comment explaining why they shouldn't be (which goes a little too far --- there are those comments all over the place).
Thanks,
Jonathan
PATCH 1/1 fetch-pack: mark die strings for translation
In reply to Duy Nguyen--- fetch-pack.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
fetch-pack.c +8 -8
@@ -1245,13 +1245,13 @@ static int process_section_header(struct packet_reader *reader,int ret;if (packet_reader_peek (reader) !=PACKET_READ_NORMAL ) -die ("error reading section header '%s'" , section); +die (_ ("error reading section header '%s'" ), section); ret = !strcmp (reader->line, section);if (!peek) {if (!ret) -die ("expected '%s', received '%s'" , +die (_ ("expected '%s', received '%s'" ), section, reader->line);packet_reader_read (reader); }
@@ -1289,12 +1289,12 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)continue ; } -die ("unexpected acknowledgment line: '%s'" , reader->line); +die (_ ("unexpected acknowledgment line: '%s'" ), reader->line); }if (reader->status !=PACKET_READ_FLUSH && reader->status !=PACKET_READ_DELIM ) -die ("error processing acks: %d" , reader->status); +die (_ ("error processing acks: %d" ), reader->status);/* return 0 if no common, 1 if there are common, or 2 if ready */ return received_ready ?2 : (received_ack ?1 :0 );
@@ -1331,7 +1331,7 @@ static void receive_shallow_info(struct fetch_pack_args *args,if (reader->status !=PACKET_READ_FLUSH && reader->status !=PACKET_READ_DELIM ) -die ("error processing shallow info: %d" , reader->status); +die (_ ("error processing shallow info: %d" ), reader->status);setup_alternate_shallow (&shallow_lock, &alternate_shallow_file,NULL ); args->deepen =1 ;
@@ -1346,7 +1346,7 @@ static void receive_wanted_refs(struct packet_reader *reader, struct ref *refs)struct ref *r =NULL ;if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) -die ("expected wanted-ref, got '%s'" , reader->line); +die (_ ("expected wanted-ref, got '%s'" ), reader->line);for (r = refs; r; r = r->next) {if (!strcmp (end, r->name)) {
@@ -1356,11 +1356,11 @@ static void receive_wanted_refs(struct packet_reader *reader, struct ref *refs) }if (!r) -die ("unexpected wanted-ref: '%s'" , reader->line); +die (_ ("unexpected wanted-ref: '%s'" ), reader->line); }if (reader->status !=PACKET_READ_DELIM ) -die ("error processing wanted refs: %d" , reader->status); +die (_ ("error processing wanted refs: %d" ), reader->status); }enum fetch_state {
Signature · 1 line
2.18.0.233.g985f88cf7e-goog
Re: [PATCH] fetch-pack: mark die strings for translation
Quoted 1 line · Brandon Williams
fetch-pack is listed as a plumbing command, which means its prime consumer is supposedly a machine; But fetch-pack is also used by git-fetch that is invoked
by a human, who prefers translations.
.. goes reads code...
This translates protocol v2 messages, and p0 messages are translated already, so this just aligns the new protocol to the old protocol w.r.t. i18n.
Sounds good,
Thanks,
Stefan
[ This message is a gentle hint for better commit messages. ;-) ]
Quoted 75 lines
Signed-off-by Brandon Williams <[email protected]>--- fetch-pack.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)fetch-pack.c +8
-9@@ -1245,13 +1245,13 @@ static int process_section_header(struct packet_reader *reader,int ret;if (packet_reader_peek (reader) !=PACKET_READ_NORMAL ) -die ("error reading section header '%s'" , section); +die (_ ("error reading section header '%s'" ), section); ret = !strcmp (reader->line, section);if (!peek) {if (!ret) -die ("expected '%s', received '%s'" , +die (_ ("expected '%s', received '%s'" ), section, reader->line);packet_reader_read (reader); }@@ -1289,12 +1289,12 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)continue ; } -die ("unexpected acknowledgment line: '%s'" , reader->line); +die (_ ("unexpected acknowledgment line: '%s'" ), reader->line); }if (reader->status !=PACKET_READ_FLUSH && reader->status !=PACKET_READ_DELIM ) -die ("error processing acks: %d" , reader->status); +die (_ ("error processing acks: %d" ), reader->status);/* return 0 if no common, 1 if there are common, or 2 if ready */ return received_ready ?2 : (received_ack ?1 :0 );@@ -1331,7 +1331,7 @@ static void receive_shallow_info(struct fetch_pack_args *args,if (reader->status !=PACKET_READ_FLUSH && reader->status !=PACKET_READ_DELIM ) -die ("error processing shallow info: %d" , reader->status); +die (_ ("error processing shallow info: %d" ), reader->status);setup_alternate_shallow (&shallow_lock, &alternate_shallow_file,NULL ); args->deepen =1 ;@@ -1346,7 +1346,7 @@ static void receive_wanted_refs(struct packet_reader *reader, struct ref *refs)struct ref *r =NULL ;if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) -die ("expected wanted-ref, got '%s'" , reader->line); +die (_ ("expected wanted-ref, got '%s'" ), reader->line);for (r = refs; r; r = r->next) {if (!strcmp (end, r->name)) {@@ -1356,11 +1356,11 @@ static void receive_wanted_refs(struct packet_reader *reader, struct ref *refs) }if (!r) -die ("unexpected wanted-ref: '%s'" , reader->line); +die (_ ("unexpected wanted-ref: '%s'" ), reader->line); }if (reader->status !=PACKET_READ_DELIM ) -die ("error processing wanted refs: %d" , reader->status); +die (_ ("error processing wanted refs: %d" ), reader->status); }enum fetch_state { --2.18.0.233.g985f88cf7e-goog
Re: [PATCH] fetch-pack: mark die strings for translation
In reply to Brandon WilliamsQuoted 4 lines · Brandon Williams
Signed-off-by Brandon Williams <[email protected]>--- fetch-pack.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
Thanks.
Re: [PATCH] fetch-pack: mark die strings for translation
In reply to Brandon WilliamsQuoted 4 lines · Brandon Williams
Signed-off-by Brandon Williams <[email protected]>--- fetch-pack.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
I do think this is a good idea, but what release does this target? It does not seem to apply to master@{4.hours.ago}, and I do not think a change as trivial like this wants to be taken of a hostage of _all_ topics that are in 'next'.
Quoted 67 lines
fetch-pack.c +8
-8@@ -1245,13 +1245,13 @@ static int process_section_header(struct packet_reader *reader,int ret;if (packet_reader_peek (reader) !=PACKET_READ_NORMAL ) -die ("error reading section header '%s'" , section); +die (_ ("error reading section header '%s'" ), section); ret = !strcmp (reader->line, section);if (!peek) {if (!ret) -die ("expected '%s', received '%s'" , +die (_ ("expected '%s', received '%s'" ), section, reader->line);packet_reader_read (reader); }@@ -1289,12 +1289,12 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)continue ; } -die ("unexpected acknowledgment line: '%s'" , reader->line); +die (_ ("unexpected acknowledgment line: '%s'" ), reader->line); }if (reader->status !=PACKET_READ_FLUSH && reader->status !=PACKET_READ_DELIM ) -die ("error processing acks: %d" , reader->status); +die (_ ("error processing acks: %d" ), reader->status);/* return 0 if no common, 1 if there are common, or 2 if ready */ return received_ready ?2 : (received_ack ?1 :0 );@@ -1331,7 +1331,7 @@ static void receive_shallow_info(struct fetch_pack_args *args,if (reader->status !=PACKET_READ_FLUSH && reader->status !=PACKET_READ_DELIM ) -die ("error processing shallow info: %d" , reader->status); +die (_ ("error processing shallow info: %d" ), reader->status);setup_alternate_shallow (&shallow_lock, &alternate_shallow_file,NULL ); args->deepen =1 ;@@ -1346,7 +1346,7 @@ static void receive_wanted_refs(struct packet_reader *reader, struct ref *refs)struct ref *r =NULL ;if (parse_oid_hex (reader->line, &oid, &end) || *end++ !=' ' ) -die ("expected wanted-ref, got '%s'" , reader->line); +die (_ ("expected wanted-ref, got '%s'" ), reader->line);for (r = refs; r; r = r->next) {if (!strcmp (end, r->name)) {@@ -1356,11 +1356,11 @@ static void receive_wanted_refs(struct packet_reader *reader, struct ref *refs) }if (!r) -die ("unexpected wanted-ref: '%s'" , reader->line); +die (_ ("unexpected wanted-ref: '%s'" ), reader->line); }if (reader->status !=PACKET_READ_DELIM ) -die ("error processing wanted refs: %d" , reader->status); +die (_ ("error processing wanted refs: %d" ), reader->status); }enum fetch_state {
Re: [PATCH] fetch-pack: mark die strings for translation
Quoted 7 lines · Junio C Hamano
Quoted 4 lines · Brandon Williams
Signed-off-by Brandon Williams <[email protected]>--- fetch-pack.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
Quoted 5 lines
I do think this is a good idea, but what release does this target? It does not seem to apply to master@{4.hours.ago}, and I do not think a change as trivial like this wants to be taken of a hostage of _all_ topics that are in 'next'.
Answering myself, this would come on top of (and would become part of) bw/ref-in-want topic, I would think. And I am perfectly OK for this change to be hostage of that topic ;-)
Re: [PATCH] fetch-pack: mark die strings for translation
Quoted 15 lines · Junio C Hamano
Quoted 7 lines · Junio C Hamano
Quoted 4 lines · Brandon Williams
Signed-off-by Brandon Williams <[email protected]>--- fetch-pack.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)Quoted 5 lines
I do think this is a good idea, but what release does this target? It does not seem to apply to master@{4.hours.ago}, and I do not think a change as trivial like this wants to be taken of a hostage of _all_ topics that are in 'next'.
Quoted 3 lines
Answering myself, this would come on top of (and would become part of) bw/ref-in-want topic, I would think. And I am perfectly OK for this change to be hostage of that topic ;-)
haha yes sorry for not mention it, but yes I built it on top of bw/ref-in-want because it was in response to this topic :)
Signature · 1 line
Brandon Williams
(replying to the original since my e-mail is about design)
Quoted 4 lines
This version of ref-in-want is a bit more restrictive than what Jonathan originally proposed (only full ref names are allowed instead of globs and OIDs), but it is meant to accomplish the same goal (solve the issues of refs changing during negotiation).
One question remains: are we planning to expand this feature (e.g. to support patterns ending in *, or to support any pattern that can appear on the LHS of a refspec), and if yes, are we OK with having 2 or more versions of the service in the wild, each having different pattern support?
Supporting patterns would mean that we would possibly be able to eliminate the ls-refs step, thus saving at least a RTT. (Originally I thought that supporting patterns would also allow us to tolerate refs being removed during the fetch process, but I see that this is already handled by the server ignoring "want-ref <ref>" wherein <ref> doesn't exist on the server.)
However, after some in-office discussion, I see that eliminating the ls-refs step means that we lose some optimizations that can only be done when we see that we already have a sought remote ref. For example, in a repo like this:
A
|
O
|
O B C
|/ /
O O
|/
O
in which we have rarely-updated branches that we still want to fetch (e.g. an annotated tag when we fetch refs/tags/* or a Gerrit refs/changes/* branch), having the ref advertisement first means that we can omit them from our "want" or "want-ref" list. But not having them means that we send "want-ref refs/tags/*" to the server, and during negotiation inform the server of our master branch (A), and since the server knows of a common ancestor of all our wants (A, B, C), it will terminate the negotiation and send the objects specific to branches B and C even though it didn't need to.
So maybe we still need to keep the ls-refs step around, and thus, this design of only accepting exact refs is perhaps good enough for now.
Quoted 45 lines · Jonathan Tan
(replying to the original since my e-mail is about design)
Quoted 4 lines
This version of ref-in-want is a bit more restrictive than what Jonathan originally proposed (only full ref names are allowed instead of globs and OIDs), but it is meant to accomplish the same goal (solve the issues of refs changing during negotiation).
One question remains: are we planning to expand this feature (e.g. to support patterns ending in *, or to support any pattern that can appear on the LHS of a refspec), and if yes, are we OK with having 2 or more versions of the service in the wild, each having different pattern support?
Supporting patterns would mean that we would possibly be able to eliminate the ls-refs step, thus saving at least a RTT. (Originally I thought that supporting patterns would also allow us to tolerate refs being removed during the fetch process, but I see that this is already handled by the server ignoring "want-ref <ref>" wherein <ref> doesn't exist on the server.)
However, after some in-office discussion, I see that eliminating the ls-refs step means that we lose some optimizations that can only be done when we see that we already have a sought remote ref. For example, in a repo like this:
A
|
O
|
O B C
|/ /
O O
|/
Oin which we have rarely-updated branches that we still want to fetch (e.g. an annotated tag when we fetch refs/tags/* or a Gerrit refs/changes/* branch), having the ref advertisement first means that we can omit them from our "want" or "want-ref" list. But not having them means that we send "want-ref refs/tags/*" to the server, and during negotiation inform the server of our master branch (A), and since the server knows of a common ancestor of all our wants (A, B, C), it will terminate the negotiation and send the objects specific to branches B and C even though it didn't need to.
Quoted 2 lines
So maybe we still need to keep the ls-refs step around, and thus, this design of only accepting exact refs is perhaps good enough for now.
I think that taking a smaller step first it probably better. This is something that we've done in the past with the shallow features and later capabilities were added to add different ways to request shallow fetches.
That being said, if we find that this feature doesn't work as-is and needs the extra complexity of patterns from the start then they should be added. But it doesn't seem like there's a concrete reason at the moment.
Signature · 1 line
Brandon Williams
[snip]
Quoted 10 lines
Quoted 10 lines
in which we have rarely-updated branches that we still want to fetch (e.g. an annotated tag when we fetch refs/tags/* or a Gerrit refs/changes/* branch), having the ref advertisement first means that we can omit them from our "want" or "want-ref" list. But not having them means that we send "want-ref refs/tags/*" to the server, and during negotiation inform the server of our master branch (A), and since the server knows of a common ancestor of all our wants (A, B, C), it will terminate the negotiation and send the objects specific to branches B and C even though it didn't need to.
Quoted 7 lines
Quoted 2 lines
So maybe we still need to keep the ls-refs step around, and thus, this design of only accepting exact refs is perhaps good enough for now.
I think that taking a smaller step first it probably better. This is something that we've done in the past with the shallow features and later capabilities were added to add different ways to request shallow fetches.
I think we're agreeing that the smaller step first is better.
Quoted 3 lines
That being said, if we find that this feature doesn't work as-is and needs the extra complexity of patterns from the start then they should be added.
I agree (although I would be OK too if we decide to do the small exact-name step now and then the pattern step later guarded by a capability, as long as the project understood that multiple support levels would then exist in the wild).
Quoted 2 lines
But it doesn't seem like there's a concrete reason at the moment.
I agree. I thought I had a reason, but not after thinking through the ideas I explained in [1].
Quoted 7 lines · Jonathan Tan
Supporting patterns would mean that we would possibly be able to eliminate the ls-refs step, thus saving at least a RTT. (Originally I thought that supporting patterns would also allow us to tolerate refs being removed during the fetch process, but I see that this is already handled by the server ignoring "want-ref <ref>" wherein <ref> doesn't exist on the server.)
What's your opinion on this? Should we keep it how it is in v2 of the series where the server ignores refs it doesn't know about or revert to what v1 of the series did and have it be a hard error?
I've gone back and forth on what I think we should do so I'd like to hear at least one more opinion :)
Signature · 1 line
Brandon Williams
Quoted 9 lines · Jonathan Tan
Quoted 7 lines
Supporting patterns would mean that we would possibly be able to eliminate the ls-refs step, thus saving at least a RTT. (Originally I thought that supporting patterns would also allow us to tolerate refs being removed during the fetch process, but I see that this is already handled by the server ignoring "want-ref <ref>" wherein <ref> doesn't exist on the server.)
Quoted 3 lines
What's your opinion on this? Should we keep it how it is in v2 of the series where the server ignores refs it doesn't know about or revert to what v1 of the series did and have it be a hard error?
I think it should be like in v2 - the server should ignore "want-ref <ref>" lines for refs it doesn't know about. And, after more thought, I think that the client should die if "fetch <exact-ref-name>" was not fulfilled, and ignore if a ref in "fetch <ref-with-wildcard>" was not fulfilled.
The advantage of doing that is that we make the protocol a bit more tolerant to adverse conditions (e.g. a rapidly changing repository or an eventually consistent load-balancing setup), while having little-to-no effect on regular conditions.
The disadvantage is that there is now one additional place where a failure can silently occur, but I think that this is a minor disadvantage. A naive script using "git fetch", in my mind, would assume that refs/heads/exact exists if "fetch refs/heads/exact:refs/heads/exact" succeeds, but would not assume that refs/heads/wildcard-something exists if "fetch refs/heads/wildcard*:refs/heads/wildcard*" succeeds, which fits in nicely with the die/ignore behavior I outlined above.