Add proto v2 archive command with HTTP support
This series adds a new protocol v2 command for archiving, and allows this command to work over HTTP(S). This was previously discussed in [1]. I've CCed everyone who participated in that discussion.
[1]: https://public-inbox.org/git/CANq=j3tK7QeBJOC7VNWkh4+WBNibMJJp5YUkd9te5NaYwukAow@mail.gmail.com/
PATCH 1/3 archive: use packet_reader for communications
Using packet_reader will simplify version detection and capability handling, which will make implementation of protocol v2 support in git-archive easier.
--- builtin/archive.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
builtin/archive.c +14 -9
@@ -27,10 +27,11 @@ static int run_remote_archiver(int argc, const char **argv, const char *remote, const char *exec, const char *name_hint) { - char *buf; int fd[2], i, rv;struct transport *transport;struct remote *_remote; +struct packet_reader reader; +enum packet_read_status status; _remote =remote_get (remote);if (!_remote->url[0 ])
@@ -38,6 +39,8 @@ static int run_remote_archiver(int argc, const char **argv, transport =transport_get (_remote, _remote->url[0 ]);transport_connect (transport,"git-upload-archive" , exec, fd); +packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); +/* * Inject a fake --format field at the beginning of the * arguments, with the format inferred from our output
@@ -53,18 +56,20 @@ static int run_remote_archiver(int argc, const char **argv,packet_write_fmt (fd[1 ],"argument %s\n" , argv[i]);packet_flush (fd[1 ]); - buf =packet_read_line (fd[0 ],NULL ); -if (!buf) + status =packet_reader_read (&reader); + +if (status ==PACKET_READ_FLUSH )die (_ ("git archive: expected ACK/NAK, got a flush packet" )); -if (strcmp (buf,"ACK" )) { -if (starts_with (buf,"NACK " )) -die (_ ("git archive: NACK %s" ), buf +5 ); -if (starts_with (buf,"ERR " )) -die (_ ("remote error: %s" ), buf +4 ); +if (strcmp (reader.buffer,"ACK" )) { +if (starts_with (reader.buffer,"NACK " )) +die (_ ("git archive: NACK %s" ), reader.buffer +5 ); +if (starts_with (reader.buffer,"ERR " )) +die (_ ("remote error: %s" ), reader.buffer +4 );die (_ ("git archive: protocol error" )); } -if (packet_read_line (fd[0 ],NULL )) + status =packet_reader_read (&reader); +if (status !=PACKET_READ_FLUSH )die (_ ("git archive: expected a flush" ));/* Now, start reading from fd[0] and spit it out to stdout */
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Re: [PATCH 1/3] archive: use packet_reader for communications
Quoted 6 lines · Josh Steadmon
Using
packet_readerwill simplify version detection and capability handling, which will make implementation of protocol v2 support in git-archive easier.Signed-off-by Josh Steadmon <[email protected]>
Re: [PATCH 1/3] archive: use packet_reader for communications
In reply to Josh SteadmonQuoted 3 lines · Josh Steadmon
Using
packet_readerwill simplify version detection and capability handling, which will make implementation of protocol v2 support in git-archive easier.
Is this meant as a change in implementation without any change in behaviour?
Quoted 32 lines
Signed-off-by Josh Steadmon <[email protected]>--- builtin/archive.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)builtin/archive.c +4
-1@@ -27,10 +27,11 @@ static int run_remote_archiver(int argc, const char **argv, const char *remote, const char *exec, const char *name_hint) { - char *buf; int fd[2], i, rv;struct transport *transport;struct remote *_remote; +struct packet_reader reader; +enum packet_read_status status; _remote =remote_get (remote);if (!_remote->url[0 ])@@ -38,6 +39,8 @@ static int run_remote_archiver(int argc, const char **argv, transport =transport_get (_remote, _remote->url[0 ]);transport_connect (transport,"git-upload-archive" , exec, fd); +packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); +/* * Inject a fake --format field at the beginning of the * arguments, with the format inferred from our output
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 62
Quoted 9 lines
builtin/archive.c +3
-2@@ -53,18 +56,20 @@ static int run_remote_archiver(int argc, const char **argv,packet_write_fmt (fd[1 ],"argument %s\n" , argv[i]);packet_flush (fd[1 ]); - buf =packet_read_line (fd[0 ],NULL ); -if (!buf) + status =packet_reader_read (&reader); + +if (status ==PACKET_READ_FLUSH )die (_ ("git archive: expected ACK/NAK, got a flush packet" ));
It is true that packet_read_line() returns a NULL on flush, but the function also returns NULL on other conditions for which underlying packet_read() returns 0 (or negative) length. EOF, normal data with zero length (i.e. packet length itself is 4), and DELIM packets would all have led to this die() in the original code. We fail to notice that we got something unexpected when we were expecting to get a normal packet with ACK or NACK on it.
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 63
Quoted 6 lines
builtin/archive.c +1
-5-if (strcmp (buf,"ACK" )) { -if (starts_with (buf,"NACK " )) -die (_ ("git archive: NACK %s" ), buf +5 ); -if (starts_with (buf,"ERR " )) -die (_ ("remote error: %s" ), buf +4 ); +if (strcmp (reader.buffer,"ACK" )) {
The way I read packet_reader_read()'s implementation and existing callers (like the ones in fetch-pack.c) tells me that consumers should not be looking at "reader.buffer"; they should instead be reading from "reader.line".
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 73
Quoted 11 lines
builtin/archive.c +6
-1+if (starts_with (reader.buffer,"NACK " )) +die (_ ("git archive: NACK %s" ), reader.buffer +5 ); +if (starts_with (reader.buffer,"ERR " )) +die (_ ("remote error: %s" ), reader.buffer +4 );die (_ ("git archive: protocol error" )); } -if (packet_read_line (fd[0 ],NULL )) + status =packet_reader_read (&reader); +if (status !=PACKET_READ_FLUSH )die (_ ("git archive: expected a flush" ));
This makes me wonder what happens if we got an EOF instead. We fail to notice protocol error here, but do the code after this part correctly handle the situation?
Quoted 1 line
builtin/archive.c +0
-0/* Now, start reading from fd[0] and spit it out to stdout */
Re: [PATCH 1/3] archive: use packet_reader for communications
Quoted 8 lines · Junio C Hamano
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 73
Quoted 4 lines
builtin/archive.c +2
-1-if (packet_read_line (fd[0 ],NULL )) + status =packet_reader_read (&reader); +if (status !=PACKET_READ_FLUSH )die (_ ("git archive: expected a flush" ));This makes me wonder what happens if we got an EOF instead. We fail to notice protocol error here, but do the code after this part correctly handle the situation?
Sorry, this part of my comment is completely backwards.
We require they send a flush, not a 0-length data packet of length 4, and otherwise we die, even though we used to treate 0-length data packet of length 4 just like a flush.
So this is making the code more strict than the original. As long as all the existing implementations correctly use flush here, there would be no unintended regression, but it bothers me that we have to even worry if these behaviour changes affect the already deployed software negatively.
Quoted 2 lines
Quoted 2 lines
/* Now, start reading from fd[0] and spit it out to stdout */
PATCH 2/3 archive: implement protocol v2 archive command
In reply to Josh SteadmonThis adds a new archive command for protocol v2. The command expects arguments in the form "argument X" which are passed unmodified to git-upload-archive--writer.
This command works over the file://, Git, and SSH transports. HTTP support will be added in a separate patch.
--- builtin/archive.c | 45 +++++++++++++++++++++++++++------------- builtin/upload-archive.c | 44 ++++++++++++++++++++++++++++++++++++--- t/t5000-tar-tree.sh | 5 +++++ 3 files changed, 77 insertions(+), 17 deletions(-)
builtin/archive.c +31 -14
@@ -5,9 +5,11 @@#include "cache.h" #include "builtin.h" #include "archive.h" +#include "connect.h" #include "transport.h" #include "parse-options.h" #include "pkt-line.h" +#include "protocol.h" #include "sideband.h" static void create_output_file (const char *output_file)
@@ -23,6 +25,13 @@ static void create_output_file(const char *output_file) } } +static int do_v2_command_and_cap (int out) +{ +packet_write_fmt (out,"command=archive\n" ); +/* Capability list would go here, if we had any. */ +packet_delim (out); +} +static int run_remote_archiver (int argc,const char **argv,const char *remote,const char *exec,const char *name_hint)
@@ -32,6 +41,7 @@ static int run_remote_archiver(int argc, const char **argv,struct remote *_remote;struct packet_reader reader;enum packet_read_status status; +enum protocol_version version; _remote =remote_get (remote);if (!_remote->url[0 ])
@@ -41,6 +51,11 @@ static int run_remote_archiver(int argc, const char **argv,packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); + version =discover_version (&reader); + +if (version == protocol_v2) +do_v2_command_and_cap (fd[1 ]); +/* * Inject a fake --format field at the beginning of the * arguments, with the format inferred from our output
@@ -56,22 +71,24 @@ static int run_remote_archiver(int argc, const char **argv,packet_write_fmt (fd[1 ],"argument %s\n" , argv[i]);packet_flush (fd[1 ]); - status =packet_reader_read (&reader); - -if (status ==PACKET_READ_FLUSH ) -die (_ ("git archive: expected ACK/NAK, got a flush packet" )); -if (strcmp (reader.buffer,"ACK" )) { -if (starts_with (reader.buffer,"NACK " )) -die (_ ("git archive: NACK %s" ), reader.buffer +5 ); -if (starts_with (reader.buffer,"ERR " )) -die (_ ("remote error: %s" ), reader.buffer +4 ); -die (_ ("git archive: protocol error" )); +if (version == protocol_v0) { + status =packet_reader_read (&reader); + +if (status ==PACKET_READ_FLUSH ) +die (_ ("git archive: expected ACK/NAK, got a flush packet" )); +if (strcmp (reader.buffer,"ACK" )) { +if (starts_with (reader.buffer,"NACK " )) +die (_ ("git archive: NACK %s" ), reader.buffer +5 ); +if (starts_with (reader.buffer,"ERR " )) +die (_ ("remote error: %s" ), reader.buffer +4 ); +die (_ ("git archive: protocol error" )); + } + + status =packet_reader_read (&reader); +if (status !=PACKET_READ_FLUSH ) +die (_ ("git archive: expected a flush" )); } - status =packet_reader_read (&reader); -if (status !=PACKET_READ_FLUSH ) -die (_ ("git archive: expected a flush" )); -/* Now, start reading from fd[0] and spit it out to stdout */ rv =recv_sideband ("archive" , fd[0 ],1 ); rv |=transport_disconnect (transport);
builtin/upload-archive.c +41 -3
69 lines not on this page · raw message
t/t5000-tar-tree.sh +5 -0
@@ -145,6 +145,11 @@ test_expect_success \ check_tar b +test_expect_success 'protocol v2 for remote' ' + GIT_PROTOCOL="version=2" git archive --remote=. HEAD >v2_remote.tar +' +check_tar v2_remote + test_expect_success 'git archive --prefix=prefix/' ' git archive --prefix=prefix/ HEAD >with_prefix.tar '
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Re: [PATCH 2/3] archive: implement protocol v2 archive command
Quoted 69 lines · Josh Steadmon
This adds a new archive command for protocol v2. The command expects arguments in the form "argument X" which are passed unmodified to git-upload-archive--writer.
This command works over the file://, Git, and SSH transports. HTTP support will be added in a separate patch.
Signed-off-by Josh Steadmon <[email protected]>--- builtin/archive.c | 45 +++++++++++++++++++++++++++------------- builtin/upload-archive.c | 44 ++++++++++++++++++++++++++++++++++++--- t/t5000-tar-tree.sh | 5 +++++ 3 files changed, 77 insertions(+), 17 deletions(-)builtin/archive.c +15
-0@@ -5,9 +5,11 @@#include "cache.h" #include "builtin.h" #include "archive.h" +#include "connect.h" #include "transport.h" #include "parse-options.h" #include "pkt-line.h" +#include "protocol.h" #include "sideband.h" static void create_output_file (const char *output_file)@@ -23,6 +25,13 @@ static void create_output_file(const char *output_file) } } +static int do_v2_command_and_cap (int out) +{ +packet_write_fmt (out,"command=archive\n" ); +/* Capability list would go here, if we had any. */ +packet_delim (out); +} +static int run_remote_archiver (int argc,const char **argv,const char *remote,const char *exec,const char *name_hint)@@ -32,6 +41,7 @@ static int run_remote_archiver(int argc, const char **argv,struct remote *_remote;struct packet_reader reader;enum packet_read_status status; +enum protocol_version version; _remote =remote_get (remote);if (!_remote->url[0 ])@@ -41,6 +51,11 @@ static int run_remote_archiver(int argc, const char **argv,packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); + version =discover_version (&reader); + +if (version == protocol_v2) +do_v2_command_and_cap (fd[1 ]); +/* * Inject a fake --format field at the beginning of the * arguments, with the format inferred from our output @@ -56,22 +71,24 @@ static int run_remote_archiver(int argc, const char **argv,packet_write_fmt (fd[1 ],"argument %s\n" , argv[i]);packet_flush (fd[1 ]);
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 68
Quoted 10 lines
builtin/archive.c +0
-10- status =packet_reader_read (&reader); - -if (status ==PACKET_READ_FLUSH ) -die (_ ("git archive: expected ACK/NAK, got a flush packet" )); -if (strcmp (reader.buffer,"ACK" )) { -if (starts_with (reader.buffer,"NACK " )) -die (_ ("git archive: NACK %s" ), reader.buffer +5 ); -if (starts_with (reader.buffer,"ERR " )) -die (_ ("remote error: %s" ), reader.buffer +4 ); -die (_ ("git archive: protocol error" ));
Maybe we also want to support v1
(which is v0 prefixed with one pkt_line saying it is v1).
If (version == protocol_v1)
/* drop version v1 line, and then follow v0 logic. */
packet_reader_read(&reader);Do we care about v1, or do we just ignore it here? why? (Don't answer me here, but rather put it in the commit message)
Quoted 52 lines
builtin/archive.c +16
-4+if (version == protocol_v0) { + status =packet_reader_read (&reader); + +if (status ==PACKET_READ_FLUSH ) +die (_ ("git archive: expected ACK/NAK, got a flush packet" )); +if (strcmp (reader.buffer,"ACK" )) { +if (starts_with (reader.buffer,"NACK " )) +die (_ ("git archive: NACK %s" ), reader.buffer +5 ); +if (starts_with (reader.buffer,"ERR " )) +die (_ ("remote error: %s" ), reader.buffer +4 ); +die (_ ("git archive: protocol error" )); + } + + status =packet_reader_read (&reader); +if (status !=PACKET_READ_FLUSH ) +die (_ ("git archive: expected a flush" )); } - status =packet_reader_read (&reader); -if (status !=PACKET_READ_FLUSH ) -die (_ ("git archive: expected a flush" )); -/* Now, start reading from fd[0] and spit it out to stdout */ rv =recv_sideband ("archive" , fd[0 ],1 ); rv |=transport_disconnect (transport);builtin/upload-archive.c +12
-0@@ -5,6 +5,7 @@#include "builtin.h" #include "archive.h" #include "pkt-line.h" +#include "protocol.h" #include "sideband.h" #include "run-command.h" #include "argv-array.h" @@ -73,13 +74,53 @@ static ssize_t process_input(int child_fd, int band)return sz; } +static int handle_v2_command_and_cap (void ) +{ +struct packet_reader reader; +enum packet_read_status status; + +packet_reader_init (&reader,0 ,NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); + +packet_write_fmt (1 ,"version 2\n" ); +/* +* We don't currently send any capabilities, but maybe we could list +* supported archival formats?
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 97
Quoted 10 lines
builtin/upload-archive.c +10
-0+*/ +packet_flush (1 ); + + status =packet_reader_read (&reader); +if (status !=PACKET_READ_NORMAL || +strcmp (reader.buffer,"command=archive" )) +die (_ ("upload-archive: expected command=archive" )); +while (status ==PACKET_READ_NORMAL ) { +/* We don't currently expect any client capabilities, but we +* should still read (and ignore) any that happen to get sent.
/*
* Makes sense to ignore the client capabilities here,
* but the multi line comments take their opening
* and closing line on a separate line. just like above. */
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 102
Quoted 5 lines
builtin/upload-archive.c +5
-0+*/ + status =packet_reader_read (&reader); + } +if (status !=PACKET_READ_DELIM ) +die (_ ("upload-archive: expected delim packet" ));
This is upload-archive, which is a low level plumbing command (see the main man page of git for an explanation of that category), so we do not translate the error/die() calls. Besides, this is executed on the server, which might have a different locale than the requesting client?
Would asking for a setlocale() on the server side be an unreasonable feature request for the capabilities (in a follow up patch, and then not just for archive but also fetch/push, etc.)?
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 119
Quoted 11 lines
builtin/upload-archive.c +4
-0int cmd_upload_archive (int argc,const char **argv,const char *prefix) {struct child_process writer = { argv }; +enum protocol_version version =determine_protocol_version_server ();if (argc ==2 && !strcmp (argv[1 ],"-h" ))usage (upload_archive_usage); +if (version == protocol_v2) +handle_v2_command_and_cap (); +else {
So if the client asked for v1, we still fall back to v0 here, which answers my question above.
Quoted 22 lines
builtin/upload-archive.c +4
-3+packet_write_fmt (1 ,"ACK\n" ); +packet_flush (1 ); + } +/* * Set up sideband subprocess. * @@ -96,9 +137,6 @@ int cmd_upload_archive(int argc, const char **argv, const char *prefix)die ("upload-archive: %s" ,strerror (err)); } -packet_write_fmt (1 ,"ACK\n" ); -packet_flush (1 ); -while (1 ) {struct pollfd pfd[2 ];t/t5000-tar-tree.sh +0
-0
t/t5000-tar-tree.sh test_expect_success \ · line 151
Quoted 7 lines
t/t5000-tar-tree.sh +4
-0@@ -145,6 +145,11 @@ test_expect_success \ check_tar b +test_expect_success 'protocol v2 for remote' ' + GIT_PROTOCOL="version=2" git archive --remote=. HEAD >v2_remote.tar +' +check_tar v2_remote
Our current standard is to keep all executions inside a test_expect_* block, but here it is hard to comply with that as the check_tar function contains test_expect_* and calling test_expect_* from within itself doesn't work with our test suite.
So bonus points for a refactoring to bring t5000 up to our current standard (c.f. t0020 for a reasonable new code, and t2002 for older code, though that only covers syntax, not functions)
The check itself is just testing that giving GIT_PROTOCOL=2 in the environment also let's you obtain an archive. It doesn't test if the actual communication *is* v2.
See 5e3548ef161 (fetch: send server options when using protocol v2, 2018-04-23) for an example how to sniff on the network traffic in tests, i.e. use GIT_TRACE_PACKET=... and grep on that?
Thanks,
Stefan
Re: [PATCH 2/3] archive: implement protocol v2 archive command
Quoted 13 lines · Stefan Beller
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 102
Quoted 5 lines · Josh Steadmon
builtin/upload-archive.c +5
-0+*/ + status =packet_reader_read (&reader); + } +if (status !=PACKET_READ_DELIM ) +die (_ ("upload-archive: expected delim packet" ));This is upload-archive, which is a low level plumbing command (see the main man page of git for an explanation of that category), so we do not translate the error/die() calls. Besides, this is executed on the server, which might have a different locale than the requesting client?
Quoted 3 lines
Would asking for a
setlocale()on the server side be an unreasonable feature request for the capabilities (in a follow up patch, and then not just for archive but also fetch/push, etc.)?
This would be very nice to have, but as you suggest in some follow-up change.
I think though that instead of doing setlocale() it would be better to pass some flag saying we're operating in a machine-readable mode, and then we'd (as part of the protocol defintion) say we're going to emit GIT_ERR_UPLOAD_ARCHIVE_EXPECTED_DELIM_PACKET or whatever.
Advantages of doing that over a server-side setlocale():
1) Purely for translation purposes, users can update to a newer client to get new translations, even though they're talking to an old server.
2) Again, only for translation purposes, servers may not have the appropriate locales generated and/or linked to libgettext.
3) Ditto, some clients that aren't git.git may want/need to emit different translation messages to their consumers than what we have, think some GUI client / Emacs magit etc. whose UI is different from ours.
4) Aside from translation purposes, getting a machine-readable "push/pull" etc. mode would be very handy. E.g. now you need to parse stderr to see why exactly your push failed (hook denied, or non-fast-forward, or non-fast-forward where there was a lock race condition? ...).
I also wonder if something like #4 wouldn't compliment something like the proposed structured logging[1]. I.e. even though we'd like to run git.git, and present exactly the message to the user we do now, we might want to run in such a machine-readable mode under the hood when talking to the server so we can log exactly how the push went / how it failed for the purposes of aggregation.
Re: [PATCH 2/3] archive: implement protocol v2 archive command
Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 3 lines · Stefan Beller
Would asking for a
setlocale()on the server side be an unreasonable feature request for the capabilities (in a follow up patch, and then not just for archive but also fetch/push, etc.)?This would be very nice to have, but as you suggest in some follow-up change.
Indeed, I think we've gone pretty far afield from the goal of this patch series.
Quoted 4 lines
I think though that instead of doing
setlocale()it would be better to pass some flag saying we're operating in a machine-readable mode, and then we'd (as part of the protocol defintion) say we're going to emitGIT_ERR_UPLOAD_ARCHIVE_EXPECTED_DELIM_PACKETor whatever.
I think you're suggesting client-side message generation, and that is one way to handle internationalization of server output.
The main downside is when the server really does want to provide a custom error message. For that, we'd need
1. To propagate LANG to the server, so it knows what human language to generate messages in.
2. On the server side, to produce messages in that language if available, with an appropriate fallback if not.
We've been thinking of doing at least (1) using the same trick as server-options use (cramming it into client capabilities).
It is difficult to use setlocale for this because it affects the whole program (problematic for a threaded server) and affects features like collation order instead of just message generation (problematic for many things). Does gettext have a variant that takes a locale_t argument?
[...]
Quoted 5 lines
4) Aside from translation purposes, getting a machine-readable "push/pull" etc. mode would be very handy. E.g. now you need to parse stderr to see why exactly your push failed (hook denied, or non-fast-forward, or non-fast-forward where there was a lock race condition? ...).
Indeed, this is a good reason to provide error codes instead of (in the case where the message doesn't add anything to it) or alongside (in case the error message is more specialized) human-oriented error messages.
Thanks,
Jonathan
Re: [PATCH 2/3] archive: implement protocol v2 archive command
Quoted 18 lines · Jonathan Nieder
Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 3 lines · Stefan Beller
Would asking for a
setlocale()on the server side be an unreasonable feature request for the capabilities (in a follow up patch, and then not just for archive but also fetch/push, etc.)?This would be very nice to have, but as you suggest in some follow-up change.
Indeed, I think we've gone pretty far afield from the goal of this patch series.
Quoted 4 lines
I think though that instead of doing
setlocale()it would be better to pass some flag saying we're operating in a machine-readable mode, and then we'd (as part of the protocol defintion) say we're going to emitGIT_ERR_UPLOAD_ARCHIVE_EXPECTED_DELIM_PACKETor whatever.
Quoted 5 lines
I think you're suggesting client-side message generation, and that is one way to handle internationalization of server output.
The main downside is when the server really does want to provide a custom error message. For that, we'd need
Yeah you can't do it for everything. E.g. hooks will want to spew out custom messages, and this hypothetical protocol extension won't have codes for those. So you'll still need to pass $LANG along.
Quoted 6 lines
1. To propagate LANG to the server, so it knows what human language to generate messages in.
2. On the server side, to produce messages in that language if available, with an appropriate fallback if not.
Quoted 8 lines
We've been thinking of doing at least (1) using the same trick as server-options use (cramming it into client capabilities).
It is difficult to use setlocale for this because it affects the whole program (problematic for a threaded server) and affects features like collation order instead of just message generation (problematic for many things). Does gettext have a variant that takes a
locale_targument?
No, its API is fairly crappy and depends on setlocale().
Keep in mind though that we're not tied to that API. E.g. one way to work around this problem is to simply loop through all the languages we have translations for at server startup, for each one call setlocale() and gettext(), and save the result in a hash table for runtime lookup, then you'd just call sprintf(hash[language][message_id], ...) at runtime.
That's all libintl is really doing under the hood, in a roundabout way where calls to setlocale() determine what table we're looking things up in.
The reason I opted to go for gettext to begin with was mainly a) it was there b) the ubiquitous availability of tooling for translators when it comes to the *.po files.
But the API for looking things up at runtime is fairly small, and easy to replace. We could e.g. replace all of our own gettext.[ch] wrapper with something that works somewhat like what I described above, with some extra build step to extract the relevant data from the *.mo files (or parse the *.po directly).
Quoted 11 lines
[...]
Quoted 5 lines
4) Aside from translation purposes, getting a machine-readable "push/pull" etc. mode would be very handy. E.g. now you need to parse stderr to see why exactly your push failed (hook denied, or non-fast-forward, or non-fast-forward where there was a lock race condition? ...).
Indeed, this is a good reason to provide error codes instead of (in the case where the message doesn't add anything to it) or alongside (in case the error message is more specialized) human-oriented error messages.
Re: [PATCH 2/3] archive: implement protocol v2 archive command
In reply to Jonathan NiederQuoted 8 lines · Jonathan Nieder
Quoted 4 lines
I think though that instead of doing
setlocale()it would be better to pass some flag saying we're operating in a machine-readable mode, and then we'd (as part of the protocol defintion) say we're going to emitGIT_ERR_UPLOAD_ARCHIVE_EXPECTED_DELIM_PACKETor whatever.I think you're suggesting client-side message generation, and that is one way to handle internationalization of server output.
Quoted 8 lines
The main downside is when the server really does want to provide a custom error message. For that, we'd need
1. To propagate LANG to the server, so it knows what human language to generate messages in.
2. On the server side, to produce messages in that language if available, with an appropriate fallback if not.
That is one way to do so, but it does not have to be the only way, I would think. You can send a machine parsable message in pieces, and assemble the parts of speech into a message at the receiving end. Like sending a msgid to identify an entry in the .pot file, and values to be filled in.
Re: [PATCH 2/3] archive: implement protocol v2 archive command
Quoted 16 lines · Junio C Hamano
Quoted 14 lines · Jonathan Nieder
Quoted 4 lines
I think though that instead of doing
setlocale()it would be better to pass some flag saying we're operating in a machine-readable mode, and then we'd (as part of the protocol defintion) say we're going to emitGIT_ERR_UPLOAD_ARCHIVE_EXPECTED_DELIM_PACKETor whatever.I think you're suggesting client-side message generation, and that is one way to handle internationalization of server output.
The main downside is when the server really does want to provide a custom error message. For that, we'd need
1. To propagate LANG to the server, so it knows what human language to generate messages in.
Quoted 8 lines
Quoted 2 lines
2. On the server side, to produce messages in that language if available, with an appropriate fallback if not.
That is one way to do so, but it does not have to be the only way, I would think. You can send a machine parsable message in pieces, and assemble the parts of speech into a message at the receiving end. Like sending a msgid to identify an entry in the .pot file, and values to be filled in.
That works if the same party controls the client and server and the client is up to date enough to know about every message the server would want to send.
It doesn't work for
- hooks
- alternate server implementations
- messages involved in an emergency fix
- ... etc ...
Don't get me wrong: for messages with a machine as an audience, error codes or similar structured errors are a great way to go, and getting client-side generation of messages for humans (not to mention styling, etc) are a nice bonus there. I stand by what's in the message you're replying to, though: if we actually want to be able to consistently provide useful messages to people who do not like to read English, then client-side generation won't get us all the way there.
Jonathan
Re: [PATCH 2/3] archive: implement protocol v2 archive command
In reply to Josh SteadmonQuoted 19 lines · Josh Steadmon
builtin/archive.c +8
-0+static int do_v2_command_and_cap (int out) +{ +packet_write_fmt (out,"command=archive\n" ); +/* Capability list would go here, if we had any. */ +packet_delim (out); +} +static int run_remote_archiver (int argc,const char **argv,const char *remote,const char *exec,const char *name_hint)@@ -32,6 +41,7 @@ static int run_remote_archiver(int argc, const char **argv,struct remote *_remote;struct packet_reader reader;enum packet_read_status status; +enum protocol_version version; _remote =remote_get (remote);if (!_remote->url[0 ])
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 54
Quoted 4 lines
builtin/archive.c +1
-0@@ -41,6 +51,11 @@ static int run_remote_archiver(int argc, const char **argv,packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); + version =discover_version (&reader);
The original version of upload-archive that is correctly running on the other end sends either NACK (unable to spawn) or ACK (ready to serve) to us without waiting for us to speak first, so peeking that with this discover_version() is a safe thing to do.
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 58
Quoted 3 lines
builtin/archive.c +3
-0+if (version == protocol_v2) +do_v2_command_and_cap (fd[1 ]); +
With proto v2, "server capabilities" have already been collected in server_capabilities_v2 array in discover_version(). We are to pick and ask the capabilities in that function and respond. Right now we do not need to do much, as we saw that very thin implementation of that function above.
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 61
Quoted 3 lines
builtin/archive.c +0
-0/* * Inject a fake --format field at the beginning of the * arguments, with the format inferred from our output
And then after that, both the original and updated protocol lets us send the archive format and arguments (like revs and pathspecs), followed by a flush packet...
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 72
Quoted 3 lines
builtin/archive.c +0
-0@@ -56,22 +71,24 @@ static int run_remote_archiver(int argc, const char **argv,packet_write_fmt (fd[1 ],"argument %s\n" , argv[i]);packet_flush (fd[1 ]);
... which is a piece of code shared between the protocol versions that ends here.
Quoted 17 lines
builtin/archive.c +7
-10- status =packet_reader_read (&reader); - -if (status ==PACKET_READ_FLUSH ) -die (_ ("git archive: expected ACK/NAK, got a flush packet" )); -if (strcmp (reader.buffer,"ACK" )) { -if (starts_with (reader.buffer,"NACK " )) -die (_ ("git archive: NACK %s" ), reader.buffer +5 ); -if (starts_with (reader.buffer,"ERR " )) -die (_ ("remote error: %s" ), reader.buffer +4 ); -die (_ ("git archive: protocol error" )); +if (version == protocol_v0) { + status =packet_reader_read (&reader); + +if (status ==PACKET_READ_FLUSH ) +die (_ ("git archive: expected ACK/NAK, got a flush packet" )); +if (strcmp (reader.buffer,"ACK" )) { +if (starts_with (reader.buffer,"NACK " ))
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 90
Quoted 10 lines
builtin/archive.c +9
-0+die (_ ("git archive: NACK %s" ), reader.buffer +5 ); +if (starts_with (reader.buffer,"ERR " )) +die (_ ("remote error: %s" ), reader.buffer +4 ); +die (_ ("git archive: protocol error" )); + } + + status =packet_reader_read (&reader); +if (status !=PACKET_READ_FLUSH ) +die (_ ("git archive: expected a flush" )); }
The original protocol lets upload-archive to report failure to spawn the writer backend process and lets us act on it. We do not need a similar support in the updated protocol and instead can jump right into receiving the archive stream because...?
Quoted 25 lines
builtin/archive.c +0
-4- status =packet_reader_read (&reader); -if (status !=PACKET_READ_FLUSH ) -die (_ ("git archive: expected a flush" )); -/* Now, start reading from fd[0] and spit it out to stdout */ rv =recv_sideband ("archive" , fd[0 ],1 ); rv |=transport_disconnect (transport);builtin/upload-archive.c +1
-0@@ -5,6 +5,7 @@#include "builtin.h" #include "archive.h" #include "pkt-line.h" +#include "protocol.h" #include "sideband.h" #include "run-command.h" #include "argv-array.h" @@ -73,13 +74,53 @@ static ssize_t process_input(int child_fd, int band)return sz;
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 84
Quoted 10 lines
builtin/upload-archive.c +8
-0} +static int handle_v2_command_and_cap (void ) +{ +struct packet_reader reader; +enum packet_read_status status; + +packet_reader_init (&reader,0 ,NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); + +packet_write_fmt (1 ,"version 2\n" );
This lets the discover_version() on the other side notice that we are speaking version 2.
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 89
Quoted 5 lines
builtin/upload-archive.c +5
-0+/* +* We don't currently send any capabilities, but maybe we could list +* supported archival formats? +*/ +packet_flush (1 );
process_capabilities_v2() expects the list of caps ends with a flush, which is given here.
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 94
Quoted 4 lines
builtin/upload-archive.c +4
-0+ status =packet_reader_read (&reader); +if (status !=PACKET_READ_NORMAL || +strcmp (reader.buffer,"command=archive" )) +die (_ ("upload-archive: expected command=archive" ));
The other side in do_v2_command_and_cap() would ask command=archive and that is verified. _() is unwanted, I suppose, by the way, as you do not know what language the other side wants anyway.
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 99
Quoted 5 lines
builtin/upload-archive.c +5
-0+while (status ==PACKET_READ_NORMAL ) { +/* We don't currently expect any client capabilities, but we +* should still read (and ignore) any that happen to get sent. +*/ + status =packet_reader_read (&reader);
It is wrong to say we should "ignore". If you are asked to behave in a certain way by a capability that is not understood, the other side expects you to honor that request and you have no idea how to comply. At least you should make sure that what is asked is among the capabilities you offered (or you understand), and you should error out when you see an unknown one, no?
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 104
Quoted 5 lines
builtin/upload-archive.c +5
-0+ } +if (status !=PACKET_READ_DELIM ) +die (_ ("upload-archive: expected delim packet" )); + +/* Let git-upload-archive--writer handle the arguments. */
The choice of DELIM here over FLUSH is a bit curious, but it is consistent between upload-archive and run-remote-archiver.
Quoted 8 lines
builtin/upload-archive.c +4
-0+return 0 ; +} +int cmd_upload_archive (int argc,const char **argv,const char *prefix) {struct child_process writer = { argv }; +enum protocol_version version =determine_protocol_version_server ();
builtin/upload-archive.c static ssize_t process_input(int child_fd, int band) · line 123
Quoted 10 lines
builtin/upload-archive.c +7
-0if (argc ==2 && !strcmp (argv[1 ],"-h" ))usage (upload_archive_usage); +if (version == protocol_v2) +handle_v2_command_and_cap (); +else { +packet_write_fmt (1 ,"ACK\n" ); +packet_flush (1 ); + } +
This breaks the original protocol, no? At this point we haven't even tried to start the writer process, and letting the other side go by giving ACK + flush prematurely. After start_command() fails, we may say NACK, but the other side is no longer listening to it.
Quoted 29 lines
builtin/upload-archive.c +0
-3/* * Set up sideband subprocess. * @@ -96,9 +137,6 @@ int cmd_upload_archive(int argc, const char **argv, const char *prefix)die ("upload-archive: %s" ,strerror (err)); } -packet_write_fmt (1 ,"ACK\n" ); -packet_flush (1 ); -while (1 ) {struct pollfd pfd[2 ];t/t5000-tar-tree.sh +5
-0@@ -145,6 +145,11 @@ test_expect_success \ check_tar b +test_expect_success 'protocol v2 for remote' ' + GIT_PROTOCOL="version=2" git archive --remote=. HEAD >v2_remote.tar +' +check_tar v2_remote + test_expect_success 'git archive --prefix=prefix/' ' git archive --prefix=prefix/ HEAD >with_prefix.tar '
Re: [PATCH 2/3] archive: implement protocol v2 archive command
In reply to Josh SteadmonHi,
Quoted 7 lines · Josh Steadmon
This adds a new archive command for protocol v2. The command expects arguments in the form "argument X" which are passed unmodified to git-upload-archive--writer.
This command works over the file://, Git, and SSH transports. HTTP support will be added in a separate patch.
Quoted 6 lines
Signed-off-by Josh Steadmon <[email protected]>--- builtin/archive.c | 45 +++++++++++++++++++++++++++------------- builtin/upload-archive.c | 44 ++++++++++++++++++++++++++++++++++++--- t/t5000-tar-tree.sh | 5 +++++ 3 files changed, 77 insertions(+), 17 deletions(-)
I like the diffstat. :)
Can this include some docs in Documentation/technical/ as well, to help other implementors to understand the protocol so they can interoperate?
Thanks,
Jonathan
PATCH 3/3 archive: allow archive over HTTP(S) with proto v2
In reply to Josh Steadmon--- builtin/archive.c | 8 +++++++- http-backend.c | 10 +++++++++- transport-helper.c | 5 +++-- 3 files changed, 19 insertions(+), 4 deletions(-)
builtin/archive.c +7 -1
@@ -87,7 +87,13 @@ static int run_remote_archiver(int argc, const char **argv, status =packet_reader_read (&reader);if (status !=PACKET_READ_FLUSH )die (_ ("git archive: expected a flush" )); - } + }else if (version == protocol_v2 && +starts_with (transport->url,"http" )) +/* +* Commands over HTTP require two requests, so there's an +* additional server response to parse. +*/ +discover_version (&reader);/* Now, start reading from fd[0] and spit it out to stdout */ rv =recv_sideband ("archive" , fd[0 ],1 );
http-backend.c +9 -1
@@ -32,6 +32,7 @@ struct rpc_service {static struct rpc_service rpc_service[] = { {"upload-pack" ,"uploadpack" ,1 ,1 }, {"receive-pack" ,"receivepack" ,0 ,-1 }, + {"upload-archive" ,"uploadarchive" ,1 ,1 }, };static struct string_list *get_parameters (void )
@@ -637,6 +638,12 @@ static void service_rpc(struct strbuf *hdr, char *service_name)struct rpc_service *svc =select_service (hdr, service_name);struct strbuf buf =STRBUF_INIT ; +if (!strcmp (service_name,"git-upload-archive" )) { +/* git-upload-archive doesn't need --stateless-rpc */ + argv[1 ] ="." ; + argv[2 ] =NULL ; + } +strbuf_reset (&buf);strbuf_addf (&buf,"application/x-git-%s-request" , svc->name);check_content_type (hdr, buf.buf);
@@ -713,7 +720,8 @@ static struct service_cmd { {"GET", "/objects/pack/pack-[0-9a-f]{40}\\.idx$", get_idx_file}, {"POST", "/git-upload-pack$", service_rpc}, - {"POST", "/git-receive-pack$", service_rpc} + {"POST", "/git-receive-pack$", service_rpc}, + {"POST", "/git-upload-archive$", service_rpc}, };static int bad_request (struct strbuf *hdr,const struct service_cmd *c)
transport-helper.c +3 -2
@@ -605,7 +605,8 @@ static int process_connect_service(struct transport *transport, ret =run_connect (transport, &cmdbuf); }else if (data->stateless_connect && (get_protocol_version_config () == protocol_v2) && - !strcmp ("git-upload-pack" , name)) { + (!strcmp ("git-upload-pack" , name) || + !strcmp ("git-upload-archive" , name))) {strbuf_addf (&cmdbuf,"stateless-connect %s\n" , name); ret =run_connect (transport, &cmdbuf);if (ret)
@@ -639,7 +640,7 @@ static int connect_helper(struct transport *transport, const char *name,/* Get_helper so connect is inited. */ get_helper (transport); -if (!data->connect) +if (!data->connect && !data->stateless_connect)die (_ ("operation not supported by protocol" ));if (!process_connect_service (transport, name, exec))
Signature · 1 line
2.19.0.397.gdd90340f6a-goog
Re: [PATCH 3/3] archive: allow archive over HTTP(S) with proto v2
Quoted 13 lines · Josh Steadmon
Signed-off-by Josh Steadmon <[email protected]>--- builtin/archive.c | 8 +++++++- http-backend.c | 10 +++++++++- transport-helper.c | 5 +++-- 3 files changed, 19 insertions(+), 4 deletions(-)builtin/archive.c +0
-0
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 91
Quoted 6 lines
builtin/archive.c +2
-1@@ -87,7 +87,13 @@ static int run_remote_archiver(int argc, const char **argv, status =packet_reader_read (&reader);if (status !=PACKET_READ_FLUSH )die (_ ("git archive: expected a flush" )); - } + }else if (version == protocol_v2 && +starts_with (transport->url,"http" ))
I assume this is a smart way to cover both http and https as the latter is prefixed by the former.
How does this interact with remote helpers
(See gitremote-helpers(1), which doesn't mention archives, but I would imagine that one would be able to use a remote helper eventually, too?
git archive --remote=abc://example.org/test ....
Quoted 13 lines
builtin/archive.c +5
-0+/* +* Commands over HTTP require two requests, so there's an +* additional server response to parse. +*/ +discover_version (&reader);/* Now, start reading from fd[0] and spit it out to stdout */ rv =recv_sideband ("archive" , fd[0 ],1 );http-backend.c +0
-0
http-backend.c struct rpc_service { · line 35
Quoted 4 lines
http-backend.c +1
-0@@ -32,6 +32,7 @@ struct rpc_service {static struct rpc_service rpc_service[] = { {"upload-pack" ,"uploadpack" ,1 ,1 }, {"receive-pack" ,"receivepack" ,0 ,-1 }, + {"upload-archive" ,"uploadarchive" ,1 ,1 },
So git archives are not supported in protocol v0 via http? Then it makes sense to see not mention of archives in the remote helpers as well.
The rest of the code is a surprisingly small patch.
Stefan
Re: [PATCH 3/3] archive: allow archive over HTTP(S) with proto v2
In reply to Josh SteadmonQuoted 13 lines · Josh Steadmon
Signed-off-by Josh Steadmon <[email protected]>--- builtin/archive.c | 8 +++++++- http-backend.c | 10 +++++++++- transport-helper.c | 5 +++-- 3 files changed, 19 insertions(+), 4 deletions(-)builtin/archive.c +0
-0@@ -87,7 +87,13 @@ static int run_remote_archiver(int argc, const char **argv, status =packet_reader_read (&reader);
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 96
Quoted 10 lines
builtin/archive.c +7
-1if (status !=PACKET_READ_FLUSH )die (_ ("git archive: expected a flush" )); - } + }else if (version == protocol_v2 && +starts_with (transport->url,"http" )) +/* +* Commands over HTTP require two requests, so there's an +* additional server response to parse. +*/ +discover_version (&reader);
What should happen if the version discovered here is different from v2 or the capabilities offered are different from what we saw before? Perhaps we need some sanity checks, as we on this side of the connection know we are making two requests, and may even end up talking with an instance of "upload-archive" that is different from the one we talked with earlier.
Quoted 50 lines
http-backend.c +9
-1@@ -32,6 +32,7 @@ struct rpc_service {static struct rpc_service rpc_service[] = { {"upload-pack" ,"uploadpack" ,1 ,1 }, {"receive-pack" ,"receivepack" ,0 ,-1 }, + {"upload-archive" ,"uploadarchive" ,1 ,1 }, };static struct string_list *get_parameters (void )@@ -637,6 +638,12 @@ static void service_rpc(struct strbuf *hdr, char *service_name)struct rpc_service *svc =select_service (hdr, service_name);struct strbuf buf =STRBUF_INIT ; +if (!strcmp (service_name,"git-upload-archive" )) { +/* git-upload-archive doesn't need --stateless-rpc */ + argv[1 ] ="." ; + argv[2 ] =NULL ; + } +strbuf_reset (&buf);strbuf_addf (&buf,"application/x-git-%s-request" , svc->name);check_content_type (hdr, buf.buf);@@ -713,7 +720,8 @@ static struct service_cmd { {"GET", "/objects/pack/pack-[0-9a-f]{40}\\.idx$", get_idx_file}, {"POST", "/git-upload-pack$", service_rpc}, - {"POST", "/git-receive-pack$", service_rpc} + {"POST", "/git-receive-pack$", service_rpc}, + {"POST", "/git-upload-archive$", service_rpc}, };static int bad_request (struct strbuf *hdr,const struct service_cmd *c)transport-helper.c +2
-1@@ -605,7 +605,8 @@ static int process_connect_service(struct transport *transport, ret =run_connect (transport, &cmdbuf); }else if (data->stateless_connect && (get_protocol_version_config () == protocol_v2) && - !strcmp ("git-upload-pack" , name)) { + (!strcmp ("git-upload-pack" , name) || + !strcmp ("git-upload-archive" , name))) {strbuf_addf (&cmdbuf,"stateless-connect %s\n" , name); ret =run_connect (transport, &cmdbuf);if (ret)
transport-helper.c static int connect_helper(struct transport *transport, const char *name, · line 644
Quoted 6 lines
transport-helper.c +1
-1@@ -639,7 +640,7 @@ static int connect_helper(struct transport *transport, const char *name,/* Get_helper so connect is inited. */ get_helper (transport); -if (!data->connect) +if (!data->connect && !data->stateless_connect)die (_ ("operation not supported by protocol" ));
This is somewhat curious. The upload-pack going over HTTP also is triggered by the same "stateless-connect" remote helper command, as we just saw in the previous hunk, and that support is not new. Why do we need this change then? What's different between the "data" that is obtained by get_helper(transport) for driving upload-pack and upload-archive? Presumably upload-pack was working without this change because it also sets the connect bit on, and upload-archive does not work without this change because it does not? Why do these two behave differently?
Quoted 1 line
transport-helper.c +0
-0if (!process_connect_service (transport, name, exec))
Re: [PATCH 3/3] archive: allow archive over HTTP(S) with proto v2
Quoted 84 lines · Junio C Hamano
Quoted 13 lines · Josh Steadmon
Signed-off-by Josh Steadmon <[email protected]>--- builtin/archive.c | 8 +++++++- http-backend.c | 10 +++++++++- transport-helper.c | 5 +++-- 3 files changed, 19 insertions(+), 4 deletions(-)builtin/archive.c +0
-0@@ -87,7 +87,13 @@ static int run_remote_archiver(int argc, const char **argv, status =packet_reader_read (&reader);builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 96
Quoted 10 lines
builtin/archive.c +7
-1if (status !=PACKET_READ_FLUSH )die (_ ("git archive: expected a flush" )); - } + }else if (version == protocol_v2 && +starts_with (transport->url,"http" )) +/* +* Commands over HTTP require two requests, so there's an +* additional server response to parse. +*/ +discover_version (&reader);What should happen if the version discovered here is different from v2 or the capabilities offered are different from what we saw before? Perhaps we need some sanity checks, as we on this side of the connection know we are making two requests, and may even end up talking with an instance of "upload-archive" that is different from the one we talked with earlier.
Quoted 51 lines
http-backend.c +9
-1@@ -32,6 +32,7 @@ struct rpc_service {static struct rpc_service rpc_service[] = { {"upload-pack" ,"uploadpack" ,1 ,1 }, {"receive-pack" ,"receivepack" ,0 ,-1 }, + {"upload-archive" ,"uploadarchive" ,1 ,1 }, };static struct string_list *get_parameters (void )@@ -637,6 +638,12 @@ static void service_rpc(struct strbuf *hdr, char *service_name)struct rpc_service *svc =select_service (hdr, service_name);struct strbuf buf =STRBUF_INIT ; +if (!strcmp (service_name,"git-upload-archive" )) { +/* git-upload-archive doesn't need --stateless-rpc */ + argv[1 ] ="." ; + argv[2 ] =NULL ; + } +strbuf_reset (&buf);strbuf_addf (&buf,"application/x-git-%s-request" , svc->name);check_content_type (hdr, buf.buf);@@ -713,7 +720,8 @@ static struct service_cmd { {"GET", "/objects/pack/pack-[0-9a-f]{40}\\.idx$", get_idx_file}, {"POST", "/git-upload-pack$", service_rpc}, - {"POST", "/git-receive-pack$", service_rpc} + {"POST", "/git-receive-pack$", service_rpc}, + {"POST", "/git-upload-archive$", service_rpc}, };static int bad_request (struct strbuf *hdr,const struct service_cmd *c)transport-helper.c +2
-1@@ -605,7 +605,8 @@ static int process_connect_service(struct transport *transport, ret =run_connect (transport, &cmdbuf); }else if (data->stateless_connect && (get_protocol_version_config () == protocol_v2) && - !strcmp ("git-upload-pack" , name)) { + (!strcmp ("git-upload-pack" , name) || + !strcmp ("git-upload-archive" , name))) {strbuf_addf (&cmdbuf,"stateless-connect %s\n" , name); ret =run_connect (transport, &cmdbuf);if (ret)@@ -639,7 +640,7 @@ static int connect_helper(struct transport *transport, const char *name,
Quoted 15 lines
transport-helper.c static int connect_helper(struct transport *transport, const char *name, · line 644
Quoted 5 lines
transport-helper.c +1
-1/* Get_helper so connect is inited. */ get_helper (transport); -if (!data->connect) +if (!data->connect && !data->stateless_connect)die (_ ("operation not supported by protocol" ));This is somewhat curious. The upload-pack going over HTTP also is triggered by the same "stateless-connect" remote helper command, as we just saw in the previous hunk, and that support is not new. Why do we need this change then? What's different between the "data" that is obtained by
get_helper(transport)for driving upload-pack and upload-archive? Presumably upload-pack was working without this change because it also sets the connect bit on, and upload-archive does not work without this change because it does not? Why do these two behave differently?
The data struct is not different between upload-pack vs. upload-archive. Neither of them set the connect bit. The difference is that upload-pack doesn't need to call transport_connect(), so it never hits connect_helper().
Re: [PATCH 3/3] archive: allow archive over HTTP(S) with proto v2
In reply to Josh SteadmonHi,
Quoted 1 line · Josh Steadmon
Subject: archive: allow archive over HTTP(S) with proto v2
It's interesting how little this has to touch the client.
Quoted 6 lines
Signed-off-by Josh Steadmon <[email protected]>--- builtin/archive.c | 8 +++++++- http-backend.c | 10 +++++++++- transport-helper.c | 5 +++-- 3 files changed, 19 insertions(+), 4 deletions(-)
[....]
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 91
Quoted 9 lines
builtin/archive.c +2
-1@@ -87,7 +87,13 @@ static int run_remote_archiver(int argc, const char **argv, status =packet_reader_read (&reader);if (status !=PACKET_READ_FLUSH )die (_ ("git archive: expected a flush" )); - } + }else if (version == protocol_v2 && +starts_with (transport->url,"http" ))
As Stefan noticed, this starts_with test seems a bit too loose. For example, what happens if I try an scp-style SSH URL like http.example.com:path/to/repo, a local path like http/foo/bar, or a custom protocol like httplikebutbetter://path/to/repo (honest
question: I haven't tried)?
builtin/archive.c static int run_remote_archiver(int argc, const char **argv, · line 96
Quoted 5 lines
builtin/archive.c +5
-0+/* +* Commands over HTTP require two requests, so there's an +* additional server response to parse. +*/ +discover_version (&reader);
Can this be made consistent with the non-http case? The original capabilities (/info/refs) response told us what protocol version the server wants to use, which means that a hypothetical protocol v3 could use a completely different request format for the followup commands:
so could the server omit the protocol version in the v2 /git-upload-archive response? Alternatively, if we want to include the protocol version again, could we do that in stateful protocols as well?
Related question: what should happen if the two responses declare different protocol versions? Should we diagnose that as a protocol error?
[...]
http-backend.c struct rpc_service { · line 35
Quoted 7 lines
http-backend.c +1
-0@@ -32,6 +32,7 @@ struct rpc_service {static struct rpc_service rpc_service[] = { {"upload-pack" ,"uploadpack" ,1 ,1 }, {"receive-pack" ,"receivepack" ,0 ,-1 }, + {"upload-archive" ,"uploadarchive" ,1 ,1 },
shell.c orders these in almost-alphabetical order (receive-pack, upload-pack, upload-archive). I guess they should both use actual alphabetical order? (If you agree, then please feel free to do that in a separate patch.)
[...]
http-backend.c static void service_rpc(struct strbuf *hdr, char *service_name) · line 642
Quoted 6 lines
http-backend.c +2
-0@@ -637,6 +638,12 @@ static void service_rpc(struct strbuf *hdr, char *service_name)struct rpc_service *svc =select_service (hdr, service_name);struct strbuf buf =STRBUF_INIT ; +if (!strcmp (service_name,"git-upload-archive" )) { +/* git-upload-archive doesn't need --stateless-rpc */
This comment doesn't seem actionable. Can it say why? E.g. "[...] because an upload-archive command always involves a single round-trip". Or alternatively, I think it's fine to omit the comment.
Quoted 3 lines
http-backend.c +3
-0+ argv[1 ] ="." ; + argv[2 ] =NULL ; + }
[...]
http-backend.c static struct service_cmd { · line 724
Quoted 7 lines
http-backend.c +2
-1@@ -713,7 +720,8 @@ static struct service_cmd { {"GET", "/objects/pack/pack-[0-9a-f]{40}\\.idx$", get_idx_file}, {"POST", "/git-upload-pack$", service_rpc}, - {"POST", "/git-receive-pack$", service_rpc} + {"POST", "/git-receive-pack$", service_rpc}, + {"POST", "/git-upload-archive$", service_rpc},
Same comment about services seeming to be in a randomish order.
[...]
transport-helper.c static int process_connect_service(struct transport *transport, · line 607
Quoted 6 lines
transport-helper.c +0
-0@@ -605,7 +605,8 @@ static int process_connect_service(struct transport *transport, ret =run_connect (transport, &cmdbuf); } else if (data->stateless_connect && (get_protocol_version_config() == protocol_v2) &&
(not about this patch) These parens don't help --- they make it harder for me to read, especially with the new parens to try to match them up with.
transport-helper.c static int process_connect_service(struct transport *transport, · line 609
Quoted 3 lines
transport-helper.c +2
-1- !strcmp("git-upload-pack", name)) { + (!strcmp("git-upload-pack", name) || + !strcmp("git-upload-archive", name))) {
A part of me wonders about the wasted cycles comparing to "git-upload-" twice, but (1) it is tiny relative to actually serving the request and (2) if we're lucky, the compiler (or a compiler of the future) inlines the strcmp call and could optimize it out.
[...]
transport-helper.c static int connect_helper(struct transport *transport, const char *name, · line 644
Quoted 7 lines
transport-helper.c +1
-1@@ -639,7 +640,7 @@ static int connect_helper(struct transport *transport, const char *name,/* Get_helper so connect is inited. */ get_helper (transport); -if (!data->connect) +if (!data->connect && !data->stateless_connect)die (_ ("operation not supported by protocol" ));
I don't understand this part. Can you explain it further (possibly by putting it in its own patch)?
Thanks for a pleasant read,
Jonathan
Hi,
Quoted 3 lines · Josh Steadmon
This series adds a new protocol v2 command for archiving, and allows this command to work over HTTP(S). This was previously discussed in [1]. I've CCed everyone who participated in that discussion.
Yay! Getting ready to read it now.
For the future, "git format-patch --cover-letter" does a few things that can be nice for this kind of opening message:
- it lists the patches in the series, and a diffstat
- it puts [PATCH 0/3] in the subject line so people know what to expect
Thanks,
Jonathan
v2 cover Add proto v2 archive command with HTTP support
In reply to Josh SteadmonThis is the second version of my series to add a new protocol v2 command for archiving, with support for HTTP(S).
NEEDSWORK: a server built with this series is not backwards-compatible with clients that set GIT_PROTOCOL=version=2 or configure protocol.version=2. The old client will unconditionally send "argument ..." packet lines, which breaks the server's expectations of a "command=archive" request, while the server's capability advertisement in turn breaks the clients expectation of either an ACK or NACK.
I've been discussing workarounds for this with Jonathan Nieder, but please let me know if you have any suggestions for v3 of this series.
Josh Steadmon (4):
archive: follow test standards around assertions
archive: use packet_reader for communications
archive: implement protocol v2 archive command
archive: allow archive over HTTP(S) with proto v2
Documentation/technical/protocol-v2.txt | 21 ++++++++- builtin/archive.c | 58 +++++++++++++++++++------ builtin/upload-archive.c | 27 ++++++++++-- http-backend.c | 13 +++++- serve.c | 7 +++ t/t5000-tar-tree.sh | 33 +++++++------- t/t5701-git-serve.sh | 1 + transport-helper.c | 7 +-- 8 files changed, 130 insertions(+), 37 deletions(-)
Range-diff against v1:
-: ---------- > 1: c2e371ad24 archive: follow test standards around assertions
1: b514184273 ! 2: a65f73f627 archive: use packet_reader for communications
@@ -6,7 +6,10 @@
handling, which will make implementation of protocol v2 support in
git-archive easier.
+ This refactoring does not change the behavior of "git archive".
+
Signed-off-by: Josh Steadmon <[email protected]>
+ Reviewed-by: Stefan Beller <[email protected]>
diff --git a/builtin/archive.c b/builtin/archive.c
@@ -42,24 +45,24 @@
- if (!buf)
+ status = packet_reader_read(&reader);
+
-+ if (status == PACKET_READ_FLUSH)
++ if (status != PACKET_READ_NORMAL || reader.pktlen <= 0)
die(_("git archive: expected ACK/NAK, got a flush packet"));
- if (strcmp(buf, "ACK")) {
- if (starts_with(buf, "NACK "))
- die(_("git archive: NACK %s"), buf + 5);
- if (starts_with(buf, "ERR "))
- die(_("remote error: %s"), buf + 4);
-+ if (strcmp(reader.buffer, "ACK")) {
-+ if (starts_with(reader.buffer, "NACK "))
-+ die(_("git archive: NACK %s"), reader.buffer + 5);
-+ if (starts_with(reader.buffer, "ERR "))
-+ die(_("remote error: %s"), reader.buffer + 4);
++ if (strcmp(reader.line, "ACK")) {
++ if (starts_with(reader.line, "NACK "))
++ die(_("git archive: NACK %s"), reader.line + 5);
++ if (starts_with(reader.line, "ERR "))
++ die(_("remote error: %s"), reader.line + 4);
die(_("git archive: protocol error"));
}
- if (packet_read_line(fd[0], NULL))
+ status = packet_reader_read(&reader);
-+ if (status != PACKET_READ_FLUSH)
++ if (status == PACKET_READ_NORMAL && reader.pktlen > 0)
die(_("git archive: expected a flush"));
/* Now, start reading from fd[0] and spit it out to stdout */
2: 1518c15dc1 < -: ---------- archive: implement protocol v2 archive command
-: ---------- > 3: 0a8cc5e331 archive: implement protocol v2 archive command
3: 1b7ad8d8f6 ! 4: 97a1424f32 archive: allow archive over HTTP(S) with proto v2
@@ -10,16 +10,20 @@
+++ b/builtin/archive.c
@@
status = packet_reader_read(&reader);
- if (status != PACKET_READ_FLUSH)
+ if (status == PACKET_READ_NORMAL && reader.pktlen > 0)
die(_("git archive: expected a flush"));
- }
+ } else if (version == protocol_v2 &&
-+ starts_with(transport->url, "http"))
++ (starts_with(transport->url, "http://") ||
++ starts_with(transport->url, "https://")))
+ /*
+ * Commands over HTTP require two requests, so there's an
-+ * additional server response to parse.
++ * additional server response to parse. We do only basic sanity
++ * checking here that the versions presented match across
++ * requests.
+ */
-+ discover_version(&reader);
++ if (version != discover_version(&reader))
++ die(_("git archive: received different protocol versions in subsequent requests"));
/* Now, start reading from fd[0] and spit it out to stdout */
rv = recv_sideband("archive", fd[0], 1);
@@ -40,7 +44,10 @@
struct strbuf buf = STRBUF_INIT;
+ if (!strcmp(service_name, "git-upload-archive")) {
-+ /* git-upload-archive doesn't need --stateless-rpc */
++ /*
++ * git-upload-archive doesn't need --stateless-rpc, because it
++ * always handles only a single request.
++ */
+ argv[1] = ".";
+ argv[2] = NULL;
+ }
@@ -63,10 +70,12 @@
--- a/transport-helper.c
+++ b/transport-helper.c
@@
+ strbuf_addf(&cmdbuf, "connect %s\n", name);
ret = run_connect(transport, &cmdbuf);
} else if (data->stateless_connect &&
- (get_protocol_version_config() == protocol_v2) &&
+- (get_protocol_version_config() == protocol_v2) &&
- !strcmp("git-upload-pack", name)) {
++ get_protocol_version_config() == protocol_v2 &&
+ (!strcmp("git-upload-pack", name) ||
+ !strcmp("git-upload-archive", name))) {
strbuf_addf(&cmdbuf, "stateless-connect %s\n", name);Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v2 1/4 archive: follow test standards around assertions
Move assertions outside of the check_tar function so that all top-level code is wrapped in a test_expect_* assertion.
--- t/t5000-tar-tree.sh | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-)
t/t5000-tar-tree.sh +10 -16
@@ -62,11 +62,9 @@ check_tar() { dir=$1 dir_with_prefix=$dir/$2 - test_expect_success ' extract tar archive' ' - (mkdir $dir && cd $dir && "$TAR" xf -) <$tarfile - ' + (mkdir $dir && cd $dir && "$TAR" xf -) <$tarfile && - test_expect_success TAR_NEEDS_PAX_FALLBACK ' interpret pax headers' ' + if test_have_prereq TAR_NEEDS_PAX_FALLBACK ; then ( cd $dir && for header in *.paxheader
@@ -82,16 +80,11 @@ check_tar() { fi done ) - ' + fi && - test_expect_success ' validate filenames' ' - (cd ${dir_with_prefix}a && find .) | sort >$listfile && - test_cmp a.lst $listfile - ' - - test_expect_success ' validate file contents' ' - diff -r a ${dir_with_prefix}a - ' + (cd ${dir_with_prefix}a && find .) | sort >$listfile && + test_cmp a.lst $listfile && + diff -r a ${dir_with_prefix}a } test_expect_success \
@@ -143,19 +136,20 @@ test_expect_success \ 'git archive' \ 'git archive HEAD >b.tar' -check_tar b +test_expect_success 'extract archive' 'check_tar b' test_expect_success 'git archive --prefix=prefix/' ' git archive --prefix=prefix/ HEAD >with_prefix.tar ' -check_tar with_prefix prefix/ +test_expect_success 'extract with prefix' 'check_tar with_prefix prefix/' test_expect_success 'git-archive --prefix=olde-' ' git archive --prefix=olde- HEAD >with_olde-prefix.tar ' -check_tar with_olde-prefix olde- +test_expect_success 'extract with olde- prefix' \ + 'check_tar with_olde-prefix olde-' test_expect_success 'git archive on large files' ' test_config core.bigfilethreshold 1 &&
Signature · 1 line
2.19.0.605.g01d371f741-goog
Re: [PATCH v2 1/4] archive: follow test standards around assertions
Quoted 3 lines · Josh Steadmon
Move assertions outside of the
check_tarfunction so that all top-level code is wrapped in a test_expect_* assertion.
Cool, I'll file this under modernizing the test infrastructure ;-)
Quoted 26 lines
t/t5000-tar-tree.sh +3
-6@@ -62,11 +62,9 @@ check_tar() { dir=$1 dir_with_prefix=$dir/$2 - test_expect_success ' extract tar archive' ' - (mkdir $dir && cd $dir && "$TAR" xf -) <$tarfile - ' + (mkdir $dir && cd $dir && "$TAR" xf -) <$tarfile && - test_expect_success TAR_NEEDS_PAX_FALLBACK ' interpret pax headers' ' + if test_have_prereq TAR_NEEDS_PAX_FALLBACK ; then ( cd $dir && for header in *.paxheader@@ -82,16 +80,11 @@ check_tar() { fi done ) - ' + fi && - test_expect_success ' validate filenames' '
t/t5000-tar-tree.sh check_tar() { · line 87
Quoted 10 lines
t/t5000-tar-tree.sh +3
-7- (cd ${dir_with_prefix}a && find .) | sort >$listfile && - test_cmp a.lst $listfile - ' - - test_expect_success ' validate file contents' ' - diff -r a ${dir_with_prefix}a - ' + (cd ${dir_with_prefix}a && find .) | sort >$listfile && + test_cmp a.lst $listfile && + diff -r a ${dir_with_prefix}a
Up to here we unwrapped code and removed test_expect_success
and just executed the code as is, so later callers would need to encapsulate the call to check_tar with test_expect_success.
However as we are touching the code here, we can go further than just unwrapping it, usually we'd format one command a line,
(
cd ${dir_with_prefix}a &&
find .
) | sort >$listfile &&
test_cmp ...I am not sure if that standard style is more legible in this case though.
t/t5000-tar-tree.sh test_expect_success \ · line 139
Quoted 9 lines
t/t5000-tar-tree.sh +1
-1} test_expect_success \@@ -143,19 +136,20 @@ test_expect_success \ 'git archive' \ 'git archive HEAD >b.tar' -check_tar b +test_expect_success 'extract archive' 'check_tar b'
Heh. Just looked into the file and the surrounding code is a wild mixture of the old style
test_expect_success \
'git archive' \
'git archive HEAD >b.tar'
check_tar b
and the new style
test_expect_success 'test name' '
<TAB> command &&
<TAB> command2
'
Maybe we could cleanup that file to look more like one of the newer tests (e.g. t3206, t0410) ?
But I guess for the purpose of getting the check_tar function usable inside a test, this would do enough.
Quoted 5 lines
t/t5000-tar-tree.sh +0
-0test_expect_success 'git archive --prefix=prefix/' ' git archive --prefix=prefix/ HEAD >with_prefix.tar '
t/t5000-tar-tree.sh test_expect_success \ · line 152
Quoted 10 lines
t/t5000-tar-tree.sh +3
-2-check_tar with_prefix prefix/ +test_expect_success 'extract with prefix' 'check_tar with_prefix prefix/' test_expect_success 'git-archive --prefix=olde-' ' git archive --prefix=olde- HEAD >with_olde-prefix.tar ' -check_tar with_olde-prefix olde- +test_expect_success 'extract with olde- prefix' \ + 'check_tar with_olde-prefix olde-'
In new style this would look like
test_expect_success 'extract with olde- prefix' '
check_tar with_olde-prefix olde-
'
PATCH v2 2/4 archive: use packet_reader for communications
In reply to Josh SteadmonUsing packet_reader will simplify version detection and capability handling, which will make implementation of protocol v2 support in git-archive easier.
This refactoring does not change the behavior of "git archive".
--- builtin/archive.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
builtin/archive.c +14 -9
@@ -27,10 +27,11 @@ static int run_remote_archiver(int argc, const char **argv, const char *remote, const char *exec, const char *name_hint) { - char *buf; int fd[2], i, rv;struct transport *transport;struct remote *_remote; +struct packet_reader reader; +enum packet_read_status status; _remote =remote_get (remote);if (!_remote->url[0 ])
@@ -38,6 +39,8 @@ static int run_remote_archiver(int argc, const char **argv, transport =transport_get (_remote, _remote->url[0 ]);transport_connect (transport,"git-upload-archive" , exec, fd); +packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); +/* * Inject a fake --format field at the beginning of the * arguments, with the format inferred from our output
@@ -53,18 +56,20 @@ static int run_remote_archiver(int argc, const char **argv,packet_write_fmt (fd[1 ],"argument %s\n" , argv[i]);packet_flush (fd[1 ]); - buf =packet_read_line (fd[0 ],NULL ); -if (!buf) + status =packet_reader_read (&reader); + +if (status !=PACKET_READ_NORMAL || reader.pktlen <=0 )die (_ ("git archive: expected ACK/NAK, got a flush packet" )); -if (strcmp (buf,"ACK" )) { -if (starts_with (buf,"NACK " )) -die (_ ("git archive: NACK %s" ), buf +5 ); -if (starts_with (buf,"ERR " )) -die (_ ("remote error: %s" ), buf +4 ); +if (strcmp (reader.line,"ACK" )) { +if (starts_with (reader.line,"NACK " )) +die (_ ("git archive: NACK %s" ), reader.line +5 ); +if (starts_with (reader.line,"ERR " )) +die (_ ("remote error: %s" ), reader.line +4 );die (_ ("git archive: protocol error" )); } -if (packet_read_line (fd[0 ],NULL )) + status =packet_reader_read (&reader); +if (status ==PACKET_READ_NORMAL && reader.pktlen >0 )die (_ ("git archive: expected a flush" ));/* Now, start reading from fd[0] and spit it out to stdout */
Signature · 1 line
2.19.0.605.g01d371f741-goog
Re: [PATCH v2 2/4] archive: use packet_reader for communications
Quoted 8 lines · Josh Steadmon
Using
packet_readerwill simplify version detection and capability handling, which will make implementation of protocol v2 support in git-archive easier.This refactoring does not change the behavior of "git archive".
Signed-off-by Josh Steadmon <[email protected]>
This patch is
Thanks!
Quoted 62 lines
--- builtin/archive.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)builtin/archive.c +14
-10@@ -27,10 +27,11 @@ static int run_remote_archiver(int argc, const char **argv, const char *remote, const char *exec, const char *name_hint) { - char *buf; int fd[2], i, rv;struct transport *transport;struct remote *_remote; +struct packet_reader reader; +enum packet_read_status status; _remote =remote_get (remote);if (!_remote->url[0 ])@@ -38,6 +39,8 @@ static int run_remote_archiver(int argc, const char **argv, transport =transport_get (_remote, _remote->url[0 ]);transport_connect (transport,"git-upload-archive" , exec, fd); +packet_reader_init (&reader, fd[0 ],NULL ,0 ,PACKET_READ_CHOMP_NEWLINE ); +/* * Inject a fake --format field at the beginning of the * arguments, with the format inferred from our output @@ -53,18 +56,20 @@ static int run_remote_archiver(int argc, const char **argv,packet_write_fmt (fd[1 ],"argument %s\n" , argv[i]);packet_flush (fd[1 ]); - buf =packet_read_line (fd[0 ],NULL ); -if (!buf) + status =packet_reader_read (&reader); + +if (status !=PACKET_READ_NORMAL || reader.pktlen <=0 )die (_ ("git archive: expected ACK/NAK, got a flush packet" )); -if (strcmp (buf,"ACK" )) { -if (starts_with (buf,"NACK " )) -die (_ ("git archive: NACK %s" ), buf +5 ); -if (starts_with (buf,"ERR " )) -die (_ ("remote error: %s" ), buf +4 ); +if (strcmp (reader.line,"ACK" )) { +if (starts_with (reader.line,"NACK " )) +die (_ ("git archive: NACK %s" ), reader.line +5 ); +if (starts_with (reader.line,"ERR " )) +die (_ ("remote error: %s" ), reader.line +4 );die (_ ("git archive: protocol error" )); } -if (packet_read_line (fd[0 ],NULL )) + status =packet_reader_read (&reader); +if (status ==PACKET_READ_NORMAL && reader.pktlen >0 )die (_ ("git archive: expected a flush" ));/* Now, start reading from fd[0] and spit it out to stdout */ --2.19.0.605.g01d371f741-goog
PATCH v2 3/4 archive: implement protocol v2 archive command
In reply to Josh SteadmonThis adds a new archive command for protocol v2. The command expects arguments in the form "argument X" which are passed unmodified to git-upload-archive--writer.
This command works over the file://, Git, and SSH transports. HTTP support will be added in a separate patch.
NEEDSWORK: this is not backwards-compatible with older clients that set GIT_PROTOCOL=version=2 or configure protocol.version=2.
--- Documentation/technical/protocol-v2.txt | 21 +++++++++++- builtin/archive.c | 45 +++++++++++++++++-------- builtin/upload-archive.c | 27 +++++++++++++-- serve.c | 7 ++++ t/t5000-tar-tree.sh | 7 ++++ t/t5701-git-serve.sh | 1 + 6 files changed, 90 insertions(+), 18 deletions(-)
Documentation/technical/protocol-v2.txt +20 -1
@@ -77,7 +77,8 @@ A v2 server would reply: S: <capability-advertisement> Subsequent requests are then made directly to the service -`$GIT_URL/git-upload-pack`. (This works the same for git-receive-pack). +`$GIT_URL/git-upload-pack`. (This works the same for git-receive-pack or +git-upload-archive). Capability Advertisement --------------------------
@@ -168,6 +169,24 @@ printable ASCII characters except space (i.e., the byte range 32 < x < and debugging purposes, and MUST NOT be used to programmatically assume the presence or absence of particular features. + archive +~~~~~~~~~ + +`archive` is the command to request an archive (such as a tarball) from a server +over v2. + +archive takes the following optional arguments: + + argument <arg> + This specifies that <arg> should be passed as an argument to the + underlying archive writer. This option can be repeated to pass + additional arguments to the archive writer. + +See linkgit:git-archive[1] for allowed values of `<arg>`. + +The output of archive is a packet-line stream of the resulting archive, with +error messages transferred over the sideband channel. + ls-refs ~~~~~~~~~
builtin/archive.c +31 -14
80 lines not on this page · raw message
builtin/upload-archive.c +24 -3
@@ -5,6 +5,8 @@#include "builtin.h" #include "archive.h" #include "pkt-line.h" +#include "protocol.h" +#include "serve.h" #include "sideband.h" #include "run-command.h" #include "argv-array.h"
@@ -76,10 +78,24 @@ static ssize_t process_input(int child_fd, int band)int cmd_upload_archive (int argc,const char **argv,const char *prefix) {struct child_process writer = { argv }; +enum protocol_version version =determine_protocol_version_server ();if (argc ==2 && !strcmp (argv[1 ],"-h" ))usage (upload_archive_usage); +if (version == protocol_v2) { +struct serve_options options =SERVE_OPTIONS_INIT ; + +/* Send "version 2" message and capabilities. */ + options.advertise_capabilities =1 ; +serve (&options); + +/* Process command request and validate client capabilities. */ + options.advertise_capabilities =0 ; + options.stateless_rpc =1 ; +serve (&options); + } +/* * Set up sideband subprocess. *
@@ -92,12 +108,17 @@ int cmd_upload_archive(int argc, const char **argv, const char *prefix) writer.git_cmd =1 ;if (start_command (&writer)) {int err = errno; -packet_write_fmt (1 ,"NACK unable to spawn subprocess\n" ); +if (version == protocol_v0 || version == protocol_v1) +packet_write_fmt (1 ,"NACK unable to spawn subprocess\n" ); +else if (version == protocol_v2) +error_clnt ("unable to spawn subprocess\n" );die ("upload-archive: %s" ,strerror (err)); } -packet_write_fmt (1 ,"ACK\n" ); -packet_flush (1 ); +if (version == protocol_v0 || version == protocol_v1) { +packet_write_fmt (1 ,"ACK\n" ); +packet_flush (1 ); + }while (1 ) {struct pollfd pfd[2 ];
serve.c +7 -0
@@ -52,8 +52,15 @@ struct protocol_capability { struct packet_reader *request); }; +static int noop (struct repository *r,struct argv_array *keys, +struct packet_reader *request) +{ +return 0 ; +} +static struct protocol_capability capabilities[] = { {"agent" , agent_advertise,NULL }, + {"archive" , always_advertise, noop }, {"ls-refs" , always_advertise, ls_refs }, {"fetch" , upload_pack_advertise, upload_pack_v2 }, {"server-option" , always_advertise,NULL },
t/t5000-tar-tree.sh +7 -0
@@ -138,6 +138,13 @@ test_expect_success \ test_expect_success 'extract archive' 'check_tar b' +test_expect_success 'protocol v2 for remote' ' + GIT_PROTOCOL="version=2" GIT_TRACE_PACKET="$(pwd)/log" git archive \ + --remote=. HEAD >v2_remote.tar && + grep "version 2" log +' +test_expect_success 'extract v2 archive' 'check_tar v2_remote' + test_expect_success 'git archive --prefix=prefix/' ' git archive --prefix=prefix/ HEAD >with_prefix.tar '
t/t5701-git-serve.sh +1 -0
@@ -8,6 +8,7 @@ test_expect_success 'test capability advertisement' ' cat >expect <<-EOF && version 2 agent=git/$(git version | cut -d" " -f3) + archive ls-refs fetch=shallow server-option
Signature · 1 line
2.19.0.605.g01d371f741-goog
PATCH v2 4/4 archive: allow archive over HTTP(S) with proto v2
In reply to Josh Steadmon--- builtin/archive.c | 12 +++++++++++- http-backend.c | 13 ++++++++++++- transport-helper.c | 7 ++++--- 3 files changed, 27 insertions(+), 5 deletions(-)
builtin/archive.c +11 -1
@@ -87,7 +87,17 @@ static int run_remote_archiver(int argc, const char **argv, status =packet_reader_read (&reader);if (status ==PACKET_READ_NORMAL && reader.pktlen >0 )die (_ ("git archive: expected a flush" )); - } + }else if (version == protocol_v2 && + (starts_with (transport->url,"http://" ) || +starts_with (transport->url,"https://" ))) +/* +* Commands over HTTP require two requests, so there's an +* additional server response to parse. We do only basic sanity +* checking here that the versions presented match across +* requests. +*/ +if (version !=discover_version (&reader)) +die (_ ("git archive: received different protocol versions in subsequent requests" ));/* Now, start reading from fd[0] and spit it out to stdout */ rv =recv_sideband ("archive" , fd[0 ],1 );
http-backend.c +12 -1
@@ -32,6 +32,7 @@ struct rpc_service {static struct rpc_service rpc_service[] = { {"upload-pack" ,"uploadpack" ,1 ,1 }, {"receive-pack" ,"receivepack" ,0 ,-1 }, + {"upload-archive" ,"uploadarchive" ,1 ,1 }, };static struct string_list *get_parameters (void )
@@ -637,6 +638,15 @@ static void service_rpc(struct strbuf *hdr, char *service_name)struct rpc_service *svc =select_service (hdr, service_name);struct strbuf buf =STRBUF_INIT ; +if (!strcmp (service_name,"git-upload-archive" )) { +/* +* git-upload-archive doesn't need --stateless-rpc, because it +* always handles only a single request. +*/ + argv[1 ] ="." ; + argv[2 ] =NULL ; + } +strbuf_reset (&buf);strbuf_addf (&buf,"application/x-git-%s-request" , svc->name);check_content_type (hdr, buf.buf);
@@ -713,7 +723,8 @@ static struct service_cmd { {"GET", "/objects/pack/pack-[0-9a-f]{40}\\.idx$", get_idx_file}, {"POST", "/git-upload-pack$", service_rpc}, - {"POST", "/git-receive-pack$", service_rpc} + {"POST", "/git-receive-pack$", service_rpc}, + {"POST", "/git-upload-archive$", service_rpc}, };static int bad_request (struct strbuf *hdr,const struct service_cmd *c)
transport-helper.c +4 -3
@@ -604,8 +604,9 @@ static int process_connect_service(struct transport *transport,strbuf_addf (&cmdbuf,"connect %s\n" , name); ret =run_connect (transport, &cmdbuf); }else if (data->stateless_connect && - (get_protocol_version_config () == protocol_v2) && - !strcmp ("git-upload-pack" , name)) { +get_protocol_version_config () == protocol_v2 && + (!strcmp ("git-upload-pack" , name) || + !strcmp ("git-upload-archive" , name))) {strbuf_addf (&cmdbuf,"stateless-connect %s\n" , name); ret =run_connect (transport, &cmdbuf);if (ret)
@@ -639,7 +640,7 @@ static int connect_helper(struct transport *transport, const char *name,/* Get_helper so connect is inited. */ get_helper (transport); -if (!data->connect) +if (!data->connect && !data->stateless_connect)die (_ ("operation not supported by protocol" ));if (!process_connect_service (transport, name, exec))
Signature · 1 line
2.19.0.605.g01d371f741-goog
Quoted 9 lines · Josh Steadmon
This is the second version of my series to add a new protocol v2 command for archiving, with support for HTTP(S).
NEEDSWORK: a server built with this series is not backwards-compatible with clients that set GIT_PROTOCOL=version=2 or configure protocol.version=2. The old client will unconditionally send "argument ..." packet lines, which breaks the server's expectations of a "command=archive" request,
So if an old client sets protocol to v2, it would only apply that protocol version
to fetch, not archive, so it would start following a v0 conversation, but as the protocol version is set, it would be transmitted to the server. This sounds like a bug in the client?
Quoted 2 lines
while the server's capability advertisement
in turn breaks the clients expectation of either an ACK or NACK.
Could a modern client send either another protocol version (3?) or a special capability along the protocol version ("fixed_archive")
Quoted 2 lines
I've been discussing workarounds for this with Jonathan Nieder, but please let me know if you have any suggestions for v3 of this series.
Care to open the discussion to the list? What are the different approaches, what are the pros/cons?
Quoted 7 lines · Stefan Beller
Quoted 2 lines · Josh Steadmon
I've been discussing workarounds for this with Jonathan Nieder, but please let me know if you have any suggestions for v3 of this series.
Care to open the discussion to the list? What are the different approaches, what are the pros/cons?
Do you mean sending video of chatting in the office?
Josh and I discussed that
1. Clients sending version=2 when they do not, in fact, speak protocol v2 for a service is a (serious) bug. (Separately from this series) we should fix it.
2. That bug is already in the wild, alas. Fortunately the semantics of GIT_PROTOCOL as a list of key/value pairs is well defined. So we have choices of (a) bump version to version=3 (b) pass another value 'version=2:yesreallyversion=2' (c) etc.
3. This is likely to affect push, too.
Thanks and hope that helps,
Jonathan
Quoted 10 lines · Jonathan Nieder
1. Clients sending version=2 when they do not, in fact, speak protocol v2 for a service is a (serious) bug. (Separately from this series) we should fix it.
2. That bug is already in the wild, alas. Fortunately the semantics of
GIT_PROTOCOLas a list of key/value pairs is well defined. So we have choices of (a) bump version to version=3 (b) pass another value 'version=2:yesreallyversion=2' (c) etc.3. This is likely to affect push, too.
Do you mean that existing "git push", "git fetch" and "git archive" sends version=2 even when they are not capable of speaking protocol v2? I thought that "git archive [--remote]" was left outside of the protocol update (that was the reason why the earlier attempt took a hacky route of "shallow clone followed by local archive"), so there is no "git archive" in the wild that can even say "version=$n" (which requires you to be at least version=1)?
Quoted 13 lines · Junio C Hamano
Quoted 10 lines · Jonathan Nieder
1. Clients sending version=2 when they do not, in fact, speak protocol v2 for a service is a (serious) bug. (Separately from this series) we should fix it.
2. That bug is already in the wild, alas. Fortunately the semantics of
GIT_PROTOCOLas a list of key/value pairs is well defined. So we have choices of (a) bump version to version=3 (b) pass another value 'version=2:yesreallyversion=2' (c) etc.3. This is likely to affect push, too.
Quoted 7 lines
Do you mean that existing "git push", "git fetch" and "git archive" sends version=2 even when they are not capable of speaking protocol v2? I thought that "git archive [--remote]" was left outside of the protocol update (that was the reason why the earlier attempt took a hacky route of "shallow clone followed by local archive"), so there is no "git archive" in the wild that can even say "version=$n" (which requires you to be at least version=1)?
Yes, the version on my desktop sends version=2 when archiving:
∫ which git/usr/bin/git
∫ git --version
git version 2.19.0.605.g01d371f741-goog
∫ GIT_TRACE_PACKET=${HOME}/server_trace git daemon \
--enable=upload-archive \
--base-path=${HOME}/src/bare-repos &
[1] 258496
∫ git archive --remote git://localhost/test-repo.git HEAD >! test.tar ∫ grep version ~/server_trace
15:31:22.377869 pkt-line.c:80 packet: git< git-upload-archive /test-repo.git\0host=localhost\0\0version=2\0
Quoted 13 lines · Josh Steadmon
Yes, the version on my desktop sends version=2 when archiving:
∫ which git
/usr/bin/git
∫ git --version
git version 2.19.0.605.g01d371f741-goog
∫ GIT_TRACE_PACKET=${HOME}/server_trace git daemon \
--enable=upload-archive \
--base-path=${HOME}/src/bare-repos &
[1] 258496
∫ git archive --remotegit://localhost/test-repo.gitHEAD >! test.tar ∫ grep version ~/server_trace
15:31:22.377869 pkt-line.c:80 packet: git< git-upload-archive /test-repo.git\0host=localhost\0\0version=2\0
Ah, that's truly broken.
Come to think of it, do we need to be using uniform versions across different endpoints? The archive request could be at v3 while fetch request could still be at v2, in which case the design to use a single protocol.version variable is probably the root cause of the confusion? Perhaps like protocol.<name>.allow, we would want protocol.<name>.version or something like that (and no protocol.version) to make it clear that protocol v2 used for fetching has nothing to do with protocol v1 or v2 or v3 used for archiving?
Luckily, protocol.version is still marked as experimental so it is not too bad that we caught the design mistake (if it is one) and can now correct it before the damage spreads too widely.
Quoted 11 lines · Stefan Beller
Quoted 9 lines · Josh Steadmon
This is the second version of my series to add a new protocol v2 command for archiving, with support for HTTP(S).
NEEDSWORK: a server built with this series is not backwards-compatible with clients that set GIT_PROTOCOL=version=2 or configure protocol.version=2. The old client will unconditionally send "argument ..." packet lines, which breaks the server's expectations of a "command=archive" request,
Quoted 5 lines
So if an old client sets protocol to v2, it would only apply that protocol version
to fetch, not archive, so it would start following a v0 conversation, but as the protocol version is set, it would be transmitted to the server. This sounds like a bug in the client?
Yeah, basically. We're telling the server we support v2, even if the specific operation we're trying to do doesn't have a v2 implementation on the client. So this is going to make it ugly to replace existing commands.
Quoted 11 lines
Quoted 2 lines
while the server's capability advertisement
in turn breaks the clients expectation of either an ACK or NACK.Could a modern client send either another protocol version (3?) or a special capability along the protocol version ("
fixed_archive")Quoted 2 lines
I've been discussing workarounds for this with Jonathan Nieder, but please let me know if you have any suggestions for v3 of this series.
Care to open the discussion to the list? What are the different approaches, what are the pros/cons?
Jonathan suggested something along the lines of what you said above, adding a new field in GIT_PROTOCOL. So we'd send something like "version=2:archive_version=2" and have the server detect the latter.
I'm not sure if that's the best way to go about this since I'm not familiar with the version detection code for other parts of the system. I worry that it will lead us down the path of having to specify a version for every command that we eventually convert to protocol v2. On the other hand, I don't see any other way to work around this, at least in the archive case. We can't peek at the client's transmissions on the server, because v2 requires that the server speaks first...