cat-file: extend remote-object-info to support %(objecttype)
cover cat-file: extend remote-object-info to support %(objecttype)
"%(objecttype)" is already known by the client's allow-list, but neither the client nor the server knows how to handle type. This series continues the work for git cat-file --batch-command extending remote-object-info to support "%(objecttype)" end to end. It is based on its predecessor series "cat-file: add remote-object-info to batch-command" [1].
Type is the last property that is identical on the server and on the client once fetched.
Whether to support more metadata such as:
- objectsize:disk
- deltabase
- objectmode (needs context)
should be discussed, unlike size and type, the rest depend on how things are packed on the server and on what the client already has, so they cannot reliably match local once fetched. IMO they are not worth supporting, I can't find a use case for them.
Adding new placeholders has become trivial.
To add a new placeholder, follow the steps in this series and add it to the client's allow-list at 'builtin/cat-file.c'.
Based-on: <[email protected]>
Github CI: https://github.com/pabloosabaterr/git/actions/runs/30155586279
[1]: https://lore.kernel.org/git/[email protected]/
---
Pablo Sabater (5):
protocol-caps: add type support to object-info
fetch-object-info: parse type from server response
fetch-object-info: request all supported options dynamically
serve: advertise type capability
cat-file: unify default format
Documentation/git-cat-file.adoc | 17 +++----- Documentation/gitprotocol-v2.adoc | 15 +++++-- builtin/cat-file.c | 7 --- fetch-object-info.c | 23 +++++++--- protocol-caps.c | 21 +++++++-- serve.c | 4 +- t/t1017-cat-file-remote-object-info.sh | 80 +++++++++++++--------------------- t/t5701-git-serve.sh | 27 ++++++++++++ 8 files changed, 113 insertions(+), 81 deletions(-)
---
base-commit: 71e19e8d2713f385c3fcef59cf6f29bcbd93d91f
change-id: 20260724-objecttype-support-ea1ef6941d07
PATCH 1/5 protocol-caps: add type support to object-info
Teach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While at it, fix requested_info->size bit field style.
--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 27 +++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 3 deletions(-)
protocol-caps.c +18 -3
@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/*
@@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string;struct object_id oid;size_t object_size; +enum object_type object_type;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) {packet_writer_error ( writer, -"object-info: protocol error, expected to get oid, not '%s'" , +"object-info: protocol error, expected to get " +"oid, not '%s'" , oid_str);continue ; }
@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }
@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); + write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);
@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;
t/t5701-git-serve.sh +27 -0
@@ -366,6 +366,33 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'type' ' + test_config transfer.advertiseObjectInfo true && + + test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type + oid $(git rev-parse two:two.t) + oid $(git rev-parse two:two.t) + 0000 + EOF + + cat >expect <<-EOF && + size + type + $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blob + $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blob + 0000 + EOF + + test-tool serve-v2 --stateless-rpc <in >out && + test-tool pkt-line unpack <out >actual && + test_cmp expect actual +' + test_expect_success 'bare OID request' ' test_config transfer.advertiseObjectInfo true &&
Signature · 1 line
2.54.0
PATCH 2/5 fetch-object-info: parse type from server response
In reply to Pablo SabaterThe server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the capability.
--- fetch-object-info.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
fetch-object-info.c +11 -1
@@ -50,6 +50,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar const int stateless_rpc, const int fd_out) { int size_index = -1; +int type_index =-1 ;switch (version) {case protocol_v2:
@@ -101,8 +102,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arfor (size_t j =0 ; j < args->oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); + }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep)); }else { -BUG ("only size is supported" ); +BUG ("unexpected object-info option: %s" , reader->line); } }
@@ -148,6 +154,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +string_list_clear (&object_info_values,0 ); }check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
Signature · 1 line
2.54.0
PATCH 3/5 fetch-object-info: request all supported options dynamically
In reply to Pablo SabaterIn send_object_info_request(), size is hardcoded to be the only option sent. In order to support type and future capabilities, replace the hardcoded size with a loop that requests everything on object_info_options list.
This is safe because the list has already been trimmed previously in fetch_object_info() to only contain options that the server supports.
--- fetch-object-info.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
fetch-object-info.c +7 -4
@@ -15,10 +15,13 @@ static void send_object_info_request(const int fd_out, struct object_info_args *write_command_and_capabilities (&req_buf,"object-info" , args->server_options); -if (unsorted_string_list_has_string (args->object_info_options,"size" )) -packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) -BUG ("only size should be in object_info_options" ); +/* +* The list is already checked to only request valid and supported fields +* no need to check, just request everything left on the list +*/ +for (size_t i =0 ; i < args->object_info_options->nr; i++) +packet_buf_write (&req_buf,"%s" , + args->object_info_options->items[i].string);if (args->oids)for (size_t i =0 ; i < args->oids->nr; i++)
Signature · 1 line
2.54.0
PATCH 4/5 serve: advertise type capability
In reply to Pablo SabaterThe server and the client can handle type requests but the client won't ask for it until the server advertises it.
Add type to the advertised capabilities so the client knows that it can request it.
--- serve.c | 4 ++-- t/t1017-cat-file-remote-object-info.sh | 10 ++++++---- 2 files changed, 8 insertions(+), 6 deletions(-)
serve.c +2 -2
@@ -97,9 +97,9 @@ static int object_info_advertise(struct repository *r, struct strbuf *value)/* disabled by default */ advertise_object_info =0 ; } -/* Currently only size is supported */ +/* Currently only size and type are supported */ if (value && advertise_object_info) -strbuf_addstr (value,"size" ); +strbuf_addstr (value,"size type" );return advertise_object_info; }
t/t1017-cat-file-remote-object-info.sh +6 -4
@@ -7,6 +7,7 @@ test_description='git cat-file --batch-command with remote-object-info command' hello_content="Hello World" hello_size=$(strlen "$hello_content") +hello_type="blob" hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) hello_short_oid=$(git rev-parse --short "$hello_oid")
@@ -19,6 +20,7 @@ unstored_oid=$(echo_without_newline "$unstored_content" | git hash-object --stdi # file name is hello, which is 5 characters # a space is 1 character and a null is 1 character tree_size=$(($(test_oid rawsz) + 13)) +tree_type="tree" commit_message="Initial commit"
@@ -31,6 +33,7 @@ commit_message="Initial commit" # An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`, # to get 177, 2. then deduct 40 hex characters to get 137 commit_size=$(($(test_oid hexsz) + 137)) +commit_type="commit" tag_header_without_oid="type blob tag hellotag
@@ -44,6 +47,7 @@ $tag_description" tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w) tag_size=$(strlen "$tag_content") +tag_type="tag" set_transport_variables () { hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin)
@@ -256,14 +260,12 @@ test_expect_success 'remote-object-info does not die on missing oid like info' ' ) ' -# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo $hello_type >expect && git cat-file --batch-command="%(objecttype)" >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOF
Signature · 1 line
2.54.0
PATCH 5/5 cat-file: unify default format
In reply to Pablo Sabater%(objecttype) is supported both by the client and by the server. Change the temporary default format to the unified version that other commands use.
Update documentation to remove %(objecttype) from the caveats of remote-object-info.
Update tests that used the default format to expect type. Update documentation to show %(objecttype) support.
--- Documentation/git-cat-file.adoc | 17 ++++----- Documentation/gitprotocol-v2.adoc | 15 ++++++-- builtin/cat-file.c | 7 ---- t/t1017-cat-file-remote-object-info.sh | 70 ++++++++++++---------------------- 4 files changed, 42 insertions(+), 67 deletions(-)
Documentation/git-cat-file.adoc +7 -10
@@ -348,15 +348,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. -The command `remote-object-info` only supports the `%(objectname)` and -`%(objectsize)` placeholders. See `CAVEATS` below for more information. +The command `remote-object-info` only supports the `%(objectname)`, +`%(objectsize)` and `%(objecttype)` placeholders. See `CAVEATS` below for more +information. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`, except for `remote-object-info` commands which -use `%(objectname) %(objectsize)` because `%(objecttype)` is not supported yet. - -WARNING: When "%(objecttype)" is supported, the default format WILL be unified, -so DO NOT RELY on the current default format to stay the same!!! +%(objecttype) %(objectsize)`. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -453,9 +450,9 @@ scripting purposes. CAVEATS ------- -Note that only `%(objectname)` and `%(objectsize)` are currently -supported by the `remote-object-info` command. Using any other placeholder in -the format string will return an empty string in its position. +Note that only `%(objectname)`, `%(objectsize)` and `%(objecttype)` are +currently supported by the `remote-object-info` command. Using any other +placeholder in the format string will return an empty string in its position. Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are
Documentation/gitprotocol-v2.adoc +11 -4
@@ -558,14 +558,17 @@ object-info `object-info` is the command to retrieve information about one or more objects. Its main purpose is to allow a client to make decisions based on this -information without having to fully fetch objects. Object size is the only -information that is currently supported. +information without having to fully fetch objects. Object size and type are the +only information that is currently supported. An `object-info` request takes the following arguments: size Requests size information to be returned for each listed object id. + type + Requests type information to be returned for each listed object id. + oid <oid> Indicates to the server an object which the client wants to obtain information for. They must be full OIDs.
@@ -580,11 +583,15 @@ space. info = *PKT-LINE(attr LF) *PKT-LINE(obj-info LF) - attr = "size" + attr = "size" | "type" obj-size = 1*DIGIT - obj-info = obj-id [SP [obj-size]] + obj-type = "blob" | "tree" | "commit" | "tag" + + obj-val = obj-size | obj-type + + obj-info = obj-id [SP [obj-val *(SP obj-val)]] If the server does not recognize the OID, the response will be `<oid> SP` regardless of the number of attributes requested.
builtin/cat-file.c +0 -7
@@ -841,15 +841,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,struct object_info *remote_object_info =NULL ;struct oid_array object_info_oids =OID_ARRAY_INIT ;struct string_list object_info_options =STRING_LIST_INIT_NODUP ; -const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )die (_ ("remote-object-info command too long" )); -/* -* TODO: Use the default format once %(objecttype) is supported. -*/ -if (!opt->format) - opt->format ="%(objectname) %(objectsize)" ; line_to_split =xstrdup (line); count =split_cmdline (line_to_split, &argv);
@@ -904,7 +898,6 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->is_remote =0 ; } data->skip_object_info =0 ; - opt->format = saved_format;for (size_t i =0 ; i < object_info_oids.nr; i++)free_object_info_contents (&remote_object_info[i]);
t/t1017-cat-file-remote-object-info.sh +24 -46
112 lines not on this page · raw message
Signature · 1 line
2.54.0
Quoted 12 lines · Pablo Sabater
"%(objecttype)" is already known by the client's allow-list, but neither the client nor the server knows how to handle type. This series continues the work for git cat-file --batch-command extending remote-object-info to support "%(objecttype)" end to end. It is based on its predecessor series "cat-file: add remote-object-info to batch-command" [1].
Type is the last property that is identical on the server and on the client once fetched.
Whether to support more metadata such as:
Quoted 8 lines
- objectsize:disk
- deltabase
- objectmode (needs context)should be discussed, unlike size and type, the rest depend on how things are packed on the server and on what the client already has, so they cannot reliably match local once fetched. IMO they are not worth supporting, I can't find a use case for them.
Makes sense to me. The metadata you listed depends on the local packfile storage where the repository lives.
objectmode depends on the tree pointing to the blob, so it wouldn't be accessible using the current infrastructure anyway.
Quoted 10 lines
Adding new placeholders has become trivial.
To add a new placeholder, follow the steps in this series and add it to the client's allow-list at 'builtin/cat-file.c'.Based-on: <[email protected]>
Github CI: https://github.com/pabloosabaterr/git/actions/runs/30155586279
Quoted 8 lines
Signed-off-by Pablo Sabater <[email protected]>---
Pablo Sabater (5):
protocol-caps: add type support to object-info
fetch-object-info: parse type from server response
fetch-object-info: request all supported options dynamically
serve: advertise type capability
cat-file: unify default format
The current incremental approach is safe and ensures every commit compiles and passes tests. However, from a storytelling perspective for the reviewers, I believe it's better to do any 'preparatory refactoring' before starting the new feature.
Patch 3 (dynamically requesting supported options) doesn't actually depend on type existing yet. I suggest bumping Patch 3 to be Patch 1 in V2. That way, the client is already dynamic and ready, and the feature patches can strictly focus on adding type:
- fetch-object-info: request all supported options dynamically (Current Patch 3)
- protocol-caps: add type support to object-info (Current Patch 1)
- fetch-object-info: parse type from server response (Current Patch 2)
- serve: advertise type capability (Current Patch 4)
- cat-file: unify default format (Current Patch 5)
Quoted 9 lines
Documentation/git-cat-file.adoc | 17 +++----- Documentation/gitprotocol-v2.adoc | 15 +++++-- builtin/cat-file.c | 7 --- fetch-object-info.c | 23 +++++++--- protocol-caps.c | 21 +++++++-- serve.c | 4 +- t/t1017-cat-file-remote-object-info.sh | 80 +++++++++++++--------------------- t/t5701-git-serve.sh | 27 ++++++++++++ 8 files changed, 113 insertions(+), 81 deletions(-)
This series is definitely a lot smaller than I thought it would be. Looks like most of the heavy lifting was already done with the previous series.
Good for us!
Re: [PATCH GSoC 1/5] protocol-caps: add type support to object-info
In reply to Pablo SabaterQuoted 38 lines · Pablo Sabater
Teach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While at it, fix
requested_info->sizebit field style.Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 27 +++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 3 deletions(-)protocol-caps.c +5
-1@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/* @@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string;
protocol-caps.c static void send_info(struct repository *r, struct packet_writer *writer, · line 90
Quoted 10 lines
protocol-caps.c +3
-1struct object_id oid;size_t object_size; +enum object_type object_type;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) { packet_writer_error( writer, - "object-info: protocol error, expected to get oid, not '%s'", + "object-info: protocol error, expected to get " + "oid, not '%s'",
I assume this is a style change? The original line doesn't seem long enough to wrap though.
Also, this would break the grep-ability of this error string.
Quoted 52 lines
protocol-caps.c +10
-1oid_str);continue ; }@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); + write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;t/t5701-git-serve.sh +9
-0@@ -366,6 +366,33 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'type' ' + test_config transfer.advertiseObjectInfo true && + + test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type
t/t5701-git-serve.sh test_expect_success 'basics of object-info' ' · line 387
Quoted 10 lines
t/t5701-git-serve.sh +10
-0+ oid $(git rev-parse two:two.t) + oid $(git rev-parse two:two.t) + 0000 + EOF + + cat >expect <<-EOF && + size + type + $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blob + $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blob
Can we not use the test_file_size tool to do this instead? That should also be much more portable.
Re: [PATCH GSoC 2/5] fetch-object-info: parse type from server response
In reply to Pablo SabaterQuoted 40 lines · Pablo Sabater
The server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the capability.
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- fetch-object-info.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)fetch-object-info.c +7
-1@@ -50,6 +50,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar const int stateless_rpc, const int fd_out) { int size_index = -1; +int type_index =-1 ;switch (version) {case protocol_v2:@@ -101,8 +102,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arfor (size_t j =0 ; j < args->oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); + }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep)); }else { -BUG ("only size is supported" ); +BUG ("unexpected object-info option: %s" , reader->line); } }
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 161
Quoted 8 lines
fetch-object-info.c +4
-0@@ -148,6 +154,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +string_list_clear (&object_info_values,0 );
Is there a risk of an out-of-bounds array access here if the server responds with a truncated or malformed packet?
If object_info_values.nr <= type_index + 1, this will segfault.
If there isn't a bounds check slightly higher up in this loop, we should add one. Either way, we should definitely add a test using a mocked server response (e.g., via test-tool pkt-line) to ensure the client gracefully dies with a protocol error rather than segfaulting when it receives a malformed packet.
Re: [PATCH GSoC 3/5] fetch-object-info: request all supported options dynamically
In reply to Pablo SabaterQuoted 21 lines · Pablo Sabater
In
send_object_info_request(), size is hardcoded to be the only option sent. In order to support type and future capabilities, replace the hardcoded size with a loop that requests everything onobject_info_optionslist.This is safe because the list has already been trimmed previously in
fetch_object_info()to only contain options that the server supports.Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- fetch-object-info.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)fetch-object-info.c +0
-0
fetch-object-info.c static void send_object_info_request(const int fd_out, struct object_info_args * · line 20
Quoted 10 lines
fetch-object-info.c +3
-4@@ -15,10 +15,13 @@ static void send_object_info_request(const int fd_out, struct object_info_args *write_command_and_capabilities (&req_buf,"object-info" , args->server_options); -if (unsorted_string_list_has_string (args->object_info_options,"size" )) -packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) -BUG ("only size should be in object_info_options" ); +/* +* The list is already checked to only request valid and supported fields +* no need to check, just request everything left on the list
Nit: ...valid and supported fields no need to check.. -> valid and supported fields.
Just request everything remaining on the list.
fetch-object-info.c static void send_object_info_request(const int fd_out, struct object_info_args * · line 24
Quoted 4 lines
fetch-object-info.c +4
-0+*/ +for (size_t i =0 ; i < args->object_info_options->nr; i++) +packet_buf_write (&req_buf,"%s" , + args->object_info_options->items[i].string);
Perfect place to use for_each_string_list_item().
Quoted 6 lines
fetch-object-info.c +0
-1if (args->oids)for (size_t i =0 ; i < args->oids->nr; i++) --2.54.0
Re: [PATCH GSoC 4/5] serve: advertise type capability
In reply to Pablo Sabater[snip]
t/t1017-cat-file-remote-object-info.sh test_expect_success 'remote-object-info does not die on missing oid like info' ' · line 271
Quoted 13 lines
t/t1017-cat-file-remote-object-info.sh +2
-4-# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo $hello_type >expect && git cat-file --batch-command="%(objecttype)" >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOF
Instead of this, what about creating a single test that verifies 'type' is supported,
and modifying this test to verify that the other options are not?
That would actually preserve this test's behaviour and make it easier to extend in the future. Something like:
+ test_expect_success 'type is supported by remote-object-info'
+ test_expect_success 'unsupported placeholder on remote returns empty string'
Re: [PATCH GSoC 5/5] cat-file: unify default format
In reply to Pablo SabaterQuoted 4 lines · Pablo Sabater
%(objecttype) is supported both by the client and by the server. Change the temporary default format to the unified version that other commands use.
s/other/the other
Quoted 4 lines
Update documentation to remove %(objecttype) from the caveats of remote-object-info.
Update tests that used the default format to expect type.
Not super accurate. We're updating the tests to expect the new default format.
Quoted 54 lines
Update documentation to show %(objecttype) support.
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- Documentation/git-cat-file.adoc | 17 ++++----- Documentation/gitprotocol-v2.adoc | 15 ++++++-- builtin/cat-file.c | 7 ---- t/t1017-cat-file-remote-object-info.sh | 70 ++++++++++++---------------------- 4 files changed, 42 insertions(+), 67 deletions(-)Documentation/git-cat-file.adoc +7
-10@@ -348,15 +348,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. -The command `remote-object-info` only supports the `%(objectname)` and -`%(objectsize)` placeholders. See `CAVEATS` below for more information. +The command `remote-object-info` only supports the `%(objectname)`, +`%(objectsize)` and `%(objecttype)` placeholders. See `CAVEATS` below for more +information. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`, except for `remote-object-info` commands which -use `%(objectname) %(objectsize)` because `%(objecttype)` is not supported yet. - -WARNING: When "%(objecttype)" is supported, the default format WILL be unified, -so DO NOT RELY on the current default format to stay the same!!! +%(objecttype) %(objectsize)`. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting@@ -453,9 +450,9 @@ scripting purposes. CAVEATS ------- -Note that only `%(objectname)` and `%(objectsize)` are currently -supported by the `remote-object-info` command. Using any other placeholder in -the format string will return an empty string in its position. +Note that only `%(objectname)`, `%(objectsize)` and `%(objecttype)` are +currently supported by the `remote-object-info` command. Using any other +placeholder in the format string will return an empty string in its position. Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects areDocumentation/gitprotocol-v2.adoc +0
-0
Documentation/gitprotocol-v2.adoc object-info · line 562
Quoted 7 lines
Documentation/gitprotocol-v2.adoc +2
-2@@ -558,14 +558,17 @@ object-info `object-info` is the command to retrieve information about one or more objects. Its main purpose is to allow a client to make decisions based on this -information without having to fully fetch objects. Object size is the only -information that is currently supported. +information without having to fully fetch objects. Object size and type are the +only information that is currently supported.
s/is currently/are currently
Quoted 92 lines
Documentation/gitprotocol-v2.adoc +9
-2An `object-info` request takes the following arguments: size Requests size information to be returned for each listed object id. + type + Requests type information to be returned for each listed object id. + oid <oid> Indicates to the server an object which the client wants to obtain information for. They must be full OIDs.@@ -580,11 +583,15 @@ space. info = *PKT-LINE(attr LF) *PKT-LINE(obj-info LF) - attr = "size" + attr = "size" | "type" obj-size = 1*DIGIT - obj-info = obj-id [SP [obj-size]] + obj-type = "blob" | "tree" | "commit" | "tag" + + obj-val = obj-size | obj-type + + obj-info = obj-id [SP [obj-val *(SP obj-val)]] If the server does not recognize the OID, the response will be `<oid> SP` regardless of the number of attributes requested.builtin/cat-file.c +0
-7@@ -841,15 +841,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,struct object_info *remote_object_info =NULL ;struct oid_array object_info_oids =OID_ARRAY_INIT ;struct string_list object_info_options =STRING_LIST_INIT_NODUP ; -const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )die (_ ("remote-object-info command too long" )); -/* -* TODO: Use the default format once %(objecttype) is supported. -*/ -if (!opt->format) - opt->format ="%(objectname) %(objectsize)" ; line_to_split =xstrdup (line); count =split_cmdline (line_to_split, &argv);@@ -904,7 +898,6 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->is_remote =0 ; } data->skip_object_info =0 ; - opt->format = saved_format;for (size_t i =0 ; i < object_info_oids.nr; i++)free_object_info_contents (&remote_object_info[i]);t/t1017-cat-file-remote-object-info.sh +4
-16@@ -139,10 +139,10 @@ test_expect_success 'batch-command remote-object-info git:// default filter' ' set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "$hello_oid $hello_size" >expect && - echo "$tree_oid $tree_size" >>expect && - echo "$commit_oid $commit_size" >>expect && - echo "$tag_oid $tag_size" >>expect && + echo "$hello_oid $hello_type $hello_size" >expect && + echo "$tree_oid $tree_type $tree_size" >>expect && + echo "$commit_oid $commit_type $commit_size" >>expect && + echo "$tag_oid $tag_type $tag_size" >>expect && git cat-file --batch-command >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid $tree_oid@@ -152,28 +152,6 @@ test_expect_success 'batch-command remote-object-info git:// default filter' ' ) ' -test_expect_success 'remote-object-info does not change the default format of info' ' - ( - set_transport_variables "$daemon_parent" && - cd "$daemon_parent/daemon_client_empty" && - - local_content="local object" && - local_oid=$(echo_without_newline "$local_content" | git hash-object -w --stdin) && - local_size=$(strlen "$local_content") && - - echo "$local_oid blob $local_size" >expect && - echo "$hello_oid $hello_size" >>expect && - echo "$local_oid blob $local_size" >>expect &&
t/t1017-cat-file-remote-object-info.sh test_expect_success 'batch-command remote-object-info git:// default filter' ' · line 176
Quoted 10 lines
t/t1017-cat-file-remote-object-info.sh +0
-10- - git cat-file --batch-command >actual <<-EOF && - info $local_oid - remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid - info $local_oid - EOF - test_cmp expect actual - ) -' -
I feel like deleting this test removes the only test in this file that validates calling info and remote-object-info in the same cat-file --batch-command session.
Instead of deleting it, we should update it. Perhaps something like this:
test "remote-object-info and info can be mixed using the unified default format":
// 1. Environment setup
// 2. Prepare a local object for the 'info' command
// 3. Construct the expected output. Since the default format is
now unified,
// both commands should output exactly: <OID> <TYPE> <SIZE>
// 4. Execute the batch command
// 5. ValidateRe: [PATCH GSoC 1/5] protocol-caps: add type support to object-info
In reply to Chandra PratapQuoted 53 lines · Chandra Pratap
Quoted 38 lines · Pablo Sabater
Teach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While at it, fix
requested_info->sizebit field style.Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 27 +++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 3 deletions(-)protocol-caps.c +5
-1@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/* @@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string;protocol-caps.c static void send_info(struct repository *r, struct packet_writer *writer, · line 90
Quoted 10 lines
protocol-caps.c +3
-1struct object_id oid;size_t object_size; +enum object_type object_type;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) { packet_writer_error( writer, - "object-info: protocol error, expected to get oid, not '%s'", + "object-info: protocol error, expected to get " + "oid, not '%s'",I assume this is a style change? The original line doesn't seem long enough to wrap though.
Quoted 1 line
Also, this would break the grep-ability of this error string.
Yes It is a style change and it's ~60columns long, I'll drop the change. Turns out my nvim settings is showing the vertical guide at ~60 columns instead of 80, but that's on me.
Quoted 58 lines
Quoted 58 lines
protocol-caps.c +10
-1oid_str);continue ; }@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); + write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;t/t5701-git-serve.sh +14
-0@@ -366,6 +366,33 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'type' ' + test_config transfer.advertiseObjectInfo true && + + test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type + oid $(git rev-parse two:two.t) + oid $(git rev-parse two:two.t) + 0000 + EOF +
Quoted 8 lines
t/t5701-git-serve.sh test_expect_success 'basics of object-info' ' · line 387
Quoted 5 lines
t/t5701-git-serve.sh +5
-0+ cat >expect <<-EOF && + size + type + $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blob + $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blobCan we not use the
test_file_sizetool to do this instead? That should also be much more portable.
Yes, I will use it, I didn't know about it.
A test on top of this one does the same pattern, I will fix it too.
Thanks for the feedback,
Pablo
Re: [PATCH GSoC 2/5] fetch-object-info: parse type from server response
In reply to Chandra PratapQuoted 53 lines · Chandra Pratap
Quoted 40 lines · Pablo Sabater
The server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the capability.
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- fetch-object-info.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)fetch-object-info.c +7
-1@@ -50,6 +50,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar const int stateless_rpc, const int fd_out) { int size_index = -1; +int type_index =-1 ;switch (version) {case protocol_v2:@@ -101,8 +102,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arfor (size_t j =0 ; j < args->oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); + }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep)); }else { -BUG ("only size is supported" ); +BUG ("unexpected object-info option: %s" , reader->line); } }fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 161
Quoted 8 lines
fetch-object-info.c +4
-0@@ -148,6 +154,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +string_list_clear (&object_info_values,0 );Is there a risk of an out-of-bounds array access here if the server responds with a truncated or malformed packet?
Quoted 1 line
If object_info_values.nr <=
type_index+ 1, this will segfault.
This shouldn't be a possible case because of:
fetch_object_info()
for (size_t i = 0; i < args->object_info_options->nr; i++) {
[snip]
} else if (!strcmp(reader->line, "type")) {
type_index = (int)i;
[snip]type_index is set based of the range of object_info_options->nr so:
type_index < object_info_options->nr
and a few lines below:
if (args->object_info_options->nr + 1 != object_info_values.nr)
die("object-info: unexpected number of attributes: %s",
reader->line);so we also know that type_index + 1 < object_info_values.nr. After that we get to those lines that this patch introduced:
+ if (type_index >= 0)
+ *object_info_data[i].typep =
+ type_from_string(object_info_values.items[type_index + 1].string);
And because type_index + 1 < object_info_values.nr we can be sure that this cannot segfault once we reach this code.
Quoted 6 lines
If there isn't a bounds check slightly higher up in this loop, we should add one. Either way, we should definitely add a test using a mocked server response (e.g., via test-tool pkt-line) to ensure the client gracefully dies with a protocol error rather than segfaulting when it receives a malformed packet.
Ok, that's sounds a good test, I think there's none where a malicious server is simulated, in part because I don't know how and I think I haven't seen a test that does that yet.
test-tool and pkt-line are used for the opposite: simulating the client to test the real server.
I'll see what I can do about it.
Thanks for the feedback,
Pablo
Re: [PATCH GSoC 3/5] fetch-object-info: request all supported options dynamically
In reply to Chandra PratapQuoted 25 lines · Chandra Pratap
Quoted 24 lines · Pablo Sabater
In
send_object_info_request(), size is hardcoded to be the only option sent. In order to support type and future capabilities, replace the hardcoded size with a loop that requests everything onobject_info_optionslist.This is safe because the list has already been trimmed previously in
fetch_object_info()to only contain options that the server supports.Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- fetch-object-info.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)fetch-object-info.c +0
-0@@ -15,10 +15,13 @@ static void send_object_info_request(const int fd_out, struct object_info_args *write_command_and_capabilities (&req_buf,"object-info" , args->server_options);
Quoted 11 lines
fetch-object-info.c static void send_object_info_request(const int fd_out, struct object_info_args * · line 20
Quoted 7 lines
fetch-object-info.c +3
-4-if (unsorted_string_list_has_string (args->object_info_options,"size" )) -packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) -BUG ("only size should be in object_info_options" ); +/* +* The list is already checked to only request valid and supported fields +* no need to check, just request everything left on the list Nit: ...valid and supported fields no need to check.. -> valid and supported fields.
Just request everything remaining on the list.
Ok, will fix it, thanks.
Quoted 7 lines
fetch-object-info.c static void send_object_info_request(const int fd_out, struct object_info_args * · line 24
Quoted 5 lines
fetch-object-info.c +4
-0+*/ +for (size_t i =0 ; i < args->object_info_options->nr; i++) +packet_buf_write (&req_buf,"%s" , + args->object_info_options->items[i].string);Perfect place to use
for_each_string_list_item().
Didn't think of that, I will change it, thanks.
Quoted 8 lines
Quoted 8 lines
fetch-object-info.c +0
-1if (args->oids)for (size_t i =0 ; i < args->oids->nr; i++) --2.54.0
Re: [PATCH GSoC 4/5] serve: advertise type capability
In reply to Chandra PratapQuoted 19 lines · Chandra Pratap
[snip]
t/t1017-cat-file-remote-object-info.sh test_expect_success 'remote-object-info does not die on missing oid like info' ' · line 271
Quoted 13 lines
t/t1017-cat-file-remote-object-info.sh +2
-4-# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo $hello_type >expect && git cat-file --batch-command="%(objecttype)" >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOFInstead of this, what about creating a single test that verifies 'type' is supported,
and modifying this test to verify that the other options are not?
Quoted 5 lines
That would actually preserve this test's behaviour and make it easier to extend in the future. Something like:
+
test_expect_success'type is supported by remote-object-info'
+test_expect_success'unsupported placeholder on remote returns empty string'
Hmmm, it is true that after this test there are no test with the git://
that test for known but unsupported placeholders returning empty strings, but there are tests from the previous series that does test for others that are still unsupported like objectmode, deltabase, objectsize:disk.
I think on more test grouping all the one that remain unsupported will be good and document.
I'll do exactly what you proposed, thanks,
Pablo
Re: [PATCH GSoC 5/5] cat-file: unify default format
In reply to Chandra PratapQuoted 7 lines · Chandra Pratap
Quoted 4 lines · Pablo Sabater
%(objecttype) is supported both by the client and by the server. Change the temporary default format to the unified version that other commands use.
s/other/the other
Will fix thanks.
Quoted 8 lines
Quoted 5 lines
Update documentation to remove %(objecttype) from the caveats of remote-object-info.
Update tests that used the default format to expect type.
Not super accurate. We're updating the tests to expect the new default format.
I will reword it to something like:
Now that type is supported and the default format unified, update the tests to expect the new default format.
Quoted 56 lines
Quoted 56 lines
Update documentation to show %(objecttype) support.
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- Documentation/git-cat-file.adoc | 17 ++++----- Documentation/gitprotocol-v2.adoc | 15 ++++++-- builtin/cat-file.c | 7 ---- t/t1017-cat-file-remote-object-info.sh | 70 ++++++++++++---------------------- 4 files changed, 42 insertions(+), 67 deletions(-)Documentation/git-cat-file.adoc +7
-10@@ -348,15 +348,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. -The command `remote-object-info` only supports the `%(objectname)` and -`%(objectsize)` placeholders. See `CAVEATS` below for more information. +The command `remote-object-info` only supports the `%(objectname)`, +`%(objectsize)` and `%(objecttype)` placeholders. See `CAVEATS` below for more +information. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`, except for `remote-object-info` commands which -use `%(objectname) %(objectsize)` because `%(objecttype)` is not supported yet. - -WARNING: When "%(objecttype)" is supported, the default format WILL be unified, -so DO NOT RELY on the current default format to stay the same!!! +%(objecttype) %(objectsize)`. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting@@ -453,9 +450,9 @@ scripting purposes. CAVEATS ------- -Note that only `%(objectname)` and `%(objectsize)` are currently -supported by the `remote-object-info` command. Using any other placeholder in -the format string will return an empty string in its position. +Note that only `%(objectname)`, `%(objectsize)` and `%(objecttype)` are +currently supported by the `remote-object-info` command. Using any other +placeholder in the format string will return an empty string in its position. Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects areDocumentation/gitprotocol-v2.adoc +0
-0@@ -558,14 +558,17 @@ object-info
Quoted 8 lines
Documentation/gitprotocol-v2.adoc object-info · line 562
Quoted 6 lines
Documentation/gitprotocol-v2.adoc +2
-2`object-info` is the command to retrieve information about one or more objects. Its main purpose is to allow a client to make decisions based on this -information without having to fully fetch objects. Object size is the only -information that is currently supported. +information without having to fully fetch objects. Object size and type are the +only information that is currently supported.s/is currently/are currently
Will fix.
Quoted 112 lines
Quoted 93 lines
Documentation/gitprotocol-v2.adoc +9
-2An `object-info` request takes the following arguments: size Requests size information to be returned for each listed object id. + type + Requests type information to be returned for each listed object id. + oid <oid> Indicates to the server an object which the client wants to obtain information for. They must be full OIDs.@@ -580,11 +583,15 @@ space. info = *PKT-LINE(attr LF) *PKT-LINE(obj-info LF) - attr = "size" + attr = "size" | "type" obj-size = 1*DIGIT - obj-info = obj-id [SP [obj-size]] + obj-type = "blob" | "tree" | "commit" | "tag" + + obj-val = obj-size | obj-type + + obj-info = obj-id [SP [obj-val *(SP obj-val)]] If the server does not recognize the OID, the response will be `<oid> SP` regardless of the number of attributes requested.builtin/cat-file.c +0
-7@@ -841,15 +841,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,struct object_info *remote_object_info =NULL ;struct oid_array object_info_oids =OID_ARRAY_INIT ;struct string_list object_info_options =STRING_LIST_INIT_NODUP ; -const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )die (_ ("remote-object-info command too long" )); -/* -* TODO: Use the default format once %(objecttype) is supported. -*/ -if (!opt->format) - opt->format ="%(objectname) %(objectsize)" ; line_to_split =xstrdup (line); count =split_cmdline (line_to_split, &argv);@@ -904,7 +898,6 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->is_remote =0 ; } data->skip_object_info =0 ; - opt->format = saved_format;for (size_t i =0 ; i < object_info_oids.nr; i++)free_object_info_contents (&remote_object_info[i]);t/t1017-cat-file-remote-object-info.sh +4
-16@@ -139,10 +139,10 @@ test_expect_success 'batch-command remote-object-info git:// default filter' ' set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "$hello_oid $hello_size" >expect && - echo "$tree_oid $tree_size" >>expect && - echo "$commit_oid $commit_size" >>expect && - echo "$tag_oid $tag_size" >>expect && + echo "$hello_oid $hello_type $hello_size" >expect && + echo "$tree_oid $tree_type $tree_size" >>expect && + echo "$commit_oid $commit_type $commit_size" >>expect && + echo "$tag_oid $tag_type $tag_size" >>expect && git cat-file --batch-command >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid $tree_oid@@ -152,28 +152,6 @@ test_expect_success 'batch-command remote-object-info git:// default filter' ' ) ' -test_expect_success 'remote-object-info does not change the default format of info' ' - ( - set_transport_variables "$daemon_parent" && - cd "$daemon_parent/daemon_client_empty" && - - local_content="local object" && - local_oid=$(echo_without_newline "$local_content" | git hash-object -w --stdin) && - local_size=$(strlen "$local_content") && - - echo "$local_oid blob $local_size" >expect && - echo "$hello_oid $hello_size" >>expect && - echo "$local_oid blob $local_size" >>expect &&t/t1017-cat-file-remote-object-info.sh test_expect_success 'batch-command remote-object-info git:// default filter' ' · line 176
Quoted 10 lines
t/t1017-cat-file-remote-object-info.sh +0
-10- - git cat-file --batch-command >actual <<-EOF && - info $local_oid - remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid - info $local_oid - EOF - test_cmp expect actual - ) -' -I feel like deleting this test removes the only test in this file that validates calling info and remote-object-info in the same cat-file --batch-command session.
Instead of deleting it, we should update it. Perhaps something like this:
test "remote-object-info and info can be mixed using the unified default format":
Quoted 7 lines
// 1. Environment setup
// 2. Prepare a local object for the 'info' command
// 3. Construct the expected output. Since the default format is
now unified,
// both commands should output exactly: <OID> <TYPE> <SIZE>
// 4. Execute the batch command
// 5. ValidateThis test was because we were modifying the default format whenever remote-object-info was used. The workaround to have them work together was to store the previous format, save the format as the capped one for remote-object-info and at the end restore it.
Now that it has been unified for both it is implicit that they can work together. However, no harm in having a test explicitly proving it. I will make what you suggest.
Thanks for the feedback,
Pablo
Quoted 22 lines · Chandra Pratap
Quoted 12 lines · Pablo Sabater
"%(objecttype)" is already known by the client's allow-list, but neither the client nor the server knows how to handle type. This series continues the work for git cat-file --batch-command extending remote-object-info to support "%(objecttype)" end to end. It is based on its predecessor series "cat-file: add remote-object-info to batch-command" [1].
Type is the last property that is identical on the server and on the client once fetched.
Whether to support more metadata such as:
Quoted 8 lines
- objectsize:disk
- deltabase
- objectmode (needs context)should be discussed, unlike size and type, the rest depend on how things are packed on the server and on what the client already has, so they cannot reliably match local once fetched. IMO they are not worth supporting, I can't find a use case for them.
Quoted 5 lines
Makes sense to me. The metadata you listed depends on the local packfile storage where the repository lives.
objectmodedepends on the tree pointing to the blob, so it wouldn't be accessible using the current infrastructure anyway.
Yup, sounds like this will be it (for now) for remote-object-info placeholders.
Quoted 25 lines
Quoted 11 lines
Adding new placeholders has become trivial.
To add a new placeholder, follow the steps in this series and add it to the client's allow-list at 'builtin/cat-file.c'.Based-on: <[email protected]>
Github CI: https://github.com/pabloosabaterr/git/actions/runs/30155586279
Quoted 8 lines
Signed-off-by Pablo Sabater <[email protected]>---
Pablo Sabater (5):
protocol-caps: add type support to object-info
fetch-object-info: parse type from server response
fetch-object-info: request all supported options dynamically
serve: advertise type capability
cat-file: unify default formatThe current incremental approach is safe and ensures every commit compiles and passes tests. However, from a storytelling perspective for the reviewers, I believe it's better to do any 'preparatory refactoring' before starting the new feature.
Quoted 4 lines
Patch 3 (dynamically requesting supported options) doesn't actually depend on type existing yet. I suggest bumping Patch 3 to be Patch 1 in V2. That way, the client is already dynamic and ready, and the feature patches can strictly focus on adding type:
True, I'll move patch 3 to be the first, thanks for noticing.
Quoted 17 lines
- fetch-object-info: request all supported options dynamically (Current Patch 3)
- protocol-caps: add type support to object-info (Current Patch 1)
- fetch-object-info: parse type from server response (Current Patch 2)
- serve: advertise type capability (Current Patch 4)
- cat-file: unify default format (Current Patch 5)Quoted 9 lines
Documentation/git-cat-file.adoc | 17 +++----- Documentation/gitprotocol-v2.adoc | 15 +++++-- builtin/cat-file.c | 7 --- fetch-object-info.c | 23 +++++++--- protocol-caps.c | 21 +++++++-- serve.c | 4 +- t/t1017-cat-file-remote-object-info.sh | 80 +++++++++++++--------------------- t/t5701-git-serve.sh | 27 ++++++++++++ 8 files changed, 113 insertions(+), 81 deletions(-)
Quoted 2 lines
This series is definitely a lot smaller than I thought it would be. Looks like most of the heavy lifting was already done with the previous series.
Yes I also got surprised by how little I had to do in order to make this series happen (I expected a lot more) but it turned out to be very brief.
Quoted 2 lines
Good for us!
Yay! :)
Re: [PATCH GSoC 1/5] protocol-caps: add type support to object-info
In reply to Chandra PratapQuoted 9 lines · Chandra Pratap
protocol-caps.c static void send_info(struct repository *r, struct packet_writer *writer, · line 90
Quoted 6 lines
if (get_oid_hex_algop(oid_str, &oid, r->hash_algo) < 0) { packet_writer_error( writer,protocol-caps.c +2
-1- "object-info: protocol error, expected to get oid, not '%s'", +"object-info: protocol error, expected to get " + "oid, not '%s'",I assume this is a style change? The original line doesn't seem long enough to wrap though.
A 92-column line is much wider than our usual ~70 column limit.
Quoted 1 line
Also, this would break the grep-ability of this error string.
I've heard this a few times, but so what? You can still grep for "object-info: protocol error," and will find it just fine.
Quoted 8 lines
t/t5701-git-serve.sh test_expect_success 'basics of object-info' ' · line 387
Quoted 5 lines
t/t5701-git-serve.sh +5
-0+ cat >expect <<-EOF && + size + type + $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blob + $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blobCan we not use the
test_file_sizetool to do this instead? That should also be much more portable.
Well spotted.
Re: [PATCH GSoC 2/5] fetch-object-info: parse type from server response
In reply to Pablo Sabater[snip]
Quoted 41 lines
Quoted 13 lines
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 161
Quoted 9 lines
fetch-object-info.c +4
-0@@ -148,6 +154,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +string_list_clear (&object_info_values,0 );Is there a risk of an out-of-bounds array access here if the server responds with a truncated or malformed packet?
Quoted 1 line
If object_info_values.nr <=
type_index+ 1, this will segfault.This shouldn't be a possible case because of:
fetch_object_info()for (size_t i = 0; i < args->object_info_options->nr; i++) { [snip] } else if (!strcmp(reader->line, "type")) { type_index = (int)i; [snip]
type_indexis set based of the range ofobject_info_options->nrso:
type_index<object_info_options->nrand a few lines below:
if (args->object_info_options->nr + 1 != object_info_values.nr) die("object-info: unexpected number of attributes: %s", reader->line);so we also know that
type_index+ 1 < object_info_values.nr. After that we get to those lines that this patch introduced:
Quoted 6 lines
+ if (
type_index>= 0)
+ *object_info_data[i].typep =
+type_from_string(object_info_values.items[type_index + 1].string);And because
type_index+ 1 < object_info_values.nr we can be sure that this cannot segfault once we reach this code.
Makes sense to me.
Quoted 7 lines
Quoted 6 lines
If there isn't a bounds check slightly higher up in this loop, we should add one. Either way, we should definitely add a test using a mocked server response (e.g., via test-tool pkt-line) to ensure the client gracefully dies with a protocol error rather than segfaulting when it receives a malformed packet.
Quoted 7 lines
Ok, that's sounds a good test, I think there's none where a malicious server is simulated, in part because I don't know how and I think I haven't seen a test that does that yet.
test-tool and pkt-line are used for the opposite: simulating the client to test the real server.I'll see what I can do about it.
Yeah, I wouldn't recommend breaking your back for it though. We already have tests exploring the happy paths, so something that simply verifies our expectations for error paths (printing an empty string in this case) should be good enough.
Re: [PATCH GSoC 1/5] protocol-caps: add type support to object-info
In reply to Pablo SabaterQuoted 31 lines · Pablo Sabater
Teach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While at it, fix
requested_info->sizebit field style.Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 27 +++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 3 deletions(-)protocol-caps.c +2
-1@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/* @@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" );
protocol-caps.c static void send_info(struct repository *r, struct packet_writer *writer, · line 85
Quoted 10 lines
protocol-caps.c +4
-0+if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string;struct object_id oid;size_t object_size; +enum object_type object_type;
This is fine, but If you do reiterate, maybe we can follow the reverse christmas tree [1] format and move this field up? Or maybe its just me...
protocol-caps.c static void send_info(struct repository *r, struct packet_writer *writer, · line 90
Quoted 6 lines
protocol-caps.c +2
-1if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) { packet_writer_error( writer, - "object-info: protocol error, expected to get oid, not '%s'", + "object-info: protocol error, expected to get " + "oid, not '%s'",
I know this is to fix styling, but we generally don't do such changes unless we're touching the code or around it. It just tends to be a distraction :)
[snip]
Re: [PATCH GSoC 2/5] fetch-object-info: parse type from server response
In reply to Pablo SabaterQuoted 4 lines · Pablo Sabater
The server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the capability.
Nit, I was a bit confused by the flow from the first to the second sentence. Maybe:
The server can handle type requests but does not advertise the capability yet. As a percursor, prepare the client to know how to parse the server response.
Quoted 35 lines
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- fetch-object-info.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)fetch-object-info.c +7
-1@@ -50,6 +50,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar const int stateless_rpc, const int fd_out) { int size_index = -1; +int type_index =-1 ;switch (version) {case protocol_v2:@@ -101,8 +102,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arfor (size_t j =0 ; j < args->oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); + }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep)); }else { -BUG ("only size is supported" ); +BUG ("unexpected object-info option: %s" , reader->line); } }
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 160
Quoted 7 lines
fetch-object-info.c +4
-0@@ -148,6 +154,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +
For size, passing the value as a number makes sense, since the value represents the field directly.
For type however, we're passing in a value, while the actual field is a string. We rely on type_from_string() to make that translation for us, which internatlly depends on object_type_strings[]. What if there is a mismatch between the server and the client? Shouldn't we be sending in the string itself?
Quoted 6 lines
fetch-object-info.c +0
-1string_list_clear (&object_info_values,0 ); }check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" ); --2.54.0
Re: [PATCH GSoC 2/5] fetch-object-info: parse type from server response
Quoted 12 lines · Karthik Nayak
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 160
Quoted 8 lines
fetch-object-info.c +4
-0@@ -148,6 +154,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +For size, passing the value as a number makes sense, since the value represents the field directly.
Quoted 12 lines
For type however, we're passing in a value, while the actual field is a string. We rely on
type_from_string()to make that translation for us, which internatlly depends onobject_type_strings[]. What if there is a mismatch between the server and the client? Shouldn't we be sending in the string itself?Quoted 6 lines
fetch-object-info.c +0
-1string_list_clear (&object_info_values,0 ); }check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" ); --2.54.0
Ah! We do send in the string itself, all good here :)
v2 cover cat-file: extend remote-object-info to support %(objecttype)
In reply to Pablo Sabater"%(objecttype)" is already known by the client's allow-list, but neither the client nor the server knows how to handle type. This series continues the work for git cat-file --batch-command extending remote-object-info to support "%(objecttype)" end to end. It is based on its predecessor series "cat-file: add remote-object-info to batch-command" [1].
Type is the last property that is identical on the server and on the client once fetched.
Whether to support more metadata such as:
- objectsize:disk
- deltabase
- objectmode (needs context)
should be discussed, unlike size and type, the rest depend on how things are packed on the server and on what the client already has, so they cannot reliably match local once fetched. IMO they are not worth supporting, I can't find a use case for them.
Adding new placeholders has become trivial.
To add a new placeholder, follow the steps in this series and add it to the client's allow-list at 'builtin/cat-file.c'.
Based-on: <[email protected]>
Github CI: https://github.com/pabloosabaterr/git/actions/runs/30660227202
[1]: https://lore.kernel.org/git/[email protected]/
---
Changes in v2:
- Changed t5701 test to use test_file_size.
- Changed the for loop to use for_each_string_list_item().
- Moved "fetch-object-info: request all supported options dynamically"
patch to be the first one as it is prep patch.
- Typos and nits.
- Restored the test mixing info and remote-object-info, adapted to the
unified format.
- Link to v1: https://lore.kernel.org/git/[email protected]/
---
Pablo Sabater (6):
fetch-object-info: request all supported options dynamically
t5701: use the test_file_size() helper
protocol-caps: add type support to object-info
fetch-object-info: parse type from server response
serve: advertise type capability
cat-file: unify default format
Documentation/git-cat-file.adoc | 17 +++----- Documentation/gitprotocol-v2.adoc | 15 +++++-- builtin/cat-file.c | 7 --- fetch-object-info.c | 23 +++++++--- protocol-caps.c | 21 +++++++-- serve.c | 4 +- t/t1017-cat-file-remote-object-info.sh | 78 +++++++++++++++++++++------------- t/t5701-git-serve.sh | 31 +++++++++++++- 8 files changed, 133 insertions(+), 63 deletions(-)
Range-diff versus v1:
-: ---------- > 1: f635361786 fetch-object-info: request all supported options dynamically
-: ---------- > 2: 5195b2db2e t5701: use the test_file_size() helper
1: acdff1bbd3 ! 3: 08a4a86ac9 protocol-caps: add type support to object-info
@@ Commit message
field. When the client includes type in its object-info request, the
server returns the requested object type.
- While at it, fix requested_info->size bit field style.
+ While touching send_info(), wrap an over-long line and fix the bit field
+ style of requested_info.size.
Mentored-by: Karthik Nayak <[email protected]>
Mentored-by: Chandra Pratap <[email protected]>
@@ protocol-caps.c: static void send_info(struct repository *r, struct packet_write
+
for_each_string_list_item (item, oid_str_list) {
const char *oid_str = item->string;
++ enum object_type object_type;
struct object_id oid;
size_t object_size;
-+ enum object_type object_type;
if (get_oid_hex_algop(oid_str, &oid, r->hash_algo) < 0) {
packet_writer_error(
@@ t/t5701-git-serve.sh: test_expect_success 'basics of object-info' '
test_cmp expect actual
'
-+test_expect_success 'type' '
++test_expect_success 'object-info supports type' '
+ test_config transfer.advertiseObjectInfo true &&
+
+ test-tool pkt-line pack >in <<-EOF &&
@@ t/t5701-git-serve.sh: test_expect_success 'basics of object-info' '
+ cat >expect <<-EOF &&
+ size
+ type
-+ $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blob
-+ $(git rev-parse two:two.t) $(wc -c <two.t | xargs) blob
++ $(git rev-parse two:two.t) $(test_file_size two.t) blob
++ $(git rev-parse two:two.t) $(test_file_size two.t) blob
+ 0000
+ EOF
+
2: c1a8a83acf = 4: 4d8e2ddb5d fetch-object-info: parse type from server response
3: ed0215c3e5 < -: ---------- fetch-object-info: request all supported options dynamically
4: 27efd907bd ! 5: 973b490b75 serve: advertise type capability
@@ t/t1017-cat-file-remote-object-info.sh: test_expect_success 'remote-object-info
git cat-file --batch-command="%(objecttype)" >actual <<-EOF &&
remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid
EOF
+@@ t/t1017-cat-file-remote-object-info.sh: test_expect_success 'unsupported placeholder on remote returns empty string' '
+ )
+ '
+
++test_expect_success 'unsupported placeholders on remote return empty string' '
++ (
++ set_transport_variables "$daemon_parent" &&
++ cd "$daemon_parent/daemon_client_empty" &&
++
++ fmt="%(objectmode) %(objectsize:disk) %(rest) %(deltabase)" &&
++
++ # The hardcoded SPs between the atoms are respected.
++ echo " " >expect &&
++ git cat-file --batch-command="$fmt" >actual <<-EOF &&
++ remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid
++ EOF
++ test_cmp expect actual
++ )
++'
++
+ test_expect_success 'requesting only objectname echoes back' '
+ (
+ set_transport_variables "$daemon_parent" &&5: ab5a44d40e ! 6: 1668033b31 cat-file: unify default format
@@ Commit message
cat-file: unify default format
%(objecttype) is supported both by the client and by the server.
- Change the temporary default format to the unified version that other
- commands use.
+ Change the temporary default format to the unified version that the
+ other commands use.
Update documentation to remove %(objecttype) from the caveats of
- remote-object-info.
+ remote-object-info and show %(objecttype) support.
- Update tests that used the default format to expect type.
- Update documentation to show %(objecttype) support.
+ Now that type is supported and the default format unified, update the
+ tests to expect the new default format.
@@ Documentation/gitprotocol-v2.adoc: object-info
-information without having to fully fetch objects. Object size is the only -information that is currently supported.
+information without having to fully fetch objects. Object size and type are the
-+only information that is currently supported.
++only information currently supported.
An object-info request takes the following arguments:
@@ t/t1017-cat-file-remote-object-info.sh: test_expect_success 'batch-command remot
'
-test_expect_success 'remote-object-info does not change the default format of info' '
-- (
-- set_transport_variables "$daemon_parent" &&
-- cd "$daemon_parent/daemon_client_empty" &&
--
-- local_content="local object" &&
-- local_oid=$(echo_without_newline "$local_content" | git hash-object -w --stdin) &&
-- local_size=$(strlen "$local_content") &&
--
-- echo "$local_oid blob $local_size" >expect &&
-- echo "$hello_oid $hello_size" >>expect &&
-- echo "$local_oid blob $local_size" >>expect &&
--
-- git cat-file --batch-command >actual <<-EOF &&
-- info $local_oid
-- remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid
-- info $local_oid
-- EOF
-- test_cmp expect actual
-- )
--'
--
- test_expect_success 'batch-command --buffer remote-object-info git://' '
++test_expect_success 'remote-object-info and info can be mixed using the unified default format' '
(
set_transport_variables "$daemon_parent" &&
+ cd "$daemon_parent/daemon_client_empty" &&
+@@ t/t1017-cat-file-remote-object-info.sh: test_expect_success 'remote-object-info does not change the default format of in
+ local_size=$(strlen "$local_content") &&
+
+ echo "$local_oid blob $local_size" >expect &&
+- echo "$hello_oid $hello_size" >>expect &&
++ echo "$hello_oid blob $hello_size" >>expect &&
+ echo "$local_oid blob $local_size" >>expect &&
+
+ git cat-file --batch-command >actual <<-EOF &&
@@ t/t1017-cat-file-remote-object-info.sh: test_expect_success 'batch-command -Z remote-object-info git:// default filter'
set_transport_variables "$daemon_parent" &&
cd "$daemon_parent/daemon_client_empty" &&
---
base-commit: 71e19e8d2713f385c3fcef59cf6f29bcbd93d91f
change-id: 20260724-objecttype-support-ea1ef6941d07
PATCH v2 1/6 fetch-object-info: request all supported options dynamically
In send_object_info_request(), size is hardcoded to be the only option sent. In order to support type and future capabilities, replace the hardcoded size with a loop that requests everything on object_info_options list.
This is safe because the list has already been trimmed previously in fetch_object_info() to only contain options that the server supports.
--- fetch-object-info.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
fetch-object-info.c +7 -4
@@ -12,13 +12,16 @@static void send_object_info_request (const int fd_out,struct object_info_args *args) {struct strbuf req_buf =STRBUF_INIT ; +struct string_list_item *item;write_command_and_capabilities (&req_buf,"object-info" , args->server_options); -if (unsorted_string_list_has_string (args->object_info_options,"size" )) -packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) -BUG ("only size should be in object_info_options" ); +/* +* The list has already been checked to only contain valid and +* supported fields, so just request everything remaining on it. +*/ +for_each_string_list_item (item, args->object_info_options) +packet_buf_write (&req_buf,"%s" , item->string);if (args->oids)for (size_t i =0 ; i < args->oids->nr; i++)
Signature · 1 line
2.54.0
PATCH v2 2/6 t5701: use the test_file_size() helper
In reply to Pablo SabaterAn object-info test uses 'wc -c <two.t | xargs' to get the file size. Update it to use the test_file_size() helper instead.
--- t/t5701-git-serve.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
t/t5701-git-serve.sh +2 -2
@@ -356,8 +356,8 @@ test_expect_success 'basics of object-info' ' cat >expect <<-EOF && size - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) + $(git rev-parse two:two.t) $(test_file_size two.t) + $(git rev-parse two:two.t) $(test_file_size two.t) 0000 EOF
Signature · 1 line
2.54.0
PATCH v2 3/6 protocol-caps: add type support to object-info
In reply to Pablo SabaterTeach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While touching send_info(), wrap an over-long line and fix the bit field style of requested_info.size.
--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 27 +++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 3 deletions(-)
protocol-caps.c +18 -3
@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/*
@@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string; +enum object_type object_type;struct object_id oid;size_t object_size;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) {packet_writer_error ( writer, -"object-info: protocol error, expected to get oid, not '%s'" , +"object-info: protocol error, expected to get " +"oid, not '%s'" , oid_str);continue ; }
@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }
@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); + write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);
@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;
t/t5701-git-serve.sh +27 -0
@@ -366,6 +366,33 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'object-info supports type' ' + test_config transfer.advertiseObjectInfo true && + + test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type + oid $(git rev-parse two:two.t) + oid $(git rev-parse two:two.t) + 0000 + EOF + + cat >expect <<-EOF && + size + type + $(git rev-parse two:two.t) $(test_file_size two.t) blob + $(git rev-parse two:two.t) $(test_file_size two.t) blob + 0000 + EOF + + test-tool serve-v2 --stateless-rpc <in >out && + test-tool pkt-line unpack <out >actual && + test_cmp expect actual +' + test_expect_success 'bare OID request' ' test_config transfer.advertiseObjectInfo true &&
Signature · 1 line
2.54.0
PATCH v2 4/6 fetch-object-info: parse type from server response
In reply to Pablo SabaterThe server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the capability.
--- fetch-object-info.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
fetch-object-info.c +11 -1
@@ -53,6 +53,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar const int stateless_rpc, const int fd_out) { int size_index = -1; +int type_index =-1 ;switch (version) {case protocol_v2:
@@ -104,8 +105,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arfor (size_t j =0 ; j < args->oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); + }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep)); }else { -BUG ("only size is supported" ); +BUG ("unexpected object-info option: %s" , reader->line); } }
@@ -151,6 +157,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +string_list_clear (&object_info_values,0 ); }check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
Signature · 1 line
2.54.0
PATCH v2 5/6 serve: advertise type capability
In reply to Pablo SabaterThe server and the client can handle type requests but the client won't ask for it until the server advertises it.
Add type to the advertised capabilities so the client knows that it can request it.
--- serve.c | 4 ++-- t/t1017-cat-file-remote-object-info.sh | 26 ++++++++++++++++++++++---- 2 files changed, 24 insertions(+), 6 deletions(-)
serve.c +2 -2
@@ -97,9 +97,9 @@ static int object_info_advertise(struct repository *r, struct strbuf *value)/* disabled by default */ advertise_object_info =0 ; } -/* Currently only size is supported */ +/* Currently only size and type are supported */ if (value && advertise_object_info) -strbuf_addstr (value,"size" ); +strbuf_addstr (value,"size type" );return advertise_object_info; }
t/t1017-cat-file-remote-object-info.sh +22 -4
@@ -7,6 +7,7 @@ test_description='git cat-file --batch-command with remote-object-info command' hello_content="Hello World" hello_size=$(strlen "$hello_content") +hello_type="blob" hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) hello_short_oid=$(git rev-parse --short "$hello_oid")
@@ -19,6 +20,7 @@ unstored_oid=$(echo_without_newline "$unstored_content" | git hash-object --stdi # file name is hello, which is 5 characters # a space is 1 character and a null is 1 character tree_size=$(($(test_oid rawsz) + 13)) +tree_type="tree" commit_message="Initial commit"
@@ -31,6 +33,7 @@ commit_message="Initial commit" # An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`, # to get 177, 2. then deduct 40 hex characters to get 137 commit_size=$(($(test_oid hexsz) + 137)) +commit_type="commit" tag_header_without_oid="type blob tag hellotag
@@ -44,6 +47,7 @@ $tag_description" tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w) tag_size=$(strlen "$tag_content") +tag_type="tag" set_transport_variables () { hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin)
@@ -256,14 +260,12 @@ test_expect_success 'remote-object-info does not die on missing oid like info' ' ) ' -# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo $hello_type >expect && git cat-file --batch-command="%(objecttype)" >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOF
@@ -271,6 +273,22 @@ test_expect_success 'unsupported placeholder on remote returns empty string' ' ) ' +test_expect_success 'unsupported placeholders on remote return empty string' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + fmt="%(objectmode) %(objectsize:disk) %(rest) %(deltabase)" && + + # The hardcoded SPs between the atoms are respected. + echo " " >expect && + git cat-file --batch-command="$fmt" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid + EOF + test_cmp expect actual + ) +' + test_expect_success 'requesting only objectname echoes back' ' ( set_transport_variables "$daemon_parent" &&
Signature · 1 line
2.54.0
PATCH v2 6/6 cat-file: unify default format
In reply to Pablo Sabater%(objecttype) is supported both by the client and by the server. Change the temporary default format to the unified version that the other commands use.
Update documentation to remove %(objecttype) from the caveats of remote-object-info and show %(objecttype) support.
Now that type is supported and the default format unified, update the tests to expect the new default format.
--- Documentation/git-cat-file.adoc | 17 +++++------ Documentation/gitprotocol-v2.adoc | 15 +++++++--- builtin/cat-file.c | 7 ----- t/t1017-cat-file-remote-object-info.sh | 52 +++++++++++++++++----------------- 4 files changed, 44 insertions(+), 47 deletions(-)
Documentation/git-cat-file.adoc +7 -10
@@ -348,15 +348,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. -The command `remote-object-info` only supports the `%(objectname)` and -`%(objectsize)` placeholders. See `CAVEATS` below for more information. +The command `remote-object-info` only supports the `%(objectname)`, +`%(objectsize)` and `%(objecttype)` placeholders. See `CAVEATS` below for more +information. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`, except for `remote-object-info` commands which -use `%(objectname) %(objectsize)` because `%(objecttype)` is not supported yet. - -WARNING: When "%(objecttype)" is supported, the default format WILL be unified, -so DO NOT RELY on the current default format to stay the same!!! +%(objecttype) %(objectsize)`. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -453,9 +450,9 @@ scripting purposes. CAVEATS ------- -Note that only `%(objectname)` and `%(objectsize)` are currently -supported by the `remote-object-info` command. Using any other placeholder in -the format string will return an empty string in its position. +Note that only `%(objectname)`, `%(objectsize)` and `%(objecttype)` are +currently supported by the `remote-object-info` command. Using any other +placeholder in the format string will return an empty string in its position. Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are
Documentation/gitprotocol-v2.adoc +11 -4
@@ -558,14 +558,17 @@ object-info `object-info` is the command to retrieve information about one or more objects. Its main purpose is to allow a client to make decisions based on this -information without having to fully fetch objects. Object size is the only -information that is currently supported. +information without having to fully fetch objects. Object size and type are the +only information currently supported. An `object-info` request takes the following arguments: size Requests size information to be returned for each listed object id. + type + Requests type information to be returned for each listed object id. + oid <oid> Indicates to the server an object which the client wants to obtain information for. They must be full OIDs.
@@ -580,11 +583,15 @@ space. info = *PKT-LINE(attr LF) *PKT-LINE(obj-info LF) - attr = "size" + attr = "size" | "type" obj-size = 1*DIGIT - obj-info = obj-id [SP [obj-size]] + obj-type = "blob" | "tree" | "commit" | "tag" + + obj-val = obj-size | obj-type + + obj-info = obj-id [SP [obj-val *(SP obj-val)]] If the server does not recognize the OID, the response will be `<oid> SP` regardless of the number of attributes requested.
builtin/cat-file.c +0 -7
@@ -841,15 +841,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,struct object_info *remote_object_info =NULL ;struct oid_array object_info_oids =OID_ARRAY_INIT ;struct string_list object_info_options =STRING_LIST_INIT_NODUP ; -const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )die (_ ("remote-object-info command too long" )); -/* -* TODO: Use the default format once %(objecttype) is supported. -*/ -if (!opt->format) - opt->format ="%(objectname) %(objectsize)" ; line_to_split =xstrdup (line); count =split_cmdline (line_to_split, &argv);
@@ -904,7 +898,6 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->is_remote =0 ; } data->skip_object_info =0 ; - opt->format = saved_format;for (size_t i =0 ; i < object_info_oids.nr; i++)free_object_info_contents (&remote_object_info[i]);
t/t1017-cat-file-remote-object-info.sh +26 -26
100 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH GSoC v2 1/6] fetch-object-info: request all supported options dynamically
In reply to Pablo SabaterQuoted 7 lines · Pablo Sabater
In
send_object_info_request(), size is hardcoded to be the only option sent. In order to support type and future capabilities, replace the hardcoded size with a loop that requests everything onobject_info_optionslist.This is safe because the list has already been trimmed previously in
fetch_object_info()to only contain options that the server supports.
Thinking along and aloud to follow the code, my understanding of how the relevant data flows in the code path to get here is as follows:
* 'cat-file --batch-command' processes the remote-object-info command, and parse_cmd_remote_object_info() populates the object_info_options string list.
* It puts 'size' and 'type' into the list if needed.
* get_remote_info() is called, and the string list is attached to the '.object_info_options' member of the '.smart_options' structure of transport.
* transport_fetch_object_info() calls fetch_object_info_via_pack(), where the local args structure receives in its '.object_info_options' member the value pointed to by the '.object_info_options' member of the '.smart_options' structure of 'gtransport'.
* fetch_object_info_via_pack() finally calls fetch_object_info(), which uses server_supports_feature() to check and discard elements from this list that are not supported by the server.
This is how the 'args' structure seen by send_object_info_request() gets prepared. The "already been checked to only contain" comment in the code refers to the loop in fetch_object_info() that uses server_supports_feature().
So, after tracing the code flow up to this point, I agree with the "This is safe" claim made in the proposed commit log message.
I always get confused while following code paths in the transport layer; my ulterior motivation for this comment is that writing it down once may help refresh my memory the next time I need it.
Thanks.
Quoted 32 lines
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- fetch-object-info.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)fetch-object-info.c +7
-4@@ -12,13 +12,16 @@static void send_object_info_request (const int fd_out,struct object_info_args *args) {struct strbuf req_buf =STRBUF_INIT ; +struct string_list_item *item;write_command_and_capabilities (&req_buf,"object-info" , args->server_options); -if (unsorted_string_list_has_string (args->object_info_options,"size" )) -packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) -BUG ("only size should be in object_info_options" ); +/* +* The list has already been checked to only contain valid and +* supported fields, so just request everything remaining on it. +*/ +for_each_string_list_item (item, args->object_info_options) +packet_buf_write (&req_buf,"%s" , item->string);if (args->oids)for (size_t i =0 ; i < args->oids->nr; i++)
Re: [PATCH GSoC v2 2/6] t5701: use the test_file_size() helper
In reply to Pablo SabaterQuoted 2 lines · Pablo Sabater
An object-info test uses 'wc -c <two.t | xargs' to get the file size. Update it to use the
test_file_size()helper instead.
What is missing from this description is what is wrong with the use of that "wc -c | xargs" construct. What benefit is this change supposed to gain?
Quoted 12 lines
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- t/t5701-git-serve.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)t/t5701-git-serve.sh +0
-0
t/t5701-git-serve.sh test_expect_success 'basics of object-info' ' · line 362
Quoted 9 lines
t/t5701-git-serve.sh +2
-2@@ -356,8 +356,8 @@ test_expect_success 'basics of object-info' ' cat >expect <<-EOF && size - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) + $(git rev-parse two:two.t) $(test_file_size two.t) + $(git rev-parse two:two.t) $(test_file_size two.t) 0000 EOF
It is not like we want to avoid piping wc -c into xargs and hide the exit status from "wc -c". We are already losing the exit status of "git rev-parse" anyway.
If the test after the change were like this
two_object=$(git rev-parse two:two.t) && two_size=$(test_file_size two.t) && cat >expect <<-EOF && size $two_object $two_size $two_object $two_size 0000 EOF
you can sell it as "we do not want to lose exit status of 'git rev-parse'", "we do not need to run the same command twice", etc. But it is unclear what we gain by rewriting the wc-piped-to-xargs to test_file_size.
Re: [PATCH GSoC v2 3/6] protocol-caps: add type support to object-info
In reply to Pablo SabaterQuoted 20 lines · Pablo Sabater
Teach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While touching
send_info(), wrap an over-long line and fix the bit field style of requested_info.size.Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 27 +++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 3 deletions(-)protocol-caps.c +0
-0
protocol-caps.c line 16
Quoted 7 lines
protocol-caps.c +2
-1@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };
OK. This matches this bit in our .clang-format file:
# Add no space around the bit field
# unsigned bf:2;
BitFieldColonSpacing: None
Quoted 23 lines
protocol-caps.c +6
-1@@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string; +enum object_type object_type;struct object_id oid;size_t object_size;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) {packet_writer_error ( writer, -"object-info: protocol error, expected to get oid, not '%s'" , +"object-info: protocol error, expected to get " +"oid, not '%s'" , oid_str);continue ; }
protocol-caps.c static void send_info(struct repository *r, struct packet_writer *writer, · line 106
Quoted 9 lines
protocol-caps.c +2
-1@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }
We were already learning the object type as part of the existence check anyway, so we will ...
protocol-caps.c static void send_info(struct repository *r, struct packet_writer *writer, · line 115
Quoted 7 lines
protocol-caps.c +3
-0@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); +
... add it to the payload.
protocol-caps.c static void send_info(struct repository *r, struct packet_writer *writer, · line 118
Quoted 3 lines
protocol-caps.c +0
-0write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);
ANd then the payload is sent in one go.
Quoted 23 lines
protocol-caps.c +5
-0@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;t/t5701-git-serve.sh +3
-0@@ -366,6 +366,33 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'object-info supports type' ' + test_config transfer.advertiseObjectInfo true && +
t/t5701-git-serve.sh test_expect_success 'basics of object-info' ' · line 381
Quoted 10 lines
t/t5701-git-serve.sh +10
-0+ test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type + oid $(git rev-parse two:two.t) + oid $(git rev-parse two:two.t) + 0000 + EOF
This is not something we can change in the middle of this topic, but the input format looks rather curious. We tell the other side that we are going to ask about size and type but on two separate lines, and then throw each object one by one.
t/t5701-git-serve.sh test_expect_success 'basics of object-info' ' · line 389
Quoted 7 lines
t/t5701-git-serve.sh +7
-0+ cat >expect <<-EOF && + size + type + $(git rev-parse two:two.t) $(test_file_size two.t) blob + $(git rev-parse two:two.t) $(test_file_size two.t) blob + 0000 + EOF
And the output format is even more curious. Again, we say size and type on two separate lines, but (object name, size, type) come on a single line. I would probably have designed the "these are the fields" declaration at the beginning to also be on a single line, in both directions. It is not like we are afraid that a line would grow too long. If we were worried that placing these "size" and "type" labels on the same line would make the line too long, we would certainly be showing object name, size, and type on separate lines.
Anyway, the code change looks like what anybody would expect to see in a change to add support of "type" to a codebase that supports "size". As an incremental change, I didn't see anything wrong in it, even though the basic protocol design smelled a bit strange.
Thanks.
Quoted 7 lines
t/t5701-git-serve.sh +5
-0+ test-tool serve-v2 --stateless-rpc <in >out && + test-tool pkt-line unpack <out >actual && + test_cmp expect actual +' + test_expect_success 'bare OID request' ' test_config transfer.advertiseObjectInfo true &&
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
In reply to Pablo Sabaterfetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 112
Quoted 9 lines · Pablo Sabater
fetch-object-info.c +5
-0@@ -104,8 +105,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arfor (size_t j =0 ; j < args->oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); + }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep));
Do object_info_data[j].typep and object_info_data[k].typep need to be independently freeable? Separate allocations by calling calloc args->oids->nr times would allow that, but if there is no such need, nr contiguous allocation of them,
enum object_type *types; *types = xcalloc(args->oids->nr, sizeof(*types)); for (size_t j = 0; j < args->oids->nr; j++) object_info_data[j].typep = &types[j];
would be simpler to manage and easier to get rid of once you are done.
Quoted 17 lines
fetch-object-info.c +5
-1}else { -BUG ("only size is supported" ); +BUG ("unexpected object-info option: %s" , reader->line); } }@@ -151,6 +157,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +string_list_clear (&object_info_values,0 ); }check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
Re: [PATCH GSoC v2 5/6] serve: advertise type capability
In reply to Pablo Sabater[snip]
t/t1017-cat-file-remote-object-info.sh test_expect_success 'remote-object-info does not die on missing oid like info' ' · line 268
Quoted 10 lines
t/t1017-cat-file-remote-object-info.sh +2
-4-# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo $hello_type >expect &&
Shouldn't this be echo "$hello_type" instead?
Quoted 30 lines
git cat-file --batch-command="%(objecttype)" >actual <<
-EOF&& remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOFt/t1017-cat-file-remote-object-info.sh +16
-1@@ -271,6 +273,22 @@ test_expect_success 'unsupported placeholder on remote returns empty string' ' ) ' +test_expect_success 'unsupported placeholders on remote return empty string' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + fmt="%(objectmode) %(objectsize:disk) %(rest) %(deltabase)" && + + # The hardcoded SPs between the atoms are respected. + echo " " >expect && + git cat-file --batch-command="$fmt" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid + EOF + test_cmp expect actual + ) +' + test_expect_success 'requesting only objectname echoes back' ' ( set_transport_variables "$daemon_parent" && --2.54.0
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
In reply to Junio C HamanoQuoted 10 lines · Junio C Hamano
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 112
Quoted 5 lines
fetch-object-info.c +5
-0+ }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep));Do object_info_data[j].typep and object_info_data[k].typep need to be independently freeable? Separate allocations by calling calloc
args->oids->nrtimes would allow that, but if there is no such need, nr contiguous allocation of them,
Stepping back a bit, the design of "odb.h:struct object_info" look rather curious.
Why does the struct store scalar values like "enum object_type" and "size_t" as a pointer to elsewhere, and does not store the values right there in the structure itself? By forcing the caller to allocate an "enum object_type" for each of these object_info[] elements, the design requires 8-byte for a pointer to the heap and malloc overhead, probably ~16 bytes or more, in addition to store a single "enum object_type" that can be stored in a single byte.
We are probably using this pointer indirection to say "ah, typep is NULL so the caller did not ask for this information and the object layer does not have to provide one", plus "typep is NULL so the engine did not give this information for the object". But we can do so with two bitfields "unsigned typep_asked:1, typep_valid:1;" instead of paying ~24-byte or more heap allocation overhead.
Again, this is not something we can change in the middle of this topic, but since I noticed it and found iffy, I'll leave a note here to stir the pot anyway.
Stepping back a bit, the design of odb.h:struct object_info looks rather curious.
Why does the struct store scalar values like enum object_type and size_t as pointers to elsewhere, rather than storing the values right there in the structure itself? By forcing the caller to allocate an enum object_type for each of these object_info[] elements, the design requires an 8-byte pointer to the heap and malloc overhead, probably ~16 bytes or more, to store a single enum object_type that could fit in a single byte.
We are probably using this pointer indirection to say "ah, '.typep' is NULL so the caller did not ask for this information and the object layer does not have to provide it", plus "'.typep' is NULL so the engine did not give this information for the object". But we can do so with two bitfields
unsigned type_asked:1,
type_valid:1;instead of paying ~24 bytes or more of heap allocation overhead.
Again, this is not something we can change in the middle of this topic, but since I noticed it and found it iffy, I'll leave a note here to stir the pot anyway. It could be something we may want to clean-up much later after all the dust settles from this year's GSoC. I dunno.
Re: [PATCH GSoC v2 2/6] t5701: use the test_file_size() helper
In reply to Junio C HamanoQuoted 8 lines · Junio C Hamano
Quoted 2 lines · Pablo Sabater
An object-info test uses 'wc -c <two.t | xargs' to get the file size. Update it to use the
test_file_size()helper instead.What is missing from this description is what is wrong with the use of that "wc -c | xargs" construct. What benefit is this change supposed to gain?
True, the patch was small and I did not write the commit message properly.
The xargs is there to strip the leading blanks that wc adds on some OS (I tested this on macOS).test_file_size() reports the size without any padding, so nothing needs to be stripped.
I'll write the log correctly next reroll.
Quoted 38 lines
Quoted 13 lines
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- t/t5701-git-serve.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)t/t5701-git-serve.sh +0
-0t/t5701-git-serve.sh test_expect_success 'basics of object-info' ' · line 362
Quoted 9 lines
t/t5701-git-serve.sh +2
-2@@ -356,8 +356,8 @@ test_expect_success 'basics of object-info' ' cat >expect <<-EOF && size - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) + $(git rev-parse two:two.t) $(test_file_size two.t) + $(git rev-parse two:two.t) $(test_file_size two.t) 0000 EOFIt is not like we want to avoid piping wc -c into xargs and hide the exit status from "wc -c". We are already losing the exit status of "git rev-parse" anyway.
If the test after the change were like this
two_object=$(git rev-parse two:two.t) && two_size=$(test_file_size two.t) && cat >expect <<-EOF && size $two_object $two_size $two_object $two_size 0000 EOF
Quoted 4 lines
you can sell it as "we do not want to lose exit status of 'git rev-parse'", "we do not need to run the same command twice", etc. But it is unclear what we gain by rewriting the wc-piped-to-xargs to
test_file_size.
I will do that, I'll move the object name and size to variables so they are not recomputed.
Thanks for the review,
Pablo
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
In reply to Junio C HamanoQuoted 17 lines · Junio C Hamano
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 112
Quoted 9 lines · Pablo Sabater
fetch-object-info.c +5
-0@@ -104,8 +105,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arfor (size_t j =0 ; j < args->oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); + }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep));Do object_info_data[j].typep and object_info_data[k].typep need to be independently freeable? Separate allocations by calling calloc
args->oids->nrtimes would allow that, but if there is no such need, nr contiguous allocation of them,
Quoted 8 lines
enum object_type *types; *types = xcalloc(args->oids->nr, sizeof(*types)); for (size_t j = 0; j < args->oids->nr; j++) object_info_data[j].typep = &types[j];would be simpler to manage and easier to get rid of once you are done.
Hmmmm, they don't need to be independently freeable but they are freed by free_object_info_contents() called at the end of parse_cmd_remote_object_info() at 'builtin/cat-file.c' in a loop:
for (size_t i = 0; i < object_info_oids.nr; i++) free_object_info_contents(&remote_object_info[i]);
free_object_info_contents() is:
void free_object_info_contents(struct object_info *object_info)
{
if (!object_info)
return;
free(object_info->typep);
free(object_info->sizep);
free(object_info->disk_sizep);
free(object_info->delta_base_oid);
}This function was implemented by the series that introduced remote-object-info (the one that this series is based on) so parse_cmd_remote_object_info() is the only caller.
Thinking about it, your suggestion can be done easily. To free types it is enough to do free(remote_object_info[0].typep); (same for sizep).
I'll make it work as a prep patch for size and modify this one to do the same.free_object_info_contents() gets dropped in the prep patch because it would have no callers after it.
Quoted 11 lines
Quoted 11 lines
fetch-object-info.c +1
-1}else { -BUG ("only size is supported" ); +BUG ("unexpected object-info option: %s" , reader->line); } }@@ -151,6 +157,10 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string);
Quoted 7 lines
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 166
Quoted 7 lines
fetch-object-info.c +4
-0+if (type_index >=0 ) + *object_info_data[i].typep = +type_from_string (object_info_values.items[type_index +1 ].string); +string_list_clear (&object_info_values,0 ); }check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
Thanks for the review,
Pablo
Re: [PATCH GSoC v2 5/6] serve: advertise type capability
In reply to Chandra PratapQuoted 9 lines · Chandra Pratap
[snip]
Quoted 8 lines
t/t1017-cat-file-remote-object-info.sh +1
-3-# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" &&
Quoted 4 lines
t/t1017-cat-file-remote-object-info.sh test_expect_success 'remote-object-info does not die on missing oid like info' ' · line 268
Quoted 2 lines
t/t1017-cat-file-remote-object-info.sh +1
-1- echo "" >expect && + echo $hello_type >expect &&Shouldn't this be echo "$hello_type" instead?
Because hello_type is "blob" (no SPs) it works but you are right, it should have quotes.
Will fix it.
Quoted 28 lines
Quoted 28 lines
git cat-file --batch-command="%(objecttype)" >actual <<
-EOF&& remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOFt/t1017-cat-file-remote-object-info.sh +16
-0@@ -271,6 +273,22 @@ test_expect_success 'unsupported placeholder on remote returns empty string' ' ) ' +test_expect_success 'unsupported placeholders on remote return empty string' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + fmt="%(objectmode) %(objectsize:disk) %(rest) %(deltabase)" && + + # The hardcoded SPs between the atoms are respected. + echo " " >expect && + git cat-file --batch-command="$fmt" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid + EOF + test_cmp expect actual + ) +' + test_expect_success 'requesting only objectname echoes back' ' ( set_transport_variables "$daemon_parent" &&
Quoted 3 lines
Quoted 3 lines
--
2.54.0
Thanks,
Pablo
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
In reply to Junio C HamanoQuoted 55 lines · Junio C Hamano
Quoted 10 lines · Junio C Hamano
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 112
Quoted 5 lines
fetch-object-info.c +5
-0+ }else if (!strcmp (reader->line,"type" )) { + type_index = (int )i; +for (size_t j =0 ; j < args->oids->nr; j++) + object_info_data[j].typep = +xcalloc (1 ,sizeof (*object_info_data[j].typep));Do object_info_data[j].typep and object_info_data[k].typep need to be independently freeable? Separate allocations by calling calloc
args->oids->nrtimes would allow that, but if there is no such need, nr contiguous allocation of them,Stepping back a bit, the design of "odb.h:struct
object_info" look rather curious.Why does the struct store scalar values like "enum
object_type" and "size_t" as a pointer to elsewhere, and does not store the values right there in the structure itself? By forcing the caller to allocate an "enumobject_type" for each of these object_info[] elements, the design requires 8-byte for a pointer to the heap and malloc overhead, probably ~16 bytes or more, in addition to store a single "enumobject_type" that can be stored in a single byte.We are probably using this pointer indirection to say "ah, typep is NULL so the caller did not ask for this information and the object layer does not have to provide one", plus "typep is NULL so the engine did not give this information for the object". But we can do so with two bitfields "unsigned typep_asked:1, typep_valid:1;" instead of paying ~24-byte or more heap allocation overhead.
Again, this is not something we can change in the middle of this topic, but since I noticed it and found iffy, I'll leave a note here to stir the pot anyway.
Stepping back a bit, the design of odb.h:struct
object_infolooks rather curious.Why does the struct store scalar values like enum
object_typeandsize_tas pointers to elsewhere, rather than storing the values right there in the structure itself? By forcing the caller to allocate an enumobject_typefor each of these object_info[] elements, the design requires an 8-byte pointer to the heap and malloc overhead, probably ~16 bytes or more, to store a single enumobject_typethat could fit in a single byte.We are probably using this pointer indirection to say "ah, '.typep' is NULL so the caller did not ask for this information and the object layer does not have to provide it", plus "'.typep' is NULL so the engine did not give this information for the object". But we can do so with two bitfields
unsigned type_asked:1, type_valid:1;
Quoted 7 lines
instead of paying ~24 bytes or more of heap allocation overhead.
Again, this is not something we can change in the middle of this topic, but since I noticed it and found it iffy, I'll leave a note here to stir the pot anyway. It could be something we may want to clean-up much later after all the dust settles from this year's GSoC. I dunno.
[CC'ing peff]
Hi!
I haven't stopped to think about that but it does look strange.
This is related to what had to be done to fix a bug at "contents" commands a few days ago [1].
In that patch it had to save the previous state of typep and then restore it, because other commands like "info" and this series one "remote-object-info" use this pointer for the "is this asked?" question.
If we take a look at expand_atom():
...
} else if (is_atom("objecttype", atom, len)) {
if (data->mark_query) {
data->info.typep = &data->type;
} else {
const char *t = type_name(data->type);
strbuf_addstr(sb, t ? t : "");
}
...expand_atom() has two responsibilities, it is called at the start to map which atoms are asked (when data->mark_query), and a second to expand those atoms.
For example, typep being non-NULL does this effect on these commands:
info: makes a type lookup, and fills type.
remote-object-info: typep is directly used to know whether a client has
asked for %(objecttype).
For both commands what we pay is extra work because at the end the data shown is the one expanded from the format.
It's out of scope for this series but I wanted to add what I know.
[1]: https://lore.kernel.org/git/[email protected]/
Thanks,
Pablo
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
Quoted 10 lines · Pablo Sabater
Quoted 10 lines
We are probably using this pointer indirection to say "ah, '.typep' is NULL so the caller did not ask for this information and the object layer does not have to provide it", plus "'.typep' is NULL so the engine did not give this information for the object". But we can do so with two bitfields
unsigned type_asked:1, type_valid:1;instead of paying ~24 bytes or more of heap allocation overhead.
Yes, this conditional loading is exactly how we use the pointers. I agree that a bool would be smaller, though I doubt it really matters in practice. You shouldn't have a large number of object_info structs. You should have one that you use over and over. And it does not point to a heap allocation, but usually to a stack variable in the caller.
I don't think you'd need type_valid (at least not as the object_info code is written now). If you ask for it, then either the query is satisfied, or we return an error.
I think the pointer system goes all the way back to 9a49059022 (sha1_object_info_extended(): expose a bit more info, 2011-05-12). It is mostly just mirroring the pointers that would be passed directly to the function (but marshalling them in a struct so callers don't have to pass a zillion NULLs). So:
read_object_info(oid, &size);
became:
struct object_info query; query.sizep = &size; read_object_info(oid, &query);
One minor benefit the pointer system gets you is that the compiler can more easily tell what has been loaded. Imagine that we had a type_asked bool, but you forgot to set it. Now you look at oi.type, and it's garbage (or maybe some sentinel value). But the compiler has no clue without looking at the innards of the read_object_info() function.
Whereas with the pointers, you do this:
enum object_type type; struct object_info oi = OBJECT_INFO_INIT;
oi.typep = &type; /* what if we forget this? */
read_object_info(oid, &oi); do_something(type);
If you forget the pointer assignment, the compiler will realize that "type" never got passed to anybody and complain.
I don't know how valuable that is in practice, though.
Anyway, that is the history.
Quoted 6 lines
This is related to what had to be done to fix a bug at "contents" commands a few days ago [1].
In that patch it had to save the previous state of typep and then restore it, because other commands like "info" and this series one "remote-object-info" use this pointer for the "is this asked?" question.
Yes, though you'd have the same thing with bools. You'd have to save type_asked, set it, and then restore it.
Quoted 12 lines
If we take a look at
expand_atom():... } else if (is_atom("objecttype", atom, len)) { if (data->mark_query) { data->info.typep = &data->type; } else { const char *t = type_name(data->type); strbuf_addstr(sb, t ? t : ""); } ...
Quoted 3 lines
expand_atom()has two responsibilities, it is called at the start to map which atoms are asked (whendata->mark_query), and a second to expand those atoms.
Yep. But again, you'd have to set the bools somewhere. And it would be here (in the mark_query half).
Quoted 9 lines
For example, typep being non-NULL does this effect on these commands:
info: makes a type lookup, and fills type.
remote-object-info: typep is directly used to know whether a client has
asked for %(objecttype).For both commands what we pay is extra work because at the end the data shown is the one expanded from the format.
There should be no extra work. We do a single read_object_info() that grabs all of the data and writes it into expand_data. If we are getting data from elsewhere (say, a remote server) then we should not be using object_info at all! The concrete data goes into expand_data, which is a data structure specific to cat-file expansion.
-Peff
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
Quoted 16 lines · Jeff King
Yes, this conditional loading is exactly how we use the pointers. I agree that a bool would be smaller, though I doubt it really matters in practice. You shouldn't have a large number of
object_infostructs. You should have one that you use over and over. And it does not point to a heap allocation, but usually to a stack variable in the caller.I don't think you'd need
type_valid(at least not as theobject_infocode is written now). If you ask for it, then either the query is satisfied, or we return an error.I think the pointer system goes all the way back to
9a49059022(sha1_object_info_extended(): expose a bit more info, 2011-05-12). It is mostly just mirroring the pointers that would be passed directly to the function (but marshalling them in a struct so callers don't have to pass a zillion NULLs). So:
Quoted 7 lines
read_object_info(oid, &size);became:
struct object_info query; query.sizep = &size; read_object_info(oid, &query);
OK, I tried to read through the patches here, not having looked at the topic previously. I think it doesn't make sent to use object_info here. It is about collecting the options to make the query for _one_ object, so it expects you to point to where it should write the results.
But what you want is to query N objects and get all of the results back. You _could_ do that with N object_info queries, one per object, like this:
enum object_type types;
struct object_info queries;
ALLOC_ARRAY(types, nr); CALLOC_ARRAY(queries, nr);
for (size_t i = 0; i < nr; i++)
queries[i].typep = &types[i];
/* you can imagine this is calling read_object_info() in a loop
* under the hood */
do_many_queries(&oids, nr, &queries);
/* now we have our answers */
for (size_t i = 0; i < nr; i++)
do_something(oids[i], types[i]);
But it's kind of silly. Every query is the same, and you'd rather just pass _one_ query struct that says what you're interested in. But since you are not calling read_object_info() yourself here, why use its query struct? You can make your own using boolean flags or whatever.
And I guess that's what started this conversation. The fundamental difference is asking about one object (and using pointers to tell where to put the answer) versus asking about N.
-Peff
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
Quoted 3 lines · Jeff King
And I guess that's what started this conversation. The fundamental difference is asking about one object (and using pointers to tell where to put the answer) versus asking about N.
Thanks for framing the trouble I had so cleanly. Yes.
The origin of the pointer pattern you mentioned, 9a49059022 (sha1_object_info_extended(): expose a bit more info, 2011-05-12), designed the object_info structure to be passed as a set of extended parameters to sha1_object_info_extended().
Instead of passing 'size_t *size_p' (which can be NULL) as a parameter to signal that (1) if NULL we are not interested in the value, and (2) if not NULL, that is where you are expected to write the answer, and having to keep adding such a pointer parameter every time we need to optionally ask the function for a different aspect of the object, it defined the function to take an object_info structure to allow us to add new members to the struct as the set of queries grows without having to change the function signature.
As a set of extended parameters, of course, it was natural for the caller's variables that receive the answers to be pointed to by members in the struct. So the pointers in the struct are justifiable, but strictly as parameters to the function.
The troubling thing I saw in the patch (and I suspect it was not a problem introduced in this series, but by earlier changes that added other kinds of fields) is exactly as you identified.
The pointers in this struct were meant to point at real variables or structure members that receive values from the function, and were never meant to be the final structure that receives and retains returned values. If we need 5 calls to the function, we either:
(1) Have a single object_info, and a set of local variables that
are pointed at by the members of the object_info structure, and
have a loop that runs 5 times where each iteration calls the
function to store the returned values in local variables and
consumes them, i.e.
struct oid oid[5];
struct object_info oi;
for (int i = 0; i < 5; i++) {
size_t size;
enum object_type type;
oi.size_p = &size;
oi.type_p = &type;
object_info_extended(oid[i], &oi);
... use 'size' and 'type' here ...
}
if you can consume and forget about the object in each
iteration, or (2) Have a single object_info, and 5 sets of local variables. A
loop runs 5 times; in the nth iteration of the loop,
object_info points at the nth set of local variables and the
function is called. After the loop runs, we have 5 sets of
local variables populated and we use them, i.e.
struct oid oid[5];
struct { size_t size; enum object_type type; } trait[5];
struct object_info oi;
for (int i = 0; i < 5; i++) {
oi.size_p = &trait[i].size;
oi.type_p = &trait[i].type;
object_info_extended(oid[i], &oi);
}
... now you have 'size' and 'type' for all these 5 objects ...
if you have to return all 5 results to your caller.In either case, you do not need more than one object_info structure. Having an array of object_info structures was what looked so weird to me.
Thanks.
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
Quoted 74 lines · Junio C Hamano
Quoted 3 lines · Jeff King
And I guess that's what started this conversation. The fundamental difference is asking about one object (and using pointers to tell where to put the answer) versus asking about N.
Thanks for framing the trouble I had so cleanly. Yes.
The origin of the pointer pattern you mentioned,
9a49059022(sha1_object_info_extended(): expose a bit more info, 2011-05-12), designed theobject_infostructure to be passed as a set of extended parameters tosha1_object_info_extended().Instead of passing '
size_t*size_p' (which can be NULL) as a parameter to signal that (1) if NULL we are not interested in the value, and (2) if not NULL, that is where you are expected to write the answer, and having to keep adding such a pointer parameter every time we need to optionally ask the function for a different aspect of the object, it defined the function to take anobject_infostructure to allow us to add new members to the struct as the set of queries grows without having to change the function signature.As a set of extended parameters, of course, it was natural for the caller's variables that receive the answers to be pointed to by members in the struct. So the pointers in the struct are justifiable, but strictly as parameters to the function.
The troubling thing I saw in the patch (and I suspect it was not a problem introduced in this series, but by earlier changes that added other kinds of fields) is exactly as you identified.
The pointers in this struct were meant to point at real variables or structure members that receive values from the function, and were never meant to be the final structure that receives and retains returned values. If we need 5 calls to the function, we either:
(1) Have a single
object_info, and a set of local variables thatare pointed at by the members of the object_info structure, and have a loop that runs 5 times where each iteration calls the function to store the returned values in local variables and consumes them, i.e. struct oid oid[5]; struct object_info oi; for (int i = 0; i < 5; i++) { size_t size; enum object_type type; oi.size_p = &size; oi.type_p = &type; object_info_extended(oid[i], &oi); ... use 'size' and 'type' here ... } if you can consume and forget about the object in each iteration, or(2) Have a single
object_info, and 5 sets of local variables. Aloop runs 5 times; in the nth iteration of the loop, object_info points at the nth set of local variables and the function is called. After the loop runs, we have 5 sets of local variables populated and we use them, i.e. struct oid oid[5]; struct { size_t size; enum object_type type; } trait[5]; struct object_info oi; for (int i = 0; i < 5; i++) { oi.size_p = &trait[i].size; oi.type_p = &trait[i].type; object_info_extended(oid[i], &oi); } ... now you have 'size' and 'type' for all these 5 objects ... if you have to return all 5 results to your caller.
Quoted 6 lines
In either case, you do not need more than one
object_infostructure. Having an array ofobject_infostructures was what looked so weird to me.Thanks.
Thanks, I think I got it.
I have the doubt of whether this change is desired for this series as prep or if I should keep on and later make a cleanup series as this doesn't make a change for a user.
What I understood is that fetch_object_info shouldn't use object_info to store the results, because it doesn't call read_object_info() like other commands like 'info' do. Then, it should use its own data structure to hold the results with flags like wants_size and wants_type. Something like:
struct object_info_results {
enum object_type *types;
size_t *sizes;
unsigned *unrecognized;
size_t nr;
unsigned wants_size:1;
unsigned wants_type:1;
};All three of the pointers are nr long.
This could be done in two patches, as I was going to do a prep to prepare the current code (size only) and this patch would add type for fetch_object_info().
At the start I read this:
Quoted 3 lines
It could be something we may want to
clean-up much later after all the dust settles from this year's GSoC. I dunno.
So I'm a bit lost about what to do, I'm happy to make that in this series or as a cleanup series later after GSoC which ends in a couple weeks.
Whatever is preferred.
Thanks,
Pablo
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
Quoted 3 lines · Pablo Sabater
I have the doubt of whether this change is desired for this series as prep or if I should keep on and later make a cleanup series as this doesn't make a change for a user.
IMHO it is worth doing now. I've spent a bit of time looking through this code and I found it kind of confusing. In particular:
- there are a lot of semi-opaque structs, like object_info_args. It would seem simpler to me to pass those elements around independently to the functions that need them. Likewise, we seem to stuff a lot of data into the transport struct rather than passing it to the relevant functions, even though many of those elements are really just used for one function call, and aren't a property of the transport at all.
- it's hard to see who is ultimately responsible for deciding whether a field is requested. I _think_ it comes down to sticking those strings into the object_info_options struct (which I might have called "attrs" or "atoms" or something; they are syntactically "options" in the v2 protocol, but I think from the perspective of this code that is not what they are).
One thing you could do is have the caller pass in pointers for "types" and "sizes", and use their NULLness as an indication of what they want. That would be more like how object_info works, and then it really becomes a low-level protocol details to form those into the v2 protocol option strings.
But unlike local object_info, we sometimes find that the remote is not willing or able to provide a particular type. So we have to return a flag somehow for "I was / was not able to get types". You could do that with a bool in the response struct.
Or you could flip it on its head. Have the caller provide a bool saying "I want types", and then the low-level code is responsible for allocating the "types" array, and leaves it NULL if types were not available. That means the caller has to clean up the result, but they'd have had to clean up their local arrays anyway.
I _think_ that's what you're getting at with your example below.
- the protocol makes this unnecessarily complex. In particular, if we ask for "type size", the server is free to return them in an arbitrary order, or even to omit one of them (even if it told us it supported it!). So we have to map their returned ordering onto our arrays. Gross. I guess we are stuck with it, though, as the server side has been around for a while. And it does indeed choose its own ordering independent of what the client sent.
I also find the framing needlessly restrictive. Rather than one pkt-line per item, we get packets with space-separated values. What happens when a future item value has spaces in it?
As a side note, I think this is a good reason not to ship half of a protocol implementation. Without seeing both sides, you don't know what gotchas are lurking. But once one side ships, then it's hard to change the protocol later. This critique may all just me being cranky, though. ;)
Quoted 15 lines
What I understood is that
fetch_object_infoshouldn't useobject_infoto store the results, because it doesn't callread_object_info()like other commands like 'info' do. Then, it should use its own data structure to hold the results with flags likewants_sizeandwants_type. Something like:struct object_info_results { enum object_type *types; size_t *sizes; unsigned *unrecognized; size_t nr; unsigned wants_size:1; unsigned wants_type:1; };
Quoted 1 line
All three of the pointers are nr long.
Yeah. At first I thought your wants_size is redundant, but I guess if the goal is for the low-level code to allocate the "sizes" array, then we cannot use it as a signal (i.e., this is the "flip it on its head" direction I gave above).
I do think you want to be careful with how wants_size interacts with the "object_info_options" string list. IMHO it would make things easier if that string-ification happened deep down, probably in send_object_info_request(). And then the rest of the code can consistently use wants_size to see if we want sizes (and checking "sizes" for NULL to cover the case that the server did not support it).
Quoted 3 lines
This could be done in two patches, as I was going to do a prep to prepare the current code (size only) and this patch would add type for
fetch_object_info().
Yeah, that would make sense.
Quoted 7 lines
Quoted 3 lines
It could be something we may want to
clean-up much later after all the dust settles from this year's GSoC. I dunno.So I'm a bit lost about what to do, I'm happy to make that in this series or as a cleanup series later after GSoC which ends in a couple weeks.
I think the "after the dust settles" suggestion was for changing the interface of "struct object_info". That all becomes moot if we stop using it here entirely. So I think you should proceed along the lines of the object_info_results you showed above.
-Peff
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
In reply to Pablo SabaterQuoted 6 lines · Pablo Sabater
What I understood is that
fetch_object_infoshouldn't useobject_infoto store the results, because it doesn't callread_object_info()like other commands like 'info' do. Then, it should use its own data structure to hold the results with flags likewants_sizeandwants_type. Something like:
Quoted 8 lines
struct object_info_results {
enum object_type *types;
size_t *sizes;
unsigned *unrecognized;
size_t nr;
unsigned wants_size:1;
unsigned wants_type:1;
};I would have expected this to be an array of struct, i.e.
struct {
struct oid *oid;
enum object_type type;
size_t size;
} *result;
size_t result_nr, result_alloc;if you do not have the number of things you query upfront, or it may be an array of fixed size (i.e. no nr/alloc, just nr).
If you'll be making the same query for many different objects, you know if you are asking for type for all of them or for none of them, so depending on how the caller uses it, you may not need the valid bit. Or type==OBJ_NONE could signal "we have no info".
And you'd be using the second pattern I outlined, i.e.
for (size_t it = 0; it < result_nr; it++) {
/*
* you may selectively populate the oi to signal
* you do not need some values, but you get the
* idea.
*/
struct object_info oi = {
type_p = &result[it].type,
size_p = &result[it].size,
...
};
... ask about result[it].oid using &oi ...
}to populate the result[] array with values, I would imagine.
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
In reply to Jeff KingQuoted 4 lines · Jeff King
I think the "after the dust settles" suggestion was for changing the interface of "struct
object_info". That all becomes moot if we stop using it here entirely. So I think you should proceed along the lines of theobject_info_resultsyou showed above.
OK, if we are not using object_info structure at all, then I do not think it matters all that much if it is a struct of arrays or an array of structs (even though I suspect the latter would be cleaner).
But I'd rather see it named differently, leaving no room to be confused with the existing object_info structure.
Thanks.
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
In reply to Junio C HamanoQuoted 26 lines · Junio C Hamano
Quoted 6 lines · Pablo Sabater
What I understood is that
fetch_object_infoshouldn't useobject_infoto store the results, because it doesn't callread_object_info()like other commands like 'info' do. Then, it should use its own data structure to hold the results with flags likewants_sizeandwants_type. Something like:Quoted 8 lines
struct object_info_results { enum object_type *types; size_t *sizes; unsigned *unrecognized; size_t nr; unsigned wants_size:1; unsigned wants_type:1; };I would have expected this to be an array of struct, i.e.
struct { struct oid *oid; enum object_type type; size_t size; } *result; size_t result_nr, result_alloc;
Quoted 2 lines
if you do not have the number of things you query upfront, or it may be an array of fixed size (i.e. no nr/alloc, just nr).
I think that could work, but two gotchas:
- an array-of-struct allocates each item for every object. So if we are only asking about type, we have to allocate nr * size_t space to hold "size" fields nobody cares about.
This is true of object_info, too, but there we don't care about memory cost because we're only using one at a time. Whereas here the intent is to hold many results at once.
- you do need to signal somewhere whether "type" is valid (i.e., whether the remote side supported it). You can put that flag into the result struct, but it is a little wasteful. It is really a property of the whole query, not of each individual object. So you'd have to carry extra flags around (one per type). Whereas NULL-ness of the array can signal that same information.
Quoted 4 lines
If you'll be making the same query for many different objects, you know if you are asking for type for all of them or for none of them, so depending on how the caller uses it, you may not need the valid bit. Or type==OBJ_NONE could signal "we have no info".
Yeah, we sometimes use OBJ_NONE or OBJ_BAD as a sentinel value for type. But if we're not asking for a type field at all, I think that gets awkward.
So for unknown objects, I think a separate bit is less awkward.
For signaling "the server refused to tell us this item" we could use sentinel types like OBJ_NONE. But I don't think that extends to other fields (e.g., there is no useful sentinel value for "size").
Quoted 16 lines
And you'd be using the second pattern I outlined, i.e.
for (size_t it = 0; it < result_nr; it++) { /* * you may selectively populate the oi to signal * you do not need some values, but you get the * idea. */ struct object_info oi = { type_p = &result[it].type, size_p = &result[it].size, ... }; ... ask about result[it].oid using &oi ... }
Quoted 1 line
to populate the result[] array with values, I would imagine.
I think that is a perfectly reasonable direction for asking many responses from read_object_info(). But ultimately this is all getting shipped to the remote over the object-info protocol. So we never need an object_info at all, and even if we used one, we really would need N of them, because we're going to fill N requests at once (to reduce server round-trips).
-Peff
Re: [PATCH GSoC v2 4/6] fetch-object-info: parse type from server response
Quoted 1 line · Jeff King
So for unknown objects, I think a separate bit is less awkward.
OK.
Quoted 3 lines
For signaling "the server refused to tell us this item" we could use sentinel types like
OBJ_NONE. But I don't think that extends to other fields (e.g., there is no useful sentinel value for "size").
True, unless we abuse things like ((size_t)-1), which I think I saw somewhere in recently posted patches.
v3 cover cat-file: extend remote-object-info to support %(objecttype)
In reply to Pablo SabaterNeither the client nor the server knows how to handle type. This series continues the work for git cat-file --batch-command extending remote-object-info to support "%(objecttype)" end to end. It is based on its predecessor series "cat-file: add remote-object-info to batch-command" [1].
Type is the last property that is identical on the server and on the client once fetched.
Whether to support more metadata such as:
- objectsize:disk
- deltabase
- objectmode (needs context)
should be discussed, unlike size and type, the rest depend on how things are packed on the server and on what the client already has, so they cannot reliably match local once fetched. IMO they are not worth supporting, I can't find a use case for them.
To add a new placeholder, follow the steps in this series and add its wants_* flag to struct fetch_object_info_results.
Patches 1-4 are preparatory. They don't change what the command does:
- 1/8 is a test cleanup.
- 2/8 fixes a possible bug in case of a truncated response.
- 3/8 and 4/8 refactor how the object data is stored and handled. The why about this refactor comes from [2].
Patches 5-8 are the actual objecttype support:
- 5/8 teaches the server to answer type.
- 6/8 teaches the client to parse it.
- 7/8 advertises the capability so the client can start asking it.
- 8/8 unifies the default format.
Note that wants_type is added in 4/8 while the types array only appears in 6/8. This is intentional, it shows that asking for an attribute the server does not advertise just outputs an empty string, as it would for any other placeholder.
Based-on: <[email protected]>
Github CI: https://github.com/pabloosabaterr/git/actions/runs/30821717187
[1]: https://lore.kernel.org/git/[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/
---
Changes in v3:
- 1/8 (was 2/6): reword the commit message to explain that "wc -c | xargs"
only strips the padding some platforms add.
- New 2/8: die if the server sends fewer object-info lines than
requested OIDs. Preexisting bug.
- New 3/8: drop struct object_info_args and pass its members directly to fetch_object_info().
- New 4/8: replace struct object_info with struct
fetch_object_info_results to hold the results. This also drops the object_info_options string list, remote_atom_map[] and free_object_info_contents().
- Dropped 1/6, which replaced the hardcoded "size" in
send_object_info_request() with a loop over object_info_options, as
that list no longer reaches it.
- 6/8: follows the new result struct, so it only adds the type array.
- 7/8: quote $hello_type in the test.
- Link to v2: https://lore.kernel.org/git/[email protected]/
Changes in v2:
- Changed t5701 test to use test_file_size.
- Changed the for loop to use for_each_string_list_item().
- Moved "fetch-object-info: request all supported options dynamically"
patch to be the first one as it is prep patch.
- Typos and nits.
- Restored the test mixing info and remote-object-info, adapted to the
unified format.
- Link to v1: https://lore.kernel.org/git/[email protected]/
---
Pablo Sabater (8):
t5701: use test_file_size() to get the size of a file
fetch-object-info: detect truncated server responses
fetch-object-info: pass arguments directly instead of a struct
fetch-object-info: use dedicated struct for the results
protocol-caps: add type support to object-info
fetch-object-info: parse type from server response
serve: advertise type capability
cat-file: unify default format
Documentation/git-cat-file.adoc | 17 ++-- Documentation/gitprotocol-v2.adoc | 18 ++++- builtin/cat-file.c | 69 +++++----------- fetch-object-info.c | 141 +++++++++++++++++++++------------ fetch-object-info.h | 37 ++++++--- object-file.c | 10 --- odb.h | 3 - protocol-caps.c | 21 ++++- serve.c | 4 +- t/t1017-cat-file-remote-object-info.sh | 78 +++++++++++------- t/t5701-git-serve.sh | 41 +++++++++- transport.c | 12 ++- transport.h | 5 +- 13 files changed, 271 insertions(+), 185 deletions(-)
Range-diff versus v2:
1: 3fb0ea6af4 < -: ---------- fetch-object-info: request all supported options dynamically
2: 641911ed87 < -: ---------- t5701: use the test_file_size() helper
-: ---------- > 1: 84810a5500 t5701: use test_file_size() to get the size of a file
-: ---------- > 2: 16e2735bd5 fetch-object-info: detect truncated server responses
-: ---------- > 3: f42882ee03 fetch-object-info: pass arguments directly instead of a struct
-: ---------- > 4: 3053d33534 fetch-object-info: use dedicated struct for the results
3: a77e38e171 ! 5: 1331fab35f protocol-caps: add type support to object-info
@@ t/t5701-git-serve.sh: test_expect_success 'basics of object-info' '
+test_expect_success 'object-info supports type' '
+ test_config transfer.advertiseObjectInfo true &&
+
++ two_oid=$(git rev-parse two:two.t) &&
++ two_size=$(test_file_size two.t) &&
++
+ test-tool pkt-line pack >in <<-EOF &&
+ command=object-info
+ object-format=$(test_oid algo)
+ 0001
+ size
+ type
-+ oid $(git rev-parse two:two.t)
-+ oid $(git rev-parse two:two.t)
++ oid $two_oid
++ oid $two_oid
+ 0000
+ EOF
+
+ cat >expect <<-EOF &&
+ size
+ type
-+ $(git rev-parse two:two.t) $(test_file_size two.t) blob
-+ $(git rev-parse two:two.t) $(test_file_size two.t) blob
++ $two_oid $two_size blob
++ $two_oid $two_size blob
+ 0000
+ EOF
+
4: 8fb55d9b54 < -: ---------- fetch-object-info: parse type from server response
-: ---------- > 6: 3929b1a6fc fetch-object-info: parse type from server response
5: 70e65acd83 ! 7: 35e6b6a068 serve: advertise type capability
@@ t/t1017-cat-file-remote-object-info.sh: test_expect_success 'remote-object-info
cd "$daemon_parent/daemon_client_empty" &&
- echo "" >expect &&
-+ echo $hello_type >expect &&
++ echo "$hello_type" >expect &&
git cat-file --batch-command="%(objecttype)" >actual <<-EOF &&
remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid
EOF6: 4fe367eb58 ! 8: fe8163ebab cat-file: unify default format
@@ Documentation/gitprotocol-v2.adoc: object-info
Its main purpose is to allow a client to make decisions based on this
-information without having to fully fetch objects. Object size is the only -information that is currently supported.
-+information without having to fully fetch objects. Object size and type are the -+only information currently supported.
++information without having to fully fetch objects. Currently only object size ++and type are supported.
An object-info request takes the following arguments:
@@ Documentation/gitprotocol-v2.adoc: space.
+ obj-val = obj-size | obj-type
+
+ obj-info = obj-id [SP [obj-val *(SP obj-val)]]
++
++The values in obj-info appear in the same order as the corresponding attr ++lines, with exactly one value per requested attribute.
If the server does not recognize the OID, the response will be <oid> SP regardless of the number of attributes requested.
## builtin/cat-file.c ##
@@ builtin/cat-file.c: static void parse_cmd_remote_object_info(struct batch_options *opt,
- struct object_info *remote_object_info = NULL;
+ char *line_to_split;
+ struct fetch_object_info_results results = FETCH_OBJECT_INFO_RESULTS_INIT;
struct oid_array object_info_oids = OID_ARRAY_INIT;
- struct string_list object_info_options = STRING_LIST_INIT_NODUP;
- const char *saved_format = opt->format;
if (strlen(line) >= MAX_REMOTE_OBJ_INFO_LINE)
@@ builtin/cat-file.c: static void parse_cmd_remote_object_info(struct batch_option
data->skip_object_info = 0;
- opt->format = saved_format;
- for (size_t i = 0; i < object_info_oids.nr; i++)
- free_object_info_contents(&remote_object_info[i]);
+ free_fetch_object_info_results(&results);
+ free(line_to_split);
## t/t1017-cat-file-remote-object-info.sh ##
@@ t/t1017-cat-file-remote-object-info.sh: test_expect_success 'batch-command remote-object-info git:// default filter' '
---
base-commit: 71e19e8d2713f385c3fcef59cf6f29bcbd93d91f
change-id: 20260724-objecttype-support-ea1ef6941d07
PATCH v3 1/8 t5701: use test_file_size() to get the size of a file
The 'basics of object-info' test runs 'wc -c | xargs' twice to get the size of two.t. The pipe to xargs is only there to strip the blanks that some platforms pad the output of wc with.
Use the test_file_size() helper, which outputs the size directly, and store the result in a variable. Because 'git rev-parse two:two.t' is also run twice, store its output in a variable as well.
--- t/t5701-git-serve.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
t/t5701-git-serve.sh +7 -4
@@ -344,20 +344,23 @@ test_expect_success 'unexpected lines are not allowed in fetch request' ' test_expect_success 'basics of object-info' ' test_config transfer.advertiseObjectInfo true && + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + test-tool pkt-line pack >in <<-EOF && command=object-info object-format=$(test_oid algo) 0001 size - oid $(git rev-parse two:two.t) - oid $(git rev-parse two:two.t) + oid $two_oid + oid $two_oid 0000 EOF cat >expect <<-EOF && size - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) + $two_oid $two_size + $two_oid $two_size 0000 EOF
Signature · 1 line
2.54.0
PATCH v3 2/8 fetch-object-info: detect truncated server responses
In reply to Pablo SabaterThe loop reading the object-info response stops as soon as the reader returns something other than PACKET_READ_NORMAL. A server that somehow answers with fewer objects leaves the end of the result arrays empty.
The caller trusts that every requested object will be filled in.
die() if the loop doesn't reach the number of oids expected.
--- fetch-object-info.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
fetch-object-info.c +8 -2
@@ -49,6 +49,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar struct packet_reader *reader, struct object_info *object_info_data, const int stateless_rpc, const int fd_out) { + size_t i;int size_index =-1 ;switch (version) {
@@ -82,7 +83,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arBUG ("unknown protocol version" ); } -for (size_t i =0 ; i < args->object_info_options->nr; i++) { +for (i =0 ; i < args->object_info_options->nr; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
@@ -106,7 +107,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; +for (i =0 ;packet_reader_read (reader) ==PACKET_READ_NORMAL && i < args->oids->nr; i++) {
@@ -150,6 +151,11 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arstring_list_clear (&object_info_values,0 ); } + +if (i != args->oids->nr) +die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), + (uintmax_t )args->oids->nr, (uintmax_t )i); +check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return 0 ;
Signature · 1 line
2.54.0
PATCH v3 3/8 fetch-object-info: pass arguments directly instead of a struct
In reply to Pablo Sabaterstruct object_info_args groups three pointers that already live in the transport and are given to fetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests that fetch_object_info() uses all three of them.
Drop the struct and pass those parameters directly to fetch_object_info() and send_object_info_request(). This should have no change in behavior.
--- fetch-object-info.c | 53 +++++++++++++++++++++++++++++++---------------------- fetch-object-info.h | 17 ++++++++--------- transport.c | 11 +++++------ 3 files changed, 44 insertions(+), 37 deletions(-)
fetch-object-info.c +31 -22
@@ -9,20 +9,24 @@#include "string-list.h" /* Sends object-info command and its arguments into the request buffer. */ -static void send_object_info_request (const int fd_out,struct object_info_args *args) +static void send_object_info_request (const int fd_out, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options) {struct strbuf req_buf =STRBUF_INIT ; -write_command_and_capabilities (&req_buf,"object-info" , args->server_options); +write_command_and_capabilities (&req_buf,"object-info" , server_options); -if (unsorted_string_list_has_string (args->object_info_options,"size" )) +if (unsorted_string_list_has_string (object_info_options,"size" ))packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) +else if (object_info_options->nr)BUG ("only size should be in object_info_options" ); -if (args->oids) -for (size_t i =0 ; i < args->oids->nr; i++) -packet_buf_write (&req_buf,"oid %s" ,oid_to_hex (&args->oids->oid[i])); +if (oids) +for (size_t i =0 ; i < oids->nr; i++) +packet_buf_write (&req_buf,"oid %s" , +oid_to_hex (&oids->oid[i]));packet_buf_flush (&req_buf);if (write_in_full (fd_out, req_buf.buf, req_buf.len) <0 )
@@ -45,8 +49,12 @@ static int parse_object_size(const char *s, size_t *res)return 0 ; } -int fetch_object_info (const enum protocol_version version,struct object_info_args *args, -struct packet_reader *reader,struct object_info *object_info_data, +int fetch_object_info (const enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options, +struct packet_reader *reader, +struct object_info *object_info_data,const int stateless_rpc,const int fd_out) {size_t i;
@@ -65,16 +73,17 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* because the number of options is a small known number (the * supported placeholders which currently are size and type). */ -for (int i = (int )args->object_info_options->nr -1 ; i >=0 ; i--) +for (int i = (int )object_info_options->nr -1 ; i >=0 ; i--)if (!server_supports_feature ("object-info" , - args->object_info_options->items[i].string,0 )) -unsorted_string_list_delete_item (args->object_info_options, i,0 ); + object_info_options->items[i].string,0 )) +unsorted_string_list_delete_item (object_info_options, i,0 );/* * Even if no options are left, we still send the oid so we get * at least an existence check. */ -send_object_info_request (fd_out, args); +send_object_info_request (fd_out, server_options, oids, + object_info_options);break ;case protocol_v1:case protocol_v0:
@@ -83,14 +92,14 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arBUG ("unknown protocol version" ); } -for (i =0 ; i < args->object_info_options->nr; i++) { +for (i =0 ; i < object_info_options->nr; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return -1 ; } -if (!unsorted_string_list_has_string (args->object_info_options, reader->line)) +if (!unsorted_string_list_has_string (object_info_options, reader->line))return -1 ;if (!strcmp (reader->line,"size" )) {
@@ -99,7 +108,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* is only size. No risk of overflow. */ size_index = (int )i; -for (size_t j =0 ; j < args->oids->nr; j++) +for (size_t j =0 ; j < oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); }else {
@@ -109,16 +118,16 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arfor (i =0 ;packet_reader_read (reader) ==PACKET_READ_NORMAL && - i < args->oids->nr; + i < oids->nr; i++) {struct string_list object_info_values =STRING_LIST_INIT_DUP ;string_list_split (&object_info_values, reader->line," " ,-1 );if (strcmp (object_info_values.items[0 ].string, -oid_to_hex (&args->oids->oid[i]))) +oid_to_hex (&oids->oid[i])))die (_ ("object-info: expected OID: %s, got %s" ), -oid_to_hex (&args->oids->oid[i]), +oid_to_hex (&oids->oid[i]), object_info_values.items[0 ].string);/*
@@ -138,7 +147,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* the server we expect the server to answer with the same * number of attributes requested. */ -if (args->object_info_options->nr +1 != object_info_values.nr) +if (object_info_options->nr +1 != object_info_values.nr)die ("object-info: unexpected number of attributes: %s" , reader->line);
@@ -152,9 +161,9 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arstring_list_clear (&object_info_values,0 ); } -if (i != args->oids->nr) +if (i != oids->nr)die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), - (uintmax_t )args->oids->nr, (uintmax_t )i); + (uintmax_t )oids->nr, (uintmax_t )i);check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
fetch-object-info.h +8 -9
@@ -4,22 +4,21 @@#include "pkt-line.h" #include "protocol.h" -struct object_info_args { -struct string_list *object_info_options; -const struct string_list *server_options; -struct oid_array *oids; -}; -struct object_info ; +struct oid_array ;/* * Sends git-cat-file object-info command into the request buf and read the * results from packets. * -* Modifies args->object_info_options, on return it contains only the supported +* Modifies object_info_options, on return it contains only the supported * options by the server. */ -int fetch_object_info (enum protocol_version version,struct object_info_args *args, -struct packet_reader *reader,struct object_info *object_info_data, +int fetch_object_info (enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options, +struct packet_reader *reader, +struct object_info *object_info_data,int stateless_rpc,int fd_out);#endif /* FETCH_OBJECT_INFO_H */
transport.c +5 -6
@@ -438,11 +438,6 @@ static int fetch_object_info_via_pack(struct transport *transport)int ret =0 ;struct git_transport_data *data = transport->data;struct packet_reader reader; -struct object_info_args args = {0 }; - - args.server_options = transport->server_options; - args.oids = transport->smart_options->object_info_oids; - args.object_info_options = transport->smart_options->object_info_options;connect_setup (transport,0 ); packet_reader_init(&reader, data->fd[0], NULL, 0,
@@ -453,7 +448,11 @@ static int fetch_object_info_via_pack(struct transport *transport) data->version =discover_version (&reader); transport->hash_algo = reader.hash_algo; - ret =fetch_object_info (data->version, &args, &reader, + ret =fetch_object_info (data->version, + transport->server_options, + transport->smart_options->object_info_oids, + transport->smart_options->object_info_options, + &reader, data->options.object_info_data, transport->stateless_rpc, data->fd[1 ]);
Signature · 1 line
2.54.0
PATCH v3 4/8 fetch-object-info: use dedicated struct for the results
In reply to Pablo Sabaterfetch_object_info() collects information about N objects, but it stores the results in an array of object_info. That struct holds the extended parameters of read_object_info() (The optional outputs the caller wants filled). Its pointers tell that function where to write the answers for a single object. object_info is not meant to be the final storage, and since fetch_object_info() does not call read_object_info(), there is no reason to use it. Using it means allocating one scalar per object per attribute just to have those pointers somewhere to point at.
Add struct fetch_object_info_results. The caller sets the wants_* flags to say what it is interested in, and fetch_object_info() allocates one array per attribute. A set wants_* flag means "asked for", while a non-NULL array means "available". The caller releases the arrays with free_fetch_object_info_results().
The object_info_options string list is no longer needed. Filtering against the server's advertisement now sets local ask_* flags, and send_object_info_request() turns those into the v2 protocol option strings. remote_atom_map[] existed only to map those strings back into atom names, so drop it and build remote_allowed_atoms from the result arrays.
free_object_info_contents() loses its only caller and is dropped.
--- builtin/cat-file.c | 59 +++++++++-------------------------- fetch-object-info.c | 90 ++++++++++++++++++++++++++++------------------------- fetch-object-info.h | 28 ++++++++++++----- object-file.c | 10 ------ odb.h | 3 -- transport.c | 3 +- transport.h | 5 +-- 7 files changed, 88 insertions(+), 110 deletions(-)
builtin/cat-file.c +15 -44
@@ -31,6 +31,7 @@#include "alias.h" #include "remote.h" #include "transport.h" +#include "fetch-object-info.h" /* * Maximum length for a remote URL. While no universal standard exists,
@@ -681,9 +682,8 @@ static void batch_one_object(const char *obj_name,static int get_remote_info (int argc,const char **argv, -struct object_info **remote_object_info, -struct oid_array *object_info_oids, -struct string_list *object_info_options) +struct fetch_object_info_results *results, +struct oid_array *object_info_oids) {int retval =0 ;struct remote *remote =NULL ;
@@ -724,11 +724,9 @@ static int get_remote_info(int argc, goto cleanup; } -CALLOC_ARRAY (*remote_object_info, object_info_oids->nr); gtransport->smart_options->object_info_oids = object_info_oids; - gtransport->smart_options->object_info_options = object_info_options; - gtransport->smart_options->object_info_data = *remote_object_info; + gtransport->smart_options->object_info_results = results; retval =transport_fetch_object_info (gtransport); cleanup:transport_disconnect (gtransport);
@@ -816,21 +814,6 @@ static void parse_cmd_mailmap(struct batch_options *opt UNUSED,load_mailmap (); } -struct protocol_placeholder_entry { -const char *option; -const char *atom; -}; - -static const struct protocol_placeholder_entry remote_atom_map[] = { - {"size" ,"objectsize" }, - {"type" ,"objecttype" }, -/* -* Add new protocol options here. Even if the server doesn't support -* them the allow_list will drop them if the server doesn't advertise -* them. -*/ -}; -static void parse_cmd_remote_object_info (struct batch_options *opt,const char *line,struct strbuf *output,struct expand_data *data)
@@ -838,9 +821,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,int count;const char **argv;char *line_to_split; -struct object_info *remote_object_info =NULL ; +struct fetch_object_info_results results =FETCH_OBJECT_INFO_RESULTS_INIT ;struct oid_array object_info_oids =OID_ARRAY_INIT ; -struct string_list object_info_options =STRING_LIST_INIT_NODUP ;const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )
@@ -861,26 +843,23 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, MAX_ALLOWED_OBJ_LIMIT);if (data->info.sizep) -string_list_append (&object_info_options,"size" ); + results.wants_size =1 ;if (data->info.typep) -string_list_append (&object_info_options,"type" ); + results.wants_type =1 ; -if (get_remote_info (count, argv, &remote_object_info, - &object_info_oids, &object_info_options)) +if (get_remote_info (count, argv, &results, &object_info_oids))die (_ ("failed to get object info from the remote: %s" ), argv[0 ]);string_list_clear (&data->remote_allowed_atoms,0 );string_list_append (&data->remote_allowed_atoms,"objectname" ); -for (size_t i =0 ; i <ARRAY_SIZE (remote_atom_map); i++) -if (unsorted_string_list_has_string (&object_info_options, remote_atom_map[i].option)) -string_list_append (&data->remote_allowed_atoms, - remote_atom_map[i].atom); +if (results.sizes) +string_list_append (&data->remote_allowed_atoms,"objectsize" ); data->skip_object_info =1 ; -for (size_t i =0 ; i < object_info_oids.nr; i++) { +for (size_t i =0 ; i < results.nr; i++) { data->oid = object_info_oids.oid[i]; -if (remote_object_info[i].unrecognized) { +if (results.unrecognized[i]) {report_object_status (opt,oid_to_hex (&data->oid), &data->oid,"missing" );continue ;
@@ -890,13 +869,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,* When reaching here, it means remote-object-info can retrieve * information from server without downloading them. */ -if (remote_object_info[i].sizep) { - data->size = *remote_object_info[i].sizep; - } - -if (remote_object_info[i].typep) { - data->type = *remote_object_info[i].typep; - } +if (results.sizes) + data->size = results.sizes[i]; opt->batch_mode =BATCH_MODE_INFO ; data->is_remote =1 ;
@@ -906,12 +880,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->skip_object_info =0 ; opt->format = saved_format; -for (size_t i =0 ; i < object_info_oids.nr; i++) -free_object_info_contents (&remote_object_info[i]); -string_list_clear (&object_info_options,0 ); +free_fetch_object_info_results (&results);free (line_to_split);free (argv); -free (remote_object_info);oid_array_clear (&object_info_oids); }
fetch-object-info.c +48 -42
155 lines not on this page · raw message
fetch-object-info.h +21 -7
@@ -4,21 +4,35 @@#include "pkt-line.h" #include "protocol.h" -struct object_info ; +struct fetch_object_info_results { +size_t *sizes; +uint8_t *unrecognized; +size_t nr; +unsigned wants_size:1 ; +unsigned wants_type:1 ; +}; + +#define FETCH_OBJECT_INFO_RESULTS_INIT { 0 } +struct oid_array ;/* -* Sends git-cat-file object-info command into the request buf and read the +* Sends git-cat-file object-info command into the request buf and reads the * results from packets. * -* Modifies object_info_options, on return it contains only the supported -* options by the server. +* The caller sets the wants_* flags in "results" to indicate which attributes +* it is interested in. On return, "results" holds one array per attribute that +* the server both advertised and answered with. An array left NULL means the +* attribute is not available. +* Release them with free_fetch_object_info_results(). */ int fetch_object_info (enum protocol_version version,const struct string_list *server_options,struct oid_array *oids, -struct string_list *object_info_options,struct packet_reader *reader, -struct object_info *object_info_data, -int stateless_rpc,int fd_out); +struct fetch_object_info_results *results, +int stateless_rpc, +int fd_out); + +void free_fetch_object_info_results (struct fetch_object_info_results *results);#endif /* FETCH_OBJECT_INFO_H */
object-file.c +0 -10
@@ -1740,13 +1740,3 @@ int odb_transaction_files_begin(struct odb_source *source,return 0 ; } - -void free_object_info_contents (struct object_info *object_info) -{ -if (!object_info) -return ; -free (object_info->typep); -free (object_info->sizep); -free (object_info->disk_sizep); -free (object_info->delta_base_oid); -}
odb.h +0 -3
@@ -635,7 +635,4 @@ void parse_alternates(const char *string,const char *relative_base, struct strvec *out); -/* Free pointers inside of object_info, but not object_info itself */ -void free_object_info_contents (struct object_info *object_info); -#endif /* ODB_H */
transport.c +1 -2
@@ -451,9 +451,8 @@ static int fetch_object_info_via_pack(struct transport *transport) ret =fetch_object_info (data->version, transport->server_options, transport->smart_options->object_info_oids, - transport->smart_options->object_info_options, &reader, - data->options.object_info_data, + data->options.object_info_results, transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);
transport.h +3 -2
@@ -7,6 +7,8 @@#include "string-list.h" #include "connect.h" +struct fetch_object_info_results ; +struct git_transport_options {unsigned thin :1 ;unsigned keep :1 ;
@@ -57,8 +59,7 @@ struct git_transport_options {struct oidset *acked_commits;struct oid_array *object_info_oids; -struct object_info *object_info_data; -struct string_list *object_info_options; +struct fetch_object_info_results *object_info_results; };enum transport_family {
Signature · 1 line
2.54.0
PATCH v3 5/8 protocol-caps: add type support to object-info
In reply to Pablo SabaterTeach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While touching send_info(), wrap an over-long line and fix the bit field style of requested_info.size.
--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 30 ++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 3 deletions(-)
protocol-caps.c +18 -3
@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/*
@@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string; +enum object_type object_type;struct object_id oid;size_t object_size;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) {packet_writer_error ( writer, -"object-info: protocol error, expected to get oid, not '%s'" , +"object-info: protocol error, expected to get " +"oid, not '%s'" , oid_str);continue ; }
@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }
@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); + write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);
@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;
t/t5701-git-serve.sh +30 -0
@@ -369,6 +369,36 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'object-info supports type' ' + test_config transfer.advertiseObjectInfo true && + + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + + test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type + oid $two_oid + oid $two_oid + 0000 + EOF + + cat >expect <<-EOF && + size + type + $two_oid $two_size blob + $two_oid $two_size blob + 0000 + EOF + + test-tool serve-v2 --stateless-rpc <in >out && + test-tool pkt-line unpack <out >actual && + test_cmp expect actual +' + test_expect_success 'bare OID request' ' test_config transfer.advertiseObjectInfo true &&
Signature · 1 line
2.54.0
PATCH v3 6/8 fetch-object-info: parse type from server response
In reply to Pablo SabaterThe server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the capability.
--- builtin/cat-file.c | 5 +++++ fetch-object-info.c | 24 +++++++++++++++++++++++- fetch-object-info.h | 2 ++ 3 files changed, 30 insertions(+), 1 deletion(-)
builtin/cat-file.c +5 -0
@@ -854,6 +854,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,string_list_append (&data->remote_allowed_atoms,"objectname" );if (results.sizes)string_list_append (&data->remote_allowed_atoms,"objectsize" ); +if (results.types) +string_list_append (&data->remote_allowed_atoms,"objecttype" ); data->skip_object_info =1 ;for (size_t i =0 ; i < results.nr; i++) {
@@ -872,6 +874,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,if (results.sizes) data->size = results.sizes[i]; +if (results.types) + data->type = results.types[i]; + opt->batch_mode =BATCH_MODE_INFO ; data->is_remote =1 ;batch_object_write (argv[i +1 ], output, opt, data,NULL ,0 );
fetch-object-info.c +23 -1
@@ -1,6 +1,7 @@#include "git-compat-util.h" #include "gettext.h" #include "hex.h" +#include "object.h" #include "pkt-line.h" #include "connect.h" #include "oid-array.h"
@@ -62,6 +63,7 @@ int fetch_object_info(const enum protocol_version version,unsigned ask_size =0 ;unsigned ask_type =0 ;int size_index =-1 ; +int type_index =-1 ;size_t wanted;size_t i;
@@ -110,8 +112,15 @@ int fetch_object_info(const enum protocol_version version,die (_ ("object-info: duplicate 'size' attribute" )); size_index = (int )i;CALLOC_ARRAY (results->sizes, results->nr); + }else if (!strcmp (reader->line,"type" )) { +if (!ask_type) +die (_ ("object-info: unrequested 'type' attribute" )); +if (results->types) +die (_ ("object-info: duplicate 'type' attribute" )); + type_index = (int )i; +CALLOC_ARRAY (results->types, results->nr); }else { -BUG ("only size is supported" ); +BUG ("unexpected object-info option: %s" , reader->line); } }
@@ -157,6 +166,18 @@ int fetch_object_info(const enum protocol_version version, object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (results->types) { +const char *type_str = + object_info_values.items[type_index +1 ].string; +int type =type_from_string_gently (type_str,-1 ,1 ); + +if (type <0 ) +die (_ ("object-info: object %s has invalid type '%s'" ), + object_info_values.items[0 ].string, type_str); + + results->types[i] = type; + } +string_list_clear (&object_info_values,0 ); }
@@ -172,6 +193,7 @@ int fetch_object_info(const enum protocol_version version,void free_fetch_object_info_results (struct fetch_object_info_results *results) {free (results->sizes); +free (results->types);free (results->unrecognized);memset (results,0 ,sizeof (*results)); }
fetch-object-info.h +2 -0
@@ -1,11 +1,13 @@ #ifndef FETCH_OBJECT_INFO_H #define FETCH_OBJECT_INFO_H +#include "object.h" #include "pkt-line.h" #include "protocol.h" struct fetch_object_info_results { size_t *sizes; + enum object_type *types; uint8_t *unrecognized; size_t nr; unsigned wants_size:1;
Signature · 1 line
2.54.0
PATCH v3 7/8 serve: advertise type capability
In reply to Pablo SabaterThe server and the client can handle type requests but the client won't ask for it until the server advertises it.
Add type to the advertised capabilities so the client knows that it can request it.
--- serve.c | 4 ++-- t/t1017-cat-file-remote-object-info.sh | 26 ++++++++++++++++++++++---- 2 files changed, 24 insertions(+), 6 deletions(-)
serve.c +2 -2
@@ -97,9 +97,9 @@ static int object_info_advertise(struct repository *r, struct strbuf *value)/* disabled by default */ advertise_object_info =0 ; } -/* Currently only size is supported */ +/* Currently only size and type are supported */ if (value && advertise_object_info) -strbuf_addstr (value,"size" ); +strbuf_addstr (value,"size type" );return advertise_object_info; }
t/t1017-cat-file-remote-object-info.sh +22 -4
@@ -7,6 +7,7 @@ test_description='git cat-file --batch-command with remote-object-info command' hello_content="Hello World" hello_size=$(strlen "$hello_content") +hello_type="blob" hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) hello_short_oid=$(git rev-parse --short "$hello_oid")
@@ -19,6 +20,7 @@ unstored_oid=$(echo_without_newline "$unstored_content" | git hash-object --stdi # file name is hello, which is 5 characters # a space is 1 character and a null is 1 character tree_size=$(($(test_oid rawsz) + 13)) +tree_type="tree" commit_message="Initial commit"
@@ -31,6 +33,7 @@ commit_message="Initial commit" # An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`, # to get 177, 2. then deduct 40 hex characters to get 137 commit_size=$(($(test_oid hexsz) + 137)) +commit_type="commit" tag_header_without_oid="type blob tag hellotag
@@ -44,6 +47,7 @@ $tag_description" tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w) tag_size=$(strlen "$tag_content") +tag_type="tag" set_transport_variables () { hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin)
@@ -256,14 +260,12 @@ test_expect_success 'remote-object-info does not die on missing oid like info' ' ) ' -# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo "$hello_type" >expect && git cat-file --batch-command="%(objecttype)" >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOF
@@ -271,6 +273,22 @@ test_expect_success 'unsupported placeholder on remote returns empty string' ' ) ' +test_expect_success 'unsupported placeholders on remote return empty string' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + fmt="%(objectmode) %(objectsize:disk) %(rest) %(deltabase)" && + + # The hardcoded SPs between the atoms are respected. + echo " " >expect && + git cat-file --batch-command="$fmt" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid + EOF + test_cmp expect actual + ) +' + test_expect_success 'requesting only objectname echoes back' ' ( set_transport_variables "$daemon_parent" &&
Signature · 1 line
2.54.0
PATCH v3 8/8 cat-file: unify default format
In reply to Pablo Sabater%(objecttype) is supported both by the client and by the server. Change the temporary default format to the unified version that the other commands use.
Update documentation to remove %(objecttype) from the caveats of remote-object-info and show %(objecttype) support.
Now that type is supported and the default format unified, update the tests to expect the new default format.
--- Documentation/git-cat-file.adoc | 17 +++++------ Documentation/gitprotocol-v2.adoc | 18 +++++++++--- builtin/cat-file.c | 7 ----- t/t1017-cat-file-remote-object-info.sh | 52 +++++++++++++++++----------------- 4 files changed, 47 insertions(+), 47 deletions(-)
Documentation/git-cat-file.adoc +7 -10
@@ -348,15 +348,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. -The command `remote-object-info` only supports the `%(objectname)` and -`%(objectsize)` placeholders. See `CAVEATS` below for more information. +The command `remote-object-info` only supports the `%(objectname)`, +`%(objectsize)` and `%(objecttype)` placeholders. See `CAVEATS` below for more +information. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`, except for `remote-object-info` commands which -use `%(objectname) %(objectsize)` because `%(objecttype)` is not supported yet. - -WARNING: When "%(objecttype)" is supported, the default format WILL be unified, -so DO NOT RELY on the current default format to stay the same!!! +%(objecttype) %(objectsize)`. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -453,9 +450,9 @@ scripting purposes. CAVEATS ------- -Note that only `%(objectname)` and `%(objectsize)` are currently -supported by the `remote-object-info` command. Using any other placeholder in -the format string will return an empty string in its position. +Note that only `%(objectname)`, `%(objectsize)` and `%(objecttype)` are +currently supported by the `remote-object-info` command. Using any other +placeholder in the format string will return an empty string in its position. Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are
Documentation/gitprotocol-v2.adoc +14 -4
@@ -558,14 +558,17 @@ object-info `object-info` is the command to retrieve information about one or more objects. Its main purpose is to allow a client to make decisions based on this -information without having to fully fetch objects. Object size is the only -information that is currently supported. +information without having to fully fetch objects. Currently only object size +and type are supported. An `object-info` request takes the following arguments: size Requests size information to be returned for each listed object id. + type + Requests type information to be returned for each listed object id. + oid <oid> Indicates to the server an object which the client wants to obtain information for. They must be full OIDs.
@@ -580,11 +583,18 @@ space. info = *PKT-LINE(attr LF) *PKT-LINE(obj-info LF) - attr = "size" + attr = "size" | "type" obj-size = 1*DIGIT - obj-info = obj-id [SP [obj-size]] + obj-type = "blob" | "tree" | "commit" | "tag" + + obj-val = obj-size | obj-type + + obj-info = obj-id [SP [obj-val *(SP obj-val)]] + +The values in `obj-info` appear in the same order as the corresponding `attr` +lines, with exactly one value per requested attribute. If the server does not recognize the OID, the response will be `<oid> SP` regardless of the number of attributes requested.
builtin/cat-file.c +0 -7
@@ -823,15 +823,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,char *line_to_split;struct fetch_object_info_results results =FETCH_OBJECT_INFO_RESULTS_INIT ;struct oid_array object_info_oids =OID_ARRAY_INIT ; -const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )die (_ ("remote-object-info command too long" )); -/* -* TODO: Use the default format once %(objecttype) is supported. -*/ -if (!opt->format) - opt->format ="%(objectname) %(objectsize)" ; line_to_split =xstrdup (line); count =split_cmdline (line_to_split, &argv);
@@ -883,7 +877,6 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->is_remote =0 ; } data->skip_object_info =0 ; - opt->format = saved_format;free_fetch_object_info_results (&results);free (line_to_split);
t/t1017-cat-file-remote-object-info.sh +26 -26
100 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH GSoC v3 1/8] t5701: use test_file_size() to get the size of a file
In reply to Pablo SabaterQuoted 7 lines · Pablo Sabater
The 'basics of object-info' test runs 'wc -c | xargs' twice to get the size of two.t. The pipe to xargs is only there to strip the blanks that some platforms pad the output of wc with.
Use the
test_file_size()helper, which outputs the size directly, and store the result in a variable. Because 'git rev-parse two:two.t' is also run twice, store its output in a variable as well.
It also has the benefit of retaining the exit status from commands run inside a $( ... ) construct placed within a HERE-document. Earlier, if your "git rev-parse" failed, you would not have noticed it directly (though you would probably have seen the "expect" file containing unexpected content). Now your assignment fails when you compute two_oid, if your "git rev-parse" segfaults.
Quoted 38 lines
Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- t/t5701-git-serve.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)t/t5701-git-serve.sh +7
-4@@ -344,20 +344,23 @@ test_expect_success 'unexpected lines are not allowed in fetch request' ' test_expect_success 'basics of object-info' ' test_config transfer.advertiseObjectInfo true && + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + test-tool pkt-line pack >in <<-EOF && command=object-info object-format=$(test_oid algo) 0001 size - oid $(git rev-parse two:two.t) - oid $(git rev-parse two:two.t) + oid $two_oid + oid $two_oid 0000 EOF cat >expect <<-EOF && size - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) + $two_oid $two_size + $two_oid $two_size 0000 EOF
Re: [PATCH GSoC v3 2/8] fetch-object-info: detect truncated server responses
In reply to Pablo SabaterQuoted 7 lines · Pablo Sabater
The loop reading the object-info response stops as soon as the reader returns something other than
PACKET_READ_NORMAL. A server that somehow answers with fewer objects leaves the end of the result arrays empty.The caller trusts that every requested object will be filled in.
die()if the loop doesn't reach the number of oids expected.
This tightening is obviously a good thing to do.
The above description makes me wonder what happens if the other side sends responses for more objects than we requested. We allocate for N objects and loop for up to N iterations, so we will not read more than N. But do we detect that we are out of sync when we read the response to our next request, or before we shut down the connection if we do not have any further requests?
Quoted 27 lines
fetch-object-info.c +3
-2@@ -49,6 +49,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar struct packet_reader *reader, struct object_info *object_info_data, const int stateless_rpc, const int fd_out) { + size_t i;int size_index =-1 ;switch (version) {@@ -82,7 +83,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arBUG ("unknown protocol version" ); } -for (size_t i =0 ; i < args->object_info_options->nr; i++) { +for (i =0 ; i < args->object_info_options->nr; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );@@ -106,7 +107,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; +for (i =0 ;packet_reader_read (reader) ==PACKET_READ_NORMAL && i < args->oids->nr; i++) {
fetch-object-info.c int fetch_object_info(const enum protocol_version version, struct object_info_ar · line 157
Quoted 7 lines
fetch-object-info.c +4
-0@@ -150,6 +151,11 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arstring_list_clear (&object_info_values,0 ); } + +if (i != args->oids->nr) +die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), + (uintmax_t )args->oids->nr, (uintmax_t )i);
OK. We give 'i' a bit longer lifetime so that we can do this comparison, which is inevitable.
Quoted 3 lines
fetch-object-info.c +0
-0check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return 0 ;
Re: [PATCH GSoC v3 3/8] fetch-object-info: pass arguments directly instead of a struct
In reply to Pablo SabaterQuoted 14 lines · Pablo Sabater
struct
object_info_argsgroups three pointers that already live in the transport and are given tofetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests thatfetch_object_info()uses all three of them.Drop the struct and pass those parameters directly to
fetch_object_info()andsend_object_info_request(). This should have no change in behavior.Helped-by Jeff King <[email protected]>Helped-by Junio C Hamano <[email protected]>Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>
Quoted 5 lines
--- fetch-object-info.c | 53 +++++++++++++++++++++++++++++++---------------------- fetch-object-info.h | 17 ++++++++--------- transport.c | 11 +++++------ 3 files changed, 44 insertions(+), 37 deletions(-)
This has lots of changes but quite straight-forward. We lose the intermediate wrapper "object_info_args" and the uses of the members of that wrapper structure are updated with direct reference to a variable.
Re: [PATCH GSoC v3 4/8] fetch-object-info: use dedicated struct for the results
In reply to Pablo SabaterQuoted 29 lines · Pablo Sabater
fetch_object_info()collects information about N objects, but it stores the results in an array ofobject_info. That struct holds the extended parameters ofread_object_info()(The optional outputs the caller wants filled). Its pointers tell that function where to write the answers for a single object.object_infois not meant to be the final storage, and sincefetch_object_info()does not callread_object_info(), there is no reason to use it. Using it means allocating one scalar per object per attribute just to have those pointers somewhere to point at.Add struct
fetch_object_info_results. The caller sets the wants_* flags to say what it is interested in, andfetch_object_info()allocates one array per attribute. A set wants_* flag means "asked for", while a non-NULL array means "available". The caller releases the arrays withfree_fetch_object_info_results().The
object_info_optionsstring list is no longer needed. Filtering against the server's advertisement now sets local ask_* flags, andsend_object_info_request()turns those into the v2 protocol option strings. remote_atom_map[] existed only to map those strings back into atom names, so drop it and buildremote_allowed_atomsfrom the result arrays.
free_object_info_contents()loses its only caller and is dropped.Helped-by Jeff King <[email protected]>Helped-by Junio C Hamano <[email protected]>Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>
Quoted 9 lines
--- builtin/cat-file.c | 59 +++++++++-------------------------- fetch-object-info.c | 90 ++++++++++++++++++++++++++++------------------------- fetch-object-info.h | 28 ++++++++++++----- object-file.c | 10 ------ odb.h | 3 -- transport.c | 3 +- transport.h | 5 +-- 7 files changed, 88 insertions(+), 110 deletions(-)
The direction this step wants to take us looks good, but at this point we only support "size" and the client side starts parsing "type" only in [6/8], while the server side starts advertising "type" only in [7/8]. If the software at this step talks to a newer server that does support "type", it will hit BUG() if the user requests %(objecttype), no? IOW, introduction of "ask_type" smells a bit premature.
Re: [PATCH GSoC v3 1/8] t5701: use test_file_size() to get the size of a file
In reply to Junio C HamanoQuoted 10 lines · Junio C Hamano
Quoted 7 lines · Pablo Sabater
The 'basics of object-info' test runs 'wc -c | xargs' twice to get the size of two.t. The pipe to xargs is only there to strip the blanks that some platforms pad the output of wc with.
Use the
test_file_size()helper, which outputs the size directly, and store the result in a variable. Because 'git rev-parse two:two.t' is also run twice, store its output in a variable as well.
Quoted 6 lines
It also has the benefit of retaining the exit status from commands run inside a $( ... ) construct placed within a HERE-document. Earlier, if your "git rev-parse" failed, you would not have noticed it directly (though you would probably have seen the "expect" file containing unexpected content). Now your assignment fails when you compute
two_oid, if your "git rev-parse" segfaults.
I will add that next reroll.
Thanks,
Pablo
Re: [PATCH GSoC v3 2/8] fetch-object-info: detect truncated server responses
In reply to Junio C HamanoQuoted 10 lines · Junio C Hamano
Quoted 7 lines · Pablo Sabater
The loop reading the object-info response stops as soon as the reader returns something other than
PACKET_READ_NORMAL. A server that somehow answers with fewer objects leaves the end of the result arrays empty.The caller trusts that every requested object will be filled in.
die()if the loop doesn't reach the number of oids expected.
Quoted 8 lines
This tightening is obviously a good thing to do.
The above description makes me wonder what happens if the other side sends responses for more objects than we requested. We allocate for N objects and loop for up to N iterations, so we will not read more than N. But do we detect that we are out of sync when we read the response to our next request, or before we shut down the connection if we do not have any further requests?
As it is now we would only notice in the stateless case. The loop will only go for N lines and then leave the rest unread, then check_stateless_delimiter() reads the next packet and dies because it is a normal packet. If it isn't stateless it will early return and we won't notice.
There is nothing to get out of sync though. The connection is started and finished for each remote-object-info command line. So a later remote-object-info starts fresh.
But even if it is harmless (I think) it's not ideal and I didn't think about this case. The fix should be easy, check the next packet for a flush after iterating, otherwise die().
[snip]
Thanks,
Pablo
Re: [PATCH GSoC v3 4/8] fetch-object-info: use dedicated struct for the results
In reply to Junio C HamanoQuoted 41 lines · Junio C Hamano
Quoted 29 lines · Pablo Sabater
fetch_object_info()collects information about N objects, but it stores the results in an array ofobject_info. That struct holds the extended parameters ofread_object_info()(The optional outputs the caller wants filled). Its pointers tell that function where to write the answers for a single object.object_infois not meant to be the final storage, and sincefetch_object_info()does not callread_object_info(), there is no reason to use it. Using it means allocating one scalar per object per attribute just to have those pointers somewhere to point at.Add struct
fetch_object_info_results. The caller sets the wants_* flags to say what it is interested in, andfetch_object_info()allocates one array per attribute. A set wants_* flag means "asked for", while a non-NULL array means "available". The caller releases the arrays withfree_fetch_object_info_results().The
object_info_optionsstring list is no longer needed. Filtering against the server's advertisement now sets local ask_* flags, andsend_object_info_request()turns those into the v2 protocol option strings. remote_atom_map[] existed only to map those strings back into atom names, so drop it and buildremote_allowed_atomsfrom the result arrays.
free_object_info_contents()loses its only caller and is dropped.Helped-by Jeff King <[email protected]>Helped-by Junio C Hamano <[email protected]>Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>Quoted 9 lines
--- builtin/cat-file.c | 59 +++++++++-------------------------- fetch-object-info.c | 90 ++++++++++++++++++++++++++++------------------------- fetch-object-info.h | 28 ++++++++++++----- object-file.c | 10 ------ odb.h | 3 -- transport.c | 3 +- transport.h | 5 +-- 7 files changed, 88 insertions(+), 110 deletions(-)
Quoted 7 lines
The direction this step wants to take us looks good, but at this point we only support "size" and the client side starts parsing "type" only in [6/8], while the server side starts advertising "type" only in [7/8]. If the software at this step talks to a newer server that does support "type", it will hit
BUG()if the user requests %(objecttype), no? IOW, introduction of "ask_type" smells a bit premature.
True, if a client asks type and size in this patch and the server supports it, "wanted" will be 2 and because in the loop over wanted we only expect size, we will end up BUG()'ing out for something that is not a BUG(), but an old client vs a newer server.
I will move ask_type int a later patch in this series where it fits correctly.
Thanks for noticing it,
Pablo
Re: [PATCH GSoC v3 3/8] fetch-object-info: pass arguments directly instead of a struct
In reply to Pablo SabaterQuoted 5 lines · Pablo Sabater
struct
object_info_argsgroups three pointers that already live in the transport and are given tofetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests thatfetch_object_info()uses all three of them.
But it does use all of them, no? Not sure I follow the reasoning here.
[snip]
Re: [PATCH GSoC v3 3/8] fetch-object-info: pass arguments directly instead of a struct
Quoted 11 lines · Karthik Nayak
Quoted 5 lines · Pablo Sabater
struct
object_info_argsgroups three pointers that already live in the transport and are given tofetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests thatfetch_object_info()uses all three of them.But it does use all of them, no? Not sure I follow the reasoning here.
[snip]
Yes, I meant that it suggests the three of them are used together, but in the end everything is accessed as args->*.
And because object_info_args is built just before calling fetch_object_info() it doesn't make much sense.
citing Peff [1]:
Quoted 7 lines
- there are a lot of semi-opaque structs, like
object_info_args. It would seem simpler to me to pass those elements around independently to the functions that need them. Likewise, we seem to stuff a lot of data into the transport struct rather than passing it to the relevant functions, even though many of those elements are really just used for one function call, and aren't a property of the transport at all.
[1]: https://lore.kernel.org/git/[email protected]/
I'll reword it in the next reroll,
Pablo
v4 cover cat-file: extend remote-object-info to support %(objecttype)
In reply to Pablo SabaterNeither the client nor the server knows how to handle type. This series continues the work for git cat-file --batch-command extending remote-object-info to support "%(objecttype)" end to end. It is based on its predecessor series "cat-file: add remote-object-info to batch-command" [1] merged into master.
Type is the last property that is identical on the server and on the client once fetched.
Whether to support more metadata such as:
- objectsize:disk
- deltabase
- objectmode (needs context)
should be discussed, unlike size and type, the rest depend on how things are packed on the server and on what the client already has, so they cannot reliably match local once fetched. IMO they are not worth supporting, I can't find a use case for them.
To add a new placeholder, follow the steps in this series and add its wants_* flag to struct fetch_object_info_results.
Patches 1-5 are preparatory. They don't change what the command does:
- [1/9] is a test cleanup.
- [2/9] fixes a possible bug in case of a malformed response.
- [3/9] and [4/9] refactor how the object data is stored and handled. The why about this refactor comes from [2].
- [5/9] drops the last error return left in fetch_object_info().
Patches 6-9 are the actual objecttype support:
- [6/9] teaches the server to answer type.
- [7/9] teaches the client to parse it.
- [8/9] advertises the capability so the client can start asking it.
- [9/9] unifies the default format.
Note that *_type appears from [7/9]. Before this series, "type" is already listed in the client's allow-list, but the prep patches drop that list, so the client knows nothing about type until [7/9] adds it back.
Github CI: https://github.com/pabloosabaterr/git/actions/runs/30938909345
[1]: https://lore.kernel.org/git/[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/
---
Changes in v4:
- Rebased onto master now that ps/cat-file-remote-object-info was merged.
- Dropped the Based-on trailer.
- Reworded [1/9] so it shows that we also benefit from preserving the
exit status.
- Changed [2/9] to also check if a response returns more packets.
- New [5/9]: fetch_object_info() dies on its last error return and
becomes void.
- Moved the introduction of ask_type, wants_type to [7/9] instead of
[4/9].
- Link to v3: https://lore.kernel.org/git/[email protected]/
Changes in v3:
- 1/8 (was 2/6): reword the commit message to explain that "wc -c | xargs"
only strips the padding some platforms add.
- New 2/8: die if the server sends fewer object-info lines than
requested OIDs. Preexisting bug.
- New 3/8: drop struct object_info_args and pass its members directly to fetch_object_info().
- New 4/8: replace struct object_info with struct
fetch_object_info_results to hold the results. This also drops the object_info_options string list, remote_atom_map[] and free_object_info_contents().
- Dropped 1/6, which replaced the hardcoded "size" in
send_object_info_request() with a loop over object_info_options, as
that list no longer reaches it.
- 6/8: follows the new result struct, so it only adds the type array.
- 7/8: quote $hello_type in the test.
- Link to v2: https://lore.kernel.org/git/[email protected]/
Changes in v2:
- Changed t5701 test to use test_file_size.
- Changed the for loop to use for_each_string_list_item().
- Moved "fetch-object-info: request all supported options dynamically"
patch to be the first one as it is prep patch.
- Typos and nits.
- Restored the test mixing info and remote-object-info, adapted to the
unified format.
- Link to v1: https://lore.kernel.org/git/[email protected]/
---
Pablo Sabater (9):
t5701: use test_file_size() to get the size of a file
fetch-object-info: detect malformed server responses
fetch-object-info: pass arguments directly instead of a struct
fetch-object-info: use dedicated struct for the results
fetch-object-info: die() on the remaining error path
protocol-caps: add type support to object-info
fetch-object-info: parse type from server response
serve: advertise type capability
cat-file: unify default format
Documentation/git-cat-file.adoc | 17 ++-- Documentation/gitprotocol-v2.adoc | 18 +++- builtin/cat-file.c | 69 +++++---------- fetch-object-info.c | 149 +++++++++++++++++++++------------ fetch-object-info.h | 37 +++++--- object-file.c | 10 --- odb.h | 3 - protocol-caps.c | 21 ++++- serve.c | 4 +- t/t1017-cat-file-remote-object-info.sh | 78 ++++++++++------- t/t5701-git-serve.sh | 41 ++++++++- transport.c | 14 ++-- transport.h | 5 +- 13 files changed, 276 insertions(+), 190 deletions(-)
Range-diff versus v3:
1: 41172a72ce ! 1: 301d7ccca9 t5701: use test_file_size() to get the size of a file
@@ Commit message
Use the test_file_size() helper, which outputs the size directly, and store the result in a variable. Because 'git rev-parse two:two.t' is
- also run twice, store its output in a variable as well.
+ also run multiple times, store its output in a variable as well.
+
+ Storing them in variables outside the HERE-document has the added
+ benefit of preserving their exit statuses.
2: a7a547c9b9 < -: ---------- fetch-object-info: detect truncated server responses -: ---------- > 2: 445d65cc7e fetch-object-info: detect malformed server responses 3: 6a9b839127 ! 3: 107315bb32 fetch-object-info: pass arguments directly instead of a struct
@@ Commit message
struct object_info_args groups three pointers that already live in the transport and are given to fetch_object_info().
Grouping them into a struct reduces the number of parameters, but it
- suggests that fetch_object_info() uses all three of them.
+ suggests that the three belong together, when they are unrelated and end
+ up being accessed as args->* independently.
Drop the struct and pass those parameters directly to
fetch_object_info() and send_object_info_request(). This should have no
@@ fetch-object-info.c: static int parse_object_size(const char *s, size_t *res)
+ struct object_info *object_info_data,
const int stateless_rpc, const int fd_out)
{
- size_t i;
+ int size_index = -1;
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version, struct object_info_ar
* because the number of options is a small known number (the
* supported placeholders which currently are size and type).
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
BUG("unknown protocol version");
}
-- for (i = 0; i < args->object_info_options->nr; i++) {
-+ for (i = 0; i < object_info_options->nr; i++) {
+- for (size_t i = 0; i < args->object_info_options->nr; i++) {
++ for (size_t i = 0; i < object_info_options->nr; i++) {
if (packet_reader_read(reader) != PACKET_READ_NORMAL) {
check_stateless_delimiter(stateless_rpc, reader,
"stateless delimiter expected");
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
xcalloc(1, sizeof(*object_info_data[j].sizep));
} else {
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version, struct object_info_ar
+ }
+ }
- for (i = 0;
- packet_reader_read(reader) == PACKET_READ_NORMAL &&
-- i < args->oids->nr;
-+ i < oids->nr;
- i++) {
+- for (size_t i = 0; i < args->oids->nr; i++) {
++ for (size_t i = 0; i < oids->nr; i++) {
struct string_list object_info_values = STRING_LIST_INIT_DUP;
+ if (packet_reader_read(reader) != PACKET_READ_NORMAL)
+ die(_("object-info: expected %" PRIuMAX " objects, got %" PRIuMAX),
+- (uintmax_t)args->oids->nr, (uintmax_t)i);
++ (uintmax_t)oids->nr, (uintmax_t)i);
+
string_list_split(&object_info_values, reader->line, " ", -1);
if (strcmp(object_info_values.items[0].string,
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
reader->line);
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version, struct object_info_ar
- string_list_clear(&object_info_values, 0);
- }
-- if (i != args->oids->nr)
-+ if (i != oids->nr)
- die(_("object-info: expected %" PRIuMAX " objects, got %" PRIuMAX),
-- (uintmax_t)args->oids->nr, (uintmax_t)i);
-+ (uintmax_t)oids->nr, (uintmax_t)i);
+ if (packet_reader_read(reader) != PACKET_READ_FLUSH)
+ die(_("object-info: expected flush after %"PRIuMAX" objects"),
+- (uintmax_t)args->oids->nr);
++ (uintmax_t)oids->nr);
check_stateless_delimiter(stateless_rpc, reader, "stateless delimiter expected");
4: 0633ddc7e8 ! 4: b942beb9d7 fetch-object-info: use dedicated struct for the results
@@ Commit message
atom names, so drop it and build remote_allowed_atoms from the result
arrays.
+ Currently for wants_* and ask_* there is only the 'size' variant but a
+ subsequent commit will add '*_type'.
+
free_object_info_contents() loses its only caller and is dropped.
+ Dropping the allow-list check makes the final else reachable from the
+ wire, so die() instead of BUG(): an unknown attribute is the server's
+ error, not ours.
+
Helped-by: Jeff King <[email protected]>
Helped-by: Junio C Hamano <[email protected]>
Mentored-by: Karthik Nayak <[email protected]>
@@ builtin/cat-file.c: static void parse_cmd_remote_object_info(struct batch_option
if (data->info.sizep)
- string_list_append(&object_info_options, "size");
-+ results.wants_size = 1;
- if (data->info.typep)
+- if (data->info.typep)
- string_list_append(&object_info_options, "type");
-+ results.wants_type = 1;
++ results.wants_size = 1;
- if (get_remote_info(count, argv, &remote_object_info,
- &object_info_oids, &object_info_options))
@@ fetch-object-info.c
const struct string_list *server_options,
struct oid_array *oids,
- struct string_list *object_info_options)
-+ unsigned ask_size,
-+ unsigned ask_type)
++ unsigned ask_size)
{
struct strbuf req_buf = STRBUF_INIT;
@@ fetch-object-info.c
packet_buf_write(&req_buf, "size");
- else if (object_info_options->nr)
- BUG("only size should be in object_info_options");
-+
-+ if (ask_type)
-+ packet_buf_write(&req_buf, "type");
if (oids)
for (size_t i = 0; i < oids->nr; i++)
@@ fetch-object-info.c: static int parse_object_size(const char *s, size_t *res)
+ const int stateless_rpc,
+ const int fd_out)
{
-- size_t i;
+ unsigned ask_size = 0;
-+ unsigned ask_type = 0;
int size_index = -1;
+ size_t wanted;
-+ size_t i;
+
+ results->nr = oids->nr;
+ CALLOC_ARRAY(results->unrecognized, results->nr);
@@ fetch-object-info.c: static int parse_object_size(const char *s, size_t *res)
+ if (results->wants_size &&
+ server_supports_feature("object-info", "size", 0))
+ ask_size = 1;
-+
-+ if (results->wants_type &&
-+ server_supports_feature("object-info", "type", 0))
-+ ask_type = 1;
/*
* Even if no options are left, we still send the oid so we get
@@ fetch-object-info.c: static int parse_object_size(const char *s, size_t *res)
*/
- send_object_info_request(fd_out, server_options, oids,
- object_info_options);
-+ send_object_info_request(fd_out, server_options, oids, ask_size,
-+ ask_type);
++ send_object_info_request(fd_out, server_options, oids, ask_size);
break;
case protocol_v1:
case protocol_v0:
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
case protocol_unknown_version:
BUG("unknown protocol version");
}
-+ wanted = ask_size + ask_type;
++ wanted = ask_size;
-- for (i = 0; i < object_info_options->nr; i++) {
-+ for (i = 0; i < wanted; i++) {
+- for (size_t i = 0; i < object_info_options->nr; i++) {
++ for (size_t i = 0; i < wanted; i++) {
if (packet_reader_read(reader) != PACKET_READ_NORMAL) {
check_stateless_delimiter(stateless_rpc, reader,
"stateless delimiter expected");
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
- xcalloc(1, sizeof(*object_info_data[j].sizep));
+ CALLOC_ARRAY(results->sizes, results->nr);
} else {
- BUG("only size is supported");
+- BUG("only size is supported");
++ die(_("object-info: unknown attribute '%s'"),
++ reader->line);
}
+ }
+
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
*/
if (object_info_values.nr >= 2 &&
@@ fetch-object-info.h
+ uint8_t *unrecognized;
+ size_t nr;
+ unsigned wants_size:1;
-+ unsigned wants_type:1;
+};
+
+#define FETCH_OBJECT_INFO_RESULTS_INIT { 0 }
-: ---------- > 5: 2e9cce44b6 fetch-object-info: die() on the remaining error path
5: 7678f2ce77 = 6: 30d9898e3b protocol-caps: add type support to object-info
6: bfff160c38 ! 7: 8ebdf712b8 fetch-object-info: parse type from server response@@ Commit message
The server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server
- response once the server advertises the capability.
+ response once the server advertises the type capability.
@@ Commit message
## builtin/cat-file.c ##
@@ builtin/cat-file.c: static void parse_cmd_remote_object_info(struct batch_options *opt,
+
+ if (data->info.sizep)
+ results.wants_size = 1;
++ if (data->info.typep)
++ results.wants_type = 1;
+
+ if (get_remote_info(count, argv, &results, &object_info_oids))
+ die(_("failed to get object info from the remote: %s"), argv[0]);
+@@ builtin/cat-file.c: static void parse_cmd_remote_object_info(struct batch_options *opt,
string_list_append(&data->remote_allowed_atoms, "objectname");
if (results.sizes)
string_list_append(&data->remote_allowed_atoms, "objectsize");
@@ fetch-object-info.c
#include "pkt-line.h"
#include "connect.h"
#include "oid-array.h"
-@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version, +@@
+ static void send_object_info_request(const int fd_out,
+ const struct string_list *server_options,
+ struct oid_array *oids,
+- unsigned ask_size)
++ unsigned ask_size,
++ unsigned ask_type)
+ {
+ struct strbuf req_buf = STRBUF_INIT;
+
+@@ fetch-object-info.c: static void send_object_info_request(const int fd_out,
+ if (ask_size)
+ packet_buf_write(&req_buf, "size");
+
++ if (ask_type)
++ packet_buf_write(&req_buf, "type");
++
+ if (oids)
+ for (size_t i = 0; i < oids->nr; i++)
+ packet_buf_write(&req_buf, "oid %s",
+@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
+ int fd_out)
+ {
unsigned ask_size = 0;
- unsigned ask_type = 0;
++ unsigned ask_type = 0;
int size_index = -1;
+ int type_index = -1;
size_t wanted;
- size_t i;
-@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
+ results->nr = oids->nr;
+@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
+ server_supports_feature("object-info", "size", 0))
+ ask_size = 1;
+
++ if (results->wants_type &&
++ server_supports_feature("object-info", "type", 0))
++ ask_type = 1;
++
+ /*
+ * Even if no options are left, we still send the oid so we get
+ * at least an existence check.
+ */
+- send_object_info_request(fd_out, server_options, oids, ask_size);
++ send_object_info_request(fd_out, server_options, oids, ask_size,
++ ask_type);
+ break;
+ case protocol_v1:
+ case protocol_v0:
+@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
+- wanted = ask_size;
++ wanted = ask_size + ask_type;
+
+ for (size_t i = 0; i < wanted; i++) {
+ if (packet_reader_read(reader) != PACKET_READ_NORMAL) {
+@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
die(_("object-info: duplicate 'size' attribute"));
size_index = (int)i;
CALLOC_ARRAY(results->sizes, results->nr);
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
+ type_index = (int)i;
+ CALLOC_ARRAY(results->types, results->nr);
} else {
-- BUG("only size is supported");
-+ BUG("unexpected object-info option: %s", reader->line);
- }
- }
-
-@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
+ die(_("object-info: unknown attribute '%s'"),
+ reader->line);
+@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
object_info_values.items[0].string, object_info_values.items[size_index + 1].string);
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
string_list_clear(&object_info_values, 0);
}
-@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version, +@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
void free_fetch_object_info_results(struct fetch_object_info_results *results) {
free(results->sizes);
@@ fetch-object-info.h
uint8_t *unrecognized;
size_t nr;
unsigned wants_size:1;
++ unsigned wants_type:1;
+ };
+
+ #define FETCH_OBJECT_INFO_RESULTS_INIT { 0 }
7: 8c58680108 = 8: acfa7cb4b1 serve: advertise type capability 8: fb5032cea5 = 9: 9dc40ba2cb cat-file: unify default format
---
base-commit: 5b2471720c93ee30e5764a19f3d3b3ae9ec9712a
change-id: 20260724-objecttype-support-ea1ef6941d07
PATCH v4 1/9 t5701: use test_file_size() to get the size of a file
The 'basics of object-info' test runs 'wc -c | xargs' twice to get the size of two.t. The pipe to xargs is only there to strip the blanks that some platforms pad the output of wc with.
Use the test_file_size() helper, which outputs the size directly, and store the result in a variable. Because 'git rev-parse two:two.t' is also run multiple times, store its output in a variable as well.
Storing them in variables outside the HERE-document has the added benefit of preserving their exit statuses.
--- t/t5701-git-serve.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
t/t5701-git-serve.sh +7 -4
@@ -344,20 +344,23 @@ test_expect_success 'unexpected lines are not allowed in fetch request' ' test_expect_success 'basics of object-info' ' test_config transfer.advertiseObjectInfo true && + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + test-tool pkt-line pack >in <<-EOF && command=object-info object-format=$(test_oid algo) 0001 size - oid $(git rev-parse two:two.t) - oid $(git rev-parse two:two.t) + oid $two_oid + oid $two_oid 0000 EOF cat >expect <<-EOF && size - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) + $two_oid $two_size + $two_oid $two_size 0000 EOF
Signature · 1 line
2.54.0
PATCH v4 2/9 fetch-object-info: detect malformed server responses
In reply to Pablo SabaterThe loop reading the object-info response stops as soon as the reader returns something other than PACKET_READ_NORMAL, or once it has read as many lines as we requested. Neither end is checked.
A server that answers with fewer objects leaves the end of the result arrays empty, and the caller trusts that every requested object was filled in. A server that answers with more leaves the extra packets unread. On stateless transports check_stateless_delimiter() notices, but on the others it passes unnoticed.
Check both limits by extracting the packet_reader_read() from the loop condition, so the loop no longer consumes the last packet (flush). If while looping the read is different from a PACKET_READ_NORMAL, die() meaning there are fewer objects than expected. After iterating, we only expect a flush, so if the last packet is not a flush, die().
--- fetch-object-info.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
fetch-object-info.c +10 -4
@@ -106,12 +106,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; -packet_reader_read (reader) ==PACKET_READ_NORMAL && - i < args->oids->nr; - i++) { +for (size_t i =0 ; i < args->oids->nr; i++) {struct string_list object_info_values =STRING_LIST_INIT_DUP ; +if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) +die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), + (uintmax_t )args->oids->nr, (uintmax_t )i); +string_list_split (&object_info_values, reader->line," " ,-1 ); if (strcmp(object_info_values.items[0].string,
@@ -150,6 +151,11 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arstring_list_clear (&object_info_values,0 ); } + +if (packet_reader_read (reader) !=PACKET_READ_FLUSH ) +die (_ ("object-info: expected flush after %" PRIuMAX" objects" ), + (uintmax_t )args->oids->nr); +check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return 0 ;
Signature · 1 line
2.54.0
PATCH v4 3/9 fetch-object-info: pass arguments directly instead of a struct
In reply to Pablo Sabaterstruct object_info_args groups three pointers that already live in the transport and are given to fetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests that the three belong together, when they are unrelated and end up being accessed as args->* independently.
Drop the struct and pass those parameters directly to fetch_object_info() and send_object_info_request(). This should have no change in behavior.
--- fetch-object-info.c | 53 +++++++++++++++++++++++++++++++---------------------- fetch-object-info.h | 17 ++++++++--------- transport.c | 11 +++++------ 3 files changed, 44 insertions(+), 37 deletions(-)
fetch-object-info.c +31 -22
@@ -9,20 +9,24 @@#include "string-list.h" /* Sends object-info command and its arguments into the request buffer. */ -static void send_object_info_request (const int fd_out,struct object_info_args *args) +static void send_object_info_request (const int fd_out, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options) {struct strbuf req_buf =STRBUF_INIT ; -write_command_and_capabilities (&req_buf,"object-info" , args->server_options); +write_command_and_capabilities (&req_buf,"object-info" , server_options); -if (unsorted_string_list_has_string (args->object_info_options,"size" )) +if (unsorted_string_list_has_string (object_info_options,"size" ))packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) +else if (object_info_options->nr)BUG ("only size should be in object_info_options" ); -if (args->oids) -for (size_t i =0 ; i < args->oids->nr; i++) -packet_buf_write (&req_buf,"oid %s" ,oid_to_hex (&args->oids->oid[i])); +if (oids) +for (size_t i =0 ; i < oids->nr; i++) +packet_buf_write (&req_buf,"oid %s" , +oid_to_hex (&oids->oid[i]));packet_buf_flush (&req_buf);if (write_in_full (fd_out, req_buf.buf, req_buf.len) <0 )
@@ -45,8 +49,12 @@ static int parse_object_size(const char *s, size_t *res)return 0 ; } -int fetch_object_info (const enum protocol_version version,struct object_info_args *args, -struct packet_reader *reader,struct object_info *object_info_data, +int fetch_object_info (const enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options, +struct packet_reader *reader, +struct object_info *object_info_data,const int stateless_rpc,const int fd_out) {int size_index =-1 ;
@@ -64,16 +72,17 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* because the number of options is a small known number (the * supported placeholders which currently are size and type). */ -for (int i = (int )args->object_info_options->nr -1 ; i >=0 ; i--) +for (int i = (int )object_info_options->nr -1 ; i >=0 ; i--)if (!server_supports_feature ("object-info" , - args->object_info_options->items[i].string,0 )) -unsorted_string_list_delete_item (args->object_info_options, i,0 ); + object_info_options->items[i].string,0 )) +unsorted_string_list_delete_item (object_info_options, i,0 );/* * Even if no options are left, we still send the oid so we get * at least an existence check. */ -send_object_info_request (fd_out, args); +send_object_info_request (fd_out, server_options, oids, + object_info_options);break ;case protocol_v1:case protocol_v0:
@@ -82,14 +91,14 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arBUG ("unknown protocol version" ); } -for (size_t i =0 ; i < args->object_info_options->nr; i++) { +for (size_t i =0 ; i < object_info_options->nr; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return -1 ; } -if (!unsorted_string_list_has_string (args->object_info_options, reader->line)) +if (!unsorted_string_list_has_string (object_info_options, reader->line))return -1 ;if (!strcmp (reader->line,"size" )) {
@@ -98,7 +107,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* is only size. No risk of overflow. */ size_index = (int )i; -for (size_t j =0 ; j < args->oids->nr; j++) +for (size_t j =0 ; j < oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); }else {
@@ -106,19 +115,19 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; i < args->oids->nr; i++) { +for (size_t i =0 ; i < oids->nr; i++) {struct string_list object_info_values =STRING_LIST_INIT_DUP ;if (packet_reader_read (reader) !=PACKET_READ_NORMAL )die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), - (uintmax_t )args->oids->nr, (uintmax_t )i); + (uintmax_t )oids->nr, (uintmax_t )i);string_list_split (&object_info_values, reader->line," " ,-1 );if (strcmp (object_info_values.items[0 ].string, -oid_to_hex (&args->oids->oid[i]))) +oid_to_hex (&oids->oid[i])))die (_ ("object-info: expected OID: %s, got %s" ), -oid_to_hex (&args->oids->oid[i]), +oid_to_hex (&oids->oid[i]), object_info_values.items[0 ].string);/*
@@ -138,7 +147,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* the server we expect the server to answer with the same * number of attributes requested. */ -if (args->object_info_options->nr +1 != object_info_values.nr) +if (object_info_options->nr +1 != object_info_values.nr)die ("object-info: unexpected number of attributes: %s" , reader->line);
@@ -154,7 +163,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arif (packet_reader_read (reader) !=PACKET_READ_FLUSH )die (_ ("object-info: expected flush after %" PRIuMAX" objects" ), - (uintmax_t )args->oids->nr); + (uintmax_t )oids->nr);check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
fetch-object-info.h +8 -9
@@ -4,22 +4,21 @@#include "pkt-line.h" #include "protocol.h" -struct object_info_args { -struct string_list *object_info_options; -const struct string_list *server_options; -struct oid_array *oids; -}; -struct object_info ; +struct oid_array ;/* * Sends git-cat-file object-info command into the request buf and read the * results from packets. * -* Modifies args->object_info_options, on return it contains only the supported +* Modifies object_info_options, on return it contains only the supported * options by the server. */ -int fetch_object_info (enum protocol_version version,struct object_info_args *args, -struct packet_reader *reader,struct object_info *object_info_data, +int fetch_object_info (enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options, +struct packet_reader *reader, +struct object_info *object_info_data,int stateless_rpc,int fd_out);#endif /* FETCH_OBJECT_INFO_H */
transport.c +5 -6
@@ -438,11 +438,6 @@ static int fetch_object_info_via_pack(struct transport *transport)int ret =0 ;struct git_transport_data *data = transport->data;struct packet_reader reader; -struct object_info_args args = {0 }; - - args.server_options = transport->server_options; - args.oids = transport->smart_options->object_info_oids; - args.object_info_options = transport->smart_options->object_info_options;connect_setup (transport,0 ); packet_reader_init(&reader, data->fd[0], NULL, 0,
@@ -453,7 +448,11 @@ static int fetch_object_info_via_pack(struct transport *transport) data->version =discover_version (&reader); transport->hash_algo = reader.hash_algo; - ret =fetch_object_info (data->version, &args, &reader, + ret =fetch_object_info (data->version, + transport->server_options, + transport->smart_options->object_info_oids, + transport->smart_options->object_info_options, + &reader, data->options.object_info_data, transport->stateless_rpc, data->fd[1 ]);
Signature · 1 line
2.54.0
PATCH v4 4/9 fetch-object-info: use dedicated struct for the results
In reply to Pablo Sabaterfetch_object_info() collects information about N objects, but it stores the results in an array of object_info. That struct holds the extended parameters of read_object_info() (The optional outputs the caller wants filled). Its pointers tell that function where to write the answers for a single object. object_info is not meant to be the final storage, and since fetch_object_info() does not call read_object_info(), there is no reason to use it. Using it means allocating one scalar per object per attribute just to have those pointers somewhere to point at.
Add struct fetch_object_info_results. The caller sets the wants_* flags to say what it is interested in, and fetch_object_info() allocates one array per attribute. A set wants_* flag means "asked for", while a non-NULL array means "available". The caller releases the arrays with free_fetch_object_info_results().
The object_info_options string list is no longer needed. Filtering against the server's advertisement now sets local ask_* flags, and send_object_info_request() turns those into the v2 protocol option strings. remote_atom_map[] existed only to map those strings back into atom names, so drop it and build remote_allowed_atoms from the result arrays.
Currently for wants_* and ask_* there is only the 'size' variant but a subsequent commit will add '*_type'.
free_object_info_contents() loses its only caller and is dropped.
Dropping the allow-list check makes the final else reachable from the wire, so die() instead of BUG(): an unknown attribute is the server's error, not ours.
--- builtin/cat-file.c | 59 +++++++++----------------------------- fetch-object-info.c | 81 ++++++++++++++++++++++++++--------------------------- fetch-object-info.h | 27 +++++++++++++----- object-file.c | 10 ------- odb.h | 3 -- transport.c | 3 +- transport.h | 5 ++-- 7 files changed, 77 insertions(+), 111 deletions(-)
builtin/cat-file.c +14 -45
@@ -31,6 +31,7 @@#include "alias.h" #include "remote.h" #include "transport.h" +#include "fetch-object-info.h" /* * Maximum length for a remote URL. While no universal standard exists,
@@ -681,9 +682,8 @@ static void batch_one_object(const char *obj_name,static int get_remote_info (int argc,const char **argv, -struct object_info **remote_object_info, -struct oid_array *object_info_oids, -struct string_list *object_info_options) +struct fetch_object_info_results *results, +struct oid_array *object_info_oids) {int retval =0 ;struct remote *remote =NULL ;
@@ -724,11 +724,9 @@ static int get_remote_info(int argc, goto cleanup; } -CALLOC_ARRAY (*remote_object_info, object_info_oids->nr); gtransport->smart_options->object_info_oids = object_info_oids; - gtransport->smart_options->object_info_options = object_info_options; - gtransport->smart_options->object_info_data = *remote_object_info; + gtransport->smart_options->object_info_results = results; retval =transport_fetch_object_info (gtransport); cleanup:transport_disconnect (gtransport);
@@ -816,21 +814,6 @@ static void parse_cmd_mailmap(struct batch_options *opt UNUSED,load_mailmap (); } -struct protocol_placeholder_entry { -const char *option; -const char *atom; -}; - -static const struct protocol_placeholder_entry remote_atom_map[] = { - {"size" ,"objectsize" }, - {"type" ,"objecttype" }, -/* -* Add new protocol options here. Even if the server doesn't support -* them the allow_list will drop them if the server doesn't advertise -* them. -*/ -}; -static void parse_cmd_remote_object_info (struct batch_options *opt,const char *line,struct strbuf *output,struct expand_data *data)
@@ -838,9 +821,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,int count;const char **argv;char *line_to_split; -struct object_info *remote_object_info =NULL ; +struct fetch_object_info_results results =FETCH_OBJECT_INFO_RESULTS_INIT ;struct oid_array object_info_oids =OID_ARRAY_INIT ; -struct string_list object_info_options =STRING_LIST_INIT_NODUP ;const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )
@@ -861,26 +843,21 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, MAX_ALLOWED_OBJ_LIMIT);if (data->info.sizep) -string_list_append (&object_info_options,"size" ); -if (data->info.typep) -string_list_append (&object_info_options,"type" ); + results.wants_size =1 ; -if (get_remote_info (count, argv, &remote_object_info, - &object_info_oids, &object_info_options)) +if (get_remote_info (count, argv, &results, &object_info_oids))die (_ ("failed to get object info from the remote: %s" ), argv[0 ]);string_list_clear (&data->remote_allowed_atoms,0 );string_list_append (&data->remote_allowed_atoms,"objectname" ); -for (size_t i =0 ; i <ARRAY_SIZE (remote_atom_map); i++) -if (unsorted_string_list_has_string (&object_info_options, remote_atom_map[i].option)) -string_list_append (&data->remote_allowed_atoms, - remote_atom_map[i].atom); +if (results.sizes) +string_list_append (&data->remote_allowed_atoms,"objectsize" ); data->skip_object_info =1 ; -for (size_t i =0 ; i < object_info_oids.nr; i++) { +for (size_t i =0 ; i < results.nr; i++) { data->oid = object_info_oids.oid[i]; -if (remote_object_info[i].unrecognized) { +if (results.unrecognized[i]) {report_object_status (opt,oid_to_hex (&data->oid), &data->oid,"missing" );continue ;
@@ -890,13 +867,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,* When reaching here, it means remote-object-info can retrieve * information from server without downloading them. */ -if (remote_object_info[i].sizep) { - data->size = *remote_object_info[i].sizep; - } - -if (remote_object_info[i].typep) { - data->type = *remote_object_info[i].typep; - } +if (results.sizes) + data->size = results.sizes[i]; opt->batch_mode =BATCH_MODE_INFO ; data->is_remote =1 ;
@@ -906,12 +878,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->skip_object_info =0 ; opt->format = saved_format; -for (size_t i =0 ; i < object_info_oids.nr; i++) -free_object_info_contents (&remote_object_info[i]); -string_list_clear (&object_info_options,0 ); +free_fetch_object_info_results (&results);free (line_to_split);free (argv); -free (remote_object_info);oid_array_clear (&object_info_oids); }
fetch-object-info.c +39 -42
147 lines not on this page · raw message
fetch-object-info.h +20 -7
@@ -4,21 +4,34 @@#include "pkt-line.h" #include "protocol.h" -struct object_info ; +struct fetch_object_info_results { +size_t *sizes; +uint8_t *unrecognized; +size_t nr; +unsigned wants_size:1 ; +}; + +#define FETCH_OBJECT_INFO_RESULTS_INIT { 0 } +struct oid_array ;/* -* Sends git-cat-file object-info command into the request buf and read the +* Sends git-cat-file object-info command into the request buf and reads the * results from packets. * -* Modifies object_info_options, on return it contains only the supported -* options by the server. +* The caller sets the wants_* flags in "results" to indicate which attributes +* it is interested in. On return, "results" holds one array per attribute that +* the server both advertised and answered with. An array left NULL means the +* attribute is not available. +* Release them with free_fetch_object_info_results(). */ int fetch_object_info (enum protocol_version version,const struct string_list *server_options,struct oid_array *oids, -struct string_list *object_info_options,struct packet_reader *reader, -struct object_info *object_info_data, -int stateless_rpc,int fd_out); +struct fetch_object_info_results *results, +int stateless_rpc, +int fd_out); + +void free_fetch_object_info_results (struct fetch_object_info_results *results);#endif /* FETCH_OBJECT_INFO_H */
object-file.c +0 -10
@@ -1340,13 +1340,3 @@ int odb_transaction_files_begin(struct odb_source *source,return 0 ; } - -void free_object_info_contents (struct object_info *object_info) -{ -if (!object_info) -return ; -free (object_info->typep); -free (object_info->sizep); -free (object_info->disk_sizep); -free (object_info->delta_base_oid); -}
odb.h +0 -3
@@ -637,7 +637,4 @@ void parse_alternates(const char *string,const char *relative_base, struct strvec *out); -/* Free pointers inside of object_info, but not object_info itself */ -void free_object_info_contents (struct object_info *object_info); -#endif /* ODB_H */
transport.c +1 -2
@@ -451,9 +451,8 @@ static int fetch_object_info_via_pack(struct transport *transport) ret =fetch_object_info (data->version, transport->server_options, transport->smart_options->object_info_oids, - transport->smart_options->object_info_options, &reader, - data->options.object_info_data, + data->options.object_info_results, transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);
transport.h +3 -2
@@ -7,6 +7,8 @@#include "string-list.h" #include "connect.h" +struct fetch_object_info_results ; +struct git_transport_options {unsigned thin :1 ;unsigned keep :1 ;
@@ -57,8 +59,7 @@ struct git_transport_options {struct oidset *acked_commits;struct oid_array *object_info_oids; -struct object_info *object_info_data; -struct string_list *object_info_options; +struct fetch_object_info_results *object_info_results; };enum transport_family {
Signature · 1 line
2.54.0
PATCH v4 5/9 fetch-object-info: die() on the remaining error path
In reply to Pablo SabaterEvery failure in fetch_object_info() dies except one: a short read while parsing the attribute lines returns -1. That -1 is then passed through fetch_object_info_via_pack() and get_remote_info() up to cat-file, only to die() with a generic message.
Die in fetch_object_info() instead, consistently with the rest of its error paths, and make fetch_object_info() void.
--- fetch-object-info.c | 19 +++++++++---------- fetch-object-info.h | 14 +++++++------- transport.c | 12 ++++++------ 3 files changed, 22 insertions(+), 23 deletions(-)
fetch-object-info.c +9 -10
@@ -47,13 +47,13 @@ static int parse_object_size(const char *s, size_t *res)return 0 ; } -int fetch_object_info (const enum protocol_version version, -const struct string_list *server_options, -struct oid_array *oids, -struct packet_reader *reader, -struct fetch_object_info_results *results, -const int stateless_rpc, -const int fd_out) +void fetch_object_info (enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct packet_reader *reader, +struct fetch_object_info_results *results, +int stateless_rpc, +int fd_out) {unsigned ask_size =0 ;int size_index =-1 ;
@@ -89,7 +89,8 @@ int fetch_object_info(const enum protocol_version version,if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" ); -return -1 ; +die (_ ("object-info: expected %" PRIuMAX" attributes, got %" PRIuMAX), + (uintmax_t )wanted, (uintmax_t )i); }if (!strcmp (reader->line,"size" )) {
@@ -156,8 +157,6 @@ int fetch_object_info(const enum protocol_version version, (uintmax_t)oids->nr);check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" ); - -return 0 ; }void free_fetch_object_info_results (struct fetch_object_info_results *results)
fetch-object-info.h +7 -7
@@ -24,13 +24,13 @@ struct oid_array;* attribute is not available. * Release them with free_fetch_object_info_results(). */ -int fetch_object_info (enum protocol_version version, -const struct string_list *server_options, -struct oid_array *oids, -struct packet_reader *reader, -struct fetch_object_info_results *results, -int stateless_rpc, -int fd_out); +void fetch_object_info (enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct packet_reader *reader, +struct fetch_object_info_results *results, +int stateless_rpc, +int fd_out);void free_fetch_object_info_results (struct fetch_object_info_results *results);
transport.c +6 -6
@@ -448,12 +448,12 @@ static int fetch_object_info_via_pack(struct transport *transport) data->version =discover_version (&reader); transport->hash_algo = reader.hash_algo; - ret =fetch_object_info (data->version, - transport->server_options, - transport->smart_options->object_info_oids, - &reader, - data->options.object_info_results, - transport->stateless_rpc, data->fd[1 ]); +fetch_object_info (data->version, + transport->server_options, + transport->smart_options->object_info_oids, + &reader, + data->options.object_info_results, + transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);if (data->fd[1 ] >=0 )
Signature · 1 line
2.54.0
PATCH v4 6/9 protocol-caps: add type support to object-info
In reply to Pablo SabaterTeach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While touching send_info(), wrap an over-long line and fix the bit field style of requested_info.size.
--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 30 ++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 3 deletions(-)
protocol-caps.c +18 -3
@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/*
@@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string; +enum object_type object_type;struct object_id oid;size_t object_size;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) {packet_writer_error ( writer, -"object-info: protocol error, expected to get oid, not '%s'" , +"object-info: protocol error, expected to get " +"oid, not '%s'" , oid_str);continue ; }
@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }
@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); + write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);
@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;
t/t5701-git-serve.sh +30 -0
@@ -369,6 +369,36 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'object-info supports type' ' + test_config transfer.advertiseObjectInfo true && + + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + + test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type + oid $two_oid + oid $two_oid + 0000 + EOF + + cat >expect <<-EOF && + size + type + $two_oid $two_size blob + $two_oid $two_size blob + 0000 + EOF + + test-tool serve-v2 --stateless-rpc <in >out && + test-tool pkt-line unpack <out >actual && + test_cmp expect actual +' + test_expect_success 'bare OID request' ' test_config transfer.advertiseObjectInfo true &&
Signature · 1 line
2.54.0
PATCH v4 7/9 fetch-object-info: parse type from server response
In reply to Pablo SabaterThe server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the type capability.
--- builtin/cat-file.c | 7 +++++++ fetch-object-info.c | 38 +++++++++++++++++++++++++++++++++++--- fetch-object-info.h | 3 +++ 3 files changed, 45 insertions(+), 3 deletions(-)
builtin/cat-file.c +7 -0
@@ -844,6 +844,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,if (data->info.sizep) results.wants_size =1 ; +if (data->info.typep) + results.wants_type =1 ;if (get_remote_info (count, argv, &results, &object_info_oids))die (_ ("failed to get object info from the remote: %s" ), argv[0 ]);
@@ -852,6 +854,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,string_list_append (&data->remote_allowed_atoms,"objectname" );if (results.sizes)string_list_append (&data->remote_allowed_atoms,"objectsize" ); +if (results.types) +string_list_append (&data->remote_allowed_atoms,"objecttype" ); data->skip_object_info =1 ;for (size_t i =0 ; i < results.nr; i++) {
@@ -870,6 +874,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,if (results.sizes) data->size = results.sizes[i]; +if (results.types) + data->type = results.types[i]; + opt->batch_mode =BATCH_MODE_INFO ; data->is_remote =1 ;batch_object_write (argv[i +1 ], output, opt, data,NULL ,0 );
fetch-object-info.c +35 -3
@@ -1,6 +1,7 @@#include "git-compat-util.h" #include "gettext.h" #include "hex.h" +#include "object.h" #include "pkt-line.h" #include "connect.h" #include "oid-array.h"
@@ -12,7 +13,8 @@static void send_object_info_request (const int fd_out,const struct string_list *server_options,struct oid_array *oids, -unsigned ask_size) +unsigned ask_size, +unsigned ask_type) {struct strbuf req_buf =STRBUF_INIT ;
@@ -21,6 +23,9 @@ static void send_object_info_request(const int fd_out,if (ask_size)packet_buf_write (&req_buf,"size" ); +if (ask_type) +packet_buf_write (&req_buf,"type" ); +if (oids)for (size_t i =0 ; i < oids->nr; i++) packet_buf_write(&req_buf, "oid %s",
@@ -56,7 +61,9 @@ void fetch_object_info(enum protocol_version version, int fd_out) {unsigned ask_size =0 ; +unsigned ask_type =0 ;int size_index =-1 ; +int type_index =-1 ;size_t wanted; results->nr = oids->nr;
@@ -71,11 +78,16 @@ void fetch_object_info(enum protocol_version version, server_supports_feature("object-info", "size", 0)) ask_size =1 ; +if (results->wants_type && +server_supports_feature ("object-info" ,"type" ,0 )) + ask_type =1 ; +/* * Even if no options are left, we still send the oid so we get * at least an existence check. */ -send_object_info_request (fd_out, server_options, oids, ask_size); +send_object_info_request (fd_out, server_options, oids, ask_size, + ask_type);break ;case protocol_v1:case protocol_v0:
@@ -83,7 +95,7 @@ void fetch_object_info(enum protocol_version version,case protocol_unknown_version:BUG ("unknown protocol version" ); } - wanted = ask_size; + wanted = ask_size + ask_type;for (size_t i =0 ; i < wanted; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {
@@ -100,6 +112,13 @@ void fetch_object_info(enum protocol_version version,die (_ ("object-info: duplicate 'size' attribute" )); size_index = (int )i;CALLOC_ARRAY (results->sizes, results->nr); + }else if (!strcmp (reader->line,"type" )) { +if (!ask_type) +die (_ ("object-info: unrequested 'type' attribute" )); +if (results->types) +die (_ ("object-info: duplicate 'type' attribute" )); + type_index = (int )i; +CALLOC_ARRAY (results->types, results->nr); }else {die (_ ("object-info: unknown attribute '%s'" ), reader->line);
@@ -149,6 +168,18 @@ void fetch_object_info(enum protocol_version version, object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (results->types) { +const char *type_str = + object_info_values.items[type_index +1 ].string; +int type =type_from_string_gently (type_str,-1 ,1 ); + +if (type <0 ) +die (_ ("object-info: object %s has invalid type '%s'" ), + object_info_values.items[0 ].string, type_str); + + results->types[i] = type; + } +string_list_clear (&object_info_values,0 ); }
@@ -162,6 +193,7 @@ void fetch_object_info(enum protocol_version version,void free_fetch_object_info_results (struct fetch_object_info_results *results) {free (results->sizes); +free (results->types);free (results->unrecognized);memset (results,0 ,sizeof (*results)); }
fetch-object-info.h +3 -0
@@ -1,14 +1,17 @@#ifndef FETCH_OBJECT_INFO_H #define FETCH_OBJECT_INFO_H +#include "object.h" #include "pkt-line.h" #include "protocol.h" struct fetch_object_info_results {size_t *sizes; +enum object_type *types;uint8_t *unrecognized;size_t nr;unsigned wants_size:1 ; +unsigned wants_type:1 ; };#define FETCH_OBJECT_INFO_RESULTS_INIT { 0 }
Signature · 1 line
2.54.0
PATCH v4 8/9 serve: advertise type capability
In reply to Pablo SabaterThe server and the client can handle type requests but the client won't ask for it until the server advertises it.
Add type to the advertised capabilities so the client knows that it can request it.
--- serve.c | 4 ++-- t/t1017-cat-file-remote-object-info.sh | 26 ++++++++++++++++++++++---- 2 files changed, 24 insertions(+), 6 deletions(-)
serve.c +2 -2
@@ -97,9 +97,9 @@ static int object_info_advertise(struct repository *r, struct strbuf *value)/* disabled by default */ advertise_object_info =0 ; } -/* Currently only size is supported */ +/* Currently only size and type are supported */ if (value && advertise_object_info) -strbuf_addstr (value,"size" ); +strbuf_addstr (value,"size type" );return advertise_object_info; }
t/t1017-cat-file-remote-object-info.sh +22 -4
@@ -7,6 +7,7 @@ test_description='git cat-file --batch-command with remote-object-info command' hello_content="Hello World" hello_size=$(strlen "$hello_content") +hello_type="blob" hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) hello_short_oid=$(git rev-parse --short "$hello_oid")
@@ -19,6 +20,7 @@ unstored_oid=$(echo_without_newline "$unstored_content" | git hash-object --stdi # file name is hello, which is 5 characters # a space is 1 character and a null is 1 character tree_size=$(($(test_oid rawsz) + 13)) +tree_type="tree" commit_message="Initial commit"
@@ -31,6 +33,7 @@ commit_message="Initial commit" # An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`, # to get 177, 2. then deduct 40 hex characters to get 137 commit_size=$(($(test_oid hexsz) + 137)) +commit_type="commit" tag_header_without_oid="type blob tag hellotag
@@ -44,6 +47,7 @@ $tag_description" tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w) tag_size=$(strlen "$tag_content") +tag_type="tag" set_transport_variables () { hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin)
@@ -256,14 +260,12 @@ test_expect_success 'remote-object-info does not die on missing oid like info' ' ) ' -# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo "$hello_type" >expect && git cat-file --batch-command="%(objecttype)" >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOF
@@ -271,6 +273,22 @@ test_expect_success 'unsupported placeholder on remote returns empty string' ' ) ' +test_expect_success 'unsupported placeholders on remote return empty string' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + fmt="%(objectmode) %(objectsize:disk) %(rest) %(deltabase)" && + + # The hardcoded SPs between the atoms are respected. + echo " " >expect && + git cat-file --batch-command="$fmt" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid + EOF + test_cmp expect actual + ) +' + test_expect_success 'requesting only objectname echoes back' ' ( set_transport_variables "$daemon_parent" &&
Signature · 1 line
2.54.0
PATCH v4 9/9 cat-file: unify default format
In reply to Pablo Sabater%(objecttype) is supported both by the client and by the server. Change the temporary default format to the unified version that the other commands use.
Update documentation to remove %(objecttype) from the caveats of remote-object-info and show %(objecttype) support.
Now that type is supported and the default format unified, update the tests to expect the new default format.
--- Documentation/git-cat-file.adoc | 17 +++++------ Documentation/gitprotocol-v2.adoc | 18 +++++++++--- builtin/cat-file.c | 7 ----- t/t1017-cat-file-remote-object-info.sh | 52 +++++++++++++++++----------------- 4 files changed, 47 insertions(+), 47 deletions(-)
Documentation/git-cat-file.adoc +7 -10
@@ -348,15 +348,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. -The command `remote-object-info` only supports the `%(objectname)` and -`%(objectsize)` placeholders. See `CAVEATS` below for more information. +The command `remote-object-info` only supports the `%(objectname)`, +`%(objectsize)` and `%(objecttype)` placeholders. See `CAVEATS` below for more +information. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`, except for `remote-object-info` commands which -use `%(objectname) %(objectsize)` because `%(objecttype)` is not supported yet. - -WARNING: When "%(objecttype)" is supported, the default format WILL be unified, -so DO NOT RELY on the current default format to stay the same!!! +%(objecttype) %(objectsize)`. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -453,9 +450,9 @@ scripting purposes. CAVEATS ------- -Note that only `%(objectname)` and `%(objectsize)` are currently -supported by the `remote-object-info` command. Using any other placeholder in -the format string will return an empty string in its position. +Note that only `%(objectname)`, `%(objectsize)` and `%(objecttype)` are +currently supported by the `remote-object-info` command. Using any other +placeholder in the format string will return an empty string in its position. Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are
Documentation/gitprotocol-v2.adoc +14 -4
@@ -558,14 +558,17 @@ object-info `object-info` is the command to retrieve information about one or more objects. Its main purpose is to allow a client to make decisions based on this -information without having to fully fetch objects. Object size is the only -information that is currently supported. +information without having to fully fetch objects. Currently only object size +and type are supported. An `object-info` request takes the following arguments: size Requests size information to be returned for each listed object id. + type + Requests type information to be returned for each listed object id. + oid <oid> Indicates to the server an object which the client wants to obtain information for. They must be full OIDs.
@@ -580,11 +583,18 @@ space. info = *PKT-LINE(attr LF) *PKT-LINE(obj-info LF) - attr = "size" + attr = "size" | "type" obj-size = 1*DIGIT - obj-info = obj-id [SP [obj-size]] + obj-type = "blob" | "tree" | "commit" | "tag" + + obj-val = obj-size | obj-type + + obj-info = obj-id [SP [obj-val *(SP obj-val)]] + +The values in `obj-info` appear in the same order as the corresponding `attr` +lines, with exactly one value per requested attribute. If the server does not recognize the OID, the response will be `<oid> SP` regardless of the number of attributes requested.
builtin/cat-file.c +0 -7
@@ -823,15 +823,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,char *line_to_split;struct fetch_object_info_results results =FETCH_OBJECT_INFO_RESULTS_INIT ;struct oid_array object_info_oids =OID_ARRAY_INIT ; -const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )die (_ ("remote-object-info command too long" )); -/* -* TODO: Use the default format once %(objecttype) is supported. -*/ -if (!opt->format) - opt->format ="%(objectname) %(objectsize)" ; line_to_split =xstrdup (line); count =split_cmdline (line_to_split, &argv);
@@ -883,7 +877,6 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->is_remote =0 ; } data->skip_object_info =0 ; - opt->format = saved_format;free_fetch_object_info_results (&results);free (line_to_split);
t/t1017-cat-file-remote-object-info.sh +26 -26
100 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH GSoC v4 2/9] fetch-object-info: detect malformed server responses
In reply to Pablo SabaterQuoted 10 lines · Pablo Sabater
The loop reading the object-info response stops as soon as the reader returns something other than
PACKET_READ_NORMAL, or once it has read as many lines as we requested. Neither end is checked.A server that answers with fewer objects leaves the end of the result arrays empty, and the caller trusts that every requested object was filled in. A server that answers with more leaves the extra packets unread. On stateless transports
check_stateless_delimiter()notices, but on the others it passes unnoticed.
Quoted 5 lines
Check both limits by extracting the
packet_reader_read()from the loop condition, so the loop no longer consumes the last packet (flush). If while looping the read is different from aPACKET_READ_NORMAL,die()meaning there are fewer objects than expected. After iterating, we only expect a flush, so if the last packet is not a flush,die().
OK.
After looking at "if (i != oids->nr)" now gone from the end of the function, but before looking at how the loop terminates its iterations, I wondered how the "there are too few" detection went. It now not just stops iterating when seeing a status other than PACKET_READ_NORMAL, but it actively barfs by dying, to detect a short read. So the only thing we need to check after the loop is if we are truly at the end of the "list of oids->nr things".
Makes sense.
Quoted 42 lines
Helped-by Junio C Hamano <[email protected]>Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>--- fetch-object-info.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)fetch-object-info.c +10
-4@@ -106,12 +106,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; -packet_reader_read (reader) ==PACKET_READ_NORMAL && - i < args->oids->nr; - i++) { +for (size_t i =0 ; i < args->oids->nr; i++) {struct string_list object_info_values =STRING_LIST_INIT_DUP ; +if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) +die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), + (uintmax_t )args->oids->nr, (uintmax_t )i); +string_list_split (&object_info_values, reader->line," " ,-1 ); if (strcmp(object_info_values.items[0].string,@@ -150,6 +151,11 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arstring_list_clear (&object_info_values,0 ); } + +if (packet_reader_read (reader) !=PACKET_READ_FLUSH ) +die (_ ("object-info: expected flush after %" PRIuMAX" objects" ), + (uintmax_t )args->oids->nr); +check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return 0 ;
Re: [PATCH GSoC v4 3/9] fetch-object-info: pass arguments directly instead of a struct
In reply to Pablo SabaterQuoted 15 lines · Pablo Sabater
struct
object_info_argsgroups three pointers that already live in the transport and are given tofetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests that the three belong together, when they are unrelated and end up being accessed as args->* independently.Drop the struct and pass those parameters directly to
fetch_object_info()andsend_object_info_request(). This should have no change in behavior.Helped-by Jeff King <[email protected]>Helped-by Junio C Hamano <[email protected]>Mentored-by Karthik Nayak <[email protected]>Mentored-by Chandra Pratap <[email protected]>Signed-off-by Pablo Sabater <[email protected]>
Quoted 5 lines
--- fetch-object-info.c | 53 +++++++++++++++++++++++++++++++---------------------- fetch-object-info.h | 17 ++++++++--------- transport.c | 11 +++++------ 3 files changed, 44 insertions(+), 37 deletions(-)
OK, in this iteration, the step is truly about just getting rid of the intermediate "args" structure and passing these three things separately, without any other changes like the (half) addition of "type" support like the previous iteration had.
Re: [PATCH GSoC v4 4/9] fetch-object-info: use dedicated struct for the results
In reply to Pablo SabaterQuoted 8 lines · Pablo Sabater
builtin/cat-file.c | 59 +++++++++----------------------------- fetch-object-info.c | 81 ++++++++++++++++++++++++++--------------------------- fetch-object-info.h | 27 +++++++++++++----- object-file.c | 10 ------- odb.h | 3 -- transport.c | 3 +- transport.h | 5 ++-- 7 files changed, 77 insertions(+), 111 deletions(-)
We lost ~30 lines mostly thanks to losing an intermediate structure and need to flip pointer members in it to point at heap allocated locations?
Quoted 4 lines
builtin/cat-file.c +0
-4-struct protocol_placeholder_entry { -const char *option; -const char *atom; -};
builtin/cat-file.c static void parse_cmd_mailmap(struct batch_options *opt UNUSED, · line 832
Quoted 10 lines
builtin/cat-file.c +0
-10- -static const struct protocol_placeholder_entry remote_atom_map[] = { - {"size" ,"objectsize" }, - {"type" ,"objecttype" }, -/* -* Add new protocol options here. Even if the server doesn't support -* them the allow_list will drop them if the server doesn't advertise -* them. -*/ -};
Hmph, it would not make any difference in the final result, as we are losing protocol_placeholder_entry and this list of atoms, but
I am somewhat confused. Why we already had "type" here, even
though we are adding support for it much later in the series (and unlike the previous iteration, this iteration did not do the ask/want_type thing in the previous step)?
Re: [PATCH GSoC v4 4/9] fetch-object-info: use dedicated struct for the results
Quoted 11 lines · Junio C Hamano
Quoted 8 lines · Pablo Sabater
builtin/cat-file.c | 59 +++++++++----------------------------- fetch-object-info.c | 81 ++++++++++++++++++++++++++--------------------------- fetch-object-info.h | 27 +++++++++++++----- object-file.c | 10 ------- odb.h | 3 -- transport.c | 3 +- transport.h | 5 ++-- 7 files changed, 77 insertions(+), 111 deletions(-)
Quoted 3 lines
We lost ~30 lines mostly thanks to losing an intermediate structure and need to flip pointer members in it to point at heap allocated locations?
Yes, also contributes that the logic that filtered the allow-list is dropped, and compared to the previous version, no type logic is in this patch.
Quoted 14 lines
Quoted 14 lines
builtin/cat-file.c +0
-13-struct protocol_placeholder_entry { -const char *option; -const char *atom; -}; - -static const struct protocol_placeholder_entry remote_atom_map[] = { - {"size" ,"objectsize" }, - {"type" ,"objecttype" }, -/* -* Add new protocol options here. Even if the server doesn't support -* them the allow_list will drop them if the server doesn't advertise -* them. -*/
Quoted 8 lines
Quoted 1 line
-};
Hmph, it would not make any difference in the final result, as we are losing
protocol_placeholder_entryand this list of atoms, but
I am somewhat confused. Why we already had "type" here, even
though we are adding support for it much later in the series (and unlike the previous iteration, this iteration did not do the ask/want_type thing in the previous step)?
The list was designed so even if new placeholder were added, it would be idemp. because nothing would happen until a server did advertise that attribute; that's why there's no change in behavior.
It was there because it was interesting for the tests. A known placeholder but unsupported in the server still outputs an empty string (this would be the case of: new client vs old server).
This way we had tested:
unknown + unsupported as %(deltabase) is.
known + unsupported as %(objecttype) was. (that's why it's on the list) knwon + supported as %(objectsize) is.
Using object_info and the list made sense because object_info already had a type field, so listing type cost nothing. Now that we have to dedicate an array per attribute it would mean carrying a types array that nobody fills, so I preferred to drop it here and implement type fully later.
Hope this makes sense,
Pablo
Re: [PATCH GSoC v4 3/9] fetch-object-info: pass arguments directly instead of a struct
In reply to Pablo SabaterQuoted 6 lines · Pablo Sabater
struct
object_info_argsgroups three pointers that already live in the transport and are given tofetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests that the three belong together, when they are unrelated and end up being accessed as args->* independently.
This reads much better and explains why we make the change.
Quoted 3 lines
Drop the struct and pass those parameters directly to
fetch_object_info()andsend_object_info_request(). This should have no change in behavior.
[snip]
The changes too look in order.
Quoted 5 lines · Pablo Sabater
Patches 1-5 are preparatory. They don't change what the command does:
- [1/9] is a test cleanup.
- [2/9] fixes a possible bug in case of a malformed response.
- [3/9] and [4/9] refactor how the object data is stored and handled. The why about this refactor comes from [2].
Thanks, I think these refactors in patches 3 and 4 make sense and address the issues raised in the earlier thread. I'd actually take patch 3 just a step further, as below (which you are welcome to put on top of your series, or work it into the middle, or even take as inspiration and rewrite as part of another patch).
-- >8 --
Subject: transport: drop remote object-info fields from transport struct
A remote object-info request needs three things: the transport for contacting the remote, the list of oids to request, and a place to store the output.
Rather than take these as function parameters, we take only the transport object, and expect the caller to have placed the other two into special fields in the transport struct. But this doesn't make much sense. The set of oids and results are really only valid for one request. There is no reason the transport would need to hang on to them outside of the single function call.
Even though we save a few lines passing the parameters around through the various vtable functions, the result is harder to understand (for example, who is responsible for cleaning up results, and when shoudl it happen?). It also opens up the possibility of a subtle bug. A caller is likely to point those fields to stack variables which could go out of scope, and the transport struct would be left holding invalid pointers.
This is mostly harmless now, as we disconnect the transport immediately after the sole caller of transport_fetch_object_info(). But conceptually we could keep we could keep the transport open and make multiple fetch calls (and reuse the same connection to the helper, to a remote HTTP server, and so on).
So let's pull these out of the struct and pass them as function parameters. It's a little more verbose, but I think more clearly illustrates the intent. I've also tweaked a few function signatures to mark the input oid array as const, since it is purely an input to the function.
---
I do think the concept of reusing the transport will become useful later. We limit a single request to 10,000 objects, so it is quite conceivable a caller would want to make several. That can mostly come later on top, though I think the design of the remote-object-info command makes it awkward. Each invocation provides a remote by name, which is then resolved to a transport. But a given caller is likely going to provide the same remote over and over again.
We probably could get away with just caching the last-used transport and reusing it when fed the same remote name again. But we could perhaps also change the protocol (which AFAICT is not yet in any released version, so still available for changes) to specify the two independently, like:
remote https://example.com/foo.git
remote-object-info objA objB objC...
remote-object-info objX objY objZ
And then it is more clear that setting "remote" is stateful, and will be used for subsequent remote-* commands. But maybe that statefulness is something we don't want. I dunno.
Anyway, either way I think the cleanup below is worth doing in the short term.
builtin/cat-file.c | 6 ++---- fetch-object-info.c | 4 ++-- fetch-object-info.h | 2 +- transport-helper.c | 7 +++++-- transport-internal.h | 4 +++- transport.c | 14 +++++++++----- transport.h | 7 +++---- 7 files changed, 25 insertions(+), 19 deletions(-)
builtin/cat-file.c +2 -4
@@ -730,10 +730,8 @@ static int get_remote_info(int argc, goto cleanup; } - gtransport->smart_options->object_info_oids = object_info_oids; - - gtransport->smart_options->object_info_results = results; - retval =transport_fetch_object_info (gtransport); + retval =transport_fetch_object_info (gtransport, object_info_oids, + results); cleanup:transport_disconnect (gtransport);return retval;
fetch-object-info.c +2 -2
@@ -12,7 +12,7 @@/* Sends object-info command and its arguments into the request buffer. */ static void send_object_info_request (const int fd_out,const struct string_list *server_options, -struct oid_array *oids, +const struct oid_array *oids,unsigned ask_size,unsigned ask_type) {
@@ -54,7 +54,7 @@ static int parse_object_size(const char *s, size_t *res) void fetch_object_info(enum protocol_version version, const struct string_list *server_options, - struct oid_array *oids, + const struct oid_array *oids, struct packet_reader *reader, struct fetch_object_info_results *results, int stateless_rpc,
fetch-object-info.h +1 -1
@@ -29,7 +29,7 @@ struct oid_array;*/ void fetch_object_info(enum protocol_version version, const struct string_list *server_options, - struct oid_array *oids, + const struct oid_array *oids, struct packet_reader *reader, struct fetch_object_info_results *results, int stateless_rpc,
transport-helper.c +5 -2
@@ -786,11 +786,14 @@ static int fetch_refs(struct transport *transport,return -1 ; } -static int fetch_object_info_helper (struct transport *transport) +static int fetch_object_info_helper (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {get_helper (transport);if (process_connect (transport,0 )) -return transport->vtable->fetch_object_info (transport); +return transport->vtable->fetch_object_info (transport, oids, + results);die (_ ("object-info requires protocol v2" )); }
transport-internal.h +3 -1
@@ -51,7 +51,9 @@ struct transport_vtable {* * Uses object-info capability of v2 protocol. */ -int (*fetch_object_info)(struct transport *transport); +int (*fetch_object_info)(struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results);/** * Push the objects and refs. Send the necessary objects, and
transport.c +9 -5
@@ -433,7 +433,9 @@ static int get_bundle_uri(struct transport *transport) transport->bundles, stateless_rpc); } -static int fetch_object_info_via_pack (struct transport *transport) +static int fetch_object_info_via_pack (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -450,9 +452,9 @@ static int fetch_object_info_via_pack(struct transport *transport)fetch_object_info (data->version, transport->server_options, - transport->smart_options->object_info_oids, + oids, &reader, - data->options.object_info_results, + results, transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);
@@ -465,11 +467,13 @@ static int fetch_object_info_via_pack(struct transport *transport)return ret; } -int transport_fetch_object_info (struct transport *transport) +int transport_fetch_object_info (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {if (!transport->vtable->fetch_object_info)die (_ ("remote does not support object-info" )); -return transport->vtable->fetch_object_info (transport); +return transport->vtable->fetch_object_info (transport, oids, results); } static int fetch_refs_via_pack(struct transport *transport,
transport.h +3 -4
@@ -57,9 +57,6 @@ struct git_transport_options {* common commits to this oidset instead of fetching any packfiles. */ struct oidset *acked_commits; - -struct oid_array *object_info_oids; -struct fetch_object_info_results *object_info_results; };enum transport_family {
@@ -317,7 +314,9 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs);/* * Fetch the object info from remote */ -int transport_fetch_object_info (struct transport *transport); +int transport_fetch_object_info (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results);/* * If this flag is set, unlocking will avoid to call non-async-signal-safe
Signature · 1 line
2.55.0.819.g4e24f5e379
Quoted 8 lines · Jeff King
Quoted 5 lines · Pablo Sabater
Patches 1-5 are preparatory. They don't change what the command does:
- [1/9] is a test cleanup.
- [2/9] fixes a possible bug in case of a malformed response.
- [3/9] and [4/9] refactor how the object data is stored and handled. The why about this refactor comes from [2].
Quoted 5 lines
Thanks, I think these refactors in patches 3 and 4 make sense and address the issues raised in the earlier thread. I'd actually take patch 3 just a step further, as below (which you are welcome to put on top of your series, or work it into the middle, or even take as inspiration and rewrite as part of another patch).
Wow, thanks a lot for getting so involved, I think I'll place it as is.
Quoted 50 lines
-- >8 --
Subject: transport: drop remote object-info fields from transport structA remote object-info request needs three things: the transport for contacting the remote, the list of oids to request, and a place to store the output.
Rather than take these as function parameters, we take only the transport object, and expect the caller to have placed the other two into special fields in the transport struct. But this doesn't make much sense. The set of oids and results are really only valid for one request. There is no reason the transport would need to hang on to them outside of the single function call.
Even though we save a few lines passing the parameters around through the various vtable functions, the result is harder to understand (for example, who is responsible for cleaning up results, and when shoudl it happen?). It also opens up the possibility of a subtle bug. A caller is likely to point those fields to stack variables which could go out of scope, and the transport struct would be left holding invalid pointers.
This is mostly harmless now, as we disconnect the transport immediately after the sole caller of
transport_fetch_object_info(). But conceptually we could keep we could keep the transport open and make multiple fetch calls (and reuse the same connection to the helper, to a remote HTTP server, and so on).So let's pull these out of the struct and pass them as function parameters. It's a little more verbose, but I think more clearly illustrates the intent. I've also tweaked a few function signatures to mark the input oid array as const, since it is purely an input to the function.
Signed-off-by Jeff King <[email protected]>---
I do think the concept of reusing the transport will become useful later. We limit a single request to 10,000 objects, so it is quite conceivable a caller would want to make several. That can mostly come later on top, though I think the design of the remote-object-info command makes it awkward. Each invocation provides a remote by name, which is then resolved to a transport. But a given caller is likely going to provide the same remote over and over again.We probably could get away with just caching the last-used transport and reusing it when fed the same remote name again. But we could perhaps also change the protocol (which AFAICT is not yet in any released version, so still available for changes) to specify the two independently, like:
Quoted 7 lines
remote https://example.com/foo.git
remote-object-info objA objB objC...
remote-object-info objX objY objZAnd then it is more clear that setting "remote" is stateful, and will be used for subsequent remote-* commands. But maybe that statefulness is something we don't want. I dunno.
Yeah, I think it is not in any released version yet as the ps/cat-file-remote-object-info (the one that precedes this series) landed in 'master' the first What's cooking of August [1].
Given that, I think that it could be a good idea to have both, if a user foresees that he's only going to make one 'remote-object-info' command he can write it as it is now:
remote-object-info <remote> objA objB
But if a user foresees that he will have to make multiple ones, we can make what you suggested:
Quoted 3 lines
remote https://example.com/foo.git
remote-object-info objA objB objC...
remote-object-info objX objY objZ
We would have to make the remote optional, if there's no remote die(), etc. We would also have to tell apart a remote from an OID in the first argument, but full OIDs and remote URLs are not very similar so that should not be hard haha.
I do like the idea, but I see it more as a follow-up series after this one, as the topic of this series is type support.
Also, I'm biased as I have little time before my deadline ends.
I'm happy to keep doing things and there are more things related to the object-info protocol that I'd like to keep working on after finishing GSoC.
Quoted 159 lines
Anyway, either way I think the cleanup below is worth doing in the short term.
builtin/cat-file.c | 6 ++---- fetch-object-info.c | 4 ++-- fetch-object-info.h | 2 +- transport-helper.c | 7 +++++-- transport-internal.h | 4 +++- transport.c | 14 +++++++++----- transport.h | 7 +++---- 7 files changed, 25 insertions(+), 19 deletions(-)builtin/cat-file.c +2
-4@@ -730,10 +730,8 @@ static int get_remote_info(int argc, goto cleanup; } - gtransport->smart_options->object_info_oids = object_info_oids; - - gtransport->smart_options->object_info_results = results; - retval =transport_fetch_object_info (gtransport); + retval =transport_fetch_object_info (gtransport, object_info_oids, + results); cleanup:transport_disconnect (gtransport);return retval;fetch-object-info.c +2
-2@@ -12,7 +12,7 @@/* Sends object-info command and its arguments into the request buffer. */ static void send_object_info_request (const int fd_out,const struct string_list *server_options, -struct oid_array *oids, +const struct oid_array *oids,unsigned ask_size,unsigned ask_type) {@@ -54,7 +54,7 @@ static int parse_object_size(const char *s, size_t *res) void fetch_object_info(enum protocol_version version, const struct string_list *server_options, - struct oid_array *oids, + const struct oid_array *oids, struct packet_reader *reader, struct fetch_object_info_results *results, int stateless_rpc,fetch-object-info.h +1
-1@@ -29,7 +29,7 @@ struct oid_array;*/ void fetch_object_info(enum protocol_version version, const struct string_list *server_options, - struct oid_array *oids, + const struct oid_array *oids, struct packet_reader *reader, struct fetch_object_info_results *results, int stateless_rpc,transport-helper.c +5
-2@@ -786,11 +786,14 @@ static int fetch_refs(struct transport *transport,return -1 ; } -static int fetch_object_info_helper (struct transport *transport) +static int fetch_object_info_helper (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {get_helper (transport);if (process_connect (transport,0 )) -return transport->vtable->fetch_object_info (transport); +return transport->vtable->fetch_object_info (transport, oids, + results);die (_ ("object-info requires protocol v2" )); }transport-internal.h +3
-1@@ -51,7 +51,9 @@ struct transport_vtable {* * Uses object-info capability of v2 protocol. */ -int (*fetch_object_info)(struct transport *transport); +int (*fetch_object_info)(struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results);/** * Push the objects and refs. Send the necessary objects, and transport.c +9
-5@@ -433,7 +433,9 @@ static int get_bundle_uri(struct transport *transport) transport->bundles, stateless_rpc); } -static int fetch_object_info_via_pack (struct transport *transport) +static int fetch_object_info_via_pack (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {int ret =0 ;struct git_transport_data *data = transport->data;@@ -450,9 +452,9 @@ static int fetch_object_info_via_pack(struct transport *transport)fetch_object_info (data->version, transport->server_options, - transport->smart_options->object_info_oids, + oids, &reader, - data->options.object_info_results, + results, transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);@@ -465,11 +467,13 @@ static int fetch_object_info_via_pack(struct transport *transport)return ret; } -int transport_fetch_object_info (struct transport *transport) +int transport_fetch_object_info (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {if (!transport->vtable->fetch_object_info)die (_ ("remote does not support object-info" )); -return transport->vtable->fetch_object_info (transport); +return transport->vtable->fetch_object_info (transport, oids, results); } static int fetch_refs_via_pack(struct transport *transport,transport.h +0
-3@@ -57,9 +57,6 @@ struct git_transport_options {* common commits to this oidset instead of fetching any packfiles. */ struct oidset *acked_commits; - -struct oid_array *object_info_oids; -struct fetch_object_info_results *object_info_results; };enum transport_family {
transport.h int transport_fetch_refs(struct transport *transport, struct ref *refs); · line 322
Quoted 10 lines
transport.h +3
-1@@ -317,7 +314,9 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs);/* * Fetch the object info from remote */ -int transport_fetch_object_info (struct transport *transport); +int transport_fetch_object_info (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results);/* * If this flag is set, unlocking will avoid to call non-async-signal-safe
I see everything all right.
There's two typos on the patch's commit message:
- s/shoudl/should/
- a duplicated "we could keep"
I will fix them, so if you see anything changed in your patch it's just that. If I end up changing anything else, I'll let you know.
[1]: https://lore.kernel.org/git/[email protected]/T/#t
Thanks, a lot,
Pablo
v5 cover cat-file: extend remote-object-info to support %(objecttype)
In reply to Pablo SabaterNeither the client nor the server knows how to handle type. This series continues the work for git cat-file --batch-command extending remote-object-info to support "%(objecttype)" end to end. It is based on its predecessor series "cat-file: add remote-object-info to batch-command" [1] merged into master.
Type is the last property that is identical on the server and on the client once fetched.
Whether to support more metadata such as: - objectsize:disk
- deltabase
- objectmode (needs context)
should be discussed, unlike size and type, the rest depend on how things are packed on the server and on what the client already has, so they cannot reliably match local once fetched. IMO they are not worth supporting, I can't find a use case for them.
To add a new placeholder, follow the steps in this series and add its wants_* flag to struct fetch_object_info_results.
Patches 1-6 are preparatory. They don't change what the command does:
- [1/10] is a test cleanup.
- [2/10] fixes a possible bug in case of a malformed response.
- [3/10] and [4/10] refactor how the object data is stored and handled. The why about this refactor comes from [2].
- [5/10] drops the last error return left in fetch_object_info().
- [6/10] drops the remote object-info fields from the transport struct.
Patches 7-10 are the actual objecttype support:
- [7/10] teaches the server to answer type.
- [8/10] teaches the client to parse it.
- [9/10] advertises the capability so the client can start asking it.
- [10/10] unifies the default format.
Note that *_type appears from [8/10]. Before this series, "type" is already listed in the client's allow-list, but the prep patches drop that list, so the client knows nothing about type until [8/10] adds it back.
Github CI: https://github.com/pabloosabaterr/git/actions/runs/30938909345
[1]: https://lore.kernel.org/git/[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/
---
Changes in v5:
- New [6/10] by Jeff King: drop the remote object-info fields from the transport struct and pass them as function parameters instead.
- [5/10]: keep the const qualifiers on parameters of fetch_object_info(), matching how the signature was.
- Link to v4: https://patch.msgid.link/[email protected]
Changes in v4:
- Rebased onto master now that ps/cat-file-remote-object-info was merged.
- Dropped the Based-on trailer.
- Reworded [1/9] so it shows that we also benefit from preserving the
exit status.
- Changed [2/9] to also check if a response returns more packets.
- New [5/9]: fetch_object_info() dies on its last error return and
becomes void.
- Moved the introduction of ask_type, wants_type to [7/9] instead of
[4/9].
- Link to v3: https://lore.kernel.org/git/[email protected]/
Changes in v3:
- 1/8 (was 2/6): reword the commit message to explain that "wc -c | xargs"
only strips the padding some platforms add.
- New 2/8: die if the server sends fewer object-info lines than
requested OIDs. Preexisting bug.
- New 3/8: drop struct object_info_args and pass its members directly to fetch_object_info().
- New 4/8: replace struct object_info with struct
fetch_object_info_results to hold the results. This also drops the object_info_options string list, remote_atom_map[] and free_object_info_contents().
- Dropped 1/6, which replaced the hardcoded "size" in
send_object_info_request() with a loop over object_info_options, as
that list no longer reaches it.
- 6/8: follows the new result struct, so it only adds the type array.
- 7/8: quote $hello_type in the test.
- Link to v2: https://lore.kernel.org/git/[email protected]/
Changes in v2:
- Changed t5701 test to use test_file_size.
- Changed the for loop to use for_each_string_list_item().
- Moved "fetch-object-info: request all supported options dynamically"
patch to be the first one as it is prep patch.
- Typos and nits.
- Restored the test mixing info and remote-object-info, adapted to the
unified format.
- Link to v1: https://lore.kernel.org/git/[email protected]/
---
Jeff King (1):
transport: drop remote object-info fields from transport struct
Pablo Sabater (9):
t5701: use test_file_size() to get the size of a file
fetch-object-info: detect malformed server responses
fetch-object-info: pass arguments directly instead of a struct
fetch-object-info: use dedicated struct for the results
fetch-object-info: die() on the remaining error path
protocol-caps: add type support to object-info
fetch-object-info: parse type from server response
serve: advertise type capability
cat-file: unify default format
Documentation/git-cat-file.adoc | 17 ++-- Documentation/gitprotocol-v2.adoc | 18 +++- builtin/cat-file.c | 73 +++++----------- fetch-object-info.c | 149 +++++++++++++++++++++------------ fetch-object-info.h | 37 +++++--- object-file.c | 10 --- odb.h | 3 - protocol-caps.c | 21 ++++- serve.c | 4 +- t/t1017-cat-file-remote-object-info.sh | 78 ++++++++++------- t/t5701-git-serve.sh | 41 ++++++++- transport-helper.c | 7 +- transport-internal.h | 4 +- transport.c | 24 +++--- transport.h | 10 +-- 15 files changed, 294 insertions(+), 202 deletions(-)
Range-diff versus v4:
1: ef807d2fa2 = 1: e5f8dd814c t5701: use test_file_size() to get the size of a file
2: c3c3e37c36 = 2: 4c108f4a59 fetch-object-info: detect malformed server responses
3: 193f71d224 = 3: 54fe4136de fetch-object-info: pass arguments directly instead of a struct
4: ee82d07710 = 4: f286159201 fetch-object-info: use dedicated struct for the results
5: b5c09feca9 ! 5: dbf5832d05 fetch-object-info: die() on the remaining error path
@@ fetch-object-info.c: static int parse_object_size(const char *s, size_t *res)
- struct fetch_object_info_results *results,
- const int stateless_rpc,
- const int fd_out)
-+void fetch_object_info(enum protocol_version version,
++void fetch_object_info(const enum protocol_version version,
+ const struct string_list *server_options,
+ struct oid_array *oids,
+ struct packet_reader *reader,
+ struct fetch_object_info_results *results,
-+ int stateless_rpc,
-+ int fd_out)
++ const int stateless_rpc,
++ const int fd_out)
{
unsigned ask_size = 0;
int size_index = -1;
-: ---------- > 6: f89a6a8c1d transport: drop remote object-info fields from transport struct
6: 2c95d76b46 = 7: 56a9cdbde6 protocol-caps: add type support to object-info
7: 8c6b342c3a ! 8: 1fd7392db9 fetch-object-info: parse type from server response @@ fetch-object-info.c
@@
static void send_object_info_request(const int fd_out,
const struct string_list *server_options,
- struct oid_array *oids,
+ const struct oid_array *oids,
- unsigned ask_size)
+ unsigned ask_size,
+ unsigned ask_type)
@@ fetch-object-info.c: static void send_object_info_request(const int fd_out,
if (oids)
for (size_t i = 0; i < oids->nr; i++)
packet_buf_write(&req_buf, "oid %s",
-@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
- int fd_out)
+@@ fetch-object-info.c: void fetch_object_info(const enum protocol_version version,
+ const int fd_out)
{
unsigned ask_size = 0;
+ unsigned ask_type = 0;
@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
size_t wanted;
results->nr = oids->nr;
-@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version, +@@ fetch-object-info.c: void fetch_object_info(const enum protocol_version version,
server_supports_feature("object-info", "size", 0))
ask_size = 1;
@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
break;
case protocol_v1:
case protocol_v0:
-@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version, +@@ fetch-object-info.c: void fetch_object_info(const enum protocol_version version,
case protocol_unknown_version:
BUG("unknown protocol version");
}
@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
for (size_t i = 0; i < wanted; i++) {
if (packet_reader_read(reader) != PACKET_READ_NORMAL) {
-@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version, +@@ fetch-object-info.c: void fetch_object_info(const enum protocol_version version,
die(_("object-info: duplicate 'size' attribute"));
size_index = (int)i;
CALLOC_ARRAY(results->sizes, results->nr);
@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
} else {
die(_("object-info: unknown attribute '%s'"),
reader->line);
-@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version, +@@ fetch-object-info.c: void fetch_object_info(const enum protocol_version version,
object_info_values.items[0].string, object_info_values.items[size_index + 1].string);
@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version,
string_list_clear(&object_info_values, 0);
}
-@@ fetch-object-info.c: void fetch_object_info(enum protocol_version version, +@@ fetch-object-info.c: void fetch_object_info(const enum protocol_version version,
void free_fetch_object_info_results(struct fetch_object_info_results *results) {
free(results->sizes);
8: 0964658eb6 = 9: 76f52c455b serve: advertise type capability 9: 77a193bf5c = 10: e5cebc08f0 cat-file: unify default format
---
base-commit: 5b2471720c93ee30e5764a19f3d3b3ae9ec9712a
change-id: 20260724-objecttype-support-ea1ef6941d07
PATCH v5 1/10 t5701: use test_file_size() to get the size of a file
The 'basics of object-info' test runs 'wc -c | xargs' twice to get the size of two.t. The pipe to xargs is only there to strip the blanks that some platforms pad the output of wc with.
Use the test_file_size() helper, which outputs the size directly, and store the result in a variable. Because 'git rev-parse two:two.t' is also run multiple times, store its output in a variable as well.
Storing them in variables outside the HERE-document has the added benefit of preserving their exit statuses.
--- t/t5701-git-serve.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
t/t5701-git-serve.sh +7 -4
@@ -344,20 +344,23 @@ test_expect_success 'unexpected lines are not allowed in fetch request' ' test_expect_success 'basics of object-info' ' test_config transfer.advertiseObjectInfo true && + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + test-tool pkt-line pack >in <<-EOF && command=object-info object-format=$(test_oid algo) 0001 size - oid $(git rev-parse two:two.t) - oid $(git rev-parse two:two.t) + oid $two_oid + oid $two_oid 0000 EOF cat >expect <<-EOF && size - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) + $two_oid $two_size + $two_oid $two_size 0000 EOF
Signature · 1 line
2.54.0
PATCH v5 2/10 fetch-object-info: detect malformed server responses
In reply to Pablo SabaterThe loop reading the object-info response stops as soon as the reader returns something other than PACKET_READ_NORMAL, or once it has read as many lines as we requested. Neither end is checked.
A server that answers with fewer objects leaves the end of the result arrays empty, and the caller trusts that every requested object was filled in. A server that answers with more leaves the extra packets unread. On stateless transports check_stateless_delimiter() notices, but on the others it passes unnoticed.
Check both limits by extracting the packet_reader_read() from the loop condition, so the loop no longer consumes the last packet (flush). If while looping the read is different from a PACKET_READ_NORMAL, die() meaning there are fewer objects than expected. After iterating, we only expect a flush, so if the last packet is not a flush, die().
--- fetch-object-info.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
fetch-object-info.c +10 -4
@@ -106,12 +106,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; -packet_reader_read (reader) ==PACKET_READ_NORMAL && - i < args->oids->nr; - i++) { +for (size_t i =0 ; i < args->oids->nr; i++) {struct string_list object_info_values =STRING_LIST_INIT_DUP ; +if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) +die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), + (uintmax_t )args->oids->nr, (uintmax_t )i); +string_list_split (&object_info_values, reader->line," " ,-1 ); if (strcmp(object_info_values.items[0].string,
@@ -150,6 +151,11 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arstring_list_clear (&object_info_values,0 ); } + +if (packet_reader_read (reader) !=PACKET_READ_FLUSH ) +die (_ ("object-info: expected flush after %" PRIuMAX" objects" ), + (uintmax_t )args->oids->nr); +check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return 0 ;
Signature · 1 line
2.54.0
PATCH v5 3/10 fetch-object-info: pass arguments directly instead of a struct
In reply to Pablo Sabaterstruct object_info_args groups three pointers that already live in the transport and are given to fetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests that the three belong together, when they are unrelated and end up being accessed as args->* independently.
Drop the struct and pass those parameters directly to fetch_object_info() and send_object_info_request(). This should have no change in behavior.
--- fetch-object-info.c | 53 +++++++++++++++++++++++++++++++---------------------- fetch-object-info.h | 17 ++++++++--------- transport.c | 11 +++++------ 3 files changed, 44 insertions(+), 37 deletions(-)
fetch-object-info.c +31 -22
@@ -9,20 +9,24 @@#include "string-list.h" /* Sends object-info command and its arguments into the request buffer. */ -static void send_object_info_request (const int fd_out,struct object_info_args *args) +static void send_object_info_request (const int fd_out, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options) {struct strbuf req_buf =STRBUF_INIT ; -write_command_and_capabilities (&req_buf,"object-info" , args->server_options); +write_command_and_capabilities (&req_buf,"object-info" , server_options); -if (unsorted_string_list_has_string (args->object_info_options,"size" )) +if (unsorted_string_list_has_string (object_info_options,"size" ))packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) +else if (object_info_options->nr)BUG ("only size should be in object_info_options" ); -if (args->oids) -for (size_t i =0 ; i < args->oids->nr; i++) -packet_buf_write (&req_buf,"oid %s" ,oid_to_hex (&args->oids->oid[i])); +if (oids) +for (size_t i =0 ; i < oids->nr; i++) +packet_buf_write (&req_buf,"oid %s" , +oid_to_hex (&oids->oid[i]));packet_buf_flush (&req_buf);if (write_in_full (fd_out, req_buf.buf, req_buf.len) <0 )
@@ -45,8 +49,12 @@ static int parse_object_size(const char *s, size_t *res)return 0 ; } -int fetch_object_info (const enum protocol_version version,struct object_info_args *args, -struct packet_reader *reader,struct object_info *object_info_data, +int fetch_object_info (const enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options, +struct packet_reader *reader, +struct object_info *object_info_data,const int stateless_rpc,const int fd_out) {int size_index =-1 ;
@@ -64,16 +72,17 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* because the number of options is a small known number (the * supported placeholders which currently are size and type). */ -for (int i = (int )args->object_info_options->nr -1 ; i >=0 ; i--) +for (int i = (int )object_info_options->nr -1 ; i >=0 ; i--)if (!server_supports_feature ("object-info" , - args->object_info_options->items[i].string,0 )) -unsorted_string_list_delete_item (args->object_info_options, i,0 ); + object_info_options->items[i].string,0 )) +unsorted_string_list_delete_item (object_info_options, i,0 );/* * Even if no options are left, we still send the oid so we get * at least an existence check. */ -send_object_info_request (fd_out, args); +send_object_info_request (fd_out, server_options, oids, + object_info_options);break ;case protocol_v1:case protocol_v0:
@@ -82,14 +91,14 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arBUG ("unknown protocol version" ); } -for (size_t i =0 ; i < args->object_info_options->nr; i++) { +for (size_t i =0 ; i < object_info_options->nr; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return -1 ; } -if (!unsorted_string_list_has_string (args->object_info_options, reader->line)) +if (!unsorted_string_list_has_string (object_info_options, reader->line))return -1 ;if (!strcmp (reader->line,"size" )) {
@@ -98,7 +107,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* is only size. No risk of overflow. */ size_index = (int )i; -for (size_t j =0 ; j < args->oids->nr; j++) +for (size_t j =0 ; j < oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); }else {
@@ -106,19 +115,19 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; i < args->oids->nr; i++) { +for (size_t i =0 ; i < oids->nr; i++) {struct string_list object_info_values =STRING_LIST_INIT_DUP ;if (packet_reader_read (reader) !=PACKET_READ_NORMAL )die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), - (uintmax_t )args->oids->nr, (uintmax_t )i); + (uintmax_t )oids->nr, (uintmax_t )i);string_list_split (&object_info_values, reader->line," " ,-1 );if (strcmp (object_info_values.items[0 ].string, -oid_to_hex (&args->oids->oid[i]))) +oid_to_hex (&oids->oid[i])))die (_ ("object-info: expected OID: %s, got %s" ), -oid_to_hex (&args->oids->oid[i]), +oid_to_hex (&oids->oid[i]), object_info_values.items[0 ].string);/*
@@ -138,7 +147,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* the server we expect the server to answer with the same * number of attributes requested. */ -if (args->object_info_options->nr +1 != object_info_values.nr) +if (object_info_options->nr +1 != object_info_values.nr)die ("object-info: unexpected number of attributes: %s" , reader->line);
@@ -154,7 +163,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arif (packet_reader_read (reader) !=PACKET_READ_FLUSH )die (_ ("object-info: expected flush after %" PRIuMAX" objects" ), - (uintmax_t )args->oids->nr); + (uintmax_t )oids->nr);check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
fetch-object-info.h +8 -9
@@ -4,22 +4,21 @@#include "pkt-line.h" #include "protocol.h" -struct object_info_args { -struct string_list *object_info_options; -const struct string_list *server_options; -struct oid_array *oids; -}; -struct object_info ; +struct oid_array ;/* * Sends git-cat-file object-info command into the request buf and read the * results from packets. * -* Modifies args->object_info_options, on return it contains only the supported +* Modifies object_info_options, on return it contains only the supported * options by the server. */ -int fetch_object_info (enum protocol_version version,struct object_info_args *args, -struct packet_reader *reader,struct object_info *object_info_data, +int fetch_object_info (enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options, +struct packet_reader *reader, +struct object_info *object_info_data,int stateless_rpc,int fd_out);#endif /* FETCH_OBJECT_INFO_H */
transport.c +5 -6
@@ -438,11 +438,6 @@ static int fetch_object_info_via_pack(struct transport *transport)int ret =0 ;struct git_transport_data *data = transport->data;struct packet_reader reader; -struct object_info_args args = {0 }; - - args.server_options = transport->server_options; - args.oids = transport->smart_options->object_info_oids; - args.object_info_options = transport->smart_options->object_info_options;connect_setup (transport,0 ); packet_reader_init(&reader, data->fd[0], NULL, 0,
@@ -453,7 +448,11 @@ static int fetch_object_info_via_pack(struct transport *transport) data->version =discover_version (&reader); transport->hash_algo = reader.hash_algo; - ret =fetch_object_info (data->version, &args, &reader, + ret =fetch_object_info (data->version, + transport->server_options, + transport->smart_options->object_info_oids, + transport->smart_options->object_info_options, + &reader, data->options.object_info_data, transport->stateless_rpc, data->fd[1 ]);
Signature · 1 line
2.54.0
PATCH v5 4/10 fetch-object-info: use dedicated struct for the results
In reply to Pablo Sabaterfetch_object_info() collects information about N objects, but it stores the results in an array of object_info. That struct holds the extended parameters of read_object_info() (The optional outputs the caller wants filled). Its pointers tell that function where to write the answers for a single object. object_info is not meant to be the final storage, and since fetch_object_info() does not call read_object_info(), there is no reason to use it. Using it means allocating one scalar per object per attribute just to have those pointers somewhere to point at.
Add struct fetch_object_info_results. The caller sets the wants_* flags to say what it is interested in, and fetch_object_info() allocates one array per attribute. A set wants_* flag means "asked for", while a non-NULL array means "available". The caller releases the arrays with free_fetch_object_info_results().
The object_info_options string list is no longer needed. Filtering against the server's advertisement now sets local ask_* flags, and send_object_info_request() turns those into the v2 protocol option strings. remote_atom_map[] existed only to map those strings back into atom names, so drop it and build remote_allowed_atoms from the result arrays.
Currently for wants_* and ask_* there is only the 'size' variant but a subsequent commit will add '*_type'.
free_object_info_contents() loses its only caller and is dropped.
Dropping the allow-list check makes the final else reachable from the wire, so die() instead of BUG(): an unknown attribute is the server's error, not ours.
--- builtin/cat-file.c | 59 +++++++++----------------------------- fetch-object-info.c | 81 ++++++++++++++++++++++++++--------------------------- fetch-object-info.h | 27 +++++++++++++----- object-file.c | 10 ------- odb.h | 3 -- transport.c | 3 +- transport.h | 5 ++-- 7 files changed, 77 insertions(+), 111 deletions(-)
builtin/cat-file.c +14 -45
@@ -31,6 +31,7 @@#include "alias.h" #include "remote.h" #include "transport.h" +#include "fetch-object-info.h" /* * Maximum length for a remote URL. While no universal standard exists,
@@ -681,9 +682,8 @@ static void batch_one_object(const char *obj_name,static int get_remote_info (int argc,const char **argv, -struct object_info **remote_object_info, -struct oid_array *object_info_oids, -struct string_list *object_info_options) +struct fetch_object_info_results *results, +struct oid_array *object_info_oids) {int retval =0 ;struct remote *remote =NULL ;
@@ -724,11 +724,9 @@ static int get_remote_info(int argc, goto cleanup; } -CALLOC_ARRAY (*remote_object_info, object_info_oids->nr); gtransport->smart_options->object_info_oids = object_info_oids; - gtransport->smart_options->object_info_options = object_info_options; - gtransport->smart_options->object_info_data = *remote_object_info; + gtransport->smart_options->object_info_results = results; retval =transport_fetch_object_info (gtransport); cleanup:transport_disconnect (gtransport);
@@ -816,21 +814,6 @@ static void parse_cmd_mailmap(struct batch_options *opt UNUSED,load_mailmap (); } -struct protocol_placeholder_entry { -const char *option; -const char *atom; -}; - -static const struct protocol_placeholder_entry remote_atom_map[] = { - {"size" ,"objectsize" }, - {"type" ,"objecttype" }, -/* -* Add new protocol options here. Even if the server doesn't support -* them the allow_list will drop them if the server doesn't advertise -* them. -*/ -}; -static void parse_cmd_remote_object_info (struct batch_options *opt,const char *line,struct strbuf *output,struct expand_data *data)
@@ -838,9 +821,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,int count;const char **argv;char *line_to_split; -struct object_info *remote_object_info =NULL ; +struct fetch_object_info_results results =FETCH_OBJECT_INFO_RESULTS_INIT ;struct oid_array object_info_oids =OID_ARRAY_INIT ; -struct string_list object_info_options =STRING_LIST_INIT_NODUP ;const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )
@@ -861,26 +843,21 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, MAX_ALLOWED_OBJ_LIMIT);if (data->info.sizep) -string_list_append (&object_info_options,"size" ); -if (data->info.typep) -string_list_append (&object_info_options,"type" ); + results.wants_size =1 ; -if (get_remote_info (count, argv, &remote_object_info, - &object_info_oids, &object_info_options)) +if (get_remote_info (count, argv, &results, &object_info_oids))die (_ ("failed to get object info from the remote: %s" ), argv[0 ]);string_list_clear (&data->remote_allowed_atoms,0 );string_list_append (&data->remote_allowed_atoms,"objectname" ); -for (size_t i =0 ; i <ARRAY_SIZE (remote_atom_map); i++) -if (unsorted_string_list_has_string (&object_info_options, remote_atom_map[i].option)) -string_list_append (&data->remote_allowed_atoms, - remote_atom_map[i].atom); +if (results.sizes) +string_list_append (&data->remote_allowed_atoms,"objectsize" ); data->skip_object_info =1 ; -for (size_t i =0 ; i < object_info_oids.nr; i++) { +for (size_t i =0 ; i < results.nr; i++) { data->oid = object_info_oids.oid[i]; -if (remote_object_info[i].unrecognized) { +if (results.unrecognized[i]) {report_object_status (opt,oid_to_hex (&data->oid), &data->oid,"missing" );continue ;
@@ -890,13 +867,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,* When reaching here, it means remote-object-info can retrieve * information from server without downloading them. */ -if (remote_object_info[i].sizep) { - data->size = *remote_object_info[i].sizep; - } - -if (remote_object_info[i].typep) { - data->type = *remote_object_info[i].typep; - } +if (results.sizes) + data->size = results.sizes[i]; opt->batch_mode =BATCH_MODE_INFO ; data->is_remote =1 ;
@@ -906,12 +878,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->skip_object_info =0 ; opt->format = saved_format; -for (size_t i =0 ; i < object_info_oids.nr; i++) -free_object_info_contents (&remote_object_info[i]); -string_list_clear (&object_info_options,0 ); +free_fetch_object_info_results (&results);free (line_to_split);free (argv); -free (remote_object_info);oid_array_clear (&object_info_oids); }
fetch-object-info.c +39 -42
147 lines not on this page · raw message
fetch-object-info.h +20 -7
@@ -4,21 +4,34 @@#include "pkt-line.h" #include "protocol.h" -struct object_info ; +struct fetch_object_info_results { +size_t *sizes; +uint8_t *unrecognized; +size_t nr; +unsigned wants_size:1 ; +}; + +#define FETCH_OBJECT_INFO_RESULTS_INIT { 0 } +struct oid_array ;/* -* Sends git-cat-file object-info command into the request buf and read the +* Sends git-cat-file object-info command into the request buf and reads the * results from packets. * -* Modifies object_info_options, on return it contains only the supported -* options by the server. +* The caller sets the wants_* flags in "results" to indicate which attributes +* it is interested in. On return, "results" holds one array per attribute that +* the server both advertised and answered with. An array left NULL means the +* attribute is not available. +* Release them with free_fetch_object_info_results(). */ int fetch_object_info (enum protocol_version version,const struct string_list *server_options,struct oid_array *oids, -struct string_list *object_info_options,struct packet_reader *reader, -struct object_info *object_info_data, -int stateless_rpc,int fd_out); +struct fetch_object_info_results *results, +int stateless_rpc, +int fd_out); + +void free_fetch_object_info_results (struct fetch_object_info_results *results);#endif /* FETCH_OBJECT_INFO_H */
object-file.c +0 -10
@@ -1340,13 +1340,3 @@ int odb_transaction_files_begin(struct odb_source *source,return 0 ; } - -void free_object_info_contents (struct object_info *object_info) -{ -if (!object_info) -return ; -free (object_info->typep); -free (object_info->sizep); -free (object_info->disk_sizep); -free (object_info->delta_base_oid); -}
odb.h +0 -3
@@ -637,7 +637,4 @@ void parse_alternates(const char *string,const char *relative_base, struct strvec *out); -/* Free pointers inside of object_info, but not object_info itself */ -void free_object_info_contents (struct object_info *object_info); -#endif /* ODB_H */
transport.c +1 -2
@@ -451,9 +451,8 @@ static int fetch_object_info_via_pack(struct transport *transport) ret =fetch_object_info (data->version, transport->server_options, transport->smart_options->object_info_oids, - transport->smart_options->object_info_options, &reader, - data->options.object_info_data, + data->options.object_info_results, transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);
transport.h +3 -2
@@ -7,6 +7,8 @@#include "string-list.h" #include "connect.h" +struct fetch_object_info_results ; +struct git_transport_options {unsigned thin :1 ;unsigned keep :1 ;
@@ -57,8 +59,7 @@ struct git_transport_options {struct oidset *acked_commits;struct oid_array *object_info_oids; -struct object_info *object_info_data; -struct string_list *object_info_options; +struct fetch_object_info_results *object_info_results; };enum transport_family {
Signature · 1 line
2.54.0
PATCH v5 5/10 fetch-object-info: die() on the remaining error path
In reply to Pablo SabaterEvery failure in fetch_object_info() dies except one: a short read while parsing the attribute lines returns -1. That -1 is then passed through fetch_object_info_via_pack() and get_remote_info() up to cat-file, only to die() with a generic message.
Die in fetch_object_info() instead, consistently with the rest of its error paths, and make fetch_object_info() void.
--- fetch-object-info.c | 19 +++++++++---------- fetch-object-info.h | 14 +++++++------- transport.c | 12 ++++++------ 3 files changed, 22 insertions(+), 23 deletions(-)
fetch-object-info.c +9 -10
@@ -47,13 +47,13 @@ static int parse_object_size(const char *s, size_t *res)return 0 ; } -int fetch_object_info (const enum protocol_version version, -const struct string_list *server_options, -struct oid_array *oids, -struct packet_reader *reader, -struct fetch_object_info_results *results, -const int stateless_rpc, -const int fd_out) +void fetch_object_info (const enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct packet_reader *reader, +struct fetch_object_info_results *results, +const int stateless_rpc, +const int fd_out) {unsigned ask_size =0 ;int size_index =-1 ;
@@ -89,7 +89,8 @@ int fetch_object_info(const enum protocol_version version,if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" ); -return -1 ; +die (_ ("object-info: expected %" PRIuMAX" attributes, got %" PRIuMAX), + (uintmax_t )wanted, (uintmax_t )i); }if (!strcmp (reader->line,"size" )) {
@@ -156,8 +157,6 @@ int fetch_object_info(const enum protocol_version version, (uintmax_t)oids->nr);check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" ); - -return 0 ; }void free_fetch_object_info_results (struct fetch_object_info_results *results)
fetch-object-info.h +7 -7
@@ -24,13 +24,13 @@ struct oid_array;* attribute is not available. * Release them with free_fetch_object_info_results(). */ -int fetch_object_info (enum protocol_version version, -const struct string_list *server_options, -struct oid_array *oids, -struct packet_reader *reader, -struct fetch_object_info_results *results, -int stateless_rpc, -int fd_out); +void fetch_object_info (enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct packet_reader *reader, +struct fetch_object_info_results *results, +int stateless_rpc, +int fd_out);void free_fetch_object_info_results (struct fetch_object_info_results *results);
transport.c +6 -6
@@ -448,12 +448,12 @@ static int fetch_object_info_via_pack(struct transport *transport) data->version =discover_version (&reader); transport->hash_algo = reader.hash_algo; - ret =fetch_object_info (data->version, - transport->server_options, - transport->smart_options->object_info_oids, - &reader, - data->options.object_info_results, - transport->stateless_rpc, data->fd[1 ]); +fetch_object_info (data->version, + transport->server_options, + transport->smart_options->object_info_oids, + &reader, + data->options.object_info_results, + transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);if (data->fd[1 ] >=0 )
Signature · 1 line
2.54.0
PATCH v5 6/10 transport: drop remote object-info fields from transport struct
In reply to Pablo SabaterFrom: Jeff King <[email protected]>
A remote object-info request needs three things: the transport for contacting the remote, the list of oids to request, and a place to store the output.
Rather than take these as function parameters, we take only the transport object, and expect the caller to have placed the other two into special fields in the transport struct. But this doesn't make much sense. The set of oids and results are really only valid for one request. There is no reason the transport would need to hang on to them outside of the single function call.
Even though we save a few lines passing the parameters around through the various vtable functions, the result is harder to understand (for example, who is responsible for cleaning up results, and when should it happen?). It also opens up the possibility of a subtle bug. A caller is likely to point those fields to stack variables which could go out of scope, and the transport struct would be left holding invalid pointers.
This is mostly harmless now, as we disconnect the transport immediately after the sole caller of transport_fetch_object_info(). But conceptually we could keep the transport open and make multiple fetch calls (and reuse the same connection to the helper, to a remote HTTP server, and so on).
So let's pull these out of the struct and pass them as function parameters. It's a little more verbose, but I think more clearly illustrates the intent. I've also tweaked a few function signatures to mark the input oid array as const, since it is purely an input to the function.
--- builtin/cat-file.c | 6 ++---- fetch-object-info.c | 4 ++-- fetch-object-info.h | 2 +- transport-helper.c | 7 +++++-- transport-internal.h | 4 +++- transport.c | 14 +++++++++----- transport.h | 7 +++---- 7 files changed, 25 insertions(+), 19 deletions(-)
builtin/cat-file.c +2 -4
@@ -724,10 +724,8 @@ static int get_remote_info(int argc, goto cleanup; } - gtransport->smart_options->object_info_oids = object_info_oids; - - gtransport->smart_options->object_info_results = results; - retval =transport_fetch_object_info (gtransport); + retval =transport_fetch_object_info (gtransport, object_info_oids, + results); cleanup:transport_disconnect (gtransport);return retval;
fetch-object-info.c +2 -2
@@ -11,7 +11,7 @@/* Sends object-info command and its arguments into the request buffer. */ static void send_object_info_request (const int fd_out,const struct string_list *server_options, -struct oid_array *oids, +const struct oid_array *oids,unsigned ask_size) {struct strbuf req_buf =STRBUF_INIT ;
@@ -49,7 +49,7 @@ static int parse_object_size(const char *s, size_t *res) void fetch_object_info(const enum protocol_version version, const struct string_list *server_options, - struct oid_array *oids, + const struct oid_array *oids, struct packet_reader *reader, struct fetch_object_info_results *results, const int stateless_rpc,
fetch-object-info.h +1 -1
@@ -26,7 +26,7 @@ struct oid_array;*/ void fetch_object_info(enum protocol_version version, const struct string_list *server_options, - struct oid_array *oids, + const struct oid_array *oids, struct packet_reader *reader, struct fetch_object_info_results *results, int stateless_rpc,
transport-helper.c +5 -2
@@ -784,11 +784,14 @@ static int fetch_refs(struct transport *transport,return -1 ; } -static int fetch_object_info_helper (struct transport *transport) +static int fetch_object_info_helper (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {get_helper (transport);if (process_connect (transport,0 )) -return transport->vtable->fetch_object_info (transport); +return transport->vtable->fetch_object_info (transport, oids, + results);die (_ ("object-info requires protocol v2" )); }
transport-internal.h +3 -1
@@ -51,7 +51,9 @@ struct transport_vtable {* * Uses object-info capability of v2 protocol. */ -int (*fetch_object_info)(struct transport *transport); +int (*fetch_object_info)(struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results);/** * Push the objects and refs. Send the necessary objects, and
transport.c +9 -5
@@ -433,7 +433,9 @@ static int get_bundle_uri(struct transport *transport) transport->bundles, stateless_rpc); } -static int fetch_object_info_via_pack (struct transport *transport) +static int fetch_object_info_via_pack (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -450,9 +452,9 @@ static int fetch_object_info_via_pack(struct transport *transport)fetch_object_info (data->version, transport->server_options, - transport->smart_options->object_info_oids, + oids, &reader, - data->options.object_info_results, + results, transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);
@@ -465,11 +467,13 @@ static int fetch_object_info_via_pack(struct transport *transport)return ret; } -int transport_fetch_object_info (struct transport *transport) +int transport_fetch_object_info (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {if (!transport->vtable->fetch_object_info)die (_ ("remote does not support object-info" )); -return transport->vtable->fetch_object_info (transport); +return transport->vtable->fetch_object_info (transport, oids, results); } static int fetch_refs_via_pack(struct transport *transport,
transport.h +3 -4
@@ -57,9 +57,6 @@ struct git_transport_options {* common commits to this oidset instead of fetching any packfiles. */ struct oidset *acked_commits; - -struct oid_array *object_info_oids; -struct fetch_object_info_results *object_info_results; };enum transport_family {
@@ -317,7 +314,9 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs);/* * Fetch the object info from remote */ -int transport_fetch_object_info (struct transport *transport); +int transport_fetch_object_info (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results);/* * If this flag is set, unlocking will avoid to call non-async-signal-safe
Signature · 1 line
2.54.0
PATCH v5 7/10 protocol-caps: add type support to object-info
In reply to Pablo SabaterTeach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While touching send_info(), wrap an over-long line and fix the bit field style of requested_info.size.
--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 30 ++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 3 deletions(-)
protocol-caps.c +18 -3
@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/*
@@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string; +enum object_type object_type;struct object_id oid;size_t object_size;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) {packet_writer_error ( writer, -"object-info: protocol error, expected to get oid, not '%s'" , +"object-info: protocol error, expected to get " +"oid, not '%s'" , oid_str);continue ; }
@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }
@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); + write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);
@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;
t/t5701-git-serve.sh +30 -0
@@ -369,6 +369,36 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'object-info supports type' ' + test_config transfer.advertiseObjectInfo true && + + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + + test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type + oid $two_oid + oid $two_oid + 0000 + EOF + + cat >expect <<-EOF && + size + type + $two_oid $two_size blob + $two_oid $two_size blob + 0000 + EOF + + test-tool serve-v2 --stateless-rpc <in >out && + test-tool pkt-line unpack <out >actual && + test_cmp expect actual +' + test_expect_success 'bare OID request' ' test_config transfer.advertiseObjectInfo true &&
Signature · 1 line
2.54.0
PATCH v5 8/10 fetch-object-info: parse type from server response
In reply to Pablo SabaterThe server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the type capability.
--- builtin/cat-file.c | 7 +++++++ fetch-object-info.c | 38 +++++++++++++++++++++++++++++++++++--- fetch-object-info.h | 3 +++ 3 files changed, 45 insertions(+), 3 deletions(-)
builtin/cat-file.c +7 -0
@@ -842,6 +842,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,if (data->info.sizep) results.wants_size =1 ; +if (data->info.typep) + results.wants_type =1 ;if (get_remote_info (count, argv, &results, &object_info_oids))die (_ ("failed to get object info from the remote: %s" ), argv[0 ]);
@@ -850,6 +852,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,string_list_append (&data->remote_allowed_atoms,"objectname" );if (results.sizes)string_list_append (&data->remote_allowed_atoms,"objectsize" ); +if (results.types) +string_list_append (&data->remote_allowed_atoms,"objecttype" ); data->skip_object_info =1 ;for (size_t i =0 ; i < results.nr; i++) {
@@ -868,6 +872,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,if (results.sizes) data->size = results.sizes[i]; +if (results.types) + data->type = results.types[i]; + opt->batch_mode =BATCH_MODE_INFO ; data->is_remote =1 ;batch_object_write (argv[i +1 ], output, opt, data,NULL ,0 );
fetch-object-info.c +35 -3
@@ -1,6 +1,7 @@#include "git-compat-util.h" #include "gettext.h" #include "hex.h" +#include "object.h" #include "pkt-line.h" #include "connect.h" #include "oid-array.h"
@@ -12,7 +13,8 @@static void send_object_info_request (const int fd_out,const struct string_list *server_options,const struct oid_array *oids, -unsigned ask_size) +unsigned ask_size, +unsigned ask_type) {struct strbuf req_buf =STRBUF_INIT ;
@@ -21,6 +23,9 @@ static void send_object_info_request(const int fd_out,if (ask_size)packet_buf_write (&req_buf,"size" ); +if (ask_type) +packet_buf_write (&req_buf,"type" ); +if (oids)for (size_t i =0 ; i < oids->nr; i++) packet_buf_write(&req_buf, "oid %s",
@@ -56,7 +61,9 @@ void fetch_object_info(const enum protocol_version version, const int fd_out) {unsigned ask_size =0 ; +unsigned ask_type =0 ;int size_index =-1 ; +int type_index =-1 ;size_t wanted; results->nr = oids->nr;
@@ -71,11 +78,16 @@ void fetch_object_info(const enum protocol_version version, server_supports_feature("object-info", "size", 0)) ask_size =1 ; +if (results->wants_type && +server_supports_feature ("object-info" ,"type" ,0 )) + ask_type =1 ; +/* * Even if no options are left, we still send the oid so we get * at least an existence check. */ -send_object_info_request (fd_out, server_options, oids, ask_size); +send_object_info_request (fd_out, server_options, oids, ask_size, + ask_type);break ;case protocol_v1:case protocol_v0:
@@ -83,7 +95,7 @@ void fetch_object_info(const enum protocol_version version,case protocol_unknown_version:BUG ("unknown protocol version" ); } - wanted = ask_size; + wanted = ask_size + ask_type;for (size_t i =0 ; i < wanted; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {
@@ -100,6 +112,13 @@ void fetch_object_info(const enum protocol_version version,die (_ ("object-info: duplicate 'size' attribute" )); size_index = (int )i;CALLOC_ARRAY (results->sizes, results->nr); + }else if (!strcmp (reader->line,"type" )) { +if (!ask_type) +die (_ ("object-info: unrequested 'type' attribute" )); +if (results->types) +die (_ ("object-info: duplicate 'type' attribute" )); + type_index = (int )i; +CALLOC_ARRAY (results->types, results->nr); }else {die (_ ("object-info: unknown attribute '%s'" ), reader->line);
@@ -149,6 +168,18 @@ void fetch_object_info(const enum protocol_version version, object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (results->types) { +const char *type_str = + object_info_values.items[type_index +1 ].string; +int type =type_from_string_gently (type_str,-1 ,1 ); + +if (type <0 ) +die (_ ("object-info: object %s has invalid type '%s'" ), + object_info_values.items[0 ].string, type_str); + + results->types[i] = type; + } +string_list_clear (&object_info_values,0 ); }
@@ -162,6 +193,7 @@ void fetch_object_info(const enum protocol_version version,void free_fetch_object_info_results (struct fetch_object_info_results *results) {free (results->sizes); +free (results->types);free (results->unrecognized);memset (results,0 ,sizeof (*results)); }
fetch-object-info.h +3 -0
@@ -1,14 +1,17 @@#ifndef FETCH_OBJECT_INFO_H #define FETCH_OBJECT_INFO_H +#include "object.h" #include "pkt-line.h" #include "protocol.h" struct fetch_object_info_results {size_t *sizes; +enum object_type *types;uint8_t *unrecognized;size_t nr;unsigned wants_size:1 ; +unsigned wants_type:1 ; };#define FETCH_OBJECT_INFO_RESULTS_INIT { 0 }
Signature · 1 line
2.54.0
PATCH v5 9/10 serve: advertise type capability
In reply to Pablo SabaterThe server and the client can handle type requests but the client won't ask for it until the server advertises it.
Add type to the advertised capabilities so the client knows that it can request it.
--- serve.c | 4 ++-- t/t1017-cat-file-remote-object-info.sh | 26 ++++++++++++++++++++++---- 2 files changed, 24 insertions(+), 6 deletions(-)
serve.c +2 -2
@@ -97,9 +97,9 @@ static int object_info_advertise(struct repository *r, struct strbuf *value)/* disabled by default */ advertise_object_info =0 ; } -/* Currently only size is supported */ +/* Currently only size and type are supported */ if (value && advertise_object_info) -strbuf_addstr (value,"size" ); +strbuf_addstr (value,"size type" );return advertise_object_info; }
t/t1017-cat-file-remote-object-info.sh +22 -4
@@ -7,6 +7,7 @@ test_description='git cat-file --batch-command with remote-object-info command' hello_content="Hello World" hello_size=$(strlen "$hello_content") +hello_type="blob" hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) hello_short_oid=$(git rev-parse --short "$hello_oid")
@@ -19,6 +20,7 @@ unstored_oid=$(echo_without_newline "$unstored_content" | git hash-object --stdi # file name is hello, which is 5 characters # a space is 1 character and a null is 1 character tree_size=$(($(test_oid rawsz) + 13)) +tree_type="tree" commit_message="Initial commit"
@@ -31,6 +33,7 @@ commit_message="Initial commit" # An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`, # to get 177, 2. then deduct 40 hex characters to get 137 commit_size=$(($(test_oid hexsz) + 137)) +commit_type="commit" tag_header_without_oid="type blob tag hellotag
@@ -44,6 +47,7 @@ $tag_description" tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w) tag_size=$(strlen "$tag_content") +tag_type="tag" set_transport_variables () { hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin)
@@ -256,14 +260,12 @@ test_expect_success 'remote-object-info does not die on missing oid like info' ' ) ' -# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo "$hello_type" >expect && git cat-file --batch-command="%(objecttype)" >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOF
@@ -271,6 +273,22 @@ test_expect_success 'unsupported placeholder on remote returns empty string' ' ) ' +test_expect_success 'unsupported placeholders on remote return empty string' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + fmt="%(objectmode) %(objectsize:disk) %(rest) %(deltabase)" && + + # The hardcoded SPs between the atoms are respected. + echo " " >expect && + git cat-file --batch-command="$fmt" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid + EOF + test_cmp expect actual + ) +' + test_expect_success 'requesting only objectname echoes back' ' ( set_transport_variables "$daemon_parent" &&
Signature · 1 line
2.54.0
PATCH v5 10/10 cat-file: unify default format
In reply to Pablo Sabater%(objecttype) is supported both by the client and by the server. Change the temporary default format to the unified version that the other commands use.
Update documentation to remove %(objecttype) from the caveats of remote-object-info and show %(objecttype) support.
Now that type is supported and the default format unified, update the tests to expect the new default format.
--- Documentation/git-cat-file.adoc | 17 +++++------ Documentation/gitprotocol-v2.adoc | 18 +++++++++--- builtin/cat-file.c | 7 ----- t/t1017-cat-file-remote-object-info.sh | 52 +++++++++++++++++----------------- 4 files changed, 47 insertions(+), 47 deletions(-)
Documentation/git-cat-file.adoc +7 -10
@@ -348,15 +348,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. -The command `remote-object-info` only supports the `%(objectname)` and -`%(objectsize)` placeholders. See `CAVEATS` below for more information. +The command `remote-object-info` only supports the `%(objectname)`, +`%(objectsize)` and `%(objecttype)` placeholders. See `CAVEATS` below for more +information. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`, except for `remote-object-info` commands which -use `%(objectname) %(objectsize)` because `%(objecttype)` is not supported yet. - -WARNING: When "%(objecttype)" is supported, the default format WILL be unified, -so DO NOT RELY on the current default format to stay the same!!! +%(objecttype) %(objectsize)`. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -453,9 +450,9 @@ scripting purposes. CAVEATS ------- -Note that only `%(objectname)` and `%(objectsize)` are currently -supported by the `remote-object-info` command. Using any other placeholder in -the format string will return an empty string in its position. +Note that only `%(objectname)`, `%(objectsize)` and `%(objecttype)` are +currently supported by the `remote-object-info` command. Using any other +placeholder in the format string will return an empty string in its position. Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are
Documentation/gitprotocol-v2.adoc +14 -4
@@ -558,14 +558,17 @@ object-info `object-info` is the command to retrieve information about one or more objects. Its main purpose is to allow a client to make decisions based on this -information without having to fully fetch objects. Object size is the only -information that is currently supported. +information without having to fully fetch objects. Currently only object size +and type are supported. An `object-info` request takes the following arguments: size Requests size information to be returned for each listed object id. + type + Requests type information to be returned for each listed object id. + oid <oid> Indicates to the server an object which the client wants to obtain information for. They must be full OIDs.
@@ -580,11 +583,18 @@ space. info = *PKT-LINE(attr LF) *PKT-LINE(obj-info LF) - attr = "size" + attr = "size" | "type" obj-size = 1*DIGIT - obj-info = obj-id [SP [obj-size]] + obj-type = "blob" | "tree" | "commit" | "tag" + + obj-val = obj-size | obj-type + + obj-info = obj-id [SP [obj-val *(SP obj-val)]] + +The values in `obj-info` appear in the same order as the corresponding `attr` +lines, with exactly one value per requested attribute. If the server does not recognize the OID, the response will be `<oid> SP` regardless of the number of attributes requested.
builtin/cat-file.c +0 -7
@@ -821,15 +821,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,char *line_to_split;struct fetch_object_info_results results =FETCH_OBJECT_INFO_RESULTS_INIT ;struct oid_array object_info_oids =OID_ARRAY_INIT ; -const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )die (_ ("remote-object-info command too long" )); -/* -* TODO: Use the default format once %(objecttype) is supported. -*/ -if (!opt->format) - opt->format ="%(objectname) %(objectsize)" ; line_to_split =xstrdup (line); count =split_cmdline (line_to_split, &argv);
@@ -881,7 +875,6 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->is_remote =0 ; } data->skip_object_info =0 ; - opt->format = saved_format;free_fetch_object_info_results (&results);free (line_to_split);
t/t1017-cat-file-remote-object-info.sh +26 -26
100 lines not on this page · raw message
Signature · 1 line
2.54.0
I just realised I posted the wrong CI, and this v5 fails static analysis.
I’ll send v6 with the fix.
Apologies,
Pablo
Quoted 9 lines · Pablo Sabater
Given that, I think that it could be a good idea to have both, if a user foresees that he's only going to make one 'remote-object-info' command he can write it as it is now:
remote-object-info <remote> objA objB
But if a user foresees that he will have to make multiple ones, we can make what you suggested:
Quoted 8 lines
Quoted 3 lines
remote https://example.com/foo.git
remote-object-info objA objB objC...
remote-object-info objX objY objZWe would have to make the remote optional, if there's no remote
die(), etc. We would also have to tell apart a remote from an OID in the first argument, but full OIDs and remote URLs are not very similar so that should not be hard haha.
Hmm, I dunno. Making it optional just seems to introduce more complications. The value of a separate "remote" is that it makes things simpler (IMHO).
So maybe it is best to leave it as-is:
remote-object-info <remote> <objs...>
and then let cat-file handle it under the hood as an optimization, leaving the transport intact between calls, in expectation of another possible request.
It does mean that the remote may sit "open" if no other call arrives. In practice that's probably not a big deal. In most cases we'd either make more requests or hang up the batch-command session soon after. And for stateless transports, the connection naturally times out (e.g., even a persistent HTTP connection will hang up after a few seconds). So it would only be things like git-over-ssh where we keep the ssh session open waiting for more commands. That seems OK to me.
Quoted 3 lines
I do like the idea, but I see it more as a follow-up series after this one, as the topic of this series is type support.
Also, I'm biased as I have little time before my deadline ends.
I think we can leave it in the state your series has it, and then later come back and do the caching optimization. If we're not changing the protocol, then those changes are invisible to the caller, so there's no rush.
-Peff
v6 cover cat-file: extend remote-object-info to support %(objecttype)
In reply to Pablo SabaterNeither the client nor the server knows how to handle type. This series continues the work for git cat-file --batch-command extending remote-object-info to support "%(objecttype)" end to end. It is based on its predecessor series "cat-file: add remote-object-info to batch-command" [1] merged into master.
Type is the last property that is identical on the server and on the client once fetched.
Whether to support more metadata such as: - objectsize:disk
- deltabase
- objectmode (needs context)
should be discussed, unlike size and type, the rest depend on how things are packed on the server and on what the client already has, so they cannot reliably match local once fetched. IMO they are not worth supporting, I can't find a use case for them.
To add a new placeholder, follow the steps in this series and add its wants_* flag to struct fetch_object_info_results.
Patches 1-6 are preparatory. They don't change what the command does:
- [1/10] is a test cleanup.
- [2/10] fixes a possible bug in case of a malformed response.
- [3/10] and [4/10] refactor how the object data is stored and handled. The why about this refactor comes from [2].
- [5/10] drops the last error return left in fetch_object_info().
- [6/10] drops the remote object-info fields from the transport struct.
Patches 7-10 are the actual objecttype support:
- [7/10] teaches the server to answer type.
- [8/10] teaches the client to parse it.
- [9/10] advertises the capability so the client can start asking it.
- [10/10] unifies the default format.
Note that *_type appears from [8/10]. Before this series, "type" is already listed in the client's allow-list, but the prep patches drop that list, so the client knows nothing about type until [8/10] adds it back.
I apologise for the spam as I've sent v5 and v6 very close together.
Github CI v6: https://github.com/pabloosabaterr/git/actions/runs/31227956862
Sorry for the inconveniences.
[1]: https://lore.kernel.org/git/[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/
---
Changes in v6:
- Fixed static analysis CI fail with two forward declarations.
- Fixed s/%"PRIuMAX"/%" PRIuMAX "/
- Link to v5: https://lore.kernel.org/git/[email protected]
Changes in v5:
- New [6/10] by Jeff King: drop the remote object-info fields from the
transport struct and pass them as function parameters instead.
- [5/10]: keep the const qualifiers on parameters of fetch_object_info(),
matching how the signature was.
- Link to v4: https://lore.kernel.org/git/[email protected]/
Changes in v4:
- Rebased onto master now that ps/cat-file-remote-object-info was merged.
- Dropped the Based-on trailer.
- Reworded [1/9] so it shows that we also benefit from preserving the
exit status.
- Changed [2/9] to also check if a response returns more packets.
- New [5/9]: fetch_object_info() dies on its last error return and
becomes void.
- Moved the introduction of ask_type, wants_type to [7/9] instead of
[4/9].
- Link to v3: https://lore.kernel.org/git/[email protected]/
Changes in v3:
- 1/8 (was 2/6): reword the commit message to explain that "wc -c | xargs"
only strips the padding some platforms add.
- New 2/8: die if the server sends fewer object-info lines than
requested OIDs. Preexisting bug.
- New 3/8: drop struct object_info_args and pass its members directly to fetch_object_info().
- New 4/8: replace struct object_info with struct
fetch_object_info_results to hold the results. This also drops the object_info_options string list, remote_atom_map[] and free_object_info_contents().
- Dropped 1/6, which replaced the hardcoded "size" in
send_object_info_request() with a loop over object_info_options, as
that list no longer reaches it.
- 6/8: follows the new result struct, so it only adds the type array.
- 7/8: quote $hello_type in the test.
- Link to v2: https://lore.kernel.org/git/[email protected]/
Changes in v2:
- Changed t5701 test to use test_file_size.
- Changed the for loop to use for_each_string_list_item().
- Moved "fetch-object-info: request all supported options dynamically"
patch to be the first one as it is prep patch.
- Typos and nits.
- Restored the test mixing info and remote-object-info, adapted to the
unified format.
- Link to v1: https://lore.kernel.org/git/[email protected]/
---
Jeff King (1):
transport: drop remote object-info fields from transport struct
Pablo Sabater (9):
t5701: use test_file_size() to get the size of a file
fetch-object-info: detect malformed server responses
fetch-object-info: pass arguments directly instead of a struct
fetch-object-info: use dedicated struct for the results
fetch-object-info: die() on the remaining error path
protocol-caps: add type support to object-info
fetch-object-info: parse type from server response
serve: advertise type capability
cat-file: unify default format
Documentation/git-cat-file.adoc | 17 ++-- Documentation/gitprotocol-v2.adoc | 18 +++- builtin/cat-file.c | 73 +++++----------- fetch-object-info.c | 149 +++++++++++++++++++++------------ fetch-object-info.h | 37 +++++--- object-file.c | 10 --- odb.h | 3 - protocol-caps.c | 21 ++++- serve.c | 4 +- t/t1017-cat-file-remote-object-info.sh | 78 ++++++++++------- t/t5701-git-serve.sh | 41 ++++++++- transport-helper.c | 7 +- transport-internal.h | 6 +- transport.c | 24 +++--- transport.h | 10 +-- 15 files changed, 296 insertions(+), 202 deletions(-)
Range-diff versus v5:
1: 77b048574a = 1: 99112fd453 t5701: use test_file_size() to get the size of a file
2: 892e149f11 ! 2: e18da979aa fetch-object-info: detect malformed server responses
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
}
+
+ if (packet_reader_read(reader) != PACKET_READ_FLUSH)
-+ die(_("object-info: expected flush after %"PRIuMAX" objects"),
++ die(_("object-info: expected flush after %" PRIuMAX " objects"),
+ (uintmax_t)args->oids->nr);
+
check_stateless_delimiter(stateless_rpc, reader, "stateless delimiter expected"); 3: 05f4d26560 ! 3: 8b4f6913c1 fetch-object-info: pass arguments directly instead of a struct
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version,
@@ fetch-object-info.c: int fetch_object_info(const enum protocol_version version, struct object_info_ar
if (packet_reader_read(reader) != PACKET_READ_FLUSH)
- die(_("object-info: expected flush after %"PRIuMAX" objects"),
+ die(_("object-info: expected flush after %" PRIuMAX " objects"),
- (uintmax_t)args->oids->nr);
+ (uintmax_t)oids->nr);
4: 832639ae1a = 4: a81e1e3c1d fetch-object-info: use dedicated struct for the results 5: ffac30cef5 = 5: a79f9957b4 fetch-object-info: die() on the remaining error path 6: 748adf35fb ! 6: f14f3b0a2a transport: drop remote object-info fields from transport struct
@@ transport-helper.c: static int fetch_refs(struct transport *transport,
}
## transport-internal.h ##
+@@ transport-internal.h: struct ref;
+ struct transport;
+ struct strvec;
+ struct transport_ls_refs_options; ++struct oid_array;
++struct fetch_object_info_results;
+
+ struct transport_vtable {
+ /**
@@ transport-internal.h: struct transport_vtable {
*
* Uses object-info capability of v2 protocol.
7: c7ab410aef = 7: c51d8ae137 protocol-caps: add type support to object-info 8: 59da95f82c = 8: 1a9b320f36 fetch-object-info: parse type from server response 9: b249e9e16c = 9: 394bdd0885 serve: advertise type capability 10: f8221fc1de = 10: 595b80c28d cat-file: unify default format
---
base-commit: 5b2471720c93ee30e5764a19f3d3b3ae9ec9712a
change-id: 20260724-objecttype-support-ea1ef6941d07
PATCH v6 1/10 t5701: use test_file_size() to get the size of a file
The 'basics of object-info' test runs 'wc -c | xargs' twice to get the size of two.t. The pipe to xargs is only there to strip the blanks that some platforms pad the output of wc with.
Use the test_file_size() helper, which outputs the size directly, and store the result in a variable. Because 'git rev-parse two:two.t' is also run multiple times, store its output in a variable as well.
Storing them in variables outside the HERE-document has the added benefit of preserving their exit statuses.
--- t/t5701-git-serve.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
t/t5701-git-serve.sh +7 -4
@@ -344,20 +344,23 @@ test_expect_success 'unexpected lines are not allowed in fetch request' ' test_expect_success 'basics of object-info' ' test_config transfer.advertiseObjectInfo true && + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + test-tool pkt-line pack >in <<-EOF && command=object-info object-format=$(test_oid algo) 0001 size - oid $(git rev-parse two:two.t) - oid $(git rev-parse two:two.t) + oid $two_oid + oid $two_oid 0000 EOF cat >expect <<-EOF && size - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) - $(git rev-parse two:two.t) $(wc -c <two.t | xargs) + $two_oid $two_size + $two_oid $two_size 0000 EOF
Signature · 1 line
2.54.0
PATCH v6 2/10 fetch-object-info: detect malformed server responses
In reply to Pablo SabaterThe loop reading the object-info response stops as soon as the reader returns something other than PACKET_READ_NORMAL, or once it has read as many lines as we requested. Neither end is checked.
A server that answers with fewer objects leaves the end of the result arrays empty, and the caller trusts that every requested object was filled in. A server that answers with more leaves the extra packets unread. On stateless transports check_stateless_delimiter() notices, but on the others it passes unnoticed.
Check both limits by extracting the packet_reader_read() from the loop condition, so the loop no longer consumes the last packet (flush). If while looping the read is different from a PACKET_READ_NORMAL, die() meaning there are fewer objects than expected. After iterating, we only expect a flush, so if the last packet is not a flush, die().
--- fetch-object-info.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
fetch-object-info.c +10 -4
@@ -106,12 +106,13 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; -packet_reader_read (reader) ==PACKET_READ_NORMAL && - i < args->oids->nr; - i++) { +for (size_t i =0 ; i < args->oids->nr; i++) {struct string_list object_info_values =STRING_LIST_INIT_DUP ; +if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) +die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), + (uintmax_t )args->oids->nr, (uintmax_t )i); +string_list_split (&object_info_values, reader->line," " ,-1 ); if (strcmp(object_info_values.items[0].string,
@@ -150,6 +151,11 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arstring_list_clear (&object_info_values,0 ); } + +if (packet_reader_read (reader) !=PACKET_READ_FLUSH ) +die (_ ("object-info: expected flush after %" PRIuMAX" objects" ), + (uintmax_t )args->oids->nr); +check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return 0 ;
Signature · 1 line
2.54.0
PATCH v6 3/10 fetch-object-info: pass arguments directly instead of a struct
In reply to Pablo Sabaterstruct object_info_args groups three pointers that already live in the transport and are given to fetch_object_info().
Grouping them into a struct reduces the number of parameters, but it suggests that the three belong together, when they are unrelated and end up being accessed as args->* independently.
Drop the struct and pass those parameters directly to fetch_object_info() and send_object_info_request(). This should have no change in behavior.
--- fetch-object-info.c | 53 +++++++++++++++++++++++++++++++---------------------- fetch-object-info.h | 17 ++++++++--------- transport.c | 11 +++++------ 3 files changed, 44 insertions(+), 37 deletions(-)
fetch-object-info.c +31 -22
@@ -9,20 +9,24 @@#include "string-list.h" /* Sends object-info command and its arguments into the request buffer. */ -static void send_object_info_request (const int fd_out,struct object_info_args *args) +static void send_object_info_request (const int fd_out, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options) {struct strbuf req_buf =STRBUF_INIT ; -write_command_and_capabilities (&req_buf,"object-info" , args->server_options); +write_command_and_capabilities (&req_buf,"object-info" , server_options); -if (unsorted_string_list_has_string (args->object_info_options,"size" )) +if (unsorted_string_list_has_string (object_info_options,"size" ))packet_buf_write (&req_buf,"size" ); -else if (args->object_info_options->nr) +else if (object_info_options->nr)BUG ("only size should be in object_info_options" ); -if (args->oids) -for (size_t i =0 ; i < args->oids->nr; i++) -packet_buf_write (&req_buf,"oid %s" ,oid_to_hex (&args->oids->oid[i])); +if (oids) +for (size_t i =0 ; i < oids->nr; i++) +packet_buf_write (&req_buf,"oid %s" , +oid_to_hex (&oids->oid[i]));packet_buf_flush (&req_buf);if (write_in_full (fd_out, req_buf.buf, req_buf.len) <0 )
@@ -45,8 +49,12 @@ static int parse_object_size(const char *s, size_t *res)return 0 ; } -int fetch_object_info (const enum protocol_version version,struct object_info_args *args, -struct packet_reader *reader,struct object_info *object_info_data, +int fetch_object_info (const enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options, +struct packet_reader *reader, +struct object_info *object_info_data,const int stateless_rpc,const int fd_out) {int size_index =-1 ;
@@ -64,16 +72,17 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* because the number of options is a small known number (the * supported placeholders which currently are size and type). */ -for (int i = (int )args->object_info_options->nr -1 ; i >=0 ; i--) +for (int i = (int )object_info_options->nr -1 ; i >=0 ; i--)if (!server_supports_feature ("object-info" , - args->object_info_options->items[i].string,0 )) -unsorted_string_list_delete_item (args->object_info_options, i,0 ); + object_info_options->items[i].string,0 )) +unsorted_string_list_delete_item (object_info_options, i,0 );/* * Even if no options are left, we still send the oid so we get * at least an existence check. */ -send_object_info_request (fd_out, args); +send_object_info_request (fd_out, server_options, oids, + object_info_options);break ;case protocol_v1:case protocol_v0:
@@ -82,14 +91,14 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arBUG ("unknown protocol version" ); } -for (size_t i =0 ; i < args->object_info_options->nr; i++) { +for (size_t i =0 ; i < object_info_options->nr; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );return -1 ; } -if (!unsorted_string_list_has_string (args->object_info_options, reader->line)) +if (!unsorted_string_list_has_string (object_info_options, reader->line))return -1 ;if (!strcmp (reader->line,"size" )) {
@@ -98,7 +107,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* is only size. No risk of overflow. */ size_index = (int )i; -for (size_t j =0 ; j < args->oids->nr; j++) +for (size_t j =0 ; j < oids->nr; j++) object_info_data[j].sizep =xcalloc (1 ,sizeof (*object_info_data[j].sizep)); }else {
@@ -106,19 +115,19 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar } } -for (size_t i =0 ; i < args->oids->nr; i++) { +for (size_t i =0 ; i < oids->nr; i++) {struct string_list object_info_values =STRING_LIST_INIT_DUP ;if (packet_reader_read (reader) !=PACKET_READ_NORMAL )die (_ ("object-info: expected %" PRIuMAX" objects, got %" PRIuMAX), - (uintmax_t )args->oids->nr, (uintmax_t )i); + (uintmax_t )oids->nr, (uintmax_t )i);string_list_split (&object_info_values, reader->line," " ,-1 );if (strcmp (object_info_values.items[0 ].string, -oid_to_hex (&args->oids->oid[i]))) +oid_to_hex (&oids->oid[i])))die (_ ("object-info: expected OID: %s, got %s" ), -oid_to_hex (&args->oids->oid[i]), +oid_to_hex (&oids->oid[i]), object_info_values.items[0 ].string);/*
@@ -138,7 +147,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_ar* the server we expect the server to answer with the same * number of attributes requested. */ -if (args->object_info_options->nr +1 != object_info_values.nr) +if (object_info_options->nr +1 != object_info_values.nr)die ("object-info: unexpected number of attributes: %s" , reader->line);
@@ -154,7 +163,7 @@ int fetch_object_info(const enum protocol_version version, struct object_info_arif (packet_reader_read (reader) !=PACKET_READ_FLUSH )die (_ ("object-info: expected flush after %" PRIuMAX" objects" ), - (uintmax_t )args->oids->nr); + (uintmax_t )oids->nr);check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" );
fetch-object-info.h +8 -9
@@ -4,22 +4,21 @@#include "pkt-line.h" #include "protocol.h" -struct object_info_args { -struct string_list *object_info_options; -const struct string_list *server_options; -struct oid_array *oids; -}; -struct object_info ; +struct oid_array ;/* * Sends git-cat-file object-info command into the request buf and read the * results from packets. * -* Modifies args->object_info_options, on return it contains only the supported +* Modifies object_info_options, on return it contains only the supported * options by the server. */ -int fetch_object_info (enum protocol_version version,struct object_info_args *args, -struct packet_reader *reader,struct object_info *object_info_data, +int fetch_object_info (enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct string_list *object_info_options, +struct packet_reader *reader, +struct object_info *object_info_data,int stateless_rpc,int fd_out);#endif /* FETCH_OBJECT_INFO_H */
transport.c +5 -6
@@ -438,11 +438,6 @@ static int fetch_object_info_via_pack(struct transport *transport)int ret =0 ;struct git_transport_data *data = transport->data;struct packet_reader reader; -struct object_info_args args = {0 }; - - args.server_options = transport->server_options; - args.oids = transport->smart_options->object_info_oids; - args.object_info_options = transport->smart_options->object_info_options;connect_setup (transport,0 ); packet_reader_init(&reader, data->fd[0], NULL, 0,
@@ -453,7 +448,11 @@ static int fetch_object_info_via_pack(struct transport *transport) data->version =discover_version (&reader); transport->hash_algo = reader.hash_algo; - ret =fetch_object_info (data->version, &args, &reader, + ret =fetch_object_info (data->version, + transport->server_options, + transport->smart_options->object_info_oids, + transport->smart_options->object_info_options, + &reader, data->options.object_info_data, transport->stateless_rpc, data->fd[1 ]);
Signature · 1 line
2.54.0
PATCH v6 4/10 fetch-object-info: use dedicated struct for the results
In reply to Pablo Sabaterfetch_object_info() collects information about N objects, but it stores the results in an array of object_info. That struct holds the extended parameters of read_object_info() (The optional outputs the caller wants filled). Its pointers tell that function where to write the answers for a single object. object_info is not meant to be the final storage, and since fetch_object_info() does not call read_object_info(), there is no reason to use it. Using it means allocating one scalar per object per attribute just to have those pointers somewhere to point at.
Add struct fetch_object_info_results. The caller sets the wants_* flags to say what it is interested in, and fetch_object_info() allocates one array per attribute. A set wants_* flag means "asked for", while a non-NULL array means "available". The caller releases the arrays with free_fetch_object_info_results().
The object_info_options string list is no longer needed. Filtering against the server's advertisement now sets local ask_* flags, and send_object_info_request() turns those into the v2 protocol option strings. remote_atom_map[] existed only to map those strings back into atom names, so drop it and build remote_allowed_atoms from the result arrays.
Currently for wants_* and ask_* there is only the 'size' variant but a subsequent commit will add '*_type'.
free_object_info_contents() loses its only caller and is dropped.
Dropping the allow-list check makes the final else reachable from the wire, so die() instead of BUG(): an unknown attribute is the server's error, not ours.
--- builtin/cat-file.c | 59 +++++++++----------------------------- fetch-object-info.c | 81 ++++++++++++++++++++++++++--------------------------- fetch-object-info.h | 27 +++++++++++++----- object-file.c | 10 ------- odb.h | 3 -- transport.c | 3 +- transport.h | 5 ++-- 7 files changed, 77 insertions(+), 111 deletions(-)
builtin/cat-file.c +14 -45
@@ -31,6 +31,7 @@#include "alias.h" #include "remote.h" #include "transport.h" +#include "fetch-object-info.h" /* * Maximum length for a remote URL. While no universal standard exists,
@@ -681,9 +682,8 @@ static void batch_one_object(const char *obj_name,static int get_remote_info (int argc,const char **argv, -struct object_info **remote_object_info, -struct oid_array *object_info_oids, -struct string_list *object_info_options) +struct fetch_object_info_results *results, +struct oid_array *object_info_oids) {int retval =0 ;struct remote *remote =NULL ;
@@ -724,11 +724,9 @@ static int get_remote_info(int argc, goto cleanup; } -CALLOC_ARRAY (*remote_object_info, object_info_oids->nr); gtransport->smart_options->object_info_oids = object_info_oids; - gtransport->smart_options->object_info_options = object_info_options; - gtransport->smart_options->object_info_data = *remote_object_info; + gtransport->smart_options->object_info_results = results; retval =transport_fetch_object_info (gtransport); cleanup:transport_disconnect (gtransport);
@@ -816,21 +814,6 @@ static void parse_cmd_mailmap(struct batch_options *opt UNUSED,load_mailmap (); } -struct protocol_placeholder_entry { -const char *option; -const char *atom; -}; - -static const struct protocol_placeholder_entry remote_atom_map[] = { - {"size" ,"objectsize" }, - {"type" ,"objecttype" }, -/* -* Add new protocol options here. Even if the server doesn't support -* them the allow_list will drop them if the server doesn't advertise -* them. -*/ -}; -static void parse_cmd_remote_object_info (struct batch_options *opt,const char *line,struct strbuf *output,struct expand_data *data)
@@ -838,9 +821,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,int count;const char **argv;char *line_to_split; -struct object_info *remote_object_info =NULL ; +struct fetch_object_info_results results =FETCH_OBJECT_INFO_RESULTS_INIT ;struct oid_array object_info_oids =OID_ARRAY_INIT ; -struct string_list object_info_options =STRING_LIST_INIT_NODUP ;const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )
@@ -861,26 +843,21 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, MAX_ALLOWED_OBJ_LIMIT);if (data->info.sizep) -string_list_append (&object_info_options,"size" ); -if (data->info.typep) -string_list_append (&object_info_options,"type" ); + results.wants_size =1 ; -if (get_remote_info (count, argv, &remote_object_info, - &object_info_oids, &object_info_options)) +if (get_remote_info (count, argv, &results, &object_info_oids))die (_ ("failed to get object info from the remote: %s" ), argv[0 ]);string_list_clear (&data->remote_allowed_atoms,0 );string_list_append (&data->remote_allowed_atoms,"objectname" ); -for (size_t i =0 ; i <ARRAY_SIZE (remote_atom_map); i++) -if (unsorted_string_list_has_string (&object_info_options, remote_atom_map[i].option)) -string_list_append (&data->remote_allowed_atoms, - remote_atom_map[i].atom); +if (results.sizes) +string_list_append (&data->remote_allowed_atoms,"objectsize" ); data->skip_object_info =1 ; -for (size_t i =0 ; i < object_info_oids.nr; i++) { +for (size_t i =0 ; i < results.nr; i++) { data->oid = object_info_oids.oid[i]; -if (remote_object_info[i].unrecognized) { +if (results.unrecognized[i]) {report_object_status (opt,oid_to_hex (&data->oid), &data->oid,"missing" );continue ;
@@ -890,13 +867,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,* When reaching here, it means remote-object-info can retrieve * information from server without downloading them. */ -if (remote_object_info[i].sizep) { - data->size = *remote_object_info[i].sizep; - } - -if (remote_object_info[i].typep) { - data->type = *remote_object_info[i].typep; - } +if (results.sizes) + data->size = results.sizes[i]; opt->batch_mode =BATCH_MODE_INFO ; data->is_remote =1 ;
@@ -906,12 +878,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->skip_object_info =0 ; opt->format = saved_format; -for (size_t i =0 ; i < object_info_oids.nr; i++) -free_object_info_contents (&remote_object_info[i]); -string_list_clear (&object_info_options,0 ); +free_fetch_object_info_results (&results);free (line_to_split);free (argv); -free (remote_object_info);oid_array_clear (&object_info_oids); }
fetch-object-info.c +39 -42
147 lines not on this page · raw message
fetch-object-info.h +20 -7
@@ -4,21 +4,34 @@#include "pkt-line.h" #include "protocol.h" -struct object_info ; +struct fetch_object_info_results { +size_t *sizes; +uint8_t *unrecognized; +size_t nr; +unsigned wants_size:1 ; +}; + +#define FETCH_OBJECT_INFO_RESULTS_INIT { 0 } +struct oid_array ;/* -* Sends git-cat-file object-info command into the request buf and read the +* Sends git-cat-file object-info command into the request buf and reads the * results from packets. * -* Modifies object_info_options, on return it contains only the supported -* options by the server. +* The caller sets the wants_* flags in "results" to indicate which attributes +* it is interested in. On return, "results" holds one array per attribute that +* the server both advertised and answered with. An array left NULL means the +* attribute is not available. +* Release them with free_fetch_object_info_results(). */ int fetch_object_info (enum protocol_version version,const struct string_list *server_options,struct oid_array *oids, -struct string_list *object_info_options,struct packet_reader *reader, -struct object_info *object_info_data, -int stateless_rpc,int fd_out); +struct fetch_object_info_results *results, +int stateless_rpc, +int fd_out); + +void free_fetch_object_info_results (struct fetch_object_info_results *results);#endif /* FETCH_OBJECT_INFO_H */
object-file.c +0 -10
@@ -1340,13 +1340,3 @@ int odb_transaction_files_begin(struct odb_source *source,return 0 ; } - -void free_object_info_contents (struct object_info *object_info) -{ -if (!object_info) -return ; -free (object_info->typep); -free (object_info->sizep); -free (object_info->disk_sizep); -free (object_info->delta_base_oid); -}
odb.h +0 -3
@@ -637,7 +637,4 @@ void parse_alternates(const char *string,const char *relative_base, struct strvec *out); -/* Free pointers inside of object_info, but not object_info itself */ -void free_object_info_contents (struct object_info *object_info); -#endif /* ODB_H */
transport.c +1 -2
@@ -451,9 +451,8 @@ static int fetch_object_info_via_pack(struct transport *transport) ret =fetch_object_info (data->version, transport->server_options, transport->smart_options->object_info_oids, - transport->smart_options->object_info_options, &reader, - data->options.object_info_data, + data->options.object_info_results, transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);
transport.h +3 -2
@@ -7,6 +7,8 @@#include "string-list.h" #include "connect.h" +struct fetch_object_info_results ; +struct git_transport_options {unsigned thin :1 ;unsigned keep :1 ;
@@ -57,8 +59,7 @@ struct git_transport_options {struct oidset *acked_commits;struct oid_array *object_info_oids; -struct object_info *object_info_data; -struct string_list *object_info_options; +struct fetch_object_info_results *object_info_results; };enum transport_family {
Signature · 1 line
2.54.0
PATCH v6 5/10 fetch-object-info: die() on the remaining error path
In reply to Pablo SabaterEvery failure in fetch_object_info() dies except one: a short read while parsing the attribute lines returns -1. That -1 is then passed through fetch_object_info_via_pack() and get_remote_info() up to cat-file, only to die() with a generic message.
Die in fetch_object_info() instead, consistently with the rest of its error paths, and make fetch_object_info() void.
--- fetch-object-info.c | 19 +++++++++---------- fetch-object-info.h | 14 +++++++------- transport.c | 12 ++++++------ 3 files changed, 22 insertions(+), 23 deletions(-)
fetch-object-info.c +9 -10
@@ -47,13 +47,13 @@ static int parse_object_size(const char *s, size_t *res)return 0 ; } -int fetch_object_info (const enum protocol_version version, -const struct string_list *server_options, -struct oid_array *oids, -struct packet_reader *reader, -struct fetch_object_info_results *results, -const int stateless_rpc, -const int fd_out) +void fetch_object_info (const enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct packet_reader *reader, +struct fetch_object_info_results *results, +const int stateless_rpc, +const int fd_out) {unsigned ask_size =0 ;int size_index =-1 ;
@@ -89,7 +89,8 @@ int fetch_object_info(const enum protocol_version version,if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" ); -return -1 ; +die (_ ("object-info: expected %" PRIuMAX" attributes, got %" PRIuMAX), + (uintmax_t )wanted, (uintmax_t )i); }if (!strcmp (reader->line,"size" )) {
@@ -156,8 +157,6 @@ int fetch_object_info(const enum protocol_version version, (uintmax_t)oids->nr);check_stateless_delimiter (stateless_rpc, reader,"stateless delimiter expected" ); - -return 0 ; }void free_fetch_object_info_results (struct fetch_object_info_results *results)
fetch-object-info.h +7 -7
@@ -24,13 +24,13 @@ struct oid_array;* attribute is not available. * Release them with free_fetch_object_info_results(). */ -int fetch_object_info (enum protocol_version version, -const struct string_list *server_options, -struct oid_array *oids, -struct packet_reader *reader, -struct fetch_object_info_results *results, -int stateless_rpc, -int fd_out); +void fetch_object_info (enum protocol_version version, +const struct string_list *server_options, +struct oid_array *oids, +struct packet_reader *reader, +struct fetch_object_info_results *results, +int stateless_rpc, +int fd_out);void free_fetch_object_info_results (struct fetch_object_info_results *results);
transport.c +6 -6
@@ -448,12 +448,12 @@ static int fetch_object_info_via_pack(struct transport *transport) data->version =discover_version (&reader); transport->hash_algo = reader.hash_algo; - ret =fetch_object_info (data->version, - transport->server_options, - transport->smart_options->object_info_oids, - &reader, - data->options.object_info_results, - transport->stateless_rpc, data->fd[1 ]); +fetch_object_info (data->version, + transport->server_options, + transport->smart_options->object_info_oids, + &reader, + data->options.object_info_results, + transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);if (data->fd[1 ] >=0 )
Signature · 1 line
2.54.0
PATCH v6 6/10 transport: drop remote object-info fields from transport struct
In reply to Pablo SabaterFrom: Jeff King <[email protected]>
A remote object-info request needs three things: the transport for contacting the remote, the list of oids to request, and a place to store the output.
Rather than take these as function parameters, we take only the transport object, and expect the caller to have placed the other two into special fields in the transport struct. But this doesn't make much sense. The set of oids and results are really only valid for one request. There is no reason the transport would need to hang on to them outside of the single function call.
Even though we save a few lines passing the parameters around through the various vtable functions, the result is harder to understand (for example, who is responsible for cleaning up results, and when should it happen?). It also opens up the possibility of a subtle bug. A caller is likely to point those fields to stack variables which could go out of scope, and the transport struct would be left holding invalid pointers.
This is mostly harmless now, as we disconnect the transport immediately after the sole caller of transport_fetch_object_info(). But conceptually we could keep the transport open and make multiple fetch calls (and reuse the same connection to the helper, to a remote HTTP server, and so on).
So let's pull these out of the struct and pass them as function parameters. It's a little more verbose, but I think more clearly illustrates the intent. I've also tweaked a few function signatures to mark the input oid array as const, since it is purely an input to the function.
--- builtin/cat-file.c | 6 ++---- fetch-object-info.c | 4 ++-- fetch-object-info.h | 2 +- transport-helper.c | 7 +++++-- transport-internal.h | 6 +++++- transport.c | 14 +++++++++----- transport.h | 7 +++---- 7 files changed, 27 insertions(+), 19 deletions(-)
builtin/cat-file.c +2 -4
@@ -724,10 +724,8 @@ static int get_remote_info(int argc, goto cleanup; } - gtransport->smart_options->object_info_oids = object_info_oids; - - gtransport->smart_options->object_info_results = results; - retval =transport_fetch_object_info (gtransport); + retval =transport_fetch_object_info (gtransport, object_info_oids, + results); cleanup:transport_disconnect (gtransport);return retval;
fetch-object-info.c +2 -2
@@ -11,7 +11,7 @@/* Sends object-info command and its arguments into the request buffer. */ static void send_object_info_request (const int fd_out,const struct string_list *server_options, -struct oid_array *oids, +const struct oid_array *oids,unsigned ask_size) {struct strbuf req_buf =STRBUF_INIT ;
@@ -49,7 +49,7 @@ static int parse_object_size(const char *s, size_t *res) void fetch_object_info(const enum protocol_version version, const struct string_list *server_options, - struct oid_array *oids, + const struct oid_array *oids, struct packet_reader *reader, struct fetch_object_info_results *results, const int stateless_rpc,
fetch-object-info.h +1 -1
@@ -26,7 +26,7 @@ struct oid_array;*/ void fetch_object_info(enum protocol_version version, const struct string_list *server_options, - struct oid_array *oids, + const struct oid_array *oids, struct packet_reader *reader, struct fetch_object_info_results *results, int stateless_rpc,
transport-helper.c +5 -2
@@ -784,11 +784,14 @@ static int fetch_refs(struct transport *transport,return -1 ; } -static int fetch_object_info_helper (struct transport *transport) +static int fetch_object_info_helper (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {get_helper (transport);if (process_connect (transport,0 )) -return transport->vtable->fetch_object_info (transport); +return transport->vtable->fetch_object_info (transport, oids, + results);die (_ ("object-info requires protocol v2" )); }
transport-internal.h +5 -1
@@ -7,6 +7,8 @@ struct ref;struct transport ;struct strvec ;struct transport_ls_refs_options ; +struct oid_array ; +struct fetch_object_info_results ;struct transport_vtable {/**
@@ -51,7 +53,9 @@ struct transport_vtable {* * Uses object-info capability of v2 protocol. */ -int (*fetch_object_info)(struct transport *transport); +int (*fetch_object_info)(struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results);/** * Push the objects and refs. Send the necessary objects, and
transport.c +9 -5
@@ -433,7 +433,9 @@ static int get_bundle_uri(struct transport *transport) transport->bundles, stateless_rpc); } -static int fetch_object_info_via_pack (struct transport *transport) +static int fetch_object_info_via_pack (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {int ret =0 ;struct git_transport_data *data = transport->data;
@@ -450,9 +452,9 @@ static int fetch_object_info_via_pack(struct transport *transport)fetch_object_info (data->version, transport->server_options, - transport->smart_options->object_info_oids, + oids, &reader, - data->options.object_info_results, + results, transport->stateless_rpc, data->fd[1 ]);close (data->fd[0 ]);
@@ -465,11 +467,13 @@ static int fetch_object_info_via_pack(struct transport *transport)return ret; } -int transport_fetch_object_info (struct transport *transport) +int transport_fetch_object_info (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results) {if (!transport->vtable->fetch_object_info)die (_ ("remote does not support object-info" )); -return transport->vtable->fetch_object_info (transport); +return transport->vtable->fetch_object_info (transport, oids, results); } static int fetch_refs_via_pack(struct transport *transport,
transport.h +3 -4
@@ -57,9 +57,6 @@ struct git_transport_options {* common commits to this oidset instead of fetching any packfiles. */ struct oidset *acked_commits; - -struct oid_array *object_info_oids; -struct fetch_object_info_results *object_info_results; };enum transport_family {
@@ -317,7 +314,9 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs);/* * Fetch the object info from remote */ -int transport_fetch_object_info (struct transport *transport); +int transport_fetch_object_info (struct transport *transport, +const struct oid_array *oids, +struct fetch_object_info_results *results);/* * If this flag is set, unlocking will avoid to call non-async-signal-safe
Signature · 1 line
2.54.0
PATCH v6 7/10 protocol-caps: add type support to object-info
In reply to Pablo SabaterTeach the server-side object-info handler to accept type as a requested field. When the client includes type in its object-info request, the server returns the requested object type.
While touching send_info(), wrap an over-long line and fix the bit field style of requested_info.size.
--- protocol-caps.c | 21 ++++++++++++++++++--- t/t5701-git-serve.sh | 30 ++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 3 deletions(-)
protocol-caps.c +18 -3
@@ -11,7 +11,8 @@#include "strbuf.h" struct requested_info { -unsigned size :1 ; +unsigned size:1 ; +unsigned type:1 ; };/*
@@ -73,15 +74,20 @@ static void send_info(struct repository *r, struct packet_writer *writer,if (info->size)packet_writer_write (writer,"size" ); +if (info->type) +packet_writer_write (writer,"type" ); + for_each_string_list_item (item, oid_str_list) {const char *oid_str = item->string; +enum object_type object_type;struct object_id oid;size_t object_size;if (get_oid_hex_algop (oid_str, &oid, r->hash_algo) <0 ) {packet_writer_error ( writer, -"object-info: protocol error, expected to get oid, not '%s'" , +"object-info: protocol error, expected to get " +"oid, not '%s'" , oid_str);continue ; }
@@ -93,7 +99,8 @@ static void send_info(struct repository *r, struct packet_writer *writer,* If an object is not recognized by the server append SP to * the response. */ -if (get_object_info (r->objects, &oid, &object_size) <=OBJ_NONE ) { + object_type =get_object_info (r->objects, &oid, &object_size); +if (object_type <=OBJ_NONE ) {strbuf_addstr (&send_buffer," " ); goto write; }
@@ -103,6 +110,9 @@ static void send_info(struct repository *r, struct packet_writer *writer, (uintmax_t)object_size); } +if (info->type) +strbuf_addf (&send_buffer," %s" ,type_name (object_type)); + write:packet_writer_write (writer,"%s" , send_buffer.buf);strbuf_reset (&send_buffer);
@@ -124,6 +134,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)continue ; } +if (!strcmp ("type" , request->line)) { + info.type =1 ; +continue ; + } +if (parse_oid (request->line, &oid_str_list))continue ;
t/t5701-git-serve.sh +30 -0
@@ -369,6 +369,36 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'object-info supports type' ' + test_config transfer.advertiseObjectInfo true && + + two_oid=$(git rev-parse two:two.t) && + two_size=$(test_file_size two.t) && + + test-tool pkt-line pack >in <<-EOF && + command=object-info + object-format=$(test_oid algo) + 0001 + size + type + oid $two_oid + oid $two_oid + 0000 + EOF + + cat >expect <<-EOF && + size + type + $two_oid $two_size blob + $two_oid $two_size blob + 0000 + EOF + + test-tool serve-v2 --stateless-rpc <in >out && + test-tool pkt-line unpack <out >actual && + test_cmp expect actual +' + test_expect_success 'bare OID request' ' test_config transfer.advertiseObjectInfo true &&
Signature · 1 line
2.54.0
PATCH v6 8/10 fetch-object-info: parse type from server response
In reply to Pablo SabaterThe server can handle type requests but does not advertise the capability yet. Prepare the client to know how to parse the server response once the server advertises the type capability.
--- builtin/cat-file.c | 7 +++++++ fetch-object-info.c | 38 +++++++++++++++++++++++++++++++++++--- fetch-object-info.h | 3 +++ 3 files changed, 45 insertions(+), 3 deletions(-)
builtin/cat-file.c +7 -0
@@ -842,6 +842,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,if (data->info.sizep) results.wants_size =1 ; +if (data->info.typep) + results.wants_type =1 ;if (get_remote_info (count, argv, &results, &object_info_oids))die (_ ("failed to get object info from the remote: %s" ), argv[0 ]);
@@ -850,6 +852,8 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,string_list_append (&data->remote_allowed_atoms,"objectname" );if (results.sizes)string_list_append (&data->remote_allowed_atoms,"objectsize" ); +if (results.types) +string_list_append (&data->remote_allowed_atoms,"objecttype" ); data->skip_object_info =1 ;for (size_t i =0 ; i < results.nr; i++) {
@@ -868,6 +872,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,if (results.sizes) data->size = results.sizes[i]; +if (results.types) + data->type = results.types[i]; + opt->batch_mode =BATCH_MODE_INFO ; data->is_remote =1 ;batch_object_write (argv[i +1 ], output, opt, data,NULL ,0 );
fetch-object-info.c +35 -3
@@ -1,6 +1,7 @@#include "git-compat-util.h" #include "gettext.h" #include "hex.h" +#include "object.h" #include "pkt-line.h" #include "connect.h" #include "oid-array.h"
@@ -12,7 +13,8 @@static void send_object_info_request (const int fd_out,const struct string_list *server_options,const struct oid_array *oids, -unsigned ask_size) +unsigned ask_size, +unsigned ask_type) {struct strbuf req_buf =STRBUF_INIT ;
@@ -21,6 +23,9 @@ static void send_object_info_request(const int fd_out,if (ask_size)packet_buf_write (&req_buf,"size" ); +if (ask_type) +packet_buf_write (&req_buf,"type" ); +if (oids)for (size_t i =0 ; i < oids->nr; i++) packet_buf_write(&req_buf, "oid %s",
@@ -56,7 +61,9 @@ void fetch_object_info(const enum protocol_version version, const int fd_out) {unsigned ask_size =0 ; +unsigned ask_type =0 ;int size_index =-1 ; +int type_index =-1 ;size_t wanted; results->nr = oids->nr;
@@ -71,11 +78,16 @@ void fetch_object_info(const enum protocol_version version, server_supports_feature("object-info", "size", 0)) ask_size =1 ; +if (results->wants_type && +server_supports_feature ("object-info" ,"type" ,0 )) + ask_type =1 ; +/* * Even if no options are left, we still send the oid so we get * at least an existence check. */ -send_object_info_request (fd_out, server_options, oids, ask_size); +send_object_info_request (fd_out, server_options, oids, ask_size, + ask_type);break ;case protocol_v1:case protocol_v0:
@@ -83,7 +95,7 @@ void fetch_object_info(const enum protocol_version version,case protocol_unknown_version:BUG ("unknown protocol version" ); } - wanted = ask_size; + wanted = ask_size + ask_type;for (size_t i =0 ; i < wanted; i++) {if (packet_reader_read (reader) !=PACKET_READ_NORMAL ) {
@@ -100,6 +112,13 @@ void fetch_object_info(const enum protocol_version version,die (_ ("object-info: duplicate 'size' attribute" )); size_index = (int )i;CALLOC_ARRAY (results->sizes, results->nr); + }else if (!strcmp (reader->line,"type" )) { +if (!ask_type) +die (_ ("object-info: unrequested 'type' attribute" )); +if (results->types) +die (_ ("object-info: duplicate 'type' attribute" )); + type_index = (int )i; +CALLOC_ARRAY (results->types, results->nr); }else {die (_ ("object-info: unknown attribute '%s'" ), reader->line);
@@ -149,6 +168,18 @@ void fetch_object_info(const enum protocol_version version, object_info_values.items[0 ].string, object_info_values.items[size_index + 1].string); +if (results->types) { +const char *type_str = + object_info_values.items[type_index +1 ].string; +int type =type_from_string_gently (type_str,-1 ,1 ); + +if (type <0 ) +die (_ ("object-info: object %s has invalid type '%s'" ), + object_info_values.items[0 ].string, type_str); + + results->types[i] = type; + } +string_list_clear (&object_info_values,0 ); }
@@ -162,6 +193,7 @@ void fetch_object_info(const enum protocol_version version,void free_fetch_object_info_results (struct fetch_object_info_results *results) {free (results->sizes); +free (results->types);free (results->unrecognized);memset (results,0 ,sizeof (*results)); }
fetch-object-info.h +3 -0
@@ -1,14 +1,17 @@#ifndef FETCH_OBJECT_INFO_H #define FETCH_OBJECT_INFO_H +#include "object.h" #include "pkt-line.h" #include "protocol.h" struct fetch_object_info_results {size_t *sizes; +enum object_type *types;uint8_t *unrecognized;size_t nr;unsigned wants_size:1 ; +unsigned wants_type:1 ; };#define FETCH_OBJECT_INFO_RESULTS_INIT { 0 }
Signature · 1 line
2.54.0
PATCH v6 9/10 serve: advertise type capability
In reply to Pablo SabaterThe server and the client can handle type requests but the client won't ask for it until the server advertises it.
Add type to the advertised capabilities so the client knows that it can request it.
--- serve.c | 4 ++-- t/t1017-cat-file-remote-object-info.sh | 26 ++++++++++++++++++++++---- 2 files changed, 24 insertions(+), 6 deletions(-)
serve.c +2 -2
@@ -97,9 +97,9 @@ static int object_info_advertise(struct repository *r, struct strbuf *value)/* disabled by default */ advertise_object_info =0 ; } -/* Currently only size is supported */ +/* Currently only size and type are supported */ if (value && advertise_object_info) -strbuf_addstr (value,"size" ); +strbuf_addstr (value,"size type" );return advertise_object_info; }
t/t1017-cat-file-remote-object-info.sh +22 -4
@@ -7,6 +7,7 @@ test_description='git cat-file --batch-command with remote-object-info command' hello_content="Hello World" hello_size=$(strlen "$hello_content") +hello_type="blob" hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) hello_short_oid=$(git rev-parse --short "$hello_oid")
@@ -19,6 +20,7 @@ unstored_oid=$(echo_without_newline "$unstored_content" | git hash-object --stdi # file name is hello, which is 5 characters # a space is 1 character and a null is 1 character tree_size=$(($(test_oid rawsz) + 13)) +tree_type="tree" commit_message="Initial commit"
@@ -31,6 +33,7 @@ commit_message="Initial commit" # An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`, # to get 177, 2. then deduct 40 hex characters to get 137 commit_size=$(($(test_oid hexsz) + 137)) +commit_type="commit" tag_header_without_oid="type blob tag hellotag
@@ -44,6 +47,7 @@ $tag_description" tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w) tag_size=$(strlen "$tag_content") +tag_type="tag" set_transport_variables () { hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin)
@@ -256,14 +260,12 @@ test_expect_success 'remote-object-info does not die on missing oid like info' ' ) ' -# This tests depends on %(objecttype) not being supported yet, once supported -# it needs to be updated. -test_expect_success 'unsupported placeholder on remote returns empty string' ' +test_expect_success 'objecttype is supported by remote-object-info' ' ( set_transport_variables "$daemon_parent" && cd "$daemon_parent/daemon_client_empty" && - echo "" >expect && + echo "$hello_type" >expect && git cat-file --batch-command="%(objecttype)" >actual <<-EOF && remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid EOF
@@ -271,6 +273,22 @@ test_expect_success 'unsupported placeholder on remote returns empty string' ' ) ' +test_expect_success 'unsupported placeholders on remote return empty string' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + fmt="%(objectmode) %(objectsize:disk) %(rest) %(deltabase)" && + + # The hardcoded SPs between the atoms are respected. + echo " " >expect && + git cat-file --batch-command="$fmt" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid + EOF + test_cmp expect actual + ) +' + test_expect_success 'requesting only objectname echoes back' ' ( set_transport_variables "$daemon_parent" &&
Signature · 1 line
2.54.0
PATCH v6 10/10 cat-file: unify default format
In reply to Pablo Sabater%(objecttype) is supported both by the client and by the server. Change the temporary default format to the unified version that the other commands use.
Update documentation to remove %(objecttype) from the caveats of remote-object-info and show %(objecttype) support.
Now that type is supported and the default format unified, update the tests to expect the new default format.
--- Documentation/git-cat-file.adoc | 17 +++++------ Documentation/gitprotocol-v2.adoc | 18 +++++++++--- builtin/cat-file.c | 7 ----- t/t1017-cat-file-remote-object-info.sh | 52 +++++++++++++++++----------------- 4 files changed, 47 insertions(+), 47 deletions(-)
Documentation/git-cat-file.adoc +7 -10
@@ -348,15 +348,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. -The command `remote-object-info` only supports the `%(objectname)` and -`%(objectsize)` placeholders. See `CAVEATS` below for more information. +The command `remote-object-info` only supports the `%(objectname)`, +`%(objectsize)` and `%(objecttype)` placeholders. See `CAVEATS` below for more +information. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`, except for `remote-object-info` commands which -use `%(objectname) %(objectsize)` because `%(objecttype)` is not supported yet. - -WARNING: When "%(objecttype)" is supported, the default format WILL be unified, -so DO NOT RELY on the current default format to stay the same!!! +%(objecttype) %(objectsize)`. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -453,9 +450,9 @@ scripting purposes. CAVEATS ------- -Note that only `%(objectname)` and `%(objectsize)` are currently -supported by the `remote-object-info` command. Using any other placeholder in -the format string will return an empty string in its position. +Note that only `%(objectname)`, `%(objectsize)` and `%(objecttype)` are +currently supported by the `remote-object-info` command. Using any other +placeholder in the format string will return an empty string in its position. Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are
Documentation/gitprotocol-v2.adoc +14 -4
@@ -558,14 +558,17 @@ object-info `object-info` is the command to retrieve information about one or more objects. Its main purpose is to allow a client to make decisions based on this -information without having to fully fetch objects. Object size is the only -information that is currently supported. +information without having to fully fetch objects. Currently only object size +and type are supported. An `object-info` request takes the following arguments: size Requests size information to be returned for each listed object id. + type + Requests type information to be returned for each listed object id. + oid <oid> Indicates to the server an object which the client wants to obtain information for. They must be full OIDs.
@@ -580,11 +583,18 @@ space. info = *PKT-LINE(attr LF) *PKT-LINE(obj-info LF) - attr = "size" + attr = "size" | "type" obj-size = 1*DIGIT - obj-info = obj-id [SP [obj-size]] + obj-type = "blob" | "tree" | "commit" | "tag" + + obj-val = obj-size | obj-type + + obj-info = obj-id [SP [obj-val *(SP obj-val)]] + +The values in `obj-info` appear in the same order as the corresponding `attr` +lines, with exactly one value per requested attribute. If the server does not recognize the OID, the response will be `<oid> SP` regardless of the number of attributes requested.
builtin/cat-file.c +0 -7
@@ -821,15 +821,9 @@ static void parse_cmd_remote_object_info(struct batch_options *opt,char *line_to_split;struct fetch_object_info_results results =FETCH_OBJECT_INFO_RESULTS_INIT ;struct oid_array object_info_oids =OID_ARRAY_INIT ; -const char *saved_format = opt->format;if (strlen (line) >=MAX_REMOTE_OBJ_INFO_LINE )die (_ ("remote-object-info command too long" )); -/* -* TODO: Use the default format once %(objecttype) is supported. -*/ -if (!opt->format) - opt->format ="%(objectname) %(objectsize)" ; line_to_split =xstrdup (line); count =split_cmdline (line_to_split, &argv);
@@ -881,7 +875,6 @@ static void parse_cmd_remote_object_info(struct batch_options *opt, data->is_remote =0 ; } data->skip_object_info =0 ; - opt->format = saved_format;free_fetch_object_info_results (&results);free (line_to_split);
t/t1017-cat-file-remote-object-info.sh +26 -26
100 lines not on this page · raw message
Signature · 1 line
2.54.0
Quoted 2 lines · Pablo Sabater
Changes in v6:
- Fixed static analysis CI fail with two forward declarations.
Sorry, these were my fault. The forward declarations are obviously the right solution.
-Peff
Re: [PATCH GSoC v6 06/10] transport: drop remote object-info fields from transport struct
In reply to Pablo SabaterQuoted 6 lines · Pablo Sabater
Rather than take these as function parameters, we take only the transport object, and expect the caller to have placed the other two into special fields in the transport struct. But this doesn't make much sense. The set of oids and results are really only valid for one request. There is no reason the transport would need to hang on to them outside of the single function call.
Thanks for injecting some sanity into the mix.
With this fixed, are we happy with the entire series by now?
Thanks.
Re: [PATCH GSoC v6 06/10] transport: drop remote object-info fields from transport struct
Quoted 10 lines · Junio C Hamano
Quoted 6 lines · Pablo Sabater
Rather than take these as function parameters, we take only the transport object, and expect the caller to have placed the other two into special fields in the transport struct. But this doesn't make much sense. The set of oids and results are really only valid for one request. There is no reason the transport would need to hang on to them outside of the single function call.
Quoted 3 lines
Thanks for injecting some sanity into the mix.
With this fixed, are we happy with the entire series by now?
Just went through the series and everything looks good to me, but I've had the same opinion for the last couple of iterations as well.
Thanks,
Chandra.
Re: [PATCH GSoC v6 06/10] transport: drop remote object-info fields from transport struct
Quoted 13 lines · Chandra Pratap
Quoted 12 lines · Junio C Hamano
Quoted 6 lines · Pablo Sabater
Rather than take these as function parameters, we take only the transport object, and expect the caller to have placed the other two into special fields in the transport struct. But this doesn't make much sense. The set of oids and results are really only valid for one request. There is no reason the transport would need to hang on to them outside of the single function call.
Thanks for injecting some sanity into the mix.
Quoted 7 lines
Quoted 1 line
With this fixed, are we happy with the entire series by now?
Just went through the series and everything looks good to me, but I've had the same opinion for the last couple of iterations as well.
Thanks,
Chandra.
Same. This version looks good to me too :)