fetch-pack: trace packfile URI downloads
PATCH 1/1 fetch-pack: trace packfile URI downloads
When a protocol v2 fetch includes packfile URIs, the client downloads each advertised pack in a separate http-fetch process. Existing Trace2 regions cover negotiation, but not the time spent downloading these packs or the number of advertised URIs.
Add a Trace2 region around the packfile URI download loop and record the number of URIs. This makes the cost of downloading external packs visible without emitting an event for each pack.
Extend the existing packfile URI test to verify the region and count.
--- fetch-pack.c | 12 ++++++++++++ t/t5702-protocol-v2.sh | 7 ++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
fetch-pack.c +12 -0
@@ -1886,6 +1886,13 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, } } +if (packfile_uris.nr) { +trace2_region_enter ("fetch-pack" ,"packfile-uris" , + the_repository); +trace2_data_intmax ("fetch-pack" , the_repository, +"packfile-uris/count" , packfile_uris.nr); + } +for (i =0 ; i < packfile_uris.nr; i++) {int j;struct child_process cmd =CHILD_PROCESS_INIT ;
@@ -1936,6 +1943,11 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,repo_get_object_directory (the_repository), packname)); } + +if (packfile_uris.nr) +trace2_region_leave ("fetch-pack" ,"packfile-uris" , + the_repository); +string_list_clear (&packfile_uris,0 );strvec_clear (&index_pack_args);
t/t5702-protocol-v2.sh +6 -1
@@ -1223,7 +1223,7 @@ configure_exclusion () { test_expect_success 'part of packfile response provided as URI' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + rm -rf "$P" http_child log trace2 && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" &&
@@ -1238,10 +1238,15 @@ test_expect_success 'part of packfile response provided as URI' ' configure_exclusion "$P" other-blob >h2 && GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + GIT_TRACE2_EVENT="$(pwd)/trace2" \ git -c protocol.version=2 \ -c fetch.uriprotocols=http,https \ clone "$HTTPD_URL/smart/http_parent" http_child && + test_grep \"event\":\"region_enter\".*\"label\":\"packfile-uris\" trace2 && + test_grep \"key\":\"packfile-uris/count\",\"value\":\"2\" trace2 && + test_grep \"event\":\"region_leave\".*\"label\":\"packfile-uris\" trace2 && + # Ensure that my-blob and other-blob are in separate packfiles. for idx in http_child/.git/objects/pack/*.idx do
Quoted 8 lines · Ted Nyman
When a protocol v2 fetch includes packfile URIs, the client downloads each advertised pack in a separate http-fetch process. Existing Trace2 regions cover negotiation, but not the time spent downloading these packs or the number of advertised URIs.
Add a Trace2 region around the packfile URI download loop and record the number of URIs. This makes the cost of downloading external packs visible without emitting an event for each pack.
Right, by having a region we can verify how long downloading the packfiles took, and by tracking the number of packfiles we know how many we fetched. What we don't know is how long fetching each of the individual packs took, but I think that omission makes sense. After all, we can reasonably expect all packs to be served by the same infra, and as such they should usually have similar download speeds.
fetch-pack.c static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, · line 1893
Quoted 13 lines
fetch-pack.c +5
-0@@ -1886,6 +1886,13 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, } } +if (packfile_uris.nr) { +trace2_region_enter ("fetch-pack" ,"packfile-uris" , + the_repository); +trace2_data_intmax ("fetch-pack" , the_repository, +"packfile-uris/count" , packfile_uris.nr);
We don't have a repository available in our context, so we have to use the_repository.
fetch-pack.c static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, · line 1898
Quoted 5 lines
fetch-pack.c +2
-0+ } +for (i =0 ; i < packfile_uris.nr; i++) {int j;struct child_process cmd =CHILD_PROCESS_INIT ;
Sensible. We don't need to track fetching if we don't have any packfiles at all.
fetch-pack.c static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, · line 1953
Quoted 12 lines
fetch-pack.c +5
-0@@ -1936,6 +1943,11 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,repo_get_object_directory (the_repository), packname)); } + +if (packfile_uris.nr) +trace2_region_leave ("fetch-pack" ,"packfile-uris" , + the_repository); +string_list_clear (&packfile_uris,0 );strvec_clear (&index_pack_args);
And likewise, we don't have to leave the region, either in that case.
Quoted 19 lines
t/t5702-protocol-v2.sh +2
-1@@ -1223,7 +1223,7 @@ configure_exclusion () { test_expect_success 'part of packfile response provided as URI' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + rm -rf "$P" http_child log trace2 && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" &&@@ -1238,10 +1238,15 @@ test_expect_success 'part of packfile response provided as URI' ' configure_exclusion "$P" other-blob >h2 && GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + GIT_TRACE2_EVENT="$(pwd)/trace2" \ git -c protocol.version=2 \
t/t5702-protocol-v2.sh test_expect_success 'part of packfile response provided as URI' ' · line 1252
Quoted 10 lines
t/t5702-protocol-v2.sh +4
-0-c fetch.uriprotocols=http,https \ clone "$HTTPD_URL/smart/http_parent" http_child && + test_grep \"event\":\"region_enter\".*\"label\":\"packfile-uris\" trace2 && + test_grep \"key\":\"packfile-uris/count\",\"value\":\"2\" trace2 && + test_grep \"event\":\"region_leave\".*\"label\":\"packfile-uris\" trace2 && + # Ensure that my-blob and other-blob are in separate packfiles. for idx in http_child/.git/objects/pack/*.idx do
It does feel a tiny bit off to piggy-back on an existing test that has nothing to do with tracing except that it requires the traces to... I dunno, what does the existing test even do with the written logfile? Doesn't seem like it's using it at all.
Anyway, having this in a separate test would've been nice, but that doesn't warrant a reroll in my eyes. So overall, this patch looks good to me, thanks!
Patrick
Quoted 3 lines · Patrick Steinhardt
Anyway, having this in a separate test would've been nice, but that doesn't warrant a reroll in my eyes. So overall, this patch looks good to me, thanks!
Thanks for reviewing, Patrick.
Junio, a gentle ping on this patch. Patrick was positive on it and didn't think the test organization warranted a reroll. Is there anything else you'd like me to address before picking it up? I'm happy to split out the tracing checks into a separate test or refresh the patch if that helps.
Thanks,
Ted
Quoted 11 lines · Ted Nyman
Quoted 3 lines · Patrick Steinhardt
Anyway, having this in a separate test would've been nice, but that doesn't warrant a reroll in my eyes. So overall, this patch looks good to me, thanks!
Thanks for reviewing, Patrick.
Junio, a gentle ping on this patch. Patrick was positive on it and didn't think the test organization warranted a reroll. Is there anything else you'd like me to address before picking it up? I'm happy to split out the tracing checks into a separate test or refresh the patch if that helps.
Thanks. This slipped below my radar. It does not look to me that the placement of the test is too bad.
Will queue.