fetch: show progress for packfile uri downloads

From: Albert Cui <[email protected]>

Git appears to hang when downloading packfiles as this part of the fetch is silent, causing user confusion. This change implements progress for the number of packfiles downloaded; a progress display for bytes would involve deeper changes at the http-fetch layer instead of fetch-pack, the caller.

---
fetch: show progress for packfile uri downloads

Git appears to hang when downloading packfiles as this part of the fetch is silent, causing user confusion. This change implements progress for the number of packfiles downloaded; a progress display for bytes would involve deeper changes at the http-fetch layer instead of fetch-pack, the caller.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-907%2Falbertcui%2Fprogress-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-907/albertcui/progress-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/907

 fetch-pack.c           | 8 ++++++++
 t/t5702-protocol-v2.sh | 5 ++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

index 6a61a464283e..5646d151293d 100644
@@ -23,6 +23,7 @@
 #include "fetch-negotiator.h"
 #include "fsck.h"
 #include "shallow.h"
+#include "progress.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;

@@ -1585,6 +1586,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 	struct fetch_negotiator *negotiator;
 	int seen_ack = 0;
 	struct string_list packfile_uris = STRING_LIST_INIT_DUP;
+	struct progress *packfile_uri_progress;
 	int i;
 	struct strvec index_pack_args = STRVEC_INIT;
 	struct oidset gitmodules_oids = OIDSET_INIT;

@@ -1689,6 +1691,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		}
 	}
 
+	packfile_uri_progress = start_progress(_("Downloading packs"), packfile_uris.nr);
+
 	for (i = 0; i < packfile_uris.nr; i++) {
 		int j;
 		struct child_process cmd = CHILD_PROCESS_INIT;

@@ -1696,6 +1700,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		const char *uri = packfile_uris.items[i].string +
 			the_hash_algo->hexsz + 1;
 
+		display_progress(packfile_uri_progress, i+1);
 		strvec_push(&cmd.args, "http-fetch");
 		strvec_pushf(&cmd.args, "--packfile=%.*s",
 			     (int) the_hash_algo->hexsz,

@@ -1739,6 +1744,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 						 get_object_directory(),
 						 packname));
 	}
+
+	stop_progress(&packfile_uri_progress);
+
 	string_list_clear(&packfile_uris, 0);
 	strvec_clear(&index_pack_args);
 

index 2e1243ca40b0..8964a4003678 100755
@@ -851,7 +851,8 @@ test_expect_success 'part of packfile response provided as URI' '
 	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
 	git -c protocol.version=2 \
 		-c fetch.uriprotocols=http,https \
-		clone "$HTTPD_URL/smart/http_parent" http_child &&
+		clone "$HTTPD_URL/smart/http_parent" http_child \
+		--progress 2>progress &&
 
 	# Ensure that my-blob and other-blob are in separate packfiles.
 	for idx in http_child/.git/objects/pack/*.idx

@@ -875,6 +876,8 @@ test_expect_success 'part of packfile response provided as URI' '
 	test -f hfound &&
 	test -f h2found &&
 
+	test_i18ngrep "Downloading packs" progress &&
+
 	# Ensure that there are exactly 3 packfiles with associated .idx
 	ls http_child/.git/objects/pack/*.pack \
 	    http_child/.git/objects/pack/*.idx >filelist &&

base-commit: a5828ae6b52137b913b978e16cd2334482eb4c1f

gitgitgadget

I think this is an improvement, but I agree that a real byte display would be much better. I actually worked on this a long time ago for a very similar feature that we never quite pushed over the finish line. See patches 11 and 12 here:

https://lore.kernel.org/git/[email protected]/

(it might need some of the earlier refactoring, too, I'm not sure; and quite likely will need forward-porting as it has been 10 years).

-Peff

... "hence we do not do so in this patch"?

That's probably a very sensible way to go.

I expect that http-fetch will in the longer term become a mere fallback default used by those who do not have anything better. Because we are not in the business of writing a performant HTTP downloader, we would be better off if we make it easy to plug an external HTTP downloader other people write in to this codepath.

		display_progress(packfile_uri_progress, i + 1);

From: Albert Cui <[email protected]>

Git appears to hang when downloading packfiles as this part of the fetch is silent, causing user confusion. This change implements progress for the number of packfiles downloaded; a progress display for bytes would involve deeper changes at the http-fetch layer instead of fetch-pack, the caller, so we do not do that in this patch.

---
fetch: show progress for packfile uri downloads

Git appears to hang when downloading packfiles as this part of the fetch is silent, causing user confusion. This change implements progress for the number of packfiles downloaded; a progress display for bytes would involve deeper changes at the http-fetch layer instead of fetch-pack, the caller.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-907%2Falbertcui%2Fprogress-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-907/albertcui/progress-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/907

Range-diff vs v1:

1: d6cf8c195111 ! 1: aedb1b33d0c4 fetch: show progress for packfile uri downloads
@@ Commit message
fetch is silent, causing user confusion. This change implements progress for the number of packfiles downloaded; a progress display for bytes would involve deeper changes at the http-fetch layer
- instead of fetch-pack, the caller.
+ instead of fetch-pack, the caller, so we do not do that in this
+ patch.

@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
const char *uri = packfile_uris.items[i].string +
the_hash_algo->hexsz + 1;

-+ display_progress(packfile_uri_progress, i+1);
++ display_progress(packfile_uri_progress, i + 1);
strvec_push(&cmd.args, "http-fetch"); strvec_pushf(&cmd.args, "--packfile=%.*s",
(int) the_hash_algo->hexsz,
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,

## t/t5702-protocol-v2.sh ##
@@ t/t5702-protocol-v2.sh: test_expect_success 'part of packfile response provided as URI' '
- GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
+ configure_exclusion "$P" my-blob >h &&
+ configure_exclusion "$P" other-blob >h2 &&
+
+- GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
++ GIT_PROGRESS_DELAY=0 GIT_TRACE=1 GIT_TRACE2_EVENT=1 \
++ GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
git -c protocol.version=2 \
-c fetch.uriprotocols=http,https \
- clone "$HTTPD_URL/smart/http_parent" http_child &&

 fetch-pack.c           | 8 ++++++++
 t/t5702-protocol-v2.sh | 8 ++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)

index 6a61a464283e..7ca85f40cf23 100644
@@ -23,6 +23,7 @@
 #include "fetch-negotiator.h"
 #include "fsck.h"
 #include "shallow.h"
+#include "progress.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;

@@ -1585,6 +1586,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 	struct fetch_negotiator *negotiator;
 	int seen_ack = 0;
 	struct string_list packfile_uris = STRING_LIST_INIT_DUP;
+	struct progress *packfile_uri_progress;
 	int i;
 	struct strvec index_pack_args = STRVEC_INIT;
 	struct oidset gitmodules_oids = OIDSET_INIT;

@@ -1689,6 +1691,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		}
 	}
 
+	packfile_uri_progress = start_progress(_("Downloading packs"), packfile_uris.nr);
+
 	for (i = 0; i < packfile_uris.nr; i++) {
 		int j;
 		struct child_process cmd = CHILD_PROCESS_INIT;

@@ -1696,6 +1700,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		const char *uri = packfile_uris.items[i].string +
 			the_hash_algo->hexsz + 1;
 
+		display_progress(packfile_uri_progress, i + 1);
 		strvec_push(&cmd.args, "http-fetch");
 		strvec_pushf(&cmd.args, "--packfile=%.*s",
 			     (int) the_hash_algo->hexsz,

@@ -1739,6 +1744,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 						 get_object_directory(),
 						 packname));
 	}
+
+	stop_progress(&packfile_uri_progress);
+
 	string_list_clear(&packfile_uris, 0);
 	strvec_clear(&index_pack_args);
 

index 2e1243ca40b0..0476b3f50455 100755
@@ -848,10 +848,12 @@ test_expect_success 'part of packfile response provided as URI' '
 	configure_exclusion "$P" my-blob >h &&
 	configure_exclusion "$P" other-blob >h2 &&
 
-	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
+	GIT_PROGRESS_DELAY=0 GIT_TRACE=1 GIT_TRACE2_EVENT=1 \
+	GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
 	git -c protocol.version=2 \
 		-c fetch.uriprotocols=http,https \
-		clone "$HTTPD_URL/smart/http_parent" http_child &&
+		clone "$HTTPD_URL/smart/http_parent" http_child \
+		--progress 2>progress &&
 
 	# Ensure that my-blob and other-blob are in separate packfiles.
 	for idx in http_child/.git/objects/pack/*.idx

@@ -875,6 +877,8 @@ test_expect_success 'part of packfile response provided as URI' '
 	test -f hfound &&
 	test -f h2found &&
 
+	test_i18ngrep "Downloading packs" progress &&
+
 	# Ensure that there are exactly 3 packfiles with associated .idx
 	ls http_child/.git/objects/pack/*.pack \
 	    http_child/.git/objects/pack/*.idx >filelist &&

base-commit: a5828ae6b52137b913b978e16cd2334482eb4c1f

gitgitgadget

... meaning, hopefully later we'd hook into transport->progress and implement the byte-level progress display down there? And when that happens, we'd remove this file-level progress as it would be too confusing to have both at the same time?

Is this start_progress() call a way to unconditionally enable the progress display? How does it interact with transport->progress that is driven by transport_set_verbosity(), which in turn is called by builtin/fetch.c and friends? If it doesn't, shouldn't this codepath pay attention to the transport->progress and enable the progress meter only when it is enabled (i.e. the stderr going to a terminal, or --progress explicitly being asked)?

Hi,

This is something that came up at the last in-person Git Contributor Summit; I'm glad to see it being taken care of.

	if (!args->quiet && !args->no_progress)
		packfile_uri_progress = ...;

	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" \
	GIT_PROGRESS_DELAY=0 GIT_TRACE2_EVENT="$(pwd)/trace2" \
	GIT_TEST_SIDEBAND_ALL=1 \
	git -c [etc]