Skip to contentSkip to conversation

t5541: test more combinations of --progress

PATCH In review David Ebbo 6 participants 14 messages Jun 15, 2016 1 message not in the archive

lore
Share & tools
lore
#1 lore raw

Re: 1.7.10 doesn't show file pushstatus

In reply to a message not in the archive
Quoted 3 lines

I just updated to msysGit 1.7.10 and I noticed I don't see any details while pushing (like file upload speed and % completion). Was this intentionally removed? If so why?

No, it's a regression. I can reproduce it easily, and it bisects to Clemens' 01fdc21 (push/fetch/clone --no-progress suppresses progress output) which went into v1.7.9.2 (and v1.7.10).

The problematic hunk is:

builtin/send-pack.c +1 -1
index 71f258e..9df341c 100644
@@ -58,7 +58,7 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
                argv[i++] = "--thin";
        if (args->use_ofs_delta)
                argv[i++] = "--delta-base-offset";
-       if (args->quiet)
+       if (args->quiet || !args->progress)
                argv[i++] = "-q";
        if (args->progress)
                argv[i++] = "--progress";

which seems wrong to me. In send-pack, args->progress may be unset if we didn't get a --progress flag on the command line. Shouldn't we be falling back to isatty in that case (or leaving "-q" unset so that pack-objects can do so)? Does it need to also be converted into a tri-state of yes/no/unknown as the other places in that patch were? Clemens?

-Peff

Signature · 12 lines
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en
#2 lore raw

Re: 1.7.10 doesn't show file pushstatus

Quoted 7 lines · Jeff King
Quoted 3 lines

I just updated to msysGit 1.7.10 and I noticed I don't see any details while pushing (like file upload speed and % completion). Was this intentionally removed? If so why?

No, it's a regression. I can reproduce it easily, and it bisects to Clemens' 01fdc21 (push/fetch/clone --no-progress suppresses progress output) which went into v1.7.9.2 (and v1.7.10).

Hmm. OK, I think I see what is going on. For most protocols, send_pack happens without a separate process these days. So the stack is like:

- transport_push
- git_transport_push
- send_pack

where the "progress" flag to send_pack is a boolean; we have already figured out at the transport layer whether we want progress, and are telling it what to do. Thus this hunk from 01fdc21 makes sense:

builtin/send-pack.c static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext · line 64

Quoted 13 lines
builtin/send-pack.c +1 -1
index 71f258e..9df341c 100644
@@ -58,7 +58,7 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
                argv[i++] = "--thin";
        if (args->use_ofs_delta)
                argv[i++] = "--delta-base-offset";
-       if (args->quiet)
+       if (args->quiet || !args->progress)
                argv[i++] = "-q";
        if (args->progress)
                argv[i++] = "--progress";

If we have been asked not to have progress, then we tell pack-objects "-q" (which is, for historical reasons, the way to spell "--no-progress" for pack-objects).

But send-pack is also a command in its own right, and gets invoked separately for the --stateless-rpc case (i.e., smart http). In that case you end up with:

- transport_push
- transport-helper.c:push_refs
- [pipes to git-remote-https]
- remote-curl.c:push_git
- [forks send-pack]
- cmd_send_pack
- send_pack

In this case, send pack gets its arguments from the command-line, not from the options set at the transport layer. Remote-curl will pass along "--quiet" if we get that from the transport layer, but it does not otherwise pass along the "progress" flag. So there are two problems:

1. send-pack defaults its progress boolean to 0. Before 01fdc21, that was OK, because it meant "don't explicitly ask for progress". But after 01fdc21 that now means "explicitly ask for no progress", and the direct-transport code paths were updated without updating cmd_send_pack.

2. There's no way to tell send-pack explicitly "yes, I would like progress, no matter what isatty(2) says". I doubt anybody cares much, but it probably makes sense to handle that for the sake of completeness.

-Peff

Signature · 12 lines
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en
#3 lore raw

Re: 1.7.10 doesn't show file pushstatus

Quoted 11 lines · Jeff King

In this case, send pack gets its arguments from the command-line, not from the options set at the transport layer. Remote-curl will pass along "--quiet" if we get that from the transport layer, but it does not otherwise pass along the "progress" flag. So there are two problems:

1. send-pack defaults its progress boolean to 0. Before 01fdc21, that was OK, because it meant "don't explicitly ask for progress". But after 01fdc21 that now means "explicitly ask for no progress", and the direct-transport code paths were updated without updating cmd_send_pack.

Quoted 4 lines

2. There's no way to tell send-pack explicitly "yes, I would like progress, no matter what isatty(2) says". I doubt anybody cares much, but it probably makes sense to handle that for the sake of completeness.

The following patch series fixes this:

  [1/3]: send-pack: show progress when isatty(2)
  [2/3]: teach send-pack about --[no-]progress
  [3/3]: t5541: test more combinations of --progress

The first patch fixes (1) above, restoring send-pack's original behavior (for remote-curl as well as anybody else who happens to call it). Note that in doing so, it breaks "push --no-progress" for http remotes that 01fdc21 tried to fix. But it didn't actually fix it; it only appeared to work because progress was _never_ on for http. Fortunately, the existing test in t5541 still passes because it's poorly written (it uses both "--quiet" and "--no-progress", unmindful of the fact that the latter does absolutely nothing).

The second patch fixes it correctly by propagating the options through remote-curl (and as a bonus, it makes (2) above work). Other transport helpers that use send-pack would want to do the same thing (but I don't know of any that exist).

The third patch just expands the test coverage.

These are prepared directly on top of 01fdc21, so they can go on the maint track.

-Peff

Signature · 12 lines
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en
#4 lore raw

PATCH 1/3 send-pack: show progress when isatty(2)

The send_pack_args struct has two verbosity flags: "quiet" and "progress". Originally, if "quiet" was set, we would tell pack-objects explicitly to be quiet, and if "progress" was set, we would tell it to show progress. Otherwise, we told it neither, and it relied on isatty(2) to make the decision itself.

However, commit 01fdc21 changed the meaning of these variables. Now both "quiet" and "!progress" instruct us to tell pack-objects to be quiet (and a non-zero "progress" means the same as before). This works well for transports which call send_pack directly, as the transport code copies transport->progress into send_pack_args->progress, and they both have the same meaning.

However, the code path of calling "git send-pack" was left behind. It always sets "progress" to 0, and thus always tells pack-objects to be quiet. We can work around this by checking isatty(2) ourselves in the cmd_send_pack code path, restoring the original behavior of the send-pack command.

---
 builtin/send-pack.c |    3 +++
 1 file changed, 3 insertions(+)
builtin/send-pack.c +3 -0
index 9df341c..7d22715 100644
@@ -492,6 +492,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		}
 	}
 
+	if (!args.quiet)
+		args.progress = isatty(2);
+
 	if (args.stateless_rpc) {
 		conn = NULL;
 		fd[0] = 0;
Signature · 15 lines
1.7.10.630.g31718

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en
#5 lore raw

PATCH 2/3 teach send-pack about --[no-]progress

In reply to Jeff King

The send_pack function gets a "progress" flag saying "yes, definitely show progress" or "no, definitely do not show progress". This gets set properly by transport_push when send_pack is called directly.

However, when the send-pack command is executed separately (as it is for the remote-curl helper), there is no way to tell it "definitely do this". As a result, we do not properly respect "git push --no-progress" for smart-http remotes; you will still get progress if stderr is a tty.

This patch teaches send-pack --progress and --no-progress, and teaches remote-curl to pass the appropriate option to override send-pack's isatty check. This fixes the --no-progress case above, and as a bonus, also makes "git push --progress" work when stderr is not a tty.

---
 builtin/send-pack.c |   14 ++++++++++++--
 remote-curl.c       |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
builtin/send-pack.c +12 -2
index 7d22715..d5d7105 100644
@@ -410,6 +410,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	const char *receivepack = "git-receive-pack";
 	int flags;
 	int nonfastforward = 0;
+	int progress = -1;
 
 	argv++;
 	for (i = 1; i < argc; i++, argv++) {
@@ -452,6 +453,14 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 				args.verbose = 1;
 				continue;
 			}
+			if (!strcmp(arg, "--progress")) {
+				progress = 1;
+				continue;
+			}
+			if (!strcmp(arg, "--no-progress")) {
+				progress = 0;
+				continue;
+			}
 			if (!strcmp(arg, "--thin")) {
 				args.use_thin_pack = 1;
 				continue;
@@ -492,8 +501,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		}
 	}
 
-	if (!args.quiet)
-		args.progress = isatty(2);
+	if (progress == -1)
+		progress = !args.quiet && isatty(2);
+	args.progress = progress;
 
 	if (args.stateless_rpc) {
 		conn = NULL;
remote-curl.c +1 -0
index d159fe7..e5e9490 100644
@@ -774,6 +774,7 @@ static int push_git(struct discovery *heads, int nr_spec, char **specs)
 		argv[argc++] = "--quiet";
 	else if (options.verbosity > 1)
 		argv[argc++] = "--verbose";
+	argv[argc++] = options.progress ? "--progress" : "--no-progress";
 	argv[argc++] = url;
 	for (i = 0; i < nr_spec; i++)
 		argv[argc++] = specs[i];
Signature · 1 line
1.7.10.630.g31718
#6 lore raw

PATCH 3/3 t5541: test more combinations of --progress

In reply to Jeff King

Previously, we tested only that "push --quiet --no-progress" was silent. However, there are many other combinations that were not tested:

1. no options at all (but stderr as a tty)
2. --no-progress by itself
3. --quiet by itself
4. --progress (when stderr not a tty)

These are tested elsewhere for general "push", but it is important to test them separately for http. It follows a very different code path than git://, and options must be relayed across a remote helper to a separate send-pack process (and in fact cases (1), (2), and (4) have all been broken just for http at some point in the past).

We can drop the "--quiet --no-progress" test, as it is not really interesting (it is already handled by testing them separately in (2) and (3) above).

---
 t/t5541-http-push.sh |   27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)
t/t5541-http-push.sh +25 -2
index d66ed24..a1b10bd 100755
@@ -215,12 +215,35 @@ test_expect_success 'push --mirror to repo with alternates' '
 	git push --mirror "$HTTPD_URL"/smart/alternates-mirror.git
 '
 
-test_expect_success TTY 'quiet push' '
+test_expect_success TTY 'push shows progress when stderr is a tty' '
+	cd "$ROOT_PATH"/test_repo_clone &&
+	test_commit noisy &&
+	test_terminal git push 2>&1 | tee output &&
+	grep "^Writing objects" output
+'
+
+test_expect_success TTY 'push --quiet silences status and progress' '
 	cd "$ROOT_PATH"/test_repo_clone &&
 	test_commit quiet &&
-	test_terminal git push --quiet --no-progress 2>&1 | tee output &&
+	test_terminal git push --quiet 2>&1 | tee output &&
 	test_cmp /dev/null output
 '
 
+test_expect_success TTY 'push --no-progress silences progress but not status' '
+	cd "$ROOT_PATH"/test_repo_clone &&
+	test_commit no-progress &&
+	test_terminal git push --no-progress 2>&1 | tee output &&
+	grep "^To http" output &&
+	! grep "^Writing objects"
+'
+
+test_expect_success 'push --progress shows progress to non-tty' '
+	cd "$ROOT_PATH"/test_repo_clone &&
+	test_commit progress &&
+	git push --progress 2>&1 | tee output &&
+	grep "^To http" output &&
+	grep "^Writing objects" output
+'
+
 stop_httpd
 test_done
Signature · 15 lines
1.7.10.630.g31718

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en
#7 lore raw

Can we add this on top or squashed in? I regret using tee when I originally wrote the test.

--8<--
Subject: [PATCH] t5541: check return codes

By piping output to tee, the return code of the command is hidden. Instead, redirect output to a file directly.

---
 t/t5541-http-push.sh |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
t/t5541-http-push.sh +4 -4
index 986210a..c07973e 100755
@@ -218,21 +218,21 @@ test_expect_success 'push --mirror to repo with alternates' '
 test_expect_success TTY 'push shows progress when stderr is a tty' '
 	cd "$ROOT_PATH"/test_repo_clone &&
 	test_commit noisy &&
-	test_terminal git push 2>&1 | tee output &&
+	test_terminal git push >output 2>&1 &&
 	grep "^Writing objects" output
 '
 
 test_expect_success TTY 'push --quiet silences status and progress' '
 	cd "$ROOT_PATH"/test_repo_clone &&
 	test_commit quiet &&
-	test_terminal git push --quiet 2>&1 | tee output &&
+	test_terminal git push --quiet >output 2>&1 &&
 	test_cmp /dev/null output
 '
 
 test_expect_success TTY 'push --no-progress silences progress but not status' '
 	cd "$ROOT_PATH"/test_repo_clone &&
 	test_commit no-progress &&
-	test_terminal git push --no-progress 2>&1 | tee output &&
+	test_terminal git push --no-progress >output 2>&1 &&
 	grep "^To http" output &&
 	! grep "^Writing objects"
 '
@@ -240,7 +240,7 @@ test_expect_success TTY 'push --no-progress silences progress but not status' '
 test_expect_success 'push --progress shows progress to non-tty' '
 	cd "$ROOT_PATH"/test_repo_clone &&
 	test_commit progress &&
-	git push --progress 2>&1 | tee output &&
+	git push --progress >output 2>&1 &&
 	grep "^To http" output &&
 	grep "^Writing objects" output
 '
Signature · 1 line
1.7.10
#8 lore raw
Quoted 8 lines · Clemens Buchacher

Can we add this on top or squashed in? I regret using tee when I originally wrote the test.

--8<--
Subject: [PATCH] t5541: check return codes

By piping output to tee, the return code of the command is hidden. Instead, redirect output to a file directly.

Ugh, absolutely. I blindly followed the existing style without thinking about whether using tee was sane or not. And it's not. Thanks.

-Peff

Signature · 12 lines
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en
author
#9 lore raw

Many thanks to Jeff King and others to help track this one down!

David

Signature · 12 lines
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en
#10 lore raw
In reply to Jeff King
Quoted 12 lines · Jeff King
t/t5541-http-push.sh +8 -2
-test_expect_success TTY 'quiet push' '
+test_expect_success TTY 'push shows progress when stderr is a tty' '
+	cd "$ROOT_PATH"/test_repo_clone &&
+	test_commit noisy &&
+	test_terminal git push 2>&1 | tee output &&
+	grep "^Writing objects" output
+'
+
+test_expect_success TTY 'push --quiet silences status and progress' '
 	cd "$ROOT_PATH"/test_repo_clone &&
 	test_commit quiet &&
-	test_terminal git push --quiet --no-progress 2>&1 | tee output &&

t/t5541-http-push.sh test_expect_success 'push --mirror to repo with alternates' ' · line 237

Quoted 10 lines
t/t5541-http-push.sh +7 -0
+	test_terminal git push --quiet 2>&1 | tee output &&
 	test_cmp /dev/null output
 '
 
+test_expect_success TTY 'push --no-progress silences progress but not status' '
+	cd "$ROOT_PATH"/test_repo_clone &&
+	test_commit no-progress &&
+	test_terminal git push --no-progress 2>&1 | tee output &&
+	grep "^To http" output &&
+	! grep "^Writing objects"

! grep "^Writing objects" output

t/t5541-http-push.sh test_expect_success 'push --mirror to repo with alternates' ' · line 247

Quoted 10 lines

+'

t/t5541-http-push.sh +9 -0
+
+test_expect_success 'push --progress shows progress to non-tty' '
+	cd "$ROOT_PATH"/test_repo_clone &&
+	test_commit progress &&
+	git push --progress 2>&1 | tee output &&
+	grep "^To http" output &&
+	grep "^Writing objects" output
+'
+

I understand that test_i18ngrep is not necessary, because pack-objects.c is not internationalized. But wouldn't it make sense to use test_i18ngrep in preparation, so that tests don't have to be modified later on?

-
Zbyszek

#11 lore raw
Quoted 4 lines · Zbigniew Jędrzejewski-Szmek
Quoted 4 lines
t/t5541-http-push.sh +4 -0
+test_expect_success 'push --progress shows progress to non-tty' '
+	cd "$ROOT_PATH"/test_repo_clone &&
+	test_commit progress &&
+	git push --progress 2>&1 | tee output &&
Quoted 8 lines

t/t5541-http-push.sh test_expect_success 'push --mirror to repo with alternates' ' · line 247

Quoted 4 lines
t/t5541-http-push.sh +4 -0
+	grep "^To http" output &&
+	grep "^Writing objects" output
+'
+

I understand that test_i18ngrep is not necessary, because pack-objects.c is not internationalized. But wouldn't it make sense to use test_i18ngrep in preparation, so that tests don't have to be modified later on?

In this test, we are not interested in making sure the progress output is properly localized. I'd rather see it keep using grep and if you really care, run "git push" under "LANG=C LC_ALL=C" or something reliable instead.

Signature · 12 lines
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en
#12 lore raw

Re: 1.7.10 doesn't show file pushstatus

In reply to Jeff King
Quoted 6 lines · Jeff King

The following patch series fixes this:

  [1/3]: send-pack: show progress when isatty(2)
  [2/3]: teach send-pack about --[no-]progress
  [3/3]: t5541: test more combinations of --progress

Thanks. Looks good to me, although I completely missed this before. But the tests should hopefully take care of any future regressions.

#14 lore raw

Re: Re: 1.7.10 doesn't show file pushstatus

In reply to Jeff King

Hi Peff,

Quoted 18 lines · Jeff King
Quoted 11 lines · Jeff King

In this case, send pack gets its arguments from the command-line, not from the options set at the transport layer. Remote-curl will pass along "--quiet" if we get that from the transport layer, but it does not otherwise pass along the "progress" flag. So there are two problems:

1. send-pack defaults its progress boolean to 0. Before 01fdc21, that was OK, because it meant "don't explicitly ask for progress". But after 01fdc21 that now means "explicitly ask for no progress", and the direct-transport code paths were updated without updating cmd_send_pack.

Quoted 4 lines

2. There's no way to tell send-pack explicitly "yes, I would like progress, no matter what isatty(2) says". I doubt anybody cares much, but it probably makes sense to handle that for the sake of completeness.

Quoted 5 lines

The following patch series fixes this:

  [1/3]: send-pack: show progress when isatty(2)
  [2/3]: teach send-pack about --[no-]progress
  [3/3]: t5541: test more combinations of --progress

Thanks. I applied and pushed it after testing on Linux.

David, if you want to relieve me of maintenance burden in the future, you

# apply the patches
# push them to a fork on GitHub
# run the tests
# report back to the mailing list (including the outcome of the test --
  detailed, if it fails)

Ciao,
Johannes

Signature · 12 lines
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en