t5541: test more combinations of --progress
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: 1.7.10 doesn't show file pushstatus
Quoted 3 lines
I just updated to msysGit 1.7.10 and I noticed I don't see any details while pushing (like file upload speed and % completion). Was this intentionally removed? If so why?
No, it's a regression. I can reproduce it easily, and it bisects to Clemens' 01fdc21 (push/fetch/clone --no-progress suppresses progress output) which went into v1.7.9.2 (and v1.7.10).
The problematic hunk is:
builtin/send-pack.c +1 -1
@@ -58,7 +58,7 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext argv[i++] ="--thin" ;if (args->use_ofs_delta) argv[i++] ="--delta-base-offset" ; -if (args->quiet) +if (args->quiet || !args->progress) argv[i++] ="-q" ;if (args->progress) argv[i++] ="--progress" ;
which seems wrong to me. In send-pack, args->progress may be unset if we didn't get a --progress flag on the command line. Shouldn't we be falling back to isatty in that case (or leaving "-q" unset so that pack-objects can do so)? Does it need to also be converted into a tri-state of yes/no/unknown as the other places in that patch were? Clemens?
-Peff
Signature · 12 lines
*** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en
Re: 1.7.10 doesn't show file pushstatus
Quoted 7 lines · Jeff King
Quoted 3 lines
I just updated to msysGit 1.7.10 and I noticed I don't see any details while pushing (like file upload speed and % completion). Was this intentionally removed? If so why?
No, it's a regression. I can reproduce it easily, and it bisects to Clemens' 01fdc21 (push/fetch/clone --no-progress suppresses progress output) which went into v1.7.9.2 (and v1.7.10).
Hmm. OK, I think I see what is going on. For most protocols, send_pack happens without a separate process these days. So the stack is like:
- transport_push
- git_transport_push
- send_pack
where the "progress" flag to send_pack is a boolean; we have already figured out at the transport layer whether we want progress, and are telling it what to do. Thus this hunk from 01fdc21 makes sense:
builtin/send-pack.c static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext · line 64
Quoted 13 lines
builtin/send-pack.c +1
-1@@ -58,7 +58,7 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext argv[i++] ="--thin" ;if (args->use_ofs_delta) argv[i++] ="--delta-base-offset" ; -if (args->quiet) +if (args->quiet || !args->progress) argv[i++] ="-q" ;if (args->progress) argv[i++] ="--progress" ;
If we have been asked not to have progress, then we tell pack-objects "-q" (which is, for historical reasons, the way to spell "--no-progress" for pack-objects).
But send-pack is also a command in its own right, and gets invoked separately for the --stateless-rpc case (i.e., smart http). In that case you end up with:
- transport_push
- transport-helper.c:push_refs
- [pipes to git-remote-https]
- remote-curl.c:push_git
- [forks send-pack]
- cmd_send_pack
- send_pack
In this case, send pack gets its arguments from the command-line, not from the options set at the transport layer. Remote-curl will pass along "--quiet" if we get that from the transport layer, but it does not otherwise pass along the "progress" flag. So there are two problems:
1. send-pack defaults its progress boolean to 0. Before 01fdc21, that was OK, because it meant "don't explicitly ask for progress". But after 01fdc21 that now means "explicitly ask for no progress", and the direct-transport code paths were updated without updating cmd_send_pack.
2. There's no way to tell send-pack explicitly "yes, I would like progress, no matter what isatty(2) says". I doubt anybody cares much, but it probably makes sense to handle that for the sake of completeness.
-Peff
Signature · 12 lines
*** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en
Re: 1.7.10 doesn't show file pushstatus
Quoted 11 lines · Jeff King
In this case, send pack gets its arguments from the command-line, not from the options set at the transport layer. Remote-curl will pass along "--quiet" if we get that from the transport layer, but it does not otherwise pass along the "progress" flag. So there are two problems:
1. send-pack defaults its progress boolean to 0. Before 01fdc21, that was OK, because it meant "don't explicitly ask for progress". But after 01fdc21 that now means "explicitly ask for no progress", and the direct-transport code paths were updated without updating
cmd_send_pack.
Quoted 4 lines
2. There's no way to tell send-pack explicitly "yes, I would like progress, no matter what isatty(2) says". I doubt anybody cares much, but it probably makes sense to handle that for the sake of completeness.
The following patch series fixes this:
[1/3]: send-pack: show progress when isatty(2) [2/3]: teach send-pack about --[no-]progress [3/3]: t5541: test more combinations of --progress
The first patch fixes (1) above, restoring send-pack's original behavior (for remote-curl as well as anybody else who happens to call it). Note that in doing so, it breaks "push --no-progress" for http remotes that 01fdc21 tried to fix. But it didn't actually fix it; it only appeared to work because progress was _never_ on for http. Fortunately, the existing test in t5541 still passes because it's poorly written (it uses both "--quiet" and "--no-progress", unmindful of the fact that the latter does absolutely nothing).
The second patch fixes it correctly by propagating the options through remote-curl (and as a bonus, it makes (2) above work). Other transport helpers that use send-pack would want to do the same thing (but I don't know of any that exist).
The third patch just expands the test coverage.
These are prepared directly on top of 01fdc21, so they can go on the maint track.
-Peff
Signature · 12 lines
*** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en
PATCH 1/3 send-pack: show progress when isatty(2)
The send_pack_args struct has two verbosity flags: "quiet" and "progress". Originally, if "quiet" was set, we would tell pack-objects explicitly to be quiet, and if "progress" was set, we would tell it to show progress. Otherwise, we told it neither, and it relied on isatty(2) to make the decision itself.
However, commit 01fdc21 changed the meaning of these variables. Now both "quiet" and "!progress" instruct us to tell pack-objects to be quiet (and a non-zero "progress" means the same as before). This works well for transports which call send_pack directly, as the transport code copies transport->progress into send_pack_args->progress, and they both have the same meaning.
However, the code path of calling "git send-pack" was left behind. It always sets "progress" to 0, and thus always tells pack-objects to be quiet. We can work around this by checking isatty(2) ourselves in the cmd_send_pack code path, restoring the original behavior of the send-pack command.
--- builtin/send-pack.c | 3 +++ 1 file changed, 3 insertions(+)
builtin/send-pack.c +3 -0
@@ -492,6 +492,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix) } } +if (!args.quiet) + args.progress =isatty (2 ); +if (args.stateless_rpc) { conn =NULL ; fd[0 ] =0 ;
Signature · 15 lines
1.7.10.630.g31718 -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en
PATCH 2/3 teach send-pack about --[no-]progress
In reply to Jeff KingThe send_pack function gets a "progress" flag saying "yes, definitely show progress" or "no, definitely do not show progress". This gets set properly by transport_push when send_pack is called directly.
However, when the send-pack command is executed separately (as it is for the remote-curl helper), there is no way to tell it "definitely do this". As a result, we do not properly respect "git push --no-progress" for smart-http remotes; you will still get progress if stderr is a tty.
This patch teaches send-pack --progress and --no-progress, and teaches remote-curl to pass the appropriate option to override send-pack's isatty check. This fixes the --no-progress case above, and as a bonus, also makes "git push --progress" work when stderr is not a tty.
--- builtin/send-pack.c | 14 ++++++++++++-- remote-curl.c | 1 + 2 files changed, 13 insertions(+), 2 deletions(-)
builtin/send-pack.c +12 -2
@@ -410,6 +410,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)const char *receivepack ="git-receive-pack" ;int flags;int nonfastforward =0 ; +int progress =-1 ; argv++;for (i =1 ; i < argc; i++, argv++) {
@@ -452,6 +453,14 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix) args.verbose =1 ;continue ; } +if (!strcmp (arg,"--progress" )) { + progress =1 ; +continue ; + } +if (!strcmp (arg,"--no-progress" )) { + progress =0 ; +continue ; + }if (!strcmp (arg,"--thin" )) { args.use_thin_pack =1 ;continue ;
@@ -492,8 +501,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix) } } -if (!args.quiet) - args.progress =isatty (2 ); +if (progress ==-1 ) + progress = !args.quiet &&isatty (2 ); + args.progress = progress;if (args.stateless_rpc) { conn =NULL ;
remote-curl.c +1 -0
@@ -774,6 +774,7 @@ static int push_git(struct discovery *heads, int nr_spec, char **specs) argv[argc++] ="--quiet" ; else if (options.verbosity > 1) argv[argc++] ="--verbose" ; + argv[argc++] = options.progress ?"--progress" :"--no-progress" ; argv[argc++] = url;for (i =0 ; i < nr_spec; i++) argv[argc++] = specs[i];
Signature · 1 line
1.7.10.630.g31718
PATCH 3/3 t5541: test more combinations of --progress
In reply to Jeff KingPreviously, we tested only that "push --quiet --no-progress" was silent. However, there are many other combinations that were not tested:
1. no options at all (but stderr as a tty)
2. --no-progress by itself
3. --quiet by itself
4. --progress (when stderr not a tty)
These are tested elsewhere for general "push", but it is important to test them separately for http. It follows a very different code path than git://, and options must be relayed across a remote helper to a separate send-pack process (and in fact cases (1), (2), and (4) have all been broken just for http at some point in the past).
We can drop the "--quiet --no-progress" test, as it is not really interesting (it is already handled by testing them separately in (2) and (3) above).
--- t/t5541-http-push.sh | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
t/t5541-http-push.sh +25 -2
@@ -215,12 +215,35 @@ test_expect_success 'push --mirror to repo with alternates' ' git push --mirror "$HTTPD_URL"/smart/alternates-mirror.git ' -test_expect_success TTY 'quiet push' ' +test_expect_success TTY 'push shows progress when stderr is a tty' ' + cd "$ROOT_PATH"/test_repo_clone && + test_commit noisy && + test_terminal git push 2>&1 | tee output && + grep "^Writing objects" output +' + +test_expect_success TTY 'push --quiet silences status and progress' ' cd "$ROOT_PATH"/test_repo_clone && test_commit quiet && - test_terminal git push --quiet --no-progress 2>&1 | tee output && + test_terminal git push --quiet 2>&1 | tee output && test_cmp /dev/null output ' +test_expect_success TTY 'push --no-progress silences progress but not status' ' + cd "$ROOT_PATH"/test_repo_clone && + test_commit no-progress && + test_terminal git push --no-progress 2>&1 | tee output && + grep "^To http" output && + ! grep "^Writing objects" +' + +test_expect_success 'push --progress shows progress to non-tty' ' + cd "$ROOT_PATH"/test_repo_clone && + test_commit progress && + git push --progress 2>&1 | tee output && + grep "^To http" output && + grep "^Writing objects" output +' + stop_httpd test_done
Signature · 15 lines
1.7.10.630.g31718 -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en
Can we add this on top or squashed in? I regret using tee when I originally wrote the test.
--8<--
Subject: [PATCH] t5541: check return codes
By piping output to tee, the return code of the command is hidden. Instead, redirect output to a file directly.
--- t/t5541-http-push.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
t/t5541-http-push.sh +4 -4
@@ -218,21 +218,21 @@ test_expect_success 'push --mirror to repo with alternates' ' test_expect_success TTY 'push shows progress when stderr is a tty' ' cd "$ROOT_PATH"/test_repo_clone && test_commit noisy && - test_terminal git push 2>&1 | tee output && + test_terminal git push >output 2>&1 && grep "^Writing objects" output ' test_expect_success TTY 'push --quiet silences status and progress' ' cd "$ROOT_PATH"/test_repo_clone && test_commit quiet && - test_terminal git push --quiet 2>&1 | tee output && + test_terminal git push --quiet >output 2>&1 && test_cmp /dev/null output ' test_expect_success TTY 'push --no-progress silences progress but not status' ' cd "$ROOT_PATH"/test_repo_clone && test_commit no-progress && - test_terminal git push --no-progress 2>&1 | tee output && + test_terminal git push --no-progress >output 2>&1 && grep "^To http" output && ! grep "^Writing objects" '
@@ -240,7 +240,7 @@ test_expect_success TTY 'push --no-progress silences progress but not status' ' test_expect_success 'push --progress shows progress to non-tty' ' cd "$ROOT_PATH"/test_repo_clone && test_commit progress && - git push --progress 2>&1 | tee output && + git push --progress >output 2>&1 && grep "^To http" output && grep "^Writing objects" output '
Signature · 1 line
1.7.10
Quoted 8 lines · Clemens Buchacher
Can we add this on top or squashed in? I regret using tee when I originally wrote the test.
--8<--
Subject: [PATCH] t5541: check return codesBy piping output to tee, the return code of the command is hidden. Instead, redirect output to a file directly.
Ugh, absolutely. I blindly followed the existing style without thinking about whether using tee was sane or not. And it's not. Thanks.
-Peff
Signature · 12 lines
*** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en
Many thanks to Jeff King and others to help track this one down!
David
Signature · 12 lines
*** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en
Quoted 12 lines · Jeff King
t/t5541-http-push.sh +8
-2-test_expect_success TTY 'quiet push' ' +test_expect_success TTY 'push shows progress when stderr is a tty' ' + cd "$ROOT_PATH"/test_repo_clone && + test_commit noisy && + test_terminal git push 2>&1 | tee output && + grep "^Writing objects" output +' + +test_expect_success TTY 'push --quiet silences status and progress' ' cd "$ROOT_PATH"/test_repo_clone && test_commit quiet && - test_terminal git push --quiet --no-progress 2>&1 | tee output &&
t/t5541-http-push.sh test_expect_success 'push --mirror to repo with alternates' ' · line 237
Quoted 10 lines
t/t5541-http-push.sh +7
-0+ test_terminal git push --quiet 2>&1 | tee output && test_cmp /dev/null output ' +test_expect_success TTY 'push --no-progress silences progress but not status' ' + cd "$ROOT_PATH"/test_repo_clone && + test_commit no-progress && + test_terminal git push --no-progress 2>&1 | tee output && + grep "^To http" output && + ! grep "^Writing objects"
! grep "^Writing objects" output
t/t5541-http-push.sh test_expect_success 'push --mirror to repo with alternates' ' · line 247
Quoted 10 lines
+'
t/t5541-http-push.sh +9
-0+ +test_expect_success 'push --progress shows progress to non-tty' ' + cd "$ROOT_PATH"/test_repo_clone && + test_commit progress && + git push --progress 2>&1 | tee output && + grep "^To http" output && + grep "^Writing objects" output +' +
I understand that test_i18ngrep is not necessary, because pack-objects.c is not internationalized. But wouldn't it make sense to use test_i18ngrep in preparation, so that tests don't have to be modified later on?
-
Zbyszek
Quoted 4 lines · Zbigniew Jędrzejewski-Szmek
Quoted 4 lines
t/t5541-http-push.sh +4
-0+test_expect_success 'push --progress shows progress to non-tty' ' + cd "$ROOT_PATH"/test_repo_clone && + test_commit progress && + git push --progress 2>&1 | tee output &&
Quoted 8 lines
t/t5541-http-push.sh test_expect_success 'push --mirror to repo with alternates' ' · line 247
Quoted 4 lines
t/t5541-http-push.sh +4
-0+ grep "^To http" output && + grep "^Writing objects" output +' +I understand that
test_i18ngrepis not necessary, becausepack-objects.cis not internationalized. But wouldn't it make sense to usetest_i18ngrepin preparation, so that tests don't have to be modified later on?
In this test, we are not interested in making sure the progress output is properly localized. I'd rather see it keep using grep and if you really care, run "git push" under "LANG=C LC_ALL=C" or something reliable instead.
Signature · 12 lines
*** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en
Re: 1.7.10 doesn't show file pushstatus
In reply to Jeff KingQuoted 6 lines · Jeff King
The following patch series fixes this:
[1/3]: send-pack: show progress when isatty(2) [2/3]: teach send-pack about --[no-]progress [3/3]: t5541: test more combinations of --progress
Thanks. Looks good to me, although I completely missed this before. But the tests should hopefully take care of any future regressions.
Re: 1.7.10 doesn't show file pushstatus
Thanks, both.
Re: Re: 1.7.10 doesn't show file pushstatus
In reply to Jeff KingHi Peff,
Quoted 18 lines · Jeff King
Quoted 11 lines · Jeff King
In this case, send pack gets its arguments from the command-line, not from the options set at the transport layer. Remote-curl will pass along "--quiet" if we get that from the transport layer, but it does not otherwise pass along the "progress" flag. So there are two problems:
1. send-pack defaults its progress boolean to 0. Before 01fdc21, that was OK, because it meant "don't explicitly ask for progress". But after 01fdc21 that now means "explicitly ask for no progress", and the direct-transport code paths were updated without updating
cmd_send_pack.Quoted 4 lines
2. There's no way to tell send-pack explicitly "yes, I would like progress, no matter what isatty(2) says". I doubt anybody cares much, but it probably makes sense to handle that for the sake of completeness.
Quoted 5 lines
The following patch series fixes this:
[1/3]: send-pack: show progress when isatty(2) [2/3]: teach send-pack about --[no-]progress [3/3]: t5541: test more combinations of --progress
Thanks. I applied and pushed it after testing on Linux.
David, if you want to relieve me of maintenance burden in the future, you
# apply the patches
# push them to a fork on GitHub
# run the tests
# report back to the mailing list (including the outcome of the test -- detailed, if it fails)
Ciao,
Johannes
Signature · 12 lines
*** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en