Test failures with GNU grep 2.23
It seems that binary file detection has changed in GNU grep 2.23 as a result of commit 40ed879 (grep: fix bug with with invalid unibyte sequence).
This causes a couple of test failures in t8005 and t9200 (the t9200 case is less obvious so I'm only including t8005 here):
-- >8 --
$ ./t8005-blame-i18n.sh -v -i
[snip]
expecting success:
git blame --incremental file | \
egrep "^(author|summary) " > actual &&
test_cmp actual expectedexpected 2016-02-07 16:14:55.359510341 +0000 renamed from actual 2016-02-07 16:14:55.372510307 +0000 +6 -1
@@ -1 +1,6 @@ -Binary file (standard input) matches +author �R�c ���Y +summary �u���[���̃e�X�g�ł��B +author �R�c ���Y +summary �u���[���̃e�X�g�ł��B +author �R�c ���Y +summary �u���[���̃e�X�g�ł��B
not ok 2 - blame respects i18n.commitencoding
#
# git blame --incremental file | \
# egrep "^(author|summary) " > actual &&
# test_cmp actual expected
#
-- 8< --
The following patch fixes the tests for me, but I wonder if "-a" is supported on all target platforms (it's not in POSIX, which specifies that the "input files shall be text files") or whether we should do something more comprehensive to provide sane_{e,f,}grep which guarantee to treat input as text.
I also tried setting POSIXLY_CORRECT but that doesn't affect the text/binary decision.
-- >8 --
t/t8005-blame-i18n.sh +4 -4
@@ -36,7 +36,7 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' git blame --incremental file | \ - egrep "^(author|summary) " > actual && + egrep -a "^(author|summary) " > actual && test_cmp actual expected '
@@ -53,7 +53,7 @@ test_expect_success !MINGW \ 'blame respects i18n.logoutputencoding' ' git config i18n.logoutputencoding eucJP && git blame --incremental file | \ - egrep "^(author|summary) " > actual && + egrep -a "^(author|summary) " > actual && test_cmp actual expected '
@@ -69,7 +69,7 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=UTF-8' ' git blame --incremental --encoding=UTF-8 file | \ - egrep "^(author|summary) " > actual && + egrep -a "^(author|summary) " > actual && test_cmp actual expected '
@@ -85,7 +85,7 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=none' ' git blame --incremental --encoding=none file | \ - egrep "^(author|summary) " > actual && + egrep -a "^(author|summary) " > actual && test_cmp actual expected '
t/t9200-git-cvsexportcommit.sh +1 -1
@@ -35,7 +35,7 @@ exit 1 check_entries () { # $1 == directory, $2 == expected - grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual + grep -a '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual if test -z "$2" then >expected
Quoted 3 lines · John Keeping
It seems that binary file detection has changed in GNU grep 2.23 as a result of commit 40ed879 (grep: fix bug with with invalid unibyte sequence).
I read this bug report a while ago when you posted it, but happily ignored it until today, when my debian unstable system pulled in the new version of grep. :)
Quoted 10 lines
This causes a couple of test failures in t8005 and t9200 (the t9200 case is less obvious so I'm only including t8005 here):
-- >8 --
$ ./t8005-blame-i18n.sh -v -i [snip] expecting success: git blame --incremental file | \ egrep "^(author|summary) " > actual && test_cmp actual expected
Just a side note while we are touching these tests:
- we probably should not pipe, so we check the exist code from git-blame
- we usually flip the test_cmp file order, to show the difference from expectation when there is a failure
- no space after ">" redirection :)
Quoted 8 lines
The following patch fixes the tests for me, but I wonder if "-a" is supported on all target platforms (it's not in POSIX, which specifies that the "input files shall be text files") or whether we should do something more comprehensive to provide sane_{e,f,}grep which guarantee to treat input as text.
I also tried setting
POSIXLY_CORRECTbut that doesn't affect the text/binary decision.
Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered. We could do:
test_lazy_prereq GREP_A '
echo foo | grep -a foo
'
and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).
Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.
I also wondered whether we could get away without grepping at all here. But the blame output has a bunch of cruft we don't care about; I think the readability of the tests would suffer if we tried to match the whole thing in a test_cmp.
-Peff
Quoted 7 lines · Jeff King
Quoted 6 lines · John Keeping
The following patch fixes the tests for me, but I wonder if "-a" is supported on all target platforms (it's not in POSIX, which specifies that the "input files shall be text files") or whether we should do something more comprehensive to provide sane_{e,f,}grep which guarantee to treat input as text.
Quoted 5 lines
Quoted 2 lines
I also tried setting
POSIXLY_CORRECTbut that doesn't affect the text/binary decision.Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered.
Mac OS X grep seems to support -a and tests in t8005 still pass with -a added to the egrep invocations.
Quoted 21 lines
We could
do:
test_lazy_prereqGREP_A'
echo foo | grep -a foo
'and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).
Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.
I also wondered whether we could get away without grepping at all here. But the blame output has a bunch of cruft we don't care about; I think the readability of the tests would suffer if we tried to match the whole thing in a
test_cmp.-Peff
Quoted 8 lines · Jeff King
Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered. We could do:
test_lazy_prereqGREP_A'
echo foo | grep -a foo
'
Quoted 7 lines
and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).
Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.
Would $PERL help, I wonder?
Quoted 15 lines · Junio C Hamano
Quoted 13 lines · Jeff King
Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered. We could do:
test_lazy_prereqGREP_A'
echo foo | grep -a foo
'and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).
Quoted 4 lines
Quoted 2 lines
Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.
Would $PERL help, I wonder?
It would, though I think you would need to call binmode to make it reliable. I was hesitant to suggest it, because I seem to recall some resistance to more perl dependencies in the test suite, but I think we may be past the point of no return there, anyway.
-Peff
Quoted 15 lines · Junio C Hamano
Quoted 13 lines · Jeff King
Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered. We could do:
test_lazy_prereqGREP_A'
echo foo | grep -a foo
'and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).
Quoted 4 lines
Quoted 2 lines
Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.
Would $PERL help, I wonder?
I suspect that any grep that lacks "-a" also lacks binary file handling that will break these tests. I found a Solaris grep that doesn't support "-a" and it treats these files as text.
Quoted 1 line
From that perspective, it would be better to have a central place that
deals with figuring out how to get grep to work for us. Perhaps we need test_grep to get this right. We already have test_cmp_bin() as a thin wrapper around cmp so I don't think this is completely unprecedented.
Quoted 8 lines · John Keeping
I suspect that any grep that lacks "-a" also lacks binary file handling that will break these tests. I found a Solaris grep that doesn't support "-a" and it treats these files as text.
From that perspective, it would be better to have a central place that deals with figuring out how to get grep to work for us. Perhaps we need
test_grepto get this right. We already havetest_cmp_bin()as a thin wrapper around cmp so I don't think this is completely unprecedented.
I think 99% of the time we are using grep for ascii text. As evidenced by the number of test failures we see with the new grep, it is a small minority that feed binary gibberish. I'd prefer if "-a" handling didn't need to pollute anything outside of this narrow range of tests (and as with my prereq suggestion, I am even find just skipping this narrow range of tests on platforms with no "-a", though falling back to running without "-a" is fine if it works).
-Peff
cover Fix test failures with GNU grep 2.23
Quoted 11 lines · Jeff King
Quoted 8 lines · John Keeping
I suspect that any grep that lacks "-a" also lacks binary file handling that will break these tests. I found a Solaris grep that doesn't support "-a" and it treats these files as text.
From that perspective, it would be better to have a central place that deals with figuring out how to get grep to work for us. Perhaps we need
test_grepto get this right. We already havetest_cmp_bin()as a thin wrapper around cmp so I don't think this is completely unprecedented.
Quoted 7 lines
I think 99% of the time we are using grep for ascii text. As evidenced by the number of test failures we see with the new grep, it is a small minority that feed binary gibberish. I'd prefer if "-a" handling didn't need to pollute anything outside of this narrow range of tests (and as with my prereq suggestion, I am even find just skipping this narrow range of tests on platforms with no "-a", though falling back to running without "-a" is fine if it works).
I went with using sed in this series because it seems to be the simplest and most compatible way to extract lines from the input. We don't need any special casing to figure out if an implementation needs "-a" or if it doesn't support that option and all the implementation I tested support the constructs used here.
John Keeping (2):
t8005: avoid grep on non-ASCII data
t9200: avoid grep on non-ASCII data
t/t8005-blame-i18n.sh | 16 ++++++++-------- t/t9200-git-cvsexportcommit.sh | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-)
Signature · 1 line
2.7.1.503.g3cfa3ac
PATCH 1/2 t8005: avoid grep on non-ASCII data
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator.
--- t/t8005-blame-i18n.sh | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
t/t8005-blame-i18n.sh +8 -8
@@ -35,8 +35,8 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' - git blame --incremental file | \ - egrep "^(author|summary) " > actual && + git blame --incremental file >output && + sed -ne "/^\(author\|summary\) /p" output >actual && test_cmp actual expected '
@@ -52,8 +52,8 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.logoutputencoding' ' git config i18n.logoutputencoding eucJP && - git blame --incremental file | \ - egrep "^(author|summary) " > actual && + git blame --incremental file >output && + sed -ne "/^\(author\|summary\) /p" output >actual && test_cmp actual expected '
@@ -68,8 +68,8 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=UTF-8' ' - git blame --incremental --encoding=UTF-8 file | \ - egrep "^(author|summary) " > actual && + git blame --incremental --encoding=UTF-8 file >output && + sed -ne "/^\(author\|summary\) /p" output >actual && test_cmp actual expected '
@@ -84,8 +84,8 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=none' ' - git blame --incremental --encoding=none file | \ - egrep "^(author|summary) " > actual && + git blame --incremental --encoding=none file >output && + sed -ne "/^\(author\|summary\) /p" output >actual && test_cmp actual expected '
Signature · 1 line
2.7.1.503.g3cfa3ac
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
Quoted 14 lines · John Keeping
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator.
Signed-off-by John Keeping <[email protected]>---
t/t8005-blame-i18n.sh EOF · line 38
Quoted 6 lines
t/t8005-blame-i18n.sh +2
-2@@ -35,8 +35,8 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' - git blame --incremental file | \ - egrep "^(author|summary) " > actual && + git blame --incremental file >output && + sed -ne "/^\(author\|summary\) /p" output >actual &&
These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:
sed -nEe "/^(author|summary) /p" ...
Quoted 35 lines
t/t8005-blame-i18n.sh +6
-6test_cmp actual expected '@@ -52,8 +52,8 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.logoutputencoding' ' git config i18n.logoutputencoding eucJP && - git blame --incremental file | \ - egrep "^(author|summary) " > actual && + git blame --incremental file >output && + sed -ne "/^\(author\|summary\) /p" output >actual && test_cmp actual expected '@@ -68,8 +68,8 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=UTF-8' ' - git blame --incremental --encoding=UTF-8 file | \ - egrep "^(author|summary) " > actual && + git blame --incremental --encoding=UTF-8 file >output && + sed -ne "/^\(author\|summary\) /p" output >actual && test_cmp actual expected '@@ -84,8 +84,8 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=none' ' - git blame --incremental --encoding=none file | \ - egrep "^(author|summary) " > actual && + git blame --incremental --encoding=none file >output && + sed -ne "/^\(author\|summary\) /p" output >actual && test_cmp actual expected '
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
Quoted 27 lines · Eric Sunshine
Quoted 14 lines · John Keeping
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator.
Signed-off-by John Keeping <[email protected]>---
t/t8005-blame-i18n.sh EOF · line 38
Quoted 6 lines
t/t8005-blame-i18n.sh +2
-2@@ -35,8 +35,8 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' - git blame --incremental file | \ - egrep "^(author|summary) " > actual && + git blame --incremental file >output && + sed -ne "/^\(author\|summary\) /p" output >actual &&These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:
Quoted 1 line
sed -nEe "/^(author|summary) /p" ...
At that point, I think we may as well use grep, because obscure platforms are probably broken either way.
I'm tempted to just go the perl route. We already depend on at least a baisc version of perl5 being installed for many of the other tests, so it's not really introducing a new dependency.
Something like the patch below works for me. I think we could make it shorter by using $PERLIO to get the raw behavior, but using binmode will work even on ancient versions of perl.
John, if you agree on the direction, feel free to combine it with your patch.
t/t8005-blame-i18n.sh +14 -4
@@ -33,10 +33,20 @@ author $SJIS_NAME summary $SJIS_MSG EOF +filter_blame () { + perl -e ' + binmode STDIN; + binmode STDOUT; + while (<>) { + print if /^(author|summary) /; + } + ' +} + test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' git blame --incremental file | \ - egrep "^(author|summary) " > actual && + filter_blame >actual && test_cmp actual expected '
@@ -53,7 +63,7 @@ test_expect_success !MINGW \ 'blame respects i18n.logoutputencoding' ' git config i18n.logoutputencoding eucJP && git blame --incremental file | \ - egrep "^(author|summary) " > actual && + filter_blame > actual && test_cmp actual expected '
@@ -69,7 +79,7 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=UTF-8' ' git blame --incremental --encoding=UTF-8 file | \ - egrep "^(author|summary) " > actual && + filter_blame >actual && test_cmp actual expected '
@@ -85,7 +95,7 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=none' ' git blame --incremental --encoding=none file | \ - egrep "^(author|summary) " > actual && + filter_blame >actual && test_cmp actual expected '
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
Quoted 29 lines · Jeff King
Quoted 27 lines · Eric Sunshine
Quoted 14 lines · John Keeping
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator.
Signed-off-by John Keeping <[email protected]>---
t/t8005-blame-i18n.sh EOF · line 38
Quoted 6 lines
t/t8005-blame-i18n.sh +2
-2@@ -35,8 +35,8 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' - git blame --incremental file | \ - egrep "^(author|summary) " > actual && + git blame --incremental file >output && + sed -ne "/^\(author\|summary\) /p" output >actual &&These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:
Quoted 4 lines
Quoted 1 line
sed -nEe "/^(author|summary) /p" ...At that point, I think we may as well use grep, because obscure platforms are probably broken either way.
Also GNU sed doesn't understand "-E", it uses "-r" for --regexp-extended.
Quoted 10 lines
I'm tempted to just go the perl route. We already depend on at least a baisc version of perl5 being installed for many of the other tests, so it's not really introducing a new dependency.
Something like the patch below works for me. I think we could make it shorter by using $PERLIO to get the raw behavior, but using binmode will work even on ancient versions of perl.
John, if you agree on the direction, feel free to combine it with your patch.
My original sed version was:
sed -ne "/^author /p" -e "/^summary /p"
which I think will work on all platforms (we already use it in t0000-basic.sh) but then I decided to be too clever :-(
I still think sed is simpler than introducing a new function to wrap a perl script.
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
Quoted 7 lines · John Keeping
Quoted 6 lines · Jeff King
Quoted 5 lines · Eric Sunshine
These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
sed -nEe "/^(author|summary) /p" ...At that point, I think we may as well use grep, because obscure platforms are probably broken either way.
Also GNU sed doesn't understand "-E", it uses "-r" for --regexp-extended.
It actually does recognize -E in all the versions I've tested, however, apparently it's undocumented (thus probably should be avoided).
Quoted 6 lines
My original sed version was:
sed -ne "/^author /p" -e "/^summary /p"which I think will work on all platforms (we already use it in
t0000-basic.sh) but then I decided to be too clever :-(
The unclever version seems fine.
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
In reply to John KeepingQuoted 9 lines · John Keeping
My original sed version was:
sed -ne "/^author /p" -e "/^summary /p"which I think will work on all platforms (we already use it in
t0000-basic.sh) but then I decided to be too clever :-(I still think sed is simpler than introducing a new function to wrap a perl script.
Yeah, I think that is good (personally I'd use a function anyway, but I think it is short enough that we could go either way).
-Peff
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
Quoted 6 lines · Jeff King
Quoted 4 lines · John Keeping
My original sed version was:
sed -ne "/^author /p" -e "/^summary /p"
Quoted 8 lines
Quoted 5 lines
which I think will work on all platforms (we already use it in
t0000-basic.sh) but then I decided to be too clever :-(I still think sed is simpler than introducing a new function to wrap a perl script.
Yeah, I think that is good (personally I'd use a function anyway, but I think it is short enough that we could go either way).
Agreed, and because there are repeated invocation of the same sed script in this file, it would be sensible to hide it behind a helper function.
Thanks.
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
In reply to John KeepingQuoted 9 lines · John Keeping
My original sed version was:
sed -ne "/^author /p" -e "/^summary /p"which I think will work on all platforms (we already use it in
t0000-basic.sh) but then I decided to be too clever :-(I still think sed is simpler than introducing a new function to wrap a perl script.
Let's do this, before everybody forgets what we discussed.
-- >8 --
From: John Keeping <[email protected]>
Date: Sun, 21 Feb 2016 17:32:21 +0000
Subject: [PATCH] t8005: avoid grep on non-ASCII data
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator. Also swap the order of the expected and actual output files given to test_cmp; we compare expect and actual to show how actual output differs from what is expected.
--- t/t8005-blame-i18n.sh | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)
t/t8005-blame-i18n.sh +16 -12
@@ -33,11 +33,15 @@ author $SJIS_NAME summary $SJIS_MSG EOF +filter_author_summary () { + sed -n -e '/^author /p' -e '/^summary /p' "$@" +} + test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' - git blame --incremental file | \ - egrep "^(author|summary) " > actual && - test_cmp actual expected + git blame --incremental file >output && + filter_author_summary output >actual && + test_cmp expected actual ' cat >expected <<EOF
@@ -52,9 +56,9 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.logoutputencoding' ' git config i18n.logoutputencoding eucJP && - git blame --incremental file | \ - egrep "^(author|summary) " > actual && - test_cmp actual expected + git blame --incremental file >output && + filter_author_summary output >actual && + test_cmp expected actual ' cat >expected <<EOF
@@ -68,9 +72,9 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=UTF-8' ' - git blame --incremental --encoding=UTF-8 file | \ - egrep "^(author|summary) " > actual && - test_cmp actual expected + git blame --incremental --encoding=UTF-8 file >output && + filter_author_summary output >actual && + test_cmp expected actual ' cat >expected <<EOF
@@ -84,9 +88,9 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=none' ' - git blame --incremental --encoding=none file | \ - egrep "^(author|summary) " > actual && - test_cmp actual expected + git blame --incremental --encoding=none file >output && + filter_author_summary output >actual && + test_cmp expected actual ' test_done
Signature · 1 line
2.7.2-532-g79873b4
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
Quoted 6 lines · Junio C Hamano
Quoted 4 lines · John Keeping
My original sed version was:
sed -ne "/^author /p" -e "/^summary /p"
Quoted 7 lines
Quoted 5 lines
which I think will work on all platforms (we already use it in
t0000-basic.sh) but then I decided to be too clever :-(I still think sed is simpler than introducing a new function to wrap a perl script.
Let's do this, before everybody forgets what we discussed.
Thanks, this looks good to me.
Quoted 87 lines
-- >8 --
From: John Keeping <[email protected]>
Date: Sun, 21 Feb 2016 17:32:21 +0000
Subject: [PATCH] t8005: avoid grep on non-ASCII dataGNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator. Also swap the order of the expected and actual output files given to
test_cmp; we compare expect and actual to show how actual output differs from what is expected.Signed-off-by John Keeping <[email protected]>Signed-off-by Junio C Hamano <[email protected]>--- t/t8005-blame-i18n.sh | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)t/t8005-blame-i18n.sh +16
-12@@ -33,11 +33,15 @@ author $SJIS_NAME summary $SJIS_MSG EOF +filter_author_summary () { + sed -n -e '/^author /p' -e '/^summary /p' "$@" +} + test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' - git blame --incremental file | \ - egrep "^(author|summary) " > actual && - test_cmp actual expected + git blame --incremental file >output && + filter_author_summary output >actual && + test_cmp expected actual ' cat >expected <<EOF@@ -52,9 +56,9 @@ EOF test_expect_success !MINGW \ 'blame respects i18n.logoutputencoding' ' git config i18n.logoutputencoding eucJP && - git blame --incremental file | \ - egrep "^(author|summary) " > actual && - test_cmp actual expected + git blame --incremental file >output && + filter_author_summary output >actual && + test_cmp expected actual ' cat >expected <<EOF@@ -68,9 +72,9 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=UTF-8' ' - git blame --incremental --encoding=UTF-8 file | \ - egrep "^(author|summary) " > actual && - test_cmp actual expected + git blame --incremental --encoding=UTF-8 file >output && + filter_author_summary output >actual && + test_cmp expected actual ' cat >expected <<EOF@@ -84,9 +88,9 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=none' ' - git blame --incremental --encoding=none file | \ - egrep "^(author|summary) " > actual && - test_cmp actual expected + git blame --incremental --encoding=none file >output && + filter_author_summary output >actual && + test_cmp expected actual ' test_doneSignature · 1 line
2.7.2-532-g79873b4
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
In reply to Jeff KingQuoted 12 lines · Jeff King
Quoted 11 lines · Eric Sunshine
Review
Quoted 4 lines · John Keeping
(quoted diff) +2
-2- git blame --incremental file | \ - egrep "^(author|summary) " > actual && + git blame --incremental file >output && + sed -ne "/^\(author\|summary\) /p" output >actual &&These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:
Quoted 4 lines
Quoted 1 line
sed -nEe "/^(author|summary) /p" ...At that point, I think we may as well use grep, because obscure platforms are probably broken either way.
I came to the same conclusion but forgot to say so at the end of my message.
t/t8005-blame-i18n.sh author $SJIS_NAME · line 39
Quoted 12 lines
I'm tempted to just go the perl route. We already depend on at least a baisc version of perl5 being installed for many of the other tests, so it's not really introducing a new dependency.
Something like the patch below works for me. I think we could make it shorter by using $PERLIO to get the raw behavior, but using binmode will work even on ancient versions of perl.
t/t8005-blame-i18n.sh +4
-0+filter_blame () { + perl -e ' + binmode STDIN; + binmode STDOUT;
I was worried about binmode() due to some vague recollection from years and years ago of it being problematic on Windows, but I see these tests are all protected by !MINGW anyhow...
Quoted 40 lines
t/t8005-blame-i18n.sh +10
-4+ while (<>) { + print if /^(author|summary) /; + } + ' +} + test_expect_success !MINGW \ 'blame respects i18n.commitencoding' ' git blame --incremental file | \ - egrep "^(author|summary) " > actual && + filter_blame >actual && test_cmp actual expected '@@ -53,7 +63,7 @@ test_expect_success !MINGW \ 'blame respects i18n.logoutputencoding' ' git config i18n.logoutputencoding eucJP && git blame --incremental file | \ - egrep "^(author|summary) " > actual && + filter_blame > actual && test_cmp actual expected '@@ -69,7 +79,7 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=UTF-8' ' git blame --incremental --encoding=UTF-8 file | \ - egrep "^(author|summary) " > actual && + filter_blame >actual && test_cmp actual expected '@@ -85,7 +95,7 @@ EOF test_expect_success !MINGW \ 'blame respects --encoding=none' ' git blame --incremental --encoding=none file | \ - egrep "^(author|summary) " > actual && + filter_blame >actual && test_cmp actual expected '
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
Quoted 4 lines · Eric Sunshine
Quoted 4 lines
Something like the patch below works for me. I think we could make it shorter by using $PERLIO to get the raw behavior, but using binmode will work even on ancient versions of perl.
Quoted 8 lines
t/t8005-blame-i18n.sh author $SJIS_NAME · line 39
Quoted 4 lines
t/t8005-blame-i18n.sh +4
-0+filter_blame () { + perl -e ' + binmode STDIN; + binmode STDOUT;I was worried about
binmode()due to some vague recollection from years and years ago of it being problematic on Windows, but I see these tests are all protected by !MINGW anyhow...
Thanks for mentioning that. I meant to put a note on that at the end of _my_ message, but forgot. :)
It does mean we won't do CRLF processing. We could get around that with some explicit chomp-ing, I think. Or just leave it as-is and assume these will lose the !MINGW prereq.
I see Junio just mentioned elsewhere that we can simply avoid the extended regular expressions by using two sed commands. That would be fine with me, too.
-Peff
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
In reply to Eric SunshineQuoted 6 lines · Eric Sunshine
These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:
sed -nEe "/^(author|summary) /p" ...
An obvious way to avoid any RE is to write it as two separate statements. As there are repeated invocations of this filtering in this script, perhaps a helper function can hide this ugliness?
Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data
Quoted 11 lines · Junio C Hamano
Quoted 6 lines · Eric Sunshine
These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:
sed -nEe "/^(author|summary) /p" ...An obvious way to avoid any RE is to write it as two separate statements.
Yes, that's even better; now I feel stupid for not thinking of it.
PATCH 2/2 t9200: avoid grep on non-ASCII data
In reply to John KeepingGNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
--- t/t9200-git-cvsexportcommit.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t9200-git-cvsexportcommit.sh +1 -1
@@ -35,7 +35,7 @@ exit 1 check_entries () { # $1 == directory, $2 == expected - grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual + sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual if test -z "$2" then >expected
Signature · 1 line
2.7.1.503.g3cfa3ac
Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data
Quoted 10 lines · John Keeping
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
Signed-off-by John Keeping <[email protected]>---
t/t9200-git-cvsexportcommit.sh exit 1 · line 37
Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1
-1@@ -35,7 +35,7 @@ exit 1 check_entries () { # $1 == directory, $2 == expected - grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual + sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
This works with BSD sed, but double negatives are confusing. Have you considered this instead?
sed -ne '/^\//p' ...
Quoted 3 lines
t/t9200-git-cvsexportcommit.sh +0
-0if test -z "$2" then >expected
Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data
Quoted 19 lines · Eric Sunshine
Quoted 10 lines · John Keeping
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
Signed-off-by John Keeping <[email protected]>---
t/t9200-git-cvsexportcommit.sh exit 1 · line 37
Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1
-1@@ -35,7 +35,7 @@ exit 1 check_entries () { # $1 == directory, $2 == expected - grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual + sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actualThis works with BSD sed, but double negatives are confusing. Have you considered this instead?
Quoted 1 line
sed -ne '/^\//p' ...
What do you mean double negatives? Do you mean using "!" as an alternative delimiter? I find changing delimters is normally simpler than following multiple levels of quoting for escaping slashes, although in this case it's simple enough that it doesn't make much difference.
Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data
Quoted 20 lines · John Keeping
Quoted 19 lines · Eric Sunshine
Quoted 10 lines · John Keeping
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
Signed-off-by John Keeping <[email protected]>---
t/t9200-git-cvsexportcommit.sh exit 1 · line 37
Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1
-1@@ -35,7 +35,7 @@ exit 1 check_entries () { # $1 == directory, $2 == expected - grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual + sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actualThis works with BSD sed, but double negatives are confusing. Have you considered this instead?
Quoted 6 lines
Quoted 1 line
sed -ne '/^\//p' ...What do you mean double negatives? Do you mean using "!" as an alternative delimiter? I find changing delimters is normally simpler than following multiple levels of quoting for escaping slashes, although in this case it's simple enough that it doesn't make much difference.
Nice, I learned something new today. If I recall correctly, historic sed did not allow the delimiter to be changed (or it wasn't documented or I simply forgot about the capability). So, feel free to ignore me.
Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data
In reply to John KeepingQuoted 20 lines · John Keeping
Quoted 19 lines · Eric Sunshine
Quoted 10 lines · John Keeping
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
Signed-off-by John Keeping <[email protected]>---
t/t9200-git-cvsexportcommit.sh exit 1 · line 37
Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1
-1@@ -35,7 +35,7 @@ exit 1 check_entries () { # $1 == directory, $2 == expected - grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual + sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actualThis works with BSD sed, but double negatives are confusing. Have you considered this instead?
Quoted 6 lines
Quoted 1 line
sed -ne '/^\//p' ...What do you mean double negatives? Do you mean using "!" as an alternative delimiter? I find changing delimters is normally simpler than following multiple levels of quoting for escaping slashes, although in this case it's simple enough that it doesn't make much difference.
I agree that changing delimiters is much nicer than backslashes. But I wonder if using "!" is more confusing than it needs to be, given its other meanings.
I dunno. I admit that the backslash threw me off, too (since it needs escaped in interactive shells, I first assumed that's what was going on). Using backslash to select the delimiter was new to me. I've usually seen:
s!/foo/!/bar/!
which is arguably a little more clear. Too bad we cannot do:
m!/foo!
which I think reads better. Oh well. Maybe:
sed -ne '\#^/#p'
would be more readable, but I'm just bikeshedding at this point. The grep invocation really was the most clear. :-/
-Peff
Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data
Quoted 42 lines · Jeff King
Quoted 20 lines · John Keeping
Quoted 19 lines · Eric Sunshine
Quoted 10 lines · John Keeping
GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.
Signed-off-by John Keeping <[email protected]>---
t/t9200-git-cvsexportcommit.sh exit 1 · line 37
Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1
-1@@ -35,7 +35,7 @@ exit 1 check_entries () { # $1 == directory, $2 == expected - grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual + sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actualThis works with BSD sed, but double negatives are confusing. Have you considered this instead?
Quoted 6 lines
Quoted 1 line
sed -ne '/^\//p' ...What do you mean double negatives? Do you mean using "!" as an alternative delimiter? I find changing delimters is normally simpler than following multiple levels of quoting for escaping slashes, although in this case it's simple enough that it doesn't make much difference.
I agree that changing delimiters is much nicer than backslashes. But I wonder if using "!" is more confusing than it needs to be, given its other meanings.
I dunno. I admit that the backslash threw me off, too (since it needs escaped in interactive shells, I first assumed that's what was going on). Using backslash to select the delimiter was new to me. I've usually seen:
s!/foo/!/bar/!
which is arguably a little more clear. Too bad we cannot do:
Quoted 8 lines
m!/foo!
which I think reads better. Oh well. Maybe:
sed -ne '\#^/#p'
would be more readable, but I'm just bikeshedding at this point. The grep invocation really was the most clear. :-/
Eric's '/^\//' was the most straight-forward and easiest to see what is going on, I would think.