Skip to contentSkip to conversation

Test failures with GNU grep 2.23

PATCH In review John Keeping 4 participants 27 messages Jun 15, 2016

lore
Share & tools
lore
#0 lore raw

It seems that binary file detection has changed in GNU grep 2.23 as a result of commit 40ed879 (grep: fix bug with with invalid unibyte sequence).

This causes a couple of test failures in t8005 and t9200 (the t9200 case is less obvious so I'm only including t8005 here):

-- >8 --

$ ./t8005-blame-i18n.sh -v -i
[snip]
expecting success: 
        git blame --incremental file | \
                egrep "^(author|summary) " > actual &&
        test_cmp actual expected
expected 2016-02-07 16:14:55.359510341 +0000 renamed from actual 2016-02-07 16:14:55.372510307 +0000 +6 -1
@@ -1 +1,6 @@
-Binary file (standard input) matches
+author �R�c ���Y
+summary �u���[���̃e�X�g�ł��B
+author �R�c ���Y
+summary �u���[���̃e�X�g�ł��B
+author �R�c ���Y
+summary �u���[���̃e�X�g�ł��B

not ok 2 - blame respects i18n.commitencoding
#
# git blame --incremental file | \
# egrep "^(author|summary) " > actual &&
# test_cmp actual expected
#
-- 8< --

The following patch fixes the tests for me, but I wonder if "-a" is supported on all target platforms (it's not in POSIX, which specifies that the "input files shall be text files") or whether we should do something more comprehensive to provide sane_{e,f,}grep which guarantee to treat input as text.

I also tried setting POSIXLY_CORRECT but that doesn't affect the text/binary decision.

-- >8 --

t/t8005-blame-i18n.sh +4 -4
index 847d098..3b6e697 100755
@@ -36,7 +36,7 @@ EOF
 test_expect_success !MINGW \
 	'blame respects i18n.commitencoding' '
 	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
+		egrep -a "^(author|summary) " > actual &&
 	test_cmp actual expected
 '
 
@@ -53,7 +53,7 @@ test_expect_success !MINGW \
 	'blame respects i18n.logoutputencoding' '
 	git config i18n.logoutputencoding eucJP &&
 	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
+		egrep -a "^(author|summary) " > actual &&
 	test_cmp actual expected
 '
 
@@ -69,7 +69,7 @@ EOF
 test_expect_success !MINGW \
 	'blame respects --encoding=UTF-8' '
 	git blame --incremental --encoding=UTF-8 file | \
-		egrep "^(author|summary) " > actual &&
+		egrep -a "^(author|summary) " > actual &&
 	test_cmp actual expected
 '
 
@@ -85,7 +85,7 @@ EOF
 test_expect_success !MINGW \
 	'blame respects --encoding=none' '
 	git blame --incremental --encoding=none file | \
-		egrep "^(author|summary) " > actual &&
+		egrep -a "^(author|summary) " > actual &&
 	test_cmp actual expected
 '
 
t/t9200-git-cvsexportcommit.sh +1 -1
index 5cfb9cf..f05578a 100755
@@ -35,7 +35,7 @@ exit 1
 
 check_entries () {
 	# $1 == directory, $2 == expected
-	grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
+	grep -a '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
 	if test -z "$2"
 	then
 		>expected
#1 lore raw
Quoted 3 lines · John Keeping

It seems that binary file detection has changed in GNU grep 2.23 as a result of commit 40ed879 (grep: fix bug with with invalid unibyte sequence).

I read this bug report a while ago when you posted it, but happily ignored it until today, when my debian unstable system pulled in the new version of grep. :)

Quoted 10 lines

This causes a couple of test failures in t8005 and t9200 (the t9200 case is less obvious so I'm only including t8005 here):

-- >8 --

$ ./t8005-blame-i18n.sh -v -i
[snip]
expecting success: 
        git blame --incremental file | \
                egrep "^(author|summary) " > actual &&
        test_cmp actual expected

Just a side note while we are touching these tests:

- we probably should not pipe, so we check the exist code from git-blame

- we usually flip the test_cmp file order, to show the difference from expectation when there is a failure

- no space after ">" redirection :)

Quoted 8 lines

The following patch fixes the tests for me, but I wonder if "-a" is supported on all target platforms (it's not in POSIX, which specifies that the "input files shall be text files") or whether we should do something more comprehensive to provide sane_{e,f,}grep which guarantee to treat input as text.

I also tried setting POSIXLY_CORRECT but that doesn't affect the text/binary decision.

Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered. We could do:

test_lazy_prereq GREP_A '
echo foo | grep -a foo
'

and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).

Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.

I also wondered whether we could get away without grepping at all here. But the blame output has a bunch of cruft we don't care about; I think the readability of the tests would suffer if we tried to match the whole thing in a test_cmp.

-Peff

#2 lore raw
Quoted 7 lines · Jeff King
Quoted 6 lines · John Keeping

The following patch fixes the tests for me, but I wonder if "-a" is supported on all target platforms (it's not in POSIX, which specifies that the "input files shall be text files") or whether we should do something more comprehensive to provide sane_{e,f,}grep which guarantee to treat input as text.

Quoted 5 lines
Quoted 2 lines

I also tried setting POSIXLY_CORRECT but that doesn't affect the text/binary decision.

Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered.

Mac OS X grep seems to support -a and tests in t8005 still pass with -a added to the egrep invocations.

Quoted 21 lines

We could
do:

test_lazy_prereq GREP_A '
echo foo | grep -a foo
'

and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).

Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.

I also wondered whether we could get away without grepping at all here. But the blame output has a bunch of cruft we don't care about; I think the readability of the tests would suffer if we tried to match the whole thing in a test_cmp.

-Peff

#3 lore raw
In reply to Jeff King
Quoted 8 lines · Jeff King

Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered. We could do:

test_lazy_prereq GREP_A '
echo foo | grep -a foo
'

Quoted 7 lines

and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).

Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.

Would $PERL help, I wonder?

#4 lore raw
Quoted 15 lines · Junio C Hamano
Quoted 13 lines · Jeff King

Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered. We could do:

test_lazy_prereq GREP_A '
echo foo | grep -a foo
'

and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).

Quoted 4 lines
Quoted 2 lines

Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.

Would $PERL help, I wonder?

It would, though I think you would need to call binmode to make it reliable. I was hesitant to suggest it, because I seem to recall some resistance to more perl dependencies in the test suite, but I think we may be past the point of no return there, anyway.

-Peff

#5 lore raw
In reply to Junio C Hamano
Quoted 15 lines · Junio C Hamano
Quoted 13 lines · Jeff King

Yeah, I'd worry that "-a" is not portable. OTOH, BSD grep seems to have it, so between that and GNU, I think most systems are covered. We could do:

test_lazy_prereq GREP_A '
echo foo | grep -a foo
'

and mark these tests with it. I'd also be happy to skip that step and just do it if and when somebody actually complains about a system without it (I wouldn't be surprised if most people on antique systems end up installing GNU grep anyway).

Quoted 4 lines
Quoted 2 lines

Another option might be using "sed -ne '/^author/p'" or similar. But that may very well just be trading one portability problem for another.

Would $PERL help, I wonder?

I suspect that any grep that lacks "-a" also lacks binary file handling that will break these tests. I found a Solaris grep that doesn't support "-a" and it treats these files as text.

Quoted 1 line

From that perspective, it would be better to have a central place that

deals with figuring out how to get grep to work for us. Perhaps we need test_grep to get this right. We already have test_cmp_bin() as a thin wrapper around cmp so I don't think this is completely unprecedented.

#6 lore raw
Quoted 8 lines · John Keeping

I suspect that any grep that lacks "-a" also lacks binary file handling that will break these tests. I found a Solaris grep that doesn't support "-a" and it treats these files as text.

From that perspective, it would be better to have a central place that deals with figuring out how to get grep to work for us. Perhaps we need test_grep to get this right. We already have test_cmp_bin() as a thin wrapper around cmp so I don't think this is completely unprecedented.

I think 99% of the time we are using grep for ascii text. As evidenced by the number of test failures we see with the new grep, it is a small minority that feed binary gibberish. I'd prefer if "-a" handling didn't need to pollute anything outside of this narrow range of tests (and as with my prereq suggestion, I am even find just skipping this narrow range of tests on platforms with no "-a", though falling back to running without "-a" is fine if it works).

-Peff

#7 lore raw

cover Fix test failures with GNU grep 2.23

Quoted 11 lines · Jeff King
Quoted 8 lines · John Keeping

I suspect that any grep that lacks "-a" also lacks binary file handling that will break these tests. I found a Solaris grep that doesn't support "-a" and it treats these files as text.

From that perspective, it would be better to have a central place that deals with figuring out how to get grep to work for us. Perhaps we need test_grep to get this right. We already have test_cmp_bin() as a thin wrapper around cmp so I don't think this is completely unprecedented.

Quoted 7 lines

I think 99% of the time we are using grep for ascii text. As evidenced by the number of test failures we see with the new grep, it is a small minority that feed binary gibberish. I'd prefer if "-a" handling didn't need to pollute anything outside of this narrow range of tests (and as with my prereq suggestion, I am even find just skipping this narrow range of tests on platforms with no "-a", though falling back to running without "-a" is fine if it works).

I went with using sed in this series because it seems to be the simplest and most compatible way to extract lines from the input. We don't need any special casing to figure out if an implementation needs "-a" or if it doesn't support that option and all the implementation I tested support the constructs used here.

John Keeping (2):
t8005: avoid grep on non-ASCII data
t9200: avoid grep on non-ASCII data

 t/t8005-blame-i18n.sh          | 16 ++++++++--------
 t/t9200-git-cvsexportcommit.sh |  2 +-
 2 files changed, 9 insertions(+), 9 deletions(-)
Signature · 1 line
2.7.1.503.g3cfa3ac
#8 lore raw

PATCH 1/2 t8005: avoid grep on non-ASCII data

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator.

---
 t/t8005-blame-i18n.sh | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
t/t8005-blame-i18n.sh +8 -8
index 847d098..0a86c72 100755
@@ -35,8 +35,8 @@ EOF
 
 test_expect_success !MINGW \
 	'blame respects i18n.commitencoding' '
-	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
+	git blame --incremental file >output &&
+	sed -ne "/^\(author\|summary\) /p" output >actual &&
 	test_cmp actual expected
 '
 
@@ -52,8 +52,8 @@ EOF
 test_expect_success !MINGW \
 	'blame respects i18n.logoutputencoding' '
 	git config i18n.logoutputencoding eucJP &&
-	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
+	git blame --incremental file >output &&
+	sed -ne "/^\(author\|summary\) /p" output >actual &&
 	test_cmp actual expected
 '
 
@@ -68,8 +68,8 @@ EOF
 
 test_expect_success !MINGW \
 	'blame respects --encoding=UTF-8' '
-	git blame --incremental --encoding=UTF-8 file | \
-		egrep "^(author|summary) " > actual &&
+	git blame --incremental --encoding=UTF-8 file >output &&
+	sed -ne "/^\(author\|summary\) /p" output >actual &&
 	test_cmp actual expected
 '
 
@@ -84,8 +84,8 @@ EOF
 
 test_expect_success !MINGW \
 	'blame respects --encoding=none' '
-	git blame --incremental --encoding=none file | \
-		egrep "^(author|summary) " > actual &&
+	git blame --incremental --encoding=none file >output &&
+	sed -ne "/^\(author\|summary\) /p" output >actual &&
 	test_cmp actual expected
 '
 
Signature · 1 line
2.7.1.503.g3cfa3ac
#9 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

Quoted 14 lines · John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator.

---

t/t8005-blame-i18n.sh EOF · line 38

Quoted 6 lines
t/t8005-blame-i18n.sh +2 -2
@@ -35,8 +35,8 @@ EOF
 test_expect_success !MINGW \
        'blame respects i18n.commitencoding' '
-       git blame --incremental file | \
-               egrep "^(author|summary) " > actual &&
+       git blame --incremental file >output &&
+       sed -ne "/^\(author\|summary\) /p" output >actual &&

These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:

    sed -nEe "/^(author|summary) /p" ...
Quoted 35 lines
t/t8005-blame-i18n.sh +6 -6
        test_cmp actual expected
 '
 
@@ -52,8 +52,8 @@ EOF
 test_expect_success !MINGW \
        'blame respects i18n.logoutputencoding' '
        git config i18n.logoutputencoding eucJP &&
-       git blame --incremental file | \
-               egrep "^(author|summary) " > actual &&
+       git blame --incremental file >output &&
+       sed -ne "/^\(author\|summary\) /p" output >actual &&
        test_cmp actual expected
 '
 
@@ -68,8 +68,8 @@ EOF
 
 test_expect_success !MINGW \
        'blame respects --encoding=UTF-8' '
-       git blame --incremental --encoding=UTF-8 file | \
-               egrep "^(author|summary) " > actual &&
+       git blame --incremental --encoding=UTF-8 file >output &&
+       sed -ne "/^\(author\|summary\) /p" output >actual &&
        test_cmp actual expected
 '
 
@@ -84,8 +84,8 @@ EOF
 
 test_expect_success !MINGW \
        'blame respects --encoding=none' '
-       git blame --incremental --encoding=none file | \
-               egrep "^(author|summary) " > actual &&
+       git blame --incremental --encoding=none file >output &&
+       sed -ne "/^\(author\|summary\) /p" output >actual &&
        test_cmp actual expected
 '
#10 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

Quoted 27 lines · Eric Sunshine
Quoted 14 lines · John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator.

---

t/t8005-blame-i18n.sh EOF · line 38

Quoted 6 lines
t/t8005-blame-i18n.sh +2 -2
@@ -35,8 +35,8 @@ EOF
 test_expect_success !MINGW \
        'blame respects i18n.commitencoding' '
-       git blame --incremental file | \
-               egrep "^(author|summary) " > actual &&
+       git blame --incremental file >output &&
+       sed -ne "/^\(author\|summary\) /p" output >actual &&

These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:

Quoted 1 line
    sed -nEe "/^(author|summary) /p" ...

At that point, I think we may as well use grep, because obscure platforms are probably broken either way.

I'm tempted to just go the perl route. We already depend on at least a baisc version of perl5 being installed for many of the other tests, so it's not really introducing a new dependency.

Something like the patch below works for me. I think we could make it shorter by using $PERLIO to get the raw behavior, but using binmode will work even on ancient versions of perl.

John, if you agree on the direction, feel free to combine it with your patch.

t/t8005-blame-i18n.sh +14 -4
index 847d098..f7a02d8 100755
@@ -33,10 +33,20 @@ author $SJIS_NAME
 summary $SJIS_MSG
 EOF
 
+filter_blame () {
+	perl -e '
+		binmode STDIN;
+		binmode STDOUT;
+		while (<>) {
+			print if /^(author|summary) /;
+		}
+	'
+}
+
 test_expect_success !MINGW \
 	'blame respects i18n.commitencoding' '
 	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
+		filter_blame >actual &&
 	test_cmp actual expected
 '
 
@@ -53,7 +63,7 @@ test_expect_success !MINGW \
 	'blame respects i18n.logoutputencoding' '
 	git config i18n.logoutputencoding eucJP &&
 	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
+		filter_blame > actual &&
 	test_cmp actual expected
 '
 
@@ -69,7 +79,7 @@ EOF
 test_expect_success !MINGW \
 	'blame respects --encoding=UTF-8' '
 	git blame --incremental --encoding=UTF-8 file | \
-		egrep "^(author|summary) " > actual &&
+		filter_blame >actual &&
 	test_cmp actual expected
 '
 
@@ -85,7 +95,7 @@ EOF
 test_expect_success !MINGW \
 	'blame respects --encoding=none' '
 	git blame --incremental --encoding=none file | \
-		egrep "^(author|summary) " > actual &&
+		filter_blame >actual &&
 	test_cmp actual expected
 '
 
#11 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

Quoted 29 lines · Jeff King
Quoted 27 lines · Eric Sunshine
Quoted 14 lines · John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator.

---

t/t8005-blame-i18n.sh EOF · line 38

Quoted 6 lines
t/t8005-blame-i18n.sh +2 -2
@@ -35,8 +35,8 @@ EOF
 test_expect_success !MINGW \
        'blame respects i18n.commitencoding' '
-       git blame --incremental file | \
-               egrep "^(author|summary) " > actual &&
+       git blame --incremental file >output &&
+       sed -ne "/^\(author\|summary\) /p" output >actual &&

These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:

Quoted 4 lines
Quoted 1 line
    sed -nEe "/^(author|summary) /p" ...

At that point, I think we may as well use grep, because obscure platforms are probably broken either way.

Also GNU sed doesn't understand "-E", it uses "-r" for --regexp-extended.

Quoted 10 lines

I'm tempted to just go the perl route. We already depend on at least a baisc version of perl5 being installed for many of the other tests, so it's not really introducing a new dependency.

Something like the patch below works for me. I think we could make it shorter by using $PERLIO to get the raw behavior, but using binmode will work even on ancient versions of perl.

John, if you agree on the direction, feel free to combine it with your patch.

My original sed version was:

	sed -ne "/^author /p" -e "/^summary /p"

which I think will work on all platforms (we already use it in t0000-basic.sh) but then I decided to be too clever :-(

I still think sed is simpler than introducing a new function to wrap a perl script.

#12 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

Quoted 7 lines · John Keeping
Quoted 6 lines · Jeff King
Quoted 5 lines · Eric Sunshine

These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
    sed -nEe "/^(author|summary) /p" ...

At that point, I think we may as well use grep, because obscure platforms are probably broken either way.

Also GNU sed doesn't understand "-E", it uses "-r" for --regexp-extended.

It actually does recognize -E in all the versions I've tested, however, apparently it's undocumented (thus probably should be avoided).

Quoted 6 lines

My original sed version was:

        sed -ne "/^author /p" -e "/^summary /p"

which I think will work on all platforms (we already use it in t0000-basic.sh) but then I decided to be too clever :-(

The unclever version seems fine.

#13 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

In reply to John Keeping
Quoted 9 lines · John Keeping

My original sed version was:

	sed -ne "/^author /p" -e "/^summary /p"

which I think will work on all platforms (we already use it in t0000-basic.sh) but then I decided to be too clever :-(

I still think sed is simpler than introducing a new function to wrap a perl script.

Yeah, I think that is good (personally I'd use a function anyway, but I think it is short enough that we could go either way).

-Peff

#14 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

Quoted 6 lines · Jeff King
Quoted 4 lines · John Keeping

My original sed version was:

	sed -ne "/^author /p" -e "/^summary /p"
Quoted 8 lines
Quoted 5 lines

which I think will work on all platforms (we already use it in t0000-basic.sh) but then I decided to be too clever :-(

I still think sed is simpler than introducing a new function to wrap a perl script.

Yeah, I think that is good (personally I'd use a function anyway, but I think it is short enough that we could go either way).

Agreed, and because there are repeated invocation of the same sed script in this file, it would be sensible to hide it behind a helper function.

Thanks.

#15 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

In reply to John Keeping
Quoted 9 lines · John Keeping

My original sed version was:

	sed -ne "/^author /p" -e "/^summary /p"

which I think will work on all platforms (we already use it in t0000-basic.sh) but then I decided to be too clever :-(

I still think sed is simpler than introducing a new function to wrap a perl script.

Let's do this, before everybody forgets what we discussed.

-- >8 --
From: John Keeping <[email protected]>
Date: Sun, 21 Feb 2016 17:32:21 +0000
Subject: [PATCH] t8005: avoid grep on non-ASCII data

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator. Also swap the order of the expected and actual output files given to test_cmp; we compare expect and actual to show how actual output differs from what is expected.

---
 t/t8005-blame-i18n.sh | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)
t/t8005-blame-i18n.sh +16 -12
index 847d098..75da219 100755
@@ -33,11 +33,15 @@ author $SJIS_NAME
 summary $SJIS_MSG
 EOF
 
+filter_author_summary () {
+	sed -n -e '/^author /p' -e '/^summary /p' "$@"
+}
+
 test_expect_success !MINGW \
 	'blame respects i18n.commitencoding' '
-	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
-	test_cmp actual expected
+	git blame --incremental file >output &&
+	filter_author_summary output >actual &&
+	test_cmp expected actual
 '
 
 cat >expected <<EOF
@@ -52,9 +56,9 @@ EOF
 test_expect_success !MINGW \
 	'blame respects i18n.logoutputencoding' '
 	git config i18n.logoutputencoding eucJP &&
-	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
-	test_cmp actual expected
+	git blame --incremental file >output &&
+	filter_author_summary output >actual &&
+	test_cmp expected actual
 '
 
 cat >expected <<EOF
@@ -68,9 +72,9 @@ EOF
 
 test_expect_success !MINGW \
 	'blame respects --encoding=UTF-8' '
-	git blame --incremental --encoding=UTF-8 file | \
-		egrep "^(author|summary) " > actual &&
-	test_cmp actual expected
+	git blame --incremental --encoding=UTF-8 file >output &&
+	filter_author_summary output >actual &&
+	test_cmp expected actual
 '
 
 cat >expected <<EOF
@@ -84,9 +88,9 @@ EOF
 
 test_expect_success !MINGW \
 	'blame respects --encoding=none' '
-	git blame --incremental --encoding=none file | \
-		egrep "^(author|summary) " > actual &&
-	test_cmp actual expected
+	git blame --incremental --encoding=none file >output &&
+	filter_author_summary output >actual &&
+	test_cmp expected actual
 '
 
 test_done
Signature · 1 line
2.7.2-532-g79873b4
#16 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

Quoted 6 lines · Junio C Hamano
Quoted 4 lines · John Keeping

My original sed version was:

	sed -ne "/^author /p" -e "/^summary /p"
Quoted 7 lines
Quoted 5 lines

which I think will work on all platforms (we already use it in t0000-basic.sh) but then I decided to be too clever :-(

I still think sed is simpler than introducing a new function to wrap a perl script.

Let's do this, before everybody forgets what we discussed.

Thanks, this looks good to me.

Quoted 87 lines

-- >8 --
From: John Keeping <[email protected]>
Date: Sun, 21 Feb 2016 17:32:21 +0000
Subject: [PATCH] t8005: avoid grep on non-ASCII data

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

While touching these lines, modernize the test style to avoid hiding the exit status of "git blame" and remove a space following a redirection operator. Also swap the order of the expected and actual output files given to test_cmp; we compare expect and actual to show how actual output differs from what is expected.

---
 t/t8005-blame-i18n.sh | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)
t/t8005-blame-i18n.sh +16 -12
index 847d098..75da219 100755
@@ -33,11 +33,15 @@ author $SJIS_NAME
 summary $SJIS_MSG
 EOF
 
+filter_author_summary () {
+	sed -n -e '/^author /p' -e '/^summary /p' "$@"
+}
+
 test_expect_success !MINGW \
 	'blame respects i18n.commitencoding' '
-	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
-	test_cmp actual expected
+	git blame --incremental file >output &&
+	filter_author_summary output >actual &&
+	test_cmp expected actual
 '
 
 cat >expected <<EOF
@@ -52,9 +56,9 @@ EOF
 test_expect_success !MINGW \
 	'blame respects i18n.logoutputencoding' '
 	git config i18n.logoutputencoding eucJP &&
-	git blame --incremental file | \
-		egrep "^(author|summary) " > actual &&
-	test_cmp actual expected
+	git blame --incremental file >output &&
+	filter_author_summary output >actual &&
+	test_cmp expected actual
 '
 
 cat >expected <<EOF
@@ -68,9 +72,9 @@ EOF
 
 test_expect_success !MINGW \
 	'blame respects --encoding=UTF-8' '
-	git blame --incremental --encoding=UTF-8 file | \
-		egrep "^(author|summary) " > actual &&
-	test_cmp actual expected
+	git blame --incremental --encoding=UTF-8 file >output &&
+	filter_author_summary output >actual &&
+	test_cmp expected actual
 '
 
 cat >expected <<EOF
@@ -84,9 +88,9 @@ EOF
 
 test_expect_success !MINGW \
 	'blame respects --encoding=none' '
-	git blame --incremental --encoding=none file | \
-		egrep "^(author|summary) " > actual &&
-	test_cmp actual expected
+	git blame --incremental --encoding=none file >output &&
+	filter_author_summary output >actual &&
+	test_cmp expected actual
 '
 
 test_done
Signature · 1 line
2.7.2-532-g79873b4
#17 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

In reply to Jeff King
Quoted 12 lines · Jeff King
Quoted 11 lines · Eric Sunshine

Review

Quoted 4 lines · John Keeping
(quoted diff) +2 -2
-       git blame --incremental file | \
-               egrep "^(author|summary) " > actual &&
+       git blame --incremental file >output &&
+       sed -ne "/^\(author\|summary\) /p" output >actual &&

These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:

Quoted 4 lines
Quoted 1 line
    sed -nEe "/^(author|summary) /p" ...

At that point, I think we may as well use grep, because obscure platforms are probably broken either way.

I came to the same conclusion but forgot to say so at the end of my message.

t/t8005-blame-i18n.sh author $SJIS_NAME · line 39

Quoted 12 lines

I'm tempted to just go the perl route. We already depend on at least a baisc version of perl5 being installed for many of the other tests, so it's not really introducing a new dependency.

Something like the patch below works for me. I think we could make it shorter by using $PERLIO to get the raw behavior, but using binmode will work even on ancient versions of perl.

t/t8005-blame-i18n.sh +4 -0
+filter_blame () {
+       perl -e '
+               binmode STDIN;
+               binmode STDOUT;

I was worried about binmode() due to some vague recollection from years and years ago of it being problematic on Windows, but I see these tests are all protected by !MINGW anyhow...

Quoted 40 lines
t/t8005-blame-i18n.sh +10 -4
+               while (<>) {
+                       print if /^(author|summary) /;
+               }
+       '
+}
+
 test_expect_success !MINGW \
        'blame respects i18n.commitencoding' '
        git blame --incremental file | \
-               egrep "^(author|summary) " > actual &&
+               filter_blame >actual &&
        test_cmp actual expected
 '
@@ -53,7 +63,7 @@ test_expect_success !MINGW \
        'blame respects i18n.logoutputencoding' '
        git config i18n.logoutputencoding eucJP &&
        git blame --incremental file | \
-               egrep "^(author|summary) " > actual &&
+               filter_blame > actual &&
        test_cmp actual expected
 '
 
@@ -69,7 +79,7 @@ EOF
 test_expect_success !MINGW \
        'blame respects --encoding=UTF-8' '
        git blame --incremental --encoding=UTF-8 file | \
-               egrep "^(author|summary) " > actual &&
+               filter_blame >actual &&
        test_cmp actual expected
 '
 
@@ -85,7 +95,7 @@ EOF
 test_expect_success !MINGW \
        'blame respects --encoding=none' '
        git blame --incremental --encoding=none file | \
-               egrep "^(author|summary) " > actual &&
+               filter_blame >actual &&
        test_cmp actual expected
 '
#18 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

Quoted 4 lines · Eric Sunshine
Quoted 4 lines

Something like the patch below works for me. I think we could make it shorter by using $PERLIO to get the raw behavior, but using binmode will work even on ancient versions of perl.

Quoted 8 lines

t/t8005-blame-i18n.sh author $SJIS_NAME · line 39

Quoted 4 lines
t/t8005-blame-i18n.sh +4 -0
+filter_blame () {
+       perl -e '
+               binmode STDIN;
+               binmode STDOUT;

I was worried about binmode() due to some vague recollection from years and years ago of it being problematic on Windows, but I see these tests are all protected by !MINGW anyhow...

Thanks for mentioning that. I meant to put a note on that at the end of _my_ message, but forgot. :)

It does mean we won't do CRLF processing. We could get around that with some explicit chomp-ing, I think. Or just leave it as-is and assume these will lose the !MINGW prereq.

I see Junio just mentioned elsewhere that we can simply avoid the extended regular expressions by using two sed commands. That would be fine with me, too.

-Peff

#19 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

In reply to Eric Sunshine
Quoted 6 lines · Eric Sunshine

These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:

    sed -nEe "/^(author|summary) /p" ...

An obvious way to avoid any RE is to write it as two separate statements. As there are repeated invocations of this filtering in this script, perhaps a helper function can hide this ugliness?

#20 lore raw

Re: [PATCH 1/2] t8005: avoid grep on non-ASCII data

Quoted 11 lines · Junio C Hamano
Quoted 6 lines · Eric Sunshine

These tests all crash and burn with BSD sed (including Mac OS X) since you're not restricting yourself to BRE (basic regular expressions). You _could_ request extended regular expressions, which do work on those platforms, as well as with GNU sed:

    sed -nEe "/^(author|summary) /p" ...

An obvious way to avoid any RE is to write it as two separate statements.

Yes, that's even better; now I feel stupid for not thinking of it.

#21 lore raw

PATCH 2/2 t9200: avoid grep on non-ASCII data

In reply to John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

---
 t/t9200-git-cvsexportcommit.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
t/t9200-git-cvsexportcommit.sh +1 -1
index 812c9cd..0765d52 100755
@@ -35,7 +35,7 @@ exit 1
 
 check_entries () {
 	# $1 == directory, $2 == expected
-	grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
+	sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
 	if test -z "$2"
 	then
 		>expected
Signature · 1 line
2.7.1.503.g3cfa3ac
#22 lore raw

Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data

Quoted 10 lines · John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

---

t/t9200-git-cvsexportcommit.sh exit 1 · line 37

Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1 -1
@@ -35,7 +35,7 @@ exit 1
 check_entries () {
        # $1 == directory, $2 == expected
-       grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
+       sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual

This works with BSD sed, but double negatives are confusing. Have you considered this instead?

    sed -ne '/^\//p' ...
Quoted 3 lines
t/t9200-git-cvsexportcommit.sh +0 -0
        if test -z "$2"
        then
                >expected
#23 lore raw

Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data

Quoted 19 lines · Eric Sunshine
Quoted 10 lines · John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

---

t/t9200-git-cvsexportcommit.sh exit 1 · line 37

Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1 -1
@@ -35,7 +35,7 @@ exit 1
 check_entries () {
        # $1 == directory, $2 == expected
-       grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
+       sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual

This works with BSD sed, but double negatives are confusing. Have you considered this instead?

Quoted 1 line
    sed -ne '/^\//p' ...

What do you mean double negatives? Do you mean using "!" as an alternative delimiter? I find changing delimters is normally simpler than following multiple levels of quoting for escaping slashes, although in this case it's simple enough that it doesn't make much difference.

#24 lore raw

Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data

Quoted 20 lines · John Keeping
Quoted 19 lines · Eric Sunshine
Quoted 10 lines · John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

---

t/t9200-git-cvsexportcommit.sh exit 1 · line 37

Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1 -1
@@ -35,7 +35,7 @@ exit 1
 check_entries () {
        # $1 == directory, $2 == expected
-       grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
+       sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual

This works with BSD sed, but double negatives are confusing. Have you considered this instead?

Quoted 6 lines
Quoted 1 line
    sed -ne '/^\//p' ...

What do you mean double negatives? Do you mean using "!" as an alternative delimiter? I find changing delimters is normally simpler than following multiple levels of quoting for escaping slashes, although in this case it's simple enough that it doesn't make much difference.

Nice, I learned something new today. If I recall correctly, historic sed did not allow the delimiter to be changed (or it wasn't documented or I simply forgot about the capability). So, feel free to ignore me.

#25 lore raw

Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data

In reply to John Keeping
Quoted 20 lines · John Keeping
Quoted 19 lines · Eric Sunshine
Quoted 10 lines · John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

---

t/t9200-git-cvsexportcommit.sh exit 1 · line 37

Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1 -1
@@ -35,7 +35,7 @@ exit 1
 check_entries () {
        # $1 == directory, $2 == expected
-       grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
+       sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual

This works with BSD sed, but double negatives are confusing. Have you considered this instead?

Quoted 6 lines
Quoted 1 line
    sed -ne '/^\//p' ...

What do you mean double negatives? Do you mean using "!" as an alternative delimiter? I find changing delimters is normally simpler than following multiple levels of quoting for escaping slashes, although in this case it's simple enough that it doesn't make much difference.

I agree that changing delimiters is much nicer than backslashes. But I wonder if using "!" is more confusing than it needs to be, given its other meanings.

I dunno. I admit that the backslash threw me off, too (since it needs escaped in interactive shells, I first assumed that's what was going on). Using backslash to select the delimiter was new to me. I've usually seen:

s!/foo/!/bar/!

which is arguably a little more clear. Too bad we cannot do:

m!/foo!

which I think reads better. Oh well. Maybe:

sed -ne '\#^/#p'

would be more readable, but I'm just bikeshedding at this point. The grep invocation really was the most clear. :-/

-Peff

#26 lore raw

Re: [PATCH 2/2] t9200: avoid grep on non-ASCII data

Quoted 42 lines · Jeff King
Quoted 20 lines · John Keeping
Quoted 19 lines · Eric Sunshine
Quoted 10 lines · John Keeping

GNU grep 2.23 detects the input used in this test as binary data so it does not work for extracting lines from a file. We could add the "-a" option to force grep to treat the input as text, but not all implementations support that. Instead, use sed to extract the desired lines since it will always treat its input as text.

---

t/t9200-git-cvsexportcommit.sh exit 1 · line 37

Quoted 4 lines
t/t9200-git-cvsexportcommit.sh +1 -1
@@ -35,7 +35,7 @@ exit 1
 check_entries () {
        # $1 == directory, $2 == expected
-       grep '^/' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual
+       sed -ne '\!^/!p' "$1/CVS/Entries" | sort | cut -d/ -f2,3,5 >actual

This works with BSD sed, but double negatives are confusing. Have you considered this instead?

Quoted 6 lines
Quoted 1 line
    sed -ne '/^\//p' ...

What do you mean double negatives? Do you mean using "!" as an alternative delimiter? I find changing delimters is normally simpler than following multiple levels of quoting for escaping slashes, although in this case it's simple enough that it doesn't make much difference.

I agree that changing delimiters is much nicer than backslashes. But I wonder if using "!" is more confusing than it needs to be, given its other meanings.

I dunno. I admit that the backslash threw me off, too (since it needs escaped in interactive shells, I first assumed that's what was going on). Using backslash to select the delimiter was new to me. I've usually seen:

s!/foo/!/bar/!

which is arguably a little more clear. Too bad we cannot do:

Quoted 8 lines

m!/foo!

which I think reads better. Oh well. Maybe:

sed -ne '\#^/#p'

would be more readable, but I'm just bikeshedding at this point. The grep invocation really was the most clear. :-/

Eric's '/^\//' was the most straight-forward and easiest to see what is going on, I would think.