t: allow use of "sed -E"
RFC 1/1 t: allow use of "sed -E"
Since early 2019 with e62e225f (test-lint: only use only sed [-n] [-e command] [-f command_file], 2019-01-20), we have been trying to limit the options of "sed" we use in our tests to "-e <pattern>", "-n", and "-f <file>".
Before the commit, we were trying to reject only "-i" (which is one of the really-not-portable options), but the commit explicitly wanted to reject use of "-E" (use ERE instead of BRE). The commit cites the then-current POSIX.1 (Issue 7, 2018 edition) to show that "even recent POSIX does not have it!", but the latest edition (Issue
8) documents "-E" as an option to use ERE.
But that was 7 years ago, and that is a long time for many things to happen.
Besides, we have been using "sed -E" without the check in question triggering in one of the scripts since 2022, with 461fec41 (bisect
run: keep some of the post-v2.30.0 output, 2022-11-10). It was hidden because the 'E' was squished with another single letter option.
t/t6030-bisect-porcelain.sh: sed -En 's/.*(bisect...
This escaped the rather simple pattern used in the checker
/\bsed\s+-[^efn]\s+/ and err 'sed option not portable...';
because -E did not appear as a singleton.
Let's change the rule to allow the "-E" option, which nobody has complained against for the past 3 years. We rewrite our first use of the "-E" option so that it is caught by the old rule, primarily because we do not want to teach our mischievous developers how to smuggle in an unwated option undetected by the test lint. And at the same time, loosen the pattern to allow "-E" the same way we allow "-n" and friends.
---
t/check-non-portable-shell.pl | 2 +- t/t6030-bisect-porcelain.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
t/check-non-portable-shell.pl +1 -1
@@ -36,7 +36,7 @@ sub err { $_ = $line; /\bcp\s+-a/ and err 'cp -a is not portable'; - /\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)'; + /\bsed\s+-[^Eefn]\s+/ and err 'sed option not portable (use only -n, -e, -f)'; /\becho\s+-[neE]/ and err 'echo with option is not portable (use printf)'; /^\s*declare\s+/ and err 'arrays/declare not portable'; /^\s*[^#]\s*which\s/ and err 'which is not portable (use type)';
t/t6030-bisect-porcelain.sh +1 -1
@@ -402,7 +402,7 @@ test_expect_success 'git bisect run: negative exit code' " git bisect good $HASH1 && git bisect bad $HASH4 && ! git bisect run ./fail.sh 2>err && - sed -En 's/.*(bisect.*code) (-?[0-9]+) (from.*)/\1 -1 \3/p' err >actual && + sed -E -n 's/.*(bisect.*code) (-?[0-9]+) (from.*)/\1 -1 \3/p' err >actual && test_cmp expect actual "
Quoted 27 lines · Junio C Hamano
Since early 2019 with
e62e225f(test-lint: only use only sed [-n] [-e command] [-fcommand_file], 2019-01-20), we have been trying to limit the options of "sed" we use in our tests to "-e <pattern>", "-n", and "-f <file>".Before the commit, we were trying to reject only "-i" (which is one of the really-not-portable options), but the commit explicitly wanted to reject use of "-E" (use ERE instead of BRE). The commit cites the then-current POSIX.1 (Issue 7, 2018 edition) to show that "even recent POSIX does not have it!", but the latest edition (Issue
8) documents "-E" as an option to use ERE.But that was 7 years ago, and that is a long time for many things to happen.
Besides, we have been using "sed -E" without the check in question triggering in one of the scripts since 2022, with
461fec41(bisect
run: keep some of the post-v2.30.0 output, 2022-11-10). It was hidden because the 'E' was squished with another single letter option.
t/t6030-bisect-porcelain.sh: sed -En 's/.*(bisect...This escaped the rather simple pattern used in the checker
/\bsed\s+-[^efn]\s+/ and err 'sed option not portable...';
Quoted 7 lines
because -E did not appear as a singleton.
Let's change the rule to allow the "-E" option, which nobody has complained against for the past 3 years. We rewrite our first use of the "-E" option so that it is caught by the old rule, primarily because we do not want to teach our mischievous developers how to smuggle in an unwated option undetected by the test lint. And at
"unwated" -> "unwanted", of course ;-)
Quoted 36 lines
the same time, loosen the pattern to allow "-E" the same way we allow "-n" and friends.
Signed-off-by Junio C Hamano <[email protected]>---
t/check-non-portable-shell.pl | 2 +- t/t6030-bisect-porcelain.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)t/check-non-portable-shell.pl +1
-1@@ -36,7 +36,7 @@ sub err { $_ = $line; /\bcp\s+-a/ and err 'cp -a is not portable'; - /\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)'; + /\bsed\s+-[^Eefn]\s+/ and err 'sed option not portable (use only -n, -e, -f)'; /\becho\s+-[neE]/ and err 'echo with option is not portable (use printf)'; /^\s*declare\s+/ and err 'arrays/declare not portable'; /^\s*[^#]\s*which\s/ and err 'which is not portable (use type)';t/t6030-bisect-porcelain.sh +1
-1@@ -402,7 +402,7 @@ test_expect_success 'git bisect run: negative exit code' " git bisect good $HASH1 && git bisect bad $HASH4 && ! git bisect run ./fail.sh 2>err && - sed -En 's/.*(bisect.*code) (-?[0-9]+) (from.*)/\1 -1 \3/p' err >actual && + sed -E -n 's/.*(bisect.*code) (-?[0-9]+) (from.*)/\1 -1 \3/p' err >actual && test_cmp expect actual "
Quoted 2 lines · Junio C Hamano
Junio C Hamano <[email protected]> writes:
[snip]
Quoted 25 lines
Quoted 6 lines
Let's change the rule to allow the "-E" option, which nobody has complained against for the past 3 years. We rewrite our first use of the "-E" option so that it is caught by the old rule, primarily because we do not want to teach our mischievous developers how to smuggle in an unwated option undetected by the test lint. And at
"unwated" -> "unwanted", of course ;-)
Quoted 16 lines
the same time, loosen the pattern to allow "-E" the same way we allow "-n" and friends.
Signed-off-by Junio C Hamano <[email protected]>---
t/check-non-portable-shell.pl | 2 +- t/t6030-bisect-porcelain.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)t/check-non-portable-shell.pl +0
-0@@ -36,7 +36,7 @@ sub err {
Quoted 4 lines
t/check-non-portable-shell.pl sub err { · line 39
Quoted 4 lines
t/check-non-portable-shell.pl +1
-1$_ = $line; /\bcp\s+-a/ and err 'cp -a is not portable'; - /\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)'; + /\bsed\s+-[^Eefn]\s+/ and err 'sed option not portable (use only -n, -e, -f)';
.. and, perhaps, add -E to the error message?
ATB,
Ramsay Jones
Quoted 17 lines
Quoted 16 lines
t/check-non-portable-shell.pl +0
-0/\becho\s+-[neE]/ and err 'echo with option is not portable (use printf)'; /^\s*declare\s+/ and err 'arrays/declare not portable'; /^\s*[^#]\s*which\s/ and err 'which is not portable (use type)';t/t6030-bisect-porcelain.sh +1
-1@@ -402,7 +402,7 @@ test_expect_success 'git bisect run: negative exit code' " git bisect good $HASH1 && git bisect bad $HASH4 && ! git bisect run ./fail.sh 2>err && - sed -En 's/.*(bisect.*code) (-?[0-9]+) (from.*)/\1 -1 \3/p' err >actual && + sed -E -n 's/.*(bisect.*code) (-?[0-9]+) (from.*)/\1 -1 \3/p' err >actual && test_cmp expect actual "
Quoted 4 lines · Ramsay Jones
Quoted 2 lines
t/check-non-portable-shell.pl sub err { · line 39
Quoted 2 lines
t/check-non-portable-shell.pl +1
-1- /\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)'; + /\bsed\s+-[^Eefn]\s+/ and err 'sed option not portable (use only -n, -e, -f)';.. and, perhaps, add -E to the error message?
Yes, indeed. Thanks.
PATCH v2 1/1 t: allow use of "sed -E"
In reply to Junio C HamanoSince early 2019 with e62e225f (test-lint: only use only sed [-n] [-e command] [-f command_file], 2019-01-20), we have been trying to limit the options of "sed" we use in our tests to "-e <pattern>", "-n", and "-f <file>".
Before the commit, we were trying to reject only "-i" (which is one of the really-not-portable options), but the commit explicitly wanted to reject use of "-E" (use ERE instead of BRE). The commit cites the then-current POSIX.1 (Issue 7, 2018 edition) to show that "even recent POSIX does not have it!", but the latest edition (Issue
8) documents "-E" as an option to use ERE.
But that was 7 years ago, and that is a long time for many things to happen.
Besides, we have been using "sed -E" without the check in question triggering in one of the scripts since 2022, with 461fec41 (bisect
run: keep some of the post-v2.30.0 output, 2022-11-10). It was hidden because the 'E' was squished with another single letter option.
t/t6030-bisect-porcelain.sh: sed -En 's/.*(bisect...
This escaped the rather simple pattern used in the checker
/\bsed\s+-[^efn]\s+/ and err 'sed option not portable...';
because -E did not appear as a singleton.
Let's change the rule to allow the "-E" option, which nobody has complained against for the past 3 years. We rewrite our first use of the "-E" option so that it is caught by the old rule, primarily because we do not want to teach our mischievous developers how to smuggle in an unwanted option undetected by the test lint. And at the same time, loosen the pattern to allow "-E" the same way we allow "-n" and friends.
--- t/check-non-portable-shell.pl | 2 +- t/t6030-bisect-porcelain.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
t/check-non-portable-shell.pl +1 -1
@@ -36,7 +36,7 @@ sub err { $_ = $line; /\bcp\s+-a/ and err 'cp -a is not portable'; - /\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)'; + /\bsed\s+-[^Eefn]\s+/ and err 'sed option not portable (use only -E, -n, -e, -f)'; /\becho\s+-[neE]/ and err 'echo with option is not portable (use printf)'; /^\s*declare\s+/ and err 'arrays/declare not portable'; /^\s*[^#]\s*which\s/ and err 'which is not portable (use type)';
t/t6030-bisect-porcelain.sh +1 -1
@@ -402,7 +402,7 @@ test_expect_success 'git bisect run: negative exit code' " git bisect good $HASH1 && git bisect bad $HASH4 && ! git bisect run ./fail.sh 2>err && - sed -En 's/.*(bisect.*code) (-?[0-9]+) (from.*)/\1 -1 \3/p' err >actual && + sed -E -n 's/.*(bisect.*code) (-?[0-9]+) (from.*)/\1 -1 \3/p' err >actual && test_cmp expect actual "
Range-diff:
1: 43a56fa521 ! 1: be430f4eaa t: allow use of "sed -E"
@@ Commit message
complained against for the past 3 years. We rewrite our first use
of the "-E" option so that it is caught by the old rule, primarily
because we do not want to teach our mischievous developers how to
- smuggle in an unwated option undetected by the test lint. And at
+ smuggle in an unwanted option undetected by the test lint. And at
the same time, loosen the pattern to allow "-E" the same way we
allow "-n" and friends.
@@ t/check-non-portable-shell.pl: sub err {
$_ = $line;
/\bcp\s+-a/ and err 'cp -a is not portable';
- /\bsed\s+-[^efn]\s+/ and err 'sed option not portable (use only -n, -e, -f)';
-+ /\bsed\s+-[^Eefn]\s+/ and err 'sed option not portable (use only -n, -e, -f)';
++ /\bsed\s+-[^Eefn]\s+/ and err 'sed option not portable (use only -E, -n, -e, -f)';
/\becho\s+-[neE]/ and err 'echo with option is not portable (use printf)';
/^\s*declare\s+/ and err 'arrays/declare not portable';
/^\s*[^#]\s*which\s/ and err 'which is not portable (use type)';Signature · 1 line
2.53.0-698-g1cc7e6b802
Quoted 21 lines · Junio C Hamano
Since early 2019 with
e62e225f(test-lint: only use only sed [-n] [-e command] [-fcommand_file], 2019-01-20), we have been trying to limit the options of "sed" we use in our tests to "-e <pattern>", "-n", and "-f <file>".Before the commit, we were trying to reject only "-i" (which is one of the really-not-portable options), but the commit explicitly wanted to reject use of "-E" (use ERE instead of BRE). The commit cites the then-current POSIX.1 (Issue 7, 2018 edition) to show that "even recent POSIX does not have it!", but the latest edition (Issue
8) documents "-E" as an option to use ERE.But that was 7 years ago, and that is a long time for many things to happen.
Besides, we have been using "sed -E" without the check in question triggering in one of the scripts since 2022, with
461fec41(bisect
run: keep some of the post-v2.30.0 output, 2022-11-10). It was hidden because the 'E' was squished with another single letter option.
Quoted 7 lines
t/t6030-bisect-porcelain.sh: sed -En 's/.*(bisect...This escaped the rather simple pattern used in the checker
/\bsed\s+-[^efn]\s+/ and err 'sed option not portable...';because -E did not appear as a singleton.
Makes me wonder whether we also want to harden this regex. But even if we started to understand that multiple single-letter options can be squished together it wouldn't be sufficient, as we don't know to process multiple arugments, either. And I guess it doesn't make sense to grow a full command line parser here.
Quoted 7 lines
Let's change the rule to allow the "-E" option, which nobody has complained against for the past 3 years. We rewrite our first use of the "-E" option so that it is caught by the old rule, primarily because we do not want to teach our mischievous developers how to smuggle in an unwanted option undetected by the test lint. And at the same time, loosen the pattern to allow "-E" the same way we allow "-n" and friends.
Feels reasonable to me.
Patrick
Quoted 12 lines · Junio C Hamano
Since early 2019 with
e62e225f(test-lint: only use only sed [-n] [-e command] [-fcommand_file], 2019-01-20), we have been trying to limit the options of "sed" we use in our tests to "-e <pattern>", "-n", and "-f <file>".Before the commit, we were trying to reject only "-i" (which is one of the really-not-portable options), but the commit explicitly wanted to reject use of "-E" (use ERE instead of BRE). The commit cites the then-current POSIX.1 (Issue 7, 2018 edition) to show that "even recent POSIX does not have it!", but the latest edition (Issue
8) documents "-E" as an option to use ERE.
Quoted 2 lines
But that was 7 years ago, and that is a long time for many things to happen.
I think this seems reasonable. I know this works on both reasonably new Linux machines and on macOS, and I did some spot checking with the FreeBSD manual page viewer[0] and it's present in the following:
* at least FreeBSD 9.0 (current is 15.0)
* at least NetBSD 7.0 (current is 10.1)
* at least OpenBSD 6.0 (current is 7.8)
* Ubuntu 18.04
so this appears to be reasonably well supported across major open source distributions. The irony is that Linux is most likely the OS holding us back here, since older versions labeled this -r and the newer -E wasn't available in Ubuntu 16.04. I myself only recently learned about the fact that -E had been officially standardized, since I'd been using -r for a long time.
And, for the record, I agree with you that -i is totally non-portable because it _requires_ an argument on BSD sed and the argument is optional on GNU sed (and cannot be specified as ''). I have a testsuite at work where I gave up and just used perl -i instead.
Quoted 5 lines
Besides, we have been using "sed -E" without the check in question triggering in one of the scripts since 2022, with
461fec41(bisect
run: keep some of the post-v2.30.0 output, 2022-11-10). It was hidden because the 'E' was squished with another single letter option.
Yes, I think if nobody has complained about it in three years, we should be fine.
[0] https://man.freebsd.org/cgi/man.cgi
Signature · 2 lines
brian m. carlson (they/them) Toronto, Ontario, CA
Quoted 2 lines · brian m. carlson
Yes, I think if nobody has complained about it in three years, we should be fine.
This actually started when I noticed a recent rewrite of a test that turned "grep -E ... | sed -e ..." into "sed -E -e ..." was flagged by the test-lint.
It is not a huge deal to rewrite the resulting sed script again to use BRE instead of ERE and it may not be a bad idea doing so anyway (and rewrite the existing t6030). That can be done independently.
Quoted 6 lines · brian m. carlson
so this appears to be reasonably well supported across major open source distributions. The irony is that Linux is most likely the OS holding us back here, since older versions labeled this
-rand the newer-Ewasn't available in Ubuntu 16.04. I myself only recently learned about the fact that-Ehad been officially standardized, since I'd been using-rfor a long time.
IIRC, GNU sed supported -E for a very long time before it was documented. In a quick search, I see it added in v4.1a, in commit 3a8e165 (treat cygwin CR/LF correctly (and by design, not by chance), 2006-08-07).
https://cgit.git.savannah.gnu.org/cgit/sed.git/diff/sed/sed.c?id=3a8e165
Signature · 1 line
Todd