2.21.0-rc0 test fixes resulting from use of /dev/zero
cover 2.21.0-rc0 test fixes resulting from use of /dev/zero
From: "Randall S. Becker" <[email protected]>
This is a candidate packages of fixes to remove dependence on /dev/zero and replaces it with the generate_zero_bytes function in test-lib-functions.sh
Randall S. Becker (3):
test-lib-functions.sh: add generate_zero_bytes function
t5318: replace use of /dev/zero with generate_zero_bytes
t5562: replace /dev/zero with a pipe from generate_zero_bytes
t/t5318-commit-graph.sh | 2 +- t/t5562-http-backend-content-length.sh | 4 ++-- t/test-lib-functions.sh | 13 +++++++++++++ 3 files changed, 16 insertions(+), 3 deletions(-)
Signature · 1 line
2.12.3
PATCH 1/3 test-lib-functions.sh: add generate_zero_bytes function
From: "Randall S. Becker" <[email protected]>
t5318 and t5562 used /dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs.
--- t/test-lib-functions.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+)
t/test-lib-functions.sh +13 -0
@@ -116,6 +116,19 @@ remove_cr () { tr '\015' Q | sed -e 's/Q$//' } +# Generate an output of $1 bytes of all zeroes (NULs, not ASCII zeroes). +# If $1 is 'infinity', output forever or until the receiving pipe stops reading, +# whichever comes first. +generate_zero_bytes () { + perl -e 'if ($ARGV[0] == "infinity") { + while (-1) { + print "\0" + } + } else { + print "\0" x $ARGV[0] + }' "$@" +} + # In some bourne shell implementations, the "unset" builtin returns # nonzero status when a variable to be unset was not set in the first # place.
Signature · 1 line
2.12.3
Re: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function
t/test-lib-functions.sh remove_cr () { · line 118
Quoted 10 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
t5318 and t5562 used
/dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs.Signed-off-by Randall S. Becker <[email protected]>---
t/test-lib-functions.sh +3
-0@@ -116,6 +116,19 @@ remove_cr () { +# Generate an output of $1 bytes of all zeroes (NULs, not ASCII zeroes). +# If $1 is 'infinity', output forever or until the receiving pipe stops reading, +# whichever comes first.
This is a somewhat unusual API. A (perhaps) more intuitive behavior would be for it to emit an infinite stream of NULs when given no argument, and a limited number of NULs when given an argument. Redefining the behavior like that also fixes the "problem" with the current implementation erroring-out if no argument is provided.
t/test-lib-functions.sh remove_cr () { · line 123
Quoted 2 lines
t/test-lib-functions.sh +2
-0+generate_zero_bytes () { + perl -e 'if ($ARGV[0] == "infinity") {
s/perl/"$PERL_PATH"/
t/test-lib-functions.sh remove_cr () { · line 126
Quoted 3 lines
t/test-lib-functions.sh +3
-0+ while (-1) { + print "\0" + }
Or, more compactly:
print "\0" while 1;
Quoted 4 lines
t/test-lib-functions.sh +4
-0+ } else { + print "\0" x $ARGV[0] + }' "$@" +}
RE: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function
Quoted 4 lines · Eric Sunshine
Quoted 3 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
t5318 and t5562 used
/dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs.
Quoted 13 lines
t/test-lib-functions.sh @@
Quoted 7 lines
Signed-off-by Randall S. Becker <[email protected]>---
t/test-lib-functions.sh @@ renamed from t/test-lib-functions.sh +3
-1-116,6 +116,19 @@ remove_cr () { +# Generate an output of $1 bytes of all zeroes (NULs, not ASCII zeroes). +# If $1 is 'infinity', output forever or until the receiving pipe +stops reading, # whichever comes first.This is a somewhat unusual API. A (perhaps) more intuitive behavior would be for it to emit an infinite stream of NULs when given no argument, and a limited number of NULs when given an argument.
Redefining the behavior like that also fixes the "problem" with the current implementation erroring-out if no argument is provided.
At this point, I've supplied three different ways to solve this incompatibility for platforms not Linux and others have also provided fixes and discussed this at length. It is not a specific fix that matters to me, but that there is a fix at all. So thanks for all your comments and I will wait on direction on what the team wants me to do about it, if anything.
Regards,
Randall
Re: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function
In reply to Eric SunshineQuoted 4 lines · Eric Sunshine
Quoted 3 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
t5318 and t5562 used
/dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs.
Quoted 13 lines
t/test-lib-functions.sh remove_cr () { · line 118
Quoted 7 lines
Signed-off-by Randall S. Becker <[email protected]>---
t/test-lib-functions.sh +3
-0@@ -116,6 +116,19 @@ remove_cr () { +# Generate an output of $1 bytes of all zeroes (NULs, not ASCII zeroes). +# If $1 is 'infinity', output forever or until the receiving pipe stops reading, +# whichever comes first.This is a somewhat unusual API. A (perhaps) more intuitive behavior would be for it to emit an infinite stream of NULs when given no argument, and a limited number of NULs when given an argument. Redefining the behavior like that also fixes the "problem" with the current implementation erroring-out if no argument is provided.
Yeah, I agree that'd be a little more idiomatic for our code base, but I'm fine with it either way.
Quoted 4 lines
t/test-lib-functions.sh remove_cr () { · line 123
Quoted 2 lines
t/test-lib-functions.sh +2
-0+generate_zero_bytes () { + perl -e 'if ($ARGV[0] == "infinity") {s/perl/"$PERL_PATH"/
This shouldn't be necessary. perl() is a function that uses $PERL_PATH (so you only need $PERL_PATH when you're writing out another script that doesn't run in the same process space as the rest of the test code).
-Peff
Re: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function
Quoted 10 lines · Jeff King
Quoted 5 lines · Eric Sunshine
t/test-lib-functions.sh remove_cr () { · line 123
Quoted 2 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
t/test-lib-functions.sh +2
-0+generate_zero_bytes () { + perl -e 'if ($ARGV[0] == "infinity") {s/perl/"$PERL_PATH"/
This shouldn't be necessary.
perl()is a function that uses $PERL_PATH (so you only need $PERL_PATH when you're writing out another script that doesn't run in the same process space as the rest of the test code).
Thanks for clarifying. I either didn't know or forgot about that.
RE: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function
Quoted 6 lines · Eric Sunshine
Quoted 5 lines · Jeff King
Quoted 4 lines · Eric Sunshine
Quoted 2 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
t/test-lib-functions.sh +2
-0+generate_zero_bytes () { + perl -e 'if ($ARGV[0] == "infinity") {
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
s/perl/"$PERL_PATH"/
This shouldn't be necessary.
perl()is a function that uses $PERL_PATH (so you only need $PERL_PATH when you're writing out another script that doesn't run in the same process space as the rest of the test code).Thanks for clarifying. I either didn't know or forgot about that.
As did I, thank you.
PATCH 2/3 t5318: replace use of /dev/zero with generate_zero_bytes
In reply to [email protected]From: "Randall S. Becker" <[email protected]>
This change removes the dependency on /dev/zero with generate_zero_bytes appending NUL values to blocks generating wrong signatures for test cases.
--- t/t5318-commit-graph.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5318-commit-graph.sh +1 -1
@@ -383,7 +383,7 @@ corrupt_graph_and_verify() { cp $objdir/info/commit-graph commit-graph-backup && printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc && dd of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=0 && - dd if=/dev/zero of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=$(($orig_size - $zero_pos)) && + generate_zero_bytes $(($orig_size - $zero_pos)) >> "$objdir/info/commit-graph" && test_must_fail git commit-graph verify 2>test_err && grep -v "^+" test_err >err && test_i18ngrep "$grepstr" err
Signature · 1 line
2.12.3
Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes
Quoted 2 lines · On Sat, Feb 9, 2019 at 2:00 PM <[email protected]> wrote:
This change removes the dependency on
/dev/zerowithgenerate_zero_bytesappending NUL values to blocks generating wrong signatures for test cases.
This commit message says what the patch does but not _why_. At minimum, it should explain that /dev/zero is not available on all platforms, therefore, not portable, and (perhaps) cite NonStop as an example.
Quoted 6 lines
Signed-off-by Randall S. Becker <[email protected]>---
t/t5318-commit-graph.sh +1
-1@@ -383,7 +383,7 @@ corrupt_graph_and_verify() { - dd if=/dev/zero of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=$(($orig_size - $zero_pos)) && + generate_zero_bytes $(($orig_size - $zero_pos)) >> "$objdir/info/commit-graph" &&
Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes
Quoted 8 lines · Eric Sunshine
Quoted 2 lines · On Sat, Feb 9, 2019 at 2:00 PM <[email protected]> wrote:
This change removes the dependency on
/dev/zerowithgenerate_zero_bytesappending NUL values to blocks generating wrong signatures for test cases.This commit message says what the patch does but not _why_. At minimum, it should explain that
/dev/zerois not available on all platforms, therefore, not portable, and (perhaps) cite NonStop as an example.
Does sombody want to do the honors? [PATCH 1/3] would become wasted effort until that happens. On the other hand, if this is not urgent (it is only urgent for those without /dev/zero, and to others it may be distraction/disruption this close to the final release to add increased risk of fat finger mistakes), obviously I can wait.
Quoted 6 lines
t/t5318-commit-graph.sh corrupt_graph_and_verify() { · line 383
Quoted 6 lines
Signed-off-by Randall S. Becker <[email protected]>---
t/t5318-commit-graph.sh +1
-1@@ -383,7 +383,7 @@ corrupt_graph_and_verify() { - dd if=/dev/zero of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=$(($orig_size - $zero_pos)) && + generate_zero_bytes $(($orig_size - $zero_pos)) >> "$objdir/info/commit-graph" &&
The original says "skip to the $zero_pos location in an existing info/commit-graph file, and overwrite its existing contents up to the $orig_size location with NULs".
If I recall the implementation of generate_zero_bytes in [PATCH 1/3] correctly, it just generated stream of NULs of given number of bytes.
The only reason why this rewrite is not wrong is because the previous line (omitted in your quote) truncates the file to $zero_pos. That is a bit tricky ;-).
Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes
Quoted 11 lines · Junio C Hamano
Quoted 8 lines · Eric Sunshine
Quoted 2 lines · On Sat, Feb 9, 2019 at 2:00 PM <[email protected]> wrote:
This change removes the dependency on
/dev/zerowithgenerate_zero_bytesappending NUL values to blocks generating wrong signatures for test cases.This commit message says what the patch does but not _why_. At minimum, it should explain that
/dev/zerois not available on all platforms, therefore, not portable, and (perhaps) cite NonStop as an example.
Quoted 5 lines
Does sombody want to do the honors? [PATCH 1/3] would become wasted effort until that happens. On the other hand, if this is not urgent (it is only urgent for those without
/dev/zero, and to others it may be distraction/disruption this close to the final release to add increased risk of fat finger mistakes), obviously I can wait.
So, before I lose the access to my primary screen (I was told that somehow I need to reimage the workstation today X-<), here is what I have now.
-- >8 --
From: "Randall S. Becker" <[email protected]>
Date: Sat, 9 Feb 2019 13:59:29 -0500
Subject: [PATCH] t5318: replace use of /dev/zero with generate_zero_bytes
There are platforms (e.g. NonStop) that lack /dev/zero; use the generate_zero_bytes helper we just introduced to append stream of NULs at the end of the file.
The original, even though it uses "dd seek=... count=..." to make it look like it is overwriting the middle part of an existing file, has truncated the file before this step with another use of "dd", which may make it tricky to see why this rewrite is a correct one.
--- t/t5318-commit-graph.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5318-commit-graph.sh +1 -1
@@ -383,7 +383,7 @@ corrupt_graph_and_verify() { cp $objdir/info/commit-graph commit-graph-backup && printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc && dd of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=0 && - dd if=/dev/zero of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=$(($orig_size - $zero_pos)) && + generate_zero_bytes $(($orig_size - $zero_pos)) >>"$objdir/info/commit-graph" && test_must_fail git commit-graph verify 2>test_err && grep -v "^+" test_err >err && test_i18ngrep "$grepstr" err
Signature · 1 line
2.21.0-rc0-36-ge9bd4aa026
RE: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes
Quoted 5 lines
-----Original Message-----
From: Junio C Hamano <[email protected]> On Behalf Of Junio C Hamano
Sent: February 13, 2019 12:25
To: Eric Sunshine <[email protected]>
Cc: [email protected]; Git List <[email protected]>; Randall
S.
Quoted 18 lines
Becker <[email protected]>
Subject: Re: [Patch v1 2/3] t5318: replace use of/dev/zerowithgenerate_zero_bytesQuoted 5 lines · Junio C Hamano
Quoted 3 lines · Eric Sunshine
Quoted 2 lines · On Sat, Feb 9, 2019 at 2:00 PM <[email protected]> wrote:
This change removes the dependency on
/dev/zerowithgenerate_zero_bytesappending NUL values to blocks generating wrongsignatures for test cases.
Quoted 6 lines
Quoted 5 lines
This commit message says what the patch does but not _why_. At minimum, it should explain that
/dev/zerois not available on all platforms, therefore, not portable, and (perhaps) cite NonStop as an example.
Quoted 7 lines
Quoted 5 lines
Does sombody want to do the honors? [PATCH 1/3] would become wasted effort until that happens. On the other hand, if this is not urgent (it is only urgent for those without
/dev/zero, and to others it may be distraction/disruption this close to the final release to add increased risk of fat finger mistakes), obviously I can wait.So, before I lose the access to my primary screen (I was told that somehow
I
Quoted 7 lines
need to reimage the workstation today X-<), here is what I have now.
-- >8 --
From: "Randall S. Becker" <[email protected]>
Date: Sat, 9 Feb 2019 13:59:29 -0500
Subject: [PATCH] t5318: replace use of/dev/zerowithgenerate_zero_bytes
Quoted 5 lines
There are platforms (e.g. NonStop) that lack
/dev/zero; use thegenerate_zero_byteshelper we just introduced to append stream of NULs at the end of the file.The original, even though it uses "dd seek=... count=..." to make it look
like it
Quoted 1 line
is overwriting the middle part of an existing file, has truncated the file
before
Quoted 1 line
this step with another use of "dd", which may make it tricky to see why
this
Quoted 1 line
rewrite is a correct one.
Here is how I interpret the test - might be wrong, but yanno... The first dd copies something looking like reasonable data from the test case.
The second dd copies zeros from seek to the end of a fixed size block.
My first attempt at a fix used truncate that extended the first to the correct size (filling with zeros). My worry there is that I'm not sure there is a guarantee of zeros, but that shouldn't matter for the test which just wants a signature mismatch.
Others suggested using yes to fill in junk.
My second attempt was to create the generate_zero_bytes function to replace exactly what the second dd was doing but not user /dev/zero. The fix was not to change the conditions of the test - not debating the correctness of that here - but to simply replicate the use of /dev/zero in context. So the resulting file contains [reasonable-stuff]{seek}[0]{orig_size-seek}, which is sufficient to satisfy the conditions of the test.
Quoted 24 lines
Signed-off-by Randall S. Becker <[email protected]>Signed-off-by Junio C Hamano <[email protected]>--- t/t5318-commit-graph.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)t/t5318-commit-graph.sh index renamed from t/t5318-commit-graph.sh +0
-0
16d10ebce8..d4bd1522fe100755t/t5318-commit-graph.sh +0
-0@@ -383,7 +383,7 @@ corrupt_graph_and_verify() { cp $objdir/info/commit-graph commit-graph-backup && printf "$data" | dd of="$objdir/info/commit-graph" bs=1seek="$pos" conv=notrunc &&
dd of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=0
&&t/t5318-commit-graph.sh +0
-1- dd if=/dev/zero of="$objdir/info/commit-graph" bs=1seek="$zero_pos" count=$(($orig_size - $zero_pos)) &&
t/t5318-commit-graph.sh +2
-0+ generate_zero_bytes $(($orig_size - $zero_pos)) +>>"$objdir/info/commit-graph" && test_must_fail git commit-graph verify 2>test_err && grep -v "^+" test_err >err && test_i18ngrep "$grepstr" err
Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes
Quoted 2 lines · Randall S. Becker
My second attempt was to create the
generate_zero_bytesfunction to replace exactly what the second dd was doing but not user/dev/zero.
Yes, and I think the patch does that ;-) It was just the original
dd if=/dev/zero of=... bs=1 seek=$there count=$this_many
would have been impossible to rewrite with the new generate_zero_bytes helper unless $there weren't seeking to the end of the file.
But the other dd before the one the patch rewrites truncates the file to make that seek=$there seeking to the end of the file, so simply appending output from genereate_zero_bytes is sufficient and correct conversion. I wanted to explain that for future readers who may wonder if the patch is doing the exact conversion.
RE: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes
Quoted 5 lines · Junio C Hamano
Quoted 2 lines · Randall S. Becker
My second attempt was to create the
generate_zero_bytesfunction to replace exactly what the second dd was doing but not user/dev/zero.
Quoted 8 lines
Yes, and I think the patch does that ;-) It was just the original
dd if=/dev/zero of=... bs=1 seek=$there count=$this_many
would have been impossible to rewrite with the new
generate_zero_byteshelper unless $there weren't seeking to the end of the file.But the other dd before the one the patch rewrites truncates the file to
make
Quoted 1 line
that seek=$there seeking to the end of the file, so simply appending
output
Quoted 1 line
from
genereate_zero_bytesis sufficient and correct conversion. I wanted
to
Quoted 2 lines
explain that for future readers who may wonder if the patch is doing the exact conversion.
Sounds like we need a documentation patch in the actual test suite rather than in the commit <ducking>.
Cheers,
Randall
PATCH 3/3 t5562: replace /dev/zero with a pipe from generate_zero_bytes
In reply to [email protected]From: "Randall S. Becker" <[email protected]>
This change removes the dependency on /dev/zero with an equivalent pipe of deliberately NUL bytes. This allows tests to proceed where /dev/zero does not exist.
--- t/t5562-http-backend-content-length.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
t/t5562-http-backend-content-length.sh +2 -2
@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' ' test_expect_success 'CONTENT_LENGTH overflow ssite_t' ' NOT_FIT_IN_SSIZE=$(ssize_b100dots) && - env \ + generate_zero_bytes infinity | env \ CONTENT_TYPE=application/x-git-upload-pack-request \ QUERY_STRING=/repo.git/git-upload-pack \ PATH_TRANSLATED="$PWD"/.git/git-upload-pack \ GIT_HTTP_EXPORT_ALL=TRUE \ REQUEST_METHOD=POST \ CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \ - git http-backend </dev/zero >/dev/null 2>err && + git http-backend >/dev/null 2>err && grep "fatal:.*CONTENT_LENGTH" err '
Signature · 1 line
2.12.3
Re: [Patch v1 3/3] t5562: replace /dev/zero with a pipe from generate_zero_bytes
Quoted 1 line · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
This change removes the dependency on
/dev/zerowith an equivalent pipe of
Too many spaces between "equivalent" and "pipe".
Quoted 2 lines
deliberately NUL bytes. This allows tests to proceed where
/dev/zerodoes not exist.
It wouldn't hurt to cite "NonStop" as an example of a platform lacking /dev/zero.
The first sentence is a bit off grammatically. Perhaps the entire commit message can be collapsed simply to:
Stop depending on /dev/zero which may not be available on all platforms (for instance, HP NonStop).
Quoted 18 lines
Signed-off-by Randall S. Becker <[email protected]>---
t/t5562-http-backend-content-length.sh +2
-2@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' ' test_expect_success 'CONTENT_LENGTH overflow ssite_t' ' NOT_FIT_IN_SSIZE=$(ssize_b100dots) && - env \ + generate_zero_bytes infinity | env \ CONTENT_TYPE=application/x-git-upload-pack-request \ QUERY_STRING=/repo.git/git-upload-pack \ PATH_TRANSLATED="$PWD"/.git/git-upload-pack \ GIT_HTTP_EXPORT_ALL=TRUE \ REQUEST_METHOD=POST \ CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \ - git http-backend </dev/zero >/dev/null 2>err && + git http-backend >/dev/null 2>err && grep "fatal:.*CONTENT_LENGTH" err '
Re: [Patch v1 3/3] t5562: replace /dev/zero with a pipe from generate_zero_bytes
Eric Sunshine <[email protected]> writes:
[jc: Summoning Dscho, J6t for their Windows expertise at the end]
Quoted 17 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
Quoted 1 line
This change removes the dependency on
/dev/zerowith an equivalent pipe ofToo many spaces between "equivalent" and "pipe".
Quoted 2 lines
deliberately NUL bytes. This allows tests to proceed where
/dev/zerodoes not exist.It wouldn't hurt to cite "NonStop" as an example of a platform lacking
/dev/zero.The first sentence is a bit off grammatically. Perhaps the entire commit message can be collapsed simply to:
Stop depending on
/dev/zerowhich may not be available on all platforms (for instance, HP NonStop).
Quoted 16 lines
Quoted 6 lines
Signed-off-by Randall S. Becker <[email protected]>---
t/t5562-http-backend-content-length.sh +0
-0@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' ' test_expect_success 'CONTENT_LENGTH overflow ssite_t' ' NOT_FIT_IN_SSIZE=$(ssize_b100dots) &&t/t5562-http-backend-content-length.sh test_expect_success GZIP 'push gzipped empty' ' · line 152
Quoted 10 lines
t/t5562-http-backend-content-length.sh +2
-2- env \ + generate_zero_bytes infinity | env \ CONTENT_TYPE=application/x-git-upload-pack-request \ QUERY_STRING=/repo.git/git-upload-pack \ PATH_TRANSLATED="$PWD"/.git/git-upload-pack \ GIT_HTTP_EXPORT_ALL=TRUE \ REQUEST_METHOD=POST \ CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \ - git http-backend </dev/zero >/dev/null 2>err && + git http-backend >/dev/null 2>err &&
Doesn't this "inifinity" mode have the same issue that was worked around by 6129c930 ("test-lib: limit the output of the yes utility", 2016-02-02) on Windows? If I read correctly, the process upstream of the pipe (in this case, perl producing a stream of infinite NULs) would not die when the downstream stops reading with SIGPIPE.
Or is it safe to assume that nobody expects to use http-backend on Windows based servers so this test is a non-issue?
Quoted 2 lines
t/t5562-http-backend-content-length.sh test_expect_success GZIP 'push gzipped empty' ' · line 155
Quoted 2 lines
t/t5562-http-backend-content-length.sh +0
-0grep "fatal:.*CONTENT_LENGTH" err '
Thanks.
Re: [Patch v1 3/3] t5562: replace /dev/zero with a pipe from generate_zero_bytes
Am 12.02.19 um 18:24 schrieb Junio C Hamano:
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
t/t5562-http-backend-content-length.sh +1
-1@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' ' test_expect_success 'CONTENT_LENGTH overflow ssite_t' ' NOT_FIT_IN_SSIZE=$(ssize_b100dots) && - env \ + generate_zero_bytes infinity | env \ CONTENT_TYPE=application/x-git-upload-pack-request \ QUERY_STRING=/repo.git/git-upload-pack \ PATH_TRANSLATED="$PWD"/.git/git-upload-pack \ GIT_HTTP_EXPORT_ALL=TRUE \ REQUEST_METHOD=POST \ CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \
Quoted 8 lines
Quoted 2 lines
t/t5562-http-backend-content-length.sh test_expect_success GZIP 'push gzipped empty' ' · line 153
Quoted 2 lines
t/t5562-http-backend-content-length.sh +1
-1- git http-backend </dev/zero >/dev/null 2>err && + git http-backend >/dev/null 2>err &&Doesn't this "inifinity" mode have the same issue that was worked around by
6129c930("test-lib: limit the output of the yes utility", 2016-02-02) on Windows? If I read correctly, the process upstream of the pipe (in this case, perl producing a stream of infinite NULs) would not die when the downstream stops reading with SIGPIPE.
I think we do not have to worry, and the reason is that the justification for 6129c930 is simply wrong.
As I did not find the patch series discussed here to pull and test, I repeated the timing tests with t7610-mergetool.sh with and without 6129c930 reverted, and the difference is only in the noise. The reason t7610 takes so long on Windows looks more like a consequence of the 10,000 processes that it spawns. It is a mystery to me how I came to the conclusion that the change in 6129c930 would make a difference. :-(
-- Hannes
Re: [Patch v1 3/3] t5562: replace /dev/zero with a pipe from generate_zero_bytes
Quoted 22 lines · Johannes Sixt
Am 12.02.19 um 18:24 schrieb Junio C Hamano:
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
t/t5562-http-backend-content-length.sh +1
-1@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' ' test_expect_success 'CONTENT_LENGTH overflow ssite_t' ' NOT_FIT_IN_SSIZE=$(ssize_b100dots) && - env \ + generate_zero_bytes infinity | env \ CONTENT_TYPE=application/x-git-upload-pack-request \ QUERY_STRING=/repo.git/git-upload-pack \ PATH_TRANSLATED="$PWD"/.git/git-upload-pack \ GIT_HTTP_EXPORT_ALL=TRUE \ REQUEST_METHOD=POST \ CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \Quoted 8 lines
Quoted 2 lines
t/t5562-http-backend-content-length.sh test_expect_success GZIP 'push gzipped empty' ' · line 153
Quoted 2 lines
t/t5562-http-backend-content-length.sh +1
-1- git http-backend </dev/zero >/dev/null 2>err && + git http-backend >/dev/null 2>err &&Doesn't this "inifinity" mode have the same issue that was worked around by
6129c930("test-lib: limit the output of the yes utility", 2016-02-02) on Windows? If I read correctly, the process upstream of the pipe (in this case, perl producing a stream of infinite NULs) would not die when the downstream stops reading with SIGPIPE.
Quoted 2 lines
I think we do not have to worry, and the reason is that the justification for
6129c930is simply wrong.
That's kinda surprising but in a pleasant way---it's good that we have one less thing we need to worry about.
Thanks.
Quoted 9 lines
As I did not find the patch series discussed here to pull and test, I repeated the timing tests with
t7610-mergetool.shwith and without6129c930reverted, and the difference is only in the noise. The reason t7610 takes so long on Windows looks more like a consequence of the 10,000 processes that it spawns. It is a mystery to me how I came to the conclusion that the change in6129c930would make a difference. :-(-- Hannes
PATCH 1/1 t5562: do not depend on /dev/zero
In reply to [email protected]It was reported [1] that NonStop platform does not have /dev/zero.
The test uses /dev/zero as a dummy input. Passing case (http-backed failed because of too big input size) should not be reading anything from it. If http-backend would erroneously try to read any data returning EOF probably would be even safer than providing some meaningless data.
Replace /dev/zero with /dev/null to avoid issues with platforms which do not have /dev/zero.
[1] https://public-inbox.org/git/[email protected]/
---
By the way, I don't think this requires such sofisticated fix. In the success case the input would not be read at all. You could replace it with /dev/null, the in failure (not immediate fail) git would fail due to truncated input or something.
Also, as you experience hang issue [2] in earlier tests, this one should not have contributed to it.
[2] https://public-inbox.org/git/[email protected]/
t/t5562-http-backend-content-length.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5562-http-backend-content-length.sh +1 -1
@@ -150,7 +150,7 @@ test_expect_success 'CONTENT_LENGTH overflow ssite_t' ' GIT_HTTP_EXPORT_ALL=TRUE \ REQUEST_METHOD=POST \ CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \ - git http-backend </dev/zero >/dev/null 2>err && + git http-backend </dev/null >/dev/null 2>err && grep "fatal:.*CONTENT_LENGTH" err '
Signature · 1 line
2.19.0.1202.g68e1e8f04e
RE: [PATCH] t5562: do not depend on /dev/zero
Quoted 4 lines · Max Kirillov
It was reported [1] that NonStop platform does not have
/dev/zero.The test uses
/dev/zeroas a dummy input. Passing case (http-backed failed because of too big input size) should not be reading anything from it. If
http-
Quoted 4 lines
backend would erroneously try to read any data returning EOF probably would be even safer than providing some meaningless data.
Replace
/dev/zerowith/dev/nullto avoid issues with platforms which do
not
Quoted 9 lines
have
/dev/zero.[1] https://public-inbox.org/git/20190209185930.5256-4-
[email protected]/Reported-by Randall S. Becker <[email protected]>Signed-off-by Max Kirillov <[email protected]>---
By the way, I don't think this requires such sofisticated fix. In the
success
Quoted 2 lines
case the input would not be read at all.
You could replace it with/dev/null, the in failure (not immediate fail)
git
Quoted 3 lines
would fail due to truncated input or something.
Also, as you experience hang issue [2] in earlier tests, this one should
not
Review
Quoted 8 lines
have contributed to it.
[2] https://public-
inbox.org/git/[email protected]/t/t5562-http-backend-content-length.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)(quoted diff) +0
-0
b/t/t5562-http-backend-
t/t5562-http-backend-content-length.sh test_expect_success 'CONTENT_LENGTH overflow ssite_t' ' · line 154
Quoted 12 lines
content-length.sh
index90d890d02f..436c261c86100755t/t5562-http-backend-content-length.sh +0
-0@@ -150,7 +150,7 @@ test_expect_success 'CONTENT_LENGTH overflow
ssite_t' 't/t5562-http-backend-content-length.sh +1
-1GIT_HTTP_EXPORT_ALL=TRUE \ REQUEST_METHOD=POST \ CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \ - git http-backend </dev/zero >/dev/null 2>err && + git http-backend </dev/null >/dev/null 2>err && grep "fatal:.*CONTENT_LENGTH" err
FTR, this particular subtest is not the one that is hanging. This subtest passes on NonStop with any and all (now) 4 solutions that have been floating around.
Cheers,
Randall
Re: [PATCH] t5562: do not depend on /dev/zero
Quoted 3 lines · Randall S. Becker
FTR, this particular subtest is not the one that is hanging. This subtest passes on NonStop with any and all (now) 4 solutions that have been floating around.
One thing I'd like to know more is if this test passes on NonStop with this patch, i.e. /dev/zero replaced with /dev/null.
Thanks.
RE: [PATCH] t5562: do not depend on /dev/zero
Quoted 5 lines · Junio C Hamano
To: Randall S. Becker <[email protected]>
Cc: 'Max Kirillov' <[email protected]>; [email protected]; 'Johannes Schindelin' <[email protected]>
Subject: Re: [PATCH] t5562: do not depend on/dev/zero
Quoted 7 lines
Quoted 3 lines · Randall S. Becker
FTR, this particular subtest is not the one that is hanging. This subtest passes on NonStop with any and all (now) 4 solutions that have been floating around.
One thing I'd like to know more is if this test passes on NonStop with
this
Quoted 1 line
patch, i.e.
/dev/zeroreplaced with/dev/null.
Yes, this particular subtest passes replacing /dev/null. The other three subtests still hang. This subtest never did.
Re: [PATCH] t5562: do not depend on /dev/zero
Quoted 12 lines · Randall S. Becker
Quoted 11 lines · Junio C Hamano
To: Randall S. Becker <[email protected]>
Cc: 'Max Kirillov' <[email protected]>; [email protected]; 'Johannes Schindelin' <[email protected]>
Subject: Re: [PATCH] t5562: do not depend on/dev/zeroQuoted 3 lines · Randall S. Becker
FTR, this particular subtest is not the one that is hanging. This subtest passes on NonStop with any and all (now) 4 solutions that have been floating around.
Quoted 6 lines
Quoted 1 line
One thing I'd like to know more is if this test passes on NonStop with
this
Quoted 1 line
patch, i.e.
/dev/zeroreplaced with/dev/null.Yes, this particular subtest passes replacing
/dev/null. The other three subtests still hang. This subtest never did.
Thanks.