Skip to contentSkip to conversation

2.21.0-rc0 test fixes resulting from use of /dev/zero

PATCH In review [email protected] 7 participants 24 messages Feb 9, 2019 – Feb 15, 2019

lore
Share & tools
lore
#0 lore raw

cover 2.21.0-rc0 test fixes resulting from use of /dev/zero

From: "Randall S. Becker" <[email protected]>

This is a candidate packages of fixes to remove dependence on /dev/zero and replaces it with the generate_zero_bytes function in test-lib-functions.sh

Randall S. Becker (3):
test-lib-functions.sh: add generate_zero_bytes function
t5318: replace use of /dev/zero with generate_zero_bytes
t5562: replace /dev/zero with a pipe from generate_zero_bytes

 t/t5318-commit-graph.sh                |  2 +-
 t/t5562-http-backend-content-length.sh |  4 ++--
 t/test-lib-functions.sh                | 13 +++++++++++++
 3 files changed, 16 insertions(+), 3 deletions(-)
Signature · 1 line
2.12.3
#1 lore raw

PATCH 1/3 test-lib-functions.sh: add generate_zero_bytes function

From: "Randall S. Becker" <[email protected]>

t5318 and t5562 used /dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs.

---
 t/test-lib-functions.sh | 13 +++++++++++++
 1 file changed, 13 insertions(+)
t/test-lib-functions.sh +13 -0
index 92cf8f812..bbf68712c 100644
@@ -116,6 +116,19 @@ remove_cr () {
 	tr '\015' Q | sed -e 's/Q$//'
 }
 
+# Generate an output of $1 bytes of all zeroes (NULs, not ASCII zeroes).
+# If $1 is 'infinity', output forever or until the receiving pipe stops reading,
+# whichever comes first.
+generate_zero_bytes () {
+	perl -e 'if ($ARGV[0] == "infinity") {
+		while (-1) {
+			print "\0"
+		}
+	} else {
+		print "\0" x $ARGV[0]
+	}' "$@"
+}
+
 # In some bourne shell implementations, the "unset" builtin returns
 # nonzero status when a variable to be unset was not set in the first
 # place.
Signature · 1 line
2.12.3
#2 lore raw

Re: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function

t/test-lib-functions.sh remove_cr () { · line 118

Quoted 10 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:

t5318 and t5562 used /dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs.

---

t/test-lib-functions.sh +3 -0
@@ -116,6 +116,19 @@ remove_cr () {
+# Generate an output of $1 bytes of all zeroes (NULs, not ASCII zeroes).
+# If $1 is 'infinity', output forever or until the receiving pipe stops reading,
+# whichever comes first.

This is a somewhat unusual API. A (perhaps) more intuitive behavior would be for it to emit an infinite stream of NULs when given no argument, and a limited number of NULs when given an argument. Redefining the behavior like that also fixes the "problem" with the current implementation erroring-out if no argument is provided.

t/test-lib-functions.sh remove_cr () { · line 123

Quoted 2 lines
t/test-lib-functions.sh +2 -0
+generate_zero_bytes () {
+       perl -e 'if ($ARGV[0] == "infinity") {

s/perl/"$PERL_PATH"/

t/test-lib-functions.sh remove_cr () { · line 126

Quoted 3 lines
t/test-lib-functions.sh +3 -0
+               while (-1) {
+                       print "\0"
+               }

Or, more compactly:

    print "\0" while 1;
Quoted 4 lines
t/test-lib-functions.sh +4 -0
+       } else {
+               print "\0" x $ARGV[0]
+       }' "$@"
+}
#3 lore raw

RE: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function

Quoted 4 lines · Eric Sunshine
Quoted 3 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:

t5318 and t5562 used /dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs.

Quoted 13 lines

t/test-lib-functions.sh @@

Quoted 7 lines

---

t/test-lib-functions.sh @@ renamed from t/test-lib-functions.sh +3 -1
-116,6 +116,19 @@ remove_cr () {
+# Generate an output of $1 bytes of all zeroes (NULs, not ASCII zeroes).
+# If $1 is 'infinity', output forever or until the receiving pipe
+stops reading, # whichever comes first.

This is a somewhat unusual API. A (perhaps) more intuitive behavior would be for it to emit an infinite stream of NULs when given no argument, and a limited number of NULs when given an argument.
Redefining the behavior like that also fixes the "problem" with the current implementation erroring-out if no argument is provided.

At this point, I've supplied three different ways to solve this incompatibility for platforms not Linux and others have also provided fixes and discussed this at length. It is not a specific fix that matters to me, but that there is a fix at all. So thanks for all your comments and I will wait on direction on what the team wants me to do about it, if anything.

Regards,
Randall

#4 lore raw

Re: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function

In reply to Eric Sunshine
Quoted 4 lines · Eric Sunshine
Quoted 3 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:

t5318 and t5562 used /dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs.

Quoted 13 lines

t/test-lib-functions.sh remove_cr () { · line 118

Quoted 7 lines

---

t/test-lib-functions.sh +3 -0
@@ -116,6 +116,19 @@ remove_cr () {
+# Generate an output of $1 bytes of all zeroes (NULs, not ASCII zeroes).
+# If $1 is 'infinity', output forever or until the receiving pipe stops reading,
+# whichever comes first.

This is a somewhat unusual API. A (perhaps) more intuitive behavior would be for it to emit an infinite stream of NULs when given no argument, and a limited number of NULs when given an argument. Redefining the behavior like that also fixes the "problem" with the current implementation erroring-out if no argument is provided.

Yeah, I agree that'd be a little more idiomatic for our code base, but I'm fine with it either way.

Quoted 4 lines

t/test-lib-functions.sh remove_cr () { · line 123

Quoted 2 lines
t/test-lib-functions.sh +2 -0
+generate_zero_bytes () {
+       perl -e 'if ($ARGV[0] == "infinity") {

s/perl/"$PERL_PATH"/

This shouldn't be necessary. perl() is a function that uses $PERL_PATH (so you only need $PERL_PATH when you're writing out another script that doesn't run in the same process space as the rest of the test code).

-Peff

#5 lore raw

Re: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function

Quoted 10 lines · Jeff King
Quoted 5 lines · Eric Sunshine

t/test-lib-functions.sh remove_cr () { · line 123

Quoted 2 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
t/test-lib-functions.sh +2 -0
+generate_zero_bytes () {
+       perl -e 'if ($ARGV[0] == "infinity") {

s/perl/"$PERL_PATH"/

This shouldn't be necessary. perl() is a function that uses $PERL_PATH (so you only need $PERL_PATH when you're writing out another script that doesn't run in the same process space as the rest of the test code).

Thanks for clarifying. I either didn't know or forgot about that.

#6 lore raw

RE: [Patch v1 1/3] test-lib-functions.sh: add generate_zero_bytes function

Quoted 6 lines · Eric Sunshine
Quoted 5 lines · Jeff King
Quoted 4 lines · Eric Sunshine
Quoted 2 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
t/test-lib-functions.sh +2 -0
+generate_zero_bytes () {
+       perl -e 'if ($ARGV[0] == "infinity") {
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

s/perl/"$PERL_PATH"/

This shouldn't be necessary. perl() is a function that uses $PERL_PATH (so you only need $PERL_PATH when you're writing out another script that doesn't run in the same process space as the rest of the test code).

Thanks for clarifying. I either didn't know or forgot about that.

As did I, thank you.

#7 lore raw

PATCH 2/3 t5318: replace use of /dev/zero with generate_zero_bytes

In reply to [email protected]

From: "Randall S. Becker" <[email protected]>

This change removes the dependency on /dev/zero with generate_zero_bytes appending NUL values to blocks generating wrong signatures for test cases.

---
 t/t5318-commit-graph.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5318-commit-graph.sh +1 -1
index 16d10ebce..65c1f45b0 100755
@@ -383,7 +383,7 @@ corrupt_graph_and_verify() {
 	cp $objdir/info/commit-graph commit-graph-backup &&
 	printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc &&
 	dd of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=0 &&
-	dd if=/dev/zero of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=$(($orig_size - $zero_pos)) &&
+	generate_zero_bytes $(($orig_size - $zero_pos)) >> "$objdir/info/commit-graph" &&
 	test_must_fail git commit-graph verify 2>test_err &&
 	grep -v "^+" test_err >err &&
 	test_i18ngrep "$grepstr" err
Signature · 1 line
2.12.3
#8 lore raw

Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes

Quoted 2 lines · On Sat, Feb 9, 2019 at 2:00 PM <[email protected]> wrote:

This change removes the dependency on /dev/zero with generate_zero_bytes appending NUL values to blocks generating wrong signatures for test cases.

This commit message says what the patch does but not _why_. At minimum, it should explain that /dev/zero is not available on all platforms, therefore, not portable, and (perhaps) cite NonStop as an example.

Quoted 6 lines

---

t/t5318-commit-graph.sh +1 -1
@@ -383,7 +383,7 @@ corrupt_graph_and_verify() {
-       dd if=/dev/zero of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=$(($orig_size - $zero_pos)) &&
+       generate_zero_bytes $(($orig_size - $zero_pos)) >> "$objdir/info/commit-graph" &&
#9 lore raw

Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes

Quoted 8 lines · Eric Sunshine
Quoted 2 lines · On Sat, Feb 9, 2019 at 2:00 PM <[email protected]> wrote:

This change removes the dependency on /dev/zero with generate_zero_bytes appending NUL values to blocks generating wrong signatures for test cases.

This commit message says what the patch does but not _why_. At minimum, it should explain that /dev/zero is not available on all platforms, therefore, not portable, and (perhaps) cite NonStop as an example.

Does sombody want to do the honors? [PATCH 1/3] would become wasted effort until that happens. On the other hand, if this is not urgent (it is only urgent for those without /dev/zero, and to others it may be distraction/disruption this close to the final release to add increased risk of fat finger mistakes), obviously I can wait.

Quoted 6 lines

t/t5318-commit-graph.sh corrupt_graph_and_verify() { · line 383

Quoted 6 lines

---

t/t5318-commit-graph.sh +1 -1
@@ -383,7 +383,7 @@ corrupt_graph_and_verify() {
-       dd if=/dev/zero of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=$(($orig_size - $zero_pos)) &&
+       generate_zero_bytes $(($orig_size - $zero_pos)) >> "$objdir/info/commit-graph" &&

The original says "skip to the $zero_pos location in an existing info/commit-graph file, and overwrite its existing contents up to the $orig_size location with NULs".

If I recall the implementation of generate_zero_bytes in [PATCH 1/3] correctly, it just generated stream of NULs of given number of bytes.

The only reason why this rewrite is not wrong is because the previous line (omitted in your quote) truncates the file to $zero_pos. That is a bit tricky ;-).

#10 lore raw

Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes

Quoted 11 lines · Junio C Hamano
Quoted 8 lines · Eric Sunshine
Quoted 2 lines · On Sat, Feb 9, 2019 at 2:00 PM <[email protected]> wrote:

This change removes the dependency on /dev/zero with generate_zero_bytes appending NUL values to blocks generating wrong signatures for test cases.

This commit message says what the patch does but not _why_. At minimum, it should explain that /dev/zero is not available on all platforms, therefore, not portable, and (perhaps) cite NonStop as an example.

Quoted 5 lines

Does sombody want to do the honors? [PATCH 1/3] would become wasted effort until that happens. On the other hand, if this is not urgent (it is only urgent for those without /dev/zero, and to others it may be distraction/disruption this close to the final release to add increased risk of fat finger mistakes), obviously I can wait.

So, before I lose the access to my primary screen (I was told that somehow I need to reimage the workstation today X-<), here is what I have now.

-- >8 --
From: "Randall S. Becker" <[email protected]>
Date: Sat, 9 Feb 2019 13:59:29 -0500
Subject: [PATCH] t5318: replace use of /dev/zero with generate_zero_bytes

There are platforms (e.g. NonStop) that lack /dev/zero; use the generate_zero_bytes helper we just introduced to append stream of NULs at the end of the file.

The original, even though it uses "dd seek=... count=..." to make it look like it is overwriting the middle part of an existing file, has truncated the file before this step with another use of "dd", which may make it tricky to see why this rewrite is a correct one.

---
 t/t5318-commit-graph.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5318-commit-graph.sh +1 -1
index 16d10ebce8..d4bd1522fe 100755
@@ -383,7 +383,7 @@ corrupt_graph_and_verify() {
 	cp $objdir/info/commit-graph commit-graph-backup &&
 	printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc &&
 	dd of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=0 &&
-	dd if=/dev/zero of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=$(($orig_size - $zero_pos)) &&
+	generate_zero_bytes $(($orig_size - $zero_pos)) >>"$objdir/info/commit-graph" &&
 	test_must_fail git commit-graph verify 2>test_err &&
 	grep -v "^+" test_err >err &&
 	test_i18ngrep "$grepstr" err
Signature · 1 line
2.21.0-rc0-36-ge9bd4aa026
#11 lore raw

RE: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes

Quoted 5 lines

-----Original Message-----
From: Junio C Hamano <[email protected]> On Behalf Of Junio C Hamano
Sent: February 13, 2019 12:25
To: Eric Sunshine <[email protected]>
Cc: [email protected]; Git List <[email protected]>; Randall

S.

Quoted 18 lines

Becker <[email protected]>
Subject: Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes

Quoted 5 lines · Junio C Hamano
Quoted 3 lines · Eric Sunshine
Quoted 2 lines · On Sat, Feb 9, 2019 at 2:00 PM <[email protected]> wrote:

This change removes the dependency on /dev/zero with generate_zero_bytes appending NUL values to blocks generating wrong

signatures for test cases.

Quoted 6 lines
Quoted 5 lines

This commit message says what the patch does but not _why_. At minimum, it should explain that /dev/zero is not available on all platforms, therefore, not portable, and (perhaps) cite NonStop as an example.

Quoted 7 lines
Quoted 5 lines

Does sombody want to do the honors? [PATCH 1/3] would become wasted effort until that happens. On the other hand, if this is not urgent (it is only urgent for those without /dev/zero, and to others it may be distraction/disruption this close to the final release to add increased risk of fat finger mistakes), obviously I can wait.

So, before I lose the access to my primary screen (I was told that somehow

I

Quoted 7 lines

need to reimage the workstation today X-<), here is what I have now.

-- >8 --
From: "Randall S. Becker" <[email protected]>
Date: Sat, 9 Feb 2019 13:59:29 -0500
Subject: [PATCH] t5318: replace use of /dev/zero with generate_zero_bytes

Quoted 5 lines

There are platforms (e.g. NonStop) that lack /dev/zero; use the generate_zero_bytes helper we just introduced to append stream of NULs at the end of the file.

The original, even though it uses "dd seek=... count=..." to make it look

like it

Quoted 1 line

is overwriting the middle part of an existing file, has truncated the file

before

Quoted 1 line

this step with another use of "dd", which may make it tricky to see why

this

Quoted 1 line

rewrite is a correct one.

Here is how I interpret the test - might be wrong, but yanno... The first dd copies something looking like reasonable data from the test case.
The second dd copies zeros from seek to the end of a fixed size block.

My first attempt at a fix used truncate that extended the first to the correct size (filling with zeros). My worry there is that I'm not sure there is a guarantee of zeros, but that shouldn't matter for the test which just wants a signature mismatch.

Others suggested using yes to fill in junk.

My second attempt was to create the generate_zero_bytes function to replace exactly what the second dd was doing but not user /dev/zero. The fix was not to change the conditions of the test - not debating the correctness of that here - but to simply replicate the use of /dev/zero in context. So the resulting file contains [reasonable-stuff]{seek}[0]{orig_size-seek}, which is sufficient to satisfy the conditions of the test.

Quoted 24 lines
---
 t/t5318-commit-graph.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5318-commit-graph.sh index renamed from t/t5318-commit-graph.sh +0 -0

16d10ebce8..d4bd1522fe 100755

t/t5318-commit-graph.sh +0 -0
@@ -383,7 +383,7 @@ corrupt_graph_and_verify() {
 	cp $objdir/info/commit-graph commit-graph-backup &&
 	printf "$data" | dd of="$objdir/info/commit-graph" bs=1

seek="$pos" conv=notrunc &&
dd of="$objdir/info/commit-graph" bs=1 seek="$zero_pos" count=0
&&

t/t5318-commit-graph.sh +0 -1
-	dd if=/dev/zero of="$objdir/info/commit-graph" bs=1

seek="$zero_pos" count=$(($orig_size - $zero_pos)) &&

t/t5318-commit-graph.sh +2 -0
+	generate_zero_bytes $(($orig_size - $zero_pos))
+>>"$objdir/info/commit-graph" &&
 	test_must_fail git commit-graph verify 2>test_err &&
 	grep -v "^+" test_err >err &&
 	test_i18ngrep "$grepstr" err
#12 lore raw

Re: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes

Quoted 2 lines · Randall S. Becker

My second attempt was to create the generate_zero_bytes function to replace exactly what the second dd was doing but not user /dev/zero.

Yes, and I think the patch does that ;-) It was just the original

dd if=/dev/zero of=... bs=1 seek=$there count=$this_many

would have been impossible to rewrite with the new generate_zero_bytes helper unless $there weren't seeking to the end of the file.

But the other dd before the one the patch rewrites truncates the file to make that seek=$there seeking to the end of the file, so simply appending output from genereate_zero_bytes is sufficient and correct conversion. I wanted to explain that for future readers who may wonder if the patch is doing the exact conversion.

#13 lore raw

RE: [Patch v1 2/3] t5318: replace use of /dev/zero with generate_zero_bytes

Quoted 5 lines · Junio C Hamano
Quoted 2 lines · Randall S. Becker

My second attempt was to create the generate_zero_bytes function to replace exactly what the second dd was doing but not user /dev/zero.

Quoted 8 lines

Yes, and I think the patch does that ;-) It was just the original

dd if=/dev/zero of=... bs=1 seek=$there count=$this_many

would have been impossible to rewrite with the new generate_zero_bytes helper unless $there weren't seeking to the end of the file.

But the other dd before the one the patch rewrites truncates the file to

make

Quoted 1 line

that seek=$there seeking to the end of the file, so simply appending

output

Quoted 1 line

from genereate_zero_bytes is sufficient and correct conversion. I wanted

to

Quoted 2 lines

explain that for future readers who may wonder if the patch is doing the exact conversion.

Sounds like we need a documentation patch in the actual test suite rather than in the commit <ducking>.

Cheers,
Randall

#14 lore raw

PATCH 3/3 t5562: replace /dev/zero with a pipe from generate_zero_bytes

In reply to [email protected]

From: "Randall S. Becker" <[email protected]>

This change removes the dependency on /dev/zero with an equivalent pipe of deliberately NUL bytes. This allows tests to proceed where /dev/zero does not exist.

---
 t/t5562-http-backend-content-length.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
t/t5562-http-backend-content-length.sh +2 -2
index 90d890d02..bbadde2c6 100755
@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' '
 
 test_expect_success 'CONTENT_LENGTH overflow ssite_t' '
 	NOT_FIT_IN_SSIZE=$(ssize_b100dots) &&
-	env \
+	generate_zero_bytes infinity  | env \
 		CONTENT_TYPE=application/x-git-upload-pack-request \
 		QUERY_STRING=/repo.git/git-upload-pack \
 		PATH_TRANSLATED="$PWD"/.git/git-upload-pack \
 		GIT_HTTP_EXPORT_ALL=TRUE \
 		REQUEST_METHOD=POST \
 		CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \
-		git http-backend </dev/zero >/dev/null 2>err &&
+		git http-backend >/dev/null 2>err &&
 	grep "fatal:.*CONTENT_LENGTH" err
 '
 
Signature · 1 line
2.12.3
#15 lore raw

Re: [Patch v1 3/3] t5562: replace /dev/zero with a pipe from generate_zero_bytes

Quoted 1 line · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:

This change removes the dependency on /dev/zero with an equivalent pipe of

Too many spaces between "equivalent" and "pipe".

Quoted 2 lines

deliberately NUL bytes. This allows tests to proceed where /dev/zero does not exist.

It wouldn't hurt to cite "NonStop" as an example of a platform lacking /dev/zero.

The first sentence is a bit off grammatically. Perhaps the entire commit message can be collapsed simply to:

Stop depending on /dev/zero which may not be available on all platforms (for instance, HP NonStop).

Quoted 18 lines

---

t/t5562-http-backend-content-length.sh +2 -2
@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' '
 test_expect_success 'CONTENT_LENGTH overflow ssite_t' '
        NOT_FIT_IN_SSIZE=$(ssize_b100dots) &&
-       env \
+       generate_zero_bytes infinity  | env \
                CONTENT_TYPE=application/x-git-upload-pack-request \
                QUERY_STRING=/repo.git/git-upload-pack \
                PATH_TRANSLATED="$PWD"/.git/git-upload-pack \
                GIT_HTTP_EXPORT_ALL=TRUE \
                REQUEST_METHOD=POST \
                CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \
-               git http-backend </dev/zero >/dev/null 2>err &&
+               git http-backend >/dev/null 2>err &&
        grep "fatal:.*CONTENT_LENGTH" err
 '
#16 lore raw

Re: [Patch v1 3/3] t5562: replace /dev/zero with a pipe from generate_zero_bytes

Eric Sunshine <[email protected]> writes:

[jc: Summoning Dscho, J6t for their Windows expertise at the end]

Quoted 17 lines · On Sat, Feb 9, 2019 at 1:59 PM <[email protected]> wrote:
Quoted 1 line

This change removes the dependency on /dev/zero with an equivalent pipe of

Too many spaces between "equivalent" and "pipe".

Quoted 2 lines

deliberately NUL bytes. This allows tests to proceed where /dev/zero does not exist.

It wouldn't hurt to cite "NonStop" as an example of a platform lacking /dev/zero.

The first sentence is a bit off grammatically. Perhaps the entire commit message can be collapsed simply to:

Stop depending on /dev/zero which may not be available on all platforms (for instance, HP NonStop).

Quoted 16 lines
Quoted 6 lines

---

t/t5562-http-backend-content-length.sh +0 -0
@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' '
 test_expect_success 'CONTENT_LENGTH overflow ssite_t' '
        NOT_FIT_IN_SSIZE=$(ssize_b100dots) &&

t/t5562-http-backend-content-length.sh test_expect_success GZIP 'push gzipped empty' ' · line 152

Quoted 10 lines
t/t5562-http-backend-content-length.sh +2 -2
-       env \
+       generate_zero_bytes infinity  | env \
                CONTENT_TYPE=application/x-git-upload-pack-request \
                QUERY_STRING=/repo.git/git-upload-pack \
                PATH_TRANSLATED="$PWD"/.git/git-upload-pack \
                GIT_HTTP_EXPORT_ALL=TRUE \
                REQUEST_METHOD=POST \
                CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \
-               git http-backend </dev/zero >/dev/null 2>err &&
+               git http-backend >/dev/null 2>err &&

Doesn't this "inifinity" mode have the same issue that was worked around by 6129c930 ("test-lib: limit the output of the yes utility", 2016-02-02) on Windows? If I read correctly, the process upstream of the pipe (in this case, perl producing a stream of infinite NULs) would not die when the downstream stops reading with SIGPIPE.

Or is it safe to assume that nobody expects to use http-backend on Windows based servers so this test is a non-issue?

Quoted 2 lines

t/t5562-http-backend-content-length.sh test_expect_success GZIP 'push gzipped empty' ' · line 155

Quoted 2 lines
t/t5562-http-backend-content-length.sh +0 -0
        grep "fatal:.*CONTENT_LENGTH" err
 '

Thanks.

#17 lore raw

Re: [Patch v1 3/3] t5562: replace /dev/zero with a pipe from generate_zero_bytes

Am 12.02.19 um 18:24 schrieb Junio C Hamano:

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
t/t5562-http-backend-content-length.sh +1 -1
@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' '
 test_expect_success 'CONTENT_LENGTH overflow ssite_t' '
        NOT_FIT_IN_SSIZE=$(ssize_b100dots) &&
-       env \
+       generate_zero_bytes infinity  | env \
                CONTENT_TYPE=application/x-git-upload-pack-request \
                QUERY_STRING=/repo.git/git-upload-pack \
                PATH_TRANSLATED="$PWD"/.git/git-upload-pack \
                GIT_HTTP_EXPORT_ALL=TRUE \
                REQUEST_METHOD=POST \
                CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \
Quoted 8 lines
Quoted 2 lines

t/t5562-http-backend-content-length.sh test_expect_success GZIP 'push gzipped empty' ' · line 153

Quoted 2 lines
t/t5562-http-backend-content-length.sh +1 -1
-               git http-backend </dev/zero >/dev/null 2>err &&
+               git http-backend >/dev/null 2>err &&

Doesn't this "inifinity" mode have the same issue that was worked around by 6129c930 ("test-lib: limit the output of the yes utility", 2016-02-02) on Windows? If I read correctly, the process upstream of the pipe (in this case, perl producing a stream of infinite NULs) would not die when the downstream stops reading with SIGPIPE.

I think we do not have to worry, and the reason is that the justification for 6129c930 is simply wrong.

As I did not find the patch series discussed here to pull and test, I repeated the timing tests with t7610-mergetool.sh with and without 6129c930 reverted, and the difference is only in the noise. The reason t7610 takes so long on Windows looks more like a consequence of the 10,000 processes that it spawns. It is a mystery to me how I came to the conclusion that the change in 6129c930 would make a difference. :-(

-- Hannes

#18 lore raw

Re: [Patch v1 3/3] t5562: replace /dev/zero with a pipe from generate_zero_bytes

Quoted 22 lines · Johannes Sixt

Am 12.02.19 um 18:24 schrieb Junio C Hamano:

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
t/t5562-http-backend-content-length.sh +1 -1
@@ -143,14 +143,14 @@ test_expect_success GZIP 'push gzipped empty' '
 test_expect_success 'CONTENT_LENGTH overflow ssite_t' '
        NOT_FIT_IN_SSIZE=$(ssize_b100dots) &&
-       env \
+       generate_zero_bytes infinity  | env \
                CONTENT_TYPE=application/x-git-upload-pack-request \
                QUERY_STRING=/repo.git/git-upload-pack \
                PATH_TRANSLATED="$PWD"/.git/git-upload-pack \
                GIT_HTTP_EXPORT_ALL=TRUE \
                REQUEST_METHOD=POST \
                CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \
Quoted 8 lines
Quoted 2 lines

t/t5562-http-backend-content-length.sh test_expect_success GZIP 'push gzipped empty' ' · line 153

Quoted 2 lines
t/t5562-http-backend-content-length.sh +1 -1
-               git http-backend </dev/zero >/dev/null 2>err &&
+               git http-backend >/dev/null 2>err &&

Doesn't this "inifinity" mode have the same issue that was worked around by 6129c930 ("test-lib: limit the output of the yes utility", 2016-02-02) on Windows? If I read correctly, the process upstream of the pipe (in this case, perl producing a stream of infinite NULs) would not die when the downstream stops reading with SIGPIPE.

Quoted 2 lines

I think we do not have to worry, and the reason is that the justification for 6129c930 is simply wrong.

That's kinda surprising but in a pleasant way---it's good that we have one less thing we need to worry about.

Thanks.

Quoted 9 lines

As I did not find the patch series discussed here to pull and test, I repeated the timing tests with t7610-mergetool.sh with and without 6129c930 reverted, and the difference is only in the noise. The reason t7610 takes so long on Windows looks more like a consequence of the 10,000 processes that it spawns. It is a mystery to me how I came to the conclusion that the change in 6129c930 would make a difference. :-(

-- Hannes

#19 lore raw

PATCH 1/1 t5562: do not depend on /dev/zero

In reply to [email protected]

It was reported [1] that NonStop platform does not have /dev/zero.

The test uses /dev/zero as a dummy input. Passing case (http-backed failed because of too big input size) should not be reading anything from it. If http-backend would erroneously try to read any data returning EOF probably would be even safer than providing some meaningless data.

Replace /dev/zero with /dev/null to avoid issues with platforms which do not have /dev/zero.

[1] https://public-inbox.org/git/[email protected]/

---
By the way, I don't think this requires such sofisticated fix. In the success case the input would not be read at all. You could replace it with /dev/null, the in failure (not immediate fail) git would fail due to truncated input or something.

Also, as you experience hang issue [2] in earlier tests, this one should not have contributed to it.

[2] https://public-inbox.org/git/[email protected]/

 t/t5562-http-backend-content-length.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
t/t5562-http-backend-content-length.sh +1 -1
index 90d890d02f..436c261c86 100755
@@ -150,7 +150,7 @@ test_expect_success 'CONTENT_LENGTH overflow ssite_t' '
 		GIT_HTTP_EXPORT_ALL=TRUE \
 		REQUEST_METHOD=POST \
 		CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \
-		git http-backend </dev/zero >/dev/null 2>err &&
+		git http-backend </dev/null >/dev/null 2>err &&
 	grep "fatal:.*CONTENT_LENGTH" err
 '
 
Signature · 1 line
2.19.0.1202.g68e1e8f04e
#20 lore raw

RE: [PATCH] t5562: do not depend on /dev/zero

Quoted 4 lines · Max Kirillov

It was reported [1] that NonStop platform does not have /dev/zero.

The test uses /dev/zero as a dummy input. Passing case (http-backed failed because of too big input size) should not be reading anything from it. If

http-

Quoted 4 lines

backend would erroneously try to read any data returning EOF probably would be even safer than providing some meaningless data.

Replace /dev/zero with /dev/null to avoid issues with platforms which do

not

Quoted 9 lines

have /dev/zero.

[1] https://public-inbox.org/git/20190209185930.5256-4-
[email protected]/

---
By the way, I don't think this requires such sofisticated fix. In the

success

Quoted 2 lines

case the input would not be read at all.
You could replace it with /dev/null, the in failure (not immediate fail)

git

Quoted 3 lines

would fail due to truncated input or something.

Also, as you experience hang issue [2] in earlier tests, this one should

not

Review

Quoted 8 lines

have contributed to it.

[2] https://public-
inbox.org/git/[email protected]/

 t/t5562-http-backend-content-length.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/t/t5562-http-backend-

t/t5562-http-backend-content-length.sh test_expect_success 'CONTENT_LENGTH overflow ssite_t' ' · line 154

Quoted 12 lines

content-length.sh
index 90d890d02f..436c261c86 100755

t/t5562-http-backend-content-length.sh +0 -0
@@ -150,7 +150,7 @@ test_expect_success 'CONTENT_LENGTH overflow

ssite_t' '

t/t5562-http-backend-content-length.sh +1 -1
 		GIT_HTTP_EXPORT_ALL=TRUE \
 		REQUEST_METHOD=POST \
 		CONTENT_LENGTH="$NOT_FIT_IN_SSIZE" \
-		git http-backend </dev/zero >/dev/null 2>err &&
+		git http-backend </dev/null >/dev/null 2>err &&
 	grep "fatal:.*CONTENT_LENGTH" err

FTR, this particular subtest is not the one that is hanging. This subtest passes on NonStop with any and all (now) 4 solutions that have been floating around.

Cheers,
Randall

#21 lore raw

Re: [PATCH] t5562: do not depend on /dev/zero

Quoted 3 lines · Randall S. Becker

FTR, this particular subtest is not the one that is hanging. This subtest passes on NonStop with any and all (now) 4 solutions that have been floating around.

One thing I'd like to know more is if this test passes on NonStop with this patch, i.e. /dev/zero replaced with /dev/null.

Thanks.

#22 lore raw

RE: [PATCH] t5562: do not depend on /dev/zero

Quoted 5 lines · Junio C Hamano

To: Randall S. Becker <[email protected]>
Cc: 'Max Kirillov' <[email protected]>; [email protected]; 'Johannes Schindelin' <[email protected]>
Subject: Re: [PATCH] t5562: do not depend on /dev/zero

Quoted 7 lines
Quoted 3 lines · Randall S. Becker

FTR, this particular subtest is not the one that is hanging. This subtest passes on NonStop with any and all (now) 4 solutions that have been floating around.

One thing I'd like to know more is if this test passes on NonStop with

this

Quoted 1 line

patch, i.e. /dev/zero replaced with /dev/null.

Yes, this particular subtest passes replacing /dev/null. The other three subtests still hang. This subtest never did.

#23 lore raw

Re: [PATCH] t5562: do not depend on /dev/zero

Quoted 12 lines · Randall S. Becker
Quoted 11 lines · Junio C Hamano

To: Randall S. Becker <[email protected]>
Cc: 'Max Kirillov' <[email protected]>; [email protected]; 'Johannes Schindelin' <[email protected]>
Subject: Re: [PATCH] t5562: do not depend on /dev/zero

Quoted 3 lines · Randall S. Becker

FTR, this particular subtest is not the one that is hanging. This subtest passes on NonStop with any and all (now) 4 solutions that have been floating around.

Quoted 6 lines
Quoted 1 line

One thing I'd like to know more is if this test passes on NonStop with

this

Quoted 1 line

patch, i.e. /dev/zero replaced with /dev/null.

Yes, this particular subtest passes replacing /dev/null. The other three subtests still hang. This subtest never did.

Thanks.