Skip to contentSkip to conversation

Run ptests via stdbuf configured to line-buffering

ptest-runner Alexander Kanavin 4 participants 8 messages Apr 4, 2019 – Apr 5, 2019

lore
Share & tools
lore
#0 lore raw

As ptest-runner communicates with child processes via pipe2(), the corresponding channels are not attached to a pty. In that situation stdio facilities like printf() or fwrite() are fully buffered. If a ptest would use them, without bothering to fflush() the output, ptest-runner will only receive what was written by the child ptest process after a buffer gets filled. If the unit tests are proceeding slowly, this may mean that ptest-runner will erroneously timeout due to an apparent lack of 'signs of life' from the child process.

stdbuf utility from coreutils adjusts the buffering to a line-buffered one, and so ptest-runner will get the lines as soon as they are written.

---
 utils.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
utils.c +2 -5
index 504df0b..1376e39 100644
@@ -243,16 +243,13 @@ filter_ptests(struct ptest_list *head, char **ptests, int ptest_num)
 static inline void
 run_child(char *run_ptest, int fd_stdout, int fd_stderr)
 {
-	char **argv = malloc(sizeof(char) * 2);
+	char* argv[] = {"stdbuf", "-oL", "-eL", "./run-ptest", NULL};
 	chdir(dirname(strdup(run_ptest)));
 
-	argv[0] = run_ptest;
-	argv[1] = NULL;
-
 	dup2(fd_stdout, STDOUT_FILENO);
 	// XXX: Redirect stderr to stdout to avoid buffer ordering problems.
 	dup2(fd_stdout, STDERR_FILENO);
-	execv(run_ptest, argv);
+	execvp(argv[0], argv);
 
 	exit(1);
 }
Signature · 1 line
2.17.1
#1 lore raw
Quoted 25 lines · Alexander Kanavin

As ptest-runner communicates with child processes via pipe2(), the corresponding channels are not attached to a pty. In that situation stdio facilities like printf() or fwrite() are fully buffered. If a ptest would use them, without bothering to fflush() the output, ptest-runner will only receive what was written by the child ptest process after a buffer gets filled. If the unit tests are proceeding slowly, this may mean that ptest-runner will erroneously timeout due to an apparent lack of 'signs of life' from the child process.

stdbuf utility from coreutils adjusts the buffering to a line- buffered
one, and so ptest-runner will get the lines as soon as they are written.

---
 utils.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
utils.c +0 -0
index 504df0b..1376e39 100644

utils.c filter_ptests(struct ptest_list *head, char · line 247

Quoted 6 lines
utils.c +1 -1
@@ -243,16 +243,13 @@ filter_ptests(struct ptest_list *head, char
 **ptests, int ptest_num)
 static inline void
 run_child(char *run_ptest, int fd_stdout, int fd_stderr)
 {
-	char **argv = malloc(sizeof(char) * 2);
+	char* argv[] = {"stdbuf", "-oL", "-eL", "./run-ptest", NULL};

Should be run_ptest the from the function argument, not the string literal "./run-ptest" ?

Quoted 17 lines
utils.c +0 -3
 	chdir(dirname(strdup(run_ptest)));
 
-	argv[0] = run_ptest;
-	argv[1] = NULL;
-
 	dup2(fd_stdout, STDOUT_FILENO);
 	// XXX: Redirect stderr to stdout to avoid buffer ordering

problems.

utils.c +1 -1
 	dup2(fd_stdout, STDERR_FILENO);
-	execv(run_ptest, argv);
+	execvp(argv[0], argv);
 
 	exit(1);
 }
Signature · 1 line
2.17.1
Signature · 1 line
Joshua Watt <[email protected]>
#2 lore raw
In reply to Alexander Kanavin
Quoted 15 lines · Alexander Kanavin

As ptest-runner communicates with child processes via pipe2(), the corresponding channels are not attached to a pty. In that situation stdio facilities like printf() or fwrite() are fully buffered. If a ptest would use them, without bothering to fflush() the output, ptest-runner will only receive what was written by the child ptest process after a buffer gets filled. If the unit tests are proceeding slowly, this may mean that ptest-runner will erroneously timeout due to an apparent lack of 'signs of life' from the child process.

stdbuf utility from coreutils adjusts the buffering to a line- buffered
one, and so ptest-runner will get the lines as soon as they are written.

Quoted 4 lines
---
 utils.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

I'm a little torn on this. I noticed some of the run-ptest scripts use "| sed -u" whilst the one you were seeing problems with uses "| sed" without -u.

We may want to consider strongly recommending -u. I'm testing a patch with some tweaks like that in it...

Cheers,

Richard

#3 lore raw
Quoted 21 lines · Richard Purdie
Quoted 15 lines · Alexander Kanavin

As ptest-runner communicates with child processes via pipe2(), the corresponding channels are not attached to a pty. In that situation stdio facilities like printf() or fwrite() are fully buffered. If a ptest would use them, without bothering to fflush() the output, ptest-runner will only receive what was written by the child ptest process after a buffer gets filled. If the unit tests are proceeding slowly, this may mean that ptest-runner will erroneously timeout due to an apparent lack of 'signs of life' from the child process.

stdbuf utility from coreutils adjusts the buffering to a line- buffered
one, and so ptest-runner will get the lines as soon as they are written.

Quoted 4 lines
---
 utils.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
Quoted 6 lines

I'm a little torn on this. I noticed some of the run-ptest scripts use "| sed -u" whilst the one you were seeing problems with uses "| sed" without -u.

We may want to consider strongly recommending -u. I'm testing a patch with some tweaks like that in it...

Please no. I'm running images without sed and using busybox sed instead, and that doesn't support -u. I'd rather be compatible with sed from busybox to keep changes to images minimal (e.g. install of additional packages) before executing ptests.

-Mikko

#4 lore raw
Quoted 33 lines · On Fri, 2019-04-05 at 06:16 +0000, [email protected] wrote:
Quoted 28 lines · Richard Purdie
Quoted 16 lines · Alexander Kanavin

As ptest-runner communicates with child processes via pipe2(), the corresponding channels are not attached to a pty. In that situation stdio facilities like printf() or fwrite() are fully buffered. If a ptest would use them, without bothering to fflush() the output, ptest-runner will only receive what was written by the child ptest process after a buffer gets filled.
If the unit tests are proceeding slowly, this may mean that ptest-runner will erroneously timeout due to an apparent lack of 'signs of life' from the child process.

stdbuf utility from coreutils adjusts the buffering to a line- buffered
one, and so ptest-runner will get the lines as soon as they are written.

Quoted 4 lines
---
 utils.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

I'm a little torn on this. I noticed some of the run-ptest scripts use
"| sed -u" whilst the one you were seeing problems with uses "| sed"
without -u.

Quoted 3 lines

We may want to consider strongly recommending -u. I'm testing a patch
with some tweaks like that in it...

Quoted 6 lines

Please no. I'm running images without sed and using busybox sed instead, and that
doesn't support -u. I'd rather be compatible with sed from busybox to keep changes
to images minimal (e.g. install of additional packages) before executing ptests.

The other alternative option being proposed is for ptest-runner to depend on coreutils which is worse?

I did test the -u option to sed in the openssh ptest runner and it did fix the problems we've been seeing.

I'm open to other alternatives but the -u option to sed is looking like the best one we have right now. These bugs are making our testing of ptests effectively useless and unpredictable so this is a serious issue...

Cheers,

Richard

#5 lore raw
Quoted 41 lines · On Fri, Apr 05, 2019 at 07:46:00AM +0100, [email protected]…
Quoted 33 lines · On Fri, 2019-04-05 at 06:16 +0000, [email protected] wrote:
Quoted 28 lines · Richard Purdie
Quoted 16 lines · Alexander Kanavin

As ptest-runner communicates with child processes via pipe2(), the corresponding channels are not attached to a pty. In that situation stdio facilities like printf() or fwrite() are fully buffered. If a ptest would use them, without bothering to fflush() the output, ptest-runner will only receive what was written by the child ptest process after a buffer gets filled.
If the unit tests are proceeding slowly, this may mean that ptest-runner will erroneously timeout due to an apparent lack of 'signs of life' from the child process.

stdbuf utility from coreutils adjusts the buffering to a line- buffered
one, and so ptest-runner will get the lines as soon as they are written.

Quoted 4 lines
---
 utils.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

I'm a little torn on this. I noticed some of the run-ptest scripts use
"| sed -u" whilst the one you were seeing problems with uses "| sed"
without -u.

Quoted 3 lines

We may want to consider strongly recommending -u. I'm testing a patch
with some tweaks like that in it...

Quoted 6 lines

Please no. I'm running images without sed and using busybox sed instead, and that
doesn't support -u. I'd rather be compatible with sed from busybox to keep changes
to images minimal (e.g. install of additional packages) before executing ptests.

Quoted 2 lines

The other alternative option being proposed is for ptest-runner to depend on coreutils which is worse?

GNU sed does not come from coreutils but from sed recipe.

Your call in the end. I just provided my point of view.

Quoted 7 lines

I did test the -u option to sed in the openssh ptest runner and it did fix the problems we've been seeing.

I'm open to other alternatives but the -u option to sed is looking like the best one we have right now. These bugs are making our testing of ptests effectively useless and unpredictable so this is a serious issue...

Understood. I hope you could also add 'set -eux' to all ptest shell scripts. Many of them seem to be missing shell script error handling and failures like providing -u to busybox sed may go unnoticed.

-Mikko

#6 lore raw
Quoted 50 lines · On Fri, 2019-04-05 at 06:54 +0000, [email protected] wrote:
Quoted 48 lines · On Fri, Apr 05, 2019 at 07:46:00AM +0100, [email protected]…
Quoted 46 lines · On Fri, 2019-04-05 at 06:16 +0000, [email protected] wrote:
Quoted 34 lines · Richard Purdie
Quoted 21 lines · Alexander Kanavin

As ptest-runner communicates with child processes via pipe2(),
the corresponding channels are not attached to a pty. In that situation stdio facilities like printf() or fwrite() are fully
buffered. If a ptest would use them, without bothering to fflush() the output, ptest-runner will only receive what was written by the child ptest process after a buffer gets filled.
If the unit tests are proceeding slowly, this may mean that ptest-runner will erroneously timeout due to an apparent lack of
'signs of life' from the child process.

stdbuf utility from coreutils adjusts the buffering to a line-
buffered
one, and so ptest-runner will get the lines as soon as they are
written.

Quoted 4 lines
---
 utils.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

I'm a little torn on this. I noticed some of the run-ptest scripts
use
"| sed -u" whilst the one you were seeing problems with uses "| sed"
without -u.

Quoted 3 lines

We may want to consider strongly recommending -u. I'm testing a patch
with some tweaks like that in it...

Please no. I'm running images without sed and using busybox sed instead, and that
doesn't support -u. I'd rather be compatible with sed from busybox to
keep changes
to images minimal (e.g. install of additional packages) before executing ptests.

Quoted 6 lines
Quoted 2 lines

The other alternative option being proposed is for ptest-runner to depend on coreutils which is worse?

GNU sed does not come from coreutils but from sed recipe.

Your call in the end. I just provided my point of view.

The original patch in this thread from Alex needs stdbuf which would add a coreutils depends to ptest-runner for everything as I understand it.

Quoted 11 lines
Quoted 10 lines

I did test the -u option to sed in the openssh ptest runner and it did
fix the problems we've been seeing.

I'm open to other alternatives but the -u option to sed is looking like
the best one we have right now. These bugs are making our testing of
ptests effectively useless and unpredictable so this is a serious issue...

Quoted 3 lines

Understood. I hope you could also add 'set -eux' to all ptest shell scripts. Many of them seem to be missing shell script error handling and failures like providing -u to busybox sed may go unnoticed.

Patches would be welcome to help clean up some of these scripts. I feel like I'm fighting a lonely battle trying to get any of this to work at times :(

The good news is the automated monitoring of the ptest results should catch problems like that as if a chunk of ptests "disappear", the new testing should quickly highlight that.

(We should improve the scripts, we just also have another way to spot problems)

Cheers,

Richard

#7 lore raw
Quoted 54 lines · On Fri, Apr 05, 2019 at 07:59:41AM +0100, [email protected]…
Quoted 53 lines · On Fri, 2019-04-05 at 06:54 +0000, [email protected] wrote:
Quoted 50 lines · On Fri, Apr 05, 2019 at 07:46:00AM +0100, [email protected]…
Quoted 39 lines · On Fri, 2019-04-05 at 06:16 +0000, [email protected] wrote:
Quoted 34 lines · Richard Purdie
Quoted 21 lines · Alexander Kanavin

As ptest-runner communicates with child processes via pipe2(),
the corresponding channels are not attached to a pty. In that situation stdio facilities like printf() or fwrite() are fully
buffered. If a ptest would use them, without bothering to fflush() the output, ptest-runner will only receive what was written by the child ptest process after a buffer gets filled.
If the unit tests are proceeding slowly, this may mean that ptest-runner will erroneously timeout due to an apparent lack of
'signs of life' from the child process.

stdbuf utility from coreutils adjusts the buffering to a line-
buffered
one, and so ptest-runner will get the lines as soon as they are
written.

Quoted 4 lines
---
 utils.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

I'm a little torn on this. I noticed some of the run-ptest scripts
use
"| sed -u" whilst the one you were seeing problems with uses "| sed"
without -u.

Quoted 3 lines

We may want to consider strongly recommending -u. I'm testing a patch
with some tweaks like that in it...

Quoted 7 lines

Please no. I'm running images without sed and using busybox sed instead, and that
doesn't support -u. I'd rather be compatible with sed from busybox to
keep changes
to images minimal (e.g. install of additional packages) before executing ptests.

The other alternative option being proposed is for ptest-runner to depend on coreutils which is worse?

Quoted 7 lines
Quoted 3 lines

GNU sed does not come from coreutils but from sed recipe.

Your call in the end. I just provided my point of view.

The original patch in this thread from Alex needs stdbuf which would add a coreutils depends to ptest-runner for everything as I understand it.

Ok, that too then. Then all hope is lost so lets include sed there as well :)

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

I did test the -u option to sed in the openssh ptest runner and it did
fix the problems we've been seeing.

I'm open to other alternatives but the -u option to sed is looking like
the best one we have right now. These bugs are making our testing of
ptests effectively useless and unpredictable so this is a serious issue...

Quoted 7 lines
Quoted 3 lines

Understood. I hope you could also add 'set -eux' to all ptest shell scripts. Many of them seem to be missing shell script error handling and failures like providing -u to busybox sed may go unnoticed.

Patches would be welcome to help clean up some of these scripts. I feel like I'm fighting a lonely battle trying to get any of this to work at times :(

We're ramping up with ptests and will provide fixes back to upstream.

Quoted 6 lines

The good news is the automated monitoring of the ptest results should catch problems like that as if a chunk of ptests "disappear", the new testing should quickly highlight that.

(We should improve the scripts, we just also have another way to spot problems)

All this work is very much appreciated here in downstream! Thanks you very much!

-Mikko