Skip to contentSkip to conversation

test: Wait for output to flush out in enable_pid

PATCH In review [email protected] 3 participants 9 messages Jun 28, 2025 – Jul 31, 2025

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 test: Wait for output to flush out in enable_pid

From: Eugene Loh <[email protected]>

Our luck with this test has been quite good, but it sometimes fails to show its last lines of output. That is, we send a USR1 to the trigger processes to set off the final output and we immediately cat the output files. If there is any delay in handling the signal, the last output will be missing.

Have the processes terminate themselves when their last output is flushed; then wait for those processes. Also, skip testing altogether if there is only a single processor to run the two, hard-spinning processes.

---
 test/unittest/usdt/tst.enable_pid.sh | 17 ++++++++++++-----
 test/unittest/usdt/tst.enable_pid.x  |  8 ++++++++
 2 files changed, 20 insertions(+), 5 deletions(-)
 create mode 100755 test/unittest/usdt/tst.enable_pid.x
test/unittest/usdt/tst.enable_pid.sh +12 -5
index 7f4f68698..296cfb382 100755
@@ -33,6 +33,8 @@ EOF
 cat > main.c <<EOF
 #include <signal.h>
 #include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
 #include "prov.h"
 
 /* We check if the is-enabled probe is or is not enabled (or unknown). */
@@ -41,7 +43,7 @@ cat > main.c <<EOF
 #define ENABLED_UNK	3
 
 /* Start with the previous probe "unknown". */
-int prv = ENABLED_UNK;
+int prv = ENABLED_UNK, nepochs_left = 4;
 long long num = 0;
 
 /* Report how many times the previous case was encountered. */
@@ -71,6 +73,9 @@ static void mark_epoch(int sig) {
 	report();
 	printf("=== epoch ===\n");
 	fflush(stdout);
+	nepochs_left--;
+	if (nepochs_left <= 0)
+		exit(0);
 }
 
 int
@@ -79,7 +84,7 @@ main(int argc, char **argv)
 	struct sigaction act;
 
 	/* Set USR1 to mark epochs. */
-	act.sa_flags = 0;
+	memset(&act, 0, sizeof(act));
 	act.sa_handler = mark_epoch;
 	if (sigaction(SIGUSR1, &act, NULL)) {
 		printf("set handler failed\n");
@@ -172,13 +177,15 @@ for pid in 1 $pid1 $pid2 '*'; do
 	kill -USR1 $pid2
 done
 
+# Wait for the processes.
+wait $pid1
+wait $pid2
+
+# Dump the output.
 echo done
 echo "========== out 1"; cat out.1
 echo "========== out 2"; cat out.2
 
 echo success
 
-kill -TERM $pid1
-kill -TERM $pid2
-
 exit 0
test/unittest/usdt/tst.enable_pid.x new file +8 -0
new file mode 100755
index 000000000..9506674ee
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+if [ `grep -c ^processor /proc/cpuinfo` -lt 2 ]; then
+	echo test should have at least two processors
+	exit 2
+fi
+
+exit 0
Signature · 1 line
2.43.5
Reviewed-by on v2 1/1
#1 lore raw
Quoted 11 lines · eugene loh

From: Eugene Loh <[email protected]>

Our luck with this test has been quite good, but it sometimes fails to show its last lines of output. That is, we send a USR1 to the trigger processes to set off the final output and we immediately cat the output files. If there is any delay in handling the signal, the last output will be missing.

Have the processes terminate themselves when their last output is flushed; then wait for those processes. Also, skip testing altogether if there is only a single processor to run the two, hard-spinning processes.

Yeah, much better design.

Quoted 1 line
Signature · 1 line
NULL && (void)
#2 lore raw
Quoted 14 lines · Nick Alcock via DTrace-devel
Quoted 11 lines · eugene loh

From: Eugene Loh <[email protected]>

Our luck with this test has been quite good, but it sometimes fails to show its last lines of output. That is, we send a USR1 to the trigger processes to set off the final output and we immediately cat the output files. If there is any delay in handling the signal, the last output will be missing.

Have the processes terminate themselves when their last output is flushed; then wait for those processes. Also, skip testing altogether if there is only a single processor to run the two, hard-spinning processes.

Quoted 5 lines

Yeah, much better design.

Quoted 1 line

Unfortunately, without this patch the test PASSes for me. With this patch it FAILs with a time out.

Not sure what is going on, but this needs investigation before the patch can go in.

#3 lore raw
Quoted 19 lines · Kris Van Hees
Quoted 18 lines · Nick Alcock via DTrace-devel
Quoted 11 lines · eugene loh

From: Eugene Loh <[email protected]>

Our luck with this test has been quite good, but it sometimes fails to show its last lines of output. That is, we send a USR1 to the trigger processes to set off the final output and we immediately cat the output files. If there is any delay in handling the signal, the last output will be missing.

Have the processes terminate themselves when their last output is flushed; then wait for those processes. Also, skip testing altogether if there is only a single processor to run the two, hard-spinning processes.

Yeah, much better design.

Quoted 1 line
Quoted 7 lines
Quoted 1 line

Unfortunately, without this patch the test PASSes for me. With this patch it FAILs with a time out.

Not sure what is going on, but this needs investigation before the patch can go in.

Actually, I am a bit off here... it is still intermittent for me on Debian and OL9. So I am not sure what else is going wrong.

#4 lore raw
In reply to [email protected]

I still see intermittent FAILs. I wonder whether there is a potentially unsafe contruct at work (see below).

Quoted 42 lines · On Sat, Jun 28, 2025 at 06:30:50PM -0400, [email protected] wrote:

From: Eugene Loh <[email protected]>

Our luck with this test has been quite good, but it sometimes fails to show its last lines of output. That is, we send a USR1 to the trigger processes to set off the final output and we immediately cat the output files. If there is any delay in handling the signal, the last output will be missing.

Have the processes terminate themselves when their last output is flushed; then wait for those processes. Also, skip testing altogether if there is only a single processor to run the two, hard-spinning processes.

---
 test/unittest/usdt/tst.enable_pid.sh | 17 ++++++++++++-----
 test/unittest/usdt/tst.enable_pid.x  |  8 ++++++++
 2 files changed, 20 insertions(+), 5 deletions(-)
 create mode 100755 test/unittest/usdt/tst.enable_pid.x
test/unittest/usdt/tst.enable_pid.sh +3 -1
index 7f4f68698..296cfb382 100755
@@ -33,6 +33,8 @@ EOF
 cat > main.c <<EOF
 #include <signal.h>
 #include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
 #include "prov.h"
 
 /* We check if the is-enabled probe is or is not enabled (or unknown). */
@@ -41,7 +43,7 @@ cat > main.c <<EOF
 #define ENABLED_UNK	3
 
 /* Start with the previous probe "unknown". */
-int prv = ENABLED_UNK;
+int prv = ENABLED_UNK, nepochs_left = 4;
 long long num = 0;
 
 /* Report how many times the previous case was encountered. */

test/unittest/usdt/tst.enable_pid.sh static void mark_epoch(int sig) { · line 78

Quoted 6 lines
test/unittest/usdt/tst.enable_pid.sh +3 -0
@@ -71,6 +73,9 @@ static void mark_epoch(int sig) {
 	report();
 	printf("=== epoch ===\n");
 	fflush(stdout);
+	nepochs_left--;
+	if (nepochs_left <= 0)
+		exit(0);

I wonder whether this could still cause some output to get lost. I would actually move the conditional and exit to the end of the loop in main(), so that we check nepochs_left and exit when <= 0 outside of the interrupt handler.

I am running a long sequence of this test to see if that helps/fixes it.

Quoted 48 lines
test/unittest/usdt/tst.enable_pid.sh +6 -4
 }
 
 int
@@ -79,7 +84,7 @@ main(int argc, char **argv)
 	struct sigaction act;
 
 	/* Set USR1 to mark epochs. */
-	act.sa_flags = 0;
+	memset(&act, 0, sizeof(act));
 	act.sa_handler = mark_epoch;
 	if (sigaction(SIGUSR1, &act, NULL)) {
 		printf("set handler failed\n");
@@ -172,13 +177,15 @@ for pid in 1 $pid1 $pid2 '*'; do
 	kill -USR1 $pid2
 done
 
+# Wait for the processes.
+wait $pid1
+wait $pid2
+
+# Dump the output.
 echo done
 echo "========== out 1"; cat out.1
 echo "========== out 2"; cat out.2
 
 echo success
 
-kill -TERM $pid1
-kill -TERM $pid2
-
 exit 0
test/unittest/usdt/tst.enable_pid.x new file +8 -0
new file mode 100755
index 000000000..9506674ee
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+if [ `grep -c ^processor /proc/cpuinfo` -lt 2 ]; then
+	echo test should have at least two processors
+	exit 2
+fi
+
+exit 0
Signature · 1 line
2.43.5
#5 lore raw
Quoted 2 lines · Kris Van Hees via DTrace-devel

I still see intermittent FAILs. I wonder whether there is a potentially unsafe contruct at work (see below).

Hm, no that does not fix it either.

Eugene - can you have a closer look at this and see what might be going on?

Quoted 110 lines · On Sat, Jun 28, 2025 at 06:30:50PM -0400, [email protected] wrote:
Quoted 42 lines

From: Eugene Loh <[email protected]>

Our luck with this test has been quite good, but it sometimes fails to show its last lines of output. That is, we send a USR1 to the trigger processes to set off the final output and we immediately cat the output files. If there is any delay in handling the signal, the last output will be missing.

Have the processes terminate themselves when their last output is flushed; then wait for those processes. Also, skip testing altogether if there is only a single processor to run the two, hard-spinning processes.

---
 test/unittest/usdt/tst.enable_pid.sh | 17 ++++++++++++-----
 test/unittest/usdt/tst.enable_pid.x  |  8 ++++++++
 2 files changed, 20 insertions(+), 5 deletions(-)
 create mode 100755 test/unittest/usdt/tst.enable_pid.x
test/unittest/usdt/tst.enable_pid.sh +3 -1
index 7f4f68698..296cfb382 100755
@@ -33,6 +33,8 @@ EOF
 cat > main.c <<EOF
 #include <signal.h>
 #include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
 #include "prov.h"
 
 /* We check if the is-enabled probe is or is not enabled (or unknown). */
@@ -41,7 +43,7 @@ cat > main.c <<EOF
 #define ENABLED_UNK	3
 
 /* Start with the previous probe "unknown". */
-int prv = ENABLED_UNK;
+int prv = ENABLED_UNK, nepochs_left = 4;
 long long num = 0;
 
 /* Report how many times the previous case was encountered. */

test/unittest/usdt/tst.enable_pid.sh static void mark_epoch(int sig) { · line 78

Quoted 6 lines
test/unittest/usdt/tst.enable_pid.sh +3 -0
@@ -71,6 +73,9 @@ static void mark_epoch(int sig) {
 	report();
 	printf("=== epoch ===\n");
 	fflush(stdout);
+	nepochs_left--;
+	if (nepochs_left <= 0)
+		exit(0);

I wonder whether this could still cause some output to get lost. I would actually move the conditional and exit to the end of the loop in main(), so that we check nepochs_left and exit when <= 0 outside of the interrupt handler.

I am running a long sequence of this test to see if that helps/fixes it.

Quoted 48 lines
test/unittest/usdt/tst.enable_pid.sh +6 -4
 }
 
 int
@@ -79,7 +84,7 @@ main(int argc, char **argv)
 	struct sigaction act;
 
 	/* Set USR1 to mark epochs. */
-	act.sa_flags = 0;
+	memset(&act, 0, sizeof(act));
 	act.sa_handler = mark_epoch;
 	if (sigaction(SIGUSR1, &act, NULL)) {
 		printf("set handler failed\n");
@@ -172,13 +177,15 @@ for pid in 1 $pid1 $pid2 '*'; do
 	kill -USR1 $pid2
 done
 
+# Wait for the processes.
+wait $pid1
+wait $pid2
+
+# Dump the output.
 echo done
 echo "========== out 1"; cat out.1
 echo "========== out 2"; cat out.2
 
 echo success
 
-kill -TERM $pid1
-kill -TERM $pid2
-
 exit 0
test/unittest/usdt/tst.enable_pid.x new file +8 -0
new file mode 100755
index 000000000..9506674ee
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+if [ `grep -c ^processor /proc/cpuinfo` -lt 2 ]; then
+	echo test should have at least two processors
+	exit 2
+fi
+
+exit 0
Signature · 1 line
2.43.5

_______________________________________________
DTrace-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/dtrace-devel

author
#6 lore raw
Quoted 6 lines · Kris Van Hees
Quoted 2 lines · Kris Van Hees via DTrace-devel

I still see intermittent FAILs. I wonder whether there is a potentially unsafe contruct at work (see below).

Hm, no that does not fix it either.

Eugene - can you have a closer look at this and see what might be going on?

I ran 40x on each of my 11 test VMs and am unable to get a FAIL with this patch.  I took a look at the test some more, but without a failure it's hard to fix.

Are you saying there is missing output (the motivating problem) or that there is a timeout (I thought you may have said this, but I cannot find where)?  Those would seem to be two separate problems, the latter being that a trigger does not see 4 signals.

Quoted 108 lines
Quoted 108 lines · On Sat, Jun 28, 2025 at 06:30:50PM -0400, [email protected] wrote:
Quoted 42 lines

From: Eugene Loh <[email protected]>

Our luck with this test has been quite good, but it sometimes fails to show its last lines of output. That is, we send a USR1 to the trigger processes to set off the final output and we immediately cat the output files. If there is any delay in handling the signal, the last output will be missing.

Have the processes terminate themselves when their last output is flushed; then wait for those processes. Also, skip testing altogether if there is only a single processor to run the two, hard-spinning processes.

---
  test/unittest/usdt/tst.enable_pid.sh | 17 ++++++++++++-----
  test/unittest/usdt/tst.enable_pid.x  |  8 ++++++++
  2 files changed, 20 insertions(+), 5 deletions(-)

create mode 100755 test/unittest/usdt/tst.enable_pid.x

test/unittest/usdt/tst.enable_pid.sh +3 -1
index 7f4f68698..296cfb382 100755
@@ -33,6 +33,8 @@ EOF
  cat > main.c <<EOF
  #include <signal.h>
  #include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
  #include "prov.h"
 
  /* We check if the is-enabled probe is or is not enabled (or unknown). */
@@ -41,7 +43,7 @@ cat > main.c <<EOF
  #define ENABLED_UNK	3
 
  /* Start with the previous probe "unknown". */
-int prv = ENABLED_UNK;
+int prv = ENABLED_UNK, nepochs_left = 4;
  long long num = 0;
 
  /* Report how many times the previous case was encountered. */

test/unittest/usdt/tst.enable_pid.sh static void mark_epoch(int sig) { · line 78

Quoted 6 lines
test/unittest/usdt/tst.enable_pid.sh +3 -0
@@ -71,6 +73,9 @@ static void mark_epoch(int sig) {
  	report();
  	printf("=== epoch ===\n");
  	fflush(stdout);
+	nepochs_left--;
+	if (nepochs_left <= 0)
+		exit(0);

I wonder whether this could still cause some output to get lost. I would actually move the conditional and exit to the end of the loop in main(), so that we check nepochs_left and exit when <= 0 outside of the interrupt handler.

I am running a long sequence of this test to see if that helps/fixes it.

Quoted 48 lines
test/unittest/usdt/tst.enable_pid.sh +6 -4
  }
 
  int
@@ -79,7 +84,7 @@ main(int argc, char **argv)
  	struct sigaction act;
 
  	/* Set USR1 to mark epochs. */
-	act.sa_flags = 0;
+	memset(&act, 0, sizeof(act));
  	act.sa_handler = mark_epoch;
  	if (sigaction(SIGUSR1, &act, NULL)) {
  		printf("set handler failed\n");
@@ -172,13 +177,15 @@ for pid in 1 $pid1 $pid2 '*'; do
  	kill -USR1 $pid2
  done
 
+# Wait for the processes.
+wait $pid1
+wait $pid2
+
+# Dump the output.
  echo done
  echo "========== out 1"; cat out.1
  echo "========== out 2"; cat out.2
 
  echo success
 
-kill -TERM $pid1
-kill -TERM $pid2
-
  exit 0
test/unittest/usdt/tst.enable_pid.x new file +8 -0
new file mode 100755
index 000000000..9506674ee
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+if [ `grep -c ^processor /proc/cpuinfo` -lt 2 ]; then
+	echo test should have at least two processors
+	exit 2
+fi
+
+exit 0
Signature · 1 line
2.43.5

_______________________________________________
DTrace-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/dtrace-devel

#7 lore raw
Quoted 9 lines · Eugene Loh
Quoted 6 lines · Kris Van Hees
Quoted 2 lines · Kris Van Hees via DTrace-devel

I still see intermittent FAILs. I wonder whether there is a potentially unsafe contruct at work (see below).

Hm, no that does not fix it either.

Eugene - can you have a closer look at this and see what might be going on?

Quoted 8 lines

I ran 40x on each of my 11 test VMs and am unable to get a FAIL with this patch.  I took a look at the test some more, but without a failure it's hard to fix.

Are you saying there is missing output (the motivating problem) or that there is a timeout (I thought you may have said this, but I cannot find where)?  Those would seem to be two separate problems, the latter being that a trigger does not see 4 signals.

Here is the issue:

Diff against expected:

tmp/runtest.18085/test.out 2025-07-30 10:23:48.697293799 -0400 renamed from unittest/usdt/tst.enable_pid.r 2025-06-08 00:17:54.789469056 -0400 +0 -2
@@ -22,8 +22,6 @@
 === epoch ===
 is not enabled
 is enabled
-is not enabled
-=== epoch ===
 ========== out 2
 is not enabled
 === epoch ===
#8 lore raw
Quoted 25 lines · Kris Van Hees
Quoted 9 lines · Eugene Loh
Quoted 6 lines · Kris Van Hees
Quoted 2 lines · Kris Van Hees via DTrace-devel

I still see intermittent FAILs. I wonder whether there is a potentially unsafe contruct at work (see below).

Hm, no that does not fix it either.

Eugene - can you have a closer look at this and see what might be going on?

Quoted 8 lines

I ran 40x on each of my 11 test VMs and am unable to get a FAIL with this patch.  I took a look at the test some more, but without a failure it's hard to fix.

Are you saying there is missing output (the motivating problem) or that there is a timeout (I thought you may have said this, but I cannot find where)?  Those would seem to be two separate problems, the latter being that a trigger does not see 4 signals.

Here is the issue:

Diff against expected:

tmp/runtest.18085/test.out 2025-07-30 10:23:48.697293799 -0400 line 27

Quoted 8 lines
tmp/runtest.18085/test.out 2025-07-30 10:23:48.697293799 -0400 renamed from unittest/usdt/tst.enable_pid.r 2025-06-08 00:17:54.789469056 -0400 +0 -2
@@ -22,8 +22,6 @@
 === epoch ===
 is not enabled
 is enabled
-is not enabled
-=== epoch ===
 ========== out 2
 is not enabled
 === epoch ===

I ran it 100 times, and got 26 failures.

OL9 - 5.15.0-304.171.4.1.el9uek.x86_64