Skip to contentSkip to conversation

perf test: Add per-test timeout support

PATCH Superseded Ian Rogers 4 participants 40 messages Sep 28 – Nov 12 perf events

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

PATCH 1/2 perf test: Add per-test timeout support

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).
- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-test.txt |   7 +-
 tools/perf/tests/builtin-test.c        | 179 ++++++++++++++++++++++---
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 174 insertions(+), 19 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
index 859df74e62ef..16d940e8d221 100644
@@ -45,10 +45,15 @@ OPTIONS
 	Run each test the given number of times, by default once. This
 	option can be useful to determine if a test is flaky.
 
+-t::
+--timeout::
+	Timeout for each forked test in seconds, by default 60 seconds. Specify
+	0 to disable the timeout. Ignored when -F/--dont-fork is used.
+
 -F::
 --dont-fork::
 	Do not fork child for each test, run all tests within single process, this
-	sets sequential mode.
+	sets sequential mode and disables the per-test timeout.
 
 --dso::
 	Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +164 -15

374 lines not on this page · raw message

tools/perf/tests/tests.h +4 -3
index 9c96f33483d1..2de66dc2e701 100644
@@ -6,9 +6,10 @@
 #include "util/debug.h"
 
 enum {
-	TEST_OK   =  0,
-	TEST_FAIL = -1,
-	TEST_SKIP = -2,
+	TEST_OK      =  0,
+	TEST_FAIL    = -1,
+	TEST_SKIP    = -2,
+	TEST_TIMEOUT = -3,
 };
 
 #define TEST_ASSERT_VAL(text, cond)					 \
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
author reviewer
#1 lore raw

PATCH 2/2 perf test: Improve JUnit XML schema compliance and completeness

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-test.txt |   5 +
 tools/perf/tests/builtin-test.c        | 309 +++++++++++++++++--------
 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
index 16d940e8d221..9b1f153ce881 100644
@@ -55,6 +55,11 @@ OPTIONS
 	Do not fork child for each test, run all tests within single process, this
 	sets sequential mode and disables the per-test timeout.
 
+-j[<file>]::
+--junit[=<file>]::
+	Generate JUnit XML test report to the specified file, or 'test.xml' if
+	no file is specified.
+
 --dso::
 	Specify a DSO for the "Symbols" test.
 
tools/perf/tests/builtin-test.c +207 -102

370 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#2 lore raw

Re: [PATCH v1 2/2] perf test: Improve JUnit XML schema compliance and completeness

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#3 lore raw
In reply to Ian Rogers
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] perf test: overly broad waitpid() condition hangs test suite on normal failures - [Medium] perf…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] perf test: overly broad waitpid() condition hangs test suite on normal failures
- [Medium] perf test: child tests inherit SA_RESTART SIGALRM handler, breaking alarm()
- [Medium] perf test: SIGKILL escalation for timed-out process groups is aborted prematurely --

commit efae32fde876d5c28dcaff79950fdf6547005215
Author: Ian Rogers <[email protected]>

perf test: Add per-test timeout support

This commit adds a new per-test timeout to perf test, defaulting to 60 seconds. Tests are placed into separate process groups, and timeouts are enforced using an ITIMER_REAL periodic timer that sends SIGTERM followed by a SIGKILL escalation after 1 second.

Quoted 3 lines
tools/perf/tests/builtin-test.c +0 -0

[ ... ]

tools/perf/tests/builtin-test.c static int run_test_child(struct child_process *process) · line 399

Quoted 7 lines
tools/perf/tests/builtin-test.c +0 -1
@@ -378,20 +395,31 @@ static int run_test_child(struct child_process *process)
 
 	check_leaks();
 err_out:
-	fflush(NULL);
 	for (size_t i = 0; i < ARRAY_SIZE(signals); i++)
 		signal(signals[i], SIG_DFL);

[Severity: Medium]
Does this omit SIGALRM from being reset to SIG_DFL?

Since the parent __cmd_test() sets up a no-op handler with SA_RESTART for SIGALRM before forking, children will inherit that handler. Will builtin tests that rely on the default SIGALRM behavior (like using alarm() to prevent internal hangs) fail to terminate because the no-op handler overrides it?

tools/perf/tests/builtin-test.c static int run_test_child(struct child_process *process) · line 408

Quoted 9 lines
tools/perf/tests/builtin-test.c +9 -0
+	if (err == TEST_FAIL) {
+		/*
+		 * Wait for any spawned child processes (such as shell scripts
+		 * running trap cleanup handlers on SIGTERM) to finish before
+		 * exiting.
+		 */
+		while (waitpid(-1, NULL, 0) > 0 || errno == EINTR)
+			;
+	}

[Severity: High]
Can this loop hang the test suite indefinitely on normal failures?

This condition catches all normal test failures (err == TEST_FAIL), not just tests interrupted by a timeout SIGTERM. If a test spawns child processes and fails normally, it will block on waitpid(-1) because the child processes were never signaled to exit.

Will this hang the entire test run until the global timeout triggers, or deadlock completely if timeouts are disabled via the dont-fork path?

Quoted 3 lines
tools/perf/tests/builtin-test.c +1 -0
+	fflush(NULL);
 	return -err;
 }

[ ... ]

Quoted 5 lines
tools/perf/tests/builtin-test.c +1 -0
@@ -748,6 +792,47 @@ static void print_test_failure_snippet(FILE *fp, const char *buf)
 	free(line_lens);
 }
 
+static void cmd_test_alarm_handler(int sig __maybe_unused)

tools/perf/tests/builtin-test.c static void print_test_failure_snippet(FILE *fp, const char *buf) · line 805

Quoted 10 lines
tools/perf/tests/builtin-test.c +10 -0
+{
+}
+
+static void check_child_timeout(struct child_test *child, const struct timespec *now)
+{
+	double elapsed;
+	pid_t pid;
+
+	if (!test_timeout || !child || child->done || child->process.finished)
+		return;

[Severity: Medium]
Does returning early when child->done is true bypass the SIGKILL escalation?

If the immediate child process exits after receiving SIGTERM, the parent marks the entire test case as done (setting child->done to true in finish_tests_parallel).

Will checking child->done here completely bypass the 1-second timer and the kill(-pid, SIGKILL) call for the broader process group, leaving any subprocesses that ignored SIGTERM to leak?

Quoted 29 lines
tools/perf/tests/builtin-test.c +29 -0
+
+	pid = child->process.pid;
+	if (pid <= 0)
+		return;
+
+	if (!child->timed_out) {
+		elapsed = (now->tv_sec - child->start_time.tv_sec) +
+			  (now->tv_nsec - child->start_time.tv_nsec) / 1000000000.0;
+		if (elapsed >= test_timeout) {
+			child->timed_out = true;
+			child->timeout_time = *now;
+			pr_debug3("Timeout (%us) for test %d.%d (pid %d), sending SIGTERM\n",
+				  test_timeout, child->suite_num + 1,
+				  child->test_case_num + 1, pid);
+			kill(-pid, SIGTERM);
+			kill(pid, SIGTERM);
+		}
+	} else if (!child->sigkill_sent) {
+		elapsed = (now->tv_sec - child->timeout_time.tv_sec) +
+			  (now->tv_nsec - child->timeout_time.tv_nsec) / 1000000000.0;
+		if (elapsed >= 1.0) {
+			child->sigkill_sent = true;
+			pr_debug3("Sending SIGKILL to timed out test %d.%d (pid %d)\n",
+				  child->suite_num + 1, child->test_case_num + 1, pid);
+			kill(-pid, SIGKILL);
+			kill(pid, SIGKILL);
+		}
+	}
+}

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author reviewer
#4 lore raw

PATCH v2 1/2 perf test: Add per-test timeout support

In reply to Ian Rogers

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).
- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro

---
v2: Address Sashiko review of v1:
- Reset SIGALRM to SIG_DFL in the forked test child so tests relying on default alarm() behaviour aren't affected by the parent's no-op handler.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when the test was interrupted by SIGTERM/SIGINT, avoiding hangs on normal test failures.
- Ensure the SIGKILL escalation reaches the whole process group even when the direct child exits after SIGTERM, by killing the group before the zombie is reaped (so the PGID can't be reused).
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.

---
 tools/perf/Documentation/perf-test.txt |   7 +-
 tools/perf/tests/builtin-test.c        | 241 +++++++++++++++++++++++--
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 233 insertions(+), 22 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
index 859df74e62ef..16d940e8d221 100644
@@ -45,10 +45,15 @@ OPTIONS
 	Run each test the given number of times, by default once. This
 	option can be useful to determine if a test is flaky.
 
+-t::
+--timeout::
+	Timeout for each forked test in seconds, by default 60 seconds. Specify
+	0 to disable the timeout. Ignored when -F/--dont-fork is used.
+
 -F::
 --dont-fork::
 	Do not fork child for each test, run all tests within single process, this
-	sets sequential mode.
+	sets sequential mode and disables the per-test timeout.
 
 --dso::
 	Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +223 -18

443 lines not on this page · raw message

tools/perf/tests/tests.h +4 -3
index 9c96f33483d1..2de66dc2e701 100644
@@ -6,9 +6,10 @@
 #include "util/debug.h"
 
 enum {
-	TEST_OK   =  0,
-	TEST_FAIL = -1,
-	TEST_SKIP = -2,
+	TEST_OK      =  0,
+	TEST_FAIL    = -1,
+	TEST_SKIP    = -2,
+	TEST_TIMEOUT = -3,
 };
 
 #define TEST_ASSERT_VAL(text, cond)					 \
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
author reviewer
#5 lore raw

PATCH v2 2/2 perf test: Improve JUnit XML schema compliance and completeness

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-test.txt |   5 +
 tools/perf/tests/builtin-test.c        | 309 +++++++++++++++++--------
 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
index 16d940e8d221..9b1f153ce881 100644
@@ -55,6 +55,11 @@ OPTIONS
 	Do not fork child for each test, run all tests within single process, this
 	sets sequential mode and disables the per-test timeout.
 
+-j[<file>]::
+--junit[=<file>]::
+	Generate JUnit XML test report to the specified file, or 'test.xml' if
+	no file is specified.
+
 --dso::
 	Specify a DSO for the "Symbols" test.
 
tools/perf/tests/builtin-test.c +207 -102

370 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#6 lore raw

Re: [PATCH v2 2/2] perf test: Improve JUnit XML schema compliance and completeness

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#7 lore raw
In reply to Ian Rogers
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf test: TOCTOU race in signal handler can kill unintended process groups --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf test: TOCTOU race in signal handler can kill unintended process groups --

commit e2e28e1fd08275414fa889f24a3483b8a0075cc0
Author: Ian Rogers <[email protected]>

perf test: Add per-test timeout support

When a test hangs in perf test, the entire test run hangs and makes it difficult to inspect output from other tests. Add a new TEST_TIMEOUT result category and a per-test timeout.

Quoted 4 lines
tools/perf/tests/builtin-test.c +0 -0
index d2f594921e25b..4ecbaf757095e 100644

[ ... ]

tools/perf/tests/builtin-test.c static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes · line 1261

Quoted 9 lines
tools/perf/tests/builtin-test.c +3 -0
@@ -1096,7 +1254,10 @@ static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes
 					close(child->process.err);
 					child->process.err = -1;
 				}
+				kill_timed_out_group(child);
 				child->result = finish_command(&child->process);
+				if (child->timed_out)
+					child->result = TEST_TIMEOUT;
 				child->process.pid = 0;

[Severity: Medium]
Is there a TOCTOU race window here in finish_tests_parallel() before child->process.pid is zeroed?

If check_if_command_finished() falls back to waitpid(..., WNOHANG) (e.g. when /proc is unmounted) and reaps the child, finish_command() won't block.

If cmd_test_sig_handler is triggered via SIGINT or SIGTERM right after finish_command() reaps the child but before child->process.pid is zeroed, the asynchronous signal handler jumps back to __cmd_test() and executes a kill on the stale PID.

[ ... ]

tools/perf/tests/builtin-test.c static int __cmd_test(struct test_suite **suites, int argc, const char *argv[], · line 1514

Quoted 9 lines
tools/perf/tests/builtin-test.c +1 -0
@@ -1327,12 +1507,27 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
 			pr_debug3("Killing %d pid %d\n",
 				  child_test->suite_num + 1,
 				  child_test->process.pid);
+			kill(-child_test->process.pid, err);
 			kill(child_test->process.pid, err);
 		}
 		goto err_out;
 	}

[Severity: Medium]
Can this escalation from kill(pid) to kill(-pid) in __cmd_test() turn the single-process race into a full process group kill?

If the race window described above occurs, and a concurrent high-rate fork workload wraps the PID space exactly to the newly reaped PID, this kill(-pid) command could wipe out an entire unrelated process group rather than just a single reused PID.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author reviewer
#8 lore raw

PATCH v3 1/2 perf test: Add per-test timeout support

In reply to Ian Rogers

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).
- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro

---
v3: Address Sashiko review of v2:
- Block SIGINT/SIGTERM while a test process may be reaped (in check_if_command_finished() and finish_command()) and until its pid is cleared, and skip already reaped tests in the signal path, so a stale pid or process group ID can never be signaled.

v2: Address Sashiko review of v1:
- Reset SIGALRM to SIG_DFL in the forked test child so tests relying on default alarm() behaviour aren't affected by the parent's no-op handler.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when the test was interrupted by SIGTERM/SIGINT, avoiding hangs on normal test failures.
- Ensure the SIGKILL escalation reaches the whole process group even when the direct child exits after SIGTERM, by killing the group before the zombie is reaped (so the PGID can't be reused).
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.

---
 tools/perf/Documentation/perf-test.txt |   7 +-
 tools/perf/tests/builtin-test.c        | 297 ++++++++++++++++++++++---
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 280 insertions(+), 31 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
index 859df74e62ef..16d940e8d221 100644
@@ -45,10 +45,15 @@ OPTIONS
 	Run each test the given number of times, by default once. This
 	option can be useful to determine if a test is flaky.
 
+-t::
+--timeout::
+	Timeout for each forked test in seconds, by default 60 seconds. Specify
+	0 to disable the timeout. Ignored when -F/--dont-fork is used.
+
 -F::
 --dont-fork::
 	Do not fork child for each test, run all tests within single process, this
-	sets sequential mode.
+	sets sequential mode and disables the per-test timeout.
 
 --dso::
 	Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +270 -27

513 lines not on this page · raw message

tools/perf/tests/tests.h +4 -3
index 9c96f33483d1..2de66dc2e701 100644
@@ -6,9 +6,10 @@
 #include "util/debug.h"
 
 enum {
-	TEST_OK   =  0,
-	TEST_FAIL = -1,
-	TEST_SKIP = -2,
+	TEST_OK      =  0,
+	TEST_FAIL    = -1,
+	TEST_SKIP    = -2,
+	TEST_TIMEOUT = -3,
 };
 
 #define TEST_ASSERT_VAL(text, cond)					 \
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
author reviewer
#9 lore raw

PATCH v3 2/2 perf test: Improve JUnit XML schema compliance and completeness

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro

---
v3: No changes.

v2: No changes.

---
 tools/perf/Documentation/perf-test.txt |   5 +
 tools/perf/tests/builtin-test.c        | 309 +++++++++++++++++--------
 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
index 16d940e8d221..9b1f153ce881 100644
@@ -55,6 +55,11 @@ OPTIONS
 	Do not fork child for each test, run all tests within single process, this
 	sets sequential mode and disables the per-test timeout.
 
+-j[<file>]::
+--junit[=<file>]::
+	Generate JUnit XML test report to the specified file, or 'test.xml' if
+	no file is specified.
+
 --dso::
 	Specify a DSO for the "Symbols" test.
 
tools/perf/tests/builtin-test.c +207 -102

370 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#10 lore raw

Re: [PATCH v3 2/2] perf test: Improve JUnit XML schema compliance and completeness

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#11 lore raw
In reply to Ian Rogers
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 23 lines

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).
- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#12 lore raw
In reply to Ian Rogers
Quoted 7 lines · Ian Rogers

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

Quoted 8 lines

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).

Do we really need this timer? I think poll() already has a timeout so it'll wake up on every 100 msec anyway?

Thanks,
Namhyung

Quoted 10 lines

- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro
author reviewer
#13 lore raw
Quoted 18 lines · Namhyung Kim
Quoted 7 lines · Ian Rogers

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

Quoted 8 lines

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).

Quoted 2 lines

Do we really need this timer? I think poll() already has a timeout so it'll wake up on every 100 msec anyway?

Agreed and that avoids async/signal issues. I will fix this in v4.

Thanks,
Ian

Quoted 14 lines

Thanks,
Namhyung

Quoted 10 lines

- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro
author reviewer
#14 lore raw

v4 cover perf test: Per-test timeouts, JUnit XML compliance, and snippet improvements

In reply to Ian Rogers

This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and refines the verbose=1 failure snippet output when tests fail with a signal or time out:

1. Add a per-test timeout (default 60s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.

2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.

3. Improve verbose=1 failure snippet selection in print_test_failure_snippet() to skip blank and stack trace lines when selecting context and scanning backwards from the end of the log, and prioritize the test output line immediately preceding '---- unexpected signal' so trailing test progress is shown instead of outermost stack frames.

v4:
- Drop the ITIMER_REAL timer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops in finish_test() and finish_tests_parallel() to detect timeouts.
- Only fall back to kill(pid, sig) when kill(-pid, sig) fails, and ignore SIGTERM/SIGINT while waiting for subprocesses in run_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.

v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.

v2:
- Reset SIGALRM to SIG_DFL in the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.

Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Avoid stack trace noise in verbose=1 failure snippets

 tools/perf/Documentation/perf-test.txt |  12 +-
 tools/perf/tests/builtin-test.c        | 668 +++++++++++++++++++------
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 543 insertions(+), 144 deletions(-)
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
author reviewer
#15 lore raw

PATCH v4 1/3 perf test: Add per-test timeout support

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes against test_timeout in the existing finish_test() and finish_tests_parallel() polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-test.txt |   7 +-
 tools/perf/tests/builtin-test.c        | 278 ++++++++++++++++++++++---
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 260 insertions(+), 32 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
index 859df74e62ef..16d940e8d221 100644
@@ -45,10 +45,15 @@ OPTIONS
 	Run each test the given number of times, by default once. This
 	option can be useful to determine if a test is flaky.
 
+-t::
+--timeout::
+	Timeout for each forked test in seconds, by default 60 seconds. Specify
+	0 to disable the timeout. Ignored when -F/--dont-fork is used.
+
 -F::
 --dont-fork::
 	Do not fork child for each test, run all tests within single process, this
-	sets sequential mode.
+	sets sequential mode and disables the per-test timeout.
 
 --dso::
 	Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +250 -28

482 lines not on this page · raw message

tools/perf/tests/tests.h +4 -3
index f379620069d3..9162068b260d 100644
@@ -6,9 +6,10 @@
 #include "util/debug.h"
 
 enum {
-	TEST_OK   =  0,
-	TEST_FAIL = -1,
-	TEST_SKIP = -2,
+	TEST_OK      =  0,
+	TEST_FAIL    = -1,
+	TEST_SKIP    = -2,
+	TEST_TIMEOUT = -3,
 };
 
 #define TEST_ASSERT_VAL(text, cond)					 \
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
#16 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 21 lines

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes against test_timeout in the existing finish_test() and finish_tests_parallel() polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author reviewer
#17 lore raw

PATCH v4 2/3 perf test: Improve JUnit XML schema compliance and completeness

In reply to Ian Rogers

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-test.txt |   5 +
 tools/perf/tests/builtin-test.c        | 309 +++++++++++++++++--------
 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
index 16d940e8d221..9b1f153ce881 100644
@@ -55,6 +55,11 @@ OPTIONS
 	Do not fork child for each test, run all tests within single process, this
 	sets sequential mode and disables the per-test timeout.
 
+-j[<file>]::
+--junit[=<file>]::
+	Generate JUnit XML test report to the specified file, or 'test.xml' if
+	no file is specified.
+
 --dso::
 	Specify a DSO for the "Symbols" test.
 
tools/perf/tests/builtin-test.c +207 -102

370 lines not on this page · raw message

Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
#18 lore raw

Re: [PATCH v4 2/3] perf test: Improve JUnit XML schema compliance and completeness

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author reviewer
#19 lore raw

PATCH v4 3/3 perf test: Avoid stack trace noise in verbose=1 failure snippets

In reply to Ian Rogers

When a test fails with a signal or times out, child_test_sig_handler() appends '---- unexpected signal (<sig>) ----' followed by a multi-line
stack trace (' #0 ...' through '_start').

In verbose=1 mode, print_test_failure_snippet() selects up to failure_snippet_lines (10) lines:
- Pass 1 selects lines matching failure keywords plus their immediate
next line (' #0 ... in child_test_sig_handler').
- Pass 2 fills the remaining quota by scanning backwards from the end of the output buffer, which consumes the remaining lines on outermost stack frames ('handle_internal_command', 'run_argv', 'main', '__libc_start_main', '_start') and hides the trailing test output showing where the test had run to.

Improve print_test_failure_snippet() by:
- Skipping blank lines and stack trace lines (' #<digit>...') when
selecting context in Pass 1 and when scanning backwards from the end in Pass 2, with a Pass 3 fallback from the top of the buffer if quota still remains.
- Prioritizing the non-blank line immediately preceding '---- unexpected signal' so there is no '...' gap before the signal message.
- Only printing '...' between selected lines when at least one non-blank line was omitted between them.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/tests/builtin-test.c | 101 +++++++++++++++++++++++++-------
 1 file changed, 81 insertions(+), 20 deletions(-)
tools/perf/tests/builtin-test.c +81 -20
index d0c2783cb312..b95e9704b79f 100644
@@ -774,6 +774,47 @@ static void print_line_highlighted(FILE *fp, const char *line, size_t len)
 }
 
 
+static bool is_blank_line(const char *line, size_t len)
+{
+	for (size_t i = 0; i < len; i++) {
+		if (!isspace((unsigned char)line[i]))
+			return false;
+	}
+	return true;
+}
+
+static bool is_stack_line(const char *line, size_t len)
+{
+	size_t i = 0;
+
+	while (i < len && isspace((unsigned char)line[i]))
+		i++;
+	return i + 1 < len && line[i] == '#' && isdigit((unsigned char)line[i + 1]);
+}
+
+static bool is_ignorable_snippet_line(const char *line, size_t len)
+{
+	return is_blank_line(line, len) || is_stack_line(line, len);
+}
+
+static bool has_non_blank_between(const char **lines, const size_t *line_lens,
+				  size_t start, size_t end)
+{
+	for (size_t j = start + 1; j < end; j++) {
+		if (!is_blank_line(lines[j], line_lens[j]))
+			return true;
+	}
+	return false;
+}
+
+static void pick_snippet_line(bool *pick, size_t idx, unsigned int *picked_count)
+{
+	if (!pick[idx] && *picked_count < failure_snippet_lines) {
+		pick[idx] = true;
+		(*picked_count)++;
+	}
+}
+
 static void print_test_failure_snippet(FILE *fp, const char *buf)
 {
 	size_t num_lines = 0;
@@ -844,43 +885,63 @@ static void print_test_failure_snippet(FILE *fp, const char *buf)
 	}
 
 	/* Pass 0: Always pick the very first line */
-	if (num_lines > 0 && picked_count < failure_snippet_lines) {
-		pick[0] = true;
-		picked_count++;
-	}
+	if (num_lines > 0)
+		pick_snippet_line(pick, 0, &picked_count);
 
 	/* Pass 1: Pick lines with failure keywords from start (Highest Priority) */
 	for (i = 0; i < num_lines && picked_count < failure_snippet_lines; i++) {
 		size_t dummy;
 
-		if (find_next_keyword(lines[i], line_lens[i], &dummy)) {
-			if (!pick[i]) {
-				pick[i] = true;
-				picked_count++;
-			}
-			/* Prioritize getting the immediate next line for context */
-			if (i + 1 < num_lines && !pick[i + 1] &&
-			    picked_count < failure_snippet_lines) {
-				pick[i + 1] = true;
-				picked_count++;
+		if (is_ignorable_snippet_line(lines[i], line_lens[i]))
+			continue;
+		if (!find_next_keyword(lines[i], line_lens[i], &dummy))
+			continue;
+
+		pick_snippet_line(pick, i, &picked_count);
+		/*
+		 * For unexpected signals, prioritize the preceding non-blank
+		 * line showing where the test had run to before the signal.
+		 */
+		if (!strncmp(lines[i], "---- unexpected signal", 22)) {
+			size_t j = i;
+
+			while (j > 0) {
+				j--;
+				if (!is_ignorable_snippet_line(lines[j], line_lens[j])) {
+					pick_snippet_line(pick, j, &picked_count);
+					break;
+				}
 			}
 		}
+		/* Prioritize getting the immediate next line for context */
+		if (i + 1 < num_lines &&
+		    !is_ignorable_snippet_line(lines[i + 1], line_lens[i + 1]))
+			pick_snippet_line(pick, i + 1, &picked_count);
 	}
 
-	/* Pass 2: Fill remaining quota from the end backwards */
+	/*
+	 * Pass 2: Fill remaining quota from the end backwards, skipping blank
+	 * and stack trace lines so trailing test progress is shown instead of
+	 * outermost stack frames like main/_start.
+	 */
 	i = num_lines;
 	while (i > 0 && picked_count < failure_snippet_lines) {
 		i--;
-		if (!pick[i]) {
-			pick[i] = true;
-			picked_count++;
-		}
+		if (!is_ignorable_snippet_line(lines[i], line_lens[i]))
+			pick_snippet_line(pick, i, &picked_count);
+	}
+
+	/* Pass 3: Fallback to top stack frames if quota remains */
+	for (i = 0; i < num_lines && picked_count < failure_snippet_lines; i++) {
+		if (!is_blank_line(lines[i], line_lens[i]))
+			pick_snippet_line(pick, i, &picked_count);
 	}
 
 	for (i = 0; i < num_lines; i++) {
 		if (!pick[i])
 			continue;
-		if (last_printed != -1 && (int)i > last_printed + 1) {
+		if (last_printed != -1 &&
+		    has_non_blank_between(lines, line_lens, last_printed, i)) {
 			if (perf_use_color_default)
 				fprintf(fp, "%s...%s\n", PERF_COLOR_BLUE, PERF_COLOR_RESET);
 			else
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
#20 lore raw

Re: [PATCH v4 3/3] perf test: Avoid stack trace noise in verbose=1 failure snippets

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 25 lines

When a test fails with a signal or times out, child_test_sig_handler() appends '---- unexpected signal (<sig>) ----' followed by a multi-line
stack trace (' #0 ...' through '_start').

In verbose=1 mode, print_test_failure_snippet() selects up to failure_snippet_lines (10) lines:
- Pass 1 selects lines matching failure keywords plus their immediate
next line (' #0 ... in child_test_sig_handler').
- Pass 2 fills the remaining quota by scanning backwards from the end of the output buffer, which consumes the remaining lines on outermost stack frames ('handle_internal_command', 'run_argv', 'main', '__libc_start_main', '_start') and hides the trailing test output showing where the test had run to.

Improve print_test_failure_snippet() by:
- Skipping blank lines and stack trace lines (' #<digit>...') when
selecting context in Pass 1 and when scanning backwards from the end in Pass 2, with a Pass 3 fallback from the top of the buffer if quota still remains.
- Prioritizing the non-blank line immediately preceding '---- unexpected signal' so there is no '...' gap before the signal message.
- Only printing '...' between selected lines when at least one non-blank line was omitted between them.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
maintainer Acked-by on the v4 series
#21 lore raw

Re: [PATCH v4 0/3] perf test: Per-test timeouts, JUnit XML compliance, and snippet improvements

In reply to Ian Rogers
Quoted 49 lines · Ian Rogers

This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and refines the verbose=1 failure snippet output when tests fail with a signal or time out:

1. Add a per-test timeout (default 60s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.

2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.

3. Improve verbose=1 failure snippet selection in print_test_failure_snippet() to skip blank and stack trace lines when selecting context and scanning backwards from the end of the log, and prioritize the test output line immediately preceding '---- unexpected signal' so trailing test progress is shown instead of outermost stack frames.

v4:
- Drop the ITIMER_REAL timer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops in finish_test() and finish_tests_parallel() to detect timeouts.
- Only fall back to kill(pid, sig) when kill(-pid, sig) fails, and ignore SIGTERM/SIGINT while waiting for subprocesses in run_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.

v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.

v2:
- Reset SIGALRM to SIG_DFL in the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.

Quoted 4 lines

Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Avoid stack trace noise in verbose=1 failure snippets

Thanks,
Namhyung

Quoted 9 lines
 tools/perf/Documentation/perf-test.txt |  12 +-
 tools/perf/tests/builtin-test.c        | 668 +++++++++++++++++++------
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 543 insertions(+), 144 deletions(-)
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
maintainer Applied
#22 lore raw

Re: [PATCH v4 0/3] perf test: Per-test timeouts, JUnit XML compliance, and snippet improvements

Quoted 4 lines · Namhyung Kim
Quoted 3 lines · Ian Rogers

Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness

Both are failing to apply to perf-tools-next, I'm pushing what I have (with 3/3) to tmp.perf-tools-next now.

Quoted 1 line
Quoted 1 line

perf test: Avoid stack trace noise in verbose=1 failure snippets

Applied.

Quoted 1 line

Thanks,

- Arnaldo

author reviewer
#23 lore raw

Re: [PATCH v4 0/3] perf test: Per-test timeouts, JUnit XML compliance, and snippet improvements

Quoted 7 lines · Arnaldo Carvalho de Melo
Quoted 4 lines · Namhyung Kim
Quoted 3 lines · Ian Rogers

Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness

Quoted 8 lines

Both are failing to apply to perf-tools-next, I'm pushing what I have (with 3/3) to tmp.perf-tools-next now.

Quoted 1 line
Quoted 1 line

perf test: Avoid stack trace noise in verbose=1 failure snippets

Applied.

Quoted 1 line

Thanks Namhyung and Arnaldo! I'll rebase and send a v5 series.

Ian

author reviewer
#24 lore raw

v5 cover perf test: Per-test timeouts and JUnit XML schema compliance

In reply to Ian Rogers

This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and ensures time spent waiting for the shared mlock budget does not count towards a test's timeout or reported duration:

1. Add a per-test timeout (default 120s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.

2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.

3. Record each mlock-serialized test's start time in the shared mlock_budget mapping when it acquires mlock_budget->lock in run_test_case_mlocked(), so tests waiting for the mlock budget do not time out while waiting in line and their reported elapsed time reflects actual execution time.

v5:
- Rebase onto tmp.perf-tools-next (patch 3 of v4 was applied upstream) and add Namhyung's Acked-by to patches 1 and 2.
- Add patch 3 so tests waiting on mlock_budget->lock don't time out while waiting.
- Increase the default per-test timeout from 60s to 120s. When running the full suite as root (where CAP_IPC_LOCK lifts the mlock limit and mlock_budget is unused), all ~180 non-exclusive tests start simultaneously at t=0 and contend heavily on CPUs, /proc scanning, and vmlinux/DWARF symbol resolution. On a 28-CPU hybrid machine, a 60s timeout resulted in 33 timeouts across the parallel burst and large sequential suites; increasing the default to 120s allowed longer tests ('kernel lock contention analysis test' at 115.3s, 'task-analyzer python test' at 94.7s, 'perf data type profiling tests' at 92.6s, 'perf metrics value validation' at 91.2s, 'perf record tests' at 79.0s, and 'python profiling with jitdump' at 62.7s) to complete, reducing timeouts from 33 to 2 ('perf record LBR tests' and 'perftool-testsuite_report') and overall suite wall-clock time from 503.5s to 363.9s.

v4:
- Drop the ITIMER_REAL timer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops in finish_test() and finish_tests_parallel() to detect timeouts.
- Only fall back to kill(pid, sig) when kill(-pid, sig) fails, and ignore SIGTERM/SIGINT while waiting for subprocesses in run_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.

v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.

v2:
- Reset SIGALRM to SIG_DFL in the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.

Assisted-by Antigravity:gemini-3.1-pro

Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Don't count mlock budget wait time towards test timeout

 tools/perf/Documentation/perf-test.txt |  12 +-
 tools/perf/tests/builtin-test.c        | 645 +++++++++++++++++++------
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 523 insertions(+), 141 deletions(-)
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
author reviewer
#25 lore raw

PATCH v5 1/3 perf test: Add per-test timeout support

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 120 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes against test_timeout in the existing finish_test() and finish_tests_parallel() polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

---
 tools/perf/Documentation/perf-test.txt |   7 +-
 tools/perf/tests/builtin-test.c        | 277 ++++++++++++++++++++++---
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 259 insertions(+), 32 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
index f66ccd77bfe7..f85e9b561ac4 100644
@@ -75,10 +75,15 @@ OPTIONS
 	Run each test the given number of times, by default once. This
 	option can be useful to determine if a test is flaky.
 
+-t::
+--timeout::
+	Timeout for each forked test in seconds, by default 120 seconds. Specify
+	0 to disable the timeout. Ignored when -F/--dont-fork is used.
+
 -F::
 --dont-fork::
 	Do not fork child for each test, run all tests within single process, this
-	sets sequential mode.
+	sets sequential mode and disables the per-test timeout.
 
 --dso::
 	Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +249 -28

482 lines not on this page · raw message

tools/perf/tests/tests.h +4 -3
index 5982c4e62cd3..c9e5f2b2bc11 100644
@@ -6,9 +6,10 @@
 #include "util/debug.h"
 
 enum {
-	TEST_OK   =  0,
-	TEST_FAIL = -1,
-	TEST_SKIP = -2,
+	TEST_OK      =  0,
+	TEST_FAIL    = -1,
+	TEST_SKIP    = -2,
+	TEST_TIMEOUT = -3,
 };
 
 #define TEST_ASSERT_VAL(text, cond)					 \
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
#26 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf test: pipe drained before kill_timed_out_group() kills subprocesses --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf test: pipe drained before kill_timed_out_group() kills subprocesses --

commit 9387c80164e94087cba7cfddba4fff510a8fabe3
Author: Ian Rogers <[email protected]>

perf test: Add per-test timeout support

This commit adds per-test timeout support to perf test, defaulting to 120 seconds and configurable via -t/--timeout. Forked test children are placed in their own process groups so timed-out tests and their subprocesses can be terminated and reported as timeouts in the test summary and JUnit output.

Quoted 4 lines
tools/perf/tests/builtin-test.c +0 -0
index 880eece0bc123..b870ca0bd8262 100644

[ ... ]

Quoted 4 lines
tools/perf/tests/builtin-test.c +0 -0
@@ -815,6 +893,106 @@ static void print_test_failure_snippet(FILE *fp, const char *buf)
 	free(line_lens);
 }
 

[ ... ]

Quoted 9 lines
tools/perf/tests/builtin-test.c +9 -0
+static void kill_timed_out_group(struct child_test *child)
+{
+	if (!child->timed_out || child->process.finished || child->process.pid <= 0)
+		return;
+
+	pr_debug3("Sending SIGKILL to remaining processes of timed out test %d.%d (pgid %d)\n",
+		  child->suite_num + 1, child->test_case_num + 1, child->process.pid);
+	kill(-child->process.pid, SIGKILL);
+}

[ ... ]

Quoted 16 lines
tools/perf/tests/builtin-test.c +13 -0
+static int reap_child_test(struct child_test *child)
+{
+	sigset_t oldset;
+	int ret;
+
+	block_term_signals(&oldset);
+	kill_timed_out_group(child);
+	ret = finish_command(&child->process);
+	child->process.pid = 0;
+	sigprocmask(SIG_SETMASK, &oldset, NULL);
+	return child->timed_out ? TEST_TIMEOUT : ret;
+}
+
 static void finish_test(struct child_test **child_tests, int running_test, int child_test_num,
 		int width)
 {

[ ... ]

Quoted 18 lines
tools/perf/tests/builtin-test.c +6 -2
@@ -904,11 +1088,15 @@ static void finish_test(struct child_test **child_tests, int running_test, int c
 					err_done = false;
 					buf[len] = '\0';
 					strbuf_addstr_safe(&err_output, buf);
+				} else if (pfds[0].revents & (POLLHUP | POLLERR | POLLNVAL)) {
+					close(err);
+					err = -1;
+					child_test->process.err = -1;
 				}
 			}
 		}
-		if (err_done)
-			err_done = check_if_command_finished(&child_test->process);
+		if (err_done || child_test->timed_out)
+			err_done = test_command_finished(child_test);
 	}
 	/* Drain any remaining data from the pipe. */
 	if (err >= 0) {

[ ... ]

Quoted 7 lines
tools/perf/tests/builtin-test.c +1 -2
@@ -925,8 +1113,7 @@ static void finish_test(struct child_test **child_tests, int running_test, int c
 		fprintf(debug_file(), PERF_COLOR_DELETE_LINE);
 	}
 	/* Clean up child process. */
-	ret = finish_command(&child_test->process);
-	child_test->process.pid = 0;
+	ret = reap_child_test(child_test);

[ ... ]

Quoted 7 lines
tools/perf/tests/builtin-test.c +1 -1
@@ -1157,14 +1350,13 @@ static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes
 			if (!child || child->done)
 				continue;
 
-			if (check_if_command_finished(&child->process)) {
+			if (test_command_finished(child)) {
 				if (child->process.err >= 0) {

tools/perf/tests/builtin-test.c static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes · line 1362

Quoted 10 lines
tools/perf/tests/builtin-test.c +1 -2
 					drain_child_process_err(child);
 					close(child->process.err);
 					child->process.err = -1;
 				}
-				child->result = finish_command(&child->process);
-				child->process.pid = 0;
+				child->result = reap_child_test(child);
 				clock_gettime(CLOCK_MONOTONIC, &child->end_time);
 				child->done = true;
 			}

[Severity: Medium]
In tools/perf/tests/builtin-test.c, when running with verbose >= 1 so child stdout and stderr are piped through process.err, both finish_test() and finish_tests_parallel() drain the pipe in an unbounded read loop before calling reap_child_test() -> kill_timed_out_group():

tools/perf/tests/builtin-test.c:finish_test() {
	...
		if (err_done || child_test->timed_out)
			err_done = test_command_finished(child_test);
	}
	/* Drain any remaining data from the pipe. */
	if (err >= 0) {
		char buf[512];
		ssize_t len;

		while ((len = read(err, buf, sizeof(buf) - 1)) > 0) {
			buf[len] = '\0';
			strbuf_addstr_safe(&err_output, buf);
		}
	}
	...
	ret = reap_child_test(child_test);
}

And in parallel mode:

tools/perf/tests/builtin-test.c:finish_tests_parallel() {
	...
	if (test_command_finished(child)) {
		if (child->process.err >= 0) {
			drain_child_process_err(child);
			close(child->process.err);
			child->process.err = -1;
		}
		child->result = reap_child_test(child);
	...
}

If a timed-out test's main child process exits after the initial termination signal while a subprocess in its process group survives and continuously writes to stdout or stderr, test_command_finished() returns true once the main child becomes a zombie. However, kill_timed_out_group() has not yet been called to terminate the remaining process group members.

Could a noisy surviving subprocess keep feeding the pipe so that the read() loop in finish_test() or drain_child_process_err() never sees -EAGAIN or end-of-file, hanging the test runner and growing err_output without bound before reap_child_test() can call kill_timed_out_group()? Should kill_timed_out_group() be invoked before draining the pipe when a timed-out child has finished?

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author reviewer
#27 lore raw

PATCH v5 2/3 perf test: Improve JUnit XML schema compliance and completeness

In reply to Ian Rogers

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

---
 tools/perf/Documentation/perf-test.txt |   5 +
 tools/perf/tests/builtin-test.c        | 309 +++++++++++++++++--------
 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
index f85e9b561ac4..ca58f8d6298f 100644
@@ -85,6 +85,11 @@ OPTIONS
 	Do not fork child for each test, run all tests within single process, this
 	sets sequential mode and disables the per-test timeout.
 
+-j[<file>]::
+--junit[=<file>]::
+	Generate JUnit XML test report to the specified file, or 'test.xml' if
+	no file is specified.
+
 --dso::
 	Specify a DSO for the "Symbols" test.
 
tools/perf/tests/builtin-test.c +207 -102

370 lines not on this page · raw message

Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
#28 lore raw

Re: [PATCH v5 2/3] perf test: Improve JUnit XML schema compliance and completeness

Sashiko has reviewed this patch and found no issues.
Quoted 19 lines

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Quoted 1 line

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author reviewer
#29 lore raw

PATCH v5 3/3 perf test: Don't count mlock budget wait time towards test timeout

In reply to Ian Rogers

When multiple parallel tests need room in kernel.perf_event_mlock_kb, they take turns acquiring the shared mlock_budget->lock mutex in run_test_case_mlocked() after being forked in start_test().

Because child->start_time is recorded in start_test() when the child process is forked, any time a test spends waiting in the mlock_budget->lock queue would otherwise be counted against both its per-test timeout (causing tests later in the queue to time out before they even start) and its JUnit XML testcase elapsed time.

Extend struct mlock_budget with a per-slot shared start_time and started flag populated in run_test_case_mlocked() once the child acquires mlock_budget->lock. In check_child_timeout(), skip tests that are still waiting for mlock_budget->lock and measure the timeout from the lock acquisition timestamp once started. Also use the post-lock start_time when computing the test's elapsed duration.

---
 tools/perf/tests/builtin-test.c | 81 +++++++++++++++++++++++++--------
 1 file changed, 63 insertions(+), 18 deletions(-)
tools/perf/tests/builtin-test.c +63 -18
index 58dc75ba82cf..e1bfde78d34d 100644
@@ -79,8 +79,17 @@ static unsigned int test_timeout = 120;
  * processes and robust so that a test killed while holding it doesn't leave
  * the budget locked, see mutex_init_pshared_robust().
  */
+struct mlock_child_time {
+	struct timespec start_time;
+	bool started;
+};
+
 struct mlock_budget {
 	struct mutex lock;
+	size_t size;
+	size_t nr_slots;
+	size_t next_slot;
+	struct mlock_child_time slots[];
 };
 
 static struct mlock_budget *mlock_budget;
@@ -384,6 +393,7 @@ struct child_test {
 	struct timespec start_time;
 	struct timespec end_time;
 	struct timespec timeout_time;
+	struct mlock_child_time *mlock_time;
 };
 
 static jmp_buf run_test_jmp_buf;
@@ -414,6 +424,8 @@ static int run_test_case_mlocked(struct child_test *child)
 	int err;
 
 	mutex_lock(&mlock_budget->lock);
+	clock_gettime(CLOCK_MONOTONIC, &child->mlock_time->start_time);
+	__atomic_store_n(&child->mlock_time->started, true, __ATOMIC_RELEASE);
 	err = run_test_case(child);
 	mutex_unlock(&mlock_budget->lock);
 
@@ -459,7 +471,6 @@ static int run_test_child(struct child_process *process)
 		SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGINT, SIGPIPE, SIGQUIT, SIGSEGV, SIGTERM,
 	};
 	struct child_test *child = container_of(process, struct child_test, process);
-	bool needs_mlock = mlock_budget && child->test->needs_mlock;
 	int err, sig;
 
 	/*
@@ -486,7 +497,7 @@ static int run_test_child(struct child_process *process)
 	pr_debug("---- start ----\n");
 	pr_debug("test child forked, pid %d\n", getpid());
 
-	if (needs_mlock)
+	if (child->mlock_time)
 		err = run_test_case_mlocked(child);
 	else
 		err = run_test_case(child);
@@ -1027,6 +1038,28 @@ static void kill_child_process(pid_t pid, int sig)
 		kill(pid, sig);
 }
 
+static bool child_test__started(const struct child_test *child, struct timespec *start_time)
+{
+	if (child->mlock_time) {
+		if (!__atomic_load_n(&child->mlock_time->started, __ATOMIC_ACQUIRE))
+			return false;
+		*start_time = child->mlock_time->start_time;
+		return true;
+	}
+	*start_time = child->start_time;
+	return true;
+}
+
+static double child_test__elapsed_secs(const struct child_test *child,
+				       const struct timespec *end_time)
+{
+	struct timespec start_time;
+
+	if (!child_test__started(child, &start_time))
+		start_time = child->start_time;
+	return timespec_diff_secs(end_time, &start_time);
+}
+
 static void check_child_timeout(struct child_test *child, const struct timespec *now)
 {
 	pid_t pid;
@@ -1043,7 +1076,12 @@ static void check_child_timeout(struct child_test *child, const struct timespec
 		return;
 
 	if (!child->timed_out) {
-		if (timespec_diff_secs(now, &child->start_time) >= test_timeout) {
+		struct timespec start_time;
+
+		if (!child_test__started(child, &start_time))
+			return;
+
+		if (timespec_diff_secs(now, &start_time) >= test_timeout) {
 			child->timed_out = true;
 			child->timeout_time = *now;
 			pr_debug3("Timeout (%us) for test %d.%d (pid %d), sending SIGTERM\n",
@@ -1258,8 +1296,7 @@ static void finish_test(struct child_test **child_tests, int running_test, int c
 		print_test_failure_snippet(stderr, err_output.buf);
 
 	clock_gettime(CLOCK_MONOTONIC, &end_time);
-	elapsed = (end_time.tv_sec - child_test->start_time.tv_sec) +
-		  (end_time.tv_nsec - child_test->start_time.tv_nsec) / 1000000000.0;
+	elapsed = child_test__elapsed_secs(child_test, &end_time);
 
 	print_test_result(t, curr_suite, curr_test_case, ret, width, /*running=*/0,
 			  err_output.buf, elapsed);
@@ -1532,9 +1569,7 @@ static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes
 				 (child->result == TEST_FAIL || child->result == TEST_TIMEOUT))
 				print_test_failure_snippet(stderr, child->err_output.buf);
 
-			elapsed = (child->end_time.tv_sec - child->start_time.tv_sec) +
-				  (child->end_time.tv_nsec -
-				   child->start_time.tv_nsec) / 1000000000.0;
+			elapsed = child_test__elapsed_secs(child, &child->end_time);
 
 			print_test_result(child->test, child->suite_num, child->test_case_num,
 					  child->result, width, 0, child->err_output.buf, elapsed);
@@ -1606,6 +1641,9 @@ static int start_test(struct test_suite *test, int curr_suite, int curr_test_cas
 		(*child)->process.stdout_to_stderr = 1;
 	}
 	(*child)->process.no_exec_cmd = run_test_child;
+	if (mlock_budget && test->needs_mlock && !sequential && pass == 1 &&
+	    mlock_budget->next_slot < mlock_budget->nr_slots)
+		(*child)->mlock_time = &mlock_budget->slots[mlock_budget->next_slot++];
 	clock_gettime(CLOCK_MONOTONIC, &(*child)->start_time);
 	err = start_command(&(*child)->process);
 	if (err)
@@ -2199,10 +2237,10 @@ static struct test_suite **build_suites(void)
 #undef for_each_suite
 }
 
-static bool perf_test__shares_mlock_budget(struct test_suite **suites, int argc,
-					   const char *argv[], struct intlist *skiplist)
+static size_t perf_test__count_mlock_budget(struct test_suite **suites, int argc,
+					    const char *argv[], struct intlist *skiplist)
 {
-	int sharing = 0;
+	size_t sharing = 0;
 
 	/* The selection rules mirrored here are __cmd_test()'s: when they change, change both. */
 	for (int curr_suite = 0; suites[curr_suite]; curr_suite++) {
@@ -2224,10 +2262,8 @@ static bool perf_test__shares_mlock_budget(struct test_suite **suites, int argc,
 					       curr_suite, argc, argv))
 				sharing += runs_per_test;
 		}
-		if (sharing > 1)
-			return true;
 	}
-	return false;
+	return sharing;
 }
 
 /*
@@ -2250,6 +2286,8 @@ static bool perf_test__mlock_budget_exhaustible(void)
 static void perf_test__setup_mlock_budget(struct test_suite **suites, int argc,
 					  const char *argv[], struct intlist *skiplist)
 {
+	size_t sharing, size;
+
 	/* Nothing can be denied the budget, nothing to take turns for. */
 	if (!perf_test__mlock_budget_exhaustible()) {
 		pr_debug("Not taking turns using the perf mlock budget: nothing can exhaust it\n");
@@ -2257,19 +2295,26 @@ static void perf_test__setup_mlock_budget(struct test_suite **suites, int argc,
 	}
 
 	/* Nothing runs at the same time, nothing to share the budget with. */
-	if (sequential || !perf_test__shares_mlock_budget(suites, argc, argv, skiplist))
+	if (sequential)
+		return;
+
+	sharing = perf_test__count_mlock_budget(suites, argc, argv, skiplist);
+	if (sharing <= 1)
 		return;
 
-	mlock_budget = mmap(NULL, sizeof(*mlock_budget), PROT_READ | PROT_WRITE,
+	size = sizeof(*mlock_budget) + sharing * sizeof(mlock_budget->slots[0]);
+	mlock_budget = mmap(NULL, size, PROT_READ | PROT_WRITE,
 			    MAP_SHARED | MAP_ANONYMOUS, /*fd=*/-1, /*offset=*/0);
 	if (mlock_budget == MAP_FAILED) {
 		mlock_budget = NULL;
 		pr_err("Failed to map the shared area used to take turns using the perf mlock budget\n");
 		return;
 	}
+	mlock_budget->size = size;
+	mlock_budget->nr_slots = sharing;
 
 	if (!mutex_init_pshared_robust(&mlock_budget->lock)) {
-		munmap(mlock_budget, sizeof(*mlock_budget));
+		munmap(mlock_budget, size);
 		mlock_budget = NULL;
 		pr_debug("Not taking turns using the perf mlock budget\n");
 	}
@@ -2281,7 +2326,7 @@ static void perf_test__teardown_mlock_budget(void)
 		return;
 
 	mutex_destroy(&mlock_budget->lock);
-	munmap(mlock_budget, sizeof(*mlock_budget));
+	munmap(mlock_budget, mlock_budget->size);
 	mlock_budget = NULL;
 }
 
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
#30 lore raw

Re: [PATCH v5 3/3] perf test: Don't count mlock budget wait time towards test timeout

Sashiko has reviewed this patch and found no issues.
Quoted 10 lines

When multiple parallel tests need room in kernel.perf_event_mlock_kb, they take turns acquiring the shared mlock_budget->lock mutex in run_test_case_mlocked() after being forked in start_test().

Because child->start_time is recorded in start_test() when the child process is forked, any time a test spends waiting in the mlock_budget->lock queue would otherwise be counted against both its per-test timeout (causing tests later in the queue to time out before they even start) and its JUnit XML testcase elapsed time.

Quoted 8 lines

Extend struct mlock_budget with a per-slot shared start_time and started flag populated in run_test_case_mlocked() once the child acquires mlock_budget->lock. In check_child_timeout(), skip tests that are still waiting for mlock_budget->lock and measure the timeout from the lock acquisition timestamp once started. Also use the post-lock start_time when computing the test's elapsed duration.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author reviewer
#31 lore raw

v6 cover perf test: Per-test timeouts and JUnit XML schema compliance

In reply to Ian Rogers

This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and ensures time spent waiting for the shared mlock budget does not count towards a test's timeout or reported duration:

1. Add a per-test timeout (default 120s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.

2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.

3. Record each mlock-serialized test's start time in the shared mlock_budget mapping when it acquires mlock_budget->lock in run_test_case_mlocked(), so tests waiting for the mlock budget do not time out while waiting in line and their reported elapsed time reflects actual execution time.

v6:
- Reap finished child tests (invoking kill_timed_out_group()) before draining process.err in finish_test() and finish_tests_parallel(), and skip draining in handle_child_pipe_activity() once a timed out test has finished, so a noisy surviving subprocess cannot block the pipe drain loop before the process group is killed.

v5:
- Rebase onto tmp.perf-tools-next (patch 3 of v4 was applied upstream) and add Namhyung's Acked-by to patches 1 and 2.
- Add patch 3 so tests waiting on mlock_budget->lock don't time out while waiting.
- Increase the default per-test timeout from 60s to 120s. When running the full suite as root (where CAP_IPC_LOCK lifts the mlock limit and mlock_budget is unused), all ~180 non-exclusive tests start simultaneously at t=0 and contend heavily on CPUs, /proc scanning, and vmlinux/DWARF symbol resolution. On a 28-CPU hybrid machine, a 60s timeout resulted in 33 timeouts across the parallel burst and large sequential suites; increasing the default to 120s allowed longer tests ('kernel lock contention analysis test' at 115.3s, 'task-analyzer python test' at 94.7s, 'perf data type profiling tests' at 92.6s, 'perf metrics value validation' at 91.2s, 'perf record tests' at 79.0s, and 'python profiling with jitdump' at 62.7s) to complete, reducing timeouts from 33 to 2 ('perf record LBR tests' and 'perftool-testsuite_report') and overall suite wall-clock time from 503.5s to 363.9s.

v4:
- Drop the ITIMER_REAL timer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops in finish_test() and finish_tests_parallel() to detect timeouts.
- Only fall back to kill(pid, sig) when kill(-pid, sig) fails, and ignore SIGTERM/SIGINT while waiting for subprocesses in run_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.

v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.

v2:
- Reset SIGALRM to SIG_DFL in the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.

Assisted-by Antigravity:gemini-3.1-pro

Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Don't count mlock budget wait time towards test timeout

 tools/perf/Documentation/perf-test.txt |  12 +-
 tools/perf/tests/builtin-test.c        | 662 +++++++++++++++++++------
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 535 insertions(+), 146 deletions(-)
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
author reviewer
#32 lore raw

PATCH v6 1/3 perf test: Add per-test timeout support

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 120 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes against test_timeout in the existing finish_test() and finish_tests_parallel() polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

---
 tools/perf/Documentation/perf-test.txt |   7 +-
 tools/perf/tests/builtin-test.c        | 294 ++++++++++++++++++++++---
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 271 insertions(+), 37 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
index f66ccd77bfe7..f85e9b561ac4 100644
@@ -75,10 +75,15 @@ OPTIONS
 	Run each test the given number of times, by default once. This
 	option can be useful to determine if a test is flaky.
 
+-t::
+--timeout::
+	Timeout for each forked test in seconds, by default 120 seconds. Specify
+	0 to disable the timeout. Ignored when -F/--dont-fork is used.
+
 -F::
 --dont-fork::
 	Do not fork child for each test, run all tests within single process, this
-	sets sequential mode.
+	sets sequential mode and disables the per-test timeout.
 
 --dso::
 	Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +261 -33

506 lines not on this page · raw message

tools/perf/tests/tests.h +4 -3
index 5982c4e62cd3..c9e5f2b2bc11 100644
@@ -6,9 +6,10 @@
 #include "util/debug.h"
 
 enum {
-	TEST_OK   =  0,
-	TEST_FAIL = -1,
-	TEST_SKIP = -2,
+	TEST_OK      =  0,
+	TEST_FAIL    = -1,
+	TEST_SKIP    = -2,
+	TEST_TIMEOUT = -3,
 };
 
 #define TEST_ASSERT_VAL(text, cond)					 \
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
#33 lore raw
Sashiko has reviewed this patch and found no issues.
Quoted 21 lines

When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.

Add a new TEST_TIMEOUT result category and a per-test timeout (default 120 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.

To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes against test_timeout in the existing finish_test() and finish_tests_parallel() polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.

Quoted 1 line

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author reviewer
#34 lore raw

PATCH v6 2/3 perf test: Improve JUnit XML schema compliance and completeness

In reply to Ian Rogers

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

---
 tools/perf/Documentation/perf-test.txt |   5 +
 tools/perf/tests/builtin-test.c        | 309 +++++++++++++++++--------
 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
index f85e9b561ac4..ca58f8d6298f 100644
@@ -85,6 +85,11 @@ OPTIONS
 	Do not fork child for each test, run all tests within single process, this
 	sets sequential mode and disables the per-test timeout.
 
+-j[<file>]::
+--junit[=<file>]::
+	Generate JUnit XML test report to the specified file, or 'test.xml' if
+	no file is specified.
+
 --dso::
 	Specify a DSO for the "Symbols" test.
 
tools/perf/tests/builtin-test.c +207 -102

370 lines not on this page · raw message

Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
#35 lore raw

Re: [PATCH v6 2/3] perf test: Improve JUnit XML schema compliance and completeness

Sashiko has reviewed this patch and found no issues.
Quoted 19 lines

Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes ('&apos;') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.

Quoted 1 line

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author reviewer
#36 lore raw

PATCH v6 3/3 perf test: Don't count mlock budget wait time towards test timeout

In reply to Ian Rogers

When multiple parallel tests need room in kernel.perf_event_mlock_kb, they take turns acquiring the shared mlock_budget->lock mutex in run_test_case_mlocked() after being forked in start_test().

Because child->start_time is recorded in start_test() when the child process is forked, any time a test spends waiting in the mlock_budget->lock queue would otherwise be counted against both its per-test timeout (causing tests later in the queue to time out before they even start) and its JUnit XML testcase elapsed time.

Extend struct mlock_budget with a per-slot shared start_time and started flag populated in run_test_case_mlocked() once the child acquires mlock_budget->lock. In check_child_timeout(), skip tests that are still waiting for mlock_budget->lock and measure the timeout from the lock acquisition timestamp once started. Also use the post-lock start_time when computing the test's elapsed duration.

---
 tools/perf/tests/builtin-test.c | 81 +++++++++++++++++++++++++--------
 1 file changed, 63 insertions(+), 18 deletions(-)
tools/perf/tests/builtin-test.c +63 -18
index ae599b6cb4c9..0732fede7d54 100644
@@ -79,8 +79,17 @@ static unsigned int test_timeout = 120;
  * processes and robust so that a test killed while holding it doesn't leave
  * the budget locked, see mutex_init_pshared_robust().
  */
+struct mlock_child_time {
+	struct timespec start_time;
+	bool started;
+};
+
 struct mlock_budget {
 	struct mutex lock;
+	size_t size;
+	size_t nr_slots;
+	size_t next_slot;
+	struct mlock_child_time slots[];
 };
 
 static struct mlock_budget *mlock_budget;
@@ -384,6 +393,7 @@ struct child_test {
 	struct timespec start_time;
 	struct timespec end_time;
 	struct timespec timeout_time;
+	struct mlock_child_time *mlock_time;
 };
 
 static jmp_buf run_test_jmp_buf;
@@ -414,6 +424,8 @@ static int run_test_case_mlocked(struct child_test *child)
 	int err;
 
 	mutex_lock(&mlock_budget->lock);
+	clock_gettime(CLOCK_MONOTONIC, &child->mlock_time->start_time);
+	__atomic_store_n(&child->mlock_time->started, true, __ATOMIC_RELEASE);
 	err = run_test_case(child);
 	mutex_unlock(&mlock_budget->lock);
 
@@ -459,7 +471,6 @@ static int run_test_child(struct child_process *process)
 		SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGINT, SIGPIPE, SIGQUIT, SIGSEGV, SIGTERM,
 	};
 	struct child_test *child = container_of(process, struct child_test, process);
-	bool needs_mlock = mlock_budget && child->test->needs_mlock;
 	int err, sig;
 
 	/*
@@ -486,7 +497,7 @@ static int run_test_child(struct child_process *process)
 	pr_debug("---- start ----\n");
 	pr_debug("test child forked, pid %d\n", getpid());
 
-	if (needs_mlock)
+	if (child->mlock_time)
 		err = run_test_case_mlocked(child);
 	else
 		err = run_test_case(child);
@@ -1027,6 +1038,28 @@ static void kill_child_process(pid_t pid, int sig)
 		kill(pid, sig);
 }
 
+static bool child_test__started(const struct child_test *child, struct timespec *start_time)
+{
+	if (child->mlock_time) {
+		if (!__atomic_load_n(&child->mlock_time->started, __ATOMIC_ACQUIRE))
+			return false;
+		*start_time = child->mlock_time->start_time;
+		return true;
+	}
+	*start_time = child->start_time;
+	return true;
+}
+
+static double child_test__elapsed_secs(const struct child_test *child,
+				       const struct timespec *end_time)
+{
+	struct timespec start_time;
+
+	if (!child_test__started(child, &start_time))
+		start_time = child->start_time;
+	return timespec_diff_secs(end_time, &start_time);
+}
+
 static void check_child_timeout(struct child_test *child, const struct timespec *now)
 {
 	pid_t pid;
@@ -1043,7 +1076,12 @@ static void check_child_timeout(struct child_test *child, const struct timespec
 		return;
 
 	if (!child->timed_out) {
-		if (timespec_diff_secs(now, &child->start_time) >= test_timeout) {
+		struct timespec start_time;
+
+		if (!child_test__started(child, &start_time))
+			return;
+
+		if (timespec_diff_secs(now, &start_time) >= test_timeout) {
 			child->timed_out = true;
 			child->timeout_time = *now;
 			pr_debug3("Timeout (%us) for test %d.%d (pid %d), sending SIGTERM\n",
@@ -1262,8 +1300,7 @@ static void finish_test(struct child_test **child_tests, int running_test, int c
 		print_test_failure_snippet(stderr, err_output.buf);
 
 	clock_gettime(CLOCK_MONOTONIC, &end_time);
-	elapsed = (end_time.tv_sec - child_test->start_time.tv_sec) +
-		  (end_time.tv_nsec - child_test->start_time.tv_nsec) / 1000000000.0;
+	elapsed = child_test__elapsed_secs(child_test, &end_time);
 
 	print_test_result(t, curr_suite, curr_test_case, ret, width, /*running=*/0,
 			  err_output.buf, elapsed);
@@ -1539,9 +1576,7 @@ static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes
 				 (child->result == TEST_FAIL || child->result == TEST_TIMEOUT))
 				print_test_failure_snippet(stderr, child->err_output.buf);
 
-			elapsed = (child->end_time.tv_sec - child->start_time.tv_sec) +
-				  (child->end_time.tv_nsec -
-				   child->start_time.tv_nsec) / 1000000000.0;
+			elapsed = child_test__elapsed_secs(child, &child->end_time);
 
 			print_test_result(child->test, child->suite_num, child->test_case_num,
 					  child->result, width, 0, child->err_output.buf, elapsed);
@@ -1613,6 +1648,9 @@ static int start_test(struct test_suite *test, int curr_suite, int curr_test_cas
 		(*child)->process.stdout_to_stderr = 1;
 	}
 	(*child)->process.no_exec_cmd = run_test_child;
+	if (mlock_budget && test->needs_mlock && !sequential && pass == 1 &&
+	    mlock_budget->next_slot < mlock_budget->nr_slots)
+		(*child)->mlock_time = &mlock_budget->slots[mlock_budget->next_slot++];
 	clock_gettime(CLOCK_MONOTONIC, &(*child)->start_time);
 	err = start_command(&(*child)->process);
 	if (err)
@@ -2206,10 +2244,10 @@ static struct test_suite **build_suites(void)
 #undef for_each_suite
 }
 
-static bool perf_test__shares_mlock_budget(struct test_suite **suites, int argc,
-					   const char *argv[], struct intlist *skiplist)
+static size_t perf_test__count_mlock_budget(struct test_suite **suites, int argc,
+					    const char *argv[], struct intlist *skiplist)
 {
-	int sharing = 0;
+	size_t sharing = 0;
 
 	/* The selection rules mirrored here are __cmd_test()'s: when they change, change both. */
 	for (int curr_suite = 0; suites[curr_suite]; curr_suite++) {
@@ -2231,10 +2269,8 @@ static bool perf_test__shares_mlock_budget(struct test_suite **suites, int argc,
 					       curr_suite, argc, argv))
 				sharing += runs_per_test;
 		}
-		if (sharing > 1)
-			return true;
 	}
-	return false;
+	return sharing;
 }
 
 /*
@@ -2257,6 +2293,8 @@ static bool perf_test__mlock_budget_exhaustible(void)
 static void perf_test__setup_mlock_budget(struct test_suite **suites, int argc,
 					  const char *argv[], struct intlist *skiplist)
 {
+	size_t sharing, size;
+
 	/* Nothing can be denied the budget, nothing to take turns for. */
 	if (!perf_test__mlock_budget_exhaustible()) {
 		pr_debug("Not taking turns using the perf mlock budget: nothing can exhaust it\n");
@@ -2264,19 +2302,26 @@ static void perf_test__setup_mlock_budget(struct test_suite **suites, int argc,
 	}
 
 	/* Nothing runs at the same time, nothing to share the budget with. */
-	if (sequential || !perf_test__shares_mlock_budget(suites, argc, argv, skiplist))
+	if (sequential)
+		return;
+
+	sharing = perf_test__count_mlock_budget(suites, argc, argv, skiplist);
+	if (sharing <= 1)
 		return;
 
-	mlock_budget = mmap(NULL, sizeof(*mlock_budget), PROT_READ | PROT_WRITE,
+	size = sizeof(*mlock_budget) + sharing * sizeof(mlock_budget->slots[0]);
+	mlock_budget = mmap(NULL, size, PROT_READ | PROT_WRITE,
 			    MAP_SHARED | MAP_ANONYMOUS, /*fd=*/-1, /*offset=*/0);
 	if (mlock_budget == MAP_FAILED) {
 		mlock_budget = NULL;
 		pr_err("Failed to map the shared area used to take turns using the perf mlock budget\n");
 		return;
 	}
+	mlock_budget->size = size;
+	mlock_budget->nr_slots = sharing;
 
 	if (!mutex_init_pshared_robust(&mlock_budget->lock)) {
-		munmap(mlock_budget, sizeof(*mlock_budget));
+		munmap(mlock_budget, size);
 		mlock_budget = NULL;
 		pr_debug("Not taking turns using the perf mlock budget\n");
 	}
@@ -2288,7 +2333,7 @@ static void perf_test__teardown_mlock_budget(void)
 		return;
 
 	mutex_destroy(&mlock_budget->lock);
-	munmap(mlock_budget, sizeof(*mlock_budget));
+	munmap(mlock_budget, mlock_budget->size);
 	mlock_budget = NULL;
 }
 
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
#37 lore raw

Re: [PATCH v6 3/3] perf test: Don't count mlock budget wait time towards test timeout

Sashiko has reviewed this patch and found no issues.
Quoted 10 lines

When multiple parallel tests need room in kernel.perf_event_mlock_kb, they take turns acquiring the shared mlock_budget->lock mutex in run_test_case_mlocked() after being forked in start_test().

Because child->start_time is recorded in start_test() when the child process is forked, any time a test spends waiting in the mlock_budget->lock queue would otherwise be counted against both its per-test timeout (causing tests later in the queue to time out before they even start) and its JUnit XML testcase elapsed time.

Quoted 8 lines

Extend struct mlock_budget with a per-slot shared start_time and started flag populated in run_test_case_mlocked() once the child acquires mlock_budget->lock. In check_child_timeout(), skip tests that are still waiting for mlock_budget->lock and measure the timeout from the lock acquisition timestamp once started. Also use the post-lock start_time when computing the test's elapsed duration.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author reviewer
#38 lore raw

Re: [PATCH v6 0/3] perf test: Per-test timeouts and JUnit XML schema compliance

In reply to Ian Rogers
Quoted 76 lines · Ian Rogers

This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and ensures time spent waiting for the shared mlock budget does not count towards a test's timeout or reported duration:

1. Add a per-test timeout (default 120s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.

2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.

3. Record each mlock-serialized test's start time in the shared mlock_budget mapping when it acquires mlock_budget->lock in run_test_case_mlocked(), so tests waiting for the mlock budget do not time out while waiting in line and their reported elapsed time reflects actual execution time.

v6:
- Reap finished child tests (invoking kill_timed_out_group()) before draining process.err in finish_test() and finish_tests_parallel(), and skip draining in handle_child_pipe_activity() once a timed out test has finished, so a noisy surviving subprocess cannot block the pipe drain loop before the process group is killed.

v5:
- Rebase onto tmp.perf-tools-next (patch 3 of v4 was applied upstream) and add Namhyung's Acked-by to patches 1 and 2.
- Add patch 3 so tests waiting on mlock_budget->lock don't time out while waiting.
- Increase the default per-test timeout from 60s to 120s. When running the full suite as root (where CAP_IPC_LOCK lifts the mlock limit and mlock_budget is unused), all ~180 non-exclusive tests start simultaneously at t=0 and contend heavily on CPUs, /proc scanning, and vmlinux/DWARF symbol resolution. On a 28-CPU hybrid machine, a 60s timeout resulted in 33 timeouts across the parallel burst and large sequential suites; increasing the default to 120s allowed longer tests ('kernel lock contention analysis test' at 115.3s, 'task-analyzer python test' at 94.7s, 'perf data type profiling tests' at 92.6s, 'perf metrics value validation' at 91.2s, 'perf record tests' at 79.0s, and 'python profiling with jitdump' at 62.7s) to complete, reducing timeouts from 33 to 2 ('perf record LBR tests' and 'perftool-testsuite_report') and overall suite wall-clock time from 503.5s to 363.9s.

v4:
- Drop the ITIMER_REAL timer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops in finish_test() and finish_tests_parallel() to detect timeouts.
- Only fall back to kill(pid, sig) when kill(-pid, sig) fails, and ignore SIGTERM/SIGINT while waiting for subprocesses in run_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.

v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.

v2:
- Reset SIGALRM to SIG_DFL in the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.

Quoted 6 lines
Assisted-by Antigravity:gemini-3.1-pro

Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Don't count mlock budget wait time towards test timeout

Ping.

The sashiko reviews are clean:
https://sashiko.dev/#/patchset/[email protected]

There's an additional patch for making the mlock waiting time not count as timeout time:
https://lore.kernel.org/linux-perf-users/[email protected]/
that is missing any tags.

Thanks,
Ian

Quoted 8 lines
 tools/perf/Documentation/perf-test.txt |  12 +-
 tools/perf/tests/builtin-test.c        | 662 +++++++++++++++++++------
 tools/perf/tests/tests.h               |   7 +-
 3 files changed, 535 insertions(+), 146 deletions(-)

--
2.56.0.307.g5fe7bd680c-goog

maintainer Applied
#39 lore raw

Re: [PATCH v6 0/3] perf test: Per-test timeouts and JUnit XML schema compliance

In reply to Ian Rogers
Quoted 14 lines · Ian Rogers

This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and ensures time spent waiting for the shared mlock budget does not count towards a test's timeout or reported duration:

1. Add a per-test timeout (default 120s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.

Quoted 1 line

[...]

Applied to perf-tools-next, thanks!

Best regards
Namhyung