perf test: Add per-test timeout support
Ian Rogers
reviewerPATCH 1/2 perf test: Add per-test timeout support
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.
To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).
- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 7 +- tools/perf/tests/builtin-test.c | 179 ++++++++++++++++++++++--- tools/perf/tests/tests.h | 7 +- 3 files changed, 174 insertions(+), 19 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
@@ -45,10 +45,15 @@ OPTIONS Run each test the given number of times, by default once. This option can be useful to determine if a test is flaky. +-t:: +--timeout:: + Timeout for each forked test in seconds, by default 60 seconds. Specify + 0 to disable the timeout. Ignored when -F/--dont-fork is used. + -F:: --dont-fork:: Do not fork child for each test, run all tests within single process, this - sets sequential mode. + sets sequential mode and disables the per-test timeout. --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +164 -15
374 lines not on this page · raw message
tools/perf/tests/tests.h +4 -3
@@ -6,9 +6,10 @@#include "util/debug.h" enum { -TEST_OK =0 , -TEST_FAIL =-1 , -TEST_SKIP =-2 , +TEST_OK =0 , +TEST_FAIL =-1 , +TEST_SKIP =-2 , +TEST_TIMEOUT =-3 , };#define TEST_ASSERT_VAL (text, cond) \
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Ian Rogers
reviewerPATCH 2/2 perf test: Improve JUnit XML schema compliance and completeness
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 5 + tools/perf/tests/builtin-test.c | 309 +++++++++++++++++-------- 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
@@ -55,6 +55,11 @@ OPTIONS Do not fork child for each test, run all tests within single process, this sets sequential mode and disables the per-test timeout. +-j[<file>]:: +--junit[=<file>]:: + Generate JUnit XML test report to the specified file, or 'test.xml' if + no file is specified. + --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +207 -102
370 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 2/2] perf test: Improve JUnit XML schema compliance and completeness
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints inxml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] perf test: overly broad waitpid() condition hangs test suite on normal failures - [Medium] perf…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] perf test: overly broad waitpid() condition hangs test suite on normal failures
- [Medium] perf test: child tests inherit SA_RESTART SIGALRM handler, breaking alarm()
- [Medium] perf test: SIGKILL escalation for timed-out process groups is aborted prematurely --
commit efae32fde876d5c28dcaff79950fdf6547005215
Author: Ian Rogers <[email protected]>
perf test: Add per-test timeout support
This commit adds a new per-test timeout to perf test, defaulting to 60 seconds. Tests are placed into separate process groups, and timeouts are enforced using an ITIMER_REAL periodic timer that sends SIGTERM followed by a SIGKILL escalation after 1 second.
Quoted 3 lines
tools/perf/tests/builtin-test.c +0
-0
[ ... ]
tools/perf/tests/builtin-test.c static int run_test_child(struct child_process *process) · line 399
Quoted 7 lines
tools/perf/tests/builtin-test.c +0
-1@@ -378,20 +395,31 @@ static int run_test_child(struct child_process *process)check_leaks (); err_out: -fflush (NULL );for (size_t i =0 ; i <ARRAY_SIZE (signals); i++)signal (signals[i],SIG_DFL );
[Severity: Medium]
Does this omit SIGALRM from being reset to SIG_DFL?
Since the parent __cmd_test() sets up a no-op handler with SA_RESTART for SIGALRM before forking, children will inherit that handler. Will builtin tests that rely on the default SIGALRM behavior (like using alarm() to prevent internal hangs) fail to terminate because the no-op handler overrides it?
tools/perf/tests/builtin-test.c static int run_test_child(struct child_process *process) · line 408
Quoted 9 lines
tools/perf/tests/builtin-test.c +9
-0+if (err ==TEST_FAIL ) { +/* +* Wait for any spawned child processes (such as shell scripts +* running trap cleanup handlers on SIGTERM) to finish before +* exiting. +*/ +while (waitpid (-1 ,NULL ,0 ) >0 || errno ==EINTR ) + ; + }
[Severity: High]
Can this loop hang the test suite indefinitely on normal failures?
This condition catches all normal test failures (err == TEST_FAIL), not just tests interrupted by a timeout SIGTERM. If a test spawns child processes and fails normally, it will block on waitpid(-1) because the child processes were never signaled to exit.
Will this hang the entire test run until the global timeout triggers, or deadlock completely if timeouts are disabled via the dont-fork path?
Quoted 3 lines
tools/perf/tests/builtin-test.c +1
-0+fflush (NULL );return -err; }
[ ... ]
Quoted 5 lines
tools/perf/tests/builtin-test.c +1
-0@@ -748,6 +792,47 @@ static void print_test_failure_snippet(FILE *fp, const char *buf)free (line_lens); } +static void cmd_test_alarm_handler (int sig__maybe_unused )
tools/perf/tests/builtin-test.c static void print_test_failure_snippet(FILE *fp, const char *buf) · line 805
Quoted 10 lines
tools/perf/tests/builtin-test.c +10
-0+{ +} + +static void check_child_timeout (struct child_test *child,const struct timespec *now) +{ +double elapsed; +pid_t pid; + +if (!test_timeout || !child || child->done || child->process.finished) +return ;
[Severity: Medium]
Does returning early when child->done is true bypass the SIGKILL escalation?
If the immediate child process exits after receiving SIGTERM, the parent marks the entire test case as done (setting child->done to true in finish_tests_parallel).
Will checking child->done here completely bypass the 1-second timer and the kill(-pid, SIGKILL) call for the broader process group, leaving any subprocesses that ignored SIGTERM to leak?
Quoted 29 lines
tools/perf/tests/builtin-test.c +29
-0+ + pid = child->process.pid; +if (pid <=0 ) +return ; + +if (!child->timed_out) { + elapsed = (now->tv_sec - child->start_time.tv_sec) + + (now->tv_nsec - child->start_time.tv_nsec) /1000000000.0 ; +if (elapsed >= test_timeout) { + child->timed_out = true; + child->timeout_time = *now; +pr_debug3 ("Timeout (%us) for test %d.%d (pid %d), sending SIGTERM\n" , + test_timeout, child->suite_num +1 , + child->test_case_num +1 , pid); +kill (-pid,SIGTERM ); +kill (pid,SIGTERM ); + } + }else if (!child->sigkill_sent) { + elapsed = (now->tv_sec - child->timeout_time.tv_sec) + + (now->tv_nsec - child->timeout_time.tv_nsec) /1000000000.0 ; +if (elapsed >=1.0 ) { + child->sigkill_sent = true; +pr_debug3 ("Sending SIGKILL to timed out test %d.%d (pid %d)\n" , + child->suite_num +1 , child->test_case_num +1 , pid); +kill (-pid,SIGKILL ); +kill (pid,SIGKILL ); + } + } +}
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH v2 1/2 perf test: Add per-test timeout support
In reply to Ian RogersWhen a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.
To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).
- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
---
v2: Address Sashiko review of v1:
- Reset SIGALRM to SIG_DFL in the forked test child so tests relying on default alarm() behaviour aren't affected by the parent's no-op handler.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when the test was interrupted by SIGTERM/SIGINT, avoiding hangs on normal test failures.
- Ensure the SIGKILL escalation reaches the whole process group even when the direct child exits after SIGTERM, by killing the group before the zombie is reaped (so the PGID can't be reused).
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.
--- tools/perf/Documentation/perf-test.txt | 7 +- tools/perf/tests/builtin-test.c | 241 +++++++++++++++++++++++-- tools/perf/tests/tests.h | 7 +- 3 files changed, 233 insertions(+), 22 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
@@ -45,10 +45,15 @@ OPTIONS Run each test the given number of times, by default once. This option can be useful to determine if a test is flaky. +-t:: +--timeout:: + Timeout for each forked test in seconds, by default 60 seconds. Specify + 0 to disable the timeout. Ignored when -F/--dont-fork is used. + -F:: --dont-fork:: Do not fork child for each test, run all tests within single process, this - sets sequential mode. + sets sequential mode and disables the per-test timeout. --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +223 -18
443 lines not on this page · raw message
tools/perf/tests/tests.h +4 -3
@@ -6,9 +6,10 @@#include "util/debug.h" enum { -TEST_OK =0 , -TEST_FAIL =-1 , -TEST_SKIP =-2 , +TEST_OK =0 , +TEST_FAIL =-1 , +TEST_SKIP =-2 , +TEST_TIMEOUT =-3 , };#define TEST_ASSERT_VAL (text, cond) \
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Ian Rogers
reviewerPATCH v2 2/2 perf test: Improve JUnit XML schema compliance and completeness
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 5 + tools/perf/tests/builtin-test.c | 309 +++++++++++++++++-------- 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
@@ -55,6 +55,11 @@ OPTIONS Do not fork child for each test, run all tests within single process, this sets sequential mode and disables the per-test timeout. +-j[<file>]:: +--junit[=<file>]:: + Generate JUnit XML test report to the specified file, or 'test.xml' if + no file is specified. + --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +207 -102
370 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 2/2] perf test: Improve JUnit XML schema compliance and completeness
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints inxml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf test: TOCTOU race in signal handler can kill unintended process groups --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf test: TOCTOU race in signal handler can kill unintended process groups --
commit e2e28e1fd08275414fa889f24a3483b8a0075cc0
Author: Ian Rogers <[email protected]>
perf test: Add per-test timeout support
When a test hangs in perf test, the entire test run hangs and makes it difficult to inspect output from other tests. Add a new TEST_TIMEOUT result category and a per-test timeout.
Quoted 4 lines
tools/perf/tests/builtin-test.c +0
-0
[ ... ]
tools/perf/tests/builtin-test.c static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes · line 1261
Quoted 9 lines
tools/perf/tests/builtin-test.c +3
-0@@ -1096,7 +1254,10 @@ static int finish_tests_parallel(struct child_test **child_tests, size_t num_tesclose (child->process.err); child->process.err =-1 ; } +kill_timed_out_group (child); child->result =finish_command (&child->process); +if (child->timed_out) + child->result =TEST_TIMEOUT ; child->process.pid =0 ;
[Severity: Medium]
Is there a TOCTOU race window here in finish_tests_parallel() before child->process.pid is zeroed?
If check_if_command_finished() falls back to waitpid(..., WNOHANG) (e.g. when /proc is unmounted) and reaps the child, finish_command() won't block.
If cmd_test_sig_handler is triggered via SIGINT or SIGTERM right after finish_command() reaps the child but before child->process.pid is zeroed, the asynchronous signal handler jumps back to __cmd_test() and executes a kill on the stale PID.
[ ... ]
tools/perf/tests/builtin-test.c static int __cmd_test(struct test_suite **suites, int argc, const char *argv[], · line 1514
Quoted 9 lines
tools/perf/tests/builtin-test.c +1
-0@@ -1327,12 +1507,27 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],pr_debug3 ("Killing %d pid %d\n" , child_test->suite_num +1 , child_test->process.pid); +kill (-child_test->process.pid, err);kill (child_test->process.pid, err); } goto err_out; }
[Severity: Medium]
Can this escalation from kill(pid) to kill(-pid) in __cmd_test() turn the single-process race into a full process group kill?
If the race window described above occurs, and a concurrent high-rate fork workload wraps the PID space exactly to the newly reaped PID, this kill(-pid) command could wipe out an entire unrelated process group rather than just a single reused PID.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH v3 1/2 perf test: Add per-test timeout support
In reply to Ian RogersWhen a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.
To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm an ITIMER_REAL periodic timer with a SIGALRM handler (using SA_RESTART so /proc status reads in check_if_command_finished() are not interrupted by EINTR while poll()/usleep() waits still wake up).
- Check elapsed child runtimes against test_timeout in finish_test() and finish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
---
v3: Address Sashiko review of v2:
- Block SIGINT/SIGTERM while a test process may be reaped (in check_if_command_finished() and finish_command()) and until its pid is cleared, and skip already reaped tests in the signal path, so a stale pid or process group ID can never be signaled.
v2: Address Sashiko review of v1:
- Reset SIGALRM to SIG_DFL in the forked test child so tests relying on default alarm() behaviour aren't affected by the parent's no-op handler.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when the test was interrupted by SIGTERM/SIGINT, avoiding hangs on normal test failures.
- Ensure the SIGKILL escalation reaches the whole process group even when the direct child exits after SIGTERM, by killing the group before the zombie is reaped (so the PGID can't be reused).
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.
--- tools/perf/Documentation/perf-test.txt | 7 +- tools/perf/tests/builtin-test.c | 297 ++++++++++++++++++++++--- tools/perf/tests/tests.h | 7 +- 3 files changed, 280 insertions(+), 31 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
@@ -45,10 +45,15 @@ OPTIONS Run each test the given number of times, by default once. This option can be useful to determine if a test is flaky. +-t:: +--timeout:: + Timeout for each forked test in seconds, by default 60 seconds. Specify + 0 to disable the timeout. Ignored when -F/--dont-fork is used. + -F:: --dont-fork:: Do not fork child for each test, run all tests within single process, this - sets sequential mode. + sets sequential mode and disables the per-test timeout. --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +270 -27
513 lines not on this page · raw message
tools/perf/tests/tests.h +4 -3
@@ -6,9 +6,10 @@#include "util/debug.h" enum { -TEST_OK =0 , -TEST_FAIL =-1 , -TEST_SKIP =-2 , +TEST_OK =0 , +TEST_FAIL =-1 , +TEST_SKIP =-2 , +TEST_TIMEOUT =-3 , };#define TEST_ASSERT_VAL (text, cond) \
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Ian Rogers
reviewerPATCH v3 2/2 perf test: Improve JUnit XML schema compliance and completeness
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
---
v3: No changes.
v2: No changes.
--- tools/perf/Documentation/perf-test.txt | 5 + tools/perf/tests/builtin-test.c | 309 +++++++++++++++++-------- 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
@@ -55,6 +55,11 @@ OPTIONS Do not fork child for each test, run all tests within single process, this sets sequential mode and disables the per-test timeout. +-j[<file>]:: +--junit[=<file>]:: + Generate JUnit XML test report to the specified file, or 'test.xml' if + no file is specified. + --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +207 -102
370 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 2/2] perf test: Improve JUnit XML schema compliance and completeness
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints inxml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewSashiko has reviewed this patch and found no issues. It looks great!
Quoted 23 lines
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new
TEST_TIMEOUTresult category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.To enforce the timeout:
- Place each forked test child into its own process group viasetpgid(0, 0)in the child andsetpgid(pid, pid)in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm anITIMER_REALperiodic timer with a SIGALRM handler (usingSA_RESTARTso /proc status reads incheck_if_command_finished()are not interrupted by EINTR while poll()/usleep() waits still wake up).
- Check elapsed child runtimes againsttest_timeoutinfinish_test()andfinish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Namhyung Kim
maintainerQuoted 7 lines · Ian Rogers
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new
TEST_TIMEOUTresult category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.
Quoted 8 lines
To enforce the timeout:
- Place each forked test child into its own process group viasetpgid(0, 0)in the child andsetpgid(pid, pid)in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm anITIMER_REALperiodic timer with a SIGALRM handler (usingSA_RESTARTso /proc status reads incheck_if_command_finished()are not interrupted by EINTR while poll()/usleep() waits still wake up).
Do we really need this timer? I think poll() already has a timeout so it'll wake up on every 100 msec anyway?
Thanks,
Namhyung
Quoted 10 lines
- Check elapsed child runtimes against
test_timeoutinfinish_test()andfinish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Ian Rogers
reviewerQuoted 18 lines · Namhyung Kim
Quoted 7 lines · Ian Rogers
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new
TEST_TIMEOUTresult category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.Quoted 8 lines
To enforce the timeout:
- Place each forked test child into its own process group viasetpgid(0, 0)in the child andsetpgid(pid, pid)in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Arm anITIMER_REALperiodic timer with a SIGALRM handler (usingSA_RESTARTso /proc status reads incheck_if_command_finished()are not interrupted by EINTR while poll()/usleep() waits still wake up).
Quoted 2 lines
Do we really need this timer? I think
poll()already has a timeout so it'll wake up on every 100 msec anyway?
Agreed and that avoids async/signal issues. I will fix this in v4.
Thanks,
Ian
Quoted 14 lines
Thanks,
NamhyungQuoted 10 lines
- Check elapsed child runtimes against
test_timeoutinfinish_test()andfinish_tests_parallel(), sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Ian Rogers
reviewerv4 cover perf test: Per-test timeouts, JUnit XML compliance, and snippet improvements
In reply to Ian RogersThis series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and refines the verbose=1 failure snippet output when tests fail with a signal or time out:
1. Add a per-test timeout (default 60s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.
2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.
3. Improve verbose=1 failure snippet selection in print_test_failure_snippet() to skip blank and stack trace lines when selecting context and scanning backwards from the end of the log, and prioritize the test output line immediately preceding '---- unexpected signal' so trailing test progress is shown instead of outermost stack frames.
v4:
- Drop the ITIMER_REAL timer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops in finish_test() and finish_tests_parallel() to detect timeouts.
- Only fall back to kill(pid, sig) when kill(-pid, sig) fails, and ignore SIGTERM/SIGINT while waiting for subprocesses in run_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.
v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.
v2:
- Reset SIGALRM to SIG_DFL in the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.
Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Avoid stack trace noise in verbose=1 failure snippets
tools/perf/Documentation/perf-test.txt | 12 +- tools/perf/tests/builtin-test.c | 668 +++++++++++++++++++------ tools/perf/tests/tests.h | 7 +- 3 files changed, 543 insertions(+), 144 deletions(-)
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
Ian Rogers
reviewerPATCH v4 1/3 perf test: Add per-test timeout support
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new TEST_TIMEOUT result category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.
To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes against test_timeout in the existing finish_test() and finish_tests_parallel() polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 7 +- tools/perf/tests/builtin-test.c | 278 ++++++++++++++++++++++--- tools/perf/tests/tests.h | 7 +- 3 files changed, 260 insertions(+), 32 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
@@ -45,10 +45,15 @@ OPTIONS Run each test the given number of times, by default once. This option can be useful to determine if a test is flaky. +-t:: +--timeout:: + Timeout for each forked test in seconds, by default 60 seconds. Specify + 0 to disable the timeout. Ignored when -F/--dont-fork is used. + -F:: --dont-fork:: Do not fork child for each test, run all tests within single process, this - sets sequential mode. + sets sequential mode and disables the per-test timeout. --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +250 -28
482 lines not on this page · raw message
tools/perf/tests/tests.h +4 -3
@@ -6,9 +6,10 @@#include "util/debug.h" enum { -TEST_OK =0 , -TEST_FAIL =-1 , -TEST_SKIP =-2 , +TEST_OK =0 , +TEST_FAIL =-1 , +TEST_SKIP =-2 , +TEST_TIMEOUT =-3 , };#define TEST_ASSERT_VAL (text, cond) \
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
[email protected]
AI reviewSashiko has reviewed this patch and found no issues. It looks great!
Quoted 21 lines
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new
TEST_TIMEOUTresult category and a per-test timeout (default 60 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.To enforce the timeout:
- Place each forked test child into its own process group viasetpgid(0, 0)in the child andsetpgid(pid, pid)in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes againsttest_timeoutin the existingfinish_test()andfinish_tests_parallel()polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH v4 2/3 perf test: Improve JUnit XML schema compliance and completeness
In reply to Ian RogersImprove the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 5 + tools/perf/tests/builtin-test.c | 309 +++++++++++++++++-------- 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
@@ -55,6 +55,11 @@ OPTIONS Do not fork child for each test, run all tests within single process, this sets sequential mode and disables the per-test timeout. +-j[<file>]:: +--junit[=<file>]:: + Generate JUnit XML test report to the specified file, or 'test.xml' if + no file is specified. + --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +207 -102
370 lines not on this page · raw message
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
[email protected]
AI reviewRe: [PATCH v4 2/3] perf test: Improve JUnit XML schema compliance and completeness
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints inxml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Ian Rogers
reviewerPATCH v4 3/3 perf test: Avoid stack trace noise in verbose=1 failure snippets
In reply to Ian RogersWhen a test fails with a signal or times out, child_test_sig_handler() appends '---- unexpected signal (<sig>) ----' followed by a multi-line
stack trace (' #0 ...' through '_start').
In verbose=1 mode, print_test_failure_snippet() selects up to failure_snippet_lines (10) lines:
- Pass 1 selects lines matching failure keywords plus their immediate
next line (' #0 ... in child_test_sig_handler').
- Pass 2 fills the remaining quota by scanning backwards from the end of the output buffer, which consumes the remaining lines on outermost stack frames ('handle_internal_command', 'run_argv', 'main', '__libc_start_main', '_start') and hides the trailing test output showing where the test had run to.
Improve print_test_failure_snippet() by:
- Skipping blank lines and stack trace lines (' #<digit>...') when
selecting context in Pass 1 and when scanning backwards from the end in Pass 2, with a Pass 3 fallback from the top of the buffer if quota still remains.
- Prioritizing the non-blank line immediately preceding '---- unexpected signal' so there is no '...' gap before the signal message.
- Only printing '...' between selected lines when at least one non-blank line was omitted between them.
--- tools/perf/tests/builtin-test.c | 101 +++++++++++++++++++++++++------- 1 file changed, 81 insertions(+), 20 deletions(-)
tools/perf/tests/builtin-test.c +81 -20
@@ -774,6 +774,47 @@ static void print_line_highlighted(FILE *fp, const char *line, size_t len) } +static bool is_blank_line (const char *line,size_t len) +{ +for (size_t i =0 ; i < len; i++) { +if (!isspace ((unsigned char )line[i])) +return false; + } +return true; +} + +static bool is_stack_line (const char *line,size_t len) +{ +size_t i =0 ; + +while (i < len &&isspace ((unsigned char )line[i])) + i++; +return i +1 < len && line[i] =='#' &&isdigit ((unsigned char )line[i +1 ]); +} + +static bool is_ignorable_snippet_line (const char *line,size_t len) +{ +return is_blank_line (line, len) ||is_stack_line (line, len); +} + +static bool has_non_blank_between (const char **lines,const size_t *line_lens, +size_t start,size_t end) +{ +for (size_t j = start +1 ; j < end; j++) { +if (!is_blank_line (lines[j], line_lens[j])) +return true; + } +return false; +} + +static void pick_snippet_line (bool *pick,size_t idx,unsigned int *picked_count) +{ +if (!pick[idx] && *picked_count < failure_snippet_lines) { + pick[idx] = true; + (*picked_count)++; + } +} +static void print_test_failure_snippet (FILE *fp,const char *buf) {size_t num_lines =0 ;
@@ -844,43 +885,63 @@ static void print_test_failure_snippet(FILE *fp, const char *buf) }/* Pass 0: Always pick the very first line */ -if (num_lines >0 && picked_count < failure_snippet_lines) { - pick[0 ] = true; - picked_count++; - } +if (num_lines >0 ) +pick_snippet_line (pick,0 , &picked_count);/* Pass 1: Pick lines with failure keywords from start (Highest Priority) */ for (i =0 ; i < num_lines && picked_count < failure_snippet_lines; i++) {size_t dummy; -if (find_next_keyword (lines[i], line_lens[i], &dummy)) { -if (!pick[i]) { - pick[i] = true; - picked_count++; - } -/* Prioritize getting the immediate next line for context */ -if (i +1 < num_lines && !pick[i +1 ] && - picked_count < failure_snippet_lines) { - pick[i +1 ] = true; - picked_count++; +if (is_ignorable_snippet_line (lines[i], line_lens[i])) +continue ; +if (!find_next_keyword (lines[i], line_lens[i], &dummy)) +continue ; + +pick_snippet_line (pick, i, &picked_count); +/* +* For unexpected signals, prioritize the preceding non-blank +* line showing where the test had run to before the signal. +*/ +if (!strncmp (lines[i],"---- unexpected signal" ,22 )) { +size_t j = i; + +while (j >0 ) { + j--; +if (!is_ignorable_snippet_line (lines[j], line_lens[j])) { +pick_snippet_line (pick, j, &picked_count); +break ; + } } } +/* Prioritize getting the immediate next line for context */ +if (i +1 < num_lines && + !is_ignorable_snippet_line (lines[i +1 ], line_lens[i +1 ])) +pick_snippet_line (pick, i +1 , &picked_count); } -/* Pass 2: Fill remaining quota from the end backwards */ +/* +* Pass 2: Fill remaining quota from the end backwards, skipping blank +* and stack trace lines so trailing test progress is shown instead of +* outermost stack frames like main/_start. +*/ i = num_lines;while (i >0 && picked_count < failure_snippet_lines) { i--; -if (!pick[i]) { - pick[i] = true; - picked_count++; - } +if (!is_ignorable_snippet_line (lines[i], line_lens[i])) +pick_snippet_line (pick, i, &picked_count); + } + +/* Pass 3: Fallback to top stack frames if quota remains */ +for (i =0 ; i < num_lines && picked_count < failure_snippet_lines; i++) { +if (!is_blank_line (lines[i], line_lens[i])) +pick_snippet_line (pick, i, &picked_count); }for (i =0 ; i < num_lines; i++) {if (!pick[i])continue ; -if (last_printed !=-1 && (int )i > last_printed +1 ) { +if (last_printed !=-1 && +has_non_blank_between (lines, line_lens, last_printed, i)) {if (perf_use_color_default)fprintf (fp,"%s...%s\n" ,PERF_COLOR_BLUE ,PERF_COLOR_RESET );else
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
[email protected]
AI reviewRe: [PATCH v4 3/3] perf test: Avoid stack trace noise in verbose=1 failure snippets
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 25 lines
When a test fails with a signal or times out,
child_test_sig_handler()appends '---- unexpected signal (<sig>) ----' followed by a multi-line
stack trace (' #0 ...' through '_start').In verbose=1 mode,
print_test_failure_snippet()selects up tofailure_snippet_lines(10) lines:
- Pass 1 selects lines matching failure keywords plus their immediate
next line (' #0 ... inchild_test_sig_handler').
- Pass 2 fills the remaining quota by scanning backwards from the end of the output buffer, which consumes the remaining lines on outermost stack frames ('handle_internal_command', 'run_argv', 'main', '__libc_start_main', '_start') and hides the trailing test output showing where the test had run to.Improve
print_test_failure_snippet()by:
- Skipping blank lines and stack trace lines (' #<digit>...') when
selecting context in Pass 1 and when scanning backwards from the end in Pass 2, with a Pass 3 fallback from the top of the buffer if quota still remains.
- Prioritizing the non-blank line immediately preceding '---- unexpected signal' so there is no '...' gap before the signal message.
- Only printing '...' between selected lines when at least one non-blank line was omitted between them.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH v4 0/3] perf test: Per-test timeouts, JUnit XML compliance, and snippet improvements
In reply to Ian RogersQuoted 49 lines · Ian Rogers
This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and refines the verbose=1 failure snippet output when tests fail with a signal or time out:
1. Add a per-test timeout (default 60s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in
finish_test()andfinish_tests_parallel()check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.
3. Improve verbose=1 failure snippet selection in
print_test_failure_snippet()to skip blank and stack trace lines when selecting context and scanning backwards from the end of the log, and prioritize the test output line immediately preceding '---- unexpected signal' so trailing test progress is shown instead of outermost stack frames.v4:
- Drop theITIMER_REALtimer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops infinish_test()andfinish_tests_parallel()to detect timeouts.
- Only fall back tokill(pid, sig)whenkill(-pid, sig)fails, and ignore SIGTERM/SIGINT while waiting for subprocesses inrun_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.v2:
- Reset SIGALRM toSIG_DFLin the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor outtimespec_diff_secs()andTEST_KILL_GRACE_SECS.
Quoted 4 lines
Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Avoid stack trace noise in verbose=1 failure snippets
Thanks,
Namhyung
Quoted 9 lines
tools/perf/Documentation/perf-test.txt | 12 +- tools/perf/tests/builtin-test.c | 668 +++++++++++++++++++------ tools/perf/tests/tests.h | 7 +- 3 files changed, 543 insertions(+), 144 deletions(-)Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
Re: [PATCH v4 0/3] perf test: Per-test timeouts, JUnit XML compliance, and snippet improvements
Quoted 4 lines · Namhyung Kim
Quoted 3 lines · Ian Rogers
Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness
Both are failing to apply to perf-tools-next, I'm pushing what I have (with 3/3) to tmp.perf-tools-next now.
Quoted 1 line
Quoted 1 line
perf test: Avoid stack trace noise in verbose=1 failure snippets
Applied.
Quoted 1 line
Acked-by Namhyung Kim <[email protected]>
Thanks,
- Arnaldo
Ian Rogers
reviewerRe: [PATCH v4 0/3] perf test: Per-test timeouts, JUnit XML compliance, and snippet improvements
Quoted 7 lines · Arnaldo Carvalho de Melo
Quoted 4 lines · Namhyung Kim
Quoted 3 lines · Ian Rogers
Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness
Quoted 8 lines
Both are failing to apply to perf-tools-next, I'm pushing what I have (with 3/3) to tmp.perf-tools-next now.
Quoted 1 line
Quoted 1 line
perf test: Avoid stack trace noise in verbose=1 failure snippets
Applied.
Quoted 1 line
Acked-by Namhyung Kim <[email protected]>
Thanks Namhyung and Arnaldo! I'll rebase and send a v5 series.
Ian
Ian Rogers
reviewerv5 cover perf test: Per-test timeouts and JUnit XML schema compliance
In reply to Ian RogersThis series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and ensures time spent waiting for the shared mlock budget does not count towards a test's timeout or reported duration:
1. Add a per-test timeout (default 120s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.
2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.
3. Record each mlock-serialized test's start time in the shared mlock_budget mapping when it acquires mlock_budget->lock in run_test_case_mlocked(), so tests waiting for the mlock budget do not time out while waiting in line and their reported elapsed time reflects actual execution time.
v5:
- Rebase onto tmp.perf-tools-next (patch 3 of v4 was applied upstream) and add Namhyung's Acked-by to patches 1 and 2.
- Add patch 3 so tests waiting on mlock_budget->lock don't time out while waiting.
- Increase the default per-test timeout from 60s to 120s. When running the full suite as root (where CAP_IPC_LOCK lifts the mlock limit and mlock_budget is unused), all ~180 non-exclusive tests start simultaneously at t=0 and contend heavily on CPUs, /proc scanning, and vmlinux/DWARF symbol resolution. On a 28-CPU hybrid machine, a 60s timeout resulted in 33 timeouts across the parallel burst and large sequential suites; increasing the default to 120s allowed longer tests ('kernel lock contention analysis test' at 115.3s, 'task-analyzer python test' at 94.7s, 'perf data type profiling tests' at 92.6s, 'perf metrics value validation' at 91.2s, 'perf record tests' at 79.0s, and 'python profiling with jitdump' at 62.7s) to complete, reducing timeouts from 33 to 2 ('perf record LBR tests' and 'perftool-testsuite_report') and overall suite wall-clock time from 503.5s to 363.9s.
v4:
- Drop the ITIMER_REAL timer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops in finish_test() and finish_tests_parallel() to detect timeouts.
- Only fall back to kill(pid, sig) when kill(-pid, sig) fails, and ignore SIGTERM/SIGINT while waiting for subprocesses in run_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.
v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.
v2:
- Reset SIGALRM to SIG_DFL in the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.
Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Don't count mlock budget wait time towards test timeout
tools/perf/Documentation/perf-test.txt | 12 +- tools/perf/tests/builtin-test.c | 645 +++++++++++++++++++------ tools/perf/tests/tests.h | 7 +- 3 files changed, 523 insertions(+), 141 deletions(-)
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
Ian Rogers
reviewerPATCH v5 1/3 perf test: Add per-test timeout support
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new TEST_TIMEOUT result category and a per-test timeout (default 120 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.
To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes against test_timeout in the existing finish_test() and finish_tests_parallel() polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 7 +- tools/perf/tests/builtin-test.c | 277 ++++++++++++++++++++++--- tools/perf/tests/tests.h | 7 +- 3 files changed, 259 insertions(+), 32 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
@@ -75,10 +75,15 @@ OPTIONS Run each test the given number of times, by default once. This option can be useful to determine if a test is flaky. +-t:: +--timeout:: + Timeout for each forked test in seconds, by default 120 seconds. Specify + 0 to disable the timeout. Ignored when -F/--dont-fork is used. + -F:: --dont-fork:: Do not fork child for each test, run all tests within single process, this - sets sequential mode. + sets sequential mode and disables the per-test timeout. --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +249 -28
482 lines not on this page · raw message
tools/perf/tests/tests.h +4 -3
@@ -6,9 +6,10 @@#include "util/debug.h" enum { -TEST_OK =0 , -TEST_FAIL =-1 , -TEST_SKIP =-2 , +TEST_OK =0 , +TEST_FAIL =-1 , +TEST_SKIP =-2 , +TEST_TIMEOUT =-3 , };#define TEST_ASSERT_VAL (text, cond) \
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf test: pipe drained before kill_timed_out_group() kills subprocesses --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf test: pipe drained before kill_timed_out_group() kills subprocesses --
commit 9387c80164e94087cba7cfddba4fff510a8fabe3
Author: Ian Rogers <[email protected]>
perf test: Add per-test timeout support
This commit adds per-test timeout support to perf test, defaulting to 120 seconds and configurable via -t/--timeout. Forked test children are placed in their own process groups so timed-out tests and their subprocesses can be terminated and reported as timeouts in the test summary and JUnit output.
Quoted 4 lines
tools/perf/tests/builtin-test.c +0
-0
[ ... ]
Quoted 4 lines
tools/perf/tests/builtin-test.c +0
-0@@ -815,6 +893,106 @@ static void print_test_failure_snippet(FILE *fp, const char *buf)free (line_lens); }
[ ... ]
Quoted 9 lines
tools/perf/tests/builtin-test.c +9
-0+static void kill_timed_out_group (struct child_test *child) +{ +if (!child->timed_out || child->process.finished || child->process.pid <=0 ) +return ; + +pr_debug3 ("Sending SIGKILL to remaining processes of timed out test %d.%d (pgid %d)\n" , + child->suite_num +1 , child->test_case_num +1 , child->process.pid); +kill (-child->process.pid,SIGKILL ); +}
[ ... ]
Quoted 16 lines
tools/perf/tests/builtin-test.c +13
-0+static int reap_child_test (struct child_test *child) +{ +sigset_t oldset; +int ret; + +block_term_signals (&oldset); +kill_timed_out_group (child); + ret =finish_command (&child->process); + child->process.pid =0 ; +sigprocmask (SIG_SETMASK , &oldset,NULL ); +return child->timed_out ?TEST_TIMEOUT : ret; +} +static void finish_test (struct child_test **child_tests,int running_test,int child_test_num,int width) {
[ ... ]
Quoted 18 lines
tools/perf/tests/builtin-test.c +6
-2@@ -904,11 +1088,15 @@ static void finish_test(struct child_test **child_tests, int running_test, int c err_done = false; buf[len] ='\0' ;strbuf_addstr_safe (&err_output, buf); + }else if (pfds[0 ].revents & (POLLHUP |POLLERR |POLLNVAL )) { +close (err); + err =-1 ; + child_test->process.err =-1 ; } } } -if (err_done) - err_done =check_if_command_finished (&child_test->process); +if (err_done || child_test->timed_out) + err_done =test_command_finished (child_test); }/* Drain any remaining data from the pipe. */ if (err >=0 ) {
[ ... ]
Quoted 7 lines
tools/perf/tests/builtin-test.c +1
-2@@ -925,8 +1113,7 @@ static void finish_test(struct child_test **child_tests, int running_test, int cfprintf (debug_file (),PERF_COLOR_DELETE_LINE ); }/* Clean up child process. */ - ret =finish_command (&child_test->process); - child_test->process.pid =0 ; + ret =reap_child_test (child_test);
[ ... ]
Quoted 7 lines
tools/perf/tests/builtin-test.c +1
-1@@ -1157,14 +1350,13 @@ static int finish_tests_parallel(struct child_test **child_tests, size_t num_tesif (!child || child->done)continue ; -if (check_if_command_finished (&child->process)) { +if (test_command_finished (child)) {if (child->process.err >=0 ) {
tools/perf/tests/builtin-test.c static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes · line 1362
Quoted 10 lines
tools/perf/tests/builtin-test.c +1
-2drain_child_process_err (child);close (child->process.err); child->process.err =-1 ; } - child->result =finish_command (&child->process); - child->process.pid =0 ; + child->result =reap_child_test (child);clock_gettime (CLOCK_MONOTONIC , &child->end_time); child->done = true; }
[Severity: Medium]
In tools/perf/tests/builtin-test.c, when running with verbose >= 1 so child stdout and stderr are piped through process.err, both finish_test() and finish_tests_parallel() drain the pipe in an unbounded read loop before calling reap_child_test() -> kill_timed_out_group():
tools/perf/tests/builtin-test.c:finish_test() {
...
if (err_done || child_test->timed_out)
err_done = test_command_finished(child_test);
}
/* Drain any remaining data from the pipe. */
if (err >= 0) {
char buf[512];
ssize_t len;
while ((len = read(err, buf, sizeof(buf) - 1)) > 0) {
buf[len] = '\0';
strbuf_addstr_safe(&err_output, buf);
}
}
...
ret = reap_child_test(child_test);
}And in parallel mode:
tools/perf/tests/builtin-test.c:finish_tests_parallel() {
...
if (test_command_finished(child)) {
if (child->process.err >= 0) {
drain_child_process_err(child);
close(child->process.err);
child->process.err = -1;
}
child->result = reap_child_test(child);
...
}If a timed-out test's main child process exits after the initial termination signal while a subprocess in its process group survives and continuously writes to stdout or stderr, test_command_finished() returns true once the main child becomes a zombie. However, kill_timed_out_group() has not yet been called to terminate the remaining process group members.
Could a noisy surviving subprocess keep feeding the pipe so that the read() loop in finish_test() or drain_child_process_err() never sees -EAGAIN or end-of-file, hanging the test runner and growing err_output without bound before reap_child_test() can call kill_timed_out_group()? Should kill_timed_out_group() be invoked before draining the pipe when a timed-out child has finished?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH v5 2/3 perf test: Improve JUnit XML schema compliance and completeness
In reply to Ian RogersImprove the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 5 + tools/perf/tests/builtin-test.c | 309 +++++++++++++++++-------- 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
@@ -85,6 +85,11 @@ OPTIONS Do not fork child for each test, run all tests within single process, this sets sequential mode and disables the per-test timeout. +-j[<file>]:: +--junit[=<file>]:: + Generate JUnit XML test report to the specified file, or 'test.xml' if + no file is specified. + --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +207 -102
370 lines not on this page · raw message
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
[email protected]
AI reviewRe: [PATCH v5 2/3] perf test: Improve JUnit XML schema compliance and completeness
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints inxml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
Quoted 1 line
Signed-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Ian Rogers
reviewerPATCH v5 3/3 perf test: Don't count mlock budget wait time towards test timeout
In reply to Ian RogersWhen multiple parallel tests need room in kernel.perf_event_mlock_kb, they take turns acquiring the shared mlock_budget->lock mutex in run_test_case_mlocked() after being forked in start_test().
Because child->start_time is recorded in start_test() when the child process is forked, any time a test spends waiting in the mlock_budget->lock queue would otherwise be counted against both its per-test timeout (causing tests later in the queue to time out before they even start) and its JUnit XML testcase elapsed time.
Extend struct mlock_budget with a per-slot shared start_time and started flag populated in run_test_case_mlocked() once the child acquires mlock_budget->lock. In check_child_timeout(), skip tests that are still waiting for mlock_budget->lock and measure the timeout from the lock acquisition timestamp once started. Also use the post-lock start_time when computing the test's elapsed duration.
--- tools/perf/tests/builtin-test.c | 81 +++++++++++++++++++++++++-------- 1 file changed, 63 insertions(+), 18 deletions(-)
tools/perf/tests/builtin-test.c +63 -18
@@ -79,8 +79,17 @@ static unsigned int test_timeout = 120;* processes and robust so that a test killed while holding it doesn't leave * the budget locked, see mutex_init_pshared_robust(). */ +struct mlock_child_time { +struct timespec start_time; +bool started; +}; +struct mlock_budget {struct mutex lock; +size_t size; +size_t nr_slots; +size_t next_slot; +struct mlock_child_time slots[]; };static struct mlock_budget *mlock_budget;
@@ -384,6 +393,7 @@ struct child_test {struct timespec start_time;struct timespec end_time;struct timespec timeout_time; +struct mlock_child_time *mlock_time; };static jmp_buf run_test_jmp_buf;
@@ -414,6 +424,8 @@ static int run_test_case_mlocked(struct child_test *child)int err;mutex_lock (&mlock_budget->lock); +clock_gettime (CLOCK_MONOTONIC , &child->mlock_time->start_time); +__atomic_store_n (&child->mlock_time->started, true, __ATOMIC_RELEASE); err =run_test_case (child);mutex_unlock (&mlock_budget->lock);
@@ -459,7 +471,6 @@ static int run_test_child(struct child_process *process) SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGINT, SIGPIPE, SIGQUIT, SIGSEGV, SIGTERM, }; struct child_test *child = container_of(process, struct child_test, process); -bool needs_mlock = mlock_budget && child->test->needs_mlock;int err, sig;/*
@@ -486,7 +497,7 @@ static int run_test_child(struct child_process *process)pr_debug ("---- start ----\n" );pr_debug ("test child forked, pid %d\n" ,getpid ()); -if (needs_mlock) +if (child->mlock_time) err =run_test_case_mlocked (child);else err =run_test_case (child);
@@ -1027,6 +1038,28 @@ static void kill_child_process(pid_t pid, int sig)kill (pid, sig); } +static bool child_test__started (const struct child_test *child,struct timespec *start_time) +{ +if (child->mlock_time) { +if (!__atomic_load_n (&child->mlock_time->started, __ATOMIC_ACQUIRE)) +return false; + *start_time = child->mlock_time->start_time; +return true; + } + *start_time = child->start_time; +return true; +} + +static double child_test__elapsed_secs (const struct child_test *child, +const struct timespec *end_time) +{ +struct timespec start_time; + +if (!child_test__started (child, &start_time)) + start_time = child->start_time; +return timespec_diff_secs (end_time, &start_time); +} +static void check_child_timeout (struct child_test *child,const struct timespec *now) {pid_t pid;
@@ -1043,7 +1076,12 @@ static void check_child_timeout(struct child_test *child, const struct timespecreturn ;if (!child->timed_out) { -if (timespec_diff_secs (now, &child->start_time) >= test_timeout) { +struct timespec start_time; + +if (!child_test__started (child, &start_time)) +return ; + +if (timespec_diff_secs (now, &start_time) >= test_timeout) { child->timed_out = true; child->timeout_time = *now; pr_debug3("Timeout (%us) for test %d.%d (pid %d), sending SIGTERM\n",
@@ -1258,8 +1296,7 @@ static void finish_test(struct child_test **child_tests, int running_test, int cprint_test_failure_snippet (stderr, err_output.buf);clock_gettime (CLOCK_MONOTONIC , &end_time); - elapsed = (end_time.tv_sec - child_test->start_time.tv_sec) + - (end_time.tv_nsec - child_test->start_time.tv_nsec) /1000000000.0 ; + elapsed =child_test__elapsed_secs (child_test, &end_time);print_test_result (t, curr_suite, curr_test_case, ret, width,/*running=*/ 0 , err_output.buf, elapsed);
@@ -1532,9 +1569,7 @@ static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes (child->result == TEST_FAIL || child->result == TEST_TIMEOUT))print_test_failure_snippet (stderr, child->err_output.buf); - elapsed = (child->end_time.tv_sec - child->start_time.tv_sec) + - (child->end_time.tv_nsec - - child->start_time.tv_nsec) /1000000000.0 ; + elapsed =child_test__elapsed_secs (child, &child->end_time);print_test_result (child->test, child->suite_num, child->test_case_num, child->result, width,0 , child->err_output.buf, elapsed);
@@ -1606,6 +1641,9 @@ static int start_test(struct test_suite *test, int curr_suite, int curr_test_cas (*child)->process.stdout_to_stderr =1 ; } (*child)->process.no_exec_cmd = run_test_child; +if (mlock_budget && test->needs_mlock && !sequential && pass ==1 && + mlock_budget->next_slot < mlock_budget->nr_slots) + (*child)->mlock_time = &mlock_budget->slots[mlock_budget->next_slot++];clock_gettime (CLOCK_MONOTONIC , &(*child)->start_time); err =start_command (&(*child)->process);if (err)
@@ -2199,10 +2237,10 @@ static struct test_suite **build_suites(void)#undef for_each_suite } -static bool perf_test__shares_mlock_budget (struct test_suite **suites,int argc, -const char *argv[],struct intlist *skiplist) +static size_t perf_test__count_mlock_budget (struct test_suite **suites,int argc, +const char *argv[],struct intlist *skiplist) { -int sharing =0 ; +size_t sharing =0 ;/* The selection rules mirrored here are __cmd_test()'s: when they change, change both. */ for (int curr_suite =0 ; suites[curr_suite]; curr_suite++) {
@@ -2224,10 +2262,8 @@ static bool perf_test__shares_mlock_budget(struct test_suite **suites, int argc, curr_suite, argc, argv)) sharing += runs_per_test; } -if (sharing >1 ) -return true; } -return false; +return sharing; }/*
@@ -2250,6 +2286,8 @@ static bool perf_test__mlock_budget_exhaustible(void)static void perf_test__setup_mlock_budget (struct test_suite **suites,int argc,const char *argv[],struct intlist *skiplist) { +size_t sharing, size; +/* Nothing can be denied the budget, nothing to take turns for. */ if (!perf_test__mlock_budget_exhaustible ()) {pr_debug ("Not taking turns using the perf mlock budget: nothing can exhaust it\n" );
@@ -2257,19 +2295,26 @@ static void perf_test__setup_mlock_budget(struct test_suite **suites, int argc, }/* Nothing runs at the same time, nothing to share the budget with. */ -if (sequential || !perf_test__shares_mlock_budget (suites, argc, argv, skiplist)) +if (sequential) +return ; + + sharing =perf_test__count_mlock_budget (suites, argc, argv, skiplist); +if (sharing <=1 )return ; - mlock_budget =mmap (NULL ,sizeof (*mlock_budget),PROT_READ |PROT_WRITE , + size =sizeof (*mlock_budget) + sharing *sizeof (mlock_budget->slots[0 ]); + mlock_budget =mmap (NULL , size,PROT_READ |PROT_WRITE ,MAP_SHARED |MAP_ANONYMOUS ,/*fd=*/ -1 ,/*offset=*/ 0 );if (mlock_budget ==MAP_FAILED ) { mlock_budget =NULL ;pr_err ("Failed to map the shared area used to take turns using the perf mlock budget\n" );return ; } + mlock_budget->size = size; + mlock_budget->nr_slots = sharing;if (!mutex_init_pshared_robust (&mlock_budget->lock)) { -munmap (mlock_budget,sizeof (*mlock_budget)); +munmap (mlock_budget, size); mlock_budget =NULL ;pr_debug ("Not taking turns using the perf mlock budget\n" ); }
@@ -2281,7 +2326,7 @@ static void perf_test__teardown_mlock_budget(void)return ;mutex_destroy (&mlock_budget->lock); -munmap (mlock_budget,sizeof (*mlock_budget)); +munmap (mlock_budget, mlock_budget->size); mlock_budget =NULL ; }
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
[email protected]
AI reviewRe: [PATCH v5 3/3] perf test: Don't count mlock budget wait time towards test timeout
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
When multiple parallel tests need room in kernel.perf_event_mlock_kb, they take turns acquiring the shared
mlock_budget->lockmutex inrun_test_case_mlocked()after being forked instart_test().Because
child->start_timeis recorded instart_test()when the child process is forked, any time a test spends waiting in themlock_budget->lockqueue would otherwise be counted against both its per-test timeout (causing tests later in the queue to time out before they even start) and its JUnit XML testcase elapsed time.
Quoted 8 lines
Extend struct
mlock_budgetwith a per-slot sharedstart_timeand started flag populated inrun_test_case_mlocked()once the child acquiresmlock_budget->lock. Incheck_child_timeout(), skip tests that are still waiting formlock_budget->lockand measure the timeout from the lock acquisition timestamp once started. Also use the post-lockstart_timewhen computing the test's elapsed duration.Signed-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Ian Rogers
reviewerv6 cover perf test: Per-test timeouts and JUnit XML schema compliance
In reply to Ian RogersThis series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and ensures time spent waiting for the shared mlock budget does not count towards a test's timeout or reported duration:
1. Add a per-test timeout (default 120s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in finish_test() and finish_tests_parallel() check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.
2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.
3. Record each mlock-serialized test's start time in the shared mlock_budget mapping when it acquires mlock_budget->lock in run_test_case_mlocked(), so tests waiting for the mlock budget do not time out while waiting in line and their reported elapsed time reflects actual execution time.
v6:
- Reap finished child tests (invoking kill_timed_out_group()) before draining process.err in finish_test() and finish_tests_parallel(), and skip draining in handle_child_pipe_activity() once a timed out test has finished, so a noisy surviving subprocess cannot block the pipe drain loop before the process group is killed.
v5:
- Rebase onto tmp.perf-tools-next (patch 3 of v4 was applied upstream) and add Namhyung's Acked-by to patches 1 and 2.
- Add patch 3 so tests waiting on mlock_budget->lock don't time out while waiting.
- Increase the default per-test timeout from 60s to 120s. When running the full suite as root (where CAP_IPC_LOCK lifts the mlock limit and mlock_budget is unused), all ~180 non-exclusive tests start simultaneously at t=0 and contend heavily on CPUs, /proc scanning, and vmlinux/DWARF symbol resolution. On a 28-CPU hybrid machine, a 60s timeout resulted in 33 timeouts across the parallel burst and large sequential suites; increasing the default to 120s allowed longer tests ('kernel lock contention analysis test' at 115.3s, 'task-analyzer python test' at 94.7s, 'perf data type profiling tests' at 92.6s, 'perf metrics value validation' at 91.2s, 'perf record tests' at 79.0s, and 'python profiling with jitdump' at 62.7s) to complete, reducing timeouts from 33 to 2 ('perf record LBR tests' and 'perftool-testsuite_report') and overall suite wall-clock time from 503.5s to 363.9s.
v4:
- Drop the ITIMER_REAL timer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops in finish_test() and finish_tests_parallel() to detect timeouts.
- Only fall back to kill(pid, sig) when kill(-pid, sig) fails, and ignore SIGTERM/SIGINT while waiting for subprocesses in run_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.
v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.
v2:
- Reset SIGALRM to SIG_DFL in the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor out timespec_diff_secs() and TEST_KILL_GRACE_SECS.
Ian Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Don't count mlock budget wait time towards test timeout
tools/perf/Documentation/perf-test.txt | 12 +- tools/perf/tests/builtin-test.c | 662 +++++++++++++++++++------ tools/perf/tests/tests.h | 7 +- 3 files changed, 535 insertions(+), 146 deletions(-)
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
Ian Rogers
reviewerPATCH v6 1/3 perf test: Add per-test timeout support
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new TEST_TIMEOUT result category and a per-test timeout (default 120 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.
To enforce the timeout:
- Place each forked test child into its own process group via setpgid(0, 0) in the child and setpgid(pid, pid) in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes against test_timeout in the existing finish_test() and finish_tests_parallel() polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 7 +- tools/perf/tests/builtin-test.c | 294 ++++++++++++++++++++++--- tools/perf/tests/tests.h | 7 +- 3 files changed, 271 insertions(+), 37 deletions(-)
tools/perf/Documentation/perf-test.txt +6 -1
@@ -75,10 +75,15 @@ OPTIONS Run each test the given number of times, by default once. This option can be useful to determine if a test is flaky. +-t:: +--timeout:: + Timeout for each forked test in seconds, by default 120 seconds. Specify + 0 to disable the timeout. Ignored when -F/--dont-fork is used. + -F:: --dont-fork:: Do not fork child for each test, run all tests within single process, this - sets sequential mode. + sets sequential mode and disables the per-test timeout. --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +261 -33
506 lines not on this page · raw message
tools/perf/tests/tests.h +4 -3
@@ -6,9 +6,10 @@#include "util/debug.h" enum { -TEST_OK =0 , -TEST_FAIL =-1 , -TEST_SKIP =-2 , +TEST_OK =0 , +TEST_FAIL =-1 , +TEST_SKIP =-2 , +TEST_TIMEOUT =-3 , };#define TEST_ASSERT_VAL (text, cond) \
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
[email protected]
AI reviewSashiko has reviewed this patch and found no issues.
Quoted 21 lines
When a test hangs in 'perf test', the entire test run hangs and makes it difficult to inspect output from other tests.
Add a new
TEST_TIMEOUTresult category and a per-test timeout (default 120 seconds, configurable via '-t'/'--timeout', with 0 to disable). When '-F' ('--dont-fork') is specified, timeouts are disabled.To enforce the timeout:
- Place each forked test child into its own process group viasetpgid(0, 0)in the child andsetpgid(pid, pid)in the parent so any subprocesses spawned by shell or workload tests can be terminated together.
- Check elapsed child runtimes againsttest_timeoutin the existingfinish_test()andfinish_tests_parallel()polling loops, sending SIGTERM first and escalating to SIGKILL after 1 second if the child process group has not exited.
- Report timed out tests in red ('TIMEOUT!'), track and display them in the test summary, and emit '<error message="TIMEOUT" type="timeout">' counted under the 'errors' attribute in JUnit XML output.
- Document '-t'/'--timeout' and '-F'/'--dont-fork' in perf-test.txt.
Quoted 1 line
Signed-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH v6 2/3 perf test: Improve JUnit XML schema compliance and completeness
In reply to Ian RogersImprove the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints in xml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
--- tools/perf/Documentation/perf-test.txt | 5 + tools/perf/tests/builtin-test.c | 309 +++++++++++++++++-------- 2 files changed, 212 insertions(+), 102 deletions(-)
tools/perf/Documentation/perf-test.txt +5 -0
@@ -85,6 +85,11 @@ OPTIONS Do not fork child for each test, run all tests within single process, this sets sequential mode and disables the per-test timeout. +-j[<file>]:: +--junit[=<file>]:: + Generate JUnit XML test report to the specified file, or 'test.xml' if + no file is specified. + --dso:: Specify a DSO for the "Symbols" test.
tools/perf/tests/builtin-test.c +207 -102
370 lines not on this page · raw message
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
[email protected]
AI reviewRe: [PATCH v6 2/3] perf test: Improve JUnit XML schema compliance and completeness
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
Improve the JUnit XML output generated by 'perf test -j' so that it validates cleanly against both the strict Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd):
- Add the required 'package', 'id', 'timestamp' (ISO 8601 UTC), 'hostname', and 'time' attributes to '<testsuite>'.
- Emit '<properties>' (populated with uname sysname, release, version, and machine when available) at the start of '<testsuite>' and '<system-out/>' and '<system-err/>' at the end, matching the element sequence required by JUnit.xsd.
- Add the required 'type="failure"' attribute to '<failure>' elements and emit self-closing tags for passing '<testcase>' elements and '<failure>'/'<error>' elements with empty error output.
- Escape apostrophes (''') and validate UTF-8 multi-byte sequences against valid XML 1.0 codepoints inxml_escape().
- Use the actual suite description and '%.2f' time format for user override skipped tests ('-s'/'--skip').
- Document '-j[<file>]'/'--junit[=<file>]' in perf-test.txt.
Quoted 1 line
Signed-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Ian Rogers
reviewerPATCH v6 3/3 perf test: Don't count mlock budget wait time towards test timeout
In reply to Ian RogersWhen multiple parallel tests need room in kernel.perf_event_mlock_kb, they take turns acquiring the shared mlock_budget->lock mutex in run_test_case_mlocked() after being forked in start_test().
Because child->start_time is recorded in start_test() when the child process is forked, any time a test spends waiting in the mlock_budget->lock queue would otherwise be counted against both its per-test timeout (causing tests later in the queue to time out before they even start) and its JUnit XML testcase elapsed time.
Extend struct mlock_budget with a per-slot shared start_time and started flag populated in run_test_case_mlocked() once the child acquires mlock_budget->lock. In check_child_timeout(), skip tests that are still waiting for mlock_budget->lock and measure the timeout from the lock acquisition timestamp once started. Also use the post-lock start_time when computing the test's elapsed duration.
--- tools/perf/tests/builtin-test.c | 81 +++++++++++++++++++++++++-------- 1 file changed, 63 insertions(+), 18 deletions(-)
tools/perf/tests/builtin-test.c +63 -18
@@ -79,8 +79,17 @@ static unsigned int test_timeout = 120;* processes and robust so that a test killed while holding it doesn't leave * the budget locked, see mutex_init_pshared_robust(). */ +struct mlock_child_time { +struct timespec start_time; +bool started; +}; +struct mlock_budget {struct mutex lock; +size_t size; +size_t nr_slots; +size_t next_slot; +struct mlock_child_time slots[]; };static struct mlock_budget *mlock_budget;
@@ -384,6 +393,7 @@ struct child_test {struct timespec start_time;struct timespec end_time;struct timespec timeout_time; +struct mlock_child_time *mlock_time; };static jmp_buf run_test_jmp_buf;
@@ -414,6 +424,8 @@ static int run_test_case_mlocked(struct child_test *child)int err;mutex_lock (&mlock_budget->lock); +clock_gettime (CLOCK_MONOTONIC , &child->mlock_time->start_time); +__atomic_store_n (&child->mlock_time->started, true, __ATOMIC_RELEASE); err =run_test_case (child);mutex_unlock (&mlock_budget->lock);
@@ -459,7 +471,6 @@ static int run_test_child(struct child_process *process) SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGINT, SIGPIPE, SIGQUIT, SIGSEGV, SIGTERM, }; struct child_test *child = container_of(process, struct child_test, process); -bool needs_mlock = mlock_budget && child->test->needs_mlock;int err, sig;/*
@@ -486,7 +497,7 @@ static int run_test_child(struct child_process *process)pr_debug ("---- start ----\n" );pr_debug ("test child forked, pid %d\n" ,getpid ()); -if (needs_mlock) +if (child->mlock_time) err =run_test_case_mlocked (child);else err =run_test_case (child);
@@ -1027,6 +1038,28 @@ static void kill_child_process(pid_t pid, int sig)kill (pid, sig); } +static bool child_test__started (const struct child_test *child,struct timespec *start_time) +{ +if (child->mlock_time) { +if (!__atomic_load_n (&child->mlock_time->started, __ATOMIC_ACQUIRE)) +return false; + *start_time = child->mlock_time->start_time; +return true; + } + *start_time = child->start_time; +return true; +} + +static double child_test__elapsed_secs (const struct child_test *child, +const struct timespec *end_time) +{ +struct timespec start_time; + +if (!child_test__started (child, &start_time)) + start_time = child->start_time; +return timespec_diff_secs (end_time, &start_time); +} +static void check_child_timeout (struct child_test *child,const struct timespec *now) {pid_t pid;
@@ -1043,7 +1076,12 @@ static void check_child_timeout(struct child_test *child, const struct timespecreturn ;if (!child->timed_out) { -if (timespec_diff_secs (now, &child->start_time) >= test_timeout) { +struct timespec start_time; + +if (!child_test__started (child, &start_time)) +return ; + +if (timespec_diff_secs (now, &start_time) >= test_timeout) { child->timed_out = true; child->timeout_time = *now; pr_debug3("Timeout (%us) for test %d.%d (pid %d), sending SIGTERM\n",
@@ -1262,8 +1300,7 @@ static void finish_test(struct child_test **child_tests, int running_test, int cprint_test_failure_snippet (stderr, err_output.buf);clock_gettime (CLOCK_MONOTONIC , &end_time); - elapsed = (end_time.tv_sec - child_test->start_time.tv_sec) + - (end_time.tv_nsec - child_test->start_time.tv_nsec) /1000000000.0 ; + elapsed =child_test__elapsed_secs (child_test, &end_time);print_test_result (t, curr_suite, curr_test_case, ret, width,/*running=*/ 0 , err_output.buf, elapsed);
@@ -1539,9 +1576,7 @@ static int finish_tests_parallel(struct child_test **child_tests, size_t num_tes (child->result == TEST_FAIL || child->result == TEST_TIMEOUT))print_test_failure_snippet (stderr, child->err_output.buf); - elapsed = (child->end_time.tv_sec - child->start_time.tv_sec) + - (child->end_time.tv_nsec - - child->start_time.tv_nsec) /1000000000.0 ; + elapsed =child_test__elapsed_secs (child, &child->end_time);print_test_result (child->test, child->suite_num, child->test_case_num, child->result, width,0 , child->err_output.buf, elapsed);
@@ -1613,6 +1648,9 @@ static int start_test(struct test_suite *test, int curr_suite, int curr_test_cas (*child)->process.stdout_to_stderr =1 ; } (*child)->process.no_exec_cmd = run_test_child; +if (mlock_budget && test->needs_mlock && !sequential && pass ==1 && + mlock_budget->next_slot < mlock_budget->nr_slots) + (*child)->mlock_time = &mlock_budget->slots[mlock_budget->next_slot++];clock_gettime (CLOCK_MONOTONIC , &(*child)->start_time); err =start_command (&(*child)->process);if (err)
@@ -2206,10 +2244,10 @@ static struct test_suite **build_suites(void)#undef for_each_suite } -static bool perf_test__shares_mlock_budget (struct test_suite **suites,int argc, -const char *argv[],struct intlist *skiplist) +static size_t perf_test__count_mlock_budget (struct test_suite **suites,int argc, +const char *argv[],struct intlist *skiplist) { -int sharing =0 ; +size_t sharing =0 ;/* The selection rules mirrored here are __cmd_test()'s: when they change, change both. */ for (int curr_suite =0 ; suites[curr_suite]; curr_suite++) {
@@ -2231,10 +2269,8 @@ static bool perf_test__shares_mlock_budget(struct test_suite **suites, int argc, curr_suite, argc, argv)) sharing += runs_per_test; } -if (sharing >1 ) -return true; } -return false; +return sharing; }/*
@@ -2257,6 +2293,8 @@ static bool perf_test__mlock_budget_exhaustible(void)static void perf_test__setup_mlock_budget (struct test_suite **suites,int argc,const char *argv[],struct intlist *skiplist) { +size_t sharing, size; +/* Nothing can be denied the budget, nothing to take turns for. */ if (!perf_test__mlock_budget_exhaustible ()) {pr_debug ("Not taking turns using the perf mlock budget: nothing can exhaust it\n" );
@@ -2264,19 +2302,26 @@ static void perf_test__setup_mlock_budget(struct test_suite **suites, int argc, }/* Nothing runs at the same time, nothing to share the budget with. */ -if (sequential || !perf_test__shares_mlock_budget (suites, argc, argv, skiplist)) +if (sequential) +return ; + + sharing =perf_test__count_mlock_budget (suites, argc, argv, skiplist); +if (sharing <=1 )return ; - mlock_budget =mmap (NULL ,sizeof (*mlock_budget),PROT_READ |PROT_WRITE , + size =sizeof (*mlock_budget) + sharing *sizeof (mlock_budget->slots[0 ]); + mlock_budget =mmap (NULL , size,PROT_READ |PROT_WRITE ,MAP_SHARED |MAP_ANONYMOUS ,/*fd=*/ -1 ,/*offset=*/ 0 );if (mlock_budget ==MAP_FAILED ) { mlock_budget =NULL ;pr_err ("Failed to map the shared area used to take turns using the perf mlock budget\n" );return ; } + mlock_budget->size = size; + mlock_budget->nr_slots = sharing;if (!mutex_init_pshared_robust (&mlock_budget->lock)) { -munmap (mlock_budget,sizeof (*mlock_budget)); +munmap (mlock_budget, size); mlock_budget =NULL ;pr_debug ("Not taking turns using the perf mlock budget\n" ); }
@@ -2288,7 +2333,7 @@ static void perf_test__teardown_mlock_budget(void)return ;mutex_destroy (&mlock_budget->lock); -munmap (mlock_budget,sizeof (*mlock_budget)); +munmap (mlock_budget, mlock_budget->size); mlock_budget =NULL ; }
Signature · 1 line
2.56.0.307.g5fe7bd680c-goog
[email protected]
AI reviewRe: [PATCH v6 3/3] perf test: Don't count mlock budget wait time towards test timeout
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
When multiple parallel tests need room in kernel.perf_event_mlock_kb, they take turns acquiring the shared
mlock_budget->lockmutex inrun_test_case_mlocked()after being forked instart_test().Because
child->start_timeis recorded instart_test()when the child process is forked, any time a test spends waiting in themlock_budget->lockqueue would otherwise be counted against both its per-test timeout (causing tests later in the queue to time out before they even start) and its JUnit XML testcase elapsed time.
Quoted 8 lines
Extend struct
mlock_budgetwith a per-slot sharedstart_timeand started flag populated inrun_test_case_mlocked()once the child acquiresmlock_budget->lock. Incheck_child_timeout(), skip tests that are still waiting formlock_budget->lockand measure the timeout from the lock acquisition timestamp once started. Also use the post-lockstart_timewhen computing the test's elapsed duration.Signed-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Ian Rogers
reviewerRe: [PATCH v6 0/3] perf test: Per-test timeouts and JUnit XML schema compliance
In reply to Ian RogersQuoted 76 lines · Ian Rogers
This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and ensures time spent waiting for the shared mlock budget does not count towards a test's timeout or reported duration:
1. Add a per-test timeout (default 120s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in
finish_test()andfinish_tests_parallel()check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.2. Update 'perf test -j' output to validate cleanly against both the Apache Ant/Windy Road JUnit schema (JUnit.xsd) and the Jenkins xUnit/Maven Surefire schema (junit-10.xsd), including required '<testsuite>' attributes, '<properties>' from uname, '<system-out/>' and '<system-err/>', self-closing tags, apostrophe escaping, and UTF-8/XML 1.0 codepoint validation.
3. Record each mlock-serialized test's start time in the shared
mlock_budgetmapping when it acquiresmlock_budget->lockinrun_test_case_mlocked(), so tests waiting for the mlock budget do not time out while waiting in line and their reported elapsed time reflects actual execution time.v6:
- Reap finished child tests (invokingkill_timed_out_group()) before draining process.err infinish_test()andfinish_tests_parallel(), and skip draining inhandle_child_pipe_activity()once a timed out test has finished, so a noisy surviving subprocess cannot block the pipe drain loop before the process group is killed.v5:
- Rebase onto tmp.perf-tools-next (patch 3 of v4 was applied upstream) and add Namhyung's Acked-by to patches 1 and 2.
- Add patch 3 so tests waiting onmlock_budget->lockdon't time out while waiting.
- Increase the default per-test timeout from 60s to 120s. When running the full suite as root (whereCAP_IPC_LOCKlifts the mlock limit andmlock_budgetis unused), all ~180 non-exclusive tests start simultaneously at t=0 and contend heavily on CPUs, /proc scanning, and vmlinux/DWARF symbol resolution. On a 28-CPU hybrid machine, a 60s timeout resulted in 33 timeouts across the parallel burst and large sequential suites; increasing the default to 120s allowed longer tests ('kernel lock contention analysis test' at 115.3s, 'task-analyzer python test' at 94.7s, 'perf data type profiling tests' at 92.6s, 'perf metrics value validation' at 91.2s, 'perf record tests' at 79.0s, and 'python profiling with jitdump' at 62.7s) to complete, reducing timeouts from 33 to 2 ('perf record LBR tests' and 'perftool-testsuite_report') and overall suite wall-clock time from 503.5s to 363.9s.v4:
- Drop theITIMER_REALtimer and SIGALRM handler in patch 1, relying on the existing 10ms/100ms polling loops infinish_test()andfinish_tests_parallel()to detect timeouts.
- Only fall back tokill(pid, sig)whenkill(-pid, sig)fails, and ignore SIGTERM/SIGINT while waiting for subprocesses inrun_test_child().
- Add patch 3 to avoid stack trace noise in verbose=1 failure snippets.v3:
- Block SIGINT/SIGTERM while a test process may be reaped and until its pid is cleared, and skip already reaped tests in the signal path.v2:
- Reset SIGALRM toSIG_DFLin the forked test child.
- Replace the blocking waitpid(-1) loop with a bounded, non-blocking wait that only runs when interrupted by SIGTERM/SIGINT.
- Ensure the SIGKILL escalation reaches the whole process group before the zombie is reaped.
- Factor outtimespec_diff_secs()andTEST_KILL_GRACE_SECS.
Quoted 6 lines
Assisted-by Antigravity:gemini-3.1-proIan Rogers (3):
perf test: Add per-test timeout support
perf test: Improve JUnit XML schema compliance and completeness perf test: Don't count mlock budget wait time towards test timeout
Ping.
The sashiko reviews are clean:
https://sashiko.dev/#/patchset/[email protected]
There's an additional patch for making the mlock waiting time not count as timeout time:
https://lore.kernel.org/linux-perf-users/[email protected]/
that is missing any tags.
Thanks,
Ian
Quoted 8 lines
tools/perf/Documentation/perf-test.txt | 12 +- tools/perf/tests/builtin-test.c | 662 +++++++++++++++++++------ tools/perf/tests/tests.h | 7 +- 3 files changed, 535 insertions(+), 146 deletions(-)--
2.56.0.307.g5fe7bd680c-goog
Re: [PATCH v6 0/3] perf test: Per-test timeouts and JUnit XML schema compliance
In reply to Ian RogersQuoted 14 lines · Ian Rogers
This series adds per-test timeout support to 'perf test', improves JUnit XML output to validate against strict JUnit schemas, and ensures time spent waiting for the shared mlock budget does not count towards a test's timeout or reported duration:
1. Add a per-test timeout (default 120s, configurable via '-t'/'--timeout', 0 to disable, disabled with '-F'/'--dont-fork'). Forked tests run in their own process groups and the existing polling loops in
finish_test()andfinish_tests_parallel()check elapsed runtimes, sending SIGTERM on timeout and escalating to SIGKILL after 1 second. Timed out tests are reported as 'TIMEOUT!', listed in the summary, and recorded as '<error message="TIMEOUT" type="timeout">' in JUnit XML output.
Quoted 1 line
[...]
Applied to perf-tools-next, thanks!
Best regards
Namhyung