selftests/bpf: fix for bpf_signal stalls, watchdog for test_progs
Subsystems (3)
Eduard Zingerman
maintainercover selftests/bpf: fix for bpf_signal stalls, watchdog for test_progs
Test case 'bpf_signal' had been recently reported to stall, both on the mailing list [1] and CI [2]. The stall is caused by CPU cycles perf event not being delivered within expected time frame, before test process enters system call and waits indefinitely.
This patch-set addresses the issue in several ways:
- A watchdog timer is added to test_progs.c runner:
- it prints current sub-test name to stderr if sub-test takes longer
than 10 seconds to finish;
- it terminates process executing sub-test if sub-test takes longer than 120 seconds to finish.
- The test case is updated to await perf event notification with a timeout and a few retries, this serves two purposes:
- busy loops longer to increase the time frame for CPU cycles event
generation/delivery;
- makes a timeout, not stall, a worst case scenario.
- The test case is updated to lower frequency of perf events, as high frequency of such events caused events generation throttling, which in turn delayed events delivery by amount of time sufficient to cause test case failure.
Note:
librt pthread-based timer API is used to implement watchdog timer. I chose this API over SIGALRM because signal handler execution within test process context was sufficient to trigger perf event delivery for send_signal/send_signal_nmi_thread_remote test case, w/o any additional changes. Thus I concluded that SIGALRM based implementation interferes with tests execution.
[1] https://lore.kernel.org/bpf/CAP01T75OUeE8E-Lw9df84dm8ag2YmHW619f1DmPSVZ5_O89+Bg@mail.gmail.com/
[2] https://github.com/kernel-patches/bpf/actions/runs/11791485271/job/32843996871
Eduard Zingerman (4):
selftests/bpf: watchdog timer for test_progs selftests/bpf: add read_with_timeout() utility function selftests/bpf: allow send_signal test to timeout selftests/bpf: update send_signal to lower perf evemts frequency
tools/testing/selftests/bpf/Makefile | 1 + tools/testing/selftests/bpf/io_helpers.c | 21 ++++ tools/testing/selftests/bpf/io_helpers.h | 7 ++ .../selftests/bpf/prog_tests/bpf_iter.c | 8 +- .../testing/selftests/bpf/prog_tests/iters.c | 4 +- .../selftests/bpf/prog_tests/send_signal.c | 35 +++--- tools/testing/selftests/bpf/test_progs.c | 104 ++++++++++++++++++ tools/testing/selftests/bpf/test_progs.h | 6 + 8 files changed, 166 insertions(+), 20 deletions(-) create mode 100644 tools/testing/selftests/bpf/io_helpers.c create mode 100644 tools/testing/selftests/bpf/io_helpers.h
Signature · 1 line
2.47.0
Eduard Zingerman
maintainerPATCH 1/4 selftests/bpf: watchdog timer for test_progs
This commit provides a watchdog timer that sets a limit of how long a single sub-test could run:
- if sub-test runs for 10 seconds, the name of the test is printed (currently the name of the test is printed only after it finishes);
- if sub-test runs for 120 seconds, the running thread is terminated with SIGSEGV (to trigger crash_handler() and get a stack trace).
Specifically:
- the timer is armed on each call to run_one_test();
- re-armed at each call to test__start_subtest();
- is stopped when exiting run_one_test().
Default timeout could be overridden using '-w' or '--watchdog-timeout' options. Value 0 can be used to turn the timer off.
Here is an example execution:
$ ./ssh-exec.sh ./test_progs -w 5 -t \
send_signal/send_signal_perf_thread_remote,send_signal/send_signal_nmi_thread_remote
WATCHDOG: test case send_signal/send_signal_nmi_thread_remote executes for 5 seconds, terminating with SIGSEGV
Caught signal #11!
Stack trace:
./test_progs(crash_handler+0x1f)[0x9049ef]
/lib64/libc.so.6(+0x40d00)[0x7f1f1184fd00]
/lib64/libc.so.6(read+0x4a)[0x7f1f1191cc4a]
./test_progs[0x720dd3]
./test_progs[0x71ef7a]
./test_progs(test_send_signal+0x1db)[0x71edeb]
./test_progs[0x9066c5]
./test_progs(main+0x5ed)[0x9054ad]
/lib64/libc.so.6(+0x2a088)[0x7f1f11839088]
/lib64/libc.so.6(__libc_start_main+0x8b)[0x7f1f1183914b]
./test_progs(_start+0x25)[0x527385]
#292 send_signal:FAIL
test_send_signal_common:PASS:reading pipe 0 nsec
test_send_signal_common:PASS:reading pipe error: size 0 0 nsec
test_send_signal_common:PASS:incorrect result 0 nsec
test_send_signal_common:PASS:pipe_write 0 nsec
test_send_signal_common:PASS:setpriority 0 nsecTimer is implemented using timer_{create,start} librt API. Internally librt uses pthreads for SIGEV_THREAD timers, so this change adds a background timer thread to the test process. Because of this a few checks in tests 'bpf_iter' and 'iters' need an update to account for an extra thread.
For parallelized scenario the watchdog is also created for each worker fork. If one of the workers gets stuck, it would be terminated by a watchdog. In theory, this might lead to a scenario when all worker threads are exhausted, however this should not be a problem for server_main(), as it would exit with some of the tests not run.
--- .../selftests/bpf/prog_tests/bpf_iter.c | 8 +- .../testing/selftests/bpf/prog_tests/iters.c | 4 +- tools/testing/selftests/bpf/test_progs.c | 104 ++++++++++++++++++ tools/testing/selftests/bpf/test_progs.h | 6 + 4 files changed, 116 insertions(+), 6 deletions(-)
tools/testing/selftests/bpf/prog_tests/bpf_iter.c +4 -4
@@ -265,10 +265,10 @@ static void *run_test_task_tid(void *arg) linfo.task.tid =0 ; linfo.task.pid =getpid (); -/* This includes the parent thread, this thread, +/* This includes the parent thread, this thread, watchdog timer thread * and the do_nothing_wait thread */ -test_task_common (&opts,2 ,1 ); +test_task_common (&opts,3 ,1 );test_task_common_nocheck (NULL , &num_unknown_tid, &num_known_tid);ASSERT_GT (num_unknown_tid,2 ,"check_num_unknown_tid" );
@@ -297,7 +297,7 @@ static void test_task_pid(void) opts.link_info = &linfo; opts.link_info_len =sizeof (linfo); -test_task_common (&opts,1 ,1 ); +test_task_common (&opts,2 ,1 ); }static void test_task_pidfd (void )
@@ -315,7 +315,7 @@ static void test_task_pidfd(void) opts.link_info = &linfo; opts.link_info_len =sizeof (linfo); -test_task_common (&opts,1 ,1 ); +test_task_common (&opts,2 ,1 );close (pidfd); }
tools/testing/selftests/bpf/prog_tests/iters.c +2 -2
@@ -192,8 +192,8 @@ static void subtest_task_iters(void)syscall (SYS_getpgid);iters_task__detach (skel);ASSERT_EQ (skel->bss->procs_cnt,1 ,"procs_cnt" ); -ASSERT_EQ (skel->bss->threads_cnt, thread_num +1 ,"threads_cnt" ); -ASSERT_EQ (skel->bss->proc_threads_cnt, thread_num +1 ,"proc_threads_cnt" ); +ASSERT_EQ (skel->bss->threads_cnt, thread_num +2 ,"threads_cnt" ); +ASSERT_EQ (skel->bss->proc_threads_cnt, thread_num +2 ,"proc_threads_cnt" );ASSERT_EQ (skel->bss->invalid_cnt,0 ,"invalid_cnt" );pthread_mutex_unlock (&do_nothing_mutex);for (int i =0 ; i < thread_num; i++)
tools/testing/selftests/bpf/test_progs.c +104 -0
168 lines not on this page · raw message
tools/testing/selftests/bpf/test_progs.h +6 -0
@@ -131,6 +131,12 @@ struct test_env {pid_t *worker_pids;/* array of worker pids */ int *worker_socks;/* array of worker socks */ int *worker_current_test;/* array of current running test for each worker */ + +pthread_t main_thread; +int secs_till_notify; +int secs_till_kill; +timer_t watchdog;/* watch for stalled tests/subtests */ +enum {WD_NOTIFY ,WD_KILL } watchdog_state; };#define MAX_LOG_TRUNK_SIZE 8192
Signature · 1 line
2.47.0
Eduard Zingerman
maintainerPATCH 2/4 selftests/bpf: add read_with_timeout() utility function
In reply to Eduard Zingermanint read_with_timeout(int fd, char *buf, size_t count, long usec)
As a regular read(2), but allows to specify a timeout in micro-seconds. Returns -EAGAIN on timeout. Implemented using select().
--- tools/testing/selftests/bpf/Makefile | 1 + tools/testing/selftests/bpf/io_helpers.c | 21 +++++++++++++++++++++ tools/testing/selftests/bpf/io_helpers.h | 7 +++++++ 3 files changed, 29 insertions(+) create mode 100644 tools/testing/selftests/bpf/io_helpers.c create mode 100644 tools/testing/selftests/bpf/io_helpers.h
tools/testing/selftests/bpf/Makefile +1 -0
@@ -742,6 +742,7 @@ TRUNNER_EXTRA_SOURCES := test_progs.c \ unpriv_helpers.c \ netlink_helpers.c \ jit_disasm_helpers.c \ + io_helpers.c \ test_loader.c \ xsk.c \ disasm.c \
tools/testing/selftests/bpf/io_helpers.c new file +21 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <sys/select.h> +#include <unistd.h> +#include <errno.h> + +int read_with_timeout (int fd,char *buf,size_t count,long usec) +{ +const long M =1000 *1000 ; +struct timeval tv = { usec /M , usec %M }; +fd_set fds; +int err; + +FD_ZERO (&fds); +FD_SET (fd, &fds); + err =select (fd +1 , &fds,NULL ,NULL , &tv); +if (err <0 ) +return err; +if (FD_ISSET (fd, &fds)) +return read (fd, buf, count); +return -EAGAIN ; +}
tools/testing/selftests/bpf/io_helpers.h new file +7 -0
@@ -0,0 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <unistd.h> + +/* As a regular read(2), but allows to specify a timeout in micro-seconds. +* Returns -EAGAIN on timeout. +*/ +int read_with_timeout (int fd,char *buf,size_t count,long usec);
Signature · 1 line
2.47.0
Eduard Zingerman
maintainerPATCH 3/4 selftests/bpf: allow send_signal test to timeout
In reply to Eduard ZingermanThe following invocation:
$ t1=send_signal/send_signal_perf_thread_remote \
t2=send_signal/send_signal_nmi_thread_remote \
./test_progs -t $t1,$t2Leads to send_signal_nmi_thread_remote to be stuck on a line 180:
/* wait for result */ err = read(pipe_c2p[0], buf, 1);
In this test case:
- perf event PERF_COUNT_HW_CPU_CYCLES is created for parent process;
- BPF program is attached to perf event, and sends a signal to child process when event occurs;
- parent program burns some CPU in busy loop and calls read() to get notification from child that it received a signal.
The perf event is declared with .sample_period = 1.
This forces perf to throttle events, and under some unclear conditions the event does not always occur while parent is in busy loop. After parent enters read() system call CPU cycles event won't be generated for parent anymore. Thus, if perf event had not occurred already the test is stuck.
This commit updates the parent to wait for notification with a timeout, doing several iterations of busy loop + read_with_timeout().
--- .../selftests/bpf/prog_tests/send_signal.c | 32 +++++++++++-------- 1 file changed, 19 insertions(+), 13 deletions(-)
tools/testing/selftests/bpf/prog_tests/send_signal.c +19 -13
@@ -3,6 +3,7 @@#include <sys/time.h> #include <sys/resource.h> #include "test_send_signal_kern.skel.h" +#include "io_helpers.h" static int sigusr1_received;
@@ -24,6 +25,7 @@ static void test_send_signal_common(struct perf_event_attr *attr,int pipe_c2p[2 ], pipe_p2c[2 ];int err =-1 , pmu_fd =-1 ;volatile int j =0 ; +int retry_count;char buf[256 ];pid_t pid;int old_prio;
@@ -163,21 +165,25 @@ static void test_send_signal_common(struct perf_event_attr *attr,/* notify child that bpf program can send_signal now */ ASSERT_EQ (write (pipe_p2c[1 ], buf,1 ),1 ,"pipe_write" ); -/* For the remote test, the BPF program is triggered from this -* process but the other process/thread is signaled. -*/ -if (remote) { -if (!attr) { -for (int i =0 ; i <10 ; i++) -usleep (1 ); - }else { -for (int i =0 ; i <100000000 ; i++) - j /= i +1 ; +for (retry_count =0 ;;) { +/* For the remote test, the BPF program is triggered from this +* process but the other process/thread is signaled. +*/ +if (remote) { +if (!attr) { +for (int i =0 ; i <10 ; i++) +usleep (1 ); + }else { +for (int i =0 ; i <100000000 ; i++) + j /= i +1 ; + } } +/* wait for result */ + err =read_with_timeout (pipe_c2p[0 ], buf,1 ,100 ); +if (err == -EAGAIN && retry_count++ <10000 ) +continue ; +break ; } - -/* wait for result */ - err =read (pipe_c2p[0 ], buf,1 );if (!ASSERT_GE (err,0 ,"reading pipe" )) goto disable_pmu;if (!ASSERT_GT (err,0 ,"reading pipe error: size 0" )) {
Signature · 1 line
2.47.0
Eduard Zingerman
maintainerPATCH 4/4 selftests/bpf: update send_signal to lower perf evemts frequency
In reply to Eduard ZingermanSimilar to commit [1] sample perf events less often in test_send_signal_nmi(). This should reduce perf events throttling.
[1] 7015843afcaf ("selftests/bpf: Fix send_signal test with nested CONFIG_PARAVIRT")
--- tools/testing/selftests/bpf/prog_tests/send_signal.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/send_signal.c +2 -1
@@ -229,7 +229,8 @@ static void test_send_signal_perf(bool signal_thread, bool remote)static void test_send_signal_nmi (bool signal_thread,bool remote) {struct perf_event_attr attr = { - .sample_period =1 , + .freq =1 , + .sample_freq =1000 , .type =PERF_TYPE_HARDWARE , .config =PERF_COUNT_HW_CPU_CYCLES , };
Signature · 1 line
2.47.0
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 14 lines · On Tue, 12 Nov 2024 03:09:02 -0800 you wrote:
Test case '
bpf_signal' had been recently reported to stall, both on the mailing list [1] and CI [2]. The stall is caused by CPU cycles perf event not being delivered within expected time frame, before test process enters system call and waits indefinitely.This patch-set addresses the issue in several ways:
- A watchdog timer is added totest_progs.crunner:
- it prints current sub-test name to stderr if sub-test takes longerthan 10 seconds to finish;- it terminates process executing sub-test if sub-test takes longer than 120 seconds to finish.
- The test case is updated to await perf event notification with a timeout and a few retries, this serves two purposes:
- busy loops longer to increase the time frame for CPU cycles event
Quoted 8 lines
generation/delivery;- makes a timeout, not stall, a worst case scenario.
- The test case is updated to lower frequency of perf events, as high frequency of such events caused events generation throttling, which in turn delayed events delivery by amount of time sufficient to cause test case failure.[...]
Here is the summary with links:
- [bpf-next,1/4] selftests/bpf: watchdog timer for test_progs
https://git.kernel.org/bpf/bpf-next/c/d9d4d127e813 - [bpf-next,2/4] selftests/bpf: add read_with_timeout() utility function https://git.kernel.org/bpf/bpf-next/c/03066ed3105a - [bpf-next,3/4] selftests/bpf: allow send_signal test to timeout https://git.kernel.org/bpf/bpf-next/c/3209139d00e5 - [bpf-next,4/4] selftests/bpf: update send_signal to lower perf evemts frequency https://git.kernel.org/bpf/bpf-next/c/4edab4c55d2d
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html