selftests/bpf: Fix flakiness of task_local_storage/sys_enter_exit
Subsystems (3)
Ihor Solodrai
reviewerPATCH 1/1 selftests/bpf: Fix flakiness of task_local_storage/sys_enter_exit
The test_sys_enter_exit test was setting target_pid before attaching the BPF programs, which causes syscalls made during the attach phase to be counted. This is flaky because, apparently, there is no guarantee that both on_enter and on_exit will trigger during the attachment.
Move the target_pid assignment to after task_local_storage__attach() so that only explicit sys_gettid() calls are counted.
---
I've been experimenting with running AI on BPF CI to investigate test failures. This is an example of a thing it may come up with.
I don't want to spam the list with these, so for starters I'll be relaying only patches that I evaluated and/or tested.
The AI generated reports will be posted with "[bpf-ci-bot]" prefix
here: https://github.com/kernel-patches/vmtest/issues
The goal of this particular application of AI is to make BPF CI more stable, less noisy/flaky, and potentially find and fix more kernel bugs. We'll see how it goes.
--- .../selftests/bpf/prog_tests/task_local_storage.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +9 -5
@@ -25,24 +25,28 @@static void test_sys_enter_exit (void ) {struct task_local_storage *skel; +pid_t pid =sys_gettid ();int err; skel =task_local_storage__open_and_load ();if (!ASSERT_OK_PTR (skel,"skel_open_and_load" ))return ; - skel->bss->target_pid =sys_gettid (); - err =task_local_storage__attach (skel);if (!ASSERT_OK (err,"skel_attach" )) goto out; +/* Set target_pid after attach so that syscalls made during +* attach are not counted. +*/ + skel->bss->target_pid = pid; +sys_gettid ();sys_gettid (); -/* 3x syscalls: 1x attach and 2x gettid */ -ASSERT_EQ (skel->bss->enter_cnt,3 ,"enter_cnt" ); -ASSERT_EQ (skel->bss->exit_cnt,3 ,"exit_cnt" ); +/* 2x gettid syscalls */ +ASSERT_EQ (skel->bss->enter_cnt,2 ,"enter_cnt" ); +ASSERT_EQ (skel->bss->exit_cnt,2 ,"exit_cnt" );ASSERT_EQ (skel->bss->mismatch_cnt,0 ,"mismatch_cnt" ); out:task_local_storage__destroy (skel);
Signature · 1 line
2.53.0
Quoted 50 lines · Ihor Solodrai
The
test_sys_enter_exittest was settingtarget_pidbefore attaching the BPF programs, which causes syscalls made during the attach phase to be counted. This is flaky because, apparently, there is no guarantee that bothon_enterandon_exitwill trigger during the attachment.Move the
target_pidassignment to aftertask_local_storage__attach()so that only explicitsys_gettid()calls are counted.Reported-by BPF CI Bot (Claude Opus 4.6) <[email protected]>Signed-off-by Ihor Solodrai <[email protected]>---
I've been experimenting with running AI on BPF CI to investigate test failures. This is an example of a thing it may come up with.
I don't want to spam the list with these, so for starters I'll be relaying only patches that I evaluated and/or tested.
The AI generated reports will be posted with "[bpf-ci-bot]" prefix
here: https://github.com/kernel-patches/vmtest/issuesThe goal of this particular application of AI is to make BPF CI more stable, less noisy/flaky, and potentially find and fix more kernel bugs. We'll see how it goes.
--- .../selftests/bpf/prog_tests/task_local_storage.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)tools/testing/selftests/bpf/prog_tests/task_local_storage.c +1
-2@@ -25,24 +25,28 @@static void test_sys_enter_exit (void ) {struct task_local_storage *skel; +pid_t pid =sys_gettid ();int err; skel =task_local_storage__open_and_load ();if (!ASSERT_OK_PTR (skel,"skel_open_and_load" ))return ; - skel->bss->target_pid =sys_gettid (); - err =task_local_storage__attach (skel);
tools/testing/selftests/bpf/prog_tests/task_local_storage.c line 45
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +5
-0if (!ASSERT_OK (err,"skel_attach" )) goto out; +/* Set target_pid after attach so that syscalls made during +* attach are not counted. +*/ + skel->bss->target_pid = pid; +sys_gettid ();sys_gettid ();
Maybe a simpler and less fragile fix would be to add syscall number filter in the BPF program, so that we don't count those unexpected syscalls.
Quoted 12 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +3
-3-/* 3x syscalls: 1x attach and 2x gettid */ -ASSERT_EQ (skel->bss->enter_cnt,3 ,"enter_cnt" ); -ASSERT_EQ (skel->bss->exit_cnt,3 ,"exit_cnt" ); +/* 2x gettid syscalls */ +ASSERT_EQ (skel->bss->enter_cnt,2 ,"enter_cnt" ); +ASSERT_EQ (skel->bss->exit_cnt,2 ,"exit_cnt" );ASSERT_EQ (skel->bss->mismatch_cnt,0 ,"mismatch_cnt" ); out:task_local_storage__destroy (skel);Signature · 1 line
2.53.0
Ihor Solodrai
reviewerQuoted 55 lines · Mykyta Yatsenko
Quoted 53 lines · Ihor Solodrai
The
test_sys_enter_exittest was settingtarget_pidbefore attaching the BPF programs, which causes syscalls made during the attach phase to be counted. This is flaky because, apparently, there is no guarantee that bothon_enterandon_exitwill trigger during the attachment.Move the
target_pidassignment to aftertask_local_storage__attach()so that only explicitsys_gettid()calls are counted.Reported-by BPF CI Bot (Claude Opus 4.6) <[email protected]>Signed-off-by Ihor Solodrai <[email protected]>---
I've been experimenting with running AI on BPF CI to investigate test failures. This is an example of a thing it may come up with.
I don't want to spam the list with these, so for starters I'll be relaying only patches that I evaluated and/or tested.
The AI generated reports will be posted with "[bpf-ci-bot]" prefix
here: https://github.com/kernel-patches/vmtest/issuesThe goal of this particular application of AI is to make BPF CI more stable, less noisy/flaky, and potentially find and fix more kernel bugs. We'll see how it goes.
--- .../selftests/bpf/prog_tests/task_local_storage.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)tools/testing/selftests/bpf/prog_tests/task_local_storage.c +1
-2@@ -25,24 +25,28 @@static void test_sys_enter_exit (void ) {struct task_local_storage *skel; +pid_t pid =sys_gettid ();int err; skel =task_local_storage__open_and_load ();if (!ASSERT_OK_PTR (skel,"skel_open_and_load" ))return ; - skel->bss->target_pid =sys_gettid (); - err =task_local_storage__attach (skel);if (!ASSERT_OK (err,"skel_attach" )) goto out;
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c line 45
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +5
-0+/* Set target_pid after attach so that syscalls made during +* attach are not counted. +*/ + skel->bss->target_pid = pid; +sys_gettid ();sys_gettid ();Maybe a simpler and less fragile fix would be to add syscall number filter in the BPF program, so that we don't count those unexpected syscalls.
Why do you think this is fragile? We are not counting the syscalls made during attachment at all this way.
You may be right, just trying to understand.
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c line 43
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +0
-1-/* 3x syscalls: 1x attach and 2x gettid */
The way the test was written, those syscalls during attachment were expected. But the flakiness wasn't obvious.
Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +3
-2-ASSERT_EQ (skel->bss->enter_cnt,3 ,"enter_cnt" ); -ASSERT_EQ (skel->bss->exit_cnt,3 ,"exit_cnt" ); +/* 2x gettid syscalls */ +ASSERT_EQ (skel->bss->enter_cnt,2 ,"enter_cnt" ); +ASSERT_EQ (skel->bss->exit_cnt,2 ,"exit_cnt" );ASSERT_EQ (skel->bss->mismatch_cnt,0 ,"mismatch_cnt" ); out:task_local_storage__destroy (skel);Signature · 1 line
2.53.0
Quoted 68 lines · Ihor Solodrai
Quoted 55 lines · Mykyta Yatsenko
Quoted 53 lines · Ihor Solodrai
The
test_sys_enter_exittest was settingtarget_pidbefore attaching the BPF programs, which causes syscalls made during the attach phase to be counted. This is flaky because, apparently, there is no guarantee that bothon_enterandon_exitwill trigger during the attachment.Move the
target_pidassignment to aftertask_local_storage__attach()so that only explicitsys_gettid()calls are counted.Reported-by BPF CI Bot (Claude Opus 4.6) <[email protected]>Signed-off-by Ihor Solodrai <[email protected]>---
I've been experimenting with running AI on BPF CI to investigate test failures. This is an example of a thing it may come up with.
I don't want to spam the list with these, so for starters I'll be relaying only patches that I evaluated and/or tested.
The AI generated reports will be posted with "[bpf-ci-bot]" prefix
here: https://github.com/kernel-patches/vmtest/issuesThe goal of this particular application of AI is to make BPF CI more stable, less noisy/flaky, and potentially find and fix more kernel bugs. We'll see how it goes.
--- .../selftests/bpf/prog_tests/task_local_storage.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)tools/testing/selftests/bpf/prog_tests/task_local_storage.c +1
-2@@ -25,24 +25,28 @@static void test_sys_enter_exit (void ) {struct task_local_storage *skel; +pid_t pid =sys_gettid ();int err; skel =task_local_storage__open_and_load ();if (!ASSERT_OK_PTR (skel,"skel_open_and_load" ))return ; - skel->bss->target_pid =sys_gettid (); - err =task_local_storage__attach (skel);if (!ASSERT_OK (err,"skel_attach" )) goto out;Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c line 45
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +5
-0+/* Set target_pid after attach so that syscalls made during +* attach are not counted. +*/ + skel->bss->target_pid = pid; +sys_gettid ();sys_gettid ();Maybe a simpler and less fragile fix would be to add syscall number filter in the BPF program, so that we don't count those unexpected syscalls.
Quoted 2 lines
Why do you think this is fragile? We are not counting the syscalls made during attachment at all this way.
I think it's more fragile if we add some code after sys_gettid() that makes syscall or for example ASSERT_EQ makes one. Another example: if someone decides to debug this test case and adds printf. I suggest to tighten up the condition in the BPF program, so potential changes in the user space have less chances breaking things.
I'm fine with your change, as it fixes the flakiness, but it is not ideal in my opinion if we use this approach in more tests widely.
Quoted 19 lines
You may be right, just trying to understand.
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c line 43
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +0
-1-/* 3x syscalls: 1x attach and 2x gettid */ The way the test was written, those syscalls during attachment were expected. But the flakiness wasn't obvious.
Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +3
-2-ASSERT_EQ (skel->bss->enter_cnt,3 ,"enter_cnt" ); -ASSERT_EQ (skel->bss->exit_cnt,3 ,"exit_cnt" ); +/* 2x gettid syscalls */ +ASSERT_EQ (skel->bss->enter_cnt,2 ,"enter_cnt" ); +ASSERT_EQ (skel->bss->exit_cnt,2 ,"exit_cnt" );ASSERT_EQ (skel->bss->mismatch_cnt,0 ,"mismatch_cnt" ); out:task_local_storage__destroy (skel);Signature · 1 line
2.53.0
Quoted 52 lines · Ihor Solodrai
The
test_sys_enter_exittest was settingtarget_pidbefore attaching the BPF programs, which causes syscalls made during the attach phase to be counted. This is flaky because, apparently, there is no guarantee that bothon_enterandon_exitwill trigger during the attachment.Move the
target_pidassignment to aftertask_local_storage__attach()so that only explicitsys_gettid()calls are counted.Reported-by BPF CI Bot (Claude Opus 4.6) <[email protected]>Signed-off-by Ihor Solodrai <[email protected]>---
I've been experimenting with running AI on BPF CI to investigate test failures. This is an example of a thing it may come up with.
I don't want to spam the list with these, so for starters I'll be relaying only patches that I evaluated and/or tested.
The AI generated reports will be posted with "[bpf-ci-bot]" prefix
here: https://github.com/kernel-patches/vmtest/issuesThe goal of this particular application of AI is to make BPF CI more stable, less noisy/flaky, and potentially find and fix more kernel bugs. We'll see how it goes.
--- .../selftests/bpf/prog_tests/task_local_storage.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)tools/testing/selftests/bpf/prog_tests/task_local_storage.c +1
-2@@ -25,24 +25,28 @@static void test_sys_enter_exit (void ) {struct task_local_storage *skel; +pid_t pid =sys_gettid ();int err; skel =task_local_storage__open_and_load ();if (!ASSERT_OK_PTR (skel,"skel_open_and_load" ))return ; - skel->bss->target_pid =sys_gettid (); - err =task_local_storage__attach (skel);if (!ASSERT_OK (err,"skel_attach" ))
tools/testing/selftests/bpf/prog_tests/task_local_storage.c line 46
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +5
-0goto out; +/* Set target_pid after attach so that syscalls made during +* attach are not counted. +*/ + skel->bss->target_pid = pid; +sys_gettid ();sys_gettid ();
The test is done at this point and perhaps we should also add skel->bss->target_pid = 0 to avoid some global variables from being changed by bpf programs again. This is already done in prog_tests/cgrp_local_storage.c. Not solving interference from parallel subtests, but at least it will be easier to reason about the test result.
Quoted 12 lines
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +3
-4-/* 3x syscalls: 1x attach and 2x gettid */ -ASSERT_EQ (skel->bss->enter_cnt,3 ,"enter_cnt" ); -ASSERT_EQ (skel->bss->exit_cnt,3 ,"exit_cnt" ); +/* 2x gettid syscalls */ +ASSERT_EQ (skel->bss->enter_cnt,2 ,"enter_cnt" ); +ASSERT_EQ (skel->bss->exit_cnt,2 ,"exit_cnt" );ASSERT_EQ (skel->bss->mismatch_cnt,0 ,"mismatch_cnt" ); out:task_local_storage__destroy (skel); --2.53.0