Global subprogs in RCU/{preempt,irq}-disabled sections
Subsystems (4)
Kumar Kartikeya Dwivedi
maintainerv3 cover Global subprogs in RCU/{preempt,irq}-disabled sections
Small change to allow non-sleepable global subprogs in RCU, preempt-disabled, and irq-disabled sections. For now, we don't lift the limitation for locks as it requires more analysis, and will do this one resilient spin locks land.
This surfaced a bug where sleepable global subprogs were allowed in RCU read sections, that has been fixed. Tests have been added to cover various cases.
Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]
* Fix broken to_be_replaced argument in the selftest.
* Adjust selftest program type.
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Rename subprog_info[i].sleepable to might_sleep, which more accurately reflects the nature of the bit. 'sleepable' means whether a given context is allowed to, while might_sleep captures if it does.
* Disallow extensions that might sleep to attach to targets that don't sleep, since they'd be permitted to be called in atomic contexts. (Eduard)
* Add tests for mixing non-sleepable and sleepable global function calls, and extensions attaching to non-sleepable global functions. (Eduard)
* Rename changes_pkt_data -> summarization
Kumar Kartikeya Dwivedi (3):
bpf: Summarize sleepable global subprogs selftests/bpf: Test sleepable global subprogs in atomic contexts selftests/bpf: Add tests for extending sleepable global subprogs
include/linux/bpf.h | 1 +
include/linux/bpf_verifier.h | 1 +
kernel/bpf/verifier.c | 62 ++++++--
.../bpf/prog_tests/changes_pkt_data.c | 107 -------------
.../selftests/bpf/prog_tests/rcu_read_lock.c | 3 +
.../selftests/bpf/prog_tests/spin_lock.c | 3 +
.../selftests/bpf/prog_tests/summarization.c | 144 ++++++++++++++++++
.../selftests/bpf/progs/changes_pkt_data.c | 39 -----
tools/testing/selftests/bpf/progs/irq.c | 71 ++++++++-
.../selftests/bpf/progs/preempt_lock.c | 68 ++++++++-
.../selftests/bpf/progs/rcu_read_lock.c | 58 +++++++
.../selftests/bpf/progs/summarization.c | 78 ++++++++++
...ta_freplace.c => summarization_freplace.c} | 17 ++-
.../selftests/bpf/progs/test_spin_lock_fail.c | 69 +++++++++
14 files changed, 558 insertions(+), 163 deletions(-)
delete mode 100644 tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/summarization.c
delete mode 100644 tools/testing/selftests/bpf/progs/changes_pkt_data.c
create mode 100644 tools/testing/selftests/bpf/progs/summarization.c
rename tools/testing/selftests/bpf/progs/{changes_pkt_data_freplace.c => summarization_freplace.c} (57%)base-commit: 0b9363131daf4227d5ae11ee677acdcfff06e938
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v3 1/3 bpf: Summarize sleepable global subprogs
The verifier currently does not permit global subprog calls when a lock is held, preemption is disabled, or when IRQs are disabled. This is because we don't know whether the global subprog calls sleepable functions or not.
In case of locks, there's an additional reason: functions called by the global subprog may hold additional locks etc. The verifier won't know while verifying the global subprog whether it was called in context where a spin lock is already held by the program.
Perform summarization of the sleepable nature of a global subprog just like changes_pkt_data and then allow calls to global subprogs for non-sleepable ones from atomic context.
While making this change, I noticed that RCU read sections had no protection against sleepable global subprog calls, include it in the checks and fix this while we're at it.
Care needs to be taken to not allow global subprog calls when regular bpf_spin_lock is held. When resilient spin locks is held, we want to potentially have this check relaxed, but not for now.
Also make sure extensions freplacing global functions cannot do so in case the target is non-sleepable, but the extension is. The other combination is ok.
Tests are included in the next patch to handle all special conditions.
--- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 62 ++++++++++++++++++++++++++++-------- 3 files changed, 50 insertions(+), 14 deletions(-)
include/linux/bpf.h +1 -0
@@ -1531,6 +1531,7 @@ struct bpf_prog_aux {bool jits_use_priv_stack;bool priv_stack_requested;bool changes_pkt_data; +bool might_sleep;u64 prog_array_member_cnt;/* counts how many times as member of prog_array */ struct mutex ext_mutex;/* mutex for is_extended and prog_array_member_cnt */ struct bpf_arena *arena;
include/linux/bpf_verifier.h +1 -0
@@ -667,6 +667,7 @@ struct bpf_subprog_info {/* true if bpf_fastcall stack region is used by functions that can't be inlined */ bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; + bool might_sleep: 1;enum priv_stack_mode priv_stack_mode;u8 arg_cnt;
kernel/bpf/verifier.c +48 -14
122 lines not on this page · raw message
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v3 2/3 selftests/bpf: Test sleepable global subprogs in atomic contexts
In reply to Kumar Kartikeya DwivediAdd tests for rejecting sleepable and accepting non-sleepable global function calls in atomic contexts. For spin locks, we still reject all global function calls. Once resilient spin locks land, we will carefully lift in cases where we deem it safe.
--- .../selftests/bpf/prog_tests/rcu_read_lock.c | 3 + .../selftests/bpf/prog_tests/spin_lock.c | 3 + tools/testing/selftests/bpf/progs/irq.c | 71 ++++++++++++++++++- .../selftests/bpf/progs/preempt_lock.c | 68 +++++++++++++++++- .../selftests/bpf/progs/rcu_read_lock.c | 58 +++++++++++++++ .../selftests/bpf/progs/test_spin_lock_fail.c | 69 ++++++++++++++++++ 6 files changed, 270 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c +3 -0
@@ -81,6 +81,9 @@ static const char * const inproper_region_tests[] = {"nested_rcu_region" ,"rcu_read_lock_global_subprog_lock" ,"rcu_read_lock_global_subprog_unlock" , +"rcu_read_lock_sleepable_helper_global_subprog" , +"rcu_read_lock_sleepable_kfunc_global_subprog" , + "rcu_read_lock_sleepable_global_subprog_indirect", };static void test_inproper_region (void )
tools/testing/selftests/bpf/prog_tests/spin_lock.c +3 -0
@@ -50,6 +50,9 @@ static struct { { "lock_id_mismatch_innermapval_mapval", "bpf_spin_unlock of different lock" }, { "lock_global_subprog_call1", "global function calls are not allowed while holding a lock" }, { "lock_global_subprog_call2", "global function calls are not allowed while holding a lock" }, + { "lock_global_sleepable_helper_subprog", "global function calls are not allowed while holding a lock" }, + { "lock_global_sleepable_kfunc_subprog", "global function calls are not allowed while holding a lock" }, + { "lock_global_sleepable_subprog_indirect", "global function calls are not allowed while holding a lock" }, };static int match_regex (const char *pattern,const char *string)
tools/testing/selftests/bpf/progs/irq.c +70 -1
81 lines not on this page · raw message
tools/testing/selftests/bpf/progs/preempt_lock.c +67 -1
78 lines not on this page · raw message
tools/testing/selftests/bpf/progs/rcu_read_lock.c +58 -0
61 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_spin_lock_fail.c +69 -0
73 lines not on this page · raw message
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v3 3/3 selftests/bpf: Add tests for extending sleepable global subprogs
In reply to Kumar Kartikeya DwivediAdd tests for freplace behavior with the combination of sleepable and non-sleepable global subprogs. The changes_pkt_data selftest did all the hardwork, so simply rename it and include new support for more summarization tests for might_sleep bit.
---
.../bpf/prog_tests/changes_pkt_data.c | 107 -------------
.../selftests/bpf/prog_tests/summarization.c | 144 ++++++++++++++++++
.../selftests/bpf/progs/changes_pkt_data.c | 39 -----
.../selftests/bpf/progs/summarization.c | 78 ++++++++++
...ta_freplace.c => summarization_freplace.c} | 17 ++-
5 files changed, 238 insertions(+), 147 deletions(-)
delete mode 100644 tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/summarization.c
delete mode 100644 tools/testing/selftests/bpf/progs/changes_pkt_data.c
create mode 100644 tools/testing/selftests/bpf/progs/summarization.c
rename tools/testing/selftests/bpf/progs/{changes_pkt_data_freplace.c => summarization_freplace.c} (57%)tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c deleted +0 -107
@@ -1,107 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include "bpf/libbpf.h" -#include "changes_pkt_data_freplace.skel.h" -#include "changes_pkt_data.skel.h" -#include <test_progs.h> - -static void print_verifier_log (const char *log) -{ -if (env.verbosity >=VERBOSE_VERY ) -fprintf (stdout,"VERIFIER LOG:\n=============\n%s=============\n" , log); -} - -static void test_aux (const char *main_prog_name, -const char *to_be_replaced, -const char *replacement, -bool expect_load) -{ -struct changes_pkt_data_freplace *freplace =NULL ; -struct bpf_program *freplace_prog =NULL ; -struct bpf_program *main_prog =NULL ; -LIBBPF_OPTS (bpf_object_open_opts, opts); -struct changes_pkt_data *main =NULL ; -char log[16 *1024 ]; -int err; - - opts.kernel_log_buf = log; - opts.kernel_log_size =sizeof (log); -if (env.verbosity >=VERBOSE_SUPER ) - opts.kernel_log_level =1 |2 |4 ; - main =changes_pkt_data__open_opts (&opts); -if (!ASSERT_OK_PTR (main,"changes_pkt_data__open" )) - goto out; - main_prog =bpf_object__find_program_by_name (main->obj, main_prog_name); -if (!ASSERT_OK_PTR (main_prog,"main_prog" )) - goto out; -bpf_program__set_autoload (main_prog, true); - err =changes_pkt_data__load (main); -print_verifier_log (log); -if (!ASSERT_OK (err,"changes_pkt_data__load" )) - goto out; - freplace =changes_pkt_data_freplace__open_opts (&opts); -if (!ASSERT_OK_PTR (freplace,"changes_pkt_data_freplace__open" )) - goto out; - freplace_prog =bpf_object__find_program_by_name (freplace->obj, replacement); -if (!ASSERT_OK_PTR (freplace_prog,"freplace_prog" )) - goto out; -bpf_program__set_autoload (freplace_prog, true); -bpf_program__set_autoattach (freplace_prog, true); -bpf_program__set_attach_target (freplace_prog, -bpf_program__fd (main_prog), - to_be_replaced); - err =changes_pkt_data_freplace__load (freplace); -print_verifier_log (log); -if (expect_load) { -ASSERT_OK (err,"changes_pkt_data_freplace__load" ); - }else { -ASSERT_ERR (err,"changes_pkt_data_freplace__load" ); -ASSERT_HAS_SUBSTR (log,"Extension program changes packet data" ,"error log" ); - } - -out: -changes_pkt_data_freplace__destroy (freplace); -changes_pkt_data__destroy (main); -} - -/* There are two global subprograms in both changes_pkt_data.skel.h: -* - one changes packet data; -* - another does not. -* It is ok to freplace subprograms that change packet data with those -* that either do or do not. It is only ok to freplace subprograms -* that do not change packet data with those that do not as well. -* The below tests check outcomes for each combination of such freplace. -* Also test a case when main subprogram itself is replaced and is a single -* subprogram in a program. -*/ -void test_changes_pkt_data_freplace (void ) -{ -struct { -const char *main; -const char *to_be_replaced; -bool changes; - } mains[] = { - {"main_with_subprogs" ,"changes_pkt_data" , true }, - {"main_with_subprogs" ,"does_not_change_pkt_data" , false }, - {"main_changes" ,"main_changes" , true }, - {"main_does_not_change" ,"main_does_not_change" , false }, - }; -struct { -const char *func; -bool changes; - } replacements[] = { - {"changes_pkt_data" , true }, - {"does_not_change_pkt_data" , false } - }; -char buf[64 ]; - -for (int i =0 ; i <ARRAY_SIZE (mains); ++i) { -for (int j =0 ; j <ARRAY_SIZE (replacements); ++j) { -snprintf (buf,sizeof (buf),"%s_with_%s" , - mains[i].to_be_replaced, replacements[j].func); -if (!test__start_subtest (buf)) -continue ; -test_aux (mains[i].main, mains[i].to_be_replaced, replacements[j].func, - mains[i].changes || !replacements[j].changes); - } - } -}
tools/testing/selftests/bpf/prog_tests/summarization.c new file +144 -0
144 lines not on this page · raw message
tools/testing/selftests/bpf/progs/changes_pkt_data.c deleted +0 -39
@@ -1,39 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 - -#include <linux/bpf.h> -#include <bpf/bpf_helpers.h> - -__noinline -long changes_pkt_data (struct __sk_buff *sk) -{ -return bpf_skb_pull_data (sk,0 ); -} - -__noinline __weak -long does_not_change_pkt_data (struct __sk_buff *sk) -{ -return 0 ; -} - -SEC("?tc") -int main_with_subprogs (struct __sk_buff *sk) -{ -changes_pkt_data (sk); -does_not_change_pkt_data (sk); -return 0 ; -} - -SEC("?tc") -int main_changes (struct __sk_buff *sk) -{ -bpf_skb_pull_data (sk,0 ); -return 0 ; -} - -SEC("?tc") -int main_does_not_change (struct __sk_buff *sk) -{ -return 0 ; -} - -char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/summarization.c new file +78 -0
78 lines not on this page · raw message
tools/testing/selftests/bpf/progs/summarization_freplace.c renamed from tools/testing/selftests/bpf/progs/changes_pkt_data_freplace.c +16 -1
@@ -1,6 +1,6 @@// SPDX-License-Identifier: GPL-2.0 -#include <linux/bpf.h> +#include <vmlinux.h> #include <bpf/bpf_helpers.h> SEC ("?freplace" )
@@ -15,4 +15,19 @@ long does_not_change_pkt_data(struct __sk_buff *sk)return 0 ; } +SEC("?freplace") +long might_sleep (struct pt_regs *ctx) +{ +int i; + +bpf_copy_from_user (&i,sizeof (i),NULL ); +return i; +} + +SEC("?freplace") +long does_not_sleep (struct pt_regs *ctx) +{ +return 0 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.43.5
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 11 lines · On Sat, 1 Mar 2025 07:18:43 -0800 you wrote:
Small change to allow non-sleepable global subprogs in RCU, preempt-disabled, and irq-disabled sections. For now, we don't lift the limitation for locks as it requires more analysis, and will do this one resilient spin locks land.
This surfaced a bug where sleepable global subprogs were allowed in RCU read sections, that has been fixed. Tests have been added to cover various cases.
[...]
Here is the summary with links:
- [bpf-next,v3,1/3] bpf: Summarize sleepable global subprogs
https://git.kernel.org/bpf/bpf-next/c/90d8c8980b5c - [bpf-next,v3,2/3] selftests/bpf: Test sleepable global subprogs in atomic contexts https://git.kernel.org/bpf/bpf-next/c/6e2cc6067e73 - [bpf-next,v3,3/3] selftests/bpf: Add tests for extending sleepable global subprogs https://git.kernel.org/bpf/bpf-next/c/ce9add7b9028
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html