Global subprogs in RCU/{preempt,irq}-disabled sections
Subsystems (4)
Kumar Kartikeya Dwivedi
maintainerv2 cover Global subprogs in RCU/{preempt,irq}-disabled sections
Small change to allow non-sleepable global subprogs in RCU, preempt-disabled, and irq-disabled sections. For now, we don't lift the limitation for locks as it requires more analysis, and will do this one resilient spin locks land.
This surfaced a bug where sleepable global subprogs were allowed in RCU read sections, that has been fixed. Tests have been added to cover various cases.
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Rename subprog_info[i].sleepable to might_sleep, which more accurately reflects the nature of the bit. 'sleepable' means whether a given context is allowed to, while might_sleep captures if it does.
* Disallow extensions that might sleep to attach to targets that don't sleep, since they'd be permitted to be called in atomic contexts. (Eduard)
* Add tests for mixing non-sleepable and sleepable global function calls, and extensions attaching to non-sleepable global functions. (Eduard)
* Rename changes_pkt_data -> summarization
Kumar Kartikeya Dwivedi (3):
bpf: Summarize sleepable global subprogs selftests/bpf: Test sleepable global subprogs in atomic contexts selftests/bpf: Add tests for extending sleepable global subprogs
include/linux/bpf.h | 1 +
include/linux/bpf_verifier.h | 1 +
kernel/bpf/verifier.c | 62 ++++++--
.../bpf/prog_tests/changes_pkt_data.c | 107 -------------
.../selftests/bpf/prog_tests/rcu_read_lock.c | 3 +
.../selftests/bpf/prog_tests/spin_lock.c | 3 +
.../selftests/bpf/prog_tests/summarization.c | 143 ++++++++++++++++++
tools/testing/selftests/bpf/progs/irq.c | 71 ++++++++-
.../selftests/bpf/progs/preempt_lock.c | 68 ++++++++-
.../selftests/bpf/progs/rcu_read_lock.c | 58 +++++++
.../{changes_pkt_data.c => summarization.c} | 38 ++++-
...ta_freplace.c => summarization_freplace.c} | 13 ++
.../selftests/bpf/progs/test_spin_lock_fail.c | 69 +++++++++
13 files changed, 513 insertions(+), 124 deletions(-)
delete mode 100644 tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/summarization.c
rename tools/testing/selftests/bpf/progs/{changes_pkt_data.c => summarization.c} (52%)
rename tools/testing/selftests/bpf/progs/{changes_pkt_data_freplace.c => summarization_freplace.c} (66%)base-commit: 0b9363131daf4227d5ae11ee677acdcfff06e938
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v2 1/3 bpf: Summarize sleepable global subprogs
The verifier currently does not permit global subprog calls when a lock is held, preemption is disabled, or when IRQs are disabled. This is because we don't know whether the global subprog calls sleepable functions or not.
In case of locks, there's an additional reason: functions called by the global subprog may hold additional locks etc. The verifier won't know while verifying the global subprog whether it was called in context where a spin lock is already held by the program.
Perform summarization of the sleepable nature of a global subprog just like changes_pkt_data and then allow calls to global subprogs for non-sleepable ones from atomic context.
While making this change, I noticed that RCU read sections had no protection against sleepable global subprog calls, include it in the checks and fix this while we're at it.
Care needs to be taken to not allow global subprog calls when regular bpf_spin_lock is held. When resilient spin locks is held, we want to potentially have this check relaxed, but not for now.
Also make sure extensions freplacing global functions cannot do so in case the target is non-sleepable, but the extension is. The other combination is ok.
Tests are included in the next patch to handle all special conditions.
--- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 62 ++++++++++++++++++++++++++++-------- 3 files changed, 50 insertions(+), 14 deletions(-)
include/linux/bpf.h +1 -0
@@ -1531,6 +1531,7 @@ struct bpf_prog_aux {bool jits_use_priv_stack;bool priv_stack_requested;bool changes_pkt_data; +bool might_sleep;u64 prog_array_member_cnt;/* counts how many times as member of prog_array */ struct mutex ext_mutex;/* mutex for is_extended and prog_array_member_cnt */ struct bpf_arena *arena;
include/linux/bpf_verifier.h +1 -0
@@ -667,6 +667,7 @@ struct bpf_subprog_info {/* true if bpf_fastcall stack region is used by functions that can't be inlined */ bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; + bool might_sleep: 1;enum priv_stack_mode priv_stack_mode;u8 arg_cnt;
kernel/bpf/verifier.c +48 -14
122 lines not on this page · raw message
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v2 2/3 selftests/bpf: Test sleepable global subprogs in atomic contexts
In reply to Kumar Kartikeya DwivediAdd tests for rejecting sleepable and accepting non-sleepable global function calls in atomic contexts. For spin locks, we still reject all global function calls. Once resilient spin locks land, we will carefully lift in cases where we deem it safe.
--- .../selftests/bpf/prog_tests/rcu_read_lock.c | 3 + .../selftests/bpf/prog_tests/spin_lock.c | 3 + tools/testing/selftests/bpf/progs/irq.c | 71 ++++++++++++++++++- .../selftests/bpf/progs/preempt_lock.c | 68 +++++++++++++++++- .../selftests/bpf/progs/rcu_read_lock.c | 58 +++++++++++++++ .../selftests/bpf/progs/test_spin_lock_fail.c | 69 ++++++++++++++++++ 6 files changed, 270 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c +3 -0
@@ -81,6 +81,9 @@ static const char * const inproper_region_tests[] = {"nested_rcu_region" ,"rcu_read_lock_global_subprog_lock" ,"rcu_read_lock_global_subprog_unlock" , +"rcu_read_lock_sleepable_helper_global_subprog" , +"rcu_read_lock_sleepable_kfunc_global_subprog" , + "rcu_read_lock_sleepable_global_subprog_indirect", };static void test_inproper_region (void )
tools/testing/selftests/bpf/prog_tests/spin_lock.c +3 -0
@@ -50,6 +50,9 @@ static struct { { "lock_id_mismatch_innermapval_mapval", "bpf_spin_unlock of different lock" }, { "lock_global_subprog_call1", "global function calls are not allowed while holding a lock" }, { "lock_global_subprog_call2", "global function calls are not allowed while holding a lock" }, + { "lock_global_sleepable_helper_subprog", "global function calls are not allowed while holding a lock" }, + { "lock_global_sleepable_kfunc_subprog", "global function calls are not allowed while holding a lock" }, + { "lock_global_sleepable_subprog_indirect", "global function calls are not allowed while holding a lock" }, };static int match_regex (const char *pattern,const char *string)
tools/testing/selftests/bpf/progs/irq.c +70 -1
81 lines not on this page · raw message
tools/testing/selftests/bpf/progs/preempt_lock.c +67 -1
78 lines not on this page · raw message
tools/testing/selftests/bpf/progs/rcu_read_lock.c +58 -0
61 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_spin_lock_fail.c +69 -0
73 lines not on this page · raw message
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v2 3/3 selftests/bpf: Add tests for extending sleepable global subprogs
In reply to Kumar Kartikeya DwivediAdd tests for freplace behavior with the combination of sleepable and non-sleepable global subprogs. The changes_pkt_data selftest did all the hardwork, so simply rename it and include new support for more summarization tests for might_sleep bit.
Sleepable extensions don't work yet, so add support but disable it for now, allow support to be tested once it's enabled (and ensure we will complain then).
---
.../bpf/prog_tests/changes_pkt_data.c | 107 -------------
.../selftests/bpf/prog_tests/summarization.c | 143 ++++++++++++++++++
.../{changes_pkt_data.c => summarization.c} | 38 ++++-
...ta_freplace.c => summarization_freplace.c} | 13 ++
4 files changed, 193 insertions(+), 108 deletions(-)
delete mode 100644 tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/summarization.c
rename tools/testing/selftests/bpf/progs/{changes_pkt_data.c => summarization.c} (52%)
rename tools/testing/selftests/bpf/progs/{changes_pkt_data_freplace.c => summarization_freplace.c} (66%)tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c deleted +0 -107
@@ -1,107 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include "bpf/libbpf.h" -#include "changes_pkt_data_freplace.skel.h" -#include "changes_pkt_data.skel.h" -#include <test_progs.h> - -static void print_verifier_log (const char *log) -{ -if (env.verbosity >=VERBOSE_VERY ) -fprintf (stdout,"VERIFIER LOG:\n=============\n%s=============\n" , log); -} - -static void test_aux (const char *main_prog_name, -const char *to_be_replaced, -const char *replacement, -bool expect_load) -{ -struct changes_pkt_data_freplace *freplace =NULL ; -struct bpf_program *freplace_prog =NULL ; -struct bpf_program *main_prog =NULL ; -LIBBPF_OPTS (bpf_object_open_opts, opts); -struct changes_pkt_data *main =NULL ; -char log[16 *1024 ]; -int err; - - opts.kernel_log_buf = log; - opts.kernel_log_size =sizeof (log); -if (env.verbosity >=VERBOSE_SUPER ) - opts.kernel_log_level =1 |2 |4 ; - main =changes_pkt_data__open_opts (&opts); -if (!ASSERT_OK_PTR (main,"changes_pkt_data__open" )) - goto out; - main_prog =bpf_object__find_program_by_name (main->obj, main_prog_name); -if (!ASSERT_OK_PTR (main_prog,"main_prog" )) - goto out; -bpf_program__set_autoload (main_prog, true); - err =changes_pkt_data__load (main); -print_verifier_log (log); -if (!ASSERT_OK (err,"changes_pkt_data__load" )) - goto out; - freplace =changes_pkt_data_freplace__open_opts (&opts); -if (!ASSERT_OK_PTR (freplace,"changes_pkt_data_freplace__open" )) - goto out; - freplace_prog =bpf_object__find_program_by_name (freplace->obj, replacement); -if (!ASSERT_OK_PTR (freplace_prog,"freplace_prog" )) - goto out; -bpf_program__set_autoload (freplace_prog, true); -bpf_program__set_autoattach (freplace_prog, true); -bpf_program__set_attach_target (freplace_prog, -bpf_program__fd (main_prog), - to_be_replaced); - err =changes_pkt_data_freplace__load (freplace); -print_verifier_log (log); -if (expect_load) { -ASSERT_OK (err,"changes_pkt_data_freplace__load" ); - }else { -ASSERT_ERR (err,"changes_pkt_data_freplace__load" ); -ASSERT_HAS_SUBSTR (log,"Extension program changes packet data" ,"error log" ); - } - -out: -changes_pkt_data_freplace__destroy (freplace); -changes_pkt_data__destroy (main); -} - -/* There are two global subprograms in both changes_pkt_data.skel.h: -* - one changes packet data; -* - another does not. -* It is ok to freplace subprograms that change packet data with those -* that either do or do not. It is only ok to freplace subprograms -* that do not change packet data with those that do not as well. -* The below tests check outcomes for each combination of such freplace. -* Also test a case when main subprogram itself is replaced and is a single -* subprogram in a program. -*/ -void test_changes_pkt_data_freplace (void ) -{ -struct { -const char *main; -const char *to_be_replaced; -bool changes; - } mains[] = { - {"main_with_subprogs" ,"changes_pkt_data" , true }, - {"main_with_subprogs" ,"does_not_change_pkt_data" , false }, - {"main_changes" ,"main_changes" , true }, - {"main_does_not_change" ,"main_does_not_change" , false }, - }; -struct { -const char *func; -bool changes; - } replacements[] = { - {"changes_pkt_data" , true }, - {"does_not_change_pkt_data" , false } - }; -char buf[64 ]; - -for (int i =0 ; i <ARRAY_SIZE (mains); ++i) { -for (int j =0 ; j <ARRAY_SIZE (replacements); ++j) { -snprintf (buf,sizeof (buf),"%s_with_%s" , - mains[i].to_be_replaced, replacements[j].func); -if (!test__start_subtest (buf)) -continue ; -test_aux (mains[i].main, mains[i].to_be_replaced, replacements[j].func, - mains[i].changes || !replacements[j].changes); - } - } -}
tools/testing/selftests/bpf/prog_tests/summarization.c new file +143 -0
143 lines not on this page · raw message
tools/testing/selftests/bpf/progs/summarization.c renamed from tools/testing/selftests/bpf/progs/changes_pkt_data.c +37 -1
@@ -16,7 +16,7 @@ long does_not_change_pkt_data(struct __sk_buff *sk) } SEC("?tc") -int main_with_subprogs (struct __sk_buff *sk) +int main_changes_with_subprogs (struct __sk_buff *sk) {changes_pkt_data (sk);does_not_change_pkt_data (sk);
@@ -36,4 +36,40 @@ int main_does_not_change(struct __sk_buff *sk)return 0 ; } +__noinline +long might_sleep (int i) +{ +bpf_copy_from_user (&i,sizeof (i),NULL ); +return i; +} + +__noinline __weak +long does_not_sleep (int i) +{ +return 0 ; +} + +SEC("?syscall") +int main_might_sleep_with_subprogs (void *ctx) +{ +might_sleep (0 ); +does_not_sleep (0 ); +return 0 ; +} + +SEC("?syscall") +int main_might_sleep (void *ctx) +{ +int i; + +bpf_copy_from_user (&i,sizeof (i),NULL ); +return i; +} + +SEC("?syscall") +int main_does_not_sleep (void *sk) +{ +return 0 ; +} + char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/summarization_freplace.c renamed from tools/testing/selftests/bpf/progs/changes_pkt_data_freplace.c +13 -0
@@ -15,4 +15,17 @@ long does_not_change_pkt_data(struct __sk_buff *sk)return 0 ; } +SEC("?freplace") +long might_sleep (int i) +{ +bpf_copy_from_user (&i,sizeof (i),NULL ); +return i; +} + +SEC("?freplace") +long does_not_sleep (int i) +{ +return 0 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v2 3/3] selftests/bpf: Add tests for extending sleepable global subprogs
Quoted 23 lines · Kumar Kartikeya Dwivedi
Add tests for freplace behavior with the combination of sleepable and non-sleepable global subprogs. The
changes_pkt_dataselftest did all the hardwork, so simply rename it and include new support for more summarization tests formight_sleepbit.Sleepable extensions don't work yet, so add support but disable it for now, allow support to be tested once it's enabled (and ensure we will complain then).
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
[...]tools/testing/selftests/bpf/prog_tests/summarization.c new file +9
-0+void test_summarization_freplace (void ) +{ +struct { +const char *main; +const char *to_be_replaced; +bool has_side_effect; + } mains[2 ][4 ] = { + { + {"main_changes_with_subprogs" ,"changes_pkt_data" , true },
tools/testing/selftests/bpf/prog_tests/summarization.c line 110
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/summarization.c new file +10
-0+ { "main_changes_with_subprogs", "does_not_change_pkt_data", false }, + { "main_changes", "main_changes", true }, + { "main_does_not_change", "main_does_not_change", false }, + }, + { + { "main_might_sleep_with_subprogs", "might_sleep", true }, + { "main_might_sleep_with_subprogs", "does_not_sleep", false }, + { "main_might_sleep", "might_sleep", true }, + { "main_does_not_sleep", "does_not_sleep", false }, + },
Sigh, I forgot to regen with git format-patch with the hunks fixing this, it is obviously incorrect.
Let me resend again.