IRQ save/restore
Subsystems (4)
Kumar Kartikeya Dwivedi
maintainerv3 cover IRQ save/restore
This set introduces support for managing IRQ state from BPF programs. Two new kfuncs, bpf_local_irq_save, and bpf_local_irq_restore are introduced to enable this functionality.
Intended use cases are writing IRQ safe data structures (e.g. memory allocator) in BPF programs natively, and use in new spin locking primitives intended to be introduced in the next few weeks.
The set begins with some refactoring patches before the actual functionality is introduced. Patch 1 consolidates all resource related state in bpf_verifier_state, and moves it out from bpf_func_state.
Patch 2 refactor acquire and release functions for reference state to make them reusable without duplication for other resource types.
After this, patch 3 refactors stack slot liveness marking logic to be shared between dynptr, and iterators, in preparation for introducing same logic for irq flag object on stack.
Finally, patch 4 and 7 introduce the new kfuncs and their selftests. For more details, please inspect the patch commit logs. Patch 5 makes the error message in case of resource leaks under BPF_EXIT a bit clearer. Patch 6 expands coverage of existing preempt-disable selftest to cover sleepable kfuncs.
See individual patches for more details.
Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]
* Drop REF_TYPE_LOCK_MASK
* Add kfunc declarations to selftest to silence s390 CI errors
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Drop reference -> resource renaming in the verifier (Eduard, Alexei)
* Change verifier log for check_resource_leak for BPF_EXIT (Eduard)
* Remove id parameter from acquire_resource_state, read s->id (Eduard)
* Rename erase to release for reference state (Eduard)
* Move resource state to bpf_verifier_state (Eduard, Alexei)
* Drop unnecessary casting to/from u64 in helpers (Eduard)
* Add test for arg != PTR_TO_STACK (Eduard)
* Drop now redundant tests (Eduard)
* Address some other misc nits
* Add Reviewed-by and Acked-by from Eduard
Kumar Kartikeya Dwivedi (7):
bpf: Consolidate locks and reference state in verifier state
bpf: Refactor {acquire,release}_reference_state
bpf: Refactor mark_{dynptr,iter}_read
bpf: Introduce support for bpf_local_irq_{save,restore}
bpf: Improve verifier log for resource leak on exit selftests/bpf: Expand coverage of preempt tests to sleepable kfunc selftests/bpf: Add IRQ save/restore tests
include/linux/bpf_verifier.h | 19 +- kernel/bpf/helpers.c | 17 + kernel/bpf/log.c | 12 +- kernel/bpf/verifier.c | 531 +++++++++++++----- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/exceptions_fail.c | 4 +- tools/testing/selftests/bpf/progs/irq.c | 397 +++++++++++++ .../selftests/bpf/progs/preempt_lock.c | 26 +- .../selftests/bpf/progs/verifier_spin_lock.c | 2 +- 9 files changed, 863 insertions(+), 147 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/irq.c
base-commit: c8d02b547363880d996f80c38cc8b997c7b90725
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v3 1/7 bpf: Consolidate locks and reference state in verifier state
Currently, state for RCU read locks and preemption is in bpf_verifier_state, while locks and pointer reference state remains in bpf_func_state. There is no particular reason to keep the latter in bpf_func_state. Additionally, it is copied into a new frame's state and copied back to the caller frame's state everytime the verifier processes a pseudo call instruction. This is a bit wasteful, given this state is global for a given verification state / path.
Move all resource and reference related state in bpf_verifier_state structure in this patch, in preparation for introducing new reference state types in the future.
--- include/linux/bpf_verifier.h | 11 ++-- kernel/bpf/log.c | 11 ++-- kernel/bpf/verifier.c | 112 ++++++++++++++++------------------- 3 files changed, 64 insertions(+), 70 deletions(-)
include/linux/bpf_verifier.h +6 -5
@@ -315,9 +315,6 @@ struct bpf_func_state {u32 callback_depth;/* The following fields should be last. See copy_func_state() */ -int acquired_refs; -int active_locks; -struct bpf_reference_state *refs;/* The state of the stack. Each element of the array describes BPF_REG_SIZE * (i.e. 8) bytes worth of stack memory. * stack[0] represents bytes [*(r10-8)..*(r10-1)]
@@ -419,9 +416,13 @@ struct bpf_verifier_state {u32 insn_idx;u32 curframe; -bool speculative; +struct bpf_reference_state *refs; +u32 acquired_refs; +u32 active_locks; +u32 active_preempt_locks;bool active_rcu_lock; -u32 active_preempt_lock; + +bool speculative;/* If this state was ever pointed-to by other state's loop_entry field * this flag would be set to true. Used to avoid freeing such states * while they are still in use.
kernel/bpf/log.c +6 -5
@@ -756,6 +756,7 @@ static void print_reg_state(struct bpf_verifier_env *env,void print_verifier_state (struct bpf_verifier_env *env,const struct bpf_func_state *state,bool print_all) { +struct bpf_verifier_state *vstate = env->cur_state;const struct bpf_reg_state *reg;int i;
@@ -843,11 +844,11 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_func_stbreak ; } } -if (state->acquired_refs && state->refs[0 ].id) { -verbose (env," refs=%d" , state->refs[0 ].id); -for (i =1 ; i < state->acquired_refs; i++) -if (state->refs[i].id) -verbose (env,",%d" , state->refs[i].id); +if (vstate->acquired_refs && vstate->refs[0 ].id) { +verbose (env," refs=%d" , vstate->refs[0 ].id); +for (i =1 ; i < vstate->acquired_refs; i++) +if (vstate->refs[i].id) +verbose (env,",%d" , vstate->refs[i].id); }if (state->in_callback_fn)verbose (env," cb" );
kernel/bpf/verifier.c +52 -60
349 lines not on this page · raw message
Signature · 1 line
2.43.5
Re: [PATCH bpf-next v3 1/7] bpf: Consolidate locks and reference state in verifier state
Quoted 8 lines · Kumar Kartikeya Dwivedi
Currently, state for RCU read locks and preemption is in
bpf_verifier_state, while locks and pointer reference state remains inbpf_func_state. There is no particular reason to keep the latter inbpf_func_state. Additionally, it is copied into a new frame's state and copied back to the caller frame's state everytime the verifier processes a pseudo call instruction. This is a bit wasteful, given this state is global for a given verification state / path.
Quoted 5 lines
Move all resource and reference related state in
bpf_verifier_statestructure in this patch, in preparation for introducing new reference state types in the future.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>
lgtm, but please fix the 'print_verifier_state' note below.
Quoted 23 lines
--- include/linux/bpf_verifier.h | 11 ++-- kernel/bpf/log.c | 11 ++-- kernel/bpf/verifier.c | 112 ++++++++++++++++------------------- 3 files changed, 64 insertions(+), 70 deletions(-)include/linux/bpf_verifier.h +0
-3@@ -315,9 +315,6 @@ struct bpf_func_state {u32 callback_depth;/* The following fields should be last. See copy_func_state() */ -int acquired_refs; -int active_locks; -struct bpf_reference_state *refs;/* The state of the stack. Each element of the array describes BPF_REG_SIZE * (i.e. 8) bytes worth of stack memory. * stack[0] represents bytes [*(r10-8)..*(r10-1)] @@ -419,9 +416,13 @@ struct bpf_verifier_state {u32 insn_idx;u32 curframe;
include/linux/bpf_verifier.h struct bpf_verifier_state { · line 425
Quoted 10 lines
include/linux/bpf_verifier.h +6
-2-bool speculative; +struct bpf_reference_state *refs; +u32 acquired_refs; +u32 active_locks; +u32 active_preempt_locks;bool active_rcu_lock; -u32 active_preempt_lock; + +bool speculative;
Nit: pahole says there are two holes here:
$ pahole kernel/bpf/verifier.o
...
struct bpf_verifier_state {
struct bpf_func_state * frame[8]; /* 0 64 */
/* --- cacheline 1 boundary (64 bytes) --- */
struct bpf_verifier_state * parent; /* 64 8 */
u32 branches; /* 72 4 */
u32 insn_idx; /* 76 4 */
u32 curframe; /* 80 4 *//* XXX 4 bytes hole, try to pack */
struct bpf_reference_state * refs; /* 88 8 */
u32 acquired_refs; /* 96 4 */
u32 active_locks; /* 100 4 */
u32 active_preempt_locks; /* 104 4 */
u32 active_irq_id; /* 108 4 */
bool active_rcu_lock; /* 112 1 */
bool speculative; /* 113 1 */
bool used_as_loop_entry; /* 114 1 */
bool in_sleepable; /* 115 1 */
u32 first_insn_idx; /* 116 4 */
u32 last_insn_idx; /* 120 4 */
/* XXX 4 bytes hole, try to pack */
/* --- cacheline 2 boundary (128 bytes) --- */
struct bpf_verifier_state * loop_entry; /* 128 8 */
u32 insn_hist_start; /* 136 4 */
u32 insn_hist_end; /* 140 4 */
u32 dfs_depth; /* 144 4 */
u32 callback_unroll_depth; /* 148 4 */
u32 may_goto_depth; /* 152 4 */
/* size: 160, cachelines: 3, members: 22 */ /* sum members: 148, holes: 2, sum holes: 8 */
maybe move the 'refs' pointer?
e.g. moving it after 'parent' makes both holes disappear.
kernel/bpf/log.c static void print_reg_state(struct bpf_verifier_env *env, · line 759
Quoted 12 lines
include/linux/bpf_verifier.h +0
-0/* If this state was ever pointed-to by other state's loop_entry field * this flag would be set to true. Used to avoid freeing such states * while they are still in use. kernel/bpf/log.c +1
-0@@ -756,6 +756,7 @@ static void print_reg_state(struct bpf_verifier_env *env,void print_verifier_state (struct bpf_verifier_env *env,const struct bpf_func_state *state,bool print_all) { +struct bpf_verifier_state *vstate = env->cur_state;
This is not always true.
For example, __mark_chain_precision does 'print_verifier_state(env, func, true)' for func obtained as 'func = st->frame[fr];' where 'st' iterates over parents of env->cur_state.
Quoted 3 lines
kernel/bpf/log.c +0
-0const struct bpf_reg_state *reg;int i;
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 1/7] bpf: Consolidate locks and reference state in verifier state
Quoted 90 lines · Eduard Zingerman
Quoted 8 lines · Kumar Kartikeya Dwivedi
Currently, state for RCU read locks and preemption is in
bpf_verifier_state, while locks and pointer reference state remains inbpf_func_state. There is no particular reason to keep the latter inbpf_func_state. Additionally, it is copied into a new frame's state and copied back to the caller frame's state everytime the verifier processes a pseudo call instruction. This is a bit wasteful, given this state is global for a given verification state / path.Quoted 5 lines
Move all resource and reference related state in
bpf_verifier_statestructure in this patch, in preparation for introducing new reference state types in the future.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>lgtm, but please fix the '
print_verifier_state' note below.Acked-by Eduard Zingerman <[email protected]>Quoted 23 lines
--- include/linux/bpf_verifier.h | 11 ++-- kernel/bpf/log.c | 11 ++-- kernel/bpf/verifier.c | 112 ++++++++++++++++------------------- 3 files changed, 64 insertions(+), 70 deletions(-)include/linux/bpf_verifier.h +0
-3@@ -315,9 +315,6 @@ struct bpf_func_state {u32 callback_depth;/* The following fields should be last. See copy_func_state() */ -int acquired_refs; -int active_locks; -struct bpf_reference_state *refs;/* The state of the stack. Each element of the array describes BPF_REG_SIZE * (i.e. 8) bytes worth of stack memory. * stack[0] represents bytes [*(r10-8)..*(r10-1)] @@ -419,9 +416,13 @@ struct bpf_verifier_state {u32 insn_idx;u32 curframe;include/linux/bpf_verifier.h struct bpf_verifier_state { · line 425
Quoted 10 lines
include/linux/bpf_verifier.h +6
-2-bool speculative; +struct bpf_reference_state *refs; +u32 acquired_refs; +u32 active_locks; +u32 active_preempt_locks;bool active_rcu_lock; -u32 active_preempt_lock; + +bool speculative;Nit: pahole says there are two holes here:
$ pahole kernel/bpf/verifier.o ... struct bpf_verifier_state { struct bpf_func_state * frame[8]; /* 0 64 */ /* --- cacheline 1 boundary (64 bytes) --- */ struct bpf_verifier_state * parent; /* 64 8 */ u32 branches; /* 72 4 */ u32 insn_idx; /* 76 4 */ u32 curframe; /* 80 4 *//* XXX 4 bytes hole, try to pack */
struct
bpf_reference_state* refs; /* 88 8 */
u32acquired_refs; /* 96 4 */
u32active_locks; /* 100 4 */
u32active_preempt_locks; /* 104 4 */
u32active_irq_id; /* 108 4 */
boolactive_rcu_lock; /* 112 1 */
bool speculative; /* 113 1 */
boolused_as_loop_entry; /* 114 1 */
boolin_sleepable; /* 115 1 */
u32first_insn_idx; /* 116 4 */
u32last_insn_idx; /* 120 4 *//* XXX 4 bytes hole, try to pack */
/* --- cacheline 2 boundary (128 bytes) --- */
structbpf_verifier_state*loop_entry; /* 128 8 */
u32insn_hist_start; /* 136 4 */
u32insn_hist_end; /* 140 4 */
u32dfs_depth; /* 144 4 */
u32callback_unroll_depth; /* 148 4 */
u32may_goto_depth; /* 152 4 */
Quoted 5 lines
/* size: 160, cachelines: 3, members: 22 */ /* sum members: 148, holes: 2, sum holes: 8 */
maybe move the 'refs' pointer?
e.g. moving it after 'parent' makes both holes disappear.
Ack, will fix.
Quoted 10 lines
Quoted 10 lines
include/linux/bpf_verifier.h +0
-0/* If this state was ever pointed-to by other state's loop_entry field * this flag would be set to true. Used to avoid freeing such states * while they are still in use. kernel/bpf/log.c +0
-0@@ -756,6 +756,7 @@ static void print_reg_state(struct bpf_verifier_env *env, void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_func_state *state,
Quoted 8 lines
kernel/bpf/log.c static void print_reg_state(struct bpf_verifier_env *env, · line 759
Quoted 3 lines
kernel/bpf/log.c +1
-0bool print_all) { +struct bpf_verifier_state *vstate = env->cur_state;This is not always true.
For example,__mark_chain_precisiondoes 'print_verifier_state(env, func, true)' for func obtained as 'func = st->frame[fr];' where 'st' iterates over parents ofenv->cur_state.
Looking through the code, I'm thinking the only proper fix is explicitly passing in the verifier state, I was hoping there would be a link from func_state -> verifier_state but it is not the case. Regardless, explicitly passing in the verifier state is probably cleaner. WDYT?
Quoted 7 lines
Quoted 4 lines
kernel/bpf/log.c +0
-0const struct bpf_reg_state *reg;int i;[...]
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 1/7] bpf: Consolidate locks and reference state in verifier state
On Thu, 2024-11-28 at 03:54 +0100, Kumar Kartikeya Dwivedi wrote:
[...]
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/log.c +0
-0@@ -756,6 +756,7 @@ static void print_reg_state(struct bpf_verifier_env *env, void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_func_state *state,Quoted 8 lines
kernel/bpf/log.c static void print_reg_state(struct bpf_verifier_env *env, · line 759
Quoted 3 lines
kernel/bpf/log.c +1
-0bool print_all) { +struct bpf_verifier_state *vstate = env->cur_state;This is not always true.
For example,__mark_chain_precisiondoes 'print_verifier_state(env, func, true)' for func obtained as 'func = st->frame[fr];' where 'st' iterates over parents ofenv->cur_state.
Quoted 4 lines
Looking through the code, I'm thinking the only proper fix is explicitly passing in the verifier state, I was hoping there would be a link from
func_state->verifier_statebut it is not the case. Regardless, explicitly passing in the verifier state is probably cleaner. WDYT?
Seems like it is (I'd also pass the frame number, instead of function state pointer, just to make it clear where the function state comes from, but feel free to ignore this suggestion).
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 1/7] bpf: Consolidate locks and reference state in verifier state
Quoted 18 lines · Eduard Zingerman
On Thu, 2024-11-28 at 03:54 +0100, Kumar Kartikeya Dwivedi wrote:
[...]
Quoted 13 lines
Quoted 12 lines
kernel/bpf/log.c static void print_reg_state(struct bpf_verifier_env *env, · line 759
Quoted 7 lines
kernel/bpf/log.c +1
-0@@ -756,6 +756,7 @@ static void print_reg_state(struct bpf_verifier_env *env,void print_verifier_state (struct bpf_verifier_env *env,const struct bpf_func_state *state,bool print_all) { +struct bpf_verifier_state *vstate = env->cur_state;This is not always true.
For example,__mark_chain_precisiondoes 'print_verifier_state(env, func, true)' for func obtained as 'func = st->frame[fr];' where 'st' iterates over parents ofenv->cur_state.
Quoted 8 lines
Quoted 4 lines
Looking through the code, I'm thinking the only proper fix is explicitly passing in the verifier state, I was hoping there would be a link from
func_state->verifier_statebut it is not the case. Regardless, explicitly passing in the verifier state is probably cleaner. WDYT?Seems like it is (I'd also pass the frame number, instead of function state pointer, just to make it clear where the function state comes from, but feel free to ignore this suggestion).
I made this change, but not passing the frame number: while most call sites have the frame number (or pass curframe), it needs to be obtained explicitly for some, so I think it won't be worth it.
Quoted 3 lines
[...]
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 1/7] bpf: Consolidate locks and reference state in verifier state
Quoted 24 lines · Kumar Kartikeya Dwivedi
Quoted 23 lines · Eduard Zingerman
On Thu, 2024-11-28 at 03:54 +0100, Kumar Kartikeya Dwivedi wrote:
[...]
Quoted 17 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/log.c +0
-0@@ -756,6 +756,7 @@ static void print_reg_state(struct bpf_verifier_env *env, void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_func_state *state,Quoted 8 lines
kernel/bpf/log.c static void print_reg_state(struct bpf_verifier_env *env, · line 759
Quoted 3 lines
kernel/bpf/log.c +1
-0bool print_all) { +struct bpf_verifier_state *vstate = env->cur_state;This is not always true.
For example,__mark_chain_precisiondoes 'print_verifier_state(env, func, true)' for func obtained as 'func = st->frame[fr];' where 'st' iterates over parents ofenv->cur_state.Looking through the code, I'm thinking the only proper fix is explicitly passing in the verifier state, I was hoping there would be a link from
func_state->verifier_statebut it is not the case. Regardless, explicitly passing in the verifier state is probably cleaner. WDYT?
Quoted 7 lines
Quoted 3 lines
Seems like it is (I'd also pass the frame number, instead of function state pointer, just to make it clear where the function state comes from, but feel free to ignore this suggestion).
I made this change, but not passing the frame number: while most call sites have the frame number (or pass curframe), it needs to be obtained explicitly for some, so I think it won't be worth it.
Understood, thank you.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 1/7] bpf: Consolidate locks and reference state in verifier state
Quoted 30 lines · Eduard Zingerman
Quoted 28 lines · Kumar Kartikeya Dwivedi
Quoted 26 lines · Eduard Zingerman
On Thu, 2024-11-28 at 03:54 +0100, Kumar Kartikeya Dwivedi wrote:
[...]
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/log.c +0
-0@@ -756,6 +756,7 @@ static void print_reg_state(struct bpf_verifier_env *env, void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_func_state *state,Quoted 8 lines
kernel/bpf/log.c static void print_reg_state(struct bpf_verifier_env *env, · line 759
Quoted 3 lines
kernel/bpf/log.c +1
-0bool print_all) { +struct bpf_verifier_state *vstate = env->cur_state;This is not always true.
For example,__mark_chain_precisiondoes 'print_verifier_state(env, func, true)' for func obtained as 'func = st->frame[fr];' where 'st' iterates over parents ofenv->cur_state.Quoted 4 lines
Looking through the code, I'm thinking the only proper fix is explicitly passing in the verifier state, I was hoping there would be a link from
func_state->verifier_statebut it is not the case. Regardless, explicitly passing in the verifier state is probably cleaner. WDYT?Seems like it is (I'd also pass the frame number, instead of function state pointer, just to make it clear where the function state comes from, but feel free to ignore this suggestion).
Quoted 6 lines
Quoted 3 lines
I made this change, but not passing the frame number: while most call sites have the frame number (or pass curframe), it needs to be obtained explicitly for some, so I think it won't be worth it.
Understood, thank you.
Ok, scratch the previous reply, I forgot you can actually do func->frameno to get it, I was trying dumb things (like func - st->frame).
I do agree it's better to pass the frameno, just for the off chance that you end up passing vstate and funcs that mismatch. So I ended up making the change in the end. Sorry for the confusion.
Kumar Kartikeya Dwivedi
maintainerPATCH v3 2/7 bpf: Refactor {acquire,release}_reference_state
In reply to Kumar Kartikeya DwivediIn preparation for introducing support for more reference types which have to add and remove reference state, refactor the acquire_reference_state and release_reference_state functions to share common logic.
The acquire_reference_state function simply handles growing the acquired refs and returning the pointer to the new uninitialized element, which can be filled in by the caller.
The release_reference_state function simply erases a reference state entry in the acquired_refs array and shrinks it. The callers are responsible for finding the suitable element by matching on various fields of the reference state and requesting deletion through this function. It is not supposed to be called directly.
Existing callers of release_reference_state were using it to find and remove state for a given ref_obj_id without scrubbing the associated registers in the verifier state. Introduce release_reference_nomark to provide this functionality and convert callers. We now use this new release_reference_nomark function within release_reference as well. It needs to operate on a verifier state instead of taking verifier env as mark_ptr_or_null_regs requires operating on verifier state of the two branches of a NULL condition check, therefore env->cur_state cannot be used directly.
--- kernel/bpf/verifier.c | 113 +++++++++++++++++++++++------------------- 1 file changed, 63 insertions(+), 50 deletions(-)
kernel/bpf/verifier.c +63 -50
@@ -196,7 +196,8 @@ struct bpf_verifier_stack_elem {#define BPF_PRIV_STACK_MIN_SIZE 64 -static int acquire_reference_state (struct bpf_verifier_env *env,int insn_idx); +static int acquire_reference (struct bpf_verifier_env *env,int insn_idx); +static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id);static int release_reference (struct bpf_verifier_env *env,int ref_obj_id);static void invalidate_non_owning_refs (struct bpf_verifier_env *env);static bool in_rbtree_lock_required_cb (struct bpf_verifier_env *env);
@@ -771,7 +772,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_if (clone_ref_obj_id) id = clone_ref_obj_id;else - id =acquire_reference_state (env, insn_idx); + id =acquire_reference (env, insn_idx);if (id <0 )return id;
@@ -1033,7 +1034,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,if (spi <0 )return spi; - id =acquire_reference_state (env, insn_idx); + id =acquire_reference (env, insn_idx);if (id <0 )return id;
@@ -1349,77 +1350,69 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state* On success, returns a valid pointer id to associate with the register * On failure, returns a negative errno. */ -static int acquire_reference_state (struct bpf_verifier_env *env,int insn_idx) +static struct bpf_reference_state *acquire_reference_state (struct bpf_verifier_env *env,int insn_idx,bool gen_id) {struct bpf_verifier_state *state = env->cur_state;int new_ofs = state->acquired_refs; -int id, err; +int err; err =resize_reference_state (state, state->acquired_refs +1 );if (err) -return err; - id = ++env->id_gen; - state->refs[new_ofs].type =REF_TYPE_PTR ; - state->refs[new_ofs].id = id; +return NULL ; +if (gen_id) + state->refs[new_ofs].id = ++env->id_gen; state->refs[new_ofs].insn_idx = insn_idx; -return id; +return &state->refs[new_ofs]; +} + +static int acquire_reference (struct bpf_verifier_env *env,int insn_idx) +{ +struct bpf_reference_state *s; + + s =acquire_reference_state (env, insn_idx, true); +if (!s) +return -ENOMEM ; + s->type =REF_TYPE_PTR ; +return s->id; }static int acquire_lock_state (struct bpf_verifier_env *env,int insn_idx,enum ref_state_type type,int id,void *ptr) {struct bpf_verifier_state *state = env->cur_state; -int new_ofs = state->acquired_refs; -int err; +struct bpf_reference_state *s; - err =resize_reference_state (state, state->acquired_refs +1 ); -if (err) -return err; - state->refs[new_ofs].type = type; - state->refs[new_ofs].id = id; - state->refs[new_ofs].insn_idx = insn_idx; - state->refs[new_ofs].ptr = ptr; + s =acquire_reference_state (env, insn_idx, false); + s->type = type; + s->id = id; + s->ptr = ptr; state->active_locks++;return 0 ; } -/* release function corresponding to acquire_reference_state(). Idempotent. */ -static int release_reference_state (struct bpf_verifier_state *state,int ptr_id) +static void release_reference_state (struct bpf_verifier_state *state,int idx) { -int i, last_idx; +int last_idx; last_idx = state->acquired_refs -1 ; -for (i =0 ; i < state->acquired_refs; i++) { -if (state->refs[i].type !=REF_TYPE_PTR ) -continue ; -if (state->refs[i].id == ptr_id) { -if (last_idx && i != last_idx) -memcpy (&state->refs[i], &state->refs[last_idx], -sizeof (*state->refs)); -memset (&state->refs[last_idx],0 ,sizeof (*state->refs)); - state->acquired_refs--; -return 0 ; - } - } -return -EINVAL ; +if (last_idx && idx != last_idx) +memcpy (&state->refs[idx], &state->refs[last_idx],sizeof (*state->refs)); +memset (&state->refs[last_idx],0 ,sizeof (*state->refs)); + state->acquired_refs--; +return ; }static int release_lock_state (struct bpf_verifier_state *state,int type,int id,void *ptr) { -int i, last_idx; +int i; - last_idx = state->acquired_refs -1 ;for (i =0 ; i < state->acquired_refs; i++) {if (state->refs[i].type != type)continue ;if (state->refs[i].id == id && state->refs[i].ptr == ptr) { -if (last_idx && i != last_idx) -memcpy (&state->refs[i], &state->refs[last_idx], -sizeof (*state->refs)); -memset (&state->refs[last_idx],0 ,sizeof (*state->refs)); - state->acquired_refs--; +release_reference_state (state, i); state->active_locks--;return 0 ; }
@@ -9666,21 +9659,41 @@ static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range reg->range =AT_PKT_END ; } +static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id) +{ +int i; + +for (i =0 ; i < state->acquired_refs; i++) { +if (state->refs[i].type !=REF_TYPE_PTR ) +continue ; +if (state->refs[i].id == ref_obj_id) { +release_reference_state (state, i); +return 0 ; + } + } +return -EINVAL ; +} +/* The pointer with the specified id has released its reference to kernel * resources. Identify all copies of the same pointer and clear the reference. +* +* This is the release function corresponding to acquire_reference(). Idempotent. +* The 'mark' boolean is used to optionally skip scrubbing registers matching +* the ref_obj_id, in case they need to be switched to some other type instead +* of havoc scalar value. */ -static int release_reference (struct bpf_verifier_env *env, -int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) { +struct bpf_verifier_state *vstate = env->cur_state;struct bpf_func_state *state;struct bpf_reg_state *reg;int err; - err =release_reference_state (env->cur_state, ref_obj_id); + err =release_reference_nomark (vstate, ref_obj_id);if (err)return err; -bpf_for_each_reg_in_vstate (env->cur_state, state, reg, ({ +bpf_for_each_reg_in_vstate (vstate, state, reg, ({if (reg->ref_obj_id == ref_obj_id)mark_reg_invalid (env, reg); }));
@@ -10774,7 +10787,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnstruct bpf_func_state *state;struct bpf_reg_state *reg; - err =release_reference_state (env->cur_state, ref_obj_id); + err =release_reference_nomark (env->cur_state, ref_obj_id);if (!err) { bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ if (reg->ref_obj_id == ref_obj_id) {
@@ -11107,7 +11120,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn/* For release_reference() */ regs[BPF_REG_0 ].ref_obj_id = meta.ref_obj_id; }else if (is_acquire_function (func_id, meta.map_ptr)) { -int id =acquire_reference_state (env, insn_idx); +int id =acquire_reference (env, insn_idx);if (id <0 )return id;
@@ -13087,7 +13100,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, }mark_btf_func_reg_size (env,BPF_REG_0 ,sizeof (void *));if (is_kfunc_acquire (&meta)) { -int id =acquire_reference_state (env, insn_idx); +int id =acquire_reference (env, insn_idx);if (id <0 )return id;
@@ -15387,7 +15400,7 @@ static void mark_ptr_or_null_regs(struct bpf_verifier_state *vstate, u32 regno,* No one could have freed the reference state before * doing the NULL check. */ -WARN_ON_ONCE (release_reference_state (vstate, id)); +WARN_ON_ONCE (release_reference_nomark (vstate, id)); bpf_for_each_reg_in_vstate(vstate, state, reg, ({ mark_ptr_or_null_reg(state, reg, id, is_null);
Signature · 1 line
2.43.5
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 2/7] bpf: Refactor {acquire,release}_reference_state
On Wed, 2024-11-27 at 08:58 -0800, Kumar Kartikeya Dwivedi wrote:
Overall looks good, but please take a look at a few notes below.
[...]
Quoted 11 lines
kernel/bpf/verifier.c +2
-2@@ -1349,77 +1350,69 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state* On success, returns a valid pointer id to associate with the register * On failure, returns a negative errno. */ -static int acquire_reference_state (struct bpf_verifier_env *env,int insn_idx) +static struct bpf_reference_state *acquire_reference_state (struct bpf_verifier_env *env,int insn_idx,bool gen_id) {struct bpf_verifier_state *state = env->cur_state;int new_ofs = state->acquired_refs; -int id, err; +int err;
kernel/bpf/verifier.c static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state · line 1363
Quoted 10 lines
kernel/bpf/verifier.c +3
-4err =resize_reference_state (state, state->acquired_refs +1 );if (err) -return err; - id = ++env->id_gen; - state->refs[new_ofs].type =REF_TYPE_PTR ; - state->refs[new_ofs].id = id; +return NULL ; +if (gen_id) + state->refs[new_ofs].id = ++env->id_gen;
Nit: state->refs[new_ods].id might end up with garbage value if 'gen_id' is false.
The resize_reference_state() uses realloc_array(),
which allocates memory with GFP_KERNEL, but without __GFP_ZERO flag. This is not a problem with current patch, as you always check reference type before checking id, but most of the data strucures in verifier are zero initialized just in case.
Quoted 5 lines
kernel/bpf/verifier.c +2
-1state->refs[new_ofs].insn_idx = insn_idx; -return id; +return &state->refs[new_ofs]; +}
[...]
Quoted 17 lines
kernel/bpf/verifier.c +2
-12-/* release function corresponding to acquire_reference_state(). Idempotent. */ -static int release_reference_state (struct bpf_verifier_state *state,int ptr_id) +static void release_reference_state (struct bpf_verifier_state *state,int idx) { -int i, last_idx; +int last_idx; last_idx = state->acquired_refs -1 ; -for (i =0 ; i < state->acquired_refs; i++) { -if (state->refs[i].type !=REF_TYPE_PTR ) -continue ; -if (state->refs[i].id == ptr_id) { -if (last_idx && i != last_idx) -memcpy (&state->refs[i], &state->refs[last_idx], -sizeof (*state->refs)); -memset (&state->refs[last_idx],0 ,sizeof (*state->refs)); - state->acquired_refs--;
kernel/bpf/verifier.c static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state · line 1405
Quoted 10 lines
kernel/bpf/verifier.c +5
-4-return 0 ; - } - } -return -EINVAL ; +if (last_idx && idx != last_idx) +memcpy (&state->refs[idx], &state->refs[last_idx],sizeof (*state->refs)); +memset (&state->refs[last_idx],0 ,sizeof (*state->refs)); + state->acquired_refs--; +return ; }
Such implementation replaces element at 'idx' with element at 'last_idx'. If the intention is to use 'state->refs' as a stack of acquired irq flags, the stack property would be broken by this trick.
E.g. consider array [a, b, c, d] where 'idx' points to 'b', after release_reference_state() the array would become [a, d, c]. You need to do 'memmove' instead.
[...]
Quoted 14 lines
kernel/bpf/verifier.c +10
-0@@ -9666,21 +9659,41 @@ static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range reg->range =AT_PKT_END ; } +static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id) +{ +int i; + +for (i =0 ; i < state->acquired_refs; i++) { +if (state->refs[i].type !=REF_TYPE_PTR ) +continue ; +if (state->refs[i].id == ref_obj_id) { +release_reference_state (state, i); +return 0 ;
kernel/bpf/verifier.c static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range · line 9681
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+ } + } +return -EINVAL ; +} +/* The pointer with the specified id has released its reference to kernel * resources. Identify all copies of the same pointer and clear the reference. +* +* This is the release function corresponding to acquire_reference(). Idempotent. +* The 'mark' boolean is used to optionally skip scrubbing registers matching
^^^^^^
Nit: this is probably a remnant of some older patch revision,
function no longer takes 'mark' parameter.
Quoted 7 lines
kernel/bpf/verifier.c +3
-2+* the ref_obj_id, in case they need to be switched to some other type instead +* of havoc scalar value. */ -static int release_reference (struct bpf_verifier_env *env, -int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) {
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 2/7] bpf: Refactor {acquire,release}_reference_state
Quoted 19 lines · Eduard Zingerman
On Wed, 2024-11-27 at 08:58 -0800, Kumar Kartikeya Dwivedi wrote:
Overall looks good, but please take a look at a few notes below.
[...]
Quoted 12 lines
kernel/bpf/verifier.c +2
-2@@ -1349,77 +1350,69 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state* On success, returns a valid pointer id to associate with the register * On failure, returns a negative errno. */ -static int acquire_reference_state (struct bpf_verifier_env *env,int insn_idx) +static struct bpf_reference_state *acquire_reference_state (struct bpf_verifier_env *env,int insn_idx,bool gen_id) {struct bpf_verifier_state *state = env->cur_state;int new_ofs = state->acquired_refs; -int id, err; +int err;
Quoted 16 lines
kernel/bpf/verifier.c static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state · line 1363
Quoted 9 lines
kernel/bpf/verifier.c +3
-4err =resize_reference_state (state, state->acquired_refs +1 );if (err) -return err; - id = ++env->id_gen; - state->refs[new_ofs].type =REF_TYPE_PTR ; - state->refs[new_ofs].id = id; +return NULL ; +if (gen_id) + state->refs[new_ofs].id = ++env->id_gen;Nit: state->refs[new_ods].id might end up with garbage value if '
gen_id' is false.
Theresize_reference_state()usesrealloc_array(),
which allocates memory withGFP_KERNEL, but without__GFP_ZEROflag. This is not a problem with current patch, as you always check reference type before checking id, but most of the data strucures in verifier are zero initialized just in case.
We end up assigning to s->id if gen_id is false, e.g. acquire_lock_state, so I think we'll be fine without __GFP_ZERO.
Quoted 35 lines
Quoted 6 lines
kernel/bpf/verifier.c +2
-1state->refs[new_ofs].insn_idx = insn_idx; -return id; +return &state->refs[new_ofs]; +}[...]
Quoted 26 lines
kernel/bpf/verifier.c +7
-16-/* release function corresponding to acquire_reference_state(). Idempotent. */ -static int release_reference_state (struct bpf_verifier_state *state,int ptr_id) +static void release_reference_state (struct bpf_verifier_state *state,int idx) { -int i, last_idx; +int last_idx; last_idx = state->acquired_refs -1 ; -for (i =0 ; i < state->acquired_refs; i++) { -if (state->refs[i].type !=REF_TYPE_PTR ) -continue ; -if (state->refs[i].id == ptr_id) { -if (last_idx && i != last_idx) -memcpy (&state->refs[i], &state->refs[last_idx], -sizeof (*state->refs)); -memset (&state->refs[last_idx],0 ,sizeof (*state->refs)); - state->acquired_refs--; -return 0 ; - } - } -return -EINVAL ; +if (last_idx && idx != last_idx) +memcpy (&state->refs[idx], &state->refs[last_idx],sizeof (*state->refs)); +memset (&state->refs[last_idx],0 ,sizeof (*state->refs)); + state->acquired_refs--; +return ;
Quoted 9 lines
Quoted 1 line
}
Such implementation replaces element at 'idx' with element at '
last_idx'. If the intention is to use 'state->refs' as a stack of acquired irq flags, the stack property would be broken by this trick.
E.g. consider array [a, b, c, d] where 'idx' points to 'b', afterrelease_reference_state()the array would become [a, d, c]. You need to do 'memmove' instead.
Wow, great catch. Thanks for spotting this. I'll fix this and let me see if I can add a selftest that would've triggered this particular pattern.
Quoted 27 lines
[...]
Quoted 14 lines
kernel/bpf/verifier.c +10
-0@@ -9666,21 +9659,41 @@ static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range reg->range =AT_PKT_END ; } +static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id) +{ +int i; + +for (i =0 ; i < state->acquired_refs; i++) { +if (state->refs[i].type !=REF_TYPE_PTR ) +continue ; +if (state->refs[i].id == ref_obj_id) { +release_reference_state (state, i); +return 0 ;kernel/bpf/verifier.c static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range · line 9681
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+ } + } +return -EINVAL ; +} +/* The pointer with the specified id has released its reference to kernel * resources. Identify all copies of the same pointer and clear the reference. +* +* This is the release function corresponding to acquire_reference(). Idempotent. +* The 'mark' boolean is used to optionally skip scrubbing registers matching ^^^^^^
Quoted 2 lines
Nit: this is probably a remnant of some older patch revision,
function no longer takes 'mark' parameter.
Yeah, this is a leftover. Sorry about that. Will fix.
Quoted 11 lines
Quoted 8 lines
kernel/bpf/verifier.c +3
-2+* the ref_obj_id, in case they need to be switched to some other type instead +* of havoc scalar value. */ -static int release_reference (struct bpf_verifier_env *env, -int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) {[...]
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 2/7] bpf: Refactor {acquire,release}_reference_state
Quoted 37 lines · Kumar Kartikeya Dwivedi
Quoted 19 lines · Eduard Zingerman
On Wed, 2024-11-27 at 08:58 -0800, Kumar Kartikeya Dwivedi wrote:
Overall looks good, but please take a look at a few notes below.
[...]
Quoted 12 lines
kernel/bpf/verifier.c +2
-2@@ -1349,77 +1350,69 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state* On success, returns a valid pointer id to associate with the register * On failure, returns a negative errno. */ -static int acquire_reference_state (struct bpf_verifier_env *env,int insn_idx) +static struct bpf_reference_state *acquire_reference_state (struct bpf_verifier_env *env,int insn_idx,bool gen_id) {struct bpf_verifier_state *state = env->cur_state;int new_ofs = state->acquired_refs; -int id, err; +int err;Quoted 16 lines
kernel/bpf/verifier.c static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state · line 1363
Quoted 9 lines
kernel/bpf/verifier.c +3
-4err =resize_reference_state (state, state->acquired_refs +1 );if (err) -return err; - id = ++env->id_gen; - state->refs[new_ofs].type =REF_TYPE_PTR ; - state->refs[new_ofs].id = id; +return NULL ; +if (gen_id) + state->refs[new_ofs].id = ++env->id_gen;Nit: state->refs[new_ods].id might end up with garbage value if '
gen_id' is false.
Theresize_reference_state()usesrealloc_array(),
which allocates memory withGFP_KERNEL, but without__GFP_ZEROflag. This is not a problem with current patch, as you always check reference type before checking id, but most of the data strucures in verifier are zero initialized just in case.
Quoted 2 lines
We end up assigning to
s->idifgen_idis false, e.g.acquire_lock_state, so I think we'll be fine without__GFP_ZERO.
Oh, I see, thank you for explaining.
[...]
Kumar Kartikeya Dwivedi
maintainerPATCH v3 3/7 bpf: Refactor mark_{dynptr,iter}_read
In reply to Kumar Kartikeya DwivediThere is possibility of sharing code between mark_dynptr_read and mark_iter_read for updating liveness information of their stack slots. Consolidate common logic into mark_stack_slot_obj_read function in preparation for the next patch which needs the same logic for its own stack slots.
--- kernel/bpf/verifier.c | 43 +++++++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 22 deletions(-)
kernel/bpf/verifier.c +21 -22
@@ -3192,10 +3192,27 @@ static int mark_reg_read(struct bpf_verifier_env *env,return 0 ; } -static int mark_dynptr_read (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +static int mark_stack_slot_obj_read (struct bpf_verifier_env *env,struct bpf_reg_state *reg, +int spi,int nr_slots) {struct bpf_func_state *state =func (env, reg); -int spi, ret; +int err, i; + +for (i =0 ; i < nr_slots; i++) { +struct bpf_reg_state *st = &state->stack[spi - i].spilled_ptr; + + err =mark_reg_read (env, st, st->parent,REG_LIVE_READ64 ); +if (err) +return err; + +mark_stack_slot_scratched (env, spi - i); + } +return 0 ; +} + +static int mark_dynptr_read (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +{ +int spi;/* For CONST_PTR_TO_DYNPTR, it must have already been done by * check_reg_arg in check_helper_call and mark_btf_func_reg_size in
@@ -3210,31 +3227,13 @@ static int mark_dynptr_read(struct bpf_verifier_env *env, struct bpf_reg_state ** bounds and spi is the first dynptr slot. Simply mark stack slot as * read. */ - ret =mark_reg_read (env, &state->stack[spi].spilled_ptr, - state->stack[spi].spilled_ptr.parent,REG_LIVE_READ64 ); -if (ret) -return ret; -return mark_reg_read (env, &state->stack[spi -1 ].spilled_ptr, - state->stack[spi -1 ].spilled_ptr.parent,REG_LIVE_READ64 ); +return mark_stack_slot_obj_read (env, reg, spi,BPF_DYNPTR_NR_SLOTS ); }static int mark_iter_read (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int spi,int nr_slots) { -struct bpf_func_state *state =func (env, reg); -int err, i; - -for (i =0 ; i < nr_slots; i++) { -struct bpf_reg_state *st = &state->stack[spi - i].spilled_ptr; - - err =mark_reg_read (env, st, st->parent,REG_LIVE_READ64 ); -if (err) -return err; - -mark_stack_slot_scratched (env, spi - i); - } - -return 0 ; +return mark_stack_slot_obj_read (env, reg, spi, nr_slots); }/* This function is supposed to be used by the following 32-bit optimization
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v3 4/7 bpf: Introduce support for bpf_local_irq_{save,restore}
In reply to Kumar Kartikeya DwivediTeach the verifier about IRQ-disabled sections through the introduction of two new kfuncs, bpf_local_irq_save, to save IRQ state and disable them, and bpf_local_irq_restore, to restore IRQ state and enable them back again.
For the purposes of tracking the saved IRQ state, the verifier is taught about a new special object on the stack of type STACK_IRQ_FLAG. This is a 8 byte value which saves the IRQ flags which are to be passed back to the IRQ restore kfunc.
Renumber the enums for REF_TYPE_* to simplify the check in find_lock_state, filtering out non-lock types as they grow will become cumbersome and is unecessary.
To track a dynamic number of IRQ-disabled regions and their associated saved states, a new resource type RES_TYPE_IRQ is introduced, which its state management functions: acquire_irq_state and release_irq_state, taking advantage of the refactoring and clean ups made in earlier commits.
One notable requirement of the kernel's IRQ save and restore API is that they cannot happen out of order. For this purpose, when releasing reference we keep track of the prev_id we saw with REF_TYPE_IRQ. Since reference states are inserted in increasing order of the index, this is used to remember the ordering of acquisitions of IRQ saved states, so that we maintain a logical stack in acquisition order of resource identities, and can enforce LIFO ordering when restoring IRQ state. The top of the stack is maintained using bpf_verifier_state's active_irq_id.
The logic to detect initialized and unitialized irq flag slots, marking and unmarking is similar to how it's done for iterators. No additional checks are needed in refsafe for REF_TYPE_IRQ, apart from the usual check_id satisfiability check on the ref[i].id. We have to perform the same check_ids check on state->active_irq_id as well.
The kfuncs themselves are plain wrappers over local_irq_save and local_irq_restore macros.
--- include/linux/bpf_verifier.h | 8 +- kernel/bpf/helpers.c | 17 +++ kernel/bpf/log.c | 1 + kernel/bpf/verifier.c | 279 ++++++++++++++++++++++++++++++++++- 4 files changed, 302 insertions(+), 3 deletions(-)
include/linux/bpf_verifier.h +6 -2
@@ -233,6 +233,7 @@ enum bpf_stack_slot_type {*/ STACK_DYNPTR ,STACK_ITER , + STACK_IRQ_FLAG, };#define BPF_REG_SIZE 8/* size of eBPF register in bytes */
@@ -254,8 +255,10 @@ struct bpf_reference_state {* default to pointer reference on zero initialization of a state. */ enum ref_state_type { -REF_TYPE_PTR =0 , -REF_TYPE_LOCK , +REF_TYPE_PTR =1 , +REF_TYPE_IRQ =2 , + +REF_TYPE_LOCK =3 , } type;/* Track each reference created with a unique id, even if the same * instruction creates the reference multiple times (eg, via CALL).
@@ -420,6 +423,7 @@ struct bpf_verifier_state {u32 acquired_refs;u32 active_locks;u32 active_preempt_locks; +u32 active_irq_id;bool active_rcu_lock;bool speculative;
kernel/bpf/helpers.c +17 -0
@@ -3057,6 +3057,21 @@ __bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __userreturn ret +1 ; } +/* Keep unsinged long in prototype so that kfunc is usable when emitted to +* vmlinux.h in BPF programs directly, but note that while in BPF prog, the +* unsigned long always points to 8-byte region on stack, the kernel may only +* read and write the 4-bytes on 32-bit. +*/ +__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag) +{ +local_irq_save (*flags__irq_flag); +} + +__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag) +{ +local_irq_restore (*flags__irq_flag); +} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (generic_btf_ids)
@@ -3149,6 +3164,8 @@ BTF_ID_FLAGS(func, bpf_get_kmem_cache) BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_local_irq_save) +BTF_ID_FLAGS(func, bpf_local_irq_restore)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
kernel/bpf/log.c +1 -0
@@ -537,6 +537,7 @@ static char slot_type_char[] = { [STACK_ZERO] = '0', [STACK_DYNPTR] = 'd', [STACK_ITER] = 'i', + [STACK_IRQ_FLAG ] ='f' }; static void print_liveness(struct bpf_verifier_env *env,
kernel/bpf/verifier.c +278 -1
421 lines not on this page · raw message
Signature · 1 line
2.43.5
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 4/7] bpf: Introduce support for bpf_local_irq_{save,restore}
Quoted 35 lines · Kumar Kartikeya Dwivedi
Teach the verifier about IRQ-disabled sections through the introduction of two new kfuncs,
bpf_local_irq_save, to save IRQ state and disable them, andbpf_local_irq_restore, to restore IRQ state and enable them back again.For the purposes of tracking the saved IRQ state, the verifier is taught about a new special object on the stack of type
STACK_IRQ_FLAG. This is a 8 byte value which saves the IRQ flags which are to be passed back to the IRQ restore kfunc.Renumber the enums for REF_TYPE_* to simplify the check in
find_lock_state, filtering out non-lock types as they grow will become cumbersome and is unecessary.To track a dynamic number of IRQ-disabled regions and their associated saved states, a new resource type
RES_TYPE_IRQis introduced, which its state management functions:acquire_irq_stateandrelease_irq_state, taking advantage of the refactoring and clean ups made in earlier commits.One notable requirement of the kernel's IRQ save and restore API is that they cannot happen out of order. For this purpose, when releasing reference we keep track of the
prev_idwe saw withREF_TYPE_IRQ. Since reference states are inserted in increasing order of the index, this is used to remember the ordering of acquisitions of IRQ saved states, so that we maintain a logical stack in acquisition order of resource identities, and can enforce LIFO ordering when restoring IRQ state. The top of the stack is maintained using bpf_verifier_state'sactive_irq_id.The logic to detect initialized and unitialized irq flag slots, marking and unmarking is similar to how it's done for iterators. No additional checks are needed in refsafe for
REF_TYPE_IRQ, apart from the usualcheck_idsatisfiability check on the ref[i].id. We have to perform the samecheck_idscheck onstate->active_irq_idas well.
Quoted 5 lines
The kfuncs themselves are plain wrappers over
local_irq_saveandlocal_irq_restoremacros.Acked-by Eduard Zingerman <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>
Sorry, two more nits below.
[...]
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+static int unmark_stack_slot_irq_flag (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +{ +struct bpf_func_state *state =func (env, reg); +struct bpf_stack_state *slot; +struct bpf_reg_state *st; +int spi, i, err; + + spi =irq_flag_get_spi (env, reg);
kernel/bpf/verifier.c static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · line 1216
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (spi <0 ) +return spi; + + slot = &state->stack[spi]; + st = &slot->spilled_ptr; + + err =release_irq_state (env->cur_state, st->ref_obj_id); +WARN_ON_ONCE (err && err != -EACCES ); +if (err) { +verbose (env,"cannot restore irq state out of order\n" );
Nit: maybe also print acquire_irq_id and an instruction where it was acquired?
Quoted 18 lines
kernel/bpf/verifier.c +18
-0+return err; + } + +__mark_reg_not_init (env, st); + +/* see unmark_stack_slots_dynptr() for why we need to set REG_LIVE_WRITTEN */ + st->live |=REG_LIVE_WRITTEN ; + +for (i =0 ; i <BPF_REG_SIZE ; i++) + slot->slot_type[i] =STACK_INVALID ; + +mark_stack_slot_scratched (env, spi); +return 0 ; +} + +static bool is_irq_flag_reg_valid_uninit (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +{ +struct bpf_func_state *state =func (env, reg);
kernel/bpf/verifier.c static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · line 1244
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+struct bpf_stack_state *slot; +int spi, i; + +/* For -ERANGE (i.e. spi not falling into allocated stack slots), we +* will do check_mem_access to check and update stack bounds later, so +* return true for that case. +*/ + spi =irq_flag_get_spi (env, reg); +if (spi == -ERANGE ) +return true;
Nit: is it possible to swap is_irq_flag_reg_valid_uninit() and
check_mem_access(), so that ERANGE special case would be not needed?
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (spi <0 ) +return false; + + slot = &state->stack[spi]; + +for (i =0 ; i <BPF_REG_SIZE ; i++) +if (slot->slot_type[i] ==STACK_IRQ_FLAG ) +return false; +return true; +}
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 4/7] bpf: Introduce support for bpf_local_irq_{save,restore}
Quoted 59 lines · Eduard Zingerman
Quoted 35 lines · Kumar Kartikeya Dwivedi
Teach the verifier about IRQ-disabled sections through the introduction of two new kfuncs,
bpf_local_irq_save, to save IRQ state and disable them, andbpf_local_irq_restore, to restore IRQ state and enable them back again.For the purposes of tracking the saved IRQ state, the verifier is taught about a new special object on the stack of type
STACK_IRQ_FLAG. This is a 8 byte value which saves the IRQ flags which are to be passed back to the IRQ restore kfunc.Renumber the enums for REF_TYPE_* to simplify the check in
find_lock_state, filtering out non-lock types as they grow will become cumbersome and is unecessary.To track a dynamic number of IRQ-disabled regions and their associated saved states, a new resource type
RES_TYPE_IRQis introduced, which its state management functions:acquire_irq_stateandrelease_irq_state, taking advantage of the refactoring and clean ups made in earlier commits.One notable requirement of the kernel's IRQ save and restore API is that they cannot happen out of order. For this purpose, when releasing reference we keep track of the
prev_idwe saw withREF_TYPE_IRQ. Since reference states are inserted in increasing order of the index, this is used to remember the ordering of acquisitions of IRQ saved states, so that we maintain a logical stack in acquisition order of resource identities, and can enforce LIFO ordering when restoring IRQ state. The top of the stack is maintained using bpf_verifier_state'sactive_irq_id.The logic to detect initialized and unitialized irq flag slots, marking and unmarking is similar to how it's done for iterators. No additional checks are needed in refsafe for
REF_TYPE_IRQ, apart from the usualcheck_idsatisfiability check on the ref[i].id. We have to perform the samecheck_idscheck onstate->active_irq_idas well.Quoted 5 lines
The kfuncs themselves are plain wrappers over
local_irq_saveandlocal_irq_restoremacros.Acked-by Eduard Zingerman <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>Sorry, two more nits below.
[...]
Quoted 12 lines
kernel/bpf/verifier.c +12
-0+static int unmark_stack_slot_irq_flag (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +{ +struct bpf_func_state *state =func (env, reg); +struct bpf_stack_state *slot; +struct bpf_reg_state *st; +int spi, i, err; + + spi =irq_flag_get_spi (env, reg); +if (spi <0 ) +return spi; + + slot = &state->stack[spi];
Quoted 8 lines
kernel/bpf/verifier.c static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · line 1216
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+ st = &slot->spilled_ptr; + + err =release_irq_state (env->cur_state, st->ref_obj_id); +WARN_ON_ONCE (err && err != -EACCES ); +if (err) { +verbose (env,"cannot restore irq state out of order\n" );Nit: maybe also print
acquire_irq_idand an instruction where it was acquired?
Ack. For printing the insn_idx, I guess just search in the refs array?
Quoted 24 lines
Quoted 24 lines
kernel/bpf/verifier.c +23
-0+return err; + } + +__mark_reg_not_init (env, st); + +/* see unmark_stack_slots_dynptr() for why we need to set REG_LIVE_WRITTEN */ + st->live |=REG_LIVE_WRITTEN ; + +for (i =0 ; i <BPF_REG_SIZE ; i++) + slot->slot_type[i] =STACK_INVALID ; + +mark_stack_slot_scratched (env, spi); +return 0 ; +} + +static bool is_irq_flag_reg_valid_uninit (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +{ +struct bpf_func_state *state =func (env, reg); +struct bpf_stack_state *slot; +int spi, i; + +/* For -ERANGE (i.e. spi not falling into allocated stack slots), we +* will do check_mem_access to check and update stack bounds later, so
Quoted 9 lines
kernel/bpf/verifier.c static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · line 1244
Quoted 5 lines
kernel/bpf/verifier.c +5
-0+* return true for that case. +*/ + spi =irq_flag_get_spi (env, reg); +if (spi == -ERANGE ) +return true;Nit: is it possible to swap
is_irq_flag_reg_valid_uninit()and
check_mem_access(), so that ERANGE special case would be not needed?
I don't think so. For dynptr, iter, irq, ERANGE indicates stack needs to be grown, so check_mem_access will naturally do that when writing. When not ERANGE, we need to catch cases where we have a bad slot_type. If we overwrote it with check_mem_access, then it would scrub the slot type as well.
When I fixed this stuff for dynptr, we had to additionally destroy_if_dynptr_stack_slot because it wasn't required to 'release' a dynptr when overwriting it.
Andrii made sure this was necessary for iters so now slot_type == STACK_ITER is just rejected instead of overwrite without a destroy operation.
Similar idea is followed for irq flag.
Just paging in context for all this, but I may be missing if you have something in mind.
Quoted 13 lines
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (spi <0 ) +return false; + + slot = &state->stack[spi]; + +for (i =0 ; i <BPF_REG_SIZE ; i++) +if (slot->slot_type[i] ==STACK_IRQ_FLAG ) +return false; +return true; +}[...]
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 4/7] bpf: Introduce support for bpf_local_irq_{save,restore}
On Thu, 2024-11-28 at 05:39 +0100, Kumar Kartikeya Dwivedi wrote:
[...]
Quoted 32 lines
Quoted 8 lines
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+static bool is_irq_flag_reg_valid_uninit (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +{ +struct bpf_func_state *state =func (env, reg); +struct bpf_stack_state *slot; +int spi, i; + +/* For -ERANGE (i.e. spi not falling into allocated stack slots), we +* will do check_mem_access to check and update stack bounds later, so Quoted 9 lines
kernel/bpf/verifier.c static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · line 1244
Quoted 5 lines
kernel/bpf/verifier.c +5
-0+* return true for that case. +*/ + spi =irq_flag_get_spi (env, reg); +if (spi == -ERANGE ) +return true;Nit: is it possible to swap
is_irq_flag_reg_valid_uninit()and
check_mem_access(), so that ERANGE special case would be not needed?I don't think so. For dynptr, iter, irq, ERANGE indicates stack needs to be grown, so
check_mem_accesswill naturally do that when writing. When not ERANGE, we need to catch cases where we have a badslot_type. If we overwrote it withcheck_mem_access, then it would scrub the slot type as well.When I fixed this stuff for dynptr, we had to additionally
destroy_if_dynptr_stack_slotbecause it wasn't required to 'release' a dynptr when overwriting it.
Andrii made sure this was necessary for iters so nowslot_type==STACK_ITERis just rejected instead of overwrite without a destroy operation.
Similar idea is followed for irq flag.
Quoted 2 lines
Just paging in context for all this, but I may be missing if you have something in mind.
I see, makes sense. And is_dynptr_reg_valid_uninit() has the same check. Thank you for explaining.
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (spi <0 ) +return false; + + slot = &state->stack[spi]; + +for (i =0 ; i <BPF_REG_SIZE ; i++) +if (slot->slot_type[i] ==STACK_IRQ_FLAG ) +return false; +return true; +}[...]
Kumar Kartikeya Dwivedi
maintainerPATCH v3 5/7 bpf: Improve verifier log for resource leak on exit
In reply to Kumar Kartikeya DwivediThe verifier log when leaking resources on BPF_EXIT may be a bit confusing, as it's a problem only when finally existing from the main prog, not from any of the subprogs. Hence, update the verifier error string and the corresponding selftests matching on it.
--- kernel/bpf/verifier.c | 2 +- .../testing/selftests/bpf/progs/exceptions_fail.c | 4 ++-- tools/testing/selftests/bpf/progs/preempt_lock.c | 14 +++++++------- .../selftests/bpf/progs/verifier_spin_lock.c | 2 +- 4 files changed, 11 insertions(+), 11 deletions(-)
kernel/bpf/verifier.c +1 -1
@@ -19088,7 +19088,7 @@ static int do_check(struct bpf_verifier_env *env)* match caller reference state when it exits. */ err =check_resource_leak (env, exception_exit, !env->cur_state->curframe, -"BPF_EXIT instruction" ); +"BPF_EXIT instruction in main prog" );if (err)return err;
tools/testing/selftests/bpf/progs/exceptions_fail.c +2 -2
@@ -131,7 +131,7 @@ int reject_subprog_with_lock(void *ctx) } SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_rcu_read_lock-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_rcu_read_lock-ed region") int reject_with_rcu_read_lock(void *ctx) {bpf_rcu_read_lock ();
@@ -147,7 +147,7 @@ __noinline static int throwing_subprog(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_rcu_read_lock-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_rcu_read_lock-ed region") int reject_subprog_with_rcu_read_lock(void *ctx) {bpf_rcu_read_lock ();
tools/testing/selftests/bpf/progs/preempt_lock.c +7 -7
@@ -6,7 +6,7 @@#include "bpf_experimental.h" SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_preempt_disable-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_preempt_disable-ed region") int preempt_lock_missing_1(struct __sk_buff *ctx) {bpf_preempt_disable ();
@@ -14,7 +14,7 @@ int preempt_lock_missing_1(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_preempt_disable-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_preempt_disable-ed region") int preempt_lock_missing_2(struct __sk_buff *ctx) {bpf_preempt_disable ();
@@ -23,7 +23,7 @@ int preempt_lock_missing_2(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_preempt_disable-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_preempt_disable-ed region") int preempt_lock_missing_3(struct __sk_buff *ctx) {bpf_preempt_disable ();
@@ -33,7 +33,7 @@ int preempt_lock_missing_3(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_preempt_disable-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_preempt_disable-ed region") int preempt_lock_missing_3_minus_2(struct __sk_buff *ctx) {bpf_preempt_disable ();
@@ -55,7 +55,7 @@ static __noinline void preempt_enable(void) } SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_preempt_disable-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_preempt_disable-ed region") int preempt_lock_missing_1_subprog(struct __sk_buff *ctx) {preempt_disable ();
@@ -63,7 +63,7 @@ int preempt_lock_missing_1_subprog(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_preempt_disable-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_preempt_disable-ed region") int preempt_lock_missing_2_subprog(struct __sk_buff *ctx) {preempt_disable ();
@@ -72,7 +72,7 @@ int preempt_lock_missing_2_subprog(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_preempt_disable-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_preempt_disable-ed region") int preempt_lock_missing_2_minus_1_subprog(struct __sk_buff *ctx) {preempt_disable ();
tools/testing/selftests/bpf/progs/verifier_spin_lock.c +1 -1
@@ -187,7 +187,7 @@ l0_%=: r6 = r0; \ SEC("cgroup/skb") __description("spin_lock: test6 missing unlock") -__failure __msg("BPF_EXIT instruction cannot be used inside bpf_spin_lock-ed region") +__failure __msg("BPF_EXIT instruction in main prog cannot be used inside bpf_spin_lock-ed region") __failure_unpriv __msg_unpriv("") __naked void spin_lock_test6_missing_unlock(void) {
Signature · 1 line
2.43.5
Re: [PATCH bpf-next v3 5/7] bpf: Improve verifier log for resource leak on exit
Quoted 7 lines · Kumar Kartikeya Dwivedi
The verifier log when leaking resources on
BPF_EXITmay be a bit confusing, as it's a problem only when finally existing from the main prog, not from any of the subprogs. Hence, update the verifier error string and the corresponding selftests matching on it.Suggested-by Eduard Zingerman <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>
[...]
Kumar Kartikeya Dwivedi
maintainerPATCH v3 6/7 selftests/bpf: Expand coverage of preempt tests to sleepable kfunc
In reply to Kumar Kartikeya DwivediFor preemption-related kfuncs, we don't test their interaction with sleepable kfuncs (we do test helpers) even though the verifier has code to protect against such a pattern. Expand coverage of the selftest to include this case.
--- tools/testing/selftests/bpf/progs/preempt_lock.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
tools/testing/selftests/bpf/progs/preempt_lock.c +12 -0
@@ -113,6 +113,18 @@ int preempt_sleepable_helper(void *ctx)return 0 ; } +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +__failure __msg("kernel func bpf_copy_from_user_str is sleepable within non-preemptible region") +int preempt_sleepable_kfunc(void *ctx) +{ +u32 data; + +bpf_preempt_disable (); +bpf_copy_from_user_str (&data,sizeof (data),NULL ,0 ); +bpf_preempt_enable (); +return 0 ; +} + int __noinline preempt_global_subprog(void) {preempt_balance_subprog ();
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v3 7/7 selftests/bpf: Add IRQ save/restore tests
In reply to Kumar Kartikeya DwivediInclude tests that check for rejection in erroneous cases, like unbalanced IRQ-disabled counts, within and across subprogs, invalid IRQ flag state or input to kfuncs, behavior upon overwriting IRQ saved state on stack, interaction with sleepable kfuncs/helpers, global functions, and out of order restore. Include some success scenarios as well to demonstrate usage.
#128/1 irq/irq_save_bad_arg:OK #128/2 irq/irq_restore_bad_arg:OK #128/3 irq/irq_restore_missing_2:OK #128/4 irq/irq_restore_missing_3:OK #128/5 irq/irq_restore_missing_3_minus_2:OK #128/6 irq/irq_restore_missing_1_subprog:OK #128/7 irq/irq_restore_missing_2_subprog:OK #128/8 irq/irq_restore_missing_3_subprog:OK #128/9 irq/irq_restore_missing_3_minus_2_subprog:OK #128/10 irq/irq_balance:OK #128/11 irq/irq_balance_n:OK #128/12 irq/irq_balance_subprog:OK #128/13 irq/irq_global_subprog:OK #128/14 irq/irq_restore_ooo:OK #128/15 irq/irq_restore_ooo_3:OK #128/16 irq/irq_restore_3_subprog:OK #128/17 irq/irq_restore_4_subprog:OK #128/18 irq/irq_restore_ooo_3_subprog:OK #128/19 irq/irq_restore_invalid:OK #128/20 irq/irq_save_invalid:OK #128/21 irq/irq_restore_iter:OK #128/22 irq/irq_save_iter:OK #128/23 irq/irq_flag_overwrite:OK #128/24 irq/irq_flag_overwrite_partial:OK #128/25 irq/irq_sleepable_helper:OK #128/26 irq/irq_sleepable_kfunc:OK #128 irq:OK
Summary: 1/26 PASSED, 0 SKIPPED, 0 FAILED
--- .../selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/irq.c | 397 ++++++++++++++++++ 2 files changed, 399 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/irq.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -98,6 +98,7 @@#include "verifier_xdp_direct_packet_access.skel.h" #include "verifier_bits_iter.skel.h" #include "verifier_lsm.skel.h" +#include "irq.skel.h" #define MAX_ENTRIES 11
@@ -225,6 +226,7 @@ void test_verifier_xdp(void) { RUN(verifier_xdp); }void test_verifier_xdp_direct_packet_access (void ) {RUN (verifier_xdp_direct_packet_access); }void test_verifier_bits_iter (void ) {RUN (verifier_bits_iter); }void test_verifier_lsm (void ) {RUN (verifier_lsm); } +void test_irq (void ) {RUN (irq); }void test_verifier_mtu (void ) {
tools/testing/selftests/bpf/progs/irq.c new file +397 -0
397 lines not on this page · raw message
Signature · 1 line
2.43.5